JavaScript Grid with One Million Records | JavaScript UI

July 5, 2013

Some time ago, I got a comment from a user who wanted to use the grid as a log table. He quickly put 10,000 records into the grid without paginating, but found that it got really, really slow. This got me wondering, can it be optimized to handle 10k of records or it is too much for the browser.

Establishing the Baseline

For my initial test I have used w2ui ver 1.2, created a simple grid with 4 columns, populated it with random data and rendered into a 1024x768 container. I have started my testing with recordsPerPage set to 50 and tried 25, 250, 2.5K, 25K, 250K and 1MIL records in the grid. To my surprise the grid performed well. It was responsive and fast, I could go from page to page and perform other grid functions.

It was an unexpected and pleasant surprise. It meant that JavaScript can handle large data sets, which was awesome. As a next step, I have set recordsPerPage to 1,000,000 and repeated my tests. The table below shows my findings:

I have found that if the number of records in the grid becomes more then just a few thousands the grid gets very slow because the rendering speed is directly related to the number of nodes in the DOM. If the number of nodes in the DOM is more then 40-50k (depending on your computer configuration and amount of memory), your browser will crash or will become unresponsive.

So, I decided to set out on a quest to do two things: (1) dynamically create records as user scrolls (2) optimize grid to handle large data sets. After a few weeks of work, the grid was optimized and ready for testing. I have repeated my original tests:

Wow! The results were awesome! Now the grid was capable of handling large data sets that not so long ago only a databas server could. If you are wondering why 2500 records performed better than 250, the answer is very simple: I have defined a parameter in the grid not to use buffered scroll if number of records below 300. So, when there are less then 300 records - all of them are rendered in the grid. If there are more then 300 records, only the ones that are in the view, plus a few items on top and botton for smooth scrolling.

Example of the grid

Below you can find an example of the grid with random 25K records. You can generate a different amount and play with it for yourself. Please note that different browsers can handle different number of records, though pretty much all of them can handle 1MIL of records.

Generate:

Sorting & Searching

The next challenge was to make local sorting and searching fast. In the table below you can see results before optimization:

As it is seen from the data, the sorting performed well (I used Array.sort() for this) and the search was increasingly slow with the increase in the number of records. I found that the biggest issue with the search was the use of eval() to parse nested record sets, which turned out to be slow. I have refactored the code and repeated the tests

Infinite Scroll

Now, rendering, sorting and searching was fast on large data sets, but still was not good enough if the number of records goes over 250K. I did not know any other way to make it faster and I think it hits the performace ceiling just because there is nothing else you can do to optimize it. There is no way to create indexes in JavaScript, but database already have this functionality. So, to make your grid work on large data set you can use Infinite Scroll, which I have also implemented in the grid.

I have created a Postgres database with 1 million records and implemented infinite scroll in the grid, buffering 100 records at a time. All my test have shown that it was never over 0.2 seconds to sort and search through this record set.

Browsers Test

Initially, I have done all my tests in Chrome. Now, I wanted to repeat them on other browsers too. Though Chrome performed better in most categories, the test have shown that other browsers can do a pretty good job with large data sets. I have recorded best of 5 tries for each browser in each category.

- (green - best, yellow - second best)

- (green - best, yellow - second best)

- (green - best, yellow - second best)

The table below shows a winner, where I have given a browser a point if it were the best in the category and half a point if it were second best:

Browser Limits

I have discovered an interested browser limitation - the height of the div has a limit (so is scrollable area in the div). Hence, number of scrollable records has a limit since one record is 25px. I have created a simple page with one div and by trial and error I have discovered the max height of the div:

Different browsers behave differently when you try to create a div with height over the limit. Opera will simply max it up to the limit and ignore anything over it. FireFox will set the height of the div to 0 if height it too big. Safari, will display distorted view if it is too large and Chrome will display a black box at the bottom. I have also discovered that Chrome has a smaller limit of height for the body. For the body, height can be no more then 16,777,201 px.

Important Lessons

During this exercise I have learned several important things:

• Large number of DOM nodes make rendering slow
• JavaScript arrays can handle large data sets
• Looping through large arrays is fast
• Sorting arrays by providing custom function to Array.sort() is fast
• eval() is slow, should not be used in large loops
• To archieve smooth scrolling render a few hidden records on top and bottom outside of the visible area

I think that 1MIL of records for JavaScript is too much, though it is doable. If user has to wait over a second it makes user experience sluggish and unpleasant. But as seen in the tables above any browser can give you a good user experience with 100K of records or less.

User Comments