App Dev · · 4 min read
NetLay: From eight seconds to a tenth
Everything felt fine on my own map of fifty devices, and at a site of eight hundred and fifty it took eight seconds to move the mouse once.
Fifty devices is not a test#
Every map I have ever opened in NetLay is my own, and mine has about fifty devices on it. It is instant, it has always been instant, and I had quietly concluded that performance was not something this application had to think about.
The performance review seeded a site with twenty switches of forty eight ports, around eight hundred and fifty devices, sixty virtual machines on one host and a hundred and fifty subnets, and then measured things.
Dragging a selection of four hundred and eighty devices took just under eight seconds. Per mouse move.
Measure, then look.#
The numbers before anything was touched, with a headless browser timing each mouse movement. Panning the map, 141 milliseconds. Dragging one device, 138. Dragging that big selection, 7,915. Dragging with automatic cable routing switched on, 935.
For reference the floor is about 33 milliseconds, because each step waits for two animation frames. So panning was four times slower than it needed to be and the selection drag was two hundred and forty times slower.
What I like about having the numbers first is that they tell you where not to look. I would have guessed the problem was drawing. It was not drawing.
One update, not four hundred and eighty.#
Dragging a selection called the move this device function once for each selected device. Each of those calls copied the entire layout and told every node on the map that something had changed.
So moving four hundred and eighty devices one pixel did four hundred and eighty full layout copies and four hundred and eighty full redraws, and all but the last were thrown away immediately.
Now the new positions are all worked out first and applied in a single update, and only the switches whose contents actually changed get new objects at all. That one change took the selection drag from eight seconds to about a tenth of a second.
Every node reading the whole map.#
The second problem is the one I would never have found by reading the code, because each individual piece of it is reasonable.
Every device on the map needs to know a few things to draw itself. Where is my host, what VLAN is my port in, am I selected. Each of those questions was answered by scanning the layout. That is around eight hundred and forty nodes each scanning around eight hundred and fifty devices, on every single update.
The fix is an index built once per version of the layout, which answers all of those questions by lookup. It hangs off the layout object itself, so when the layout is replaced, which happens on every change, the old index goes with it and there is nothing to invalidate by hand. Selection became a set in the same way.
With that in place the map items could be memoised properly, and lines reuse their previous point arrays when nothing has moved, so panning now redraws almost nothing.
Routing, but not during a drag.#
Automatic cable routing from post twenty re-ran its grid search for every cable on every step of a drag, which is a lot of searching for a picture that is about to change again in sixteen milliseconds.
During a drag, cables that have not moved keep the routes they already had and the ones that have moved are drawn straight. Let go, and everything is routed properly.
Nobody is studying the exact path of a cable while they are dragging it, and the routing is correct the moment the mouse comes up, which is when anyone would look.
The server had its own problems.#
Ping now queued behind a full round, which with a lot of unreachable devices took twenty eight seconds. It has its own lane now, rounds run sixty four at a time, and results arrive at the map in batches rather than all at the end.
The hourly history roll up deleted two hundred thousand rows one at a time while holding the database lock, which took five and a third seconds and made everything else wait almost five seconds. One statement instead of a loop took it to 1.1 seconds, and other writes now wait about six tenths of a second.
SQLite went into WAL mode, so readers do not wait for the writer, with a fifteen second busy timeout, events pruned after a year, and the layout download gzipped.
After.#
Panning, 55 milliseconds. Dragging a device, 50. Dragging the four hundred and eighty device selection, 96. Dragging with routing on, 63.
All of it close enough to the floor that the remaining time is the browser doing its job rather than my code doing too much of it.
The honest lesson is in the first paragraph of this post. Everything was fine on the only map I ever opened, and a map five times bigger is not five times slower, it is two hundred times slower, because the mistakes multiply by each other rather than adding up.
Next time, the last phase of the review, and the jobs that took five clicks when they should have taken one.