NetLay: Remembering what happened, and asking the network how it got there

The map only knew about now#

A green dot tells you a device is answering this minute. It cannot answer either of the two questions anybody actually asks.

Was it down last night, and how often does that access point drop?

So NetLay needed a memory, and the constraint was that it had to fit inside what is already there. One process, one SQLite file, no separate time-series database to install and look after, because the moment this thing needs a second service to run, it stops being something you can unzip on a laptop.

Two tables and a fold.#

Raw samples are kept for twenty six hours. That is deliberately just over a day, so you can always see last night in full detail.

Every twenty polls those raw samples get folded into hourly rows holding the number of samples, how many were up, and the minimum, average and maximum round trip. Those are kept for ninety days. An hour of a device's life becomes one small row, and three months of history stays comfortably inside a file you can copy.

Separately from both, up and down transitions are stored as events in their own right. Not derived at query time, written down when they happen, because an event is a fact and recomputing facts from samples is how you end up with history that changes depending on when you ask.

Two bugs from that work are worth passing on. Summing a boolean column needs an explicit cast to an integer or you get an answer that is technically correct and completely useless. And SQLite hands datetimes back without timezone information, so everything now gets normalised to UTC on the way in and on the way out, after an afternoon of wondering why an outage appeared to last minus one hour.

There is one more piece of defensiveness in there. When the pinger restarts, or a device's address changes, it has no idea what state that device was in a minute ago, so it falls back to the last stored event rather than assuming. Otherwise every restart invents a transition that never happened.

Showing it without a dashboard.#

The inspector for a device now has a small sparkline, raw samples and hourly averages merged into one line, with red bars where it was down. Underneath it, uptime over the last day, week and month.

There is an events dialog as well, grouped by day with a time range picker and a count of outages, and clicking one jumps to the device on the map. That is the view I open on a Monday morning.

Alerts that do not nag.#

Alerting is easy to build and hard to make bearable. The rule I settled on is that NetLay tells you a thing once.

A device down for five minutes produces one message. Its recovery produces one message. A device flapping, by default six state changes in an hour, produces at most one message an hour saying it is flapping, because the entire problem with a flapping device is that it generates a hundred notifications and teaches everyone to mute the channel.

It is off until an admin turns it on. Webhooks send a payload carrying both of the field names the common chat tools expect, so the same message works in Slack, Teams or Discord without three code paths, and there is SMTP for email with a save and send test button, because nobody should discover their alert configuration is wrong during an outage.

Devices with monitoring switched off are never pinged, so they never alert, which is the behaviour you want from the pause switch I added a couple of posts back.

Asking the network how it actually gets there.#

The last piece came from this note.

"I love the idea of the history, could we perhap do something with tracert to capture the jumps and devices through the network also."

So the server can run a traceroute at a device, numeric only, twenty hops maximum, and parse what comes back into a list. The useful part is what happens next: each hop address is matched against the devices and switch management addresses already in your map, and the ones it recognises are highlighted using the same glow as the path trace.

That gives you two routes on one picture. The one you documented, and the one the operating system says it is actually taking. When those disagree, you have learned something, and it is usually that there is a router in the building nobody put on the map.

The honest limits are stated in the app. It runs from the server, so it is the server's route and not yours, and a hop that matches nothing is not an error, it is simply a device you have not documented yet. The last few traces per device are kept so you can compare.

Next time, teaching the app to read a switch configuration, so you can stop typing all of this in by hand.

NetLayNetworking