Edge of Space, Part Seven: The Bug That Only Happened in Release

Every developer has a bug they tell people about. This is mine.

The one player who mattered#

It arrived as a support ticket from the most important player in the game at the time, which was me. Switching between my two planets in the browser froze Edge of Space solid. Not slow, not a stutter you could live with. The tab stopped responding and stayed that way until I killed it, and it did it every single time I swapped.

So I did the obvious thing and tried to reproduce it properly. In the editor it worked perfectly. The desktop build was perfect. The browser build against a server running on my own machine was perfect, fifty planet swaps in a row without so much as a hiccup. The only place it went wrong was the one place I could not attach a debugger to it, which was the real web build, talking to the real server, logged in as the real me.

Making the bug come to me.#

If you can't go to where the bug lives then you have to build something that goes there for you.

What I ended up with was a small Python script that launched a headless browser, loaded the actual production build with my login details baked into the page arguments, and hooked itself into the browser's debugging protocol. From there it did two jobs. Every couple of seconds it sent a tiny "are you still alive" question to the page, and every two and a half seconds it swapped between my planets. If the answers stopped coming back, the main thread had frozen, and I would know exactly when it happened and what had just occurred.

The first run against production froze inside thirty seconds. I have rarely been so pleased to break something.

Then I could not do it again. The next run was fine, and so was the one after that. I cloned my account's planets onto a test account so I could hammer them without wrecking my own empire, and that was fine. I copied the same data onto the development server and that was fine as well. Same build, same account, same coordinates, and the freeze had simply evaporated. I went to bed that night having caught it twice but did not understood it, not at all.

The clue was in what I wasn't testing.#

It came back the next day, and this time I spotted the pattern I had been staring straight past for two days.

Every test account I had been using sat idle with nothing in its build queue, because that is what test accounts do. My own account always had something under construction, because I actually play the game. And the freeze only ever happened when there was a construction site standing on the planet at the moment I swapped away from it.

Once I could make it happen on demand, the editor gave the whole thing away in a single line.

Infinite loop detected. Check set_loops() description.

The blinking work light on top of a construction scaffold is a looping animation. That animation was being created on the planet view, which lives for a long time, but it was pointed at a node that gets destroyed whenever the world rebuilds itself. A looping tween whose targets have all been freed advances no time at all on each step, so it never reaches the end, so it loops again. Forever, and on the main thread.

The editor notices that and stops it with an error, which is why I never saw it there. Release builds, and the browser is a release build, have no such check. They just go round. The main thread never comes back, the server's send buffer fills up behind it, and about eleven seconds later the connection dies too, which is why it felt like the entire game falling over rather than one small animation misbehaving.

The fix was one line. The tween now belongs to the light itself, so it dies when the light does. It has not happened since.

What I actually learned#

The bug was trivial. The hunt was not, and everything worth keeping is in the hunt.

The gap between a debug build and a release build is a place where bugs live. I knew that in the way you know things you have read. I know it properly now. Anything that only shows up in release has earned a reproduction rig rather than a shrug and another attempt in the editor.

Reproduce with the real thing, not something near it. The headless browser driving the genuine production build with a copy of my genuine account was the only setup that ever showed me the freeze. Every sensible, convenient, close enough substitute I tried hid it completely, and each one cost me an evening.

The state that triggers a bug is usually state your tests never have. Test accounts are tidy because nobody plays them. Real players are not tidy. My empire always had a half built something sat on it, and that half built something turned out to be the entire story.

There is a quieter lesson in there too. I shipped two rounds of instrumentation before I ever found the cause, timing on the planet refresh and a watchdog that names any frame taking longer than a second, both shouting into the browser console. Neither of them caught this one. Both are still in the game, waiting for the next bug that only ever happens to one person.

Next time, the part I had been putting off for three months, letting actual human beings into the Universe.

You can play Edge of Space for yourself at eos.simontasker.co.uk.

EOSOGameGodot