This blog runs on a single laptop from 2012. A Sony VAIO SVS15125CKB with one 5400 rpm hard disk. That machine holds the web server, the statistics API, and the database.
In early August it started disappearing several times a day. It came back on its own after about six minutes, and for those six minutes the site was simply dead.
The conclusion first: the cause was a fault in the wireless card's firmware. And what I fixed was not the firmware. The firmware fault still occurs, roughly every four days, and the server has not gone down once in 25 days. This is a record of what happened in between.
The server that came back after six minutes
The symptom was vague. By the time I noticed the site was unreachable, it had already recovered. Pulling the reboot history showed a pattern.
Wed Jul 29 10:15 (up 7 days 11 hours)
Tue Aug 4 21:13 (up 1 day 3 minutes)
Wed Aug 5 21:16 (20 minutes)
Wed Aug 5 21:37
Thu Aug 6 14:46 ← here
Thu Aug 6 14:50
The 12 July boot had lasted 24 days. From the end of July the intervals shrank to a day, then twenty minutes. Something was getting worse.
Two lines in the log
The kernel log had the answer, in a pair of lines.
iwlwifi 0000:02:00.0: Microcode SW error detected. Restarting 0x2000000.
iwlwifi 0000:02:00.0: Loaded firmware version: 18.168.6.1 6000g2b-6.ucode
iwlwifi 0000:02:00.0: Start IWL Error Log Dump:
iwlwifi 0000:02:00.0: 0x0000198A | ADVANCED_SYSASSERT
The wireless card is an Intel Centrino Advanced-N 6235, the part that came with a 2012 machine.
The driver is iwldvm and the firmware is 18.168.6.1. No newer firmware is coming.
ADVANCED_SYSASSERT is the firmware running inside the card deciding for itself that "from
here on I am not in a state I recognise" and stopping. That alone does not kill the computer;
the driver resets the card and reattaches it.
The problem was what came next. During the reset, the kernel took a general protection fault in
ieee80211_free_chanctx(). That is the code cleaning up the channel context, the record of
which channel is in use. Firmware dying and the kernel dying are separate events, and these two
were overlapping.
It was moving every 37 seconds
Why the channel cleanup code specifically? I pulled the wireless association records in time order.
14:53:01 disconnect from AP ...:FB for new auth to ...:FA
14:53:37 disconnect from AP ...:FA for new auth to ...:FB
14:54:12 disconnect from AP ...:FB for new auth to ...:FA
Thirty-six seconds, thirty-five, thirty-seven. This laptop was shuttling between two access points without pause. Measuring all 19 transitions gave a median of 37 seconds and a minimum of 35.
The two access point addresses differ only in the final digit. It was not moving between two mesh routers but between the 2.4 GHz and 5 GHz radios of one router. Band steering, where the router nudges a client toward the band it prefers, was enabled.
Every move changes channel, and every channel change tears down that channel context and builds a new one. Once every 37 seconds. When the firmware fault happens to land in that moment, the kernel dies.
The access point addresses in this post are not the real values. The MAC address of a wireless access point can be traced back to a location through public collection databases, so they have been changed, keeping only the difference in the final digit. All that matters for what follows is that the two addresses are consecutive.
Fixing the band on the router made it worse
By this point the fix looks obvious. Go into the router app and pin this device to 2.4 GHz. So I did.
Wrong. It got worse.
New lines appeared in the log.
send auth to ...:FA (try 1/3)
send auth to ...:FA (try 2/3)
send auth to ...:FA (try 3/3)
And the association responses came back carrying status=37. Looking that number up in the
kernel headers:
WLAN_STATUS_REQUEST_DECLINED = 37,Declined. A router can only refuse, saying "go over there"; it cannot stop the laptop from trying to move. So what had been "one clean transition" became "drop → attempt → refusal → retry → timeout → reconnect". The channel context churned more, not less.
The laptop decides whether to move
This is where I had it wrong.
In Wi-Fi, the device connecting is what decides which access point to attach to. What the router does is broadcast, and accept or refuse requests that arrive. Band steering is a nudge, not a command. Refusing and making it inconvenient usually persuades a device to go elsewhere, but the decision remains the device's.
So ending this meant changing the laptop's mind. No amount of router configuration helps while the laptop keeps looking around.
Pinning the access point on the laptop
NetworkManager on Linux has a setting on a connection profile that specifies the access point address directly. Fill it in and no other address is even considered a candidate.
I added one more thing. The profile with the pinned address got a high priority, and a profile with the address left blank was kept alongside at low priority. If the pinned access point disappears, because the router is replaced or channels are reassigned, the low-priority profile takes over and attaches to whatever is there. Pinning alone would take the server offline the day that access point stops existing.
This system has its network configuration managed by netplan, so the file that actually operates
lives in a temporary area that disappears on reboot. The originals are YAML under
/etc/netplan/, and the access point address is a field netplan knows natively, so it is stored
as-is. The priority is a field netplan does not know but passes through wholesale, so in the end
both survived a reboot.
Applied at 15:15 on 6 August.
What I found 25 days later
Today I pulled the logs again from the same machine. It has been up since 14:50 on 6 August, 25 days.
Access point transitions: the last one was at 15:13. After the pin was applied at 15:15, zero in 25 days. That stopped completely.
The firmware faults did not stop.
Aug 6 15:05 Aug 18 09:05 Aug 23 10:13
Aug 7 19:54 Aug 19 15:54 Aug 24 07:25
Aug 9 13:55 Aug 22 19:32 Aug 24 18:19
Aug 12 17:43 Aug 22 22:59 Aug 25 20:41
Aug 18 02:00 Aug 22 23:07 Aug 30 16:17
Fifteen in 25 days, more often than every four days. On the last boot before the fix there were three in 17 hours, so per day it fell from 4.2 to 0.6. Not moving between access points appears to stress the firmware less. But it did not reach zero.
Instead, all fifteen passed without incident. The fault occurs, the driver resets the card, it reattaches to the same access point a few seconds later, and that is the end of it. Reboots: zero.
Put together: this outage had two causes and only fired when they overlapped. Leaving the firmware fault in place and removing the thing it overlapped with was enough to stop the service going down. That you can prevent an incident without fixing the root cause is something I had not felt directly before.
What I did not fix
Recorded honestly.
- The firmware fault itself is untouched. This card's firmware is the final version and nothing further is coming. The next thing to try is a driver option reducing interference between Bluetooth and the 2.4 GHz band, which I have not applied. This machine uses a Bluetooth keyboard, and the band I pinned happens to be 2.4 GHz.
- Why 2.4 GHz rather than 5 GHz was because the signal was more stable at the time, not because I measured both bands side by side over several days. It is a weakly grounded choice.
- I have not tried a different kernel. This machine has stayed on the same one, so whether the same thing happens elsewhere is unknown.
- There is also an option making the driver give up on resetting after a firmware fault. I decided against it. Without the reset the kernel survives, and the wireless stays dead with no automatic reboot to recover it. A six-minute outage becomes an indefinite one.
- Twenty-five days is short for a verdict. Even before the fix, one boot lasted 24 days. The difference is that it was moving between access points constantly then and is not now, so I am not treating the two 25-day periods as equivalent.
Running something on an old machine leaves you with things like this. In the next part I will count how many people a day tried to get in through the ports this laptop had open.