An access point at a client site dropped offline. Not flapping, not degraded. Gone. It stayed gone for eleven days, through two work sessions and one Sunday that mattered, while coverage in that corner of the building quietly got worse and nobody could say why.
When we finally closed it out, the fix took about ninety seconds and zero dollars in parts. The hardware was fine. The cable was fine. The run in the wall was fine. What failed was a line of paperwork that never got written, and I think that failure is worth more to you than any product recommendation I could publish.
Everything that was not broken
When an AP goes dark, the suspect list writes itself: the AP died, the switch port died, the PoE budget ran out, or the cable run is damaged. All reasonable theories. All wrong here, and each one costs real money if you act on it before you diagnose. A replacement AP is a few hundred dollars. A re-pulled cable run is several hundred more. An eleven-day mystery invites both purchases at once.
The five-minute cable verdict. Unplug the suspect run from its device and plug it into a laptop. Read the negotiated link speed. Gigabit means the run is good, because gigabit requires all four pairs to be healthy. A 100 Mbps sync means damaged pairs, because 100 Mbps only needs two, which is exactly why a wounded run still appears to work. No link at all means a dead run, or nothing behind the wall plate. This one test separates a termination problem from a hardware problem before you spend a dollar.
What the records said, and why they lied
The site documentation we inherited said this AP lived on a specific switch port. The switch's live table said that port belonged to an entirely different room. Both could not be true, and the live table wins every time, because the live table is reporting what is physically negotiating electrons right now and the binder is reporting what somebody believed months ago.
This was not the first time inherited records lied to us at this site. Across the engagement, prior documentation had one access point misnamed, two that did not appear in any record at all, and one declared dead that was alive the whole time. The lesson got promoted to a standing rule: the MAC address printed on the physical hardware is ground truth. Names, spreadsheets, and port maps are claims. The label is evidence.
What the logs actually said
The controller kept a precise record of the moment the AP disconnected. That timestamp landed squarely inside a documented work session eleven days earlier, a morning of heavy physical work in the same rack. Two days after that, a planned rack consolidation moved and re-dressed most of the cabling in the area, and the loose end got buried behind clean cable management.
Reconstruct it and the story is mundane. A cable was unplugged during legitimate work, for a legitimate reason, by someone doing their job. It never got plugged back in, and the unplug never got written down. Every day after that, the outage looked more like a hardware failure and less like what it was.
The ninety-second fix
Once the timeline pointed at the rack instead of the hardware, the fix was walking to the switch with the run in one hand and landing it on a port. Gigabit sync. Both radios up. And for the first time in the engagement's recorded history, every single device on that network showed online at once. Nothing was purchased. Nothing was replaced. Nothing had ever been broken.
What actually failed
I spent my career on submarines, where physical state changes are controlled by tag-out discipline. Nothing gets unplugged, de-energized, or valved shut without a tag on the equipment and a line in the log, because underway, an unexplained state change is not an inconvenience. It is a casualty investigation.
Civilian networks skip that paperwork because the cost is invisible at the moment you skip it. Unplugging a cable without writing it down costs nothing today. The bill shows up later, as an eleven-day outage nobody can explain, a replacement AP nobody needed, or a re-pulled cable run that was never damaged. The failure mode is not technical. It is a state change without a record, and no amount of hardware spend prevents it.
The four rules we adopted that afternoon
- If it gets unplugged and not re-plugged, it gets flagged. A tag on the cable or a line in the day's notes before anyone goes home. No exceptions for "I'll remember."
- Trust the MAC label, never the inherited name. Physical verification beats every database, spreadsheet, and well-meaning predecessor.
- The port map is a living document. It gets updated at the moment of the change, not at the end of the project. A port map updated later is a port map that is wrong now.
- Diagnose by link speed before replacing anything. The laptop test above is free and takes five minutes. Start there, every time.
Why this matters if you run a small organization
A business with an IT department can absorb an eleven-day mystery. A twenty-person company, a nonprofit, or a church running a Sunday livestream cannot. You do not have the slack, and you should not need it, because the countermeasure costs almost nothing. Documentation is the cheapest component in the building. It is also the only component that turns the next mystery into a ninety-second fix.
This is the actual product of a disciplined IT practice. Not the router brand, not the WiFi generation. The binder: an accurate port map, an asset register verified against physical labels, and a change log that records what changed hands and when. Every engagement I run leaves the client holding that binder, because the day I am unreachable, the binder is the engineer.
If your network has a mystery like this one, or you cannot say with confidence what is plugged into what, that is a solvable problem and solving it is not expensive. Reach me at travis@buteranet.com, or see how the managed practice works at msp.buteranet.com.
Travis D. Butera