Some outages are caused by power failures. Others by network issues.
And then there are the ones caused by a vendor who decided—without telling you—to deactivate the literal lifeline of your organization.
That was my Wednesday. Right before Juneteenth.
The Main Line: Disconnected
The District’s main customer service number—the one printed on every bill, posted on the website, and used for water and sewer emergencies—suddenly stopped working.
No dial tone. No ring. Just silence.
The kind of silence that makes an IT admin’s stomach drop.
The Root Cause
After racing through internal checks, system reboots, and a wild ride through AT&T’s Tier 1 to Tier 3 support tiers, we got our answer:
A vendor had accidentally included our main number and its circuit in a service deactivation request.
That’s right. The line didn’t break.
It was intentionally shut off—just not by us.
Apparently, it was part of a list for disconnection. A well-meaning click by someone on the other end turned into a full-blown communications outage for a public utility.
Activating Plan B
We moved quickly.
- Outreach was coordinated
- Messaging was updated
- Staff were notified
The team acted swiftly to ensure that customers still had a way to reach the District during the outage.
No technician was dispatched. No immediate fix was offered.
Just me, a solo IT admin, navigating chaos with a Surface and a lot of patience.
The Vendor Did Step Up
To their credit, once the issue was identified, the vendor responded quickly and gave the situation the support it needed. They worked with urgency to correct the mistake and provided technical assistance — even on a holiday.
Mistakes happen. What matters is how fast they’re fixed — and in this case, they followed through.
Juneteenth Complications
Because outages love holidays, this one hit the day before Juneteenth—a federal holiday when offices close but emergencies don’t.
The line was restored later that day, but because of the timing, there was still work to be done:
- Coordinating public messaging during a holiday
- Updating social media and the District website
- Verifying that critical calls were routing correctly
Even on a day meant to reflect, pause, and honor history—we were troubleshooting, documenting, and pushing out updates.
What We Learned (Again)
- Always confirm what’s being deactivated—especially when vendors are involved
- Assume nothing is safe, even numbers you think are untouchable
- Have your comms and contingency protocols ready before you need them
- Vendor mistakes become your emergencies. Plan accordingly
Final Thoughts
It wasn’t a catastrophic failure.
But it was a reminder: in public service, every detail matters.
A single mistake by a third party can disconnect your entire organization from the people who rely on you.
And if you’re the solo IT guy?
You are the helpdesk, the escalation path, and the damage control team—even during holidays.
We’re back online. Messaging went out. Customers were informed.
The inbox quieted just enough for a deep breath.
So here’s to redundancy, Surface-powered persistence, and the kind of weekends where nothing breaks.
Have a good one. You’ve earned it.
—
Written by your solo IT Admin, protecting infrastructure and sanity—one unexpected outage at a time.