Sleepless In Oakland

Talk about a really bad day

Talk about a run of bad days and weeks keeping you up 20 out of 24 hours a day and then when you do try to sleep, your brain won’t let you rest. This is the nightmare story of a ransomware attack that shut down a city’s entire network. Police. Fire. City administration. Municipal government. Public works. City employees were instructed to shut down their work computers—no remote connections—and I.T. scrambled to install new software on every computer to stave off the next possible attack.

And if it happens again, you can count on Perry Lamont, emergency medical services coordinator and QA/QI Officer for the Oakland City Fire Department Communication Center, California (USA), taking an extended vacation without voice mail access or a remote connection. Don’t believe that for a second.

The “away from my office” comment was simply what Lamont said in jest five months after the ransomware attack devasted Oakland City services. Lamont is a QA fanatic, particularly in relation to maintaining accreditation. If it happens again—banish that thought—he'd be the mound of melted wax from burning the candle at both ends.

There’s no opportune time for a ransomware attack triggered by a spear phishing email inadvertently opened by a city employee. The attack in Oakland, however, couldn’t have been at a worse time, as far as Lamont was concerned. He’d submitted an application in January to complete their fifth medical accreditation through the International Academies of Emergency Dispatch® (IAED™). Most of the reports had been updated and were ready before the deadline since, by nature, Lamont never waits until the last minute, especially when it comes to ACE. “We take accreditation very seriously,” he said.

But then the ransomware attack happened. The biggest problem for Lamont resided in documentation. All the data necessary to re-accredit was gone and held for ransom, the same as for the other public entities in Oakland. As the QA/QI, Lamont is directly responsible for at least 10 of the Twenty Points of Accreditation that concentrate on quality assurance. Requirements include providing copies of 25 case review audio files with completed Case Evaluation Records (CER) and Incident Performance Reports (IPR), two calls from the one-month period immediately preceding the application, and compliance scores at or above expected minimum performance levels for at least the three months preceding the application.

That’s just a tip for the Twenty Points. There’s also policies and procedures to update, processes to describe, and details such as EMD certification numbers to verify (add and delete).

All of it was gone. It was either on the dark web or destroyed. But ACE would not be sacrificed.

“It hit us so hard; it was a nightmare,” Lamont said, and one that demanded immediate attention and laser-sharp focus to resolve. In the back of his mind, Lamont knew the Academy and the ACE reviewer overseeing Oakland’s application—Christoph Högl, EFD-Q™, EMD-Q®, 144 Notruf Niederösterreich, St. Pölten, Austria—would grant a grace period due to the “unusual” circumstances. But Lamont wasn’t going to take that route. The ACE deadline was a target, and he wasn’t going to miss the bull’s-eye.

Oakland Fire Communications Manager David Ebarle confirmed the seriousness of their ACE plight. “ACE makes a huge difference,” he said. “It confirms to our constituents they can trust they’re getting the highest level of service in emergency dispatch. We are very careful to never lose it.”

Ebarle had been there a year when the attack occurred. He deferred to Lamont in explaining what they were up against and what it took to get the ACE ball rolling again. “Perry has QA down to a science,” Ebarle said. He was the “rock star” in meeting the ransomware head on.

Lamont's personal life was non-existent for the ensuing two months. He didn’t take time off. He worked nights, weekends, and holidays. He was exhausted at times, but he couldn’t stop. “I kept thinking I need to get this done and how was I going to get this done.”

He went back to old-school methods of using the telephone to contact co-workers and paper and pen notetaking. He tracked data in reverse, from the end of the period to the beginning. He entered data manually into his home computer and transferred it into his work computer. He revised policies and procedures for a second time and called each EMD to verify their certification numbers. “I literally went through things with a fine-tooth comb,” he said. “I don’t do anything halfway.”

Lamont walked on stage at NAVIGATOR during the opening ceremonies (April 25) to accept the ACE certificate from Christof Chwojka, Accreditation Board Chair, and Kim Rigden, Associate Director of Accreditation. Lamont was elated. “It was a great feeling knowing what the city had gone through.”

The attack was the most challenging experience in Lamont’s 30-year career in emergency services. There was no instruction book to follow and since everyone in Oakland public services was knee-deep in the same quagmire, he hesitated to ask for assistance. “We were all, in every department, trying to get through this.”

He gives hats off to Ebarle and the Academy, namely Chwojka, Ridgen, and Högl. “They were with me the whole way,” he said. Although they were helpless to remedy the ransomware blues, they offered him a world of support, never underestimating his tenacity and his drive to make it through hell and high water to get it done.

Their commitment was recognized beyond those directly involved. Ivan Whitaker, Vice President, MD Ally, IAED ED-Q Curriculum Board Member, said their dedication to excellence and accreditation was apparent during a recent visit to Oakland dispatch. “Their unwavering professionalism and precise adherence to protocols were evident, and the citizens of Oakland should take immense pride in their exceptional service,” he said.

There were lessons learned, Lamont said. Backup is a priority and the more data backed up the better. He uses full backup, incremental backup, and differential backup. He stores data on thumb drives. He said the malicious breech was a wake-up call. Who knew something like this could take down an entire city?

Lamont takes nothing for granted, as far as data protection goes. The chance an attack could happen again is definitely stored in the back of his mind. “No matter the upgrade, there’s always someone in the dark web finding new ways to get back in.”

The Fire Dispatch Center (FDC) receives around 60,000 emergency calls for service annually, and most calls involve medical emergencies. There are 18 dispatchers and five dispatcher supervisors, QA/QI (Lamont), and management (Ebarle).