Por Jaijongkit, Tim Drugan and John Herrick contributed reporting to this story.
Update: As of 1:20 p.m. on July 19, all City of Boulder utility billing and online permitting were back online.
A software update from CrowdStrike, a cybersecurity firm serving various industries, appears to have triggered a global IT outage, causing machines running Microsoft Windows to crash and impacting numerous businesses and government services, including those in Boulder.
The City of Boulder, which employs the equivalent of 1,540 full-time people, issued an employee alert on Friday morning stating that city computers were affected by the antivirus software update. It said it was prioritizing first responders’ computers at police headquarters and establishing a separate triage team for anyone with a non-working computer.
“City technical staff are working as quickly as possible to reset the impacted computers,” the city said in a news release. Some city services, such as utility billing and online permitting, won’t be available until restored, it said.
“Our emergency services are not impacted. 911 and Dispatch are working,” Dionne Waugh, spokesperson for the Boulder Police Department, told Boulder Reporting Lab. “Our other services, like in-car computers, are currently being worked on.” Fire response services in the county appeared mostly unaffected, while the Boulder wildland fire division was impacted, it said, though it did not provide details.
Sarah Huntley, city spokesperson, told Boulder Reporting Lab that systems necessary for public health, such as water quality treatment facilities, were not affected. Many of the city’s computers, however, were inoperable. The outage was first noticed at 3 a.m. “We are working as quickly as we can to get every computer back online,” she said.
Huntley explained that this process required every impacted computer to be manually reset by city IT staff, who were working on the issue since the early morning hours. “We appreciate the public’s patience while we work through this,” Huntley said. She expected the repair process to take all day.
Hospitals in the area experienced different levels of impact. While both Foothills Hospital in Boulder and AdventHealth Avista in Louisville were affected, Lafayette’s hospital appeared to have escaped unscathed.
“We have not been affected by it at all, thank goodness,” said Michelle McCarty, emergency management communication specialist at Good Samaritan Medical Center in Lafayette.
Staff at Foothills told Boulder Reporting Lab that while they were affected, the hospital was still functioning and the outage was nothing “that would be concerning to patients.” Staff at AdventHealth said the hospital was directly affected with leadership “basically held up in a conference room.” Testing on emergency systems like fire alarms was in progress at about 9:30 a.m.
The outage is being called potentially the largest IT outage of its kind in history. In an update Friday morning, CrowdStrike said it had deployed a fix.
“This is not a security incident or cyberattack,” it said. “The issue has been identified, isolated, and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.”