Illinois Water SCADA Reporting Shows Need for Analytical Competence

SCADA hacking

Guest author Sean McBride is the Director of Analysis and Co-founder of Critical Intelligence, a company that provides Cyber Situational Awareness and Threat Intelligence services for Industrial Control System Owner/Operators, Vendors and Government stakeholders.

A pair of investigations, one by the Department of Homeland Security [1], and one by reporters from Wired [2], provided a conclusion to the Curran-Gardner Water Disrict (Illinois) hacking incident. In short, the suspected intrusion from a Russian IP address was legitimate traffic from a SCADA engineer on travel in that country. The log-in did not seem to be related to the failure of a pump under the system’s control.

The story originally broke on November 17, when Joe Weiss, an outspoken ICS security professional, noted having received a report from a government body that described a compromised water SCADA system. Over the following weeks numerous news outlets, including the BBC [3] covered the incident – which is now known to be a false alarm.

Several facets of the incident, its reporting and coverage, show the importance of developing  sound analytical competence.

First, and most obviously, the report, which came from the Illinois Statewide Terrorism and Intelligence Center, was unverified analysis. As a consumer of information, one must accept that when you receive analysis from a third party there may be errors and omissions, lack of expertise, logical fallacies, false assumption, and biases. The old adage still applies: “You can’t believe everything you read in the paper.” Of course, this challenge of discerning truth and error is compounded when information is provided by an “official” government source – which was the case here.

Close examination of the Illinois report however, shows several reasons to call it into question:

  • The report was released on November 10, 2011 and provides extensive details of an event that reportedly occurred on November 8, 2011. That timeline seems unreasonable.
  • The report claimed that the intrusion “is the same method of attack recently used against the Massachusetts Institute of Technology (MIT) server” because “The water district’s attack and the MIT attack both had references to ‘phpMyAdmin’ in the log files of the computer systems.”

This is inaccurate, showing a fundamental misunderstanding and lack of research. The compromised MIT server attacked other systems that exhibited the phpMyAdmin vulnerability– it was not the means of its own compromise. Moreover, it does not seem likely that the SCADA system would be running phpMyAdmin (though not impossible).

  • The report also states “It is believed the hackers had acquired unauthorized access to the software company’s database and retrieved the usernames and passwords of various SCADA systems, including the water district’s system.”

This seems to conflict with the idea that the compromise occurred via phpMyAdmin. One possible way to sync the two named attack vectors is that the SCADA vendor was running phpMyAdmin and the SCADA vendor’s networks were compromised by that vulnerability – but that would probably have required analysis of the SCADA vendor’s logs in addition to the Curran-Gardner logs, and the report itself implies that the vendor’s logs have not been reviewed: “It is unknown at this time the number of SCADA usernames and passwords acquired from the software company’s database and if any additional SCADA systems have been attacked as a result of this theft.”

Second, DHS’s (INL/Battelle) handling of the situation represents a continual squandering of taxpayer resources.

  • “Rules” for government engagement in the private sector seem to have been disregarded.

It is my understanding that ICS-CERT or any other federal agency providing a public-private service can only intervene upon the request of an affected entity. In the Curran-Gardner case, ICS-CERT actively sought to contact the reportedly affected entity: “ICS-CERT would like to thank the Curran-Gardner Public Water District and the SCADA systems integrator (vendor) for their cooperativeness in pulling all available resources in order to conduct a thorough and exhaustive investigation” [1].

While this may seem like the right thing to do, it is, at a minimum, inconsistent with previous statements that incident response requires “voluntary” participation with the government. Imagine that the DHS and FBI come calling, “We heard you had a problem…we are here to investigate. You need our help. Can we help?” This is hardly voluntary.

  • What was the evidence that prompted the intervention/response? A failed pump that helped provide water to 2000 customers. What risk does this represent to the homeland? There was no threat, and very little consequence. Apparently risk assessment is NOT prerequisite to or part of conducting a “thorough and exhaustive investigation.”
  • If Curran-Garnder wanted an investigation, they could have hired qualified parties to help out (in fact, their logs had already been reviewed by a computer support company, that correctly concluded that someone had logged in from Russia).

If DHS continues at this rate, it might want to move into the system integrator business, and to operating water facilities in small towns – just to make sure the Russians don’t get in. The parties managing the Control Systems Security Program (as well as the shoe-in contractors) wouldn’t mind the increased budget.

  • Reporters from Wired ran the story to ground in ways that surpassed DHS efforts (at no public fee).

Third, DHS ignores important issues and hypes the worthless.

  • DHS continued to make an issue of the story by sending emails, releasing documents, and making it a prominent item at industry briefings. There was no actionable intelligence from this effort. Why continue beating the drum?
  • DHS reports fail to mention that Curran-Gardner has experienced SCADA issues for well over a year, undergoing several pump repairs in 2009 and 2010 [4-6], and ultimately receiving an insurance check for a “SCADA failure” in September of 2010 [7].
  • The reporting did not mention the fact that if Curran-Gardner didn’t want folks logging in from Russia, they could have blocked IP addresses from Russia (or any other country).
  • DHS has yet to publicly address the four SCADA systems accessed/trespassed by pr0f – which are real intrusions. Some information about all of these is publicly available.

In short, a little bit of analytical thinking and a few phone calls could have saved unnecessary taxpayer expenditures, the dissemination of misinformation, and the wasting of valuable time. This seems to underscore that as a group of practitioners, we are quick to lend weight to government agencies (such as the Illinois Statewide Terrorism and Intelligence Center and DHS in this case) despite a proven record of analytical shortcomings. Until we all (government, reporters, solution providers, and operators of critical infrastructure) increase our analytical competence we can only expect to believe (and promote) things that aren’t accurate.

[1] http://www.us-cert.gov/control_systems/pdf/ICSB-11-327-01.pdf

[2] http://www.wired.com/threatlevel/2011/11/water-pump-hack-mystery-solved/

[3] http://www.bbc.co.uk/news/technology-15817335

[4] http://www.currangardner.com/Archives/Minutes/2010/20100309minutes.pdf

[5] http://www.currangardner.com/Archives/Minutes/2010/20100608minutes.pdf

[6] http://www.currangardner.com/Archives/Minutes/2009/20090210minutes.pdf

[7] http://www.currangardner.com/Archives/Minutes/2010/20100914minutes.pdf

Image by Chris_Short

5 comments to Illinois Water SCADA Reporting Shows Need for Analytical Competence

  • Sean, you’re the man. This is a terrific article and a much-needed one. I hope you continue writing them.

  • RonF

    Excellent article Sean. This ICS-CERT alert just fuels the fire: http://www.us-cert.gov/control_systems/pdf/ICS-ALERT-11-343-01.pdf While I appreciate that they give credit to those independent researchers using google/shodan to reveal internet connected devices, it just tells me that without those independents, DHS would have been totally in the dark, and only reactionary when a utility contacts them.

    This hasn’t been a good month for DHS/ICS-CERT =\

  • Bryan Owen

    Stuxnet to the Curran-Gardner in just over a year.

    Weather this debacle significantly undermines support for improving ICS practices related to cyber remains to be seen.

    In addition to Sean’s call for compentent analytics perhaps these incidents (including pr0f) will provide impetus for consensus on acceptable ICS remote access patterns.

  • mtoecker

    The one bright ray of sunshine in this debacle is this: A cyber security event was actually considered as the cause of the event.

    Usually, failures like this are investigated down the engineering path only, and don’t include anyone with any cyber security expertise. Additionally, these investigations are often limited by the logging capabilities of the systems in question.

    So, we actually have two successes, mired in this false alarm… First, engineering and IT got together to share information and collaborate on an (albeit incorrect) cause. Second, there was a reasonable amount of logging and monitoring surrounding remote access to the systems, such that they were able to verify source, destination, and the unique ID of the account used.

    All in all, an educational experience, and an example for future collaboration by engineering and IT. Just a few less assumptions next time.

    Did they ever find out what actually killed the pump?

    Mike “Mr. Ray of Sunshine” Toecker
    //My opinions are my own

  • RonF

    “Did they ever find out what actually killed the pump?”

    It appears that as soon as wired debunked the ‘Russian Attacker’, DHS/Ill STIC brushed everything else under the rug. As Sean pointed out, forget the years worth of problems.

Leave a Reply