Lies, Damned Lies and Statistics – Part 2


Part 1 covered the need to pull and publish more useful information from the gathered ICS incident and vulnerability data. Part 2 covers “Are the numbers intentionally misleading?

245 Incidents Reported To ICS-CERT in 2014 Means What?

The big statistic picked up by the press from the front page of the latest ICS-CERT Monitor is in 2014 ICS-CERT “received and responded to 245 incidents reported by asset owners and industry partners.” This number is meaningless, but it gets the lead from DHS/ICS-CERT/INL.

In fact, this number is actually slightly less than the number of incidents reported in 2013. If the gross number has meaning, then we should saying we are succeeding in reducing ICS attacks.

Or perhaps it is evidence that asset owners, vendors and researchers had bad experiences and are choosing to no longer work with ICS-CERT.

The number of vulnerabilities handled by ICS-CERT was also down from 187 in 2013 to 159 in 2014. ICS-CERT’s method of counting “vulnerabilities” is so broken and wrong that the gross numbers have little value beyond PR for ICS-CERT. For example, some vulnerabilities in ICS common components are counted individually for each product that integrated the component. Others are counted as one vulnerability. And still others are completely ignored by ICS-CERT leading to 100’s of vulnerable products being counted as zero.

Looked at another way, when an ICS application distributed from an ICS vendor contains Havex, Conficker being found on an ICS computer, and an Internet scan of a building automation controller are each counted as one incident, the gross number of incidents has little value.

Everything ICS is “Sophisticated”

The most misleading number in the statistics from the ICS-CERT Monitor is 55%. “Of the total number of incidents reported to ICS-CERT, roughly 55 percent involved advanced persistent threats (APT) or sophisticated actors.” This is despite “In many cases, the threat actors were unknown due to a lack of attributional data.” So this means almost every incident reported to ICS-CERT is considered APT or sophisticated actor.

ICS-CERT needs to publish the criteria of what makes one a “sophisticated actor”, but based on earlier Alerts and Advisories it is a low bar.

Busy Work

The last set of numbers has Idaho National Labs and DHS/ICS-CERT increasingly competing with industry for some undefined reason. ICS-CERT performed 37 onsite assessments in the six months from Sept 14 to Feb 15. Why?

Why 37? Why not 10 or 100?

What is anyone to take from reporting this number and the sectors the assessments took place in?

Where are the statistics showing progress from this work? And what is the progress they were hoping to achieve?

Why is DHS/ICS-CERT doing this work? Why are they doing it for free for critical infrastructure companies?

There are a number of companies, including Digital Bond, that do onsite assessments. There are also companies that do quality ICSsec incident response and ICSsec training.

This work and related reported numbers feign useful activity and avoid the reality that DHS/ICS-CERT is not taking the leadership role in providing technical expertise to help in developing secure ICS protocols and standards, accurately informing government and industry, analyzing ICS attack code and performing the role that ICS-CERT was created to do.

Lies, Damned Lies and Statistics

“There are three kinds of lies: lies, damned lies, and statistics.” Mark Twain (purportedly quoting Benjamin Disraeli)

The latest edition of the ICS Monitor, last week’s USA Today articles and the reemergence of Joe Weiss’s secret database warrant a hard look at the numbers coming out. Specifically two questions:

  1. Are the numbers intentionally misleading?
  2. Are there additional insights that should be pulled from collected data?

The second question is more interesting and involves less speculation on the motives. Consider Joe Weiss’s database of 400 Cyber Incidents (NIST Definition). No doubt there was a lot of labor collecting and documenting those Cyber Incidents, and there are trust issues that probably limit the information that can be distributed. That said, the number 400 is of little or no help.

Start to break them down by reason for incident (attacker, mistake, malware, etc.), category of impact, qualitative measure of impacts across different categories (cost, loss of life, environmental, etc.), and other divisions and the information begins to be of value. Even then we have to be aware of collection bias. Joe’s background in the electric sector would likely lead to an overweighting of Cyber Incidents in that sector, but looking at each sector as a separate population could provide some helpful information.

ICS-CERT is the champion of intentionally misleading statistics, see ICS-CERT Monthly Monitor for prime examples, but I’ll cover that in part 2 of this article. Let’s focus now on how ICS-CERT could provide useful numbers from the data they have.

  • ICS-CERT provides data on the number of vulnerabilities reported. In 2014, they just provided a raw number of 159 and a trend line. Here would be some useful info, especially if they were tracking this over years.
    • How many of the vendors had a process to deal with the reported vulnerability? This could be broken down further to include a PGP key, security@ email, CERT and other categories.
    • What percentage of the vulnerabilities were verified fixed by ICS-CERT, 3rd party, or the disclosing party?
    • A distribution of the timeline from disclosure to fix … especially now that Google has set the bar at 90 days.
    • Severity distribution to US critical infrastructure (since it is US ICS-CERT)
    • … you probably have already thought of more
  • ICS-CERT provided data on “incidents”. It is very hard to hold off on misleading and just plain wrong, but here are some examples of useful data to pull from the 245 incidents in 2014.
    • Impact, similar to the discussion on the Weiss database
    • Identification and traits of what they called APT/Sophisticated Actors. Somehow they determined 55% fell in this category. What was the distribution of characteristics that led to the 55% being called APT?
    • Discovery, a distribution of how the attackers were discovered, perhaps broken down by type of attack or APT/non-APT in their parlance.
    • Most common attack trees.
    • many, many more

If you really have hundreds of incidents that are not simply corporate network incident noise on big companies that run ICS, then there is a lot of useful information there.

Coming Soon: Part 2 – The Fine Print and Methodology

  • CyberLens

S4x15 Video – Ginter on Embedding Malware in ICS Protocols

Andrew Ginter of Waterfall Security Solutions speaks on Embedding Malware in ICS Protocols. His conclusion is this is harder than one thinks. The easier solution might be to use the SQL server, web server, ftp server, or other commonly exploited protocols that ICS applications integrate.

Fair warning – the second half of the session gets a bit commercial on his/Waterfall’s view on why unidirectional security solves ICS security challenges.

ICSage Video: Eireann Leverett on Catastronomics

Eireann Leverett of the University of Cambridge Centre for Risk Studies looks at control system related catastrophe scenarios and the economic impact of these scenarios with an eye towards how insurance and reinsurance policies will be written and priced.

Admittedly critical infrastructure cyber security is a new topic in an insurance industry that has been around hundreds of years. Eireann points out that insuring against malicious attacks is not new to the insurance company. They insured against piracy on the seas.

The session provides some relevant macro economics in easy to understand language and graphs, and Eireann admits “we’re inventing rough metrics in a land of no metrics”.

His initial efforts are related to an important cyber incident that could impact the US, UK and European bulk electric system. The % loss of GDP due to an incident sounds like a good measure if it can be credibly calculated.

The Q&A in this session was particularly good, which is understandable since there are more questions than answers at this time. It’s a fertile field for those looking for an important economic problem.

For what it’s worth … this was my 18-month old daughter’s favorite session.

Unsolicited Response Podcast: SANS ICS 410 Course & GICSP

Episode 2015:2 SANS ICS Security Training and Certification

SANS provided four individuals for our Unsolicited Response podcast on the 5-day ICS 410: ICS/SCADA Security Essentials training course and the related Global Industrial Cyber Security Professional (GICSP) certification.

  • Scott Cassity, Managing Director of GIAC
  • Mike Assante, SANS Lead for ICS/SCADA security training
  • Justin Searle, SANS Instructor and major course content creator
  • Graham Speake, SANS Instructor and participant in GICSP creation

In the hour long discussion we cover:

  • Why SANS developed an ICS certification and training course
  • The difficulties and benefits of having a mix of IT Security and OT students in the class
  • How the course and certification were created and how they are structured
  • Early feedback on the course and certification along with likely changes and possible future courses (2-day course) and certifications
  • The number of students trained and the number of certified GICSP
  • Why SANS courses are so expensive relative to other courses
  • How should a potential employer view an individual who has the GICSP certification

I also gave the SANS team a chance to answer the criticism that this is an IT Security course from an IT security organization.

I appreciate SANS providing so much time and resources to the podcast. I think there is a fair argument on how the SANS course rates in comparison to the competition, and it might depend on the attendee profile and goal of taking the course. The one thing that SANS has going for it is they know how to scale up to train thousands of students, and this is needed in the ICS security space.

Related Links:

I’m committed to a minimum of 20 podcasts in 2015; this is episode 2. We will wait until five episodes are recorded before bringing on podcast sponsors, but let us know if you are interested in sponsoring Unsolicited Response.

Subscribe to the Unsolicited Response Podcast in iTunes.

Save The Date: S4xJapan is Nov 5-6 in Tokyo

HankoDigital Bond is pleased to announce the 2nd edition of S4xJapan will be held on November 5 – 6 in Tokyo.

The event will be in the Mori Building, Roppongi Hills. The Academy Hills facilities on the 49th floor were perfect for the event last year. The room where the sessions are held have desks/power/Internet for all with a tiered seating so there isn’t a bad seat in the venue.

Thursday, November 5th will be Operations Technology Day (OTDay) where we will focus on real world examples of operating and securing critical control systems implementations.

Friday, November 6th will be the S4 Technical Sessions. You will hear the latest offensive and defensive ICS security research and technicals in technical detail.

Our goal is to have the sessions in English and half in Japan, with simultaneous translation as appropriate. You can see some of last years sessions our video site, and we will be posting the remainder over the next two months.

We also will have a social event after the Thursday sessions in the Academy Hills library. Last year it was food, drink and the Kaspersky ICS Simulation Game. We are planning something fun for this year as well.

The Call For Presentations will be released just after Golden Week, so be thinking about what research you will want to submit to be a part of the event.

Tool Release – CANBus Protector

beagle-canbusContinuing in the line of CANBus research and tools release I’d like to announce some quick work on a proof-of-concept CANBus IPS called, unoriginally, the CANBus Protector. I took some time to work on defense of CAN after conducting a lot of vulnerability research in recent weeks.

The trouble with trying to defend insecure by design protocols is that you can’t. CANBus is a protocol that cannot be done correctly….ever. I don’t think I’m being unreasonable in stating so. When considering defensive security methods there really is only one design that makes any sense.

CANBus protector is a system built on two separate pieces of hardware that use one-way communication to get information out of the “trusted” vehicle network. The only way I could see protecting the bus was to create a “server” which would sit on the actual CAN and perform the queries for vehicle information. This is done through standard set of OBDII queries made by the server (e.g. Vehicle Speed, RPM, VIN, etc). Future expansion of the project would include more queries and manufacturer specific information. The server then publishes that information via one-way communication to a “client”. The client sets up an entirely separate CANBus where any third party systems would sit. These third party systems requiring vehicle information would not be aware they are not speaking to the actual vehicle network.

This limits the attack surface by removing the risk of third party dongles plugged into a vehicle OBDII port. This device does not attempt to address the vehicle manufacturer itself attaching a network-enabled system directly to the CAN (which is happening). That particular action cannot be protected against except by vehicle manufacturers. The best one could hope for out of a third party solution is alert the user if a certain type of message is seen on the bus. If that message is malicious, however, it may be too late. It could certainly work as a “black box” of sorts though. Hmm…perhaps vehicle CAN forensics will be the next project.

If you are so inclined you can download the source and create your own CANBus Protector. The repo can be found at

Happy Hacking,

S4x15 CTF ICS Village Page

The Capture The Flag (CTF) contest in the ICS Village at S4x15 was a big hit. We have had numerous requests from attendees and those that heard about it for more information and data. So Stephen has put together a page of information. The page includes:

  • Examples of flags in each of the five categories
  • Packet captures with ICS protocol and attack data (the most requested item)
  • Screenshots of detected data and the scoreboard
  • Pictures from the ICS Village
  • An explanation of the event

You may also want to watch an interview with the team that won the CTF.

Great job by Stephen and the team of volunteers who put the CTF together and kept it running under three days of attacks. It puts a lot of pressure on the team to make it bigger and better for S4x16.

S4x15 Video: Ralph Langner or Cyber-Physical Attack Engineering

Ralph Langner presented at ICSage: ICS Cyber Weapons during S4x15 Week. As always Ralph is introducing new thoughts to push the industry forward, but this session is more on how to orient and organize the ICS communities’ thinking on attack / defense on ICS.

There is entirely too much attention paid to 0days and compromising an ICS computer or application. This is still trivial to do based on code quality and is almost always unnecessary. A more useful line of thinking is what would or could an attacker do with this access, what would be the intended result, and what can we do to defend against it.

  • At the 9 minute mark, Ralph discusses different types of ICS cyber-physical attacks.
  • At the 22 minute mark, he breaks down impact categories of cyber-physical attacks.
  • At the 29 minute mark, he discusses examples of how to identify the defensive controls to prevent catastrophic results.

The pull quote, in my opinion, was “is there any combination of bits and bytes that if I throw that at this plant will result in harmful physical effects? This is a question that can be answered through engineering methodology”.

Tool Release – Digital Bond CANBus-Utils

beagle-canbusI’d like to make a quick post with the release of some CANBus analysis tools I wrote.

The tools are written in javascript using nodejs, which comes preinstalled on the Beaglebone black — my hardware of choice when doing CAN analysis. I wrote up a brief README on setting up a Beaglebone for use as a CAN analyzer which you can find at That README provides links to the hardware setup we used at the S4 CANBus Hacking Class along with some instructions for getting an environment set up to start doing vehicle work in javascript or python.

For Version 0.1.0 we are releasing four tools:

  1. unique_ids: Watches the CAN and logs all IDs on which data is sent
  2. watch_id: Prints out every data packet for a given ID or IDs. Bytes that change from one packet to another are colorized for easy viewing.
  3. decode_obdii: Decodes any OBDII standard request/reply messages sent over the CANBus. Includes a description of the message type according to the spec.
  4. canbus_IDS: A very simple Intrusion Detection System for CANBus that will take configuration via JSON and alert on messages of a given ID. TODOs are extending alerts to use beaglebone GPIO and allowing ctype configuration of message data to alert on things like a bit flag being set.

The code is licensed under MIT. Contributions, questions, complaints are welcome. Check out the README file for more details on using the tools.

The repository is available at

Happy Vehicle Hacking,