Comment from Ralph Langner
Time: November 6, 2008, 6:15 pm

Disclaimer: The following statement is not meant as criticism on Miles’ paper, which I do not know more about than any other blog reader. (I will, however, try to come up with a sophisticated three-part question after Miles’ presentation — inside joke.)

“How do you estimate something that you don’t know” asks Dale, referring to 0days of control systems. I must say that I find the question severely misleading. The sorry fact is that most vulnerabilities we have to deal with in real life are very well known. To mention a few:

- lack of authentication in Modbus/TCP, Ethernet/IP, PROFINET, proprietary protocols, you name it
- backdoors in SCADA/DCS products for vendor access
- “open” (=insecure) software interfaces in SCADA/DCS products for easy integration with 3rd party applications
- “open” technology such as OPC which was designed with no considerations about security
- firmware uploads that are not checked for validity (a.k.a. Boreas)
- etc, etc, etc.

There are three problems with these:
1. They are known for years
2. They are not bugs, but features, and therefore:
3. They won’t get “fixed” in short term.

I don’t worry about 0days. I worry about attackers who at some point in time might get smart enough to notice the obvious, and find out about those safe bets. For the attacker, the safe bet is the vulnerability that won’t get fixed because it is considered a “feature”, and therefore is likely to be available in a certain product regardless of patch level and firmware version.

Comment from Éireann Leverett
Time: November 7, 2008, 9:41 am

@Ralph Langner
Have you got any metrics for your list?

Kidding aside, I am looking forward to more metrics in this space. In general they will help with the ‘battle in the boardroom’.

Comment from Ralph Langner
Time: November 7, 2008, 3:56 pm

Éireann,

I’m afraid that if you and your boss are looking for metrics, you might not get what was advertised here. Again, no criticism on the work of the INL fellows here who I am certain will come up with some interesting and useful stuff. Some things puzzle me though.

Metrics are not ESTIMATED but MEASURED. According to Jaquith, a good metric should be
- consistently measured,
- cheap to gather,
- expressed as a cardinal number or percentage,
- expressed using at least one unit of measure, and
- be relevant to decision makers so that they can take action.

“’Metrics’ that depend on the subjective judgments of … humans are not metrics at all”, says Jaquith. Unfortunately it looks like we must assume some sort of subjective judgment in the estimate, because, as Dale says, the numbers that we’re talking about are not KNOWN. They are probably also not cheap to gather, as we cannot expect that any of us would be able to measure or compute the number of 0days in our control systems every month or so just like that. What’s the unit of measure here? Number of 0day vulnerabilities, I think. Unfortunately it is unclear what constitutes a vulnerability. For example, several weeks ago we disclosed what we believe is a critical vulnerability of a specific PLC product line to a very large vendor of automation systems (you send some command to the box which then halts with all outputs frozen in the present condition, no fallback to safe). Surprise! The folks from the vendor told us that they “would not call this a vulnerability, but thank you anyway”. Now who defines the term vulnerability, a global player in the automation business, or a tiny consulting firm? Regardless how you decide, “number of vulnerabilities” is not a unit of measure. Same with 0day… The term comes from office IT where Microsoft once started to be friendly enough to disclose their vulnerabilities in public. On Patch Tuesday, everyone can start counting the calendar and know what defines an 0day – consistently measured, easy to gather etc. On the other hand, the term 0day HERE is used as a vulnerability that is UNDISCLOSED, but “known to someone”. Even if we buy this definition, it seems clear to me that we will have a hard time measuring how many things are – or might be – known to someone. Finally, what’s the relevance in the number of 0days of control systems for decision makers? I don’t know, but I would rather start with all the many known and undisputed vulnerabilities of a typical control system environment, such as those of the NT4 operating system that the application in question is running on, than being concerned with the unknown. In the average control system installation, we have so many KNOWN problems that we don’t need to dig for further ones in order to justify taking action.

No offense for Miles and his crew, but I think we should be very careful here with our terminology and about keeping things in perspective.

Comment from Matt Franz
Time: November 7, 2008, 5:41 pm

Ralph,

But unlike 0-days, you can’t write Metasploit modules to generate articles on The Register for all those vulns you are really worried about

Country First!

(so you know it as actually me!)

- mdf

Comment from Ron Southworth
Time: November 9, 2008, 1:09 am

Hi Gents,

Having had the pleasure of meeting Miles I am certain that he would be happy to read that people working in the industry are discussing the topic even if he does not reply here in an open forum.

Eric Byres and David Leversarge presented / prepaired a paper in a “response” to Miles paper and it was equally thought provoking.

There is an information systems standard in draft form IEC27004 if I recall correctly that would address this need to measure.

Éireann’s boss, Mine and many others would love such a tool to justify validate the realisation for return on investment and it would be a good for the industry to have however we need to take on board the very valid points from Ralph’s posting.

The threat aspect is difficult to measure.

The mean time to compromise is difficult to measure as the it is hard to quantify how much knowledge the attacker (or group) has we don’t know for certain what the threat is.

And from that basis I think it’s strength is modelling contingencies and scenario’s the what if’s. I think this is the area that security metrics of this type will be useful for especially seeing how much push back from CIO level ther was with 27004.

It would be reasonable to assert that these are intangeable values and I would contend that it is a good litmus to get technical people discussing security however we should be looking at other aspects that are more sound, redily understood, such as risk assessment, to deliver the message and quantifying what risk we can remove from the table. Measurements, consequences and concepts are all well understood and management is used to this tool. It puts everything on a level playing field for the enterprise to allocate resources to.

I think we are so used to working with risk we forget to upload to management what risk the enterprise operates under on a daily basis.

This is my take on it.

Comment from Éireann Leverett
Time: November 10, 2008, 6:28 am

I am big fan of Jacquith, and while it is difficult to be precise and not estimate or use subjective quantifications at the moment, looking in that direction is the right way to go. This discussion has all the right points.

On the vuln definition level, I personally think the best way to address that is US-CERT at the moment. A third party is the best judge until the politeness of vulnerability management takes hold.

For us mediators, trying to cross ‘the line’ to talk to each other, metrics is one of the best ways to find an understanding.

So summing up, we don’t have the right metrics yet, but we’re discussing what they are…and that is a step towards finding common ground.