I’ve been meaning to chime on some of the issues that surfaced over the weekend at SANS, but real work has been getting in the way of my blogging.

Shame on me.

In particular, I wanted to respond to some of the issues that were raised when Digital Bond’s work with DHS/US-CERT & CERT/CC last spring appeared as a bullet point (with some editorializing from Jason Larsen) an INL SANS slide in less than favorable terms.

Here goes. Warning, this is long!!!

If you hadn’t read Dale’s blog on how business models impact disclosure practices you might find it ironic that two of the first three SCADA Vulnerabilities to be disclosed (and reported to) US-CERT were not the result of commercial vulnerability assessments or the result of an well-marketed INL-CSSC Vendor Assessments.

The reality is the first two advisories (LiveData and Tamarack) were done in our spare time and on our own dollar. And for what is is worth, probably 15-20 times the effort spent was on communicating with vendors and working with coordination centers than was spent finding the flaws. This either means the bugs were so easy to find (they were!) or that the people/organizational challenges were hard to work out (sort of true)

In any case, if you did cheat and read Dale’s blog, you know exactly why these two flaws were the first to go through (and helped refine) the US-CERT’s process for handling vulnerabilities.

It is simple.

There were no commercial/contractual obligations standing in the way of disclosure–which is not the case for the majority of SCADA/DCS product/device/application assessments being conducted and the dozens (to hundreds) of product vulnerabilities that may have been discovered and that may (or may not be!) fixed as a result of assessments.

(NOTE: I’m making these numbers up, but they are probably conservative given the number of assessment that have been conducted. Or maybe they are lower because I’ve heard CSSC is only doing assessment of “non-Legacy” systems. Which is another reason why end-users need to do testing themselves or get someone to help them test systems that are actually deployed and get vulnerabilities addressed vs. securing forklift-upgrades)

My not so subtle point here is that relying exclusively contract vulnerability testing/assessment that is bound by NDA/CREDA to “raise the bar” is unhealthy for the industry.

It is also difficult for outsiders to measure the impact or effectiveness of these assurance activities.

(ASIDE: When I was at Cisco, some of the metrics we used for ourselves/for product teams was the number of defects discovered, their severity, how many long the bugs remained open, how many had been resolved, how quick defects were assigned to engineers, how simple/complex were the vulns to discover and exploit. Product teams that took security serious oftened assigned bugs in a matter of days while others let bugs go into the triple digits. It wonder if DHS (or even INL) has this sort of data on issues found by CSSC assessments.)

But do end users ever see the results of contract applications assessments?

Yes, end users sometimes see commercial assessment results when vendor contracts for a 3rd party assessment. Or at least some subset of the results. I’ve seen a fair number of these type of reports for non-SCADA products and applications. The way it works is that when commercial application security firms (such as @stake or Foundstone or others that have not been swallowed up by security vendors) do assessments, there are usually two sets of deliverables: the one the vendor uses internally to get a true assessment of the product and the sanitized version used for marketing purposes that can shared with customers. The consulting companies also sometimes give you (meaning the vendor) custom tools that were developed for the assessment.

Some examples of the 2nd type of deliverable are Cisco Whitepaper on VLAN Security for Catalyst Switches and the @stake report. SCADA Asset owners might have seen similar types of documents for their current SCADA/DCS system–or more likely, for a new system they might want to upgrade to. You will notice you don’t see any defect information, like you would in a product advisory. Something else is that these sort of deliverable highlight the security configuration features that are available vs. mentioning implementation vulns. That being said these sorts of white papers are far more meaningful than the “SANS Super-Secure SCADA Vendor Awards”

It’s a win win: the vendor shows they’ve had work done and the consultants use this show they’ve worked with the big vendor to get more assessment contracts, perhaps with the vendor’s competitors.

What is my point? Should vendors contract security companies or government entities to do these sort of assessment? Of course. Should asset owners sponsor assessments and testing of SCADA products and applications? You bet.

But these activities have some inherent constraints due to their commercial nature.

The sad fact of life as a security consultant is that your clients are free to ignore your advice, no matter how much they are paying you. This is just as true at Digital Bond as when I worked with product teams at Cisco and found bugs in security products.

A client can decide not to take action on an assessment findings. Whether is a firewall rule change or a data validation strategy in a web application. Development managers can and will ignore your advice or junk your bugs in the defect tracking system. And there may be valid technical or business justifications.

But there always are. Based on experience, this is less true for bugs that are not protected from disclosure, that are “free.”

If a researcher reports the bug to a vendor security team (assuming they exist) the security team esclates these issues within customer support organizations. Quie often these security teams are in “customer advocacy” groups and work to get the bugs fixed. This happens less for internally discovered bugs or others that will never go public.

It might be counterintuitive but some folks in vendors thing external discovery and disclosure is a good thing. If I go back in time 2-3 years, I was actually glad when researchers found and responsibly reported flaws in Cisco products. When customers found and angrily reported default undocumented administrator credentials in shipping products, I cheered.

Product management or sales teams might not like the bad PR caused by a BlackHat presentation done by Mike Lynn or FX, but I believe the impact of these external inside development organizations was positive–and ultimately made products and customer network more secure. It opened eyes. Folks “got it” when “hackers” on the outside found bugs. Less true for internal findings. It also gave security engineers involved in product security that much more incentive to do a good job on assessing products prior to release so that outsiders did not find the easy low-hanging fruit that unfortunately makes up a lot of vendor advisories.

And we kicked ourselves when we “missed one” and the Engineering Director sent us an email asking for explanation.

But in the SCADA Security world, this viewpoint appears to be a minority one.

At the PCSF meeting in LaJolla it was clear that most vendors did not think SCADA vulnerabilities should be reported to US-CERT. US-CERT vulnerability notes were “just arming hackers.” Besides, the control system community didn’t read them anyway.

And if I have my facts straight (and I hope someone will correct me and I’ll be glad to issue a mea culpa), at the SANS meeting one or more folks from INL (representing DHS sponsored program) also feel that disclosure of SCADA vulnerabilities (to and by US-CERT?) is inappropriate.

The bottom line (assuming you made it this far)

There is a need for SCADA Vulnerability research/assessments that can be disclosed to trusted third parties to ensure that vulnerabilities are addressed in a timely manner and so that end users actually have the information they need to make informed risk decisions.

I would like to think this is happening, but if it is not — what is to be done?

Perhaps ethical independent security researchers should pool their dollars to buy 2nd-hand equipment off of eBay in search of SCADA vulnerabilities.

Or maybe there should be new government initiatives that are less captive to industry pressure than current programs.

Any other ideas?