What does it mean? INL Testing LiveData’s ICCP Server
For a while now I have had a number or questions and concerns about this “testing”.
1) What exactly is the INL test procedure for these vendor assessments?
I have attended numerous control system security events and never have observed anything more than vague hand waving on this question on stage. Something to the effect of INL runs rigorous tests on the systems. INL has identified vulnerabilities that vendors have addressed resulting in more secure control systems. Then there are motherhood and apple pie generic findings such as closing ports, missing patches, changing default passwords, …
The reports of the results are very closely held and disclosure is solely at the discretion of the vendor. Some of the vendors have released sanitized versions of the reports to their customers and potential customers. I have had the opportunity, probably unauthorized, to read two of the sanitized reports, and based on this information and what is presented at conferences it appears that the INL test procedure is similar to a security assessment Digital Bond or our competitors would perform for an asset owner.
Based on what I have seen, INL has been given a scholarship because they are a National Lab funded by the USG. The limited evidence that is available indicates this is a classic asset owner / end user assessment rather than the testing a vendor application should undergo.
2) What is the responsibility of INL or any National Lab for sharing information with the community when most of their expertise and control system security program is the result of taxpayer money?
INL, Sandia and PNL have received many tens of millions of US taxpayer dollars over the past five years to build up a control system security program. In addition to the money, they have received the intangible benefit of being the go to organizations for DHS and Dept. of Energy in this area. This is in large part the reason that LiveData and other vendors go to INL for testing.
Is it appropriate then that INL provide test results generated from taxpayer funded programs and expertise only to the vendor? The vendor has the sole decision whether to fix the vulnerabilities and to inform affected asset owners? How is it in the national interest that taxpayer funding is used to create a program that potentially lets vendors ignore any identified vulnerabilities and not inform affected asset owners?
Let me clear - - to date - - not a single INL discovered vulnerability has resulted in a US CERT vulnerability note telling asset owners they need to apply a patch to their system.
We have clients that use applications tested by INL, and I have yet to see them receive a support notice saying our system needs to be patched to address identified vulnerabilities from our INL testing. There are probably a number of silent fixes in new releases, and admittedly we only see a fraction of the systems. I would be very interested to hear and write on how a vendor proactively notified asset owners of a security patch resulting from the INL testing. It would be great to be able to blog on a concrete, positive result.
Given the strong position INL and the other National Labs are in, it does not seem unreasonable for them to put something in the contract saying all vulnerabilities will be reported to US CERT within some time frame, 3 months? 6 months? I have suggested this to the INL team a number of times.
Again, to be clear, I’m not suggesting exploit code or detailed vulnerability descriptions be published. Simply that the version of vulnerable applications be published along with patching information or compensating control recommendations at the appropriate time as determined by US CERT.
3) How can INL allow the marketing of ‘INL testing’ without putting any requirements on vendor to fix identified problems?
Being “INL tested” is viewed in the community as a significant marketing statement. In fact I have heard asset owners ask if a system has been “INL tested”. The marketing benefits of this testing is one of the major reasons vendors are spending big bucks to get their systems tested.
Yet this “INL tested” means nothing from a practical standpoint. Some sort of unspecified testing occurs and the vendor decides whether to disclose any results or fix any problem. This seems very irresponsible for any consulting company, National Lab or any other testing body.
Digital Bond performs white box security assessments of vendor applications and systems under NDA, most often prior to release. The vendors typically adopt some but not all of our recommendations to address vulnerabilities and follow good practice, and it is their choice. We would never let a vendor say “Digital Bond Tested” and imply this means more secure unless a description of the testing methodology and any patches or other changes required to address all identified vulnerabilities were made available. This is not necessary if the goal is to improve an application’s security, but it is necessary if an implication of security is made by a public “Digital Bond Tested” announcement.
If the goal is to identify and fix problems, why the press releases and PR?
Why is INL, and by reference the USG, giving vendors their halo without requiring some civic minded action by the vendor to correct vulnerabilities and notify affected asset owners?
4) LiveData Issues
The LiveData stack is used by many vendors. In the previous vulnerabilities identified by Digital Bond and published by US CERT, 12 different vendors were identified as having the vulnerable LiveData stack. Nine of them still have not responded to US CERT and must be considered vulnerable.
So if INL identifies vulnerabilities in the LiveData ICCP stack, and if LiveData contacts all their customers with patch information, it still will only filter down to a small percentage of the affected asset owners. We may not agree on what is proper in vulnerability disclosure, but I think we can all agree this is improper.
After rereading this post I’m sure it can be viewed as bitter jealously from a competitor. There may be some truth in this. INL is in fact our biggest competitor for both control system security research and consulting, and Digital Bond is small enough that we probably don’t register on their radar.
That said, INL has some very sharp and talented employees on the control systems security team. I’m happy to compliment them in this blog when they do great things such as the CSSP training program, procurements language project and CS2SAT. In fact, I’ll have a blog and our first SCADApedia entry on CS2SAT later this week.
Author: Dale Peterson
Posted: March 19th, 2007 under Assessment Tools, National Labs, Vulnerability Disclosure.
Comments: 4
Comments
Comment from Ron Southworth
Time: March 20, 2007, 9:42 am
Hi Dale,
It is interesting to read your comments on the work of INL. We are all entitled to vent frustration at times.
I have been involved with the guys there for a while with respect to reviewing documentation on recommended practices where the effort being produced is actually from a number of research institutes such as INL and LL under the umbrella of DHS activities.
From my perspective their efforts to date have been quite non-partisan in nature and I would commend all of the people involved and the results and goals they are trying to set and maintain as to be quite exemplary. I would also add that I consider the work you do at Digital Bond for the good of the community to be of similar standards and ideals (By no means a complete list of organisations all working in the same space)
I think that any of the knowledge learned through work with vendors via the assessment processes they conduct, based on public info surely must be seen as flowing on to the documents and standards work in perhaps something tangible like the common procurement language?
After experiencing first hand very recently a response from a couple of vendors over releasing trivial vulnerability info and subsequent over reaction and obvious sensitivities, perhaps to alleviate this sort of problem arising not a lot of detailed disclosure occurs as a result.
Good bad or otherwise as to how the timing or methodologies for such disclosures occur, this is something I have discussed in some email round robins in recent times and I have seen some different perspectives on the issue and I for one would like to see some more debate and shared opinion form the industry before reaching a solid personal opinion on this issue.
I think I have mentioned previously about the need for us all in the industry to derive income and so this can create some competitive issues and headaches- we all have to market our wares and feelings can be understandingly bruised quite easily at times, as specially those that are so dedicated.
We need to try to work co-operatively as much as we can and share the burden around government, vendor, support agency, and end users . I think you genuinely try to do that and offer praise to others in the market and I hope you receive similar praise from others in the market place as like so many it is well deserved.
At the end of the day the forum you have established here is a means to express your thoughts ideas etc so expressing your feelings is quite logical and I hope helps to keep some balance.
Remember healthy competition is a good thing it can bring out some of the most creative results at times.
Keep smiling
Comment from Matt Franz
Time: March 20, 2007, 9:43 am
First of all I think you reiterate some of the issues I first raised in my fee/free vuln research blog. And I think we have seen seen the non-institutional security research community step up to some extent, as measured by the number of new vulns being disclosed by US-CERT. That is easier for software such as OPC servers, for which free/demo versions are available. The sad reality is that research programs (whether in academia or government) are are a business and the labs and vendors (and maybe even end users) benefit from these arrangements and increased transparency is in no ones interest.
Comment from Jake Brodsky
Time: March 21, 2007, 2:24 pm
The problem from this user’s perspective is that it’s hard to translate from the actual vulnerability documentation to something which may be relevant to our existing SCADA system. The question here is whether there is enough value to be had by revealing it to a customer.
I hate to admit that my industry is so far behind, but it is. Most of the folks I know from other water utilities have difficulty describing what SCADA is, let alone how it works. I even know of engineering and controls integration firms who have trouble with this concept.
You seem to suggest that by airing this dirty laundry via US-CERT that something good will come of it through public awareness. I have misgivings about this. Honestly, Dale, I don’t have an answer. However, airing this information, even without exploits for the script kiddies, means the information will be available to any Red Team that decides to launch an attack. Might we be better served if such information was kept under wraps?
The problem is that the users are not savvy enough to do anything with such information, and the vendors aren’t all that happy to share such ugly background either. So how DO these CERT listed vulnerabilities help? I hate to say it, we’re damned if we publish and we’re damned if we don’t.
The folks at INL have made a judgment call and decided to try advising the vendors directly in the hope that by keeping such information under wraps that overall security purposes will be served. You should know that I have some doubts that this is a good idea as well. But, since I really don’t have a good answer to this problem, I can’t say they’re wrong any more than I can definitively say you’re right.
I know we’d like better transparency, but perhaps this isn’t the time or the issue to call for it. I’d like to think that some day we’ll see enough utilities watching CERT so that they’ll know they have to take action, or hound their vendors or integrators. Right now I don’t think that’s the case. I think the only thing we can do is to educate the masses in the hope that some day they will be capable of reading such warnings and doing something about it.
Jake Brodsky
Comment from Dale Peterson
Time: March 22, 2007, 11:51 am
Jake - I think you make an intellectually honest, if not terrifying argument.
History in the IT and control system world have proven that the decision on what, when and how to disclose cannot be left solely to the vendor. I’m sure somewhere there is a vendor that is an exception to the rule, but I haven’t run into one yet.
But your comment is different - we can’t trust the users/asset owners to do the right thing such as monitoring for published vulnerabilities and determining the appropriate response. So if the users don’t respond - - does any value to disclosure disappear and the balance tip to the attackers?
We are fortunate that we work with leading edge asset owners (they are the ones that pay for security consulting services), and I know they have proven processes in place to deal with published vulnerabilities. Ignorance is not bliss for them. I also think we should aim high rather than accept a low status quo.
That said, I can’t say you are wrong. One of the reason we blog on the US-CERT Vulnerability Notes is because we are unsure even if our loyal blog readers are subscribed to US CERT.
I have also been getting a lot of off-the-record comments on this post as you might imagine. I’ll have a follow up that will bring up a few points/clarifications without violating anyone’s desire and need for anonymity.
Dale
Write a comment