Wonderware Disclosure Saga
Saga may be overstated since the process did not take that long, but it was a classic example of why we don’t agree with leaving disclosure decisions up to the vendor - - or the researcher. Our approach is to let a coordination center, US-CERT in this case, determine what disclosure is appropriate.
On April 17th Xavier Panadero of Neutralbit contacted Wonderware about the InTouch 8.0 vulnerability. After some back and forth, Wonderware indicated in June that the vulnerability was not present in InTouch 9.0 and Xavi was able to verify this. So far, so good. A solution to remove the vulnerability and a reasonably prompt vendor response by disclosure standards. Then the problems began.
What was Wonderware going to do to notify InTouch 8.0 customers of the vulnerability and the fix?
After all, InTouch 8.0 is still being sold to existing users through the end of the year. Given the long lifecycle of control system devices and applications there will likely be 8.0 systems for at least another five years.
Wonderware’s answer was fuzzy. They sent out a very large whitepaper “Securing Industrial Control Systems” and implied that customers needed to read this to fix the vulnerability. The whitepaper did not address the vulnerability. They mentioned customers with support contracts could upgrade to 9.0. They said the product is just a “toolbox”, and it is impossible for them to control how customers use the product. Still no answer to the question of whether Wonderware had told their customers or planned on telling their customers.
At this point Xavi asked for our assistance in working the issue with Wonderware and US-CERT if appropriate. I contacted Wonderware and got the same answer, and they felt the case was closed. It was only when I told them this vulnerability was being reported to US-CERT and we wer just trying to be accurate on their disclosure to date that the tenor changed.
To the best of my knowledge, prior to our disclosure to US-CERT Wonderware had not disclosed, nor did they intend to disclose, the vulnerability. Not even on their limited accss support site. I checked with friends who had access to this site, and they found no notice. After all, it was fixed in 9.0.
This is a classic example of the silent fix. An asset owner does not have the information to determine if they should upgrade or not. They may decide the features do not warrant an upgrade, but if the security issue was disclosed they may reach a different decision. The risk is the asset owners so they need the information to make a decision.
We can argue how much information should be disclosed, by whom and to whom, but I think almost all would agree that customers should be notified in at least a vague way of a serious security problem with an existing solution.
The question I have is how many other security vulnerabilities is Wonderware sitting on?
And this may be quite petty, but in the last hour of the last day at WeissCon a Wonderware employee stood up twice and said if you need secure systems they had it. Problem has been solved. If it had not been so late in a long three day event I think many in the room, including myself, would have jumped on it. Not because of any one vulnerability, but because it showed a lack of understanding of security.
Wonderware has published a public announcement and a private tech note. The public announcement downplays the vulnerability saying “The presence of the share can cause some confusion or concern as related to NetDDE security. The share name appears in a security scan of the computer, even if the share is not usable.” I have seen the exploit from Xavi, and he has compromised systems in his lab with the default install.
It seems that Wonderware still does not get it. Software has bugs and some are going to lead to vulnerabilities. When one is found you deal with it honestly and encourage your customers to apply the patch. I really wonder what the private technical note says.
After all that bashing, let me say Wonderware is not alone and is in many ways better than numerous control system vendors. The majority of the vulnerabilities we identify are in assessments where we are restricted by NDA. All too often the vendor responds with a yes you are correct, and either a sizable engineering fee to fix their mistake or a polite way of saying it will not be fixed. At least Wonderware had a fix, and they do emphasize putting security features in their product.
We continue to push our clients to disclose vulnerabilities to US-CERT and their user groups to put more pressure on the vendors to fix the problems.
Author: Dale Peterson
Posted: November 20th, 2007 under Vulnerability Disclosure.
Comments: 6
Comments
Comment from Jake Brodsky
Time: November 21, 2007, 4:53 pm
Dale, it was never news to me. NetDDE has always been a GAPING security hole. Any protocol that allows you to remotely execute applications is dangerous to say the least. The way Wonderware told their users to set it up was no different than the instructions for OPC via DCOM. In essense, they wanted you to enable execution to the whole world.
This isn’t news. This is why we control systems engineers have always tried to create borders around our gear. I’m not sure how many people knew this about NetDDE, but it hasn’t exactly been a secret.
On top of that, you can run applications and ActiveX drop-ins from inside InTouch. Yes, this is a gaping hole. Welcome to reality. I’ve said this for years: You can’t bolt on security after the fact. You have to start from scratch.
The folks who wrote InTouch are using similar architecture to many other HMI applications. Anyone who expects better from RSView, Intellution, or others are deluding themselves.
Right now, the best we can do is to carefully shield these networks from the outside. Those who expose these HMI work stations, OPC, or heaven forbit, NetDDE, to the outside world are just asking for trouble.
Comment from Ralph Langner
Time: November 22, 2007, 2:48 pm
That’s what I was going to say, Jake… I am not puzzled by some vulnerability found in a NetDDE application. I am puzzled that folks still use NetDDE. Well, probably it’s because it is a “proven standard”.
Comment from Jake Brodsky
Time: November 23, 2007, 11:33 am
Ralph, in Windows based HMI software there really aren’t many secure choices. Both DCOM+OPC and NetDDE have a sad history of instructions telling integrators to open the whole computer wide to execute everything.
I don’t pretend for a minute that any of this stuff is secure. Even if it could be secured, it is still a scary thing to put trust in.
The ActiveX container architecture so often used on Windows based HMI displays leaves a very large surface area of attack for malware. With a container HMI, and little or no validation for most drop-ins, script injection attacks are easy.
What we need is a new architecture. Fixing something like this is not feasible. Even if Vista checked the signature of every drop-in, the surface area of attack is still frighteningly large.
We need to figure out how to slim down these clients. We need to figure out how to make SCADA systems flexible without excessive centralization. My guess is that this will require an entirely new Operating System architecture.
We have bounced back and forth from POSIX based X-Windows to ActiveX containers. These are two extreme aspects of something we’re trying to accomplish: Real time display of data and dispatch of commands to and from a tag server using locally executing programs.
We’ve tried all the easy stuff. Now it’s time to figure out the hard aspects to this.
Comment from Ralph Langner
Time: November 24, 2007, 11:28 am
Jake, what I don’t get is that the market doesn’t put pressure on the industry to come up with a simple, streamlined and easy to protect protocol for interfacing automation peripherals. Once, when software vendors started writing all those NetDDE and OPC connectors, peripherals used all kinds of protocols via all kinds of media. Today, virtually any peripheral that you can buy uses TCP/IP and Modbus. Now it would already be a relief if you would just use an embedded Modbus/TCP driver in your HMI instead of any DDE, OPC, DCOM, SOAP or whatever. If you would also replace Modbus by a modern protocol some time in the future, you could virtually forget about all the hassle with component software. What most people don’t realize is that the interaction between a SCADA system and peripherals is very, very simple, and that there is absolutely no need for some high tech stuff like DCOM and Web services. This would be achievable in near term, but it won’t happen. Instead, end users and the industry have come to favor “proven standards”, no matter how insecure and oversized they are. So what we ARE going to see that the “proven standards” get enhancements to fix some of their flaws, and we are even seeing that a piece of nightmare technology — DCOM — moves into controllers, namely those that support PROFINET CBA. Although controller/SCADA interaction hasn’t changed a bit, the development of IT procedures and products over the last years has always involved more and more complex stuff. As we all know, a complex system is an insecure system. Funny enough, it looks like nobody wants to reverse this trend.
Comment from Ron Southworth
Time: November 28, 2007, 6:26 pm
Hi Ralph I would refine your first paragraph slightly from our perspective here in AU.
We (end users across multiple verticals) are placing a lot of pressure on vendors to secure field devices, protocols and systems and are experiencing a lot of resistance from some, not all.
I beleive the message is being heard at last and it is starting to be addressed. For the players that are early adopters of the cultural shift, They will have a market advantage and this is something that is attractive and tangeable. We recently held a brief for our new systems requirements and some vendors nodded knowingly and some wiere almost in a heightend form of being out of their comfort zone visibly squirming in their seats. More people knodded than werre squirming however. We are hoping to improve our vendor involvement with our SCADA CoI and there is a very positive level of interest and involvement.
I lament along with Jake as to how complicted sytems can become. Complexity quite often translates into a reduction in availability.
With best practice principals effective compartmentalisation of the industrial process control systems can be acheived and availability can be maintained perhaps even improved with some lateral and realistic objective risk assessment. The effect of “from plant floor to boardroom” in terms of information visibility is really the aspect that has to be carefully considered. Jake often highlights the advantages of machine to machine interfacing and this is a valid approach to increasing availability of equipment to the operation of the process and forces a more physical attach profile.
I do think this is a bit unfair to single out Wonderware as you say Ralph this is a broader issue.
Comment from Byron Sonne
Time: December 3, 2007, 12:16 pm
Having worked in the vulnerability management space for a company that made products targetted for nets of 100k and more nodes… I’ve come across Wonderware. Definitely has issues! Mind you my knowledge is over a year old by now, but I doubt the situation has changed.
Surprised at NetDDE - really? Then you’d be positively disturbed at the amount of WinNT and DOS stuff still in use for process control, or for that matter, in datacentres.
Write a comment