My point — we, the SCADA Security community, need to put all our efforts and emphasis in the PLC, RTU, controller space on getting vendors to add basic security features to their models available for sale today. Beginning with authenticating the source and data sent and received from the PLC and continuing with other Security 101 features. We should not say or pretend that any other solution besides this is acceptable. Fix the problem! We have lived with this and PLC vendor inaction far too long, and it is pathetic that there is no serious secure PLC offering.
Last week I had a public back and forth with Joel Langill, @scadahacker, on Twitter. Here is a excerpt from the conversation.
My main disagreement with Joel is not on the value of NAC. It would not be on the top of my list of new technical controls like white-listing / HIPS, but it is probably worth considering its value as a compensating control. I hope to have Joel on a podcast soon to talk about his advocacy of NAC as an important tool for ICS security.
The problem is we should not be providing any cover or any excuses for PLC/RTU/PAC/Controller vendors to further avoid designing security into their products. This has happened for years with the air gap fantasy and then the firewall “preventing access” to the PLC. PLC’s were not considered vulnerable even though they were vulnerable by design because the attacker shouldn’t be able to reach them. The last thing we need is another silver bullet technology that allows vendors to avoid fixing this gaping ICS security hole.
Let’s take it a step further. In two weeks Joel will be presenting on Stuxnet, other attack vectors and stopping them at the Siemens Automation Summit. Unless Siemens is prepared to announce a new line of PLC’s or major upgrade that will have the Security 101 features, this is a huge mistake. The only message that security professionals should have at that meeting is how wrong it is that the Siemens PLC’s are designed with little or no security; that Siemens response has been late and misleading marketing spin on Stuxnet and now the Beresford vulnerabilities; and that Siemens’ customers should revolt and apply all pressure possible to make the vendor truly address the problem. I make the same plea to John Cusimano of Exida and Eric Byres of Byres Security who will also be there presenting on ways to address Stuxnet and other ICS security issues. Remember this almost one year after Stuxnet and three years after a Siemens requested INL assessment pointed out many of these Vulnerable by Design problems. After all this time there should at least be a detailed announced plan to address it.
The last message that needs to be delivered at a Siemens User Group is that all will be ok because you can deploy NAC technology, set of IDS signatures, or Tofino field firewall and be secure. This is not to knock those or any other compensating controls. These are worthwhile presentations in other venues, but definitely not the message to deliver at any user group meeting where the vendor continues to ignore designing basic security features into even their flagship, new controller product lines.
I considered adding some analogies here where influential people knew there was a big problem and chose to stay silent or pretend like it would be ok if we just did these other things, but they all were loaded with sensitive issues that would cause the conversation would veer off topic. Joel, Eric and John are all smart guys that I would gladly work with on projects, hire or work for. I have in the past and would again. They have all been on the TMICSS podcast because they know their stuff. It is time for the leaders in this industry to stop being sensitive to offending large vendors, making excuses, offering alternatives and just come out and speak the truth.
We are not going to solve this problem quickly with the large deployed base of field devices out there. I’m sure that the compensating control suggestions in their presentations are helpful, but only after a lengthy and pointed discussion of the real problem of vulnerable by design PLC’s. It is definitely time to have a secure PLC that can be purchased and deployed so the problem doesn’t get bigger, and so the most critical ICS can decide when their risk management decides it is necessary to solve the problem.
Mr. Peterson:
One thing that Ralph Langner and I have been discussing lately – the Stuxnet people had the entire PLC code BEFORE they launched Stuxnet. And they did not breach the security around the SCADA castle to get it. The whole code was probably on a laptop.
Think about how we write the ladder logic for the PLC. We do it on a laptop then download it to the PLC but the whole code is still on the laptop plus a whole bunch of other information. Use an ADF Triage G2 and you never even have to go near the castle defences. With a floorplan of the castle you can breech any defences.
Dale,
I’m surprised you haven’t pulled out the Technical Feasibility Exception (TFE) card in this argument. Under NERC, there is a responsibility to identify deficiencies in products, and actively work to remediate them.
While those deficiencies are all limited to what is in the NERC CIP standards, the CIP covers many Security 101 practices.
In addition, once those Security 101 parts are taken care of, the information shown by increased account management, increased logging, identification of ports and services, etc would then need to be managed. As someone who has dealt with security management in an automation environment, there are ways to manage security controls, and there are ways to manager security controls CHEAPER. Oftentimes, a bare bones implementation of security is not cost effective, and a vendor can earn bonus points by having both the capability for security and cost effective management of security.
Mike T.
Private Citizen
you’ve still not said how “fixing PLC vulns” would have prevented Stuxnet.
placing the blame solely or primarily on the PLC vendor is misguided and naive. vendors are not in the business of pissing off customers (though of course it happens all the time) or setting out to make them unhappy. They generally are going to put the finite R&D money on things that solve customer problems, and that the customers (who are paying the bills after all) are asking for. That you do not see more security capabilities is an indication of what end users are asking for. If the sales guy tells customer XYZ that he can’t get this other feature — that he asked for — because he can instead get this nifty other security feature (that he didn’t ask for) … what is the customer response?
Fair question anon. I’m looking at a SCADA or DCS and saying what is the:
- least secure IP connected system on a newly purchased ICS
- IP connected component where little or no security progress has been made in the last ten years
- component that can actually directly affect the process if hacked
The answer for all three are the PLC/field devices. SCADA and DCS application vendors have made significant security progress, some more than others.
We shouldn’t get fixated on Stuxnet. What Stuxnet and Beresford and others are showing is how easy it is to make these PLC’s do whatever you want if you get logical access.
And my emphasis in this blog post is that ICS Security Experts or Gurus, talented people, should not be telling customers at user conferences that this total lack of security from the PLC vendor is ok if you just deploy this compensating control.
Amazing how we wouldn’t dream of paying for something online without end to end ssl encryption, but somehow in a control network it’s ok. We’re playing with a lot more than a Visa with a $2500 limit here folks. Also, I get the deterministic argument that so often gets bantered about when end to end encryption is brought up. Make it a choice to use it or not. There are so many applications that do not require determinism that could benefit from encryption. If you are building applications that require >10 ms of precision you should really have a gapped dedicated network with harsh access controls anyways.
Vendors need to start taking action on this!
It’s simple.
End to end encryption plus an app whitelisting product closes the circle on zero days. The app whitelisting prevents the windows vuln access, and the end to end encryption protects the controller. Couple that with some good secure network architecture and things start looking a whole lot better.
The problem with end-to-end encryption is what are the endpoints? Since the endpoints are sensors and actuators, then the PLC just introduces a key management nightmare as well as a central point of failure, since it needs to decrypt the messages and do logic operations.
My question: why do we even have PLCs? Processors are cheap these days. I buy dev boards with 400Mhz ARM11s which feature all kinds of niceness, like an MMU and on-die high-precision A2D for only $50-100 (including a nice LCD). Removing PLCs could open some interesting opportunities to make control systems more secure, while only increasing the cost of the field devices a little bit.
Put the brains of a plant in the sensors and the actuators. The HMI could monitor the sensors and actuators (essentially sniffing the network) to make a nice representation of system state and could ultimately override actuator decisions. Messages would originate from sensors and be timestamped+digitally signed. Messages would also originate from the HMI, where employees could digitally sign their commands with their smartcard. Actuators would perform logic based on the HMI input (if any) and status from the data monitors, verifying the digital signature of each message before executing their logic.
The glue required to put it all together is a fairly simple compiler that understands which actuators need which inputs to perform the control logic prescribed by the programmer. The protocol between sensors and actuators just needs to be smart enough to get sensor messages to the actuators that need them, and to perform some basic time synchronization. You could have a box that you call a PLC (really a network router or something) if it would make the control systems people feel better. The worst that a bad guy could do with such a system is denial of service. Actuators should be programmed fail-safe, and ought to move to the fail position if they don’t see updated sensor information.
I wonder if any standards group would be interested in making such a protocol a reality? It seems like such a system would go a long way to preventing future problems. I wonder a bit what it would take to get manufacturers to start building the things…
Reid,
Not all ICS is created equal. While your approach would work well in a traditional PLC environment, it’s not a good model for a full Distributed Control System (DCS) environment.
In a DCS, all the logic for governing everything is within the DCS controller, not at the HMI. This logic is developed, and then loaded into the controller, where it runs autonomously with minimal operator intervention.
The controller hardware is specifically optimized for these control related functions: Most contain an FPGA that allows specific logic to be ‘burned in’ which makes operations run much faster than any general purpose processor could run the logic(analogous to a dedicated GPU for graphics). It also has significant capability with multiple automation buses, protocols, methods of interface, etc that are superior to interfaces by the standard HMI platform.
Its conceivable to add this capability to specific servers in the DCS, but this approach was abandoned in favor of the controller a long, long time ago. Maybe some of those reasons are less relevant now.
This is why controller vulnerabilities are such a big deal. Once that logic is in the controller, there’s not many mechanisms outside of standard integrity verification being done to verify that it’s the logic is functioning as designed.
In addition, there’s a problem with scale. If we eliminate the ‘middleware’ between the HMIs and the sensor/actuators, there will be a large increase the level of traffic that the network must support. Typically, the way to deal with this is the use of multicast groups or broadcast, but this adds processing overhead to your sensors, which then increases cost per sensor and maintenance overhead per sensor. That middleware serves a useful purpose right now.
Mike T
Private Citizen
I am glad that what started out as a casual conversation on Twitter has caused so many to offer suggestions on addressing what is obviously a very serious problem that is not going to get solved in the near future.
To start, it would be awesome to offer end-to-end encryption of ICS comms like “anon” mentions, and guess what … Siemens has announced exactly such a product. They have also announced application whitelisting as well. So you can see, Siemens is trying to address these weaknesses and make their systems more secure. Can you say the same for Rockwell Automation, Honeywell or Emerson?
Just a point of clarification … end-to-end encryption would not have stopped Stuxnet … which is why I think those that have not should read the paper I co-authored with Eric Byres and Andrew Ginter.
It is easy to say we need an automobile that gets 100 miles per gallon or more, but technology may not be there at this time. For ICS and PLCs, most legacy products just do not have the horsepower to perform advanced security controls, which is why compensating controls must be a part of any defense-in-depth strategy.
Dillon’s recent disclosure on the S7-1200 exploits vulns within a web server that is an optional component of the CPU, and is something that anyone serious about security would not even consider. This is why Siemens offers this functionality as an “option”. I have had a high rate of success exploiting these trivial web engines inside many PLCs, including those from major suppliers, but rather than make a big deal about the vuln, I suggest to clients to stop using a technology that is not really appropriate for ICS control applications! Most agree.
Having done extensive research with Stuxnet in both the way it infects hosts and propagates on networks, very few technologies would have stopped it! Where nearly all systems fail assessments today is the lack of their ability to (1) detect an attack in progrss, and (2) contain and minimize the consequences of the attack. We need to stop thinking that ICS hosts can prevent all attacks, and start addressing what we do when an attack occurs. I would like vendors to start by allowing security assessments as part of their customer witnessed tests prior to commissioning. I have been pushing this for several years with little acceptance from the vendors.
I hope to see many of you at Siemens Automation Summit in Florida!
Here are our comments on security:
a. Web security may make it harder for hackers to get in, but won’t keep adept hackers out. There exist holes in software to jump through for a hacker that understands programming.
b. A good programmer can figure out how control system software works and can change the code to do what they want it to do. It’s a matter then of finding a vulnerability to allow access to the control system to drop in and run the modified code.
c. With systems connected to the internet, hackers will always be at least one step ahead. Hackers have time to spend looking for application vulnerabilities. The people programming applications aren’t really good at secure coding practices, and even if they are, a hacker who understands coding can usually find a way around it.
d. As long as a system is communicating wired or wirelessly to other systems that are eventually communicating with the internet wired or wirelessly, they can be hacked.
@Krag and others: Note that even if there were keys present on the network, it does not protect those who have contractors/employees descending on to plants to put code in to PLC gear from their laptops.
The keys are already on those laptops. They would have to be if one is to get anything done. Furthermore, insisting on keys makes the job of interfacing with an HMI that much harder. An attacker only has to get to the HMI or to the Laptop development environment. And since the keys are already there, why did we bother ourselves with this hassle? Just hook the software when it downloads code to the PLC and insert a little something extra, the way Stuxnet did.
Furthermore, we still have to get setpoints and commands in to the PLC in real time, sometimes from other PLCs across the plant. To issue those commands and setpoints, one would presumably also need keys under Dale’s scheme. Because these calculations need to happen in real time (the scan times of many PAC gear systems can be as little as one millisecond), it is doubtful that the keys will ever look like the full blown X.509 keys we use on the Internet. More likely, it will be a symmetric key and there won’t be many of them and they won’t change very often.
So you’re back to square one.
Dale, Joel, Krag, and so on: I disagree with the notion that you should maintain keys for operational PLC work. It is one thing to sign code and to maintain traceability. It is another thing entirely to say that we should process these keys in real time. I see enormous technical obstacles to making that work and I don’t see much benefit in return.
Disclaimer: I’m not an IT Security expert, and I don’t even play one on TV. I’m just another process engineer who doesn’t understand the way security people think.
Jake Brodsky
In a way, Dale has set the authoritative point of conclusion in his blog post two weeks ago, titled The Lost Decade (http://www.digitalbond.com/2011/05/31/the-lost-decade/). Many of us, including myself, have reason to question what we did wrong. Right now I cannot fight the feeling that we’re seeing ICS security imploding: We see people who always argued that default-allow is nonsense advertising bolt-on security products (bolt-on is similar to your specific deny rule), and others who tell their customers to be strict on passwords while continiuing to use hard-coded passwords in their products, or who discourage USB thumb drive use while at the same time requiring the user to active their software products by an activation key that’s distributed on… a USB thumb drive. While even security experts will hardly be able to make sense out of this, control system engineers in the trenches certainly won’t. I believe it’s time for a change, and that we must find ways for the cited folks in the trenches to get out of the haze. ICS security never sold well; it certainly won’t sell better if major stakeholders and thought leaders start to communicate contradictory or meaningless messages.
I agree that we should not be letting the PLC/DCS/SCADA vendors off the hook with regards to security. Letting Siemens off easy certainly will not be my message to the attendees at the Siemens Summit in Orlando. I don’t think that is John’s or Joel’s message either. And to their credit, Siemens did not place any restrictions on my talk. To even invite me to speak at their event is commendable – I have not pulled my punches in my criticism of Siemens disclosure process during the whole S7-1200 vulnerability issue (http://www.tofinosecurity.com/blog/protecting-siemens-s7-1200-plcs-against-security-vulnerabilities-part-33). I don’t intend to hold back at this event either.
What I want to do is to improve the overall security of SCADA and ICS through as many channels as possible. I believe that Stuxnet has a lot to teach the operators of control systems on where the failings are in typical control system architectures, security processes and products. Certainly Siemens has to take some responsibility for the security short comings in their product, but the process designer, the integrator, and end-user also must accept their share of responsibility going forward. After all, end-users could have demanded secure systems a decade ago via their RFPs. For a number of reasons they did not and still don’t.
In my opinion, it is best that we use Stuxnet (and other worms) as an opportunity to learn the realities of ICS security and how the responsibilities for security need to be assigned. For example, you bring up the very valid point of the fantasy of the air gap (which is a topic I plan to discuss in depth in a future blog). Vendors should not hide behind the air gap fantasy in their security notices, but neither should the end user when he/she talks to his management or designs a control system. The entire industry needs to move past this and learn to deal with the reality of connected SCADA and ICS. Stuxnet is an excellent vehicle to drive that point home.
In an ideal world, any other solution besides more secure PLC products would be unacceptable. Unfortunately, we all know that creating a truly well designed and secure product takes a lot of time. It takes even more time for the end-users to deploy these patches and/or new products in the field. Thus the world is stuck with insecure legacy systems for many years to come. And while we wait for truly secure ICS / SCADA product (a holy grail if there ever was one), the end user needs compensating controls to mitigate the risks present in today’s systems. The end user also needs to understand where he or she needs to focus their efforts – no one has unlimited resources, so clear guidance in this area is also important. It is these three messages – what the control system owner can learn from Stuxnet, what they can do to mitigate current security failings and what they need to focus on. That I hope people will hear Joel’s and my presentation on June 28 at the Siemens conference (http://www.tofinosecurity.com/article/2011-siemens-automation-summit).
Eric
NSA will help the networks scan for known threats/signatures, at the rate of 100 Gb/millisecond. This scanning does not help find the unknown ones. The malefactors are always ahead. It is a never ending battle.
@Ralph Amen to that! The message needs to be stronger and more clear. The operators I have met through my travels need help in digesting the ramifications and understanding the potential solutions. Those charged with managing their ICS environment have been doing the same job in the same manner for decades. The ICS security story is riddled with confusing and contradictory statements – not to mention the security regulation language they have been exposed to of late doesn’t exactly speak to the issue using a system operator lexicon.
I have yet to meet a systems control engineer who didn’t take amazing pride in his/her role in assuring the reliability of the bulk electric system (or BPS if you prefer). I’ve met those who have been on the job for a couple weeks and those who built from the ground up a large majority of the western interconnect – they all really, really want to do the right thing. As security practitioners and tellers of security concerns we have to continue to find better ways to tell the story.
I truly appreciate the work of Dale, Eric, Joel, Ralph and many others. This is important and getting out there sharing the knowledge is ever so helpful and encouraging to me personally as I work toward building solutions to improve collaboration and information sharing.
[...] his blog article, Fix the Problem, Stop Bailing out Vendors, Dale Peterson made a brief comment about “fantasy of the air gap”. It was an important [...]