.gif)
Where is the outrage?
We hoped for at least the start of outrage demanding fragile and insecure PLC’s in the critical infrastructure be either fixed or replaced.
Of course, we expected some aimed at us for pointing out the problem and creating tools to make it easy to see. Unfortunately that seems to be the extent of the outrage and most are settling back into the “we all knew PLC’s are vulnerable” status quo. This is not a complete surprise given Siemens was able to avoid admitting or fixing the S7 Stuxnet issues despite the publicity.
But it’s not over, there are more Basecamp Metasploit modules and other tools coming, and we are still hopeful that as people see how easy it is to compromise a PLC the reality will be too much for continued inaction.
Important Clarification
The biggest Basecamp misunderstanding is the GE D20 Metasploit Modules take advantage of some 0day vulnerability. This is not true.
The GE D20 Metasploit Modules do not leverage a vulnerability at all. One module just downloads the configuration file and extracts the user credentials – a feature in the product. The other uploads (TFTP) a text file containing commands to the GE D20, which the D20 then happily executes and writes results to another file — again a feature of the product.
Reid did find strcpy and memcpy overflows and other “0days” in the product, but those are not worth the trouble to an attacker. The main reasons we did the additional testing was to complete the results table and determine how fragile the device was. Extremely fragile as frustrated students in the PLC Hacking class found out. Even an nmap fingerprint crashes the D20.
Interim Results
The vendor response has been mixed and incomplete after one week. Fixes would be a surprise in one week, but preliminary analysis and communication should be done by now.
GE – Complete radio silence. We did not expect a lot of questions since this device is beyond saving, but we haven’t been able to find a customer bulletin or public statement. Please send us a link if you see one. How hard is it to say: “the GE D20 is an old system designed well before ICS security was an issue. We have a secure replacement, the xxx, that will be available in yyy. It will have the following security features and we strongly recommend any of our customers concerned with security plan on moving to this replacement product as soon as possible.”?
Rockwell Automation – They reached out to get the details, beyond what they heard at S4. RA issued a broadly worded bulletin that there are security issues in the ControlLogix and MicroLogix, and their Security Taskforce is looking into them. There are three more bulletins, but they just provide generic guidance. It’s a bit disappointing that the Security Taskforce has not put out more helpful info to customers one week later. We will have to wait for the more detailed bulletin and any patches or other remediation recommendations.
Schneider Modicon – Nothing. No questions, no contact. There may be a bulletin out to customers. Send us a link if you have one. Unlike the GE device, the Quantum should be able to be fixed/patched.
SEL – They reached out to get all the details and already have begun an effort to document the CAL level authentication in the older products. Hopefully they will respond to the SEL blog entry to explain why they believe the CAL level is required.
US Government – Incomplete for Basecamp. Fail on PLC Security. ICS-CERT quickly published the Basecamp bulletins, but nowhere does ICS-CERT or anyone else in the USG come out and say that owner/operators must replace the fragile and insecure PLC’s. Instead its simply publish the information, a la Stuxnet, issue some generic guidance such as defense-in-depth, and go back to sleep hoping nothing else bad happens.
When will they just bluntly tell owner/operators that they need to replace these fragile and insecure PLC’s? It’s not going to be done in months or even a year, but we are going on year 11 now. My expectation is any government would identify the most critical ICS and put them on a timetable either through the bully pulpit or regulation if it is necessary. Regulation has not gone well, but when has the USG or other governments spoken honestly about what needs to be done and the negligence of owner/operators if they don’t do it?
Automation Press – Fail. Basically a repeat of the behavior when Langner found the reality of Stuxnet. In that case members of the automation press had a two week head start and failed to write anything until the mainstream press picked it up. In this case, silence. It would be understandable if they did not like the result and chastised us, or even didn’t mention the Basecamp team in an effort to not encourage such efforts.
But how do you not tell your owner/operator readers that their PLC’s have proven to be fragile and vulnerable AND that there are tools out there to make it easily demonstrable. I don’t expect them to pick it up until it is everywhere else, and they have no choice. Friends have told me to get off this rant, but the automation press is a major source of information for ICS engineers and what they get are thinly disguised advertising.
My greater hope is the non-automation press, IT, IT Security and mainstream will eventually get the message to C-level executives.
Image by RTP
Come on most vendors will not want to highlight the issue most customers are not plugged into this. Even if they are there’s no incentive to deal with it. In fact there’s a strong incentive to do nothing – it costs money. If you want action these events will not work. Money is the only language they understand.
Anyone that hangs a PLC on the internet with no protection deserves to get hacked. Reasonable network security makes the majority of your findings worthless. I agree with your friends, get off this rant!
Jim
Jim – I agree with you that a very small number of the Internet accessible ICS are part of the critical infrastructure, and I talked with Eireann Leverett who agreed after all his Shodan work.
But I’m not sure we ever mentioned once that the outrage with the insecure and fragile PLC’s has anything to do with Internet accessibility. We typically view the corporate network as untrusted, and there is almost always communication between the corporate network and the control center.
Dale
I don’t think Project Basecamp is going to have the impact you guys were hoping for. Expecting it to be ICS vendors “Firesheep” moment just isn’t practical, and not even a remotely close comparison. To fix Firesheep, the affected websites simply had to make a relatively simple change on their end (as simple as anything is for very large scale websites), and it immediately went into affect for all their users. Even if this triggered massive security focus by the vendors, I would be shocked if more than a tiny fraction of PLCs got the required updates. The mindset of most in the ICS world is “if ain’t broke, don’t fix it.” We know security lackings technically should classify things as “broken”, but the general perception of the majority isn’t going to be as such.
I’m not sure we’ll ever see the kind of security improvements across the board in the ICS world that we’ve seen from other vendors, Microsoft in particular is a good example. Around 10 years ago, Microsoft had one nasty, easily exploitable vulnerability after another, which frequently turned into mass-replicating worms. Their customers demanded better after being bitten over and over and over by their vulnerabilities. Microsoft still isn’t perfect, nor is any vendor, but their focus on security has paid significant dividends. They still have serious vulnerabilities on occasion as any complex software will, but things are vastly improved from a decade ago. It’s been around 8-9 years since the last widespread self-replicating worm exploiting Microsoft software, which in and of itself speaks to the success of their focus on security, considering the frequency of such events in the years prior.
You could point to the same with Adobe and their horrid security track record with Acrobat/Reader. After a huge number of machines were exploited by malicious PDFs, they’ve finally added sandboxing to the latest version in addition to more focus on security in general, which appears will greatly reduce the impact of any future vulnerabilities. It took a bunch of machines getting owned by malicious PDFs, pressure from their customers, and IT people starting to recommend using anything other than Adobe Reader to view PDFs, for them to put focus on it.
History shows significant security strides have almost exclusively only been made not where security researchers start putting out vulnerabilities, but when those things are actually being widely exploited and impacting customers. Are the ICS vendors going to wait until that happens? Will it ever happen regardless of how many vulnerabilities are made public? Given the general lack of connectivity these systems have to the Internet and untrusted networks (in most cases at least, not that air gapping is perfect or even exists in a lot of instances, but it definitely makes exploitation less likely and more difficult), I’m not so sure it will.
Some ICS vendors are at least paying more attention to security these days, but don’t seem to be taking things as seriously as the vendors whose customers have been owned by vulnerabilities in their software. That’s why the outrage is missing in my opinion – nobody has felt the impact of any of this (outside of Iran with Stuxnet), and I get the feeling most control engineers aren’t paying a lot of attention to the security community. Just like most IT people didn’t pay a whole lot of attention to vulnerability researchers putting out Windows holes back in the ’90s. I thought Stuxnet may have been the trigger to get people to pay more attention, and to some degree it was, but I think it’ll take another instance or two of similar malware that more widely impacts ICS to start creating widespread customer outrage.
Chris,
Thanks for taking the time to write that long comment. Two quick comments on your comment:
- The “Firesheep Moment” was aimed at the owner/operators and government regulators, not the vendors. Only when the end users say I can’t live with this crap will the vendors change.
- I have heard numerous times and for years that nothing is going to change until systems go down. Natanz went down and there were no changes. Now people are saying it has to be a widespread attack affecting large number of people in countries where the vendors are located. I guess I’m not willing to sit and wait for that to happen. If I was, I’d get out of the ICS security space and do something else. Why waste one’s time.
Dale
Dale,
The issues that Chris raises are valid – I found myself nodding my head vigorously as I read his response (which was way more eloquent than I could’ve managed).
In addition, you’re going after the audience least able to effect the change. It can happen during refresh cycles, perhaps, but consider a utility that has a huge investment in a single vendor and a multi-stage replacement cycle. Given the natural support advantages of having a homogeneous environment (at least for a given operation), there’s a pressure to keep going with 1) what you know, and 2) what’s easiest (and cheapest) to support.
Arguing that security vulnerabilities that MAY be released at some point in the product’s lifecycle will skew the cost equation is difficult to evidence as it requires a long-range view that ignores short-term needs.
I was fortunate enough to see a very similar thing play out in the mid ’90s when I was heading up network engineering for a major bank. Cisco was the dominant player, though you had Juniper, 3Com, Cabletron(!) and a few others with some technically better products. When the slew of Cisco vulns came out (there were, if I recall correctly, at least half a dozen in a 6-month window), we had to decide whether we were 1) going to trust Cisco to eventually (that is, before we were attacked) fix their product, or 2) going to incur a management nightmare by purchasing devices that weren’t vulnerable AT THIS TIME.
Now, the argument in favor of disclosing vulnerabilities is that we can see how responsive certain vendors are to fixing their systems, and we can use that history to make strategic purchasing decisions. But this argues against a “firesheep moment” and more towards a “firesheep era”. It’s not going to be a seminal moment. It’s going to be a historical one.
What would be awesome is if someone kept track of vendors and their response times to fixing security issues (time from disclosure to patch). That would be a metric that could be widely used as a discriminator, and one that asset owners could use to put pressure on their own vendors (e.g., “Why aren’t you in the top 3?”)
Anyway, just my thoughts on this. Reasonable people can disagree vehemently on this issue, which is why we haven’t seen an industry consensus. Thanks for all your work in bringing the issues to light.
Seth,
It is real easy to get lost in this gray area that you describe, but let’s look at something that is black and white – - the GE D20ME. As I’m sure you know this device is widely used in electric substations as well as other critical infrastructure SCADA and DCS.
1. It is an ancient device, that already had obsolete hardware and software when 9/11 occurred.
2. Most security types who have encountered it quickly learn how insecure and fragile it is.
3. It is insecure by design, and it can’t be fixed.
4. Over the past ten years GE has provided their clients with no upgrade path, and to my knowledge does not have one now (based on what GE customers are telling me)
Question: At what point does an owner/operator who is supposedly running something that can’t ever go down say I need to replace this fragile and insecure device that has no future? (and remember we are going on eleven years now)
Question: At what point does DHS, DoE, NERC, FERC say this device must, or at least should, be pulled?
I think the ICS security community gets hung up some time on solving all the problems, when in fact we need to just start with the easy ones. In this case, develop a plan to replace all your GE D20ME’s as soon as practical. Or demand an answer on how Modicon is going to fix the insecure by design problems in the Quantum, and then determine if you can wait for the fix or develop a plan to replace.
Dale
Dale,
It’s an extreme example, but I think you can look at it like this (answers to your two questions and some add’l commentary follow):
1) You replace it when it’s fully depreciated, or when the loss expectancy of keeping it in service outweighs its residual value (see below for more on this). If the device has a usable life of 20 years, then the AO would have to find a way to write off 9 years of depreciation. (I don’t know whether this is the case with the D20ME, having never seen one.) Measuring the loss expectancy requires access to information (see below).
2) “Must”? When those agencies are accountable to the shareholders and customers, or are required do so out of some exigent national security concern. “Should?” This one’s not as easy – when they’re comfortable accepting the risk that the vendor will take them to court for anticompetitive activity (or other such nonsense). I don’t think the government should be (or currently is) in the business of saying, “buy this, don’t buy that” – the risk decisions are left to the asset owners in a privatized environment. I believe (via experience) that vendor neutrality is a paramount goal of these organizations.
Finally: are there NO other mitigating controls that can be put in place to reduce the risk of exploitation? I find that hard to believe. It’s not necessarily a matter of pulling the device, it’s figuring out how to protect it against the known attacks (and, if you’re really on your game, perhaps knocking out some theoretical ones at the same time). There are lots of devices like this – for one admittedly less-important example, look at the consumer-grade VOIP TAs, where the vendor is no longer offering updates, the devices are fundamentally insecure, and the customer must find a way to securely maintain them since they’re all that their VOIP provider offers. Is it worth breaking a contract and paying an ETF to move to another provider that may or may not have similar issues with their OEM equipment?
But to suggest that AOs aren’t capable of making these risk decisions is probably not the right way to go. What I think we can say with certainty is that without good information at hand, the AOs aren’t going to be able to make correct decisions – and historically, the AOs haven’t been given this information (the vendors have had exclusive access to vulnerability information supplied by the labs via CRADA, for example). This is why responsible disclosure is so important.
The AOs should have access to the information so they can 1) make decisions at the right time based on it, and 2) be held accountable for having known that information. I don’t think public disclosure is necessarily the right approach here – specifically because of the long implementation time for remediation, but again, this is where reasonable people can disagree.
Thanks for the discussion.
PS: An interesting exercise might be to roleplay the discussion with someone being the security guy, someone else being the CFO, and another person being the person in charge of operations. You could even add some govvies to the mix, and some regulators. Maybe it’ll be an easier discussion than I currently think it will be.
Seth – not sure we are going to find common ground on this. A few last points:
1. Our hope is you are correct that Asset Owners are capable of making risk decisions and are just lacking the info. Basecamp is an effort to make the information abundantly clear.
2. Your example “9 years depreciation” vs the cost of the entire plant. If you consider it is simple for a skilled attacker with logical access to plant and the “insecure by design” field devices to cause massive damage, then either Asset Owners don’t know this or are counting on no one wanting to launch such an attack. Neither is very encouraging. People keep telling me nothing will change until there are significant and very harmful attacks — again not very encouraging and defeatist.
3. An engineer would never design a process with this level of fragility, lack of understanding of what is happening, lack of certainty in operation, … that exists with the “cyber” components. They would lose their PE and be the laughingstock of the profession.
Personally I’m going to try to stop using the term critical infrastructure, because it cannot be truly critical if it continues to be allowed to be in this fragile and insecure state.
Dale
Hi Dale,
Can they ever really be “last points?” ;-)
In response…
1. I believe most Asset Owners are capable of making risk decisions and are just lacking the info. Unfortunately, I believe the reality is much more embarrassing – that the organizational structure of our industry substantially inhibits how speed and effectiveness of a “Firesheep Moment.” I’m going to speak to the electric power industry in my examples because that is the one I know. [Insert disclaimer here about not understanding the corresponding nuances of other ICS sectors…]
The electric power industry is absurdly capital-heavy compared to the general IT market. If you look at how many $$$ worth of equipment each engineer is responsible for acquiring compared to their counterpart in commodity IT, the dollars get pretty distorted. This, coupled with the size heterogeneity of Asset Owners means that a vendor can garner a notable portion of market share by targeting specific individuals. This pushes the battle toward the case-by-case end of the spectrum and away from the pissed off masses with pitchforks and torches. Psychologically, this removes a substantial amount of fear on the vendor end of the equation – vendors are no longer scared that they are going to lose all their market share overnight because that fundamentally can’t happen. All they have to do is figure out if their next customer knows about the issue, figure out how to spin the whole story, send an army of slick sales engineers to overwhelm the already overtaxed AO, push the discussion in every direction except security (functionality, purchase price, etc.), and just for insurance, make sure all of the executives are the best friends EVAR.
2. Pointing to the absurdity of “9 years depreciation” vs the cost of the entire plant sounds great in blog discussions. Unfortunately it is not how most projects of which I am aware are funded in the electric power sector. Most AO’s negotiate sizable contracts with vendors for a fleet of equipment that will serve a specific function across their infrastructure in the interest of making the price somewhat manageable. These assets are actually purchased and deployed over a long period of time and across multiple projects (i.e., plants), albeit under the umbrella of the large contract. Only in the event of a “green field” site is a plant built entirely at once, and even then most of the equipment is already chosen before the design even begins. At this point it is just an exercise of how to piece it together.
Even if we could hypothetically move to the residential house-building model where each plant was designed as a whole and all the parts were picked for that plant, what kind of impact would this have on the AO’s management overhead? How many different types of systems would they have to maintain? So now we are stuck with the dilemma of justifying the very tangible purchase price of one brand of device vs another against this hypothetical cost-of-the-plant thing. Have you ever been in a discussion where an AO is trying to justify cost to a public service commission?
3. Your point about engineers designing fragile systems, becoming a laughing stock, losing their PE, etc. is your strongest. Why can’t we hold engineers accountable when their systems fail?
Best regards,
Darren