The Japanese Ministry of Economy, Trade and Industry (METI) is stepping up its ICS security activities by forming a private sector led Control System Security Center. A recent Yomiuri Shinbun article also describes 10 virus incidents in Japanese control systems.
Another possible US ICS security hack makes the news, this time purportedly affecting railway signaling for two days in December. From Zetter’s article, “A DHS official added that after more in-depth analysis of the incident, it did not appear to be a targeted attack aimed at the railway and halting service, but was more of a random incident that simply hit the transportation entity.” A subsequent Mills article has DHS disputing hacking as the cause of the incident.
The US Government Accountability Office (GAO) issued a report titled, “Cybersecurity Guidance Is Available, but More Can Be Done to Promote Its Use“. The title is an accurate report summary, and the report suggests that DHS should tell sectors which guidance documents they should follow out of the plethora that are available. There also is a section that compares CIP to SP800-53 and comments they are very similar.
Industrial Defender announced their Automation Systems Manager (ASM) this week and was showing it at the SANS event. It is billed as one management application to handle everything, compliance, configuration, event, … We will review this product offering in more detail soon.
We hoped for at least the start of outrage demanding fragile and insecure PLC’s in the critical infrastructure be either fixed or replaced.
Of course, we expected some aimed at us for pointing out the problem and creating tools to make it easy to see. Unfortunately that seems to be the extent of the outrage and most are settling back into the “we all knew PLC’s are vulnerable” status quo. This is not a complete surprise given Siemens was able to avoid admitting or fixing the S7 Stuxnet issues despite the publicity.
But it’s not over, there are more Basecamp Metasploit modules and other tools coming, and we are still hopeful that as people see how easy it is to compromise a PLC the reality will be too much for continued inaction.
Important Clarification
The biggest Basecamp misunderstanding is the GE D20 Metasploit Modules take advantage of some 0day vulnerability. This is not true.
The GE D20 Metasploit Modules do not leverage a vulnerability at all. One module just downloads the configuration file and extracts the user credentials – a feature in the product. The other uploads (TFTP) a text file containing commands to the GE D20, which the D20 then happily executes and writes results to another file — again a feature of the product.
Reid did find strcpy and memcpy overflows and other “0days” in the product, but those are not worth the trouble to an attacker. The main reasons we did the additional testing was to complete the results table and determine how fragile the device was. Extremely fragile as frustrated students in the PLC Hacking class found out. Even an nmap fingerprint crashes the D20.
Interim Results
The vendor response has been mixed and incomplete after one week. Fixes would be a surprise in one week, but preliminary analysis and communication should be done by now.
GE – Complete radio silence. We did not expect a lot of questions since this device is beyond saving, but we haven’t been able to find a customer bulletin or public statement. Please send us a link if you see one. How hard is it to say: “the GE D20 is an old system designed well before ICS security was an issue. We have a secure replacement, the xxx, that will be available in yyy. It will have the following security features and we strongly recommend any of our customers concerned with security plan on moving to this replacement product as soon as possible.”?
Rockwell Automation – They reached out to get the details, beyond what they heard at S4. RA issued a broadly worded bulletin that there are security issues in the ControlLogix and MicroLogix, and their Security Taskforce is looking into them. There are three more bulletins, but they just provide generic guidance. It’s a bit disappointing that the Security Taskforce has not put out more helpful info to customers one week later. We will have to wait for the more detailed bulletin and any patches or other remediation recommendations.
Schneider Modicon – Nothing. No questions, no contact. There may be a bulletin out to customers. Send us a link if you have one. Unlike the GE device, the Quantum should be able to be fixed/patched.
SEL – They reached out to get all the details and already have begun an effort to document the CAL level authentication in the older products. Hopefully they will respond to the SEL blog entry to explain why they believe the CAL level is required.
US Government – Incomplete for Basecamp. Fail on PLC Security. ICS-CERT quickly published the Basecamp bulletins, but nowhere does ICS-CERT or anyone else in the USG come out and say that owner/operators must replace the fragile and insecure PLC’s. Instead its simply publish the information, a la Stuxnet, issue some generic guidance such as defense-in-depth, and go back to sleep hoping nothing else bad happens.
When will they just bluntly tell owner/operators that they need to replace these fragile and insecure PLC’s? It’s not going to be done in months or even a year, but we are going on year 11 now. My expectation is any government would identify the most critical ICS and put them on a timetable either through the bully pulpit or regulation if it is necessary. Regulation has not gone well, but when has the USG or other governments spoken honestly about what needs to be done and the negligence of owner/operators if they don’t do it?
Here it is – the video of Reid Wightman’s Project Basecamp presentation. It is full of technical details and a real example of the technical detail we aim for S4 presentations. Reid did a fantastic job on the GE D20 and Modicon Quantum (with an assist from Ruben), and he coordinated and presented the other Basecamp researchers findings. It’s 90 minutes and 1.28GB.
Our only regret was Dillon Beresford and Ruben Santamarta were not able to attend and present their own work on the SEL-2032 and Rockwell Automation ControlLogix respectively. I’m sure they could have added even more detail and deserve credit for their strong work.
You will also notice that there is no SCADASEC 101. Reid just dives into the technical details which is what we ask our S4 speakers to do.
A Project Basecamp result that has been widely overlooked is that some of the devices were much more robust and secure than others. The Modicon Quantum and GE D20 were clearly the worst. The Rockwell Automation / Allen Bradley controllers and Koyo/Direct LOGIC were in the middle, the SCADAPack bricked early so it is incomplete, and the SEL-2032 Communications Processor was the least fragile and most secure out of the devices tested.
It is a slightly unfair to directly compare the devices since some are full featured, deluxe PLC’s and the SEL-2032 is a communication processor that typically allows a WAN connection to access multiple serial devices in a substation. However many of the “Insecure by Design” features and vulnerabilities in these PLC’s could have been in the SEL-2032.
The most interesting and potentially serious issue with the SEL-2032 was the hidden CAL access level and default password. The CAL access level provides debugger-like access to the device, e.g. list running processes, dump and overwrite arbitrary memory, … Dillon Beresford found this in his firmware analysis, a very impressive piece of work, but it actually is listed in some product documentation – just not SEL-2032 documentation.
To get to the CAL access level you need to login to the ACC access level, authenticate to the second level (2AC), and then you can get to the CAL access level. Dillon and the Basecamp team discussed whether the CAL access level is a backdoor. The consensus was it’s not a backdoor because of the authentication requirement to ACC and 2AC access levels to reach the CAL access level. Dillon was calling it “a hidden debug environment for internal diagnostics and SEL engineers.”
However, debuggers used in product development are typically removed prior to release. At a minimum SEL should better document the CAL access level so owner/operators understand the risk if an attacker can reach this access level and recommend the default password be changed. We would rather see it removed because a skilled attacker with debugger access is likely to do some damage.
Other positive and negative findings include: Read More
With the previous vigilante article answering the why of Project Basecamp, let’s get into what was found. This will take a while, and the full content will make its way onto the Basecamp research page on digitalbond.com and in the S4 Proceedings e-book.
The GE D20ME (worked on by Reid Wightman) faired the worst in Basecamp, and this is an extremely popular PLC in electric sector. There are numerous vulnerabilities and bugs that can either crash it or let an attacker take complete control. Let’s start with the two Metasploit modules that have been released via Rapid7 for the GE D20ME.
d20pass module – the complete configuration of the D20 is downloaded from the TFTP server, including the usernames and passwords. The username and passwords are displayed and then stored in the Metasploit database for future use.
d20tftpdb module - This module demonstrates what Reid calls an asynchronous command line. The D20 has a “feature” where you can TFTP a text file with a set of commands to command.log — commands such as edit arbitrary memory, list processes, debug processes, see where in memory processes live, … The D20 will run the commands and write the results to response.log. Since it is TFTP there is no authentication. The Metasploit module automates all this so it appears you have a command line. After a command is entered, the module creates the file, sends it to the D20 and reads the response. (Rapid7 Note: this module is currently still in the unstable Metasploit branch pending a little more QA work on getting this, pretty unique, command and channel all nice and automated.)
These are examples of “Insecure by Design”. They are features intentionally built into the product. But when you combine these two features you have a very scary and dangerous situation. An attacker can download the entire configuration, study it, determine what series of commands he wants to run to alter or crash the process, and then TFTP that set of commands to the device at a time of his choosing.
In his Basecamp presentation Reid went into a number of buffer overflows that could be turned into exploits that run arbitrary code, but it probably isn’t worth the trouble.
The bigger issue with the code quality is the fragility of the device. It is so easy to crash this device through fuzzing or other testing. Reid had one crash that took 50 hours to recover from despite his precautions and experience.
Eric Butler’s Firesheep plugin for the Firefox browser made it simple for anyone who could operate a browser to hijack Twitter, Facebook and Hotmail http sessions in a coffee shop’s wifi. This security problem related to cleartext cookies that had not been addressed 2+ years after researchers disclosed it. After Firesheep the outcry from the users was so widespread that https quickly became a configurable option and in a few more months the default.
Eric Byres’ in his S4 recap mistakenly wrote that the Project Basecamp goal was to shame vendors into fixing the problem. We don’t expect vendor shame to result in any change. Our hope is that the resulting Basecamp tools, particularly the Metasploit modules, will make demonstrating the ease of compromise and potential catastrophic impact possible for any owner/operator, vendor, consultant or anyone else involved in ICS. C-level executives running the critical infrastructure SCADA and DCS will know beyond any doubt the fragility and insecurity of these devices. Hopefully they will find this unacceptable, demand their vendors offer a secure replacement, and spend the money to replace the PLC’s.
Admittedly this will take time, but ten years has passed already and there has been scant progress. If owner/operators don’t replace these PLC’s then the idea that these are truly critical systems that can never go down is a myth and significant fragility and insecurity is an acceptable reality. I covered this and more in my introduction to Project Basecamp.
Direct link to the Introduction to Project Basecamp. Note: Reid’s technical portion of Project Basecamp video will be online tomorrow.
The formal S4 Conference is over, and the researchers and attendees were great. This year we had more technical papers submitted and actually had to shorten times and reject some good work to fit it all in. The quality and technical detail in the presentations was at an all time high for S4. We challenged the researchers to present a lot of technical meat for this unique audience, and they came through. Thanks again to the researchers who presented and to their co-authors who were unable to make it.
We recorded the event and will be issuing the presentations over the next few weeks. Two presentations, Ralph Langner explaining the Stuxnet S7 code in line-by-line commented detail and the Basecamp PLC hacking presentation, got most of the attention outside S4. However there were many more that were strong, technical and important presentations that you will want to see, and we will write some blogs around the release of the videos that will help you decide whether you need to view it.
The other big part of S4 is the audience/attendees. Not only to they have to be skilled in ICS and IT security to understand the research, but they need to participate in the in session and out of session discussions. The audience got the research points, didn’t always agree, but voiced questions, suggestions, experiences. As the moderator of a completely open one hour “Great Debate” session, it is a relief to have so many attendee participants.
This morning, at our S4 Conference, Reid Wightman gave a detailed two-hour presentation on the Project Basecamp results. Project Basecamp had six great researchers looking for vulnerabilities in six different PLC’s / field devices, and the PLC’s took a beating. There were backdoors, weak credential storage, ability to change ladder logic and firmware, command line interface, overflows galore, TFTP for important files and so much more.
Digital Bond’s S4 has us flat out this week, but we will be blogging in detail on this next week, but here are some of the Basecamp basics.
Just a quick note of apology to loyal blog readers for the lack of information this week. This week is our S4 event in Miami Beach, and there are so many great presentations, people to talk to and hosting activities that it is hard to find time to write. I suggest you follow #s4ics to stay up to speed.
Day One was fantastic with talks on witch doctors, Langner’s amazing Stuxnet deep dive into S7 code, EMET, lost decade ICS vuln analysis, whitelisting x 2, 60 Minutes, S4 cocktail party and more.
We will have a blog and press releases tomorrow on Basecamp, and will catch up with all the news and details in the next week. We are recording S4 and will put up the videos soon.
The UK Government Centre for Protection of National Infrastructure (CPNI) published a list of 20 Critical Controls for Cyber Defence in conjunction with SANS. Many in the ICS world don’t follow SANS, so this distribution may reach a broader ICS audience.
The list is simply written, easy to understand and worth a read.
My immediate question while reading the list was the prioritization of these twenty security controls. The document addresses this at the end:
“The twenty controls are a baseline of high-priority ‘technical’ information security measures and controls that can be applied across an organisation to improve its cyber defence.”
CPNI could do better in prioritization. For example, #4 Continuous Vulnerability Assessment and Remediation and #20 Penetration Tests and Red Team Exercises would be an inefficient use of resources until a significant number of other security controls were in place.
Conversely, #5 Malware Defences, #8 Data Recovery Capability, #13 Boundary Defence, and #19 Secure Network Engineering would be items that would rate in the highest priority.
ICS and other networks will vary so it is hard to come up with a detailed prioritization that even most would agree on. That said, CPNI could provide additional value to the document by at least grouping them in some priority categories. They also could develop an ICS specific prioritization, and in that case could consider the security program maturity in the prioritization.
A two-week catch-up edition.
Where is the outrage?
A Project Basecamp result that has been widely overlooked is that some of the devices were much more robust and secure than others. The Modicon Quantum and GE D20 were clearly the worst. The Rockwell Automation / Allen Bradley controllers and Koyo/Direct LOGIC were in the middle, the SCADAPack bricked early so it is incomplete, and the SEL-2032 Communications Processor was the least fragile and most secure out of the devices tested.
While 
This morning, at our S4 Conference, Reid Wightman gave a detailed two-hour presentation on the Project Basecamp results. Project Basecamp had six great researchers looking for vulnerabilities in six different PLC’s / field devices, and the PLC’s took a beating. There were backdoors, weak credential storage, ability to change ladder logic and firmware, command line interface, overflows galore, TFTP for important files and so much more.
