Many asset owners would like a check box approach to security, where some independent, reputable organization certifies the system or component is secure by design. There are a growing number of security certifications that are trying to meet this need. Even if every ICS is a semi-custom system, a security certification could eliminate the need for each asset owner to assess the same security basics such as technical security controls, security development lifecycle and fuzz testing.
The problem today is insecure by design ICS software and devices can actually be certified as secure. ICS suffers from, in the words of the great orator President George W Bush, “the soft bigotry of low expectations”.
The best example is the ISASecure embedded device security assurance (EDSA) certification. ISASecure/ISCI was created by ISA originally to have a certification capability for ISA99 security standards. When the ISA99 standards that could serve as the basis for certification were delayed, ISASecure certification criteria were independently developed by the member companies (Chevron, ExxonMobil, Honeywell, Invensys, Siemens, Yokogawa). There are a lot of positives to this effort, but currently the overwhelming problem is the false sense of security an ISASecure Certification provides.
There are three parts to the certification testing:
Functional Security Assessment. This is where the technical security controls in the PLC/RTU are defined, and there are no data integrity requirements for Level 1 ISASecure Certification. A PLC can be certified as ISASecure and have no authentication of the source or data related to critical request packets. Look through the document and you will see that Level 1 has minimal required security controls. Even the nine controls with the word basic in the title are not required to be certified as ISASecure
Communication Robustness Testing (fuzz testing). The devices protocol stack is subjected to positive and negative testing … well, sort of. There are tests for Ethernet, IP, ICMP, TCP and UDP, but no tests for fuzzing of the control system protocols. The control system protocols are where the protocol stack bugs are most likely to be found.
Security Development Security Assessment. The SDL portion of the ISASecure certification. This section actually has stringent SDL requirements, even in Level 1. It is hard to see how a product developed under these requirements could have such weak technical security controls or vulnerable protocol stacks. The escape clause is usually related to the environment assumptions. If you assume this is running on a secure network that cannot be accessed by an adversary the security requirements melt away.
ISASecure would likely respond to these issues with the Levels concept. Certifications begin at Level 1 and gets tougher at Levels 2 and 3. And to be fair, Level 2 adds a substantial amount of required technical security controls. That said, would you ever assume any PLC that received an ISASecure Certification is insecure by design? Isn’t this terribly misleading? When you read the press releases from ISASecure and the vendor you would assume that purchasing an ISASecure certified PLC or RTU meant it had basic security controls.
And ISASecure may be the best ICS certification available today.
We can argue about how long it should take to replace existing insecure by design components and protocols in the critical infrastructure, but there should be no argument about new purchases and deployments. We need to request and expect more.
At this point in time, be very careful about accepting an industry created security certification as evidence of a product or system being secure by design.
Thanksgiving is over and S4x14 is filling up. Now is the time to guarantee your spot. Check out the agenda and register for Digital Bond’s S4x14, January 14-17 in Miami Beach.
Hotel Rooms
The last date the conference hotels are holding rooms is 14 December. After that they will be available to the general public. If rooms are available you can still get the S4x14 rate up until the event, but January in Miami Beach is in demand.
All of the classic rooms at the Marenas Resort are booked now, but there are still suites available as well as rooms at the Trump International … both hotels are gorgeous and right on the beach. I highly recommend the Marenas suites if you are bringing a family or the Trump International if you are looking for more night activity.
We have added two new sessions for OTDay on Tuesday:
Chris Hughes of Freeport-McMoRan will tell you how they have virtualized a number of the DCS at their mine sites. They have done this with and without DCS vendor support. Here the pro’s and con’s, lessons learned and more.
Tiptoe Through The Network: Practical Vulnerability Assessments in Control System Environments with Paul Asadoorian of Tenable Network Security and pauldotcom fame. Paul has his finger on the pulse of assessment tools and methodologies both from personal experience and over 350 weekly podcasts that interviewed top talent from around the world.
New Sponsors
Sponsors help us keep the price of S4x14 down, allow us to hold more social events, and help a great deal with the ICS village. In addition to our previously announced sponsors Siemens and Check Point, we are pleased to announce:
We have a number of additional sponsors in process of becoming official.
Mobile App
The S4x14 Mobile App will be officially released on December 9th, but it is now available on Apple App Store and Google Play if you can’t wait. Only about 50% of the content is in the app, and we will be uploading the balance this week. After that, the app will be kept current and is a great way to track developments on the event.
Plus, if you can keep a secret, I’ll be hosting a tasting of Binary Brew Works craft beer in the roof top suite in the Marenas Tuesday at sunset. Good beer with ocean and intracoastal views.
President Obama tasked NIST to develop a Cybersecurity Framework ”to reduce cyber risks to critical infrastructure (the “Cybersecurity Framework”). The Cybersecurity Framework shall include a set of standards, methodologies, procedures, and processes that align policy, business, and technological approaches to address cyber risks. The Cybersecurity Framework shall incorporate voluntary consensus standards and industry best practices to the fullest extent possible.”
In this edition of the Unsolicited Response Podcast I talk with Jack Whitsitt of EnergySec. Jack attended all Cybersecurity Framework workshops and was actively involved in providing input and comments in a variety of formats. He also is a very unique and creative thinker.
I talk with Jack about the good and bad about the process of creating the framework and the end result, as well as what to expect in the use of the framework and further development of the framework.
ISA announced that Codenomicon’s fuzzing tools are approved for use in the Communications Robustness Testing (CRT) portion of the ISASecure certification. This is a positive step forward for ISASecure. Wurldtech’s Achilles was used as the de facto tool at the start due to the difficulty of writing a detailed tool criteria, but the vision was to allow multiple fuzzing tools to be used as long as they meet a certain level of rigor. This is now achieved with certification organizations having the choice between the Wurldtech and Codenomicon tools.
FERC approved NERC CIP V5 as expected, but there are some changes they will require in V6. As always Tom Aldrich has the best coverage.
Nice job by ICS-CERT to clarify the DNP3 vulnerability impact to the master station in their updates on Nov 14th and 21st. Still it has the incredibly weak statement “Impact to individual organizations depends on many factors that are unique to each organization.” Yes. Of course. Is there another place in DHS where they say this is a serious vuln, particularly for DNP3 in SCADA?
Critical Intelligence provides reports and other information products on Cyber Situational Awareness and Threat Intelligence services for Industrial Control System Owner/Operators, Vendors and Government stakeholders.
We have covered Insecure By Design issues in ICS repeatedly on this site and at S4, resulting in some challenges to define what would make a PLC Secure By Design. This is a much harder task, but I will present some thoughts in a series of articles beginning here.
The first thought is not all that insightful – a vendor needs to integrate security into the product development lifecycle. Simple, right? Yet we often talk with vendors, as part of the asset owner team, who cannot produce even the security requirements from the definition and design phases of a product development project.
You do not need to be a security or development expert to make the first pass decision on whether the vendor has implemented a SDL.
Ask the vendor what their SDL is. If they can’t easily provide the documentation of the SDL it doesn’t exist in a practical and effective manner.
After you see the SDL, find a few key activities and ask for evidence these activities took place for the product or system you are looking to buy. If they can’t easily provide this evidence of these activities, then it took place informally at best.
This does not guarantee that the resulting product is Secure By Design, but the lack of a documented and followed SDL is evidence that it is not Secure By Design.
My personal view of the three most important items in an SDL are:
Threat Model – Every time we have done a threat model there have been a number of unaddressed and obvious threats that have fallen out, and often the fix would have been simple if it had been addressed prior to product release to production. Often these are related to reducing the attack surface, but they can also require additional security functionality to address the threat. Also, how do you know what security controls are required if you have not determined and documented the threats? (Warning – watch out for threat models that say everything is in a secure zone and eliminates most if not all of the threats. More on threat models in future parts of this series.)
Secure Coding Standards – A large percentage of the ICS vulnerabilities that have been identified in the post-Stuxnet era would not have occurred if reasonable secure coding standards had been followed by the vendor. Do the standards exist? How are engineers and developers trained on these standards? How does the vendor verify the standards have been adhered to in QA or elsewhere?
Fuzzing – This is included on the shortlist because Steve Lipner at Microsoft, and co-author of the book on SDL, said it was fuzzing and threat modeling identified the most bugs that could lead to vulnerabilities. The evidence should be more than a checkbox for fuzzing. (Check out a S4x13 session on OSIsoft’s fuzzing program)
You should inspect the SDL prior to awarding a contract, as part of your decision in selecting the winner. We saw this first hand when a RFP was awarded and the SDL audit was part of Factory Acceptance Testing (FAT). Many of the claimed SDL elements had no evidence and were clearly not performed at all, let alone as specified in the RFP response. At FAT the options are limited, particularly related to product or solution development.
There are multiple vendors in the ICS space that have been integrating security into their development process, many following the Microsoft model, for about five years now. It is a significant investment, with many failures in the early years. The training requirements alone so that everyone understands the tasks they need to perform and have a common language to discuss this are substantial. The results are showing in many of the ICS workstation and server applications from these companies.
Quick post on some big names making moves to new companies:
Ralph Langner announced today that he is forming the Langner Group in the US, and the first hire is Perry Pederson. Perry led the DHS Control System Security Program a few years back and most recently was with Nuclear Regulatory Commission (NRC).
Brad Hegrat left Rockwell recently and joined IOActive. This is an interesting, and I believe good, hire for IOActive. Most of the IOActive talent in the ICS space are top notch researchers. Guys who do a great job black box testing. Brad complements that talent with decades of working with owner/operators to secure their systems. It’s not enough to find 0days. To instigate change the owner/operator needs to understand why it matters.
Billy Rios and Terry McCorkle have left Cylance and joined Qualys. They only joined Cylance back in January, announced at S4x13, and were Cylance’s ICS security efforts. Billy is now the Director of Vulnerability Research at Qualys and will be working on more than ICS. Hopefully both Billy and Terry will continue to spend some time on ICS.
I’m Mike Toecker, Computer Engineer. I’ve been working in the Electric Power industry for about 8 years now, doing cyber security and compliance work associated with the NERC CIP regulations. I’ve worked for a major electric power consulting engineering firm for 6 years, a utility for a year, and now I work for an ICS security company.
I’ve had the opportunity during my career to assess control systems in control centers, transmission substations, and generation plants all across North America (now internationally). As part of NERC CIP compliance, I’ve had to assess impact from loss of a computer system, and how that could affect bulk system reliability. I’ve had to dig deep into how these systems function and into network and system architecture of ~38 different installations.
So, when I say that Master Fuzzing and Bare Serial vulnerabilities, like the 25 that Crain and Sistrunk responsibly disclosed via ICS-CERT, concern me, you need to pay attention. The Crain/Sistrunk vulnerabilities are the first ones we know of to demonstrate attacks on two unknown/underassessed threat vectors:
Compromising a Master from a Slave
Attacks without use of more modern (read: IP based) communications
I’m not going to talk about these new threats in detail today, they require a lot of time to cover. In this post, I’m talking history of development regarding the NERC regulations.
In the original development of both the NERC CIP and the previous NERC 1200 regulations, serial channels were considered less risky than their IP-based counterparts. And rightly so, at the time. There was almost no awareness of computer security issues in the planning and deployment of these systems, and insecure practices were rampant. Most of these practices revolved around truly horrible design and implementation of IP based networks, which were becoming the norm in electric power. Additionally, development took place after and during some of the roughest Microsoft Windows vulnerabilities, those like Blaster, SQL-Slammer, Sasser, and others. So, the developers of the original NERC regulations focused on these high profile vulnerabilities and the threat vectors they would come in over.
This led to a regulation where network security was the main technical component, where emphasis was on identifying critical systems, putting those systems in a perimeter, and locking down that perimeter. This has been the direction for the past ~10 years.
But, as we track the development of exploits, vulnerabilities, and viruses after this, NERC has not made significant changes to this approach. As everyone on the ‘Corporate’ side tightened their perimeters, restricted services, and locked the internet out from their networks, attackers moved to new threat vectors. These new threat vectors were not based on network services and direct communication via the internet, they were based in email, browsers, and browser based components such as Flash, Java, and others.
We’re once again in a new world of threat vectors, but this time they are ICS based. Compromise of the industrial controllers for Stuxnet should have rung a bell in NERC regulation development, it didn’t. Industry still has no strong standards for ensuring that logic designed is the logic running. The Crain/Sistrunk vulnerabilities are similar, we have research showing that Master stations can be compromised from the Slaves as early as June. And a few months later, we get another NERC CIP regulation submitted to FERC that explicitly exempts serial based connections from any regulatory security measures.
I find myself saying this often: The Crain/Sistrunk vulns are not the big deal. The big deal is that they demonstrate a general case that bare serial based communications are vulnerable to the same types of vulnerabilities that exist in IP communications. It shouldn’t be a surprise, but apparently it is. The NERC regulations have measures to protect, and detect, on threats against the IP communication vulnerabilities, but we do nothing against bare serial.
Crain/Sistrunk also demonstrate a general case where communications originating from a high security zone (a control center) to a low security zone (a non-critical substation) can be modified to compromise the high security zone. This vector was considered low-risk (I even heard the word ‘impossible’ uttered at least once) at the time. There were various reasons for this, most centered around the simplicity of many SCADA protocols, the lack of bandwidth on these connections, etc. This low risk was translated into a blanket exemption, which has been adopted by the industry as law.
Well, things are different now. While we’ve been lucky over the past ~10 years, and no threat vectors have required updates to the core of the NERC regulations, the general cases presented by the Crain/Sistrunk vulnerabilities require industry to take a second look at all our assumptions, exemptions, and protections from the past. How would we react if this were a vulnerability in lean burning gas turbines that caused them to burn out under certain grid conditions? What about if verbal communications between operators were often garbled or confused, and this led to problems coordinating activities between different entities?
Normally, NERC would start a workgroup to study the problem and make appropriate recommendations for addition to the regulations. Why this isn’t done, I don’t know. Most of the development activities for NERC CIP come straight out of FERC, trying to meet requirements in NOPR documents. Maybe I’m talking to the wrong people, and should start talking to FERC and the PUCs.
Press – We do allow a limited number of press to attend the event free of charge with priority given to the press that understands and covers ICS. If that describes you, and you would like to cover S4x14, send us an email.
Speakers – All Speakers with a 30-minute or longer session at S4x14 or ICSage are automatically registered for S4x14 and ICSage. We will be doing that this week and sending out the first speaker update.
Introduction to Hardware Hacking for ICS Security Professionals with Kevin Finisterre and Josh Thomas
Our Stephen Hilt took this class at DerbyCon and raved about it. We asked Kevin and Josh to give it an ICS spin and teach it at S4x14. There are not a lot of opportunities to learn hardware hacking, let alone hardware hacking focused on ICS.
Response and Serial Fuzzing ICS Protocol Stacks with Adam Crain
Adam will be presenting the technical details on his DNP3 response fuzzing at S4x14. Most of the protocol stacks fell over quite easily with special crafted response packets. The availability and integrity of the master / polling engine in the control center is crucial. Learn how to do this type of fuzz testing in the class. We are particularly interested in factors involved in creating the fuzzed response packets and serial fuzzing.
Space is limited to these courses, and all the S4x13 courses sold out. Register now to guarantee your spot.
IMPORTANT NOTE — If you want to register for an Advanced ICS Security Training course select the S4x14 Plus OTDay option in registration. The next screen will give you an opportunity to register for the training courses.
I’ll tackle that good question on when is a PLC Secure By Design in the next article, but the tweets demonstrate that Insecure By Design in ICS is not understood. I’m not surprised because it is hard to comprehend that critical infrastructure controllers are still Insecure By Design twelve years after 9/11.
Insecure By Design is not the defined by a lack of a successful Secure By Design process. No. Insecure By Design is when the vendor intentionally adds documented “features” to a product that provide an attacker with the capability to compromise the availability or integrity of the PLC and underlying process.
Our definition of Insecure By Design would not cover a lack of security coding practices, using vulnerable libraries, poor QA, lack of fuzz testing, threat modeling, etc. Those are real deficiencies that can lead to bugs and exploitable vulns. They are not however the vendor consciously deciding to add features that allow an attacker with logical access to take complete control of the controller or application.
A PLC is Insecure By Design when an attacker can use documented features to achieve his goals. Examples:
Open administrative command shells, documented hard coded support accounts, …
We coined the term of Insecure By Design as part of Project Basecamp. Many in the community were upset that Digital Bond was disclosing so many PLC vulnerabilities, but they weren’t vulnerabilities in the sense of bugs and coding errors that could be exploited. They were documented features. We didn’t consider it vulnerability disclosure if the feature was defined in the protocol specification or user manual.
Ralph Langner gave one of my favorite ICSsec quotes related to this … “The pro’s don’t bother with vulnerabilities; they use features to compromise the ICS”. Most ICS vulnerabilities matter little because most ICS protocols and controllers are Insecure By Design. Great Mr. Vendor, you fixed a cross site scripting vuln in the PLC web interface but I can still upload firmware and logic, stop the device, change the process using published features.
Hopefully if you have made it this far you now realize there is a big chasm between Insecure By Design and Secure By Design. It should be a very low bar to clear to remove Insecure By Design features in ICS devices and applications, yet minimal progress has been made in over a decade.
So to answer Joel “SCADAhacker” Langill’s tweet — anyone who can identify a device or application feature that can be used as designed and documented to compromise the integrity or availability in an ICS can call Insecure By Design. I’m sure there are a couple of features we could argue about, but most of them are straightforward.
A PLC is not Secure By Design once the Insecure By Design features are removed. I’m not bold enough to think Digital Bond or any one organization can stamp a PLC as Secure By Design, but I have some thoughts on how to address this challenge. In the next article, I’ll tackle the question looking at the ISASecure certifications and digging out an older PLC Protection Profile we wrote for NIST/PCSRF.
DHS’s ICSJWG is next week in Rockville, MD??? I guess it is still happening, but there isn’t a published agenda for the Nov 6-7 event on the ICSJWG web site area. Click on the announcement picture and you go 404. Plus there is the added bonus of no food at the event “Because meetings are no longer co-located with a hotel, members will need provide for their local accommodations and arrange for travel and meals.” How important is that public/private partnership?
Finally the latest in the Battelle/INL v. Southfork Security case. An interim win for Southfork as the temporary restraining order is lifted and a Battelle requested preliminary injunction is denied. The rest of the news is ominous for Southfork Security as the court wrote “Battelle is likely to succeed on its contract claims, including its third claim for breach of contract and its seventh claim for breach of the implied covenant of good faith and fair dealing.” Also, the court seems to be taking the national security claim seriously, which is embarrassing.
… and I still owe the promised Insecure By Design / Secure By Design explanation. Next week.
Tweet(s) of the Week
Devote 20% of your cycles to pay down technical debt. Via @joshcorman#rsac
Critical Intelligence provides reports and other information products on Cyber Situational Awareness and Threat Intelligence services for Industrial Control System Owner/Operators, Vendors and Government stakeholders.
Many asset owners would like a check box approach to security, where some independent, reputable organization certifies the system or component is secure by design. There are a growing number of security certifications that are trying to meet this need. Even if every ICS is a semi-custom system, a security certification could eliminate the need for each asset owner to assess the same security basics such as technical security controls, security development lifecycle and fuzz testing.
