- Bryan Singer has a magnum opus post and a few predictions about upcoming cyber attacks and events on control systems. I know of a few non-governmental people working on some pretty compelling scenarios as part of quantifying risk projects.
- Tomorrow, Sept 1st, is the deadline to join the ISA Security Compliance Institute (SCI) as a founding member. On Wednesday, Andre Ristaino posted that interest in SCI has been “overwhelming” and they have “commitments for Founding Strategic Membership from a balance of leading asset owners and suppliers in the control system community”. Their goal is to begin testing compliance to some yet to be defined criteria in June, 2009.
- PennWell is now producing a quarterly e-newsletter titled Security The Power Grid.
- Triangle MicroWorks has released the Secure DNP3 protocol enhancements into their source code library. This is big because many DNP3 implementations use the Triangle MicroWorks stack. (hat tip: Grant Gilchrist)
- Beyond Security found and disclosed a vulnerability in the Wireshark DNP3 dissector that could prevent Wireshark from capturing data. This should not affect any control systems since Wireshark is a security and communication analysis tool rather than part of a control system. It is yet another example of control system security by obscurity slipping away.
Archives for August 2007
After a fair amount of soul searching and delay, Digital Bond is finally releasing our iccpsic tool set to subscribers who are vetted asset owners.
This was a difficult decision because this tool set will crash vulnerable ICCP servers. It was what we developed and used to find a number of ICCP protocol implementation vulnerabilities, including some of those responsibly disclosed by US-CERT.
Here is why we are releasing it to subscribers who are vetted asset owners:
- The vulnerabilities disclosed by US-CERT have been properly addressed by the vendors who make the ICCP stacks, primarily SISCO and LiveData, but many vendors who integrate and resell SISCO and LiveData stacks under private label have not issued security bulletins notifying their customers of the need to patch or upgrade the ICCP server to address the vulnerability. This tool will allow the asset owner to identify if they have a vulnerable ICCP server irrespective of the ICCP vendor’s disclosure decision.
- Digital Bond has limited access to ICCP servers. We have tested versions of the more popular stacks, but new vulnerabilities could be introduced in future versions. It is not rare to see vulnerabilities reappear. There are many ICCP stacks with smaller distributions we have not tested and probably don’t know about. Asset owners can test the server they use and report any crashes/vulnerabilities to the vendor. We encourage and will support disclosing newly identified vulnerabilities to US-CERT.
- Not all vulnerabilities get disclosed by US-CERT. If the vendor does not patch the vulnerability, US-CERT typically will not issue a Vulnerability Note. In other cases, vulnerabilities are not disclosed to US-CERT based on NDA’s or other reasons. Asset owners can take matters into their own hands and test their ICCP server.
This was not an easy decision, but we feel selectively releasing these types of tools is critical to achieving our primary goal of assisting asset owners in securing control systems through information, tools and services.
In our web site redesign we envisioned releasing tools to vetted subscribers and built in the mechanism. We went through our list of subscribers and marked an initial set of asset owners as having vetted accounts. You will be able to download the iccpsic tool set and documentation from our Resources section after accepting the license terms.
For those subscribers who believe they warrant vetted status please send us an email and request vetted status.
- IEEE P1686 passed Working Group balloting, but the ballot is being recirculated after a few changes were made to the document.
- A reasonable article from the business press on SCADA security this week at Forbes.com.
- If you are in the electric sector in the Western US check out the Energy Security NW annual event on Sept 18/19 in Portland. Click on events to see the draft agenda with FERC, NERC, WECC, DHS and more. This is an event led by and for asset owners.
- InfraGard cancelled their annual conference this year, but they are participating and recommending the 2007 Critical Infrastructure Protection Conference, October 2-3 in San Diego. This event is broad and control system cyber security is only a small fraction. The dates conflict with ISA Expo in Houston so make your choice.
- And one more event, SANS will be holding another SCADA Security Summit in New Orleans on January 14-15, no online info yet. As mentioned earlier this is a great event for newcomers to the SCADA security issue and for status updates, best practice presentations, etc. DHS will likely have some of their half day and full day introductory courses at the event as well.
- The guys at Matasano have a great and easy to understand blog on how computer memory works and some security and performance ramifications. The memory hierarchy diagram using Sesame Street characters is a nice touch.
- Amusing final note – – Jake Brodsky is a category over at the ControlGlobal blog
It is so disheartening.
Secure By Default is a straightforward and critically important security concept. The default settings for a device or application should be secure settings so an administrator must turn off security to weaken rather than turn on security to strengthen.
My Secure By Default tale starts in June at the ISA SP99 Working Group 4 meeting to begin work on Part 4 of the standard. Part 4 is a normative standard meaning there are must or shall statements rather than guidance. Applications and devices can be compliant and eventually certified to a normative standard.
During the initial meetings the group was outlining the standard, determining what was in and out of scope, and dealing with other broad issues. I raised the point that since products are going to claim to be Part 4 compliant, should we require the default settings meet Part 4 to claim compliance? In other words, does ISA SP99 Part 4 require Secure By Default?
In my mind, this was simply a clarifying statement that all would agree to. Much to my surprise about 90% disagreed with the concept of Secure By Default in control systems. Obviously I did not explain it correctly. Second try. Do we as SP99 want an asset owner to purchase a product that is certified as SP99 Part 4 compliant with all of the security requirements in Part 4 disabled at delivery? Of course not. Next informal vote and Secure By Default loses big again. The group understands the issue but disagrees with the concept of Secure By Default. Depressing and surprising since there are a lot of smart people comprising this group.
So over the last two months I’ve used the opportunity of industry events and telephone conversations to ask a lot of people about Secure By Default.
Here’s the straight answer: the vast majority of asset owners do not want security and vendors are not going to spend money or infuriate asset owners by forcing security on them.
The most compelling discussion was with a product manager of a very popular product line in the control system community. The product manager said Secure By Default almost cost him his job. I chuckled figuring this was hyperbole. The manager was completely serious and repeated this multiple times. They implemented Secure By Default which just required the asset owner to configure some authentication at installation. This caused widespread customer complaints as they locked themselves out, large support cost increases and actual loss of customers.
Secure By Default was removed in the next version and support has gone back down. No one is using the security features. Imagine the response this manager gets when asking for new security features or security resources. It was a long, sad emotional story.
I heard similar, if less dramatic, versions of this story repeated by many vendors. During our recent OPC testing one of the vendors said:
In our OPC server product the user can configure whether or not to obey the DCOM security settings. Do I like having a setting like this? No, but at the end of the day a large majority of customers pushed for the ability to turn DCOM security off (by default) since their environments were not tied to the outside world, so we listened to the customers and made this an option.
Note the default setting is DCOM security off.
I can’t blame the vendors this time. They are delivering what customers are demanding. I can’t blame SP99 WG4 because they are in line with the sentiment of the community.
At a minimum it would be good if a vendor offered a Secure By Default purchase option, much like you can by a FIPS 140 certified version of some products, but today I would have a hard time honestly saying this made business sense for a vendor.
At the recent Weiss event, one luminary from a chemical manufacturing asset owner made an impassioned plea for the asset owners to demand security during procurement. Even pounding on the table in an effort to wake people up. We need a lot more of that, but right now it does not look good for broad based improvement in the control system security posture over the next five years unless there is an event that wakes the crowd up like some of the worms did for Microsoft users. Let’s hope that doesn’t happen, but it is frightening to rely on hope.
My expectation is many regular readers are going to comment on how availability is more important than anything else, and we always welcome divergent views. However there are ways to pre-stage Secure By Default devices, train installers on the appropriate techniques, and be successful. Vendors need to make installing with security as simple as possible, and since they are new to this there is probably room for improvement.
When a system or device is installed there are requirements such as connecting power and cables, loading databases and setting parameters, and placing it in an operating mode. Until we start to view initializing the authentication, authorization or crypto keys in the same vein the community is simply relying on the isolation and lack of motivated attackers for the safety and reliability of the critical infrastructure.
The 2008 Edition of the SCADA Security Scientific Symposium (S4) is January 23-24 in beautiful Miami Beach, Florida.
Remember the Call for Papers deadline is September 15th.
We are searching the world for the best research on control system security, and we want your help. Do you know a researcher doing important work? Send me his or her contact info.
We are chasing for a great paper on anomaly detection in control systems. Seems like a natural given the more predictable nature of communications in these systems and DHS HSARPA even funded a couple of efforts. We are chasing papers performing security analysis on non-windows OS used in control system devices such as VxWorks and QNX, although embedded XP would be interesting as well. What don’t we know about that we should be chasing?
Last year we found groundbreaking analysis on OPC vulnerabilities in Hamburg and Barcelona. We found a grad student at University of Milan taking an innovative SCADA cryptography approach. (FYI – that students skills with a little help from S4 publicity landed him at the University of Illinois/TCIP for PhD studies). We found approaches to calculating risk and controller test methodologies in Canada. And of course some of the traditional research institutions, Sandia, Mitre and SRI, presented great work on current research at the event.
Some of you may be aware that SANS recently announced they will be holding another SCADA Security Summit in New Orleans the week prior to S4. Although both events address control system security, they are targeted at very different audiences. Per the SANS email, they focus on things like “best practices … overviews … education and awareness aimed at the CEO and executive level … procurement guide …”. These events, whether they be Joe Weiss’s event, SANS, PCSF, or ISA Expo, are all highly useful for the broad community learning about SCADA security.
S4 is different. Executives, marketing and those that are non-technical should not attend. It is for the person who has been to those other events and said where’s the beef. It is detailed code analysis, vulnerability testing methodologies, mathematics and statistics, protocol generation and analysis, crypto, hardware and software performance, … We give our speakers an hour because the level of detail required cannot be presented in twenty minutes. Speakers provide a technical paper that is published in the proceedings book. Very importantly we insure the papers and presentations do not go over basic material the attendees have heard multiple times, such as the differences between control systems and IT systems or what a PLC is.
The headline on this blog is hardly shocking, but software quality does not get enough attention in the control system community. We now have three strong data points that show all OPC servers are not created equal.
1. The latest is Landon’s work to verify configuration recommendations in Part III of the OPC Security whitepaper series. Specifically setting the DCOM endpoint configuration and port restrictions to allow minimal ports through a firewall or ACL. Only three of the six tested OPC servers bothered to check the DCOM configuration. Note this is not a vulnerability that would lead to the loss of availability, integrity or confidentiality. Not all bugs lead to vulnerabilities.
2. Lluis Mora of Neutralbit (Barcelona, Spain) ran 24 test cases that would identify bugs likely to lead to vulnerabilities on 75 OPC servers. He found approximately 33% failed one or more test cases. He submitted 25 to US-CERT and a few have been published. The lack of the more advisories points to vendors not patching problems identified in January 2007. You would want a vendor that either survived the testing or was responsive in fixing software quality problems with security ramifications. An overview of his testing is blogged, and a great paper is available in the S4 Proceedings.
3. Ralph Langner of Langner Communications (Hamburg, Germany) presented at S4 serious differences in the handling of resource exhaustion attacks such as resources exhaustion through client connections, memory exhaustion through group names,
and CPU overload through new client threads.
It is not surprising that software quality varies since we are dealing with human beings coding under different, if any, security development lifecycle policies and procedures. That said, the OPC Foundation’s Product Certification appears to be the one and only software quality factor considered in a purchase. This certification is useful for positive testing and interoperability, but it would only be a minimum requirement for consideration in any product selection criteria Digital Bond ran in one of our architecture or RFI/RFP engagements.
The results from recent Landon’s testing were interesting. Not only did some products ignore DCOM settings, some of the better products allowed DCOM security to be configured during the installation process. We have not done a rigorous product comparison to make public recommendations, but based on what we know now there would be some vendors we would stay away from and others we would look to based on the three data points if forced to make a quick decision.
Our conclusion is it is worth taking some time selecting your OPC vendor – – admittedly limited data points to serious software quality problems in many implementations.
- The Big News this week is the rumor that Perry Pederson will be leaving DHS NCSD is in fact true. He is leaving the government for a job in private industry at the end of the month. This is a big loss for DHS, but best of luck to Perry in his new career.
- Mu Security has come out with an Achilles like protocol stack certification. Honeywell is the first company to have a MUSIC certified device.
- There is an organization I’ve been trying to chase down for a couple of weeks now, the Open SCADA Security Project. They have a wiki and say they are going to take the OWASP approach. My caution is I don’t know any of the leaders of the project and have had difficulty getting any information. What is their control system security experience? Are they with a vendor, asset-owner, university? All from one organization or an industry effort? Any of these answers are fine, but transparency is very important if you are asking people to trust and contribute to an effort like this.
- ISA is considering distributing DHS’s CS2SAT self-assessment tool in the near future.
- The DHS CSSP Catalog of Control System Requirements that was released a few weeks ago was pulled out of circulation for some editing. There is some politics here that I was unable to learn the dirty details. This is a shame because it was very useful, if not perfect, in its current form.
- The Industrial Ethernet book has a copy of Byres’ et al paper on the statistics from their Industrial Security Incident Database. This is not new information, but I’m not sure I have seen this paper available on the web before.
- On a personal note – I’m proud to say that this week the number of total number of comments on this blog (582) exceeded the total number of blog entries (578). This is big because the blog entries had about a 300 entry lead last year. A large number of the comments are well written, informative, logical and highly interesting – – even when they completely disagree with something we wrote. Thanks to all who have commented and improved the content in the blog.
I pulled out the Mobile Podcast rig, a new toy, and took advantage of the gathering of experts to do a few interviews. Listen to them all or skip to the one you are interested in by noting the start time in the stream.
- Introduction (0:00)
- Dilemma of Water Sector Security with Jake Brodsky and Cheryl Santor (0:22)
- US-CERT Control System Vulnerability Disclosure with Art Manion (17:44)
- HP Trusted Compliance Solution for Energy (think NERC CIP) with Steve Scott (25:21)
- Proposed ISA Security Division with Bryan Singer (29:25)
- IEC 60870-5-104 deep inspection firewall with Erik Hjelmvik (34:05)
- A raspy Joe Weiss post event interview (39:54)
First a little clean-up from yesterday on the demos in the afternoon.
- The demonstrations showing DNP3 has no authentication to prevent an attacker issuing commands and the fuzzing of protocols caused denial of service is in fact almost identical to what Ganesh presented at Defcon – – only in a lot more detail. Both of these demos have been done at SCADA security events for years. So if the press had been accurate the real story was a gleeful SCADA “hacking” presentation was made at Defcon, a venue that attracts hackers of all hat colors.
- There was a bit of a stir that the controller vendor and model was displayed on the screen for a period of time in the Mu demo with a stated “500 ICMP vulnerabilities and TCP vulnerabilities” and some demonstrations of serious DoS attacks. So this was vulnerability disclosure to the 70 or so people in the room prior to notifying the vendor or US-CERT. It was not intentional, but it happened. Given the grief Digital Bond gets for responsibly disclosing vulnerabilities just to US-CERT, this is a pretty big oops in the eyes of the control system community.
Today the morning is the conclusion of Joe Weiss’s event and the afternoon is a NIST workshop on control system security.
10:30AM – Perry Pederson led off the day with a presentation on the DHS NCSD programs. A few new items:
- DHS has developed a control system security training program in a box. The box contains the presentation and other materials for the class. It will be piloted in October and if successful released through PCSF after that. So if you are knowledgeable on control system security you should be able to take this box and teach a course at your company or industry group.
- DHS has developed a curriculum for a university course on control system security at the Graduate level. It is for non-technical, management types and is available from the PCSF site.
Perry introduced Simon Hennin of Raytheon who has a TSWG contract for the Cyber Attack Alert Tool (CAAT). This is another effort to collect and share threat information. Phase 1 is to “define standard data schema and protocol for communicating attack data”. This is similar to what we tried and failed to do in a PCSF working group. Only Lurhq (a MSSP) submitted data for a few months on about 5 control systems. There was a related S4 paper from Sandia/I3P on anonymous, authenticated information sharing. Simon is looking for participation if you are interested. Tough problem with the “what’s in it for me and is it worth the risk sharing my information”.
Mike Peters from FERC presented later in the morning. He opened by saying he was not going to talk about the NOPR on NERC CIP, but he did strongly encourage everyone to read and comment on the NOPR. This was echoed by Scott Mix of NERC.
I’m not going to go into details on Mike’s interesting talk to avoid writing anything that would prevent him from speaking at other events. What he did was take real world events and extrapolated on how cyber methods could have caused a similar incident.
Mike did discuss the risk equation and made the case for setting threat to 1 like they do in the RAM assessments. Essentially remove threat from the equation and focus on vulnerability x consequences. In my opinion this is a classic government viewpoint, and maybe appropriate for government. History proves this approach does not cut it with C-level executives. They require threat info to justify spending.
6:00PM – I attended the first half of the NIST Workshop on Applying NIST SP800-53. There were some strong example presentations from TVA and Entergy. Then the groups broke up to answer the questions:
- Do you think that convergence of standards is important. Why?
- Are the NIST RMF and the ICS augmentation of SP800-53, Revision 1 a good basis for convergence?
The effort to answer these questions will continue tomorrow. Our group started by trying to determine what NIST meant by convergence. My opinion is if convergence means everyone agreeing to use SP800-53, this will never happen and isn’t worth the effort to try. Many in SP99 envision their standard being used cross industry. NERC isn’t going to move to SP800-53, and I have a hard time imagining other vertical sectors not issuing their own standards. You could even argue the question if convergence is helpful.
This does not mean that other standards bodies don’t look at SP800-53. SP99 WG4 is using it as a reference and may develop a mapping. In fact, the DHS Catalog of Security Requirements is a useful reference document, and if convergence meant that each standard body issued an Appendix mapping their standard’s requirements to the SP800-53 requirements this is probably achievable.
It will be very interesting to see what if and what consensus the group at the workshop achieves.
Back again semi-liveblogging on day two at Joe Weiss’s conference. I think the day two agenda is the most interesting with sessions in the afternoon on field device vulnerabilities. Check back often for updates.
9:15AM – An overview of the Chemical Sectors efforts in control system security is presented by two individuals from Dow Chemicals. Two great things about the Chemical Sector:
1. The large companies from the Chemical Sector are some of the most active participants and leaders in the control system security standards efforts.
2. The Chemical Sector makes their work product in this area available to everyone at www.chemicalcybersecurity.com.
Chemical industry will be participating in DHS’s Cyberstorm II.
A brief and interesting discussion in the Q&A on the new DHS regulations on the chemical sector. DHS’s Chemical Security Compliance Sector is working out the details to implement the law now. The large Tier 1 companies will be audited in person by DHS or their representatives. The regulations are “primarily physical with cyber overtones”. If you are familiar with Sandia’s RAM (Risk Assessment Methodologies such as RAM-W for water and RAMCAP for chemical), the emphasis on physical is similar.
11:45AM – Marshall Abrams from MITRE, with help from Joe Weiss, did a case study on the Olympic Pipe Line incident in Bellingham, WA on June 10, 1999. While the cause of the rupture and ensuing fire and death were not due to a cyber security incident, the fact that the leak was not detected for approximately 90 minutes was a fault of the SCADA and leak detection system. See the description of the SCADA failures starting on page 61 of the NTSB report.
Interestingly some of the key personnel invoked their Fifth Amendment rights and refused to testify and some logs were missing. There are still some unknowns as to what actually caused the SCADA failure.
2:15PM – Perry Pederson of DHS NCSD is talking about one of Digital Bond’s hot topics, vulnerability disclosure. There is a large US-CERT part to this story. It is good to see DHS promoting this.
I wish DHS would use their muscle and power of the purse to force the National Labs to report identified vulnerabilities to US-CERT. There are lab-vendor agreements that prevent this now, but the labs have enough clout in this industry to include a report to US-CERT in 3 or 6 months clause in those agreements.
3:15PM – A couple of live demos this afternoon. First Mark Hadley of PNL showed that he could run a DoS attack or control a remote device via the DNP3 protocol. This is true of most control protocols because they lack user/device and data authentication. This seems old hat and has been shown for years, but it is effective for those new to control system security. Watch. See me change the lights. Watch me turn the lights out.
Of course Secure DNP3 has the promise to change this.
Following Mark is a team from Mu Security. They are demonstrating their fuzzing/security analysis platform on the ICMP and TCP protocols in a controller which I’m tempted to name, but will not. Claimed to find 500 vulnerabilities in the ICMP protocol alone. The impact of the vulnerabilities varies from a 4 second hang to requiring a reboot. Run a series of fuzzed packets and watch the controller crash.
Loyal readers of this blog, and users of controllers, know how fragile many controller protocol stacks are. This is an effective demo and highlights a large problem.
The issue that controller vendors often purchase stacks from third parties is raised. This is a problem when it comes to remediation.