- The “news” that an attacker with network access could upload firmware to many controllers came out this week. This FOUO report has been floating around, and it seemed hard to believe it was FOUO. It is common knowledge in the control system space, not to diminish the fact it is another serious widespread control system security flaw. In fact, firmware uploads have been on the Quickdraw event list almost from the start because it sure would be nice to know when this has happened. If you want to read some of the leaked document and additional info check out the liquidmatrix blog entry.
- DHS’s Control System Security Program issued a Recommended Practice for Creating Cyber Forensics Plans for Control Systems.
- Joe Weiss wrote a white paper including recommendations for the Blue Ribbon Commission on Cyber Security. This Commission will be providing a set of recommendations for the next US President. The paper is available after registering on the Control site.
Archives for August 2008
UPDATE: 6:30PM, Dale
PCSF is not perfect, but it is my favorite event in the control system security space by far. One main reason is the number, variety and quality of attendees. The lunch, evening, break discussions were highly interesting and even three days had me scrambling to talk with all the people I’d like to. The venue and schedule helped maximize opportunities for these discussions.
The program was mixed. I was not a big fan of the all day plenary session on Tuesday. Some of the panels had format challenges. The quality of the sessions may have been down slightly, but that is subjective. There were some very strong sessions, and I even missed some of the more highly reviewed sessions, and the days when there were 3 or 4 tracks usually meant something interesting was going on. There may be a need to spice up the next events, more shorter presentations, perhaps PCSF classic presentations for newcomers, more livelier debate and discussion sessions, etc.
I believe it is essential that PCSF continue and grow mainly because there isn’t a good alternative and starting over would be difficult. The information exchange and education at PCSF is needed. 200 people from 17 countries with little notice the week before Labor Day is impressive. Four tracks on Wednesday; three tracks on Thursday that were easily filled as submissions exceeded time. Hopefully whatever issue prevented DHS from attending will be resolved, and whatever format PCSF ends up in the future can focus on how to make this annual event and other events even stronger.
UPDATE: 6PM, Dale
The Vulnerability Disclosure Workshop followed up the panel. There is never a shortage of opinions on this subject. Not sure we made any progress. It was interesting that Daniel and I from Digital Bond were the only ones in the room that would disclose a vuln to anyone besides the vendor [we disclose to US-CERT and Core had left].
Back to the Plenary to wrap up. A report by PCSF Brazil – – not directly affiliated with PCSF, but there have been interesting discussions about PCSF Europe and other international locations.
Jason Holcomb, Bandolier
I started the morning going to Jason’s Bandolier presentation at 8AM for support. Nice job and the presentation will be posted on our site shortly.
Included in the presentation is the updated list of planned Bandolier security audit files. It is great that we were able to add Areva, Emerson Ovation and others to the list. We will update the SCADApedia page shortly.
I moved over to the vendor panel in progress, interesting group with smart guys and gals from ABB, Emerson, Honeywell, Invensys, Siemens, Telvent, and Yokogawa. Doing a little liveblogging during the Q&A
– Love the point of needing to move by Secure by Default from the ABB rep
– The Honeywell rep indicated the lifecycle may need to be reduced from 15 years to 10 years.
– Maybe it is no longer realistic to expect to have a control system with equipment and applications from 20 different vendors, Invensys rep.
– Don’t touch your switches, update IOS after installed and working??? Hello, McFly, won’t attribute that comment.
– Interesting comment from Telvent that some of the customers have them physically disable, burn out, the USB ports and other unused ports so they can never be used even if enabled in software.
– Discussion on encryption, not sure why because as one of the panelists noted integrity is much
– They asked my question “Do vendors have any obligation to provide security vulnerability mitigation for customers who do not have a current support contract?” Invensys says definitely. Siemens frames it well, out of warranty, no support contract, not current contact info . . . talks about User Group, we will always help them in an emergency – – vague but it sounds like they will help on a time and materials basis or some other cost basis. They move on to the next question.
– Do you have 3, 5, 10 year plan? Telvent focused on 5 year plans and defined a bit. Interesting they have a plan on how to bolster legacy systems until they are replaced.
FOUO filetype:pdf shows the number of FOUO documents (pdf only try it on doc and see what you find) available via google.
scada filetype:doc shows just how easy it is to find critical control system information. Such a document can be seen at:
- Sedapal – Water Treatment Plant – Jaime G Uchuya with diagrams and screen shots.
UPDATE: Next day, Dale Peterson
I missed the Waterfall Solutions Unidirectional Connectivity presentation but caught up with them at the evening exhibit. They have a product that through hardware, I heard the term diode and optical communications, only allows one way communication. Hence they use the term unidirectional. It is an interesting concept that could be useful if you are pushing data from a more secure zone to a less secure zone, such as control center to DMZ. It is purely one way, so there are no acks, resend, recovery, etc. Where is this a good option?
UPDATE: 4PM, Dale Peterson
I also attended the RISI / incident database talk. I’m convinced it can work, because it has worked. The question is whether there is enough interest to do this pro bono or receive funding. Interestingly, I was thinking why would a business want to go through the effort to collect and maintain this database. Maybe one with a portal strategy??? Maybe we should talk to Mark Fabro and Eric Byres.
Bryan Singer of Wurldtech had the long slot after lunch to talk about Achilles inside. [Full disclosure: Wurldtech is a past client and current advertiser]. Actually have a few comments about this. After the 1:30 presentation I still can’t tell you what Achilles Inside is. I asked a few others, and they couldn’t either. Perhaps it was to avoid commercialism, and it could be the greatest thing ever, but the message needs some work.
There were some interesting parts of the presentation such as “Safety does not deal with intentional actions” and the impact of bridging the traffic for monitoring. Wurldtech had to be specify their own hardware to minimize the impact of monitoring during testing.
A bit of discussion on vulnerability disclosure as well. Wurldtech will not release vulnerability information and is very sympathetic to the problems of patching.
UPDATE : Morning Recap, Jason Holcomb
Several good presentations and side conversations so far today.
I attended the first one “Are You Compliant or Liable? Industrial Security and Compliance Using the Holistic Lifecycle Model” with a bit of a personal agenda. I assumed those attending might also be interested in our Bandolier project so I wanted to listen any issues that may be relevant.
(Side Note: This was presented by Clint Bodungen of CIDG, Chris Paul of Joyce and Paul, and Jeff Whitney of Berkana Resources Corporation). I do appreciate the holistic approach to compliance (CIDG’s model). In fact, I have worked on something very similar for another organization only we called it the “security framework”.
Not sure if I’m convinced on all the legal arguments made by attorney Chris Paul but IANAL, as they say. He talked a lot about potential criminal or civil liabilities based on security negligence. I’m just not sure if avoiding a lawsuit is the right motivation for control system security but I suppose it can help get the attention of some.
Next up for me was Eric Byres’ and Mark Fabro’s presentation about the Repository for Industrial Security Incidents (RISI). This is a spinoff of the work Eric did at BCIT with ISID (Industrial Security Incident Database). Here’s the overview:
- You will need to submit an incident to the database in order to have full access (this is the same policy used with the ISID)
- The difference with this system is there will be online access
- There will be a paid quarterly newsletter that will provide summary information from the database — statistics, sector-specific data, etc…
- There will be somewhere between 75 and 150 incidents in the database from the beginning
They are actively gathering input on if and how to carry out this project so I’m sure they would love to hear from you if you have an opinion. There will be some challenges for them but I am definitely curious to see what this looks like in final form.
I rounded out the morning with “Control Systems Threat Awareness” by Robert Huber and Sean McBride of INL. These guys have used various data collection points to help understand the current threat and trends over time. It was a good follow-up to yesterday’s presentation by Stephen Gill of Team Cymru. It was a well-organized compilation of threat data. They’ve taken many of the things you’ve heard, such as control system presentations at hacker conferences, and plotted them in a measurable way that illustrates an increasing “adversary interest”.
One of the really interesting slides did a comparison of how control system application vendors make their security contact information available versus that of the big traditional IT software companies. It measured the percentage of the two groups that had a /security web page and a dedicated e-mail address for security issues, a standard of sorts for interfacing with the security research community. As you might imagine, the results showed tat only a very small minority of the control system application vendors followed the practice.
Thinking back on day one, the highlights for me were Phyliss Schneck’s keynote and Mark Fabro’s closed to press presentation. Plenary sessions are tough because it is hard to calibrate the presentation to a large audience with very different experience and interest levels.
Day two is called solution day. There are four tracks going on and then an exhibit tonight. I find these sessions more interesting than the plenary event. There are more details and more focused.
When Good Traffic Goes Bad: When is Application Traffic Too Much?
Daniel Peck from Digital Bond joined Tom Maufer of Mu Dynamics and Kevin McGrath of ABB in this presentation. Interesting denial of service examples from Brown’s Ferry Unit 3 Scram [too much traffic to a PLC], Amazon S3 [too many logins], and Ralph Langner’s OPC DoS paper from S4. Ralph showed how very long group names and too many client connections could exhaust all resources and cause a DoS. The OPC applications did not have any limits.
Vendors can improve the situation through rate limiting, syn cookies and source filtering, as well as beefing up their logging. Asset owners should consider quality of service measures, and maybe there is a case for looking at load balancing rather than purely redundancy?
Lots of good talk on the importance and methods for vendor testing, followed now by Mu doing a demo of some testing options with their product.
Guess what – – the demo didn’t work – – may have been for the best as the Q&A was more interesting.
Vulnerability Disclosure Panel
Ted Angevaare of Shell is only interested in sharing any vulnerabilities with the vendor. Not a coordination center, not any public disclosure.
Nate Kube of Wurldtech does not believe in widespread dissemination of control system vulnerability information.
Art Manion of CERT/CC believes there needs to be a public record of vulnerabilities, including control system vulnerabilities. There may be considerations about the amount of information. Interesting IT Vulnerability Response Evolution slide, Denial followed by Anger followed by Acceptance.
Yurie Ito from Japanese CERT, JPCERT, is providing overview of an international cert. They use the same disclosure model for IT and control system vulns. “Make sure the system users are aware of the risk and can make a decision how to respond.” They have a policy that if a vendor knows and can contact all users there may not be a public disclosure. [I don’t like that]. They also have a vendor point of contact list so they know who do contact [that would be a big help in the control system space where it can take 6 months plus to find the right person.]
Kevin Sullivan of Microsoft – – “we need the help of security researchers; no vendor can imagine or identify all vulnerabilities in their code; our updates must run correctly on every single machine”. ICASI is an effort among global IT vendors to coordinate response to a widespread problem with a multi-vendor response. Open lines of communication – – where can researchers report vulns.
Aaaahh . . . finally a control system vendor on the big panel, Al Rivero from Telvent. Their customers do not want the vulnerability publicly disclosed, but they want to know about it from Telvent. Their patch mgmt program has a 5-day business commitment for patch verification, service packs take longer. Information available to customers through a secure Extranet. They have an RSS feed for this info, nice. Not really talking at all about what happens when a vuln is found in their software.
Ivan Arce of Core Security Technologies is the last on the big panel. They were the ones who recently found and disclosed the Wonderware and Citect vulns. Core disclosure is not a revenue generation activity. They do it to gain knowledge, promote brand and name, and “help vulnerable organizations understand and mitigate risk”. [Well maybe not direct revenue, but it is part of their strategy. Which is fine; no one is pure] “Extend the vendor the courtesy of notifying them first, but do not rely on the vendor to solve the problem”. Believe if they found the vuln, others have found it. They do coordinate with other organizations when necessary, but not most of the time only when they hit roadblocks. They push for transparency all communications are documented [this makes for interesting reading]. They assume the vulnerability is exploitable unless there is some strong evidence it cannot be.
Ted of Shell is making a big point that systems cannot be shut down to be patched without having a business impact. [this should not be a problem if adequate redundancy is in place, but of course it takes time and $]. Ted’s second comment, what about vulns in the vendor’s freezer – – vulns the vendor knows about but hasn’t fixed.
Question from the audience – – companies that say we know and don’t fix. How long does CERT wait? Art says that threat to disclosure is the only stick. Ivan says they don’t see disclosure as a threat. If vendor is not addressing it they disclose so end users can address the problem.
Question about IPv6.
Summary: Lots of interesting comments from the panel, but it was way too big. Should have been cut in half so the intro comments did not take up 75% of the session time.
UPDATE: 4 PM PDT, Dale Peterson
The law enforcement panel included RCMP and FBI. FBI has trained cyber squads in all 56 field offices, and they even have dedicated analysts focused on control systems.
Most interesting factoid is the first SCADA-related prosecution is under way for a crime in the water sector.
Safeguarding asset owner information is now possible given the Protected Critical Infrastructure Information Act which eliminates FOIA access. Confiscating equipment was an issue they were ‘sensitive’ to for critical infrastructure equipment. Not sure ‘sensitive’ gives me comfort.
I’ll be live blogging the vulnerability disclosure panel.
UPDATE: 3:00 PM PDT, Jason Holcomb
Feedback from the those around my table is that the plenary sessions have been interesting but lacking some detail. The first day tackles some high level issues. The workshops and demonstrations over the next two days should be a little more detail-oriented.
One of our DOE-funded research projects, Bandolier, got a brief plug earlier today in the Energy Sector Roadmap Update. I think the immediate, usable results make it an attractive project. I’m looking forward to talking about it more in our Thursday session.
Some observations in the current session “Control System Cyber Incident Handling: A Law Enforcement Perspective Panel”… This may be old news to some, but just in case you didn’t know, the FBI has its own Process Control Systems Analyst. And it’s good to see participation from the RCMP, also represented on the panel.
UPDATE: 2PM PDT, Dale Peterson
Tim Roxey of Constellation Energy and the Nuclear Sector Coordinating Council provided some detail on the AURORA vulnerability including the equipment necessary for an attack, the access and knowledge required to launch the attack, and the time [less than one minute] to execute the attack. A four question check list was created, simplified version in the presentation such as “Does the facility contain rotating AC electrical equipment identified as a critical asset? and Is this rotating equipment connected to commercial power at any time?, was provided to asset owners to determine if their was vulnerable. There is an Official Use Only version of this briefing with more info.
Next up is the Law Enforcement and Vulnerability Disclosure panels.
– – – – –
The 2008 PCSF Annual Meeting kicked off this morning in La Jolla just outside of San Diego. Well … actually it is now the Process Control System Industry Conference and DHS is not here! Officially they were “not able to attend” and were “frustrated and disappointed” by this. The unofficial buzz was there was a last minute issue about whether it was legal/allowed for DHS to participate or spend money on this type of event. Can’t say I know the exact story, and it really doesn’t matter. It is just highly regrettable that the team from DHS couldn’t participate in what is largely a DHS sponsored event.
On a happier note there are about 200 people here from 17 different countries, and the weather is beautiful when the marine fog burns off around noon.
Keynote – Phyillis Schneck, Founding Chairman and Chairman Emeritus of InfraGard, currently with Secure Computing
This was an interesting choice for a keynote. In some ways InfraGard, especially with SCADAgard, overlaps what PCSF does, but when asked about that Phyllis sidestepped the question. However, InfraGard went through a number of changes in approach and structure over 8 years to become what it is today – – a successful organization with 86 chapters and more than 26K members. It shows those trying to make PCSF a success that it takes persistence and an ability to adjust. You don’t usually get it right the first time.
One of the main benefits of InfraGard, according to Phyllis, is the ability to “know who you’re going to call before you need to”. InfraGard members have a relationship with their local FBI agents, and these relationships can dramatically reduce response times. One other interesting point is each of the 26K members of InfraGard have undergone a records check.
Two Closed Presentations
The next two presentations are closed to the press, which seems unnecessary, but I’ll respect that. I attended the first closed presentation, “Security Challenges Facing the Control Systems Environment”. No comment allowed.
I skipped the second session “Should We Be Scared-a SCADA?” by Team Cymru. If you want to see what I imagine will be the Cymru presentation see http://www.cylab.cmu.edu/seminars/default.asp and click on Page 2. It requires a Windows machine to view the presentation. Some interesting info on Internet traffic on control system ports, but my expectation is PCSF attendees would not be using the Internet as a plaintext control system WAN.
Check back throughout the day for updates.
Next week should be a lot of info with the PCSF annual meeting and three from our team in San Diego. Only a couple of items this week.
- Telvent issued a press release discussing their participation in Bandolier. The team there has been a great help in improving the OASyS DNA Security audit files.
- Tripwire joins the NERC CIP party and discusses how their product will help meet some requirements.
- UPDATE: A few weeks back NERC announced they would hire a Chief Security Officer. This week they announced that Michael Assante will fill that role. Nice hire as he has the knowledge, people skills and experience from INL and AEP. [hat tip: Ron Southworth]
Believe it or not research teams are not always marketing wizards, and even the best results can have little impact if the potential users don’t understand the value of the solution. So the DHS Science and Technology Directorate is putting a representative from all the research teams in the recently awarded contracts through SRI’s Value Creation workshop. I attended the two-day workshop this week.
The group arrived from a variety of different organizations – academia, small companies, massive companies, very different research projects, and different levels of experience doing this sort of thing. And I think it was fair to say many arrived as skeptics. SRI put us through a set of instructions and exercises in a structured process to describe the need, approach, benefits and competition of our projects. First in a 1-minute elevator pitch, then adding more elements in a 3-minute presentation and finally a 5-minute presentation. As you can imagine the brevity forced focus.
A key part of the process was the way the attendee and teacher feedback was presented and acted on in the class.
It is hard to do the workshop justice. It is a combination of the environment, energy, process, materials, teachers, and fellow students. The best way to convey it is every attendee had dramatic improvement in their ability to present the value of their project. Dramatic is not hyperbole here. You had to see it to believe it.
Hats off to DHS S&T for realizing the importance of this effort to maximizing the impact of their research dollars and SRI for a strong program. [FD: Obviously DHS S&T has funded Digital Bond research projects]
There has been a lot of talk about disclosure of control system vulnerabilities. We have been laying low on this issue and letting it percolate after disclosing to US-CERT the initial control system vulnerabilities and kicking the issue off at PCSF two years ago.
With another PCSF annual meeting and disclosure panel coming up next week in San Diego, it is time to reengage. So take a look at our narrated 20 slides, 20 seconds each Pecha Kucha presentation on the topic.
If you can’t spend 6’40”, I’ll sum it up in 4 sentences. Fighting over the ‘proper’ control system vulnerability disclosure procedure and putting up new organizations is a waste of time because the decisions of vendors, asset owners, academia, government, and coordination center do not matter. The only policy that matters is the policy of the person or organization that finds the vulnerability, and many will not play ball with all these new proposed methods and organizations. I know Digital Bond wouldn’t, because we are happy with the results of our policy to disclose to US-CERT. [and we are quite conservative compared to most vuln discoverers] Instead the community, especially vendors but also asset owners, should be focused on how they will process the inevitable vulnerabilities as they arise in increasing numbers.
Some companies, both vendors and asset owners, continue to give away the proverbial “baby with the bath water.” Case in point (from an article at automation.com but which was a general press release):
August 7, 2008 – Reykjavik Energy selected ABB to upgrade and integrate five utility automation systems – geothermal power plants, district heating, water and wastewater – into a single state-of-the-art 800xA extended automation system that will be operated from a central control room.
Reykjavik Energy is Iceland’s largest utility, providing almost 70 percent of the country’s population – including the capital, Reykjavik – with electric power, district heating, hot and cold water, and wastewater treatment.
The company operates two geothermal power plants which provide 240 megawatts (MW) of electricity and 700 MW of heat to some 26,000 homes and businesses in 20 communities.
According to Reykjavik Energy, the district heating system is the “largest and most sophisticated in the world.” Hot water is pumped via 2,500 kilometers of pipes to heat buildings and keep pavement and outdoor parking lots free of ice in winter, as well as supply the many outdoor swimming pools and spas with a constant stream of hot geothermal water.
Cold groundwater is distributed to consumers throughout the Greater Reykjavik area, and the region’s sewage is piped to several wastewater treatment plants throughout the area.
System 800xA as the integration platform for current and future systems
ABB supplied the original distributed control systems for the utilities in the late 1990s and has evolved all five systems to its state-of-the-art Extended Automation System 800xA in accordance with the advanced and growing requirements of Reykjavik Energy.
ABB has sold more than 4,000 of these systems since its introduction in 2004, improving industrial productivity, safety, and operational profitability for customers in virtually every industry.
Reykjavik Energy is in an expansive phase of development, acquiring and integrating other utilities and extending its geographical reach and customer base. System 800xA enables the customer to easily integrate the control systems, databases and automation hardware of future acquisitions.
The customer’s previous investments in PLCs (programmable logic controllers) supplied by a diverse array of vendors have all been integrated into the System 800xA solution. Some 50,000 I/Os (input/output) data signals are processed by the solution.
ABB is a leader in power and automation technologies that enable utility and industry customers to improve performance while lowering environmental impact. The ABB Group of companies operates in around 100 countries and employs more than 115,000 people.
Why is such a press release a bad idea? Well if I were a bad guy with malicious intentions I now know that the whole of Iceland’s power generation and heating (hot water distribution) is now controlled by a single, centralized control system. But, not only that I know that it is an ABB system. Specifically an: Extended Automation System 800xA.
So now if I was a bad guy, I could research or perhaps even purchase an Extended Automation System 800xA an probe it for vulnerabilities. Through this press release my reconnaissance was greatly simplified. I now know that there is a single point of failure that I need to pw0n in order to control, or DOS the network.
Instead of diversifying their network, effectively making it more complicated to attack. Reykjavik Energy has conveniently collapsed the entire nation’s energy control grid into one nicely packaged application and broadcasted it to the world.
So what do I mean by “just not getting it”? Well perhaps this type of public announcement, though attractive to the vendor “touting their PR horn” and to the asset owner, really doesn’t serve ethers’ best interests in terms of security.
More exciting news from the Bandolier project… we are wrapping up some extensive collaborative testing with one of our vendor partners. It is the most thorough outside review of the Bandolier audit files to date and we are very pleased with the results. With each development and testing cycle, we are able to apply what we have learned to audit files from other vendors as well as improve the assessment methodology. The review process has solidified our opinion on application vendor involvement, re-iterated the importance of the OS-level checks, and pushed us to develop a better set of checks.
We blogged recently on the benefit of vendor involvement and this most recent testing process has certainly corroborated our feelings on the subject. The ability to tap into the application’s developers and top security talent definitely helps us create the best set of checks possible. The vendors are also intimately familiar with development and testing processes and can provide QA assistance by testing the audit files in their labs.
We’ve observed that application vendors that are serious about security are taking steps to deliver the underlying operating systems in a hardened state. This sometimes includes additional security hardening documentation and, for Windows systems, group policies that dictate a number of important security settings. We discussed before how our expectations about the value of the app checks vs. the OS checks shifted a little — the OS checks, tailored to the application, have proven to be a very important part of Bandolier.
To capture Windows security settings, we use a variety of collection techniques. Tenable provides some tools that have been helpful. The first one is a simple executable that is launched on a Windows machine known as the Windows Nessus Policy Creator (WNPC). It gathers many of the typical Windows security settings and translates them into audit checks. The second is called i2a (inf to audit) and generates an audit file based on a Microsoft .inf policy file. (We’ll discuss the configuration to audit tool (c2a) for Linux in a later post.)
An important lesson learned during the recent testing process concerned review of the checks developed from the automated tools. The vendor pushed us to improve the baseline OS files with better organization, descriptions, and additional checks. These improvements will lead to better Windows server and workstation OS files for all the Bandolier applications.
Having this thorough, outside review was definitely beneficial. In the meantime, we are seeing support and interest in this project continue to ramp up. Other big news for Bandolier… next week at PCSF we will announce the final list of audit files.