S4x15 Week: Call for Papers/Presentations

ICS Security

The S4x15 Week Call for Papers/Presentations is now out.

Send us your session ideas asap to have the best chance of getting on the agenda. All we need is a short description and time requirement mailed to s4@digitalbond.com.

We are calling it S4x15 Week now because it goes Tuesday – Friday (Jan 13-16 in Miami Beach):

  • Tuesday – OTDay and ICS Village Opens
  • Wednesday – Day 1 of S4x15
  • Thursday – Day 2 of S4x15
  • Friday – ICSage:ICS Cyberweapons and Advanced Topics ICS Security Training

The CFP gives more detail on each day and the type of sessions we are looking for.

I wish that we could sit back and wait for all the great sessions to come in, but history has shown that we need to hunt for this great work and unknown talent. If you see or hear about anything that we should chase for S4x15 week, please let us know.

Last year was a big step forward for the ICS security community. We moved past low hanging fruit; we brought in some top security researchers from outside the ICS space; and there was a new focus on what an attacker would do after successful exploit.

We are looking forward to seeing some new and amazing work for S4x15.

Digital Bond Labs Open For Business

open_chipgriffinWay back at the Spring 2014 ICSJWG meeting, Dale announced that Digital Bond is opening a new division — Digital Bond Labs.  This week, we are officially opened for business…and we are hiring.

Digital Bond has a long reputation for building the tools that other ICS consultants use ten years down the road.  It seems that every other talk given in the ICS security community lately make reference to Digital Bond’s intrusion detection signatures, Nessus audit files, or Project Basecamp exploit demos.

At Digital Bond Labs, we aim to provide the best in the business at breaking control systems software, security add-ons, and access control systems for both end-users and vendors.  Our goal is simple: break all the things that make all the things so that we can rebuild them to be more robust and more secure.  In the Digital Bond tradition, we will also continue to focus on valuable research to share with the ICS community.

Read More

  • Siemens Industrial Security
  • Critical Intelligence

Even Little Bobby Knows

Remote AccessWe are working with Robert M. Lee and his publisher to get SCADA and Me in Japanese for a giveaway on OTDay of S4xJapan (agenda and registration open on Aug 4th). I wish I had the page above as a hidden slide to pull out at ICSJWG last month.

While most of my presentation involved the secure and insecure way to use the cloud in the future for analysis of process data, the most contentious point was on remote access. The easiest way to get into an ICS with a good security perimeter is to compromise an administrator, engineer or technician that has remote access to the ICS. The ICS Spear Phishing session at S4x13 showed how something as simple as a fake maintenance bulletin would have led to compromise of over 25% of the targeted users with remote access to the ICS.

Here’s the basic solution. Push the data out so the right people can view it without jeopardising the integrity and availability of the ICS. Have a physical disconnect for the remote access, and close the connection only in emergency situations following a defined process. Use your automation skills to put this capability on a display in the control room with the appropriate alarms and logging, and auto open after a time limit. If you are having multiple emergencies a week that require remote access your system is not under control or you are understaffed.

Someone in the audience, who actually is in the business of advising industry, pushed back hard at these limitations on remote access. Paraphrasing he said “c’mon we all know that this generation is going to demand and have remote access with a control and admin capability from their smart phone in their basement. What do you recommend to secure this?” This is when I needed the SCADA and Me page. “If you can control it from a phone — so can Bad People.”

Lior Frenkel of Waterfall said something after my session that I told him I’ll be stealing from now on. “You’re part of the critical infrastructure. Act like it!”

Friday News & Notes

SCADA SecurityGive eWON some credit. They released information that their website was compromised for a short time in January, and issued an updated notice late last week on their home page. Still nothing on the MB Connect or Swiss vendor site to tell customers they may be compromised by Havex if they downloaded and ran their software. Companies are going to have security incidents; customers should be looking and considering how they respond.

Alstom Grid has a new product coming out in response to the PG&E substation shootings called e-terrasheriff. It will detect and report gunshots at unmanned substations, and presumedly integrate this into the e-terra SCADA displays.

DHS has opened the CFP for the ICSJWG Fall Meeting. After attending and speaking at the Spring Meeting I was going to pass on this one, but holding it in Idaho Falls will dampen attendance.

The first release of Automotive Grade Linux is out. “Each component includes a detailed Design Requirements Document (DRD) with descriptions, use cases, HMI flows, graphical assets, architecture diagrams and more.”

We have always appreciated the Swedish contingent that has supported S4 since the start. Now they have created their own event, 4SICS, Oct 22-23 in Stockholm. They are working on the agenda, but they already have some great technical ICS talent in Europe lined up. Lueders, Santamarta, Hjelmvik, …

DHS is looking for a lead and partners for their Critical Infrastructure Resilience Center of Excellence (CIRC). “Each COE is led by a U.S. college or university and has multiple partners. COE partners include other academic institutions, private industry, DHS components, DOE National Laboratories and other Federally-Funded Research and Development Centers (FFRDCs), other federal agencies that have homeland security-relevant missions, state/local/tribal governments, and first responder organizations.”

A Honeywell help wanted ad is illustrative of how ICS vendors are trying to generate revenue from cyber security. “This position will be responsible for leading, managing and growing the Honeywell Process Solutions (HPS) industrial cyber security global remote managed services business.”

IETF has an initial draft standard out for “Two Way Authentication for IoT“.

USG Aurora Data Dump

Thanks Dan for the tip.

First a reading tip to save you time. Most of the 840 pages are weekly reports from the DHS Control System Security Program (CSSP). There is a ton of repetition as each week’s report carries forward all of the previous week’s items. So go straight to page 750 and you will see the reports going backwards from 19-23 Nov 2007 to 22-26 Jan 2007.

The most interesting excerpt is from the 12-16 March 2007 report:

The CSSP large scale validation test of a significant control systems vulnerability (Pandora) was successfully completed at the Idaho National Laboratory on March 4, 2007. Results and findings from the test are being documented and significant follow-on activities are anticipated. The Tiger Team formed to coordinate activities for this vulnerability will meet on March 13. to U/S Foresman is scheduled to be briefed the afternoon of March 13. Briefings to Secretary Chertoff, House Homeland Security Committee and White House Homeland Security Council are anticipated.

After that Pandora entry there is no other mention of Pandora in the weekly reports. It evidently was classified and changed its name to Aurora. A meeting to discuss the technical details of the Aurora vulnerability appears next in the 19-23 Nov 2007 weekly report on page 751.

There were mentions of this “large scale validation test of a significant control system vulnerability (Pandora)” in the weekly reports prior to the test. A few other tidbits:

  • It was a large scale test with an estimated cost of $2.8M (page 57).
  • There are some good pictures of the physical site beginning on page 100.
  • Pages 70 and 71 have some good examples of specific systems that could be affected by Aurora.
  • There is mention of a Control System Malware Identification Team being formed by the CSSP back in Jan 2007 (page 233). Let’s put this team on Havex.
  • A Firmware Upgrade Vulnerability report is discussed on page 165. I don’t remember this being issued, but it was seven years ago and DHS was calling these insecure by design features as vulnerabilities back then.
  • The mention of CSSP working with JASON – an independent group of scientists that advises the USG and particularly intelligence is interesting, especially back in 2007. Stu….
  • The mitigation strategy memos start on page 36. The early briefing milestones were met, but little else after that seems to have been accomplished and much of the detail on what was to be done is redacted. They do show a plan for software and hardware fixes being developed and deployed within two to three years.
  • The technical team memo on page 821 is worth a read.


It’s been seven years since that turbine shook and the smoke came out, yet I always thought Aurora was a lost opportunity.

The real beauty of the Aurora demonstration was it clearly showed that a cyber attack could affect a physical process. The specific vulnerability they chose to achieve this, while not unimportant, was not the main point to take from Aurora. It was an effective and dramatic demonstration.

Aurora should have led a massive DHS and US Government push to address the insecure by design ICS that run the critical infrastructure. Instead of taking this and leading a massive PR and bully pulpit campaign building off of this expensive but effective demonstration, people lost their jobs because the video and secret got out.

Perhaps the idea of physical damage through a cyber attack struck too close to Stuxnet, or maybe it didn’t have the internal support and program to leverage the successful demo. Whatever the reason it was a lost opportunity.

I knew it was lost during the Congressional Hearings. Senators and Representatives asked the august panel from DHS, NERC, utilities, etc. if the Aurora problem had been fixed. Rather than use the question to pivot and highlight Aurora is a small symptom of the larger problem, the experts would go into the plan in place to address Aurora.

I can’t end this long post without a nod to my friend Joe Weiss. He has beating the Aurora drum harder and longer than anyone else. Perhaps this will give him more ammunition for his cause. It is difficult to reconcile Pandora being called “a significant control system vulnerability”, being classified, resulting in all those briefings, tiger teams, remediation plans, … and the relatively small expenditure and effort to address the “Aurora vulnerability”.

and just in case you want to see the video again:


Havex Hype & Unhelpful Mystery

ICS MalwareUnhelpful Mystery

Why hasn’t ICS-CERT or some other CERT or the security vendors issuing bulletins announced publicly the three ICS vendors that were distributing malware with their ICS software and the energy sector websites redirecting to a malware delivering site?

It’s baffling. Perhaps the security vendors have a valid profit motive for keeping it secret, but the CERT’s are largely in place to aggregate and spread this information. I’m told the information is in the US-CERT Secure Portal, but this is a terrible way of alerting the affected asset owners.

If the names of the vendors that unwittingly spread Havex were made public, the wide coverage would likely reach most of the affected asset owners.

It is also regrettable that most of the ICS vendors involved in the Havex distribution have not come clean on their web site to warn their customers, more on this below.

Next: The Hype

For these attacks to have a significant impact on the US or other countries’ energy sector the vendors distributing the software with malware would have to a good size client list in the sector. (And we would have to make the leap that asset owners actually update software)

A profile of the compromised vendors’ customers would help understand how widespread the impact is and perhaps what specific asset owner, sector or country is being targeted. So who are the compromised vendors?

MB Connect Line

Michael Toecker quickly identified MB Connect Line as one of the vendors by looking at some public malware samples.

This is likely company #3 in the Symantec post. The MB Connect Line site states wind turbines and biogas plants, along with other energy infrastructure systems are the applications for their products. Ironically they also highlight their mbEagle product, secure detection of Stuxnet and other manipulations, and mbSECBOX, security for S7 PLCs. We also have a few independent sources confirming MB Connect Line is the German company.

This is a very small company outside of Stuttgart trying to gain a foothold providing remote access solutions in tdistributed energy resources. The impact to the critical infrastructure of this company distributing malware along with their software would be minimal in Europe, and minuscule in the US.

I could not find any mention on the MB Connect Line site that they had unknowingly distributed Havex and what action customers should take.


Search for “VPN access to PLCs” and the first response was eWON in Belgium. Multiple other trusted sources independently told us or confirmed that #1 in the Symantec post was eWON.

We have never seen this company’s products in the US. Their impact to the US energy sector would be minimal. Perhaps they could have an impact in Europe. We will ask around with our European friends and find out more. It is clearly not one of the major vendors that would have a widespread impact.

eWON disclosed the website breach back on January 30th (note the 250 download number matches Symantec’s description), but they did not appear to know the OPC aspect of the Trojan and have not issued an update now that it is high profile.

Swiss Company

The F-Secure article stated that the three vendors were in Germany, Belgium and Switzerland, so the last affected vendor is a Swiss “manufacturer of specialist PLC type devices”. We eventually found the name of the vendor, but not in a way that we can disclose at this time.

If our sources are correct, this company would have a smaller impact on energy sector than eWON or MB Connect Line. There is also no notice of the Havex distribution on their site.

Energy Related Site Redirects

Symantec describes the other avenue of infection as:

comprising a number of energy-related websites and injecting an iframe into each which redirected visitors to another compromised legitimate website hosting the Lightsout exploit kit.

Symantec provides a redacted list, on page 15 of their report, of five “energy control system” companies and six “energy” companies that were redirecting visitors to a compromised site. These companies were in France, India, Italy and Norway.

Again it would be helpful if these energy control system and energy sites were made public so asset owners could be alerted that they may have been compromised. We do not know these sites, but we have been told they are not big or even medium players in the energy sector. They are closer to a MB Connect Line or eWON rather than an ABB or Siemens.

Hype Summary

A few sentences out of longer articles from Symantec and F-Secure, mixed with some selected quotes from ICSsec pundits, and combined with an absence of information on what software and sites were compromised has led to the hype in the press.

The Target

The three ICS vendors distributing software with Havex are terrible watering holes if you want to attack the US energy sector and not great watering holes even for European energy sector. A couple of possibilities:

  • The attacker was going after a specific target that the attacker knew was going to use the compromised ICS software. Note I said going to use, not is using. The attacker needed the target to download the compromised software, and it is still rare for asset owners to update software.
  • The attacker was trying a proof of concept attack. How effective could this software be at finding and enumerating OPC servers? An attacker might want to know this before they compromised more popular energy sector software being deployed in their actual target organizations.
  • ??? who knows ??? The key is the customer base of these three companies. While small, perhaps they had significant penetration in a sector in a country. Take a look at the intersection of the MB Connect Line, eWON and Swiss company’s customer lists.

The ICS Portion of the Attack

The Havex code itself is highly interesting for the ICS community because it is only the second publicly acknowledged occurrence of an attack using the insecure by design ICS protocols as part of the attack. I’m wary of the early returns fully understanding the impact of the ICS code, but it is safe to say now that it is at least doing some identification and enumeration of OPC servers.

While OPC can be used for monitoring and control, it rarely is in critical infrastructure or any SCADA or DCS of any size for a variety of performance and historical reasons. Perhaps that will change with OPC UA in the future, but today you see it used primarily for passing data to and from systems from different manufacturers. For example, the OPC interface is used over 50% of the time to get data in and out of the very popular OSISoft PI Server even though OSIsoft has 100′s of interfaces.

Attacking OPC servers can be a good way to get through the corporate/ICS security perimeter and also to jump from ICS to ICS. It is a good target.

One last note … the ICS-CERT advisory states:

ICS-CERT testing has determined that the Havex payload has caused multiple common OPC platforms to intermittently crash. This could cause a denial of service effect on applications reliant on OPC communications.

This may be nothing more than poor code quality of the OPC servers they are testing. We have personal experience and seen multiple S4 talks that show how easy it is to crash an OPC server.

Image by James Marvin Philips

Michael Toecker Starts Context Industrial Security

Michael ToeckerMichael Toecker recently has joined the ranks of Digital Bond alumni and is starting his own firm. Here is his farewall blog entry. Best of luck Mike and welcome to the world of being a small business owner.

A few others have known this for a while, but I’ve left Digital Bond to form a new engineering firm to focus on cyber security for electrical power systems.

The past two years at Digital Bond has been a fantastic experience. I’ve had the opportunity to conduct assessments of Critical Infrastructure that I hadn’t had a chance to see before, and work with operators on improving the security of their control systems.

While working at Digital Bond, everywhere we went, every critical infrastructure had one requirement in common: a reliable source of electric power. Without this basic resource, most critical infrastructure would not function, and would not provide benefit to society. So, I’m refocusing efforts on the security of electric power systems. The new company is called Context Industrial Security, and is focused on providing cyber security consulting and design within the context of the process being controlled, and is focused on the security of electric power systems and the unique characteristics that affect their vulnerability.

I’m grateful to Digital Bond and to Dale for giving this engineer a chance to work on the bleeding edge of industrial control system security, and to interact with security, process, and operations persons who are dedicated to the security of control systems. I will still tweet and blog, but it will be from the (in development) website of Context Industrial Security (www.context-is.com), and my personal twitter account (@mtoecker) for now.

Havex / Stuxnet / ICS-CERT / DHS

Havex RatI believe the last time ICS-CERT announced malware that specifically attacked a control system product or protocol was back on July 20, 2010. At that time I naively railed that DHS / INL / ICS-CERT should be thoroughly investigating this and determining the impact to control systems. After they essentially and intentionally dropped the ball, I was encouraging the guy in the ICSsec community that knew the most about Siemens’ products and protocols to dig into it. Ralph Langner and his team, along with some great work from Symantec, did the analysis that led to the world learning about Stuxnet.

Now we have the announcement from F-Secure on the Havex RAT. There are two reasons to believe this is attack is targeted at ICS. First, it is doing something with the OPC protocol. OPC is often used to transfer process data between systems from different vendors. Almost every ICS made in the last decade has an OPC interface; it is the ICS universal translator.

What it is doing to OPC servers is still unclear. If this is an early phase of an attack it could simply be running OpcEnum to gather information about what OPC servers are on the network. Most do not deploy the available security controls in OPC because it is difficult (even after reading a three part white paper from Byres / Digital Bond), and it breaks necessary comms if not done right. Also, the lack of good coding practices leaves many OPC servers with vulnerabilities, some disclosed and many just waiting to be found or used.

The other reason to believe it is targeting ICS comes from F-Secure:

Of more interest is the third channel, which could be considered a form of “watering-hole attack”, as the attackers chose to compromise an intermediary target – the ICS vendor site – in order to gain access to the actual targets.

It appears the attackers abuse vulnerabilities in the software used to run the websites to break in and replace legitimate software installers available for download to customers.

Our research uncovered three software vendor sites that were compromised in this manner. The software installers available on the sites were trojanized to include the Havex RAT. We suspect more similar cases exist but have not been identified yet.

Based on the content of their websites, all three companies are involved in development of applications and appliances for use in industrial applications. These organizations are based in Germany, Switzerland and Belgium. Two of them are suppliers of remote management software for ICS systems and the third develops high-precision industrial cameras and related software.

Let’s watch the post-discovery DHS / INL / ICS-CERT analysis of the Havex RAT. I’ll keep my tinfoil hat in the closet for now.

F-Secure’s discovery of this ICS malware leads to a question … shouldn’t DHS / INL / ICS-CERT be scouring malware data and samples to identify ICS malware?

I have to give Michael Toecker credit for being prescient on this. While at Digital Bond he had a Mining Malware research project and wrote a bit about it. Time and other limitations left it to primarily conceptual conclusions, but the idea had merit and likely would have identified this as ICS malware if the F-Secure or VirusTotal sample was scanned.

Developing a process and tools to identify potential ICS malware in large samples seems like an ideal project for DHS / INL / ICS-CERT. Then give it, don’t try to sell it a la Sophia, to those with large samples with some agreement to share the results. The ICS world would get some great threat data from the often touted, but rarely of value, public/private partnership. Big win.

Image by turiskopio

South Beach Hotel for S4x15

surfcomberI came a day early to South Florida this week to check out the newest official S4x15 hotel: the Surfcomber Hotel in South Beach.

Those still wanting large rooms and suites, luxury, quieter beach and close to the best malls and the Kovens Center can stay at the Trump International. This got high marks from attendees the last two years. To meet the request for a South Beach hotel we are making the Surfcomber Hotel the second official S4x15 hotel … and we got a great rate for attendees.

The Surfcomber is right in the middle of the South Beach action. It’s close to Lincoln Road, all the great art deco hotels and neon signs, restaurants of all types and price ranges, great people watching and bars and clubs.

Of course this location and activity means smaller rooms in an older (1948) hotel and noisier atmosphere. Kimpton bought and did a major update and refurbishment a couple of years ago, but it is a smaller art deco hotel in a party area of South Beach. So attendees will have a good choice between the Trump and Surfcomber.

The main reason I wanted to see the place was to go over ideas for the social event we will hold there on Thursday night out at the pool. The pool is somewhat famous for parties, and Daley Direction has some unique and fun ideas for the S4 / ICSage convergence event.


Friday News & Notes

ICS Security NewsBloomberg published more detail on the “UglyGorilla” attack on pipeline SCADA. It’s worth reading past some of the hyperbole in the article to learn what information was taken. “Operatives vacuumed up caches of e-mails, engineering PDFs and other documents, but it was their focus on supervisory control and data acquisition, or SCADA, systems in industrial computers that most concerned U.S. officials.” We have heard more detail on what was taken, and some of it would be very helpful in crafting an attack.

EnergySec/The Anfield Group has published the Agenda for their 10th Anniversary Summit, August 18-21 in Austin, Texas. Days 1 and 2 are training and workshops. Day 3 is a day primarily devoted to sponsor presentations. This is a bold move given the general revulsion to vendor presentations that have even a whiff of commercialism, but give them credit for being clear that they are sponsor presentations. Day 4 looks like the best day with a solid agenda.

The PHDays blog has more detail about the Critical Infrastructure Attack contest at PHDays 4. “Organizers added new SCADA systems (such as Siemens TIA Portal 13 Pro and Schneider Electric ClearSCADA 2014) and various OPC servers (Kepware KepServerEX, Honeywell Matrikon OPC). New HMI devices, the operator panel Siemens KTP 600, PLC (Siemens Simatic S7-300 and S7-1500)and remote control devices (ICP DAS PET-7067) were presented as well. Schneider Electric MiCOM C264 was provided by CROC.” Impressive.

The Kuwait Industrial Automation and Industrial Control System (KIACS) Cyber Security event graciously put videos of the sessions on YouTube. The production quality is first class. After watching a few sessions it appears to be an excellent event for those new to the field of ICS security.

Joe Weiss participated in the filming of a television show on ICS security. He took the camera crew out to a couple of transmission substation sites and found they were left alone while parked in an unmarked van and filming the substation. Expect this to generate some articles when the show is on air.

Image by ChrisinPlymouth