On Mobile Device ICS App Security

redphone_ekilbyI was talking a while ago to Justin Engler, a friend who also happens to be a really talented web app and mobile app security researcher, about the popping-up of ICS management software for mobile devices.  He theorized that mobile apps for ICS would be an interesting place to watch for bugs nearly three years ago.  Dale’s recent ICSJWG Q&A over mobile device security gave me a little motivation to dig into some sample apps and see how the field actually looks.  The results highlight some of the issues that your organization will run into if and when you decide to adopt mobile.

The focus of this post is not just application security.  While there are a few specific vulnerable applications mentioned, I think that the big lessons should be ones of architecture and integration challenges.  The current lot of ICS management apps pay little mind to securing access or preventing bad operation.  Even an app with ‘secure’ on its product homepage may leave you wide open.

I decided to pick on Android simply because my only jailbroken iOS device at the moment is so terribly destroyed from years of abuse that installing new apps is a nonstarter. There also seems to be more interesting control systems apps for Android at the moment.

A quick survey of Google’s Play store for terms such as ‘SCADA’’, ‘PLC’, and ‘OPC’ turns up a few applications worth checking out.  Unfortunately there are no apps that I could find which do what Dale prescribes: obtain safe, accurate, remote, ‘read-only’ access to control system data.  Doing so will require a lot of backend work on your part.

Let’s take a look at two interesting vulnerable applications.

Read More

OT Is Mission Critical IT

applesorgangesYou are pounded with the message: ICS security is different than IT security. The fact is the Operations Technology (OT) in an ICS is a mission critical / high value IT system and needs to be treated like one. Don’t let the ICS is different argument allow you to accept fragility and insecurity in your OT.

The trigger for this entry is Bob Huba’s article in ISA’s Intech: Top Ten Differences Between ICS and IT Cybersecurity. The article is well written and accurate in comparing ICS and IT Desktop management, but that is the wrong comparison. ICS should be compared to mission critical / high value IT where companies can tell you how much money they lose, and it’s often a big number, for every minute of downtime.

There are plenty of mission critical IT systems that have availability requirements that rival ICS (1: Security Objectives). They have a better availability strategy than the too often ICS approach of make it redundant and don’t touch it if it is working. This is software and hardware that must be managed and supported to meet the availability requirements and not devolve into fragility.

Bob’s 8: Untested Software applies to both high value IT and OT. The change management around both includes rigorous testing, phased deployment, rollback strategy, etc. If you load up a patch or upgrade in high value IT and bring something down, you will see a similar management reaction to an ICS outage. Clearly the physical danger in a lack of integrity or availability is a factor in ICS, but do you honestly believe that a large financial impact is not treated with a similar level of concern to a business?

#9: Patching and #10: Security Inconveniences are ICS excuses and actually make the ICS more insecure and fragile than the high value IT. An ICS needs to be designed with a plan to support and maintain the mission critical IT system that it is. Would you put in physical equipment and not consider what maintenance will be required to the physical equipment to support reliable operation? I always point to Langner’s Robust Control System Networks as the book to hand an engineer who needs to understand why a fragile ICS is a problem.

The #3: Network Topology for ICS actually sounds like high value IT systems. #2 Network Segmentation is a partially valid difference. Some high value IT systems are Internet accessible, others are not. The high value IT systems will use DMZ’s and least privilege rulesets that quite frankly are much more restrictive than the average ICS firewall ruleset.

#6: User Accounts really is not a valid difference between IT and ICS, except that ICS too often accepts poor identification. Role based authorization is common in both as are other authorization methods and principles. Authorization has traditionally been an area of strength for ICS, and increasingly this is being performed in Active Directory.

There are a few areas of genuine ICS and High Value IT difference that Bob points out:

  • High value IT does not typically have Safety Instrumented Systems (#7: SIS)
  • The Purdue model does not apply to IT (#4: Functional Modeling), but you could argue that they have similar data flow and responsibility models with different names.
  • Many of the #5: Physical Components are unique to ICS … although you are seeing some of these physical components in a Data Center.

I agree with Bob that if you treat ICS HMI, EWS, Historians, Real Time Servers, PLC’s, RTU’s … like desktop user PC’s you will cause some serious problems, but this is a flawed comparison.

There is a lot the ICS community can learn and emulate from Mission Critical / High Value IT in designing, deploying and maintaining a robust OT environment. We should be leveraging their knowledge and processes rather than pushing them away. And certainly we shouldn’t continue down the path of install and don’t maintain because we didn’t plan to support the OT and don’t know how.

This is not to say that High Value IT is perfect. There are systems running with uncertainty and fragility and just hoping things won’t break … a similarity with many critical infrastructure ICS.

Image by Dan McKay

  • Critical Intelligence

Friday News & Notes

SCADA Security NewsAfter the PG&E substation shooting, FERC had ordered NERC, as the ERO, to develop and submit a Physical Security Reliability Standard within a very short time frame for this type of work. NERC complied and now FERC says they will approve the standard with two changes. FERC wants the ability to add or remove facilities from the critical facilities list. While they say this would be “exercised only rarely”, this is a crack in the door or slippery slope that is likely to give utilities heartburn. FERC also wants to replace “widespread instability” with “instability”. There needs to be an adjective in front of instability.

Critical Intelligence is holding a one day conference and two days of training called CounterIntel, Sept 16-18 in Park City, UT. The two day training is to help you be a more effective Cyber Intelligence Analyst, and the whole event is limited to owner/operators. Living in the Park City area, I can tell you it is a great time to hold a conference here.

Read the Kyle Wilhoit of FireEye article on how Havex enumerates OPC Servers. Great work.

The automobile sector has started the Auto Information Sharing and Analysis Center (Auto-ISAC). ISAC’s have a very mixed record based, but it seems every sector will have one.

Image by ChrisinPlymouth (the F king)

S4x15 Week: Call for Papers/Presentations

ICS Security

The S4x15 Week Call for Papers/Presentations is now out.

Send us your session ideas asap to have the best chance of getting on the agenda. All we need is a short description and time requirement mailed to s4@digitalbond.com.

We are calling it S4x15 Week now because it goes Tuesday – Friday (Jan 13-16 in Miami Beach):

  • Tuesday – OTDay and ICS Village Opens
  • Wednesday – Day 1 of S4x15
  • Thursday – Day 2 of S4x15
  • Friday – ICSage:ICS Cyberweapons and Advanced Topics ICS Security Training

The CFP gives more detail on each day and the type of sessions we are looking for.

I wish that we could sit back and wait for all the great sessions to come in, but history has shown that we need to hunt for this great work and unknown talent. If you see or hear about anything that we should chase for S4x15 week, please let us know.

Last year was a big step forward for the ICS security community. We moved past low hanging fruit; we brought in some top security researchers from outside the ICS space; and there was a new focus on what an attacker would do after successful exploit.

We are looking forward to seeing some new and amazing work for S4x15.

Digital Bond Labs Open For Business

open_chipgriffinWay back at the Spring 2014 ICSJWG meeting, Dale announced that Digital Bond is opening a new division — Digital Bond Labs.  This week, we are officially opened for business…and we are hiring.

Digital Bond has a long reputation for building the tools that other ICS consultants use ten years down the road.  It seems that every other talk given in the ICS security community lately make reference to Digital Bond’s intrusion detection signatures, Nessus audit files, or Project Basecamp exploit demos.

At Digital Bond Labs, we aim to provide the best in the business at breaking control systems software, security add-ons, and access control systems for both end-users and vendors.  Our goal is simple: break all the things that make all the things so that we can rebuild them to be more robust and more secure.  In the Digital Bond tradition, we will also continue to focus on valuable research to share with the ICS community.

Read More

Even Little Bobby Knows

Remote AccessWe are working with Robert M. Lee and his publisher to get SCADA and Me in Japanese for a giveaway on OTDay of S4xJapan (agenda and registration open on Aug 4th). I wish I had the page above as a hidden slide to pull out at ICSJWG last month.

While most of my presentation involved the secure and insecure way to use the cloud in the future for analysis of process data, the most contentious point was on remote access. The easiest way to get into an ICS with a good security perimeter is to compromise an administrator, engineer or technician that has remote access to the ICS. The ICS Spear Phishing session at S4x13 showed how something as simple as a fake maintenance bulletin would have led to compromise of over 25% of the targeted users with remote access to the ICS.

Here’s the basic solution. Push the data out so the right people can view it without jeopardising the integrity and availability of the ICS. Have a physical disconnect for the remote access, and close the connection only in emergency situations following a defined process. Use your automation skills to put this capability on a display in the control room with the appropriate alarms and logging, and auto open after a time limit. If you are having multiple emergencies a week that require remote access your system is not under control or you are understaffed.

Someone in the audience, who actually is in the business of advising industry, pushed back hard at these limitations on remote access. Paraphrasing he said “c’mon we all know that this generation is going to demand and have remote access with a control and admin capability from their smart phone in their basement. What do you recommend to secure this?” This is when I needed the SCADA and Me page. “If you can control it from a phone — so can Bad People.”

Lior Frenkel of Waterfall said something after my session that I told him I’ll be stealing from now on. “You’re part of the critical infrastructure. Act like it!”

Friday News & Notes

SCADA SecurityGive eWON some credit. They released information that their website was compromised for a short time in January, and issued an updated notice late last week on their home page. Still nothing on the MB Connect or Swiss vendor site to tell customers they may be compromised by Havex if they downloaded and ran their software. Companies are going to have security incidents; customers should be looking and considering how they respond.

Alstom Grid has a new product coming out in response to the PG&E substation shootings called e-terrasheriff. It will detect and report gunshots at unmanned substations, and presumedly integrate this into the e-terra SCADA displays.

DHS has opened the CFP for the ICSJWG Fall Meeting. After attending and speaking at the Spring Meeting I was going to pass on this one, but holding it in Idaho Falls will dampen attendance.

The first release of Automotive Grade Linux is out. “Each component includes a detailed Design Requirements Document (DRD) with descriptions, use cases, HMI flows, graphical assets, architecture diagrams and more.”

We have always appreciated the Swedish contingent that has supported S4 since the start. Now they have created their own event, 4SICS, Oct 22-23 in Stockholm. They are working on the agenda, but they already have some great technical ICS talent in Europe lined up. Lueders, Santamarta, Hjelmvik, …

DHS is looking for a lead and partners for their Critical Infrastructure Resilience Center of Excellence (CIRC). “Each COE is led by a U.S. college or university and has multiple partners. COE partners include other academic institutions, private industry, DHS components, DOE National Laboratories and other Federally-Funded Research and Development Centers (FFRDCs), other federal agencies that have homeland security-relevant missions, state/local/tribal governments, and first responder organizations.”

A Honeywell help wanted ad is illustrative of how ICS vendors are trying to generate revenue from cyber security. “This position will be responsible for leading, managing and growing the Honeywell Process Solutions (HPS) industrial cyber security global remote managed services business.”

IETF has an initial draft standard out for “Two Way Authentication for IoT“.

USG Aurora Data Dump

Thanks Dan for the tip.

First a reading tip to save you time. Most of the 840 pages are weekly reports from the DHS Control System Security Program (CSSP). There is a ton of repetition as each week’s report carries forward all of the previous week’s items. So go straight to page 750 and you will see the reports going backwards from 19-23 Nov 2007 to 22-26 Jan 2007.

The most interesting excerpt is from the 12-16 March 2007 report:

The CSSP large scale validation test of a significant control systems vulnerability (Pandora) was successfully completed at the Idaho National Laboratory on March 4, 2007. Results and findings from the test are being documented and significant follow-on activities are anticipated. The Tiger Team formed to coordinate activities for this vulnerability will meet on March 13. to U/S Foresman is scheduled to be briefed the afternoon of March 13. Briefings to Secretary Chertoff, House Homeland Security Committee and White House Homeland Security Council are anticipated.

After that Pandora entry there is no other mention of Pandora in the weekly reports. It evidently was classified and changed its name to Aurora. A meeting to discuss the technical details of the Aurora vulnerability appears next in the 19-23 Nov 2007 weekly report on page 751.

There were mentions of this “large scale validation test of a significant control system vulnerability (Pandora)” in the weekly reports prior to the test. A few other tidbits:

  • It was a large scale test with an estimated cost of $2.8M (page 57).
  • There are some good pictures of the physical site beginning on page 100.
  • Pages 70 and 71 have some good examples of specific systems that could be affected by Aurora.
  • There is mention of a Control System Malware Identification Team being formed by the CSSP back in Jan 2007 (page 233). Let’s put this team on Havex.
  • A Firmware Upgrade Vulnerability report is discussed on page 165. I don’t remember this being issued, but it was seven years ago and DHS was calling these insecure by design features as vulnerabilities back then.
  • The mention of CSSP working with JASON – an independent group of scientists that advises the USG and particularly intelligence is interesting, especially back in 2007. Stu….
  • The mitigation strategy memos start on page 36. The early briefing milestones were met, but little else after that seems to have been accomplished and much of the detail on what was to be done is redacted. They do show a plan for software and hardware fixes being developed and deployed within two to three years.
  • The technical team memo on page 821 is worth a read.


It’s been seven years since that turbine shook and the smoke came out, yet I always thought Aurora was a lost opportunity.

The real beauty of the Aurora demonstration was it clearly showed that a cyber attack could affect a physical process. The specific vulnerability they chose to achieve this, while not unimportant, was not the main point to take from Aurora. It was an effective and dramatic demonstration.

Aurora should have led a massive DHS and US Government push to address the insecure by design ICS that run the critical infrastructure. Instead of taking this and leading a massive PR and bully pulpit campaign building off of this expensive but effective demonstration, people lost their jobs because the video and secret got out.

Perhaps the idea of physical damage through a cyber attack struck too close to Stuxnet, or maybe it didn’t have the internal support and program to leverage the successful demo. Whatever the reason it was a lost opportunity.

I knew it was lost during the Congressional Hearings. Senators and Representatives asked the august panel from DHS, NERC, utilities, etc. if the Aurora problem had been fixed. Rather than use the question to pivot and highlight Aurora is a small symptom of the larger problem, the experts would go into the plan in place to address Aurora.

I can’t end this long post without a nod to my friend Joe Weiss. He has beating the Aurora drum harder and longer than anyone else. Perhaps this will give him more ammunition for his cause. It is difficult to reconcile Pandora being called “a significant control system vulnerability”, being classified, resulting in all those briefings, tiger teams, remediation plans, … and the relatively small expenditure and effort to address the “Aurora vulnerability”.

and just in case you want to see the video again:


Havex Hype & Unhelpful Mystery

ICS MalwareUnhelpful Mystery

Why hasn’t ICS-CERT or some other CERT or the security vendors issuing bulletins announced publicly the three ICS vendors that were distributing malware with their ICS software and the energy sector websites redirecting to a malware delivering site?

It’s baffling. Perhaps the security vendors have a valid profit motive for keeping it secret, but the CERT’s are largely in place to aggregate and spread this information. I’m told the information is in the US-CERT Secure Portal, but this is a terrible way of alerting the affected asset owners.

If the names of the vendors that unwittingly spread Havex were made public, the wide coverage would likely reach most of the affected asset owners.

It is also regrettable that most of the ICS vendors involved in the Havex distribution have not come clean on their web site to warn their customers, more on this below.

Next: The Hype

For these attacks to have a significant impact on the US or other countries’ energy sector the vendors distributing the software with malware would have to a good size client list in the sector. (And we would have to make the leap that asset owners actually update software)

A profile of the compromised vendors’ customers would help understand how widespread the impact is and perhaps what specific asset owner, sector or country is being targeted. So who are the compromised vendors?

MB Connect Line

Michael Toecker quickly identified MB Connect Line as one of the vendors by looking at some public malware samples.

This is likely company #3 in the Symantec post. The MB Connect Line site states wind turbines and biogas plants, along with other energy infrastructure systems are the applications for their products. Ironically they also highlight their mbEagle product, secure detection of Stuxnet and other manipulations, and mbSECBOX, security for S7 PLCs. We also have a few independent sources confirming MB Connect Line is the German company.

This is a very small company outside of Stuttgart trying to gain a foothold providing remote access solutions in tdistributed energy resources. The impact to the critical infrastructure of this company distributing malware along with their software would be minimal in Europe, and minuscule in the US.

I could not find any mention on the MB Connect Line site that they had unknowingly distributed Havex and what action customers should take.


Search for “VPN access to PLCs” and the first response was eWON in Belgium. Multiple other trusted sources independently told us or confirmed that #1 in the Symantec post was eWON.

We have never seen this company’s products in the US. Their impact to the US energy sector would be minimal. Perhaps they could have an impact in Europe. We will ask around with our European friends and find out more. It is clearly not one of the major vendors that would have a widespread impact.

eWON disclosed the website breach back on January 30th (note the 250 download number matches Symantec’s description), but they did not appear to know the OPC aspect of the Trojan and have not issued an update now that it is high profile.

Swiss Company

The F-Secure article stated that the three vendors were in Germany, Belgium and Switzerland, so the last affected vendor is a Swiss “manufacturer of specialist PLC type devices”. We eventually found the name of the vendor, but not in a way that we can disclose at this time.

If our sources are correct, this company would have a smaller impact on energy sector than eWON or MB Connect Line. There is also no notice of the Havex distribution on their site.

Energy Related Site Redirects

Symantec describes the other avenue of infection as:

comprising a number of energy-related websites and injecting an iframe into each which redirected visitors to another compromised legitimate website hosting the Lightsout exploit kit.

Symantec provides a redacted list, on page 15 of their report, of five “energy control system” companies and six “energy” companies that were redirecting visitors to a compromised site. These companies were in France, India, Italy and Norway.

Again it would be helpful if these energy control system and energy sites were made public so asset owners could be alerted that they may have been compromised. We do not know these sites, but we have been told they are not big or even medium players in the energy sector. They are closer to a MB Connect Line or eWON rather than an ABB or Siemens.

Hype Summary

A few sentences out of longer articles from Symantec and F-Secure, mixed with some selected quotes from ICSsec pundits, and combined with an absence of information on what software and sites were compromised has led to the hype in the press.

The Target

The three ICS vendors distributing software with Havex are terrible watering holes if you want to attack the US energy sector and not great watering holes even for European energy sector. A couple of possibilities:

  • The attacker was going after a specific target that the attacker knew was going to use the compromised ICS software. Note I said going to use, not is using. The attacker needed the target to download the compromised software, and it is still rare for asset owners to update software.
  • The attacker was trying a proof of concept attack. How effective could this software be at finding and enumerating OPC servers? An attacker might want to know this before they compromised more popular energy sector software being deployed in their actual target organizations.
  • ??? who knows ??? The key is the customer base of these three companies. While small, perhaps they had significant penetration in a sector in a country. Take a look at the intersection of the MB Connect Line, eWON and Swiss company’s customer lists.

The ICS Portion of the Attack

The Havex code itself is highly interesting for the ICS community because it is only the second publicly acknowledged occurrence of an attack using the insecure by design ICS protocols as part of the attack. I’m wary of the early returns fully understanding the impact of the ICS code, but it is safe to say now that it is at least doing some identification and enumeration of OPC servers.

While OPC can be used for monitoring and control, it rarely is in critical infrastructure or any SCADA or DCS of any size for a variety of performance and historical reasons. Perhaps that will change with OPC UA in the future, but today you see it used primarily for passing data to and from systems from different manufacturers. For example, the OPC interface is used over 50% of the time to get data in and out of the very popular OSISoft PI Server even though OSIsoft has 100′s of interfaces.

Attacking OPC servers can be a good way to get through the corporate/ICS security perimeter and also to jump from ICS to ICS. It is a good target.

One last note … the ICS-CERT advisory states:

ICS-CERT testing has determined that the Havex payload has caused multiple common OPC platforms to intermittently crash. This could cause a denial of service effect on applications reliant on OPC communications.

This may be nothing more than poor code quality of the OPC servers they are testing. We have personal experience and seen multiple S4 talks that show how easy it is to crash an OPC server.

Image by James Marvin Philips

Michael Toecker Starts Context Industrial Security

Michael ToeckerMichael Toecker recently has joined the ranks of Digital Bond alumni and is starting his own firm. Here is his farewall blog entry. Best of luck Mike and welcome to the world of being a small business owner.

A few others have known this for a while, but I’ve left Digital Bond to form a new engineering firm to focus on cyber security for electrical power systems.

The past two years at Digital Bond has been a fantastic experience. I’ve had the opportunity to conduct assessments of Critical Infrastructure that I hadn’t had a chance to see before, and work with operators on improving the security of their control systems.

While working at Digital Bond, everywhere we went, every critical infrastructure had one requirement in common: a reliable source of electric power. Without this basic resource, most critical infrastructure would not function, and would not provide benefit to society. So, I’m refocusing efforts on the security of electric power systems. The new company is called Context Industrial Security, and is focused on providing cyber security consulting and design within the context of the process being controlled, and is focused on the security of electric power systems and the unique characteristics that affect their vulnerability.

I’m grateful to Digital Bond and to Dale for giving this engineer a chance to work on the bleeding edge of industrial control system security, and to interact with security, process, and operations persons who are dedicated to the security of control systems. I will still tweet and blog, but it will be from the (in development) website of Context Industrial Security (www.context-is.com), and my personal twitter account (@mtoecker) for now.