Kaspersky issued a research report on Havex they called Energetic Bear – Crouching Yeti after the threat actor. It’s probably worth it’s own post and worth reading but here are three highlights.
On page 15 (HT: Damiano Bolzoni) they describe the Network Scanning Module that looks for much more than OPC servers. It is scanning for Modbus (502), Siemens S7 (102), EtherNet/IP (44818) and ports for two proprietary ICS vendor protocols. Much like Stuxnet, I expect we are just starting to learn what Havex’s ICS capabilities are. Is it asking too much for DHS/INL to actually perform research and inform the community? It’s understandable, after the fact, why they didn’t research Stuxnet, but this is only the second piece of public ICS malware. Stop sending fly away teams for telnet password cracking attacks and other corporate network exploits and use that pricey ICSsec expertise developed over the last decade.
Kaspersky identifies the Swiss company, Mesa Imaging. This is what we were told and is very helpful for identifying the target. Mesa Imaging is not an ICS vendor. So what company or country sector is using eWon, MB Connect and Mesa Imaging products? That is the best clue so far for who the threat actor was targeting with that phase of the attack.
Kaspersky states there is not enough data to identify the Crouching Yeti threat actor. Some have pointed the finger at Russia, but I’d agree that there is not dispositive evidence in the public at this time.
Belden announced Tofino 2.0 this week. Lot’s of good technical info surrounding the obligatory marketing hype in Eric’s blog entry. I want to dig into the technical details more in a future article and perhaps podcast. The EtherNet/IP Deep Packet Inspection had to be a bear to write, and I’m looking forward to running it through some use cases. When are we going to see this technology integrated into a PLC Ethernet module?
Take note of the latest ICS-CERT advisory from the Crain/Sistrunk DNP3 Telegyr 8979 master fuzzing. This one is related to a SUBNET Solutions product. Most important is a 13-word sentence: “SUBNET had also determined the root issue was in the GPT software library“. The GPT software library was sold by ASE to a number of ICS vendors that now have a latent, remotely exploitable vuln that is available to all. Shouldn’t ICS-CERT be disclosing these vendor names so affected electric utilities can take action? This could be a difficult fix because ASE is no longer selling the GPT/Protocol Pak, but SUBNET found a way.
All software needs to be part of your security patching program … including your security software. Latest example is a new 0day in Symantec Endpoint Protection (SEP). It’s also why you keep your attack surface as small as possible.
Cisco takes your IoT and raises it to Internet of Everything (IoE). They announced partnerships with Rockwell Automation and Yokogawa to support “risk management and compliance for industrial control environments.” It’s one of those press releases that is hard to digest. “It addresses risks using a combination of people, process and technology.” The best I can tell is Cisco will be helping Rockwell Automation and Yokogawa design Cisco products into their ICS.
Graham Speake made the move from Yokogawa to NexDefense. NexDefense is another Mike Assante venture that is trying to commercialize the INL Sophia product. Mike also heads up the SANS ICS security program, and Graham is a SANS ICS security instructor as well as longtime active participant in ISA99. Good luck Graham.
You can get some great tools at reasonable prices by backing the right Kickstarter campaigns. We will be taking the RFIDIer out to an assessment next week, and we should be soon getting our HackRF’s. Look for soon to follow reviews so you can consider them for your toolkit.
I was talking a while ago to Justin Engler, a friend who also happens to be a really talented web app and mobile app security researcher, about the popping-up of ICS management software for mobile devices. He theorized that mobile apps for ICS would be an interesting place to watch for bugs nearly three years ago. Dale’s recent ICSJWG Q&A over mobile device security gave me a little motivation to dig into some sample apps and see how the field actually looks. The results highlight some of the issues that your organization will run into if and when you decide to adopt mobile.
The focus of this post is not just application security. While there are a few specific vulnerable applications mentioned, I think that the big lessons should be ones of architecture and integration challenges. The current lot of ICS management apps pay little mind to securing access or preventing bad operation. Even an app with ‘secure’ on its product homepage may leave you wide open.
I decided to pick on Android simply because my only jailbroken iOS device at the moment is so terribly destroyed from years of abuse that installing new apps is a nonstarter. There also seems to be more interesting control systems apps for Android at the moment.
A quick survey of Google’s Play store for terms such as ‘SCADA’’, ‘PLC’, and ‘OPC’ turns up a few applications worth checking out. Unfortunately there are no apps that I could find which do what Dale prescribes: obtain safe, accurate, remote, ‘read-only’ access to control system data. Doing so will require a lot of backend work on your part.
Let’s take a look at two interesting vulnerable applications.
You are pounded with the message: ICS security is different than IT security. The fact is the Operations Technology (OT) in an ICS is a mission critical / high value IT system and needs to be treated like one. Don’t let the ICS is different argument allow you to accept fragility and insecurity in your OT.
The trigger for this entry is Bob Huba’s article in ISA’s Intech: Top Ten Differences Between ICS and IT Cybersecurity. The article is well written and accurate in comparing ICS and IT Desktop management, but that is the wrong comparison. ICS should be compared to mission critical / high value IT where companies can tell you how much money they lose, and it’s often a big number, for every minute of downtime.
There are plenty of mission critical IT systems that have availability requirements that rival ICS (1: Security Objectives). They have a better availability strategy than the too often ICS approach of make it redundant and don’t touch it if it is working. This is software and hardware that must be managed and supported to meet the availability requirements and not devolve into fragility.
Bob’s 8: Untested Software applies to both high value IT and OT. The change management around both includes rigorous testing, phased deployment, rollback strategy, etc. If you load up a patch or upgrade in high value IT and bring something down, you will see a similar management reaction to an ICS outage. Clearly the physical danger in a lack of integrity or availability is a factor in ICS, but do you honestly believe that a large financial impact is not treated with a similar level of concern to a business?
#9: Patching and #10: Security Inconveniences are ICS excuses and actually make the ICS more insecure and fragile than the high value IT. An ICS needs to be designed with a plan to support and maintain the mission critical IT system that it is. Would you put in physical equipment and not consider what maintenance will be required to the physical equipment to support reliable operation? I always point to Langner’s Robust Control System Networks as the book to hand an engineer who needs to understand why a fragile ICS is a problem.
The #3: Network Topology for ICS actually sounds like high value IT systems. #2 Network Segmentation is a partially valid difference. Some high value IT systems are Internet accessible, others are not. The high value IT systems will use DMZ’s and least privilege rulesets that quite frankly are much more restrictive than the average ICS firewall ruleset.
#6: User Accounts really is not a valid difference between IT and ICS, except that ICS too often accepts poor identification. Role based authorization is common in both as are other authorization methods and principles. Authorization has traditionally been an area of strength for ICS, and increasingly this is being performed in Active Directory.
There are a few areas of genuine ICS and High Value IT difference that Bob points out:
High value IT does not typically have Safety Instrumented Systems (#7: SIS)
The Purdue model does not apply to IT (#4: Functional Modeling), but you could argue that they have similar data flow and responsibility models with different names.
Many of the #5: Physical Components are unique to ICS … although you are seeing some of these physical components in a Data Center.
I agree with Bob that if you treat ICS HMI, EWS, Historians, Real Time Servers, PLC’s, RTU’s … like desktop user PC’s you will cause some serious problems, but this is a flawed comparison.
There is a lot the ICS community can learn and emulate from Mission Critical / High Value IT in designing, deploying and maintaining a robust OT environment. We should be leveraging their knowledge and processes rather than pushing them away. And certainly we shouldn’t continue down the path of install and don’t maintain because we didn’t plan to support the OT and don’t know how.
This is not to say that High Value IT is perfect. There are systems running with uncertainty and fragility and just hoping things won’t break … a similarity with many critical infrastructure ICS.
After the PG&E substation shooting, FERC had ordered NERC, as the ERO, to develop and submit a Physical Security Reliability Standard within a very short time frame for this type of work. NERC complied and now FERC says they will approve the standard with two changes. FERC wants the ability to add or remove facilities from the critical facilities list. While they say this would be “exercised only rarely”, this is a crack in the door or slippery slope that is likely to give utilities heartburn. FERC also wants to replace “widespread instability” with “instability”. There needs to be an adjective in front of instability.
Send us your session ideas asap to have the best chance of getting on the agenda. All we need is a short description and time requirement mailed to s4@digitalbond.com.
We are calling it S4x15 Week now because it goes Tuesday – Friday (Jan 13-16 in Miami Beach):
Tuesday – OTDay and ICS Village Opens
Wednesday – Day 1 of S4x15
Thursday – Day 2 of S4x15
Friday – ICSage:ICS Cyberweapons and Advanced Topics ICS Security Training
I wish that we could sit back and wait for all the great sessions to come in, but history has shown that we need to hunt for this great work and unknown talent. If you see or hear about anything that we should chase for S4x15 week, please let us know.
Last year was a big step forward for the ICS security community. We moved past low hanging fruit; we brought in some top security researchers from outside the ICS space; and there was a new focus on what an attacker would do after successful exploit.
We are looking forward to seeing some new and amazing work for S4x15.
Way back at the Spring 2014 ICSJWG meeting, Dale announced that Digital Bond is opening a new division — Digital Bond Labs. This week, we are officially opened for business…and we are hiring.
Digital Bond has a long reputation for building the tools that other ICS consultants use ten years down the road. It seems that every other talk given in the ICS security community lately make reference to Digital Bond’s intrusion detection signatures, Nessus audit files, or Project Basecamp exploit demos.
At Digital Bond Labs, we aim to provide the best in the business at breaking control systems software, security add-ons, and access control systems for both end-users and vendors. Our goal is simple: break all the things that make all the things so that we can rebuild them to be more robust and more secure. In the Digital Bond tradition, we will also continue to focus on valuable research to share with the ICS community.
We are working with Robert M. Lee and his publisher to get SCADA and Me in Japanese for a giveaway on OTDay of S4xJapan (agenda and registration open on Aug 4th). I wish I had the page above as a hidden slide to pull out at ICSJWG last month.
While most of my presentation involved the secure and insecure way to use the cloud in the future for analysis of process data, the most contentious point was on remote access. The easiest way to get into an ICS with a good security perimeter is to compromise an administrator, engineer or technician that has remote access to the ICS. The ICS Spear Phishing session at S4x13 showed how something as simple as a fake maintenance bulletin would have led to compromise of over 25% of the targeted users with remote access to the ICS.
Here’s the basic solution. Push the data out so the right people can view it without jeopardising the integrity and availability of the ICS. Have a physical disconnect for the remote access, and close the connection only in emergency situations following a defined process. Use your automation skills to put this capability on a display in the control room with the appropriate alarms and logging, and auto open after a time limit. If you are having multiple emergencies a week that require remote access your system is not under control or you are understaffed.
Someone in the audience, who actually is in the business of advising industry, pushed back hard at these limitations on remote access. Paraphrasing he said “c’mon we all know that this generation is going to demand and have remote access with a control and admin capability from their smart phone in their basement. What do you recommend to secure this?” This is when I needed the SCADA and Me page. “If you can control it from a phone — so can Bad People.”
Lior Frenkel of Waterfall said something after my session that I told him I’ll be stealing from now on. “You’re part of the critical infrastructure. Act like it!”
Give eWON some credit. They released information that their website was compromised for a short time in January, and issued an updated notice late last week on their home page. Still nothing on the MB Connect or Swiss vendor site to tell customers they may be compromised by Havex if they downloaded and ran their software. Companies are going to have security incidents; customers should be looking and considering how they respond.
Alstom Grid has a new product coming out in response to the PG&E substation shootings called e-terrasheriff. It will detect and report gunshots at unmanned substations, and presumedly integrate this into the e-terra SCADA displays.
DHS has opened the CFP for the ICSJWG Fall Meeting. After attending and speaking at the Spring Meeting I was going to pass on this one, but holding it in Idaho Falls will dampen attendance.
The first release of Automotive Grade Linux is out. “Each component includes a detailed Design Requirements Document (DRD) with descriptions, use cases, HMI flows, graphical assets, architecture diagrams and more.”
We have always appreciated the Swedish contingent that has supported S4 since the start. Now they have created their own event, 4SICS, Oct 22-23 in Stockholm. They are working on the agenda, but they already have some great technical ICS talent in Europe lined up. Lueders, Santamarta, Hjelmvik, …
DHS is looking for a lead and partners for their Critical Infrastructure Resilience Center of Excellence (CIRC). “Each COE is led by a U.S. college or university and has multiple partners. COE partners include other academic institutions, private industry, DHS components, DOE National Laboratories and other Federally-Funded Research and Development Centers (FFRDCs), other federal agencies that have homeland security-relevant missions, state/local/tribal governments, and first responder organizations.”
A Honeywell help wanted ad is illustrative of how ICS vendors are trying to generate revenue from cyber security. “This position will be responsible for leading, managing and growing the Honeywell Process Solutions (HPS) industrial cyber security global remote managed services business.”
First a reading tip to save you time. Most of the 840 pages are weekly reports from the DHS Control System Security Program (CSSP). There is a ton of repetition as each week’s report carries forward all of the previous week’s items. So go straight to page 750 and you will see the reports going backwards from 19-23 Nov 2007 to 22-26 Jan 2007.
The most interesting excerpt is from the 12-16 March 2007 report:
The CSSP large scale validation test of a significant control systems vulnerability (Pandora) was successfully completed at the Idaho National Laboratory on March 4, 2007. Results and findings from the test are being documented and significant follow-on activities are anticipated. The Tiger Team formed to coordinate activities for this vulnerability will meet on March 13. to U/S Foresman is scheduled to be briefed the afternoon of March 13. Briefings to Secretary Chertoff, House Homeland Security Committee and White House Homeland Security Council are anticipated.
After that Pandora entry there is no other mention of Pandora in the weekly reports. It evidently was classified and changed its name to Aurora. A meeting to discuss the technical details of the Aurora vulnerability appears next in the 19-23 Nov 2007 weekly report on page 751.
There were mentions of this “large scale validation test of a significant control system vulnerability (Pandora)” in the weekly reports prior to the test. A few other tidbits:
It was a large scale test with an estimated cost of $2.8M (page 57).
There are some good pictures of the physical site beginning on page 100.
Pages 70 and 71 have some good examples of specific systems that could be affected by Aurora.
There is mention of a Control System Malware Identification Team being formed by the CSSP back in Jan 2007 (page 233). Let’s put this team on Havex.
A Firmware Upgrade Vulnerability report is discussed on page 165. I don’t remember this being issued, but it was seven years ago and DHS was calling these insecure by design features as vulnerabilities back then.
The mention of CSSP working with JASON – an independent group of scientists that advises the USG and particularly intelligence is interesting, especially back in 2007. Stu….
The mitigation strategy memos start on page 36. The early briefing milestones were met, but little else after that seems to have been accomplished and much of the detail on what was to be done is redacted. They do show a plan for software and hardware fixes being developed and deployed within two to three years.
The technical team memo on page 821 is worth a read.
_____
It’s been seven years since that turbine shook and the smoke came out, yet I always thought Aurora was a lost opportunity.
The real beauty of the Aurora demonstration was it clearly showed that a cyber attack could affect a physical process. The specific vulnerability they chose to achieve this, while not unimportant, was not the main point to take from Aurora. It was an effective and dramatic demonstration.
Aurora should have led a massive DHS and US Government push to address the insecure by design ICS that run the critical infrastructure. Instead of taking this and leading a massive PR and bully pulpit campaign building off of this expensive but effective demonstration, people lost their jobs because the video and secret got out.
Perhaps the idea of physical damage through a cyber attack struck too close to Stuxnet, or maybe it didn’t have the internal support and program to leverage the successful demo. Whatever the reason it was a lost opportunity.
I knew it was lost during the Congressional Hearings. Senators and Representatives asked the august panel from DHS, NERC, utilities, etc. if the Aurora problem had been fixed. Rather than use the question to pivot and highlight Aurora is a small symptom of the larger problem, the experts would go into the plan in place to address Aurora.
I can’t end this long post without a nod to my friend Joe Weiss. He has beating the Aurora drum harder and longer than anyone else. Perhaps this will give him more ammunition for his cause. It is difficult to reconcile Pandora being called “a significant control system vulnerability”, being classified, resulting in all those briefings, tiger teams, remediation plans, … and the relatively small expenditure and effort to address the “Aurora vulnerability”.
Why hasn’t ICS-CERT or some other CERT or the security vendors issuing bulletins announced publicly the three ICS vendors that were distributing malware with their ICS software and the energy sector websites redirecting to a malware delivering site?
It’s baffling. Perhaps the security vendors have a valid profit motive for keeping it secret, but the CERT’s are largely in place to aggregate and spread this information. I’m told the information is in the US-CERT Secure Portal, but this is a terrible way of alerting the affected asset owners.
If the names of the vendors that unwittingly spread Havex were made public, the wide coverage would likely reach most of the affected asset owners.
It is also regrettable that most of the ICS vendors involved in the Havex distribution have not come clean on their web site to warn their customers, more on this below.
Next: The Hype
For these attacks to have a significant impact on the US or other countries’ energy sector the vendors distributing the software with malware would have to a good size client list in the sector. (And we would have to make the leap that asset owners actually update software)
A profile of the compromised vendors’ customers would help understand how widespread the impact is and perhaps what specific asset owner, sector or country is being targeted. So who are the compromised vendors?
MB Connect Line
Michael Toecker quickly identified MB Connect Line as one of the vendors by looking at some public malware samples.
So, it looks like these guys that got owned and waterholed by #Havex. http://t.co/7JiS7WsNXD Wonder who the other ones were?
This is likely company #3 in the Symantec post. The MB Connect Line site states wind turbines and biogas plants, along with other energy infrastructure systems are the applications for their products. Ironically they also highlight their mbEagle product, secure detection of Stuxnet and other manipulations, and mbSECBOX, security for S7 PLCs. We also have a few independent sources confirming MB Connect Line is the German company.
This is a very small company outside of Stuttgart trying to gain a foothold providing remote access solutions in tdistributed energy resources. The impact to the critical infrastructure of this company distributing malware along with their software would be minimal in Europe, and minuscule in the US.
I could not find any mention on the MB Connect Line site that they had unknowingly distributed Havex and what action customers should take.
eWON
Search for “VPN access to PLCs” and the first response was eWON in Belgium. Multiple other trusted sources independently told us or confirmed that #1 in the Symantec post was eWON.
We have never seen this company’s products in the US. Their impact to the US energy sector would be minimal. Perhaps they could have an impact in Europe. We will ask around with our European friends and find out more. It is clearly not one of the major vendors that would have a widespread impact.
eWON disclosed the website breach back on January 30th (note the 250 download number matches Symantec’s description), but they did not appear to know the OPC aspect of the Trojan and have not issued an update now that it is high profile.
Swiss Company
The F-Secure article stated that the three vendors were in Germany, Belgium and Switzerland, so the last affected vendor is a Swiss “manufacturer of specialist PLC type devices”. We eventually found the name of the vendor, but not in a way that we can disclose at this time.
If our sources are correct, this company would have a smaller impact on energy sector than eWON or MB Connect Line. There is also no notice of the Havex distribution on their site.
Energy Related Site Redirects
Symantec describes the other avenue of infection as:
comprising a number of energy-related websites and injecting an iframe into each which redirected visitors to another compromised legitimate website hosting the Lightsout exploit kit.
Symantec provides a redacted list, on page 15 of their report, of five “energy control system” companies and six “energy” companies that were redirecting visitors to a compromised site. These companies were in France, India, Italy and Norway.
Again it would be helpful if these energy control system and energy sites were made public so asset owners could be alerted that they may have been compromised. We do not know these sites, but we have been told they are not big or even medium players in the energy sector. They are closer to a MB Connect Line or eWON rather than an ABB or Siemens.
Hype Summary
A few sentences out of longer articles from Symantec and F-Secure, mixed with some selected quotes from ICSsec pundits, and combined with an absence of information on what software and sites were compromised has led to the hype in the press.
The Target
The three ICS vendors distributing software with Havex are terrible watering holes if you want to attack the US energy sector and not great watering holes even for European energy sector. A couple of possibilities:
The attacker was going after a specific target that the attacker knew was going to use the compromised ICS software. Note I said going to use, not is using. The attacker needed the target to download the compromised software, and it is still rare for asset owners to update software.
The attacker was trying a proof of concept attack. How effective could this software be at finding and enumerating OPC servers? An attacker might want to know this before they compromised more popular energy sector software being deployed in their actual target organizations.
??? who knows ??? The key is the customer base of these three companies. While small, perhaps they had significant penetration in a sector in a country. Take a look at the intersection of the MB Connect Line, eWON and Swiss company’s customer lists.
The ICS Portion of the Attack
The Havex code itself is highly interesting for the ICS community because it is only the second publicly acknowledged occurrence of an attack using the insecure by design ICS protocols as part of the attack. I’m wary of the early returns fully understanding the impact of the ICS code, but it is safe to say now that it is at least doing some identification and enumeration of OPC servers.
While OPC can be used for monitoring and control, it rarely is in critical infrastructure or any SCADA or DCS of any size for a variety of performance and historical reasons. Perhaps that will change with OPC UA in the future, but today you see it used primarily for passing data to and from systems from different manufacturers. For example, the OPC interface is used over 50% of the time to get data in and out of the very popular OSISoft PI Server even though OSIsoft has 100′s of interfaces.
Attacking OPC servers can be a good way to get through the corporate/ICS security perimeter and also to jump from ICS to ICS. It is a good target.
ICS-CERT testing has determined that the Havex payload has caused multiple common OPC platforms to intermittently crash. This could cause a denial of service effect on applications reliant on OPC communications.
This may be nothing more than poor code quality of the OPC servers they are testing. We have personal experience and seen multiple S4 talks that show how easy it is to crash an OPC server.
Kaspersky issued a research report on Havex they called Energetic Bear – Crouching Yeti after the threat actor. It’s probably worth it’s own post and worth reading but here are three highlights.
