Havex Hype & Unhelpful Mystery

ICS MalwareUnhelpful Mystery

Why hasn’t ICS-CERT or some other CERT or the security vendors issuing bulletins announced publicly the three ICS vendors that were distributing malware with their ICS software and the energy sector websites redirecting to a malware delivering site?

It’s baffling. Perhaps the security vendors have a valid profit motive for keeping it secret, but the CERT’s are largely in place to aggregate and spread this information. I’m told the information is in the US-CERT Secure Portal, but this is a terrible way of alerting the affected asset owners.

If the names of the vendors that unwittingly spread Havex were made public, the wide coverage would likely reach most of the affected asset owners.

It is also regrettable that most of the ICS vendors involved in the Havex distribution have not come clean on their web site to warn their customers, more on this below.

Next: The Hype

For these attacks to have a significant impact on the US or other countries’ energy sector the vendors distributing the software with malware would have to a good size client list in the sector. (And we would have to make the leap that asset owners actually update software)

A profile of the compromised vendors’ customers would help understand how widespread the impact is and perhaps what specific asset owner, sector or country is being targeted. So who are the compromised vendors?

MB Connect Line

Michael Toecker quickly identified MB Connect Line as one of the vendors by looking at some public malware samples.

This is likely company #3 in the Symantec post. The MB Connect Line site states wind turbines and biogas plants, along with other energy infrastructure systems are the applications for their products. Ironically they also highlight their mbEagle product, secure detection of Stuxnet and other manipulations, and mbSECBOX, security for S7 PLCs. We also have a few independent sources confirming MB Connect Line is the German company.

This is a very small company outside of Stuttgart trying to gain a foothold providing remote access solutions in tdistributed energy resources. The impact to the critical infrastructure of this company distributing malware along with their software would be minimal in Europe, and minuscule in the US.

I could not find any mention on the MB Connect Line site that they had unknowingly distributed Havex and what action customers should take.

eWON

Search for “VPN access to PLCs” and the first response was eWON in Belgium. Multiple other trusted sources independently told us or confirmed that #1 in the Symantec post was eWON.

We have never seen this company’s products in the US. Their impact to the US energy sector would be minimal. Perhaps they could have an impact in Europe. We will ask around with our European friends and find out more. It is clearly not one of the major vendors that would have a widespread impact.

eWON disclosed the website breach back on January 30th (note the 250 download number matches Symantec’s description), but they did not appear to know the OPC aspect of the Trojan and have not issued an update now that it is high profile.

Swiss Company

The F-Secure article stated that the three vendors were in Germany, Belgium and Switzerland, so the last affected vendor is a Swiss “manufacturer of specialist PLC type devices”. We eventually found the name of the vendor, but not in a way that we can disclose at this time.

If our sources are correct, this company would have a smaller impact on energy sector than eWON or MB Connect Line. There is also no notice of the Havex distribution on their site.

Energy Related Site Redirects

Symantec describes the other avenue of infection as:

comprising a number of energy-related websites and injecting an iframe into each which redirected visitors to another compromised legitimate website hosting the Lightsout exploit kit.

Symantec provides a redacted list, on page 15 of their report, of five “energy control system” companies and six “energy” companies that were redirecting visitors to a compromised site. These companies were in France, India, Italy and Norway.

Again it would be helpful if these energy control system and energy sites were made public so asset owners could be alerted that they may have been compromised. We do not know these sites, but we have been told they are not big or even medium players in the energy sector. They are closer to a MB Connect Line or eWON rather than an ABB or Siemens.

Hype Summary

A few sentences out of longer articles from Symantec and F-Secure, mixed with some selected quotes from ICSsec pundits, and combined with an absence of information on what software and sites were compromised has led to the hype in the press.

The Target

The three ICS vendors distributing software with Havex are terrible watering holes if you want to attack the US energy sector and not great watering holes even for European energy sector. A couple of possibilities:

  • The attacker was going after a specific target that the attacker knew was going to use the compromised ICS software. Note I said going to use, not is using. The attacker needed the target to download the compromised software, and it is still rare for asset owners to update software.
  • The attacker was trying a proof of concept attack. How effective could this software be at finding and enumerating OPC servers? An attacker might want to know this before they compromised more popular energy sector software being deployed in their actual target organizations.
  • ??? who knows ??? The key is the customer base of these three companies. While small, perhaps they had significant penetration in a sector in a country. Take a look at the intersection of the MB Connect Line, eWON and Swiss company’s customer lists.

The ICS Portion of the Attack

The Havex code itself is highly interesting for the ICS community because it is only the second publicly acknowledged occurrence of an attack using the insecure by design ICS protocols as part of the attack. I’m wary of the early returns fully understanding the impact of the ICS code, but it is safe to say now that it is at least doing some identification and enumeration of OPC servers.

While OPC can be used for monitoring and control, it rarely is in critical infrastructure or any SCADA or DCS of any size for a variety of performance and historical reasons. Perhaps that will change with OPC UA in the future, but today you see it used primarily for passing data to and from systems from different manufacturers. For example, the OPC interface is used over 50% of the time to get data in and out of the very popular OSISoft PI Server even though OSIsoft has 100′s of interfaces.

Attacking OPC servers can be a good way to get through the corporate/ICS security perimeter and also to jump from ICS to ICS. It is a good target.

One last note … the ICS-CERT advisory states:

ICS-CERT testing has determined that the Havex payload has caused multiple common OPC platforms to intermittently crash. This could cause a denial of service effect on applications reliant on OPC communications.

This may be nothing more than poor code quality of the OPC servers they are testing. We have personal experience and seen multiple S4 talks that show how easy it is to crash an OPC server.

Image by James Marvin Philips

21 comments to Havex Hype & Unhelpful Mystery

  • “The Havex code itself is highly interesting for the ICS community because it is only the second publicly acknowledged occurrence of an attack using the insecure by design ICS protocols as part of the attack.”

    Exactly. Therefore it’s surprising to see that the ICS (security) community, and most of all ICS-CERT, so far has made no effort to analyze and understand this aspect of the malware.

  • Michael Thib

    To piggy back on the ICS-CERT thing I am also surprised to notice (or not notice) that the German, Belgian, nor the Swiss Certs; as well as, the ENISA Cert are not making statements on this.

    Just shows that fact that Europeans are left more often out in the rain than the US colleagues. It is a sad fact. Maybe Anapur will start to bring something forward in the October conference with ENISA on board.

    Michael

  • Could you care to elaborate on why you think OPC is “insecure by design”, Ralph?

    The protocol itself is not insecure, as it does provide reasonable authentication of both client and server to prevent unauthorized access to tag data. Enumeration is part of the standard, and does not translate into direct access to tag data once enumeration is complete.

    Again, we are faced with a poor implementation of the protocol, and not necessarily the mechanics of the protocol itself. Any subsequent attack that the media to refer to as “sabotage” would easily be prevented using basic security controls that anyone who has read basic OPC documents would understand.

    I would be interested to hear your thoughts on this.

  • Dale Peterson

    Ralph – This is where I get into tinfoil hat territory. We eventually learned why ICS-CERT did not investigate Stuxnet.

    Joel – I think if you look at the OPC Foundations efforts and statements on OPC UA it is clear even they realized “OPC Classic” lacked security. OPC UA is a big step forward in adding security capabilities, although the default installs often allow things like connections from any client with a self signed cert.

  • John Dickinson

    Wonder how the Swiss vendor who ‘develops high-precision industrial cameras and related software’ (from the f-secure glog) morphed into a ‘manufacturer of specialist PLC type devices’?

  • Michael Werthschitzky

    I think it’s too early to see what’s behind. Too many persons involved we don’t know with intentions we don’t know either PLUS 2 Companies that would like to make business on the plant floor. However for me it’s very calming that our ICS hero’s Joel, Dale & Ralph had start the discussion……please keep it ongoing !!

  • Dale – though Classic OPC has its pitfalls, OPC-UA is more than just adding security. It is moving away from the MS dependence and frameworks to a more open architecture. The problems with Classic do not necessary lie in the protocol (OPC is an application layer protocol that is pretty good), but rather the underlying dependence on the MS infrastructure lower in the stack (RPC, DCOM, etc.).

    I just wanted to offer a different point of view when OPC is referred to as an “insecure by design” protocol and then gets thrown into the bucket with truly insecure industrial protocols like Modbus – which I think we all agree is not the same! Modbus, also an application layer protocol, has security deficiencies at the top and has nothing to do with the layers below it.

    Cheers.

  • Joel, let’s end this useless discussion by simply asserting that today no vendor would be so silly to suggest building a technology for accessing control systems (which then even gets abused to access DCS servers) based on DCOM.

  • Not useless, but important topic. OPC was never meant to leave the security perimeter. It has never worked well across firewalls, and for that reason, all best practices suggest not to do this.

    Again, because people take a product and deploy it using poor practices and engineering design is not a reason to discredit the technology.

    FYI … would suggest you dig a bit deeper into MOST leading ICS technologies, and you will find that RPC is the underlying protocol used in most systems for their interprocess communications over local area networks. Note that I am careful to say LAN and not WAN.

    Cheers.

  • Dale – the fact that you would say that you have never seen eWon is not really a credible argument.

    eWon is carried by Industrial Networking Solutions, which is one of the largest distributors of networking and security related equipment for industrial systems in the USA. Trust me, they wouldn’t care a product if it didn’t sell in the USA!

    And another point of fact … “While OPC can be used for monitoring and control, it rarely is in critical infrastructure or any SCADA or DCS of any size for a variety of performance and historical reasons.” couldn’t be further from the truth. In large DCS architectures, it is not uncommon for >50% of the total tag quantities to come from third-party interfaces. It is the cornerstone of most integration technologies in use today, and there are even systems like Emerson’s DeltaV (one of the world’s leading ICS) that depends almost entirely on OPC to bring in third-party protocols (Modbus, DNP, AS-I, Profi, plus many more depending on industry) into their system for L2 integration. Honeywell also uses it extensively in their Experion system lineup for L2 data. This is way below the PI components in the archtiecture, and is common directly with the control (L1) and supervisory networks (L2).

    All in good fun, but this shows that architectures can vary significantly as you move from industry sector to sector, and why generalizations from one should not be assumed accurate in others.

  • Dale Peterson

    Joel – It’s not a surprise that we disagree.

    As I have said I agree that OPC is the universal translator. In fact the sentence after the one you quoted said ” today you see it used primarily for passing data to and from systems from different manufacturers”. Perhaps to clarify, OPC today is rarely used for reads/writes/config to PLCs/RTUs/controllers.

    I’m not sure that this is the important point though. The attacker was wise to select OPC because an OPC server or OPC server interface is found in a high percentage of ICS. I assume we can agree on that.

    Your point that my not seeing eWON does not necessarily equal small market share in US critical infrastructure or elsewhere is accurate. That said, I’m confident the eWON, MB Connect Line and the Swiss ICS vendor are not major players in the energy sector in the US or elsewhere. This is not to denigrate those companies, Digital Bond is even smaller.

    I hope the relevant gov’s and CERT’s are looking at those three ICS vendors’ client lists to see if there is anything in common to try to identify a target or targets.

    Dale

  • Hate to say it … but I concur with Dale on all but one part … (I guess having cross 50YO, I begin to take things more in stride!)

    OPC is definitely used to read/write values to PLCs, RTUs, etc. – that is the cornerstone of the technology. These devices are primary control devices that receive supervisory read/write commands from the DCS or SCADA. That is exactly why we originally developed OPC – as a “device” driver replacement. It has evolved over time and is used for inter-system comms, but still one of the most widely used applications is for direct communication with control devices. That is why OPC-DA is the oldest in the family … and why the system-to-system comms needed additional features later provided by OPC-AE and -HDA.

    Cheers.

    … and have a great Thursday!

  • Hello?……I have the slightly feeling that the discussion turns in the wrong direction and gets lost in details about OPC protocol….Is it possible for you to step back from this an focus more on the WHY? Why was it pushed so hard by Symantec/F-Secure and why ICS-CERT continue with that on the same level….did they know more? Or was it only the attempt from Symantec/F-Secure to hive us the impression that they can make it everywhere..?

    Cheers
    Michael

  • Dale Peterson

    Michael – To be fair, I don’t see any hype in the F-Secure article. They state it is targeted at Europe and profile a few of the target systems.

    The Symantec article and white paper have a few hype sentences that are balanced by other sentences that reduce the hype even describing the affected US sites as “collateral damage”. It’s not a surprise that reporters pick out the juiciest quotes.

    Any ICS related malware is going to get a lot of press and hype. Remember how a potentially compromised water pump in a small Illinois water utility generated a flood of articles.

    My contention is if the CERTs or security vendors had published the vendors names that had been providing compromised ICS software or redirecting to a compromised site, then the stories would have changed to “if you downloaded software from one of these sites in this time period you may be compromised and need to do xyz”. Still would be high volume, sensational articles, but at least directed in a positive way.

    I’ve been encouraging writers and .govs to dig into the client base of the compromised vendors. This will better identify the targets and potential impact.

    ——-

    All that is secondary to digging into the ICS component of Havex and understanding what it is doing.

    Dale

  • Michael Werthschitzky

    Dale …….I didn’t say hype, let me say it is more a way of how they make their business which I sometimes dislike…..but on the other hand I must accept it, keep in mind the huge amount of data they analyze from their installations at their customers networks….today mostly on the business side of the LAN but tomorrow on the plant floor too ?

    Regards,
    Michael

  • Dave

    Coming from a security rather than SCADA background, I was wondering what the OPC was that everyone was talking about. OLE for Process Control (OK, “Open Productivity something”, the important part is the “OLE”). Despite intensive efforts from Microsoft, this has been an attack vector for Windows from the day it was announced until until, well, there isn’t an end in sight. So if Microsoft, with their massive resource and focus on security, can’t get it right, I shudder to think how vulnerable the typical SCADA implementation is. What’s worse is that attackers can then leverage the tools and attack principles they’ve tuned on Windows OLE to target OPC.

    (I’m not saying the decision to use OLE was a bad one, you need to use some protocol for the job, but talk about choosing a target-rich environment…).

  • Dale Peterson

    Dave – It’s funny that the OPC Foundation dropped the OLE for Process Control acronym when Microsoft moved away from the OLE term to DCOM and ActiveX. The use of COM and DCOM make OPC Classic hard to secure and nasty for firewalls. We worked with Eric Byres to write a three part whitepaper series on securing OPC. What we wrote was accurate, but proved not to be practical for a variety of reasons.

    Today about the best you can do is limit access and use tunneling products like the Matrikon OPC Tunneler. Tofino offers a deep packet inspection capability for OPC, but we haven’t checked that out yet.

    OPC UA is what to look towards and ask vendors for. It doesn’t rely on COM/DCOM and has built in security including authentication of the source and data. A big step forward (and nothing to do with OLE).

  • Dave

    @Dale: Thanks for that. I’ve had a quick look at the little bit of OPC-UA security architecture information that’s available online (coincidentally, http://www.digitalbond.com/scadapedia/protocols/opc-ua/ seems to have the best information) and it looks interesting. As a security guy I wish there was something a bit more detailed available to look over(without having to spend a fortune on the standard just to satisfy my curiosity).

  • It is important to not confuse the weaknesses and shortcomings of DCOM/RPC with those of the OPC Application Protocol. I doubt anyone would disagree that there are inherent security problems in DCOM/RPC, which is not the fault of OPC. DCOM and RPC are at the cornerstone of most network-based interprocess communications on many leading ICS systems (both DCS and SCADA). So if you want to talk about problems, open the hood and take a look and you will see what I am talking about.

    Next, though OPC-UA was first released in 2009, there were few products available that supported it during the first 2-3 years. The reason is that legacy system support is one of the primary objectives when evaluating ICS technologies. OPC-DA had a good 10-year head start on UA, which means there were countless installations deployed that could not simply be migrated. Today, there are more than 20,000 OPC products (mostly “Classic”) from some 3,500 vendors – not a trivial upgrade task! OPC-UA undoubtedly adds much needed capabilities to the OPC family, but like all technology deployed in industries that require solid reliability and availability, it is going to take some time. Vendors only began introducing support for UA beginning in 2010, and in “ICS Years”, 4 years is still considered a “new product” that could also introduce new risk to operational integrity that is greater than the risk that you tried to mitigate by migrating from Classic to UA in the first place! (FYI – there are several vendors that even today do not offer OPC-UA support, yet their products are some of the most secure ones available!)

    Being involved in OPC from an implementation standpoint from the early days, those of us that understood the protocol knew how to properly install it in an acceptable manner that provided sufficient resilience. As an example, we never allowed OPC through a security perimeter that required any L3 or L4+ controls. When it was used for site-wide historians, we placed the historian’s collector (acting as an OPC Client) and the ICS servers (acting as an OPC Server) within the same Security Zone, and then allowed more secure application-to-application (like the OSIsoft PI-PI connection) communications across the perimeter. Application Servers that needed real-time data were placed in the same Security Zone as the servers hosting the data, so again, we were able to manage security within the Zone while containing possible vectors and associated risk. Sure, people could take shortcuts and try to save some money, but these same people can equally install a firewall and configure the first rule as an “any any any allow” rule and then complain that their perimeter is not secure! Tunnelers are a nice product, but we still had two obstacles: (1) not all vendors endorsed the installation of these tunnelers on their servers, and (2) it also introduced new risks associated with credential management that could provide a new set of attack vectors.

    Remember, a typical industrial system lifecycle can exceed 15-20 years in the ICS world – compared to a much shorter 3-5 year lifecycle in traditional IT terms. In terms of “real” ICS security vulnerabilities, we haven’t even scratched the surface. This is one reason why it is so vital that users, integrators, vendors, etc. learn how to properly assess their current security risk, and learn how to secure their systems. Glad that I am still the only one offering a course devoted to applied security to ICS! I am excited to introduce a new 3-day course in Fall 2014 that will be entirely devoted to applied security for ICS focusing on three critical areas!

  • Dale Peterson

    Dave – We released a paper on it about four years ago. I’ll try to pull it out and put an updated link.

    The biggest security issue with OPC UA is the way it is delivered and installed. The default for most OPC UA servers, the last time we looked at them, was to accept self-signed certs as valid. So even though the OPC UA server requires a client cert by the protocol spec, any attacker can say here is my cert and the OPC UA server would likely accept it unless the asset owner or integrator had properly created a CA, issued certs and modified the OPC UA server config.

    As a security guy I’d rather see secure by default, but this is still a challenge in the ICS world. They want solutions to work out of the box, and anything IT or security related that makes installation more likely to fail is often removed from the default install. Of course, this leads to availability without integrity.

  • Dave

    I wouldn’t dismiss using self-signed certs out of hand, they can actually be a lot more secure than CA-issued ones. Consider the case of a system trusting self-signed certs based on their SHA-1 fingerprints vs. one trusting CA-issued ones, e.g. anything you buy from GoDaddy, the system using the self-signed certs will be vastly more secure. It really depends on how you authenticate the access, allowing any CA-issued cert in is just as bad as allowing any self-signed cert in. Either running your own CA and only allowing certs from that in, or using a certificate whitelist (the fingerprint-based system I mentioned before) works equally well, the only problem is that most people managing a SCADA infrastructure really aren’t set up to run a PKI as well. By that I mean that if your job is to run a SCADA infrastructure then you shouldn’t also be required to run a PKI alongside it just to make it work.

    (This is a very brief summary of what could be an enormously long discussion :-).

Leave a Reply