Friday News & Notes

We added a bunch of info to the S4x15 site including the newly designed banner, see below. We are almost through the first 50 tier ticket pricing (42 sold).


“DHS ICS-CERT” and FBI announced, a bit clumsily, that they will be touring 13 cities across the US and providing ”a series of SECRET briefings …for cleared asset owners/operators. …  These briefings will provide additional context and information about the BlackEnergy campaign as well as the Havex malware that both targeted industrial control systems.” Sounds like a worthwhile program if they have unique information. I always wonder why these briefings happen after, rather than before, the information is released publicly by researchers and vendors. This is related to an ICS-CERT Alert issued this week.

Some good news on the INL front, they recently added Andy Bochman to the team. I’ve always admired Andy’s writing on Smart Grid security and other ICSsec matters when at IBM and in his own startup. Good luck Andy.

Fireeye released a whitepaper on a Russian organization they are calling APT28. It does not appear to have any critical infrastructure ICS aspects, although some of the government systems being attacked or having intelligence gathered could be ICS.

The team at Netrecsec wrote a nice blog summarizing the three vendors who were distributing Havex infected software.

What Does ICS-CERT Do?

This post was inspired by two tweets from Reid.


First, DHS needs to stop putting everything they do under the ICS-CERT umbrella. There is a CERT function, and there is a bunch of other non-CERT activity. The naming confuses everyone, and you would almost think that is intentional.

Next, as Reid suggested they should be very clear about their vulnerability handling processes. Right now it is coordination of what researchers submit and the vendor response. There is no analysis, no evaluation of impact, no validation of the vulnerabilities, and no validation of the fix. If the vendor says it is fixed, the alert or advisory says it is fixed. The vendor is not even asked how they fixed the vuln. The process, the best that we can tell, is simply coordination of messaging from other peoples info. Figure out what boilerplate fits best, pull some info from the vendor announcements, and put out an Alert or Advisory.

You probably surmise from my tone that I think this is inadequate and actually of little use. It is particularly harmful when they measure success based on the number of alerts and advisories issued. My recommendation would be to shut ICS-CERT down and just roll it into US-CERT. The whole purpose of ICS-CERT at its creation was to provide second level support for US-CERT when ICS vulns were found. We did not need to replicate the existing coordination function.

However, I realize some see value from the Alerts and Advisories, so I would count it a success if ICS-CERT was simply forthright about how they handle ICS vulnerabilities and generate Alerts and Advisories. Reid is right that the public has assumptions about what they are doing that are totally wrong.


I’ve been sitting back and watching to see what activity Reid’s S4xJapan talk would generate. When he found the vulnerabilities in Version 2 of CoDeSys it generated some Advisories that eventually stated the problem was fixed in Version 3 based on the vendor provided information. As we now know Version 3 has the same vulnerabilities as Version 2.

Yet two weeks later there has been no correction or updated Advisory. This is an issue that affects PLC’s and RTU’s from over 100 different vendors, and many of these vendors and their customers believe all is well since they are running on Version 3 of CoDeSys.

Reid showed the exploits on two Japanese products, one from Hitachi and the other from Sanyo-Denki. The later is used to control robot arms. There have been no Alerts or Advisories for these specific examples or the 100′s of affected products.


To be clear, I’m not saying ICS-CERT should jump every time a researcher demonstrates a vulnerability. The whole vulnerability in ICS is overplayed given that ICS-CERT does not consider insecure by design as a vulnerability.

They should have a clear and public set of procedures for vulnerability handling so the community can understand what they can expect and how they should interpret the Alerts and Advisories.

  • Critical Intelligence

Dynamic Zoning / S4x15 Great Debate

One of the most thought provoking sessions at S4xJapan was Wataru Machii of the Nagoya Institute of Technology’s session on Dynamic Zoning in an ICS. One of the great things about S4xJapan is it provides videos and sessions in the Japanese language. The downside is it is not accessible if you don’t speak or read Japanese.

The basic concept is that the security zones and conduits in an ICS should change dynamically based on the state of the ICS. There are two parts to this. The first is how to set up the zone and conduit states that you will switch between based on ICS state. And second, what triggers the change in state. Machii san had good ideas on both of these questions, but it is an area worth further investigation to identify a methodology that can be applied across sectors and customized by owner/operators.

This session was the inspiration for our S4x15 Great Debate: Can Operators Use a Security Display. The control room is often staffed 24×7 by Operators, but they have little security knowledge. The S4xJapan session made me consider if the Security Display could be simple enough that an Operator could trigger a change in the dynamic zone based on the information in a security display.

This is only one possibility.

In the Great Debate we will have attendees submit their single screen security display, and importantly, explain the defined Operator actions based on information shown in the security display. A handful of attendees will explain their security displays, others will be flashed on the screen for consideration, and their will be skeptics voices heard I’m certain.

By the way, this was one of two sessions at S4xJapan from the Nagoya Institute of Technology. They have an active ICS security program there and seem to be working on research with real world implications.

Havex Deep Dive

IMG_20141013_045855909_HDR At S4xJapan in Tokyo I presented on a couple things, this post is about Havex. During the talk I am speaking slowly and plainly as the conference was being simultaneously translated into Japanese. Altering your speaking style to help translators is a good exercise that everyone should do. It forces you to be concise and use simple language but warning: it’s a bit dry.

There has already been some excellent articles/research published on the ICS relevant aspects to Havex. Regarded as the second major ICS malware, Havex garnered some media attention which prompted the need for more analysis, writeups, and talks like this. The goal of the talk is to give an overview of what Havex is, what ICS components it has, and then to dive in to the codeflow of the downloadable OPC scanning module. At the end of the talk hopefully the What and How questions are answered but Who and Why still remain.

After the presentation we had some good discussion about OPC module internals/encryption as well as general ICS malware campaigns. The conference did well to foster that type of communication and I appreciated working with everyone there.

I received my samples from insecure Command & Control servers as well as from professional contacts. Shoutouts to Kyle Wilhoit, Daavid, other Kyle, Kaspersky, and Daniel.


Google is maybe a little TOO helpful in trying to save us from ourselves. In attempting to forward on samples I discovered that Google seems to try basic password attempts on encrypted zip files. Putting the samples in a zip archive with the standard password “infected” was insufficient to get past Google virus detection but changing to “infected1234″ worked fine (without changing any file names). Creepy….


arch_snowpeak_John_FowlerIf you haven’t read up on the latest debacle in hardware security, I recommend reading EEVBlog’s writeup, or Sparkfun’s blog post, or follow the FTDIGate hashtag on Twitter …

For a summary, FTDI (Future Technology Devices, Inc) released a driver update via Windows Update yesterday. The driver update intentionally bricks unauthorized copies of FTDI’s popular USB to Serial converter by overwriting the USBID. Reports are that their mechanism for selecting devices was actually imperfect, resulting in the driver also bricking one of FTDI’s own legitimate chipsets including the FT2232H.

In Digital Bond Labs, one of the things that we discuss with vendor clients is hardware security. When considering how to build hardware security, one of the items we consider is, what can a vendor do to protect their intellectual property from being cloned?

We’ve seen and purchased a lot of cloned hardware for sale on eBay over the years.  The equipment mostly comes from the Shenzhen/Guangzhou areas of China, and sells for a small fraction of the normal retail price of legitimate equipment. Everything from specialized PLC programming cables (chiefly for Siemens equipment) to knockoff digital protective relays can be found in this way. Knockoff quality ranges from pretty good and fully functional equipment, to equipment that is well-packaged and turns on, but which does nothing functionally.

FTDI’s response is a bit surprising, especially given where its devices are used. FTDI chipsets are a daily part of life for many engineers who work with critical infrastructure, medical devices, and other big industries. It is basically impossible for an end user to know their supply chain, to know whether every FTDI chip in USB devices they own is legitimate. Cables using FTDI chips often come included with hardware that has a serial port, such as network switches, PLCs, and other embedded devices. The chips are also frequently used internally in devices with USB ports. An end user will have a difficult time telling which of their cables or devices is using a legitimate FTDI chip, and which may contain a clone. It can be difficult for manufacturers of these chips to even know, as supply chains can be surprisingly obtuse.  There are some good hackers working on detection scripts, although there may always be corner cases that are not properly detected.

Unauthorized hardware cloning is like insecure-by-design software in one way: once cloned hardware is out in the wild, it’s really too late to try and secure it. While FTDI found a way to disable the unauthorized equipment, the blowback from doing so is going to be extreme. This is precisely because it is the end users that wind up with the broken equipment. I immediately picture a nightmare scenario in which an engineer absolutely needs to reprogram a piece of field equipment in an emergency, and cannot because they didn’t know their chip was not legitimate.  Cutting that engineer off is clearly not the right solution. I for one will recommend that my vendor clients discontinue using FTDI’s chipsets in future products in favor of competitors like Prolific. This incident suggests that FTDI’s management is too reactionary and is not thinking in the long-term about their own security or the safety of end users.  A better idea would have been to implement detection in the driver similar to the script linked above, and simply warned the end user that the driver would stop communicating with the illegitimate chip after a certain date.

Adding good security to a chipset, or any hardware design, takes some dedication and has a cost associated with it. In this case, FTDI wants it both ways: they want to avoid paying the upfront cost of building their device in a manner that is difficult to clone, yet they also want to stop illegitimate chips. Thankfully, they have relented, but only after seriously harming both their reputation and the legitimacy of software updates.

Image by John Fowler

S4x15 Is Open For Registration!

We have opened the S4x15 website and registration. There still is a lot to add to the site, like the Conference Hotel, ICS Village CTF, Social Events, Area Info, FAQ, … But we have always believed it is important to provide attendees with information on the sessions and speakers so you can make an informed decision.

The agenda looks great and very different than anything you have seen before at an ICSsec event.

Register right away if you want to get one of the first 50 tickets at the same price we have charged every year since 2007. We will be providing event updates on this site. There is a lot to say about the event, but we wanted to get this open for registration.

S4x15 Registration Opens Tomorrow

Registration for S4x15 was scheduled to open today at noon. We have a one day delay, and registration will open tomorrow, Friday, at noon EDT.

Sorry for the one day delay, but we wanted to get all of the accepted sessions into the site so you know what you will be registering for.

Duplicity, Ineffectiveness & Challenge Pass/Fail

CoDeSysReid Wightman of Digital Bond Labs presented Vulnerability Inheritance in ICS at S4xJapan, and he posted the video and a technical article yesterday. I’d like to weigh in on the duplicity of 3S, the ineffectiveness of ICS-CERT, and the challenge passed and failed by integrators.

What Reid showed clearly in his presentation, and in the tools he released, is that the six categories of Version 2 vulnerabilities had not been fixed in Version 3. All that CoDeSys did was modify the software slightly so the previous tools did not work.

Here is a simple analogy. Imagine Version 2 was a door that had no lock. All a burglar had to do was turn the doorknob clockwise and the door opened. Rather than putting a lock on the door in Version 3, CoDeSys simply made it so the door would not open if you turned the doorknob clockwise. But if you turned the doorknob counterclockwise, the door opens. What was needed was security, a lock on the door, rather than some trick.

I don’t know what else to say about 3S/CoDeSys except they have done their vendor customers and the end users a major disservice by saying Version 3 fixes the security problems. At least Festo was honest when they said they were not going to fix the vulns.


ICS-CERT is a vendor megaphone, little more. I know they should expect a forthright answer from the vendor, but is it too much for them to ask a couple of questions on how the vulnerabilities were fixed? These are vulnerabilities that affect 100′s of vendor products.

DHS touts the number of vulnerabilities they have handled as a measure of their value and effectiveness. They add little or no useful information to other disclosures, and they don’t perform even basic evaluation of the information.

It is hard to see any benefit to the DHS/ICS-CERT role in disclosure. Close down shop and move the resources to something more useful. My recommendations for years now is for ICS-CERT to ignore 95% of the vulns and to a great job providing value information on the 5% they deem important.


Which leaves the vendors that integrated the CoDeSys software. We are aware of two vendors that looked at the 3S fixes in Version 3, realized they didn’t address the security problems, and built their own protection into the integrated product. A great example of internal Red Teams and SDL doing its job.

The examples of Hitachi and Sanyo-Denki that Reid used at S4xJapan are the case where the vendor did not adequately test third party software that is integrated into their product. Hopefully this will be a learning experience for the CoDeSys customer base.

All ICS vendors are going to have security issues. The important point in evaluating ICS vendor security is how they fix identified problems, and the root cause of the problem in the development lifecycle.

Image by Lyn Matthews

Vulnerability Inheritance in PLCs – CoDeSys V3 Edition

shibuya_guwashi999At last week’s S4xJapan conference, I gave a talk about insecure-by-design vulnerabilities inherited in PLCs, and provide two vulnerable Japanese PLC vendors as examples of those inheriting security issues.

During the talk, I am speaking purposefully slowly — the conference was being simultaneously translated into Japanese and it was important to allow the translators time to catch up.

Inheriting insecure-by-design vulnerability problems is particularly troublesome.  In the talk, I look at two Japanese vendors who have inherited vulnerabilities from third-party software.  In both cases, it appears that the vendors affected are either unwilling (Sanyo Denki) or unable (Hitachi) to fix their systems.  Even if they would or could produce patches for their PLCs, external software dependencies mean it is unlikely that anyone will actually apply those patches.

Sanyo Denki is actually the more interesting case to me, because the same hardware and firmware is in use by many vendors. Their SanMotion C controller is used primarily for servo-actuated robotic arms.  While that isn’t likely to be critical infrastructure to the general public, it is likely a critical system to a manufacturer that relies upon a robot arm in its manufacturing process.

I’ve said this before — 3S Software is in a unique position in the industry.  Hundreds of vendors run their PLC ladder logic runtime.  I wish that 3S Software would introduce some security into the runtime, for example integrating authentication against a RADIUS server or other network management server, and add proper protocol integrity (encryption or at least message signing). They could revolutionize the industry and become the secure logic runtime of choice for security-concious vendors. But…in the words of Ellie Arroway…that continues to be my wish.

There were some fun challenges in making a tool to speak the newer CoDeSys V3 protocol. Continue reading for a little bit of reverse engineering and code-gawking that didn’t end up in my talk.

Read More

S4x15 Theme & Other ICS Security Events

ICS Security Event

Registration for S4x15 Week will open this Thursday, and be ready if you want to get one of the 50 lowest cost tickets to the event.

We are still working on the one word theme for the event. Some of the leading contenders are Advance, Beyond, and Push. I’ve seen the session abstracts and it is going to be a novel and exciting event, a significant leap forward in the ICS security research community. The gap between S4 and other ICS security events has grown significantly over the last three years and S4x15 will extend that even further. In fact, the technical research and discussions at S4 are going so far beyond the standard ICS security event that it is almost unrecognizable that they are all in the same general category ICS security events

This is not a negative comment on SANS, ICSJWG, WeissCon and the international events. There is still a need to provide basic ICS security education and awareness to a huge portion of the ICS community. In fact, the number of people who need one of these traditional and excellent events is 100x or even 1000x the number of people who need an event like S4.

The problem is the top researchers and thought leaders in this space need to continue to push forward. I guess we could worry about getting too far ahead, outrunning the supply lines. However if we have an event that is accessible and understandable to the newcomer to ICS security, or even an advanced beginner or intermediate, it is worthless to the leaders in the ICSsec space. The S4 target attendee is the type that has long outgrown the other ICSsec events.

A very brief history of recent S4 conferences:

  • S4x12 was Project Basecamp (Insecure By Design), Stuxnet Deep Dive (Detailed discussion of first ICS cyber weapon) and the first session on Internet connected ICS. It opened a lot of fronts and took off the gloves.
  • S4x13 was ICS Exploitapolooza. There was session after session showing a pathetically insecure ICS application or device and watch the speaker exploit it. We had over 50 0days at the event. It brought a number of new researchers into the space, but the point was beaten to death for the S4 audience. This was a turning point.
  • S4x14 was a big step forward. ICS low-hanging fruit exploits were banned. Novel attack techniques for ICS and a greater exploration of what an attacker would do post exploit were the highlights. Some big names in security research stepped into the ICS realm. Plus we moved up to the ballroom, added OTDay, ICS Village, and ICSage: ICS Cyber Weapons as well as a lot more fun at the social events.

So what is in store for the main two days of S4x15? It is a continuation of what was hinted at and started at S4x14. The focus is on the engineering and automation aspects of attacking and defending ICS. We have some great session on simulation for analysis and defense, some novel attack techniques, basically things that you will not see anywhere else. … and there will be triangles.

We have said from the first S4x07 that this event is not for everyone. If you want to discuss OT vs IT or information sharing or what some government agency is doing, go to one of the other great events. If you want a lot of technical meat, new concepts and to mingle with best minds in the ICS security space you should grab a ticket for S4x15.