While progress on adding basic security to PLC/RTU/Controllers, Level 1 of the Purdue Model, continues to be excruciatingly slow, there is much good news from vendors that make the applications that reside at Level 2.
Vendors offering HMI, Engineering Workstations, Historians, SCADA and DCS Servers and other Level 2 applications are adding security features to their software. Equally important is they are providing quality configuration and deployment advice to their customers in the form of white papers and videos. This is a growing trend for at least the past five years.
Check out Bryan Owen’s 2014 and 205 User Group presentations on securely deploying the OSIsoft PI Server. These are two of many videos and tools they provide to help customers secure their PI deployments.
OSIsoft is not alone. We work with numerous vendors who provide quality secure deployment advice for the architecture, OS hardening and ICS application security settings. We have also seen this type of great information at S4 OTDay and other vendor User Group events.
The ICS Community should recognize and appreciate the vendors that are stepping up to this challenge, but … the unfortunate reality is this quality guidance seems to be rarely followed and new system deployments are not following even the key items stressed in the vendor deployment guidance.
In some cases this is an integrator, consultant or customer that is installing the ICS incorrectly. This is bad enough, but the real shame is when the ICS vendor installs their own solution without following their own security advice.
Many times a year we have a disagreement with the ICS vendor’s deployment team on how their product should be installed, and we have to pull out the ICS vendor’s own documentation to show them they are installing it wrong. Even then more often than not they fight back and tell us why it is impossible to install the ICS the way their own product team recommends.
It’s not hard to understand why this happens. The team that works on design and implementation has a difficult job, and many of these team members have been installing systems for decades. They have a way to make a complex deployment work and are not looking to make their job harder. If they know how to install a legacy configuration, they will install a legacy configuration even on a completely new system.
The ICS vendors need to step up even more and push this secure deployment information and processes down to the deployment teams. I’ll fully admit this is a big task that is unlikely to be 100% in the near term, but we are not seeing installs following ICS vendor guidance even 25% of the time.
Asset owners purchasing new ICS are going to have to step up as well. They will need to get the ICS vendor secure deployment recommendations and add secure deployment requirements to the acceptance tests.
This is a great session for power engineers and those involved in substations to watch. It is an extremely technical session by Dr. Chee-Wooi Ten of Michigan Technological University.
The key point is actually easy to understand. The most critical substations to secure may not be the highest voltage substations, and this session provides a set of mathematical equations to perform an impact analysis to identify the most critical substations.
Dr. Ten gets into the modeling and mathematics in significant detail in the video.
In my talk way back at S4x15 I briefly mentioned a few techniques at identifying interesting parts of an application for reverse engineers.
A lot of times we as reverse engineers load up a firmware or DLL or executable. We want to get hunting for bugs as quickly as possible. Hunting for bugs usually means rapidly identifying the data entrypoints into an application. In the case of networked equipment and networked software, data entrypoints mean network protocol parsers.
One of the techniques that I talked about was using CRC/hashing/crypto algorithm precomputed tables for identifying interesting functions. I thought a little post about this technique was worth writing.
Obviously there is an easy button for some application software: searching for OS functions like recv() is a sure way to find your protocol receiver (and perhaps your parser as well). For deeply embedded systems this is a bit trickier though, as a disassembler will not necessarily know what function in the disassembly constitutes recv() or any other C library call.
Most protocols in the ICS space have a lot of legacy cruft hanging around, though, which can help us a lot. The ‘cruft’ parts of protocols are those that are no longer necessary in the Ethernet and IP worlds, but are left in the protocol to simplify the lives of implementers. Usually, the IP variants of protocols are just wrapped-up versions of the serial protocols. Serial protocols usually included CRCs, since the serial physical layer provides for no error detection or correction.
One of the neat tricks that was invented long ago was to partially precompute CRCs using a table. The algorithm, known as the Rocksoft Model CRC Algorithm, is documented in a nice paper, which includes the source code needed to generate the precomputation table given any CRCs exponents.
We generally do not allow product presentations at S4, but occasionally there is a technology that is novel or potentially important that we make an exception. For example, we had Kaspersky present on their ICS operating system at S4x15.
A second exception was made for Carlos Aguayo Gonzalez of PFP Cybersecurity to present the idea of using Power Fingerprinting to identify changes in PLC or RTU logic or firmware.
I won’t attempt to summarize the technical details; watch the video. It includes a demo of the technology.
However it is interesting that the Power Fingerprinting sensor is in fact not connected to the device it is monitoring. Hello air gap. It also is a potential tool for addressing the supply chain problem.
WhiteScope – an online ICS/SCADA whitelist that is trying to solve the last mile supply chain problem until vendors start signing their code. The WhiteScope data repository is available to all, free of charge.
Medical Device Security – an area that Billy is a pioneer on. We discuss progress, FDA involvement and how similar or different it is as compared to the classic SCADA/DCS/Process Control.
In part 1 we looked at what CAN is and what the difference between CAN and OBDII traffic is on a vehicle network. In this part we’re going to look at simple reverse engineering techniques to determine which CAN IDs are of interest to us. For this exercise, we’d like to see if we can confuse the tachometer display.
First, we need to identify which ID Toyota is using for the Engine RPM on the CAN. Manufacturer specific IDs are not published so we’re going to have to figure it out ourselves. For this work, I hooked up my beaglebone canbus tool (you can build your own by following instructions here: https://github.com/digitalbond/canbus-beaglebone). Then I started my truck and let it run for about 15 seconds before powering it off. Dumping all CAN messages shows us a lot of traffic and it can be difficult to make sense of what’s going on. We’re going to use the unique_ids tool (part of the canbus-utils repo available at https://github.com/digitalbond/canbus-utils) l to get a list of all of the IDs that were seen while the tool was running. My list was:
We need to do some filtering and looking at the data to identify which data bytes are of importance. We’re going to be using the watch_id tool that I wrote. Cheating, we’ll start by looking at ID 0x2C4. The bytes that change are color coded red for ease of display. Some fields may be a single byte, multiple bytes, or have multiple values in a bitmask.
$ node watch_id.js –id 0x2c4
The results from the tool were:
Looking at the results we can see that the first two bytes remained at zero until a point where they increase and remain steady at around 0x08CC (2252 decimal). The 5th byte goes from zero to 08 and then never changes. The last byte changes frequently. Later, the 4th byte changes from 0x10 to 0x0F.
We can figure that the first two bytes are a 16 bit integer, the 4th byte contains some kind of bitmask, the 5th likely contains a bitmask or flag, and the 4th is a single byte integer. Converting the first two bytes to a decimal integer we see that it appears to be a direct value of the current engine RPM which starts at 0, ramps up, and then settles lower as the engine comes to idle. We correlate this with the readout from the tachometer and can be pretty confident that we’ve found the RPM.
In this case, the RPM was a direct value. For other values (or even other manufacturers) there may be a static value or formula applied to get the proper result (e.g. RPM=bytes 1&2 + 1k). Now, we know the ID and packet formula for the RPM on the CAN. We can write a little tool that spams the bus with our “attack” messages in order to confuse the tachometer into displaying an incorrect value.
This is a classic art deco venue where The Dick Clark Show, The Ed Sullivan Show and the Miss USA and Miss Universe Pageants were often filmed in the auditorium. In 1964, Jackie Gleason even moved his hugely popular tv show there, hence the name.
In 2007, the Jackie Gleason Theater underwent a major renovation and now has state of the art lighting, sound, video and just about any staging option we can think of — all without losing that cool art deco feel that is South Beach. In fact, the planning team is feeling the challenge of coming up with unique content, formats and staging worthy of this non-traditional ICSsec conference facility.
You often hear this security professional or that automation engineer is a “rock star”. At S4x16 we will see who the real rock stars are on a real stage that today is a major concert venue. Speakers will literally be in the spotlight on the big stage.
In addition to the main theater, the venue has so many interesting rooms for us to use. For example, the ICS Village will be held in the Red Star Lounge, a VIP lounge for concerts with coffee and cocktails and comfortable couches and tables to better get to know your fellow S4 attendees. Much like the staging, taking full advantage of all the rooms is a fun, creative challenge.
The Kimpton Surfcomber will again be an official S4x16 hotel, and it is only a 3 block walk to the Fillmore Miami Beach at Jackie Gleason Theater. Attendees raved about this hotel last year … except for the bus ride to Kovens. No buses for S4x16!
Note: We want to publicly thank the Kovens Conference Center that has been the home of S4 since the inaugural event back in 2007. They did a fantastic job for us each and every year. Unfortunately we outgrew the ballroom there and wanted to move somewhere that buses were not required. We highly recommend Kovens if you need a South Florida venue for an event.
I thought I’d take a step back after releasing tools and presenting on CAN to do a quick intro into what communications are going on inside a vehicle anyway. What is CANBus? What is OBDII? Is there a difference?
We’re going to skip all the electrical fun parts, the packet formats, and the theory and move right into using some tools to look at CAN traffic. This means my language is simplified for example purposes. For this exercise I’m going to be analyzing traffic from my 2013 Toyota Tundra.
This is CANBus (image courtesy Wikipedia):
As you can see, it is a bus. We have a bunch of systems on a 2 wire bus. Each system has a particular job. In the case of a vehicle that might be the Engine controller, instrument display, or the tire pressure sensor. CAN is also used in industrial control as well.
The first number is the ID or address of the device sending the message (e.g. 0x630, 0x638). This is not unlike an ethernet MAC address or an IP address. The following numbers are the data bytes (up to 8 per message).
ANY device on the CAN is able to view and send a message to ANY ID. Therein lies the entire problem with vehicle security. There is zero authentication, verification, non-repudiation, or any other accountability whatsoever. Anything on the bus can pretend to be anything else per the spec. Further, you only have 8 bytes of data so adding in something like encryption or cryptoverification is particularly challenging. Basically in order to not suck we have to ditch the whole thing (sound familiar ICS people?).
Ok so that’s CAN in a nutshell. What is OBDII? OBDII is a standard SUBPROTOCOL to CANBus. It’s a set of ID’s that all the vehicle manufacturers must support.
Every manufacturer builds their cars differently. In a vehicle, for example, the RPM may show up on the CANBus on ID 0x2C4 as a 16-bit integer on bytes 1 and 2. Another manufacturer may decide to put the RPM on CAN at ID 0x310 bytes 7-8. OBDII is an attempt to create a standard method of finding out that information. Rather than standardize on IDs, however, a query/response system was created.
The method for obtaining the engine RPM via OBDII is:
Diagnostic tool sends a message on ID 0x7DF, the standard OBDII query ID. The message contains the number of additional data bytes, the “Mode”, and the “PID” of the requested information.
For querying the RPM the Mode is 1 and the PID is 0x0C. The message would look like:
0x7DF  02 01 0C 00 00 00 00 00
The modules in a vehicle respond to these diagnostics messages. The ID for the response depends on the module but the IDs start at 0x8 higher than the standard query ID. Thus, the ECU responding to an RPM request will respond on ID 0x7E8. The message contains the number of data bytes, the mode + 0x40, the PID, and the value.
For responding to RPM the Mode is 0x41, the PID is 0x0C. That message may look like:
0x7E8  03 41 0C 09 C4 00 00 00
In summary, OBDII is simply a sub-protocol that runs on top of CAN. It uses standardized CAN IDs and message formats to request vehicle information. The problem with plugging in third party dongles to the OBDII port is that that port still gives you direct CAN access. The OBDII port can be used to write arbitrary CAN IDs to the bus. In practice, this is used to do things like provide vehicle software updates (e.g. flashing new firmware to the ECU).
An attacker is not interested in the OBDII spec or most of the information it provides. An attacker is not interested in querying a vehicle for diagnostic information. Instead, an attacker wants to see what kind of control over vehicle operations is possible by writing to the CAN.
In part two of this series, we’ll do some CANBus reverse engineering to figure out what CAN messages we have an interest in if we want to cause the vehicle to behave erratically.
Belden buys Tofino, GE buys Wurldtech, Lockheed Martin buys Industrial Defender and now iSight Partners acquires Critical Intelligence. The trend continues of larger organizations buying ICS security expertise.
Bob Huber and Sean McBride left Idaho National Labs (INL), after being involved in setting up what became ICS-CERT, to form Critical Intelligence. Critical Intelligence in many ways competed, or augmented if you want to play nice, the information ICS-CERT provided. However, the depth and breadth of the Critical Intelligence product far exceeded what ICS-CERT provided. Whether this was due to the talent disparity, fewer restrictions on what could be written, or both is not known.
I spoke with Bob and Steve Ward of iSight yesterday to understand the motivation for partnering and what future ICS services, products and events will look like. It is too early to answer the later question, but the motivation was clear.
iSight is looking to improve their threat intelligence in the ICS area, basic and easily understood reason. From a Critical Intelligence standpoint it’s more interesting. iSight has 200+ analysts that speak 16 different languages.
A lot of the important ICS threat info is written in Chinese, Russian and Arabic, not to mention the videos and podcasts that require the ability to understand the spoken language.
A fair amount of the technical analysis of malware and other attack code is not ICS specific. Look at the work that Kyle Wilhoit is doing over at Trend Micro on Havex and Black Energy for an example.
iSight has a methodology that will add rigor to the analysis and reporting process.
What iSight was missing was an understanding of what matters in an ICS, who are the players, important protocols and products, and the ability to task all those resources in a smart way that would lead to useful product. If the two companies can integrate the capabilities well the result should be more than the sum of the parts.
The biggest question then will be are the asset owners able to take in and act on this better threat intel?
Admittedly I’m a big fan of Critical Intelligence’s work. They helped with content on our site for a couple of years, were guests on the podcast, speakers at S4 and one of the people I talked to when I was trying to figure out who was doing what to whom.
Another thing this latest acquisition has in common with the other ICSsec acquisition is the price and terms were not disclosed.
Congratulations to Bob and Sean and the rest of the team at Critical Intelligence.
At S4x14 Adam Crain of Automatak, along with Chris Sistrunk, presented the results of their Project Robus that fuzzed DNP3 stacks and found most had problems with processing malformed or illegal responses. This year at S4x15 Adam talked about Avoiding Insecurity in ICS Protocols.
Adam compares Schweitzer’s Streaming Encryption Protocol (SEP) with DNP3 Secure Authentication Version 5 (SAv5).
Two of the main criteria he discusses and demonstrates with those two protocols are 1. have a clear trust boundary and 2. keep it simple. It is clear why there were so many bugs that led to vulnerabilities in the DNP3 protocol stacks.
This is a must watch for any group adding security to an ICS protocol or those that need to start this important and necessary ICS protocol feature.