We have another control system incident in the news that will surely fill up slidedecks for the next decade.

News became public yesterday of an arrest of security guard involved in a compromise of the HVAC system, and likely the rest of the hospital network,  at the Carrel Clinic in Dallas, Texas.  Thankfully nothing was done to disturb the operation of the HVAC system before the arrest could be made, but it seems plans were in place for it on or around July 4th.  Anyone who has been to Texas in July knows that this could have easily been life threatening for those already at the hospital.

From the information availible it looks like this is going to be a case of no one watching the watchers combined with poor separation of corporate and control systems, assuming that the computer compromised in the video wasn’t controlling the HVAC.  From my experience hospitals usually have a pretty good handle on systems that directly affect patients, we’re not seeing machines controlling a morphine drip connected to open wireless (though some of the information systems with patient data make me worry). Perhaps this will be a good example for other healthcare groups on how to better protect not only their data and systems that directly affect patients, but also their systems that have an indirect affect on protecting and maintaining life.

It looks like we all were a little lucky in this case due to the perpetrator doing a bit too much bragging and leaving quite a bit of information on the web for some one to dig up.  It seems that the lionshare of the credit for that digging goes to Wesley McGrew, a research assistant in the Critical Infrastructure Protection Center at Mississippi State, who tracked down the information neccessary for the arrest to be made.  Details,including the criminal complaint, can be found on his blog and it looks like there is loads more information about the investigation and the trail of bits he that the attacker left behind.

The Portaledge team is pleased to announce the beta release of the Enumeration Event Class. Portaledge is Digital Bond’s Dept. of Energy program that leverages OSISoft’s PI ACE engine to provide Security Event Management, detecting, alerting and logging of security events on SCADA and DCS.

The Enumeration Event class currently has modules for detecting common port scans and other enumeration scans; TCP, FIN, SYN, UDP, ICMP and for detecting account enumeration via Finger. These scans are often the first part of an attack scenario, and therefore by detecting and alerting on these attacks an asset owner is better able to stop the attack before it is successful or at least limit the impact of the attack.

Most of these events will identify scanning two ways. The first is a single system being scanned on multiple ports, the default is 3, and these alerts are labelled by Portaledge as a “scan” such as TCP scan, FIN scan, SYN scan, … The second way is multiple systems being scanned on the same port. For example, a scanner looking for web servers or ICCP servers. If three systems, the default but this can be changed, being monitored are scanned on the same port a “port sweep” alert is issued.

In addition to detecting common enumeration scans, the Enumeration Event Class provides a Traffic Monitor that monitors for anomalous communications. The Traffic Monitor allows the user to profile normal communications on the system and add the “allowed” communications on a per system basis. Portaledge will then monitor and alert on communications outside of those allowed. As communications patterns on control systems under normal conditions are very repetitive, the detection of abnormal communications is a strong indicator of enumeration and other malicious activities.

The Enumeration Release include the Enumeration Event Class event (see last weeks discussion of Event Class Events and Event Chains) that correlates events on four types of commonalities, and links events into chains to provide a better understanding of what is occurring on the system.

The Enumeration and previously released Availability events are available for download to Digital Bond content subscribers. Any feedback on Portaledge is appreciated.

Other Links:

Ok brilliant researchers and thought leaders in the control system security community – - send in a brief abstract for S4 2010. This will be the 4th edition of Digital Bond’s SCADA Security Scientific Symposium. It draws 50 top technical talent from around the world and a virtual audience watching the simulcast.

Read the Call for Papers

There are many professional benefits in presenting at S4, but it also has the added benefit of being in Miami Beach in January [20 - 21].

We have always tried to have a speaker package that would allow even a humble grad student with a great paper to attend. This continues in 2010. Speakers will get to attend S4 for free, have their hotel room covered for up to three nights and get a $400 stipend to help with expenses.

Remember this is a technical event in front of a friendly, but very capable audience. You will need to bring your A game in the paper and presentation.

If you want to discuss a possible S4 paper or know someone we should chase for the event please contact S4@digitalbond.com.

We routinely use file content checking to retrieve and evaluate configuration settings for the Bandolier security audit files. This is a function of the Compliance Checks plugins for Windows and Unix. It works well as long as the file name is known. What if you want to search for specific content but do not know the file name? There is another type of file content searching in Nessus designed to do just that for Windows servers and workstations. The Windows File Contents Compliance Checks plugin is found alongside the other compliance checks in the Policy Compliance family.

In the traditional IT world, this type of file content search is often used to locate things like credit card numbers, social security numbers, or other sensitive content hanging around in places where it should not be. Tenable provides an extensive library of audit files for searching various types of sensitive content. For control systems, I see a couple of potential uses for this type of auditing.

1.) Identifying unencrypted sensitive data in installation and temporary files. Working on Bandolier, we’ve seen that some installations leave behind temp files that include things like configuration details and even user and password information. You can use your imagination for other ways sensitive data might get left behind–from an operator keeping information in a text file to an undocumented application function that leaves sensitive data behind. A file content audit file can help identify these scenarios. It is only as difficult as deciding on the search terms and the file types you want to inspect.

2.) Identifying unencrypted sensitive control system data on non-control networks. Whether you’re concerned with PCII, NERC CIP-003 R4 or just good security, keeping sensitive control system data from floating around on the business network is a valid concern and one that is not easily controlled. Speaking from experience, you may be surprised how much infromation you can find hanging around on local drives and even semi-public shared drives. Again, you can use your imagination for search terms and locations but a place to start might be searching for files that contain IP addresses from your control system subnets.

Let’s say you have a control network subnet that is 172.16.50.xxx. Here’s a simple example of an audit file that will search for it:

—

<check_type : “WindowsFiles”>

<item>
type: FILE_CONTENT_CHECK
description: “The file contains sensitive SCADA IP addressing information”
file_extension: “txt” |”doc” |”xls” |”pdf”
regex: “172.16.50.*”
expect: “172.16.50.*”
max_size : “50K”
</item>

</check_type>

—

Before you run off to search your entire network, there are some performance issues to consider. As you can imagine, searching file content can be resource intensive. Start by testing in a non-production environment so you can evaluate the impact of different types of searches. You can hone in on particular file types and limit file size to save resources.

For Unix systems, there is not a separate plugin for file content searching but there is an option in the regular audit file language called “CMD_EXEC”. As the name implies, you can essentially run any Unix command including grep. This comes with the same performance impact warning as above – be careful how you use it and please remember to test.

Have thoughts or ideas on other ways this can be used? Would love to see your comments.

Portaledge has an event hierarchy. The hierarchy (from smallest to largest) consists of: Event Triggers, which cause Events, which are correlated in a class into Event Class Events. Events and Event Class Events can be correlated across classes into Meta Events.

Today I am going to discuss Event Class Events. Triggers and Events were covered last week in Part 5 of this series. Meta Events will be described in an upcoming post.

To date for Portaledge we have released one set of Event Class modules, the Availability Class. And there is another set, the Enumeration Class forthcoming. The other classes to be developed are: Communication, Escalation, Exploitation, Obfuscation, Process Manipulation, and possibly Reconnaissance. More information on the Portaledge classes and hierarchy can be seen on the SCADApedia.

Each Event occurs in one and only one Event Class. For example in the Enumeration Event Class there are the; UDP Port Scan, SYN Port Scan, ICMP Scan, TCP Port Scan, and FIN Port Scan events as well as the associated Port Sweep events. The Enumeration Class also contains a Finger Detection Event and an Event that monitors for out of range communications, namely the Traffic Monitor Event. These Enumeration Events, scheduled for release shortly, only represent a subsection of the total events defined in the Enumeration Event Class.

As Events occur they are logged and written to the historical database. When the Event Class Event module periodically fires it examines the Events associated with it’s Event Class across a time slice and correlates the Events on commonalities. The Events sharing commonalities are “chained” into Event Chains within the Event Class.

For clarification of this process, consider the following examples:

An attacker runs a Nmap UDP scan on a subset of systems on a network segment. As each system in the subset is scanned an Enumeration Class Event reporting that the system has been scanned, the source IP and port of the scan, and the number of ports touched on the system being scanned is created, and written to the historical database. When the Enumeration Event Class module fires it gathers all of these data points across a time slice and correlates them on the common source IP and protocol of the scan. The common events are sorted on time and chained into an “Enumeration Event Class Chain.” The chain is composed of the same number of parts as the number of events detected and each part contains data relative to the event that caused it.

In the second example consider the case where an attacker performs a TCP port scan of a single system. This port scan creates an TCP Port Scan Enumeration Event indicating that a number of ports were scanned from the attacker’s system and port. The attacker, noticing that the finger port is enable next tries to enumerate user accounts using finger. This creates a number of Finger Enumeration Events. The Event Class Event module correlates these events based on the common source IP of the attackers system and the common destination IP of the system scanned and fingered. An Enumeration Event Class Chain is created containing as its first part the TCP port scan and then the subsequent finger events.

We are pleased to announce the beta release of some Quickdraw software components today. Quickdraw is a Digital Bond research project funded by the US Department of Homeland Security (DHS). This beta release is the first three SCADA IDS preprocessors that were the crux of the Quickdraw project. They are:

• DNP3
• Ethernet Industrial Protocol (EtherNet/IP and the encapsulated CIP protocol)
• Modbus TCP

We’ll follow up with some more detailed blog posts about functionality in the next few days, but for now here are some of the basics. This package adds three preprocessors to the Snort IDS/IPS application, these do the heavy lifting and parse out the protocol into nice structures for later use. We’ve also included several detection plugins that expand the Snort language to allow matched based on the data that the preprocessors have given us. From there you can send off an alert using the standard Snort mechanisms or the syslog support.

Specifically, the plugins in this release include matching on the Modbus/TCP function code or unit code, the DNP3 checksum and internal indications, EtherNet/IP CIP service and several others. And if your comfortable with the Snort source code you can easily add more of these plugins yourself, but if your not then you’ll have to wait on our next release thats coming soon. We plan on adding many more plugins so writing Snort IDS rules is simple and have many examples of where this would be useful not only for detecting attacks, but also for troubleshooting.

We appreciate any feedback you have and will continue working on this project to make these modules as useful as possible. Look for updates coming regularly, and more specific details on using and extending
Quickdraw here and on the Scadapedia.

Key Links:

• Main SCADApedia documentation page on SCADA IDS Preprocessors
• SCADA IDS Preprocessors download page

Dale’s Note - Like many research projects, we have learned a lot in the program. My guess is that two or three years from now these SCADA preprocessors will be viewed as the major contribution from this research program. Not only are they needed to detect and write security events for legacy field devices – - Quickdraw, but they are also hugely useful in enabling and making more effective many more SCADA IDS/IPS rules, adding deep inspection to field firewalls and probably three or four uses we have not thought about yet. Once you have easy access to the decoded SCADA protocol fields there is a lot that becomes much easier.

Congratulations to Daniel and Victor Julien from the Netherlands for some really great work.

As Daniel said, we will follow up with some very practical posts and examples on how these SCADA IDS Preprocessors can be used.

• The Microsoft Manufacturing User Group [MSMUG] will hold its annual summit in conjunction with ISA’s annual event in Houston on October 8th rather than out in Redmond. Whether this is due to limited travel budgets or to take advantage of the built in crowd at ISA it should boost attendance at MSMUG.
• UPDATE: This just in . . . the DHS ICSJWG [Industrial Control System Joint Working Group] will hold their First Annual Conference in conjunction with the ISA Expo, Oct 5 – 7 in Houston. We seem to be nearing a harmonic convergence. There is a short fuse for the Call for Papers that ends on July 6th. Now here are some of my questions: what is the ICSJWG trying to accomplish with this conference? What has ICSJWG done that past 3 months and what specific efforts are underway? What presentations at the Conference would help ICSJWG meet its mission? I attended the planning/kick off in Denver and left with many questions, and I have heard nothing since. I’m sure this will draw a good crowd, but is holding an annual event that is similar to the old PCSF annual meeting the goal? That would be great, especially if it was community driven like the most successful PCSF meetings. But it was my impression that this was specifically not the goal, although admittedly the goal beyond industry <> government information sharing was unclear.
• Wurldtech announced the HIMA HIMax safety system has achieved Achilles Certification. This is the 13th control system to get Achilles certified [FD: Wurldtech is a past Digital Bond client]

Lately I’ve been working with the SEL-2032, learning the device capabilities provided through the SEL protocol, Modbus, DNP3, UCA2, GOOSE, etc. GOOSE (which stands for Generic Object Oriented Substation Events) is particularly interesting. GOOSE (a component of the 61850 protocol) is a mechanism for fast transmission via ethernet of substation commands and sensor data used for tripping switchgear, starting a disturbance recorder, and in the implementation of, among other things, safety interlock systems. GOOSE stipulates VLAN technology and priority tagging as per IEEE 802.1Q to have separate virtual network within the same physical network and sets appropriate message priority level. (Which means it MAY be vulnerable to a denial of service attack originating from and even targeting a different network that happens to share the frame-relay level resources with the GOOSE VLAN, and MAY be vulnerable to VLAN Hopping attacks, perhaps using a fast traffic generator e.g. Mausezahn ). Cisco has identified at least 8 different possible VLAN attacks, but to be fair Cisco also asserts:

“The security of VLAN technology has proven to be far more reliable than its detractors had hoped for and only user misconfiguration or improper use of features have been pointed out as ways to undermine its robustness. The most serious mistake that a user can make is to underestimate the importance of the Data Link layer, and of VLANs in particular, in the sophisticated architecture of switched networks. It should not be forgotten that the OSI stack is only as robust as its weakest link, and that therefore an equal amount of attention should be paid to any of its layers so as to make sure that its entire structure is sound.”

The GOOSE level remote switching and overall grid control is mission critical functionality requiring an extremely high-level of assurance. Currently storing very large quantities of electrical power is not very feasible (except storing potential electricity in the form of water behind a damn). So the power grids basically operate within very defined boundaries. Supply must be matched to demand; over voltage, under voltage and a host of other power quality conditions can result in economic inefficiency, equipment damage, and temporary or prolonged power outages, and conceivably significant safety hazards. The root causes of these problems may have originated up or down stream of where the impact is most severely felt. For this reason power grid control is extremely dependent on upstream and downstream data & commands.

As I learned about GOOSE I learned that it is part of the IEC-61850 family of protocols, that 61850 is being adopted rapidly especially in Europe, and is proposed for that panacea for all our power reliability and efficiency problems, the ‘Smart Grid’. So I decided to review some of the recent Smart Grid technology articles. The Economist review of the “Smart Grid” this week is a real nice popular introduction to the concept.
The recent article by CNN ['Smart Grid' may be vulnerable to Hackers] pointed out that the advance metering infrastructure of the smart grid presents some security issues as has been highlighted by research done by IOActive. IOActive asserts that a determined attacker with $500 of equipment and materials and a background in electronics and software could “take command and control of the [advanced metering infrastructure]. The CNN article contrasted that while a representative of Edison Electric Institute has asserted that “We are not going to manufacture this car without a seat belt.” – the article also noted that “as of now there are no clear-cut Smart Grid cybersecurity standards”. According to the Economist article, the United States is a bit behind in terms of the advanced metering technology adoption rate, behind especially Italy. The Economist sites ABI Research which say’s that 76 Million smart meters have already been installed world-wide and also states that about 12 million smart meters will be installed in California over the next few years.

The advanced metering used to determine if I am drawing from the grid today or selling excess power from my roof top solar panels or the 25 kWhr Lithium Ion battery pack of the Wrightspeed X1 in my garage (just kidding) back to the grid may not seem as mission critical as the relays controlling breakers for a major metropolitan area. But the thing is, to be effective the data from my house must be fused with the data from other residences as well as with data from substations, generation plants, etc.

And so there is the current GOOSE information domain which goes from secure node to secure node and is clearly mission critical functionality requiring an extremely high-level of assurance. On the other hand, the Smart Grid’s private residence based advanced metering constitutes a different information domain. Assurance is important for the advanced metering but to be effective advanced metering must be bidirectional and it is doubtful that an information domain extending to private residences can attain the level of assurance absolutely critical to the current GOOSE information domain. This is the makings of a Multi-Level security (MLS) problem, a kind of problem here to fore largely germane only to military and intelligence organizations.

A system is said to be Multi-Level Secure when it meets the following criteria:

1. The system must process information for more than one information domain; and

2. The system must be implemented with assurance that the security policies within each domain are enforced.

(citation from AEPOS Technology)

MLS sounds like a simple problem to the layman. In fact MLS Research began in the 60’s and has eaten through literally millions of research dollars at this point. So far we’ve learned, among other things, that the coverage you get per dollar expended on MLS technology tends to be lower than one might expect, that the ‘covert channel problem’ is pernicious and really difficult, that even the most successful technologies sometimes have significant performance (network transfer speed) impact which would be a non-starter in the 4 millisecond time frame of the GOOSE world, and that application-level-only solutions are insufficient, rather hardware and operating system level guards are essential. For a good review of MLS research see the “Introduction to Multilevel Security” by Dr. Rick Smith of the University of St. Thomas, Minnesota, and the Wikipedia MLS article.

All of which isn’t to say that a secure “Smart Grid” with an advanced metering infrastructure that also supports data fusion with the mission critical grid functionality is impossible. I’m just saying it might be a good idea to send our best cyber security minds to the ISA99 10/7 – Morning Panel: Cyber Security Considerations for Smart Grid Technologies

As our second release of Portaledge Event Modules is forthcoming, I am continuing with a series of posts on Portaledge fundamentals. My goal is to provide an overview of how Portaledge functions, and it role as a Security Event Manager for control systems.

Portaledge relies on a variety of data sources to monitor a system and detect security events. For each security event detected and recorded by Portaledge there is a corresponding event trigger. Each event module in a Portaldge event class monitors the historical data for one or more sets of criteria, watching for potential event triggers and correspondingly, creating and logging the associated event.

A Porteledge event trigger is a single data point or set of aggregated data points sent from the various PI interfaces, such as the PI IP Flow, Ping, and Syslog IT Monitor Interfaces, the OPC and other SCADA Protocol Interfaces, SCADA vendor specific interfaces or other security event data sources. Portaledge polls data from the historical data base and evaluates the data points or aggregated data set against specific criteria. If the criteria meets a trigger definition then an event is created and, the data point/set that caused the event is the event trigger.

Portaledge aggregates data from various Pi points to monitor for triggers. For example; the majority of the enumeration events poll data from over 8 data sources to create a data set representative of a network communication session. The data polled and aggregated is: source system name, source system IP address, source system port, destination system name, destination system IP address, destination system port, session length, and session protocol e.g. TCP, UDP, ICMP etc.

As Portaledge creates and stores events, an “event” is then the base level of correlated data. An event is targeted at a single system or device, i.e. a single IP address. Correlated and chained events within an event class or across event classes will be discussed in future postings.

To better understand the trigger to event process consider the following example: The Enumeration Traffic Monitor module monitors for communications between systems that are “out of bounds” meaning they are not on a list of “approved” communications. For each system in a network the administrator can create a system entry, and with that entry keep a list of systems that are allowed to talk to the system, their associated source and destination ports, and what protocol.

The Enumeration Traffic Monitor, when it executes, creates and parses a list of all the session that occurred over a given time slice. It creates the list by polling the 8 data points mentioned above from the historical data base. It then sorts and parses the list, comparing the sessions against the list of allowed communications. When a sessions is detected that is not allowed an event alert is created and logged.

The trigger is the set of of data creating the out of bounds communication fed to the historian by the IP Flow Interface. The event is a “Enumeration Traffic Monitor Event” noted with an alert that shows the source and destination IPs, the ports, the sessions size and protocol and the time stamp of when the session info was written to the historian.

We have been blogging about the benefits of virtualization in control systems, see the blog posts here. Asset owners have been reluctant to embrace virtualization until it was blessed by the vendor, and this is understandable. A few vendors have been working on virtualization support, and the highlight for me at the AREVA User Group meeting was to hear that they working with virtualization.

The first step is to ’support’ virtualized environments. This means when a customer calls vendor support with a problem and the discussion reveals he is running it on ESX or some other platform, the support person does not reject the call. AREVA is now supporting the most popular virtualized environments and looking to expand the list of supported virtualization solutions. They are treating it like any other OS. For example, we support our application on Windows XP, Red Hat Linux Version X.X and VMware ESX.

The second step is to actually deploy new SCADA systems on virtualized systems. AREVA has its first customer in the process of deploying a new e-terra on a virtualized solution. Virtualization is now another customer option.

When the community gets more comfortable with virtualization performance and understands the benefits in fast recovery, patch testing / rollback, redundancy, legacy OS support, … the virtualization will be common. In fact, my prediction is in 3 years more than half of all new deployments will include the substantial use of virtualization.