Digital Bond

For Secure & Robust ICS

How Deep Is Your ICS Deep Packet Inspection (DPI)

by 5 Comments

Check out the S4x17 Agenda At A Glance and Register Now

The industrial firewall and ICS anomaly detection markets are getting very crowded. The industrial firewall market is older, but it is still expanding both in specialized ICS firewalls and enterprise firewalls adding ICS protocol support. The ICS anomaly detection market has exploded with a new entrant almost every month and millions of dollars of funding.

The benefits of these product categories are heavily based on their ability to perform deep packet inspection (DPI) of ICS protocols. Firewalls do this for more granular control of a security perimeter (and some IDS/IPS), and anomaly detection rely on DPI to identify unusual or potentially damaging use of ICS protocols.

These products are typically promoted by the breadth and depth of the ICS protocol support. The breadth is easy to compare and somewhat useless. A vendor can easily list the protocols they support at some unspecified level of depth. I say breadth is somewhat useless because an ICS asset owner doesn’t care if the vendor supports 10 or 30 protocols; the ICS asset owner only cares if the product supports the protocols they use.

Depth in DPI of the protocols an asset owner uses should be one of the key decision factors, along with company viability, ease of use, reporting, support, interoperability with SIEM’s, … Depth can vary figuratively from inches to a mile deep, and depth can vary a lot per protocol in the same product. We worked with one client considering an enterprise firewall with tremendous breadth of ICS protocol support. The firewall vendor was only checking the TCP port number and a single byte in the request packet, inches deep, in the protocol our client was most concerned with. We know that the same vendor has very deep DPI for other ICS protocols including proprietary extensions of the protocol to cover engineering work station actions.

Talk to the anomaly detection vendors and they will typically tell you not only how completely they inspect the ICS protocol, but also how they do this to a much greater degree than their competitors. When asked for more details and reasoning it devolves into emphatic assertion, and they cannot all be right. It is likely that simple protocols have similar levels of depth, but more complex protocols will vary as will support for proprietary extensions.

At S4x17 we are trying to help asset owners and the ICS community compare and contrast ICS DPI with two sessions on Stage 2 titled How Deep Is Your ICS DPI? The speakers have been challenged with developing a structured method to evaluate the depth and value of the DPI of an ICS protocol. Ideally this would come down to a method of comparatively score the solutions. Given the number of vendors and asset owners looking at this issue we are hopeful we can at least narrow down the approach to comparisons.

Check out the S4x17 Agenda At A Glance and Register Now

ICS vendors still falling short on security response

by 1 Comment

While at S4, Digital Bond Labs had a security advisory published by ICS-CERT (see ICSA-15-013-03).  One thing that we tried to do differently with releasing information on the issue this time around was to reach out to vendors that were obviously using the affected software as part of their control system.

The results were pretty strange.  Most companies contacted had no security@ email address, and no /security URI on their website.

Of the 32 affected vendors that we reached out to, only 3 had a ‘security@‘ email address that did not bounce (the remaining 29 we contacted using a general ‘support’ email address listed on the vendor website).  Two months later, zero of the affected vendors have provided a response beyond an automated ticket.

When I worked for a vendor, I had an internal pep-talk informally called ‘dealing with researchers.’  There are a lot of cheap and easy lessons to learn, based on how the bug comes to you:

[Read more…]

Responding to Stinky Vulnerabilities

by Leave a Comment

So, I hear there's a problem with your Control System application...Responding to cyber vulnerabilities as a vendor is a lot like responding to diaper issues. No matter what, you are going to handle a lot of crap from both ends. As a vendor, all you want to do is clean it up, and move on with operation. But just like diapers, doing it wrong invites a smelly mess that everyone you invite over will comment on. And blaming the producer just isn’t realistic, because it happens whether you like it or not. Instead, change how you react, rather than trying to convince the producer to quit.

At the S4 Conference this year, I presented on using the Microsoft Attack Surface Analyzer to critique protective relay programs. While not a ‘bloodbath’, the research brought to light some fundamental cyber issues with the development of these programs, and I picked two programs as showpieces in the presentation.

The two programs, Schweitzer Engineering Labs’ (SEL) AcSELerator and Schneider’s MiCOM S1 Studio, both demonstrated file permissions issues that made the installed system less secure. Additionally, Schneider had a service vulnerable to privilege escalation, due to the file permissions. The issues were swiftly found through the use of the Attack Surface Analyzer, and recommendations to fix were included. I reported these issues to the vendors, and worked through the process of giving them details.

Comparing and contrasting the responses allows me to comment on vendor response in general.

First off, the responses highlight a very real problem in vulnerability response; authority for cyber security response is rarely concentrated in one part of the organization, it is distributed out to the various business units. This usually reflects a lack of a top down approach to security, where individual units and groups can decide on a response based on their own judgement of risk. How would investors respond if the risk was from accounting practices, or safety, or diversity in the workplace instead of software development? That’s why authority to respond to those issues is held at the top level.

Second, vendors must recognize that their capability to respond scales with the size of their organization, and allocate resources to combat the scale. The responses provide a case study. SEL, a smaller company with a core product set and a generally flat organization, responded with timelines, actions, and full fix promises, generally considered the gold standard. On the other hand, Schneider is a massive holdings company with diverse business units, and it’s response was more legal and liability related. To me, this reflected an inability by the Schneider cyber security response folks to muster resources in time to meet the gold standard, which would not be an issue if they had the authority to muster those resources. The advantage is to the smaller company, unless action is taken to bring authority for response development up to a higher level, and push accountability down.

Third, don’t let the vulnerability researcher drive your process, and don’t try to manage him/her. I’m not saying minimize the researcher, because we have voices, but response is YOUR process. The researcher is not the problem, the vulnerability is the problem, so address the vulnerability. If your focus is to completely address the vulnerability, and not how the researcher responds to your response, you’ll do better than if you try to minimize your effort by divining how the researcher might react. Neither Schneider nor SEL attempted to manage me during this process, which I’m quite happy about (I really didn’t have much time to get managed that week).

Lastly, notification of users is a necessary step. Vulnerabilities in products are a vendor’s responsibility to recognize, triage, and fix, but the customer has a responsibility as well. Notification, with a discussion of the risk and possible mitigation measures, allows customers to identify vulnerable systems, and ensure they are protected. The need for notification to be prompt must be weighed against the need for accuracy as well. We’ve seen the reports where a fix turns out to not be a fix, which just increases effort over the entire process. Fundamentally, a vulnerability is a newly recognized defect. How do you normally let customers know of defects, and are you using that process for cyber security notification?  SEL showed exactly how they were going to alert their users, but no such luck on the Schneider response.

Vulnerabilities happen, just don’t let them stink up the house. As always, questions, comments, and criticisms welcome below!

title image by Spigoo

Update: Removed the Vendor Responses, which were not professional for me to post. MT

S4: Wightman’s Tofino Raves & Limitations

by 1 Comment

When Reid Wightman was still at Digital Bond in 2012 we discussed how to follow up Project Basecamp. The idea was to give field firewalls a hard shake. Fortunately he was able to continue the work and present at S4 after moving to IOActive. I have a lot to say about field / industrial / SCADA firewalls, but here are the takeaways from Reid’s S4 presentation.

  • Reid gave the Tofino Security Appliance high marks for providing the promised protection and for resisting a variety of attacks that would cause a poorly designed device to fail. He had some great pro-Tofino quotes such as: “I really like it … I would buy one of these.”
  • The ICS protocols lack of integrity can still be exploited. It’s not a fault of the Tofino firewall, but if you let insecure protocols cross security zones bad things can happen. Key takeaway is endpoint security is still needed; Tofino does not change that.
  • Extending tunneling attacks, such as DNS tunneling, to ICS protocol tunneling is an important attack concept to defeat firewall protection. Reid has released Modshaft to demonstrate this. Also Reid will soon release an ettercap tool for recording and replaying Modbus TCP, what he calls a Modbus VCR.
  • Huge respect to Eric Byres, Tofino Security and Belden for providing devices to an ICS security researcher who they know is looking to find vulns in their product. It shows confidence in the product and a desire to find and fix problems.
  • Reid, and other researchers, will give credit where credit is due. Many people after Project Basecamp just assumed any research effort by that crew would have negative results. The truth is the Project Basecamp PLC’s were pathetically weak and fragile and the Tofino was strong.

and just watch Reid present at S4x13.

Eric Byres and the Tofino team were rightly proud of results in a post S4x13 blog. I do think that title of their blog describing the results was exactly wrong: Digital Bond Testing Proves Tofino Hardens Vulnerable SCADA Protocols. It showed the value of application layer firewalls being applied in ICS and the good work by their R&D team in producing a secure device, but it didn’t reduce the risk of any part of the vulnerable SCADA protocol allowed through the firewall.

Additionally if you watched my NOW 10-minute keynote you can understand I’m disappointed that Eric went the SCADA apologist route. Eric states my urging of replacing the insecure by design PLC’s in the critical infrastructure in the next 1 – 3 years as “completely unrealistic”. How much longer should we wait? I wouldn’t argue if he or anyone else states it will take 1 – 4 years or whatever time frame they believe is more realistic, but it needs to start now. If industry experts like Eric say it is “completely unrealistic” to expect to secure critical infrastructure ICS why would any policy decision maker or C-level executive concern themselves with securing CI ICS?

Devices like Tofino have their place. We typically recommend the Read Only versions for sharing information between Safety Systems and DCS or anywhere else where you can use it to prevent writes or other function codes that would affect PLC and process integrity.

However, the idea of putting a Tofino in front of every critical infrastructure PLC seems like a waste of time and money. If you are going to let write requests or proprietary function codes that can load ladder or application logic through the firewall what are you really accomplishing? The time and non-trivial expense is better spent upgrading to a secure PLC.

In the end, technology like Tofino should and will be integrated on the Ethernet boards in the PLC’s and controllers themselves.

The Value of Security, And Some History

by 2 Comments

Last week, Dale had difficult conversations regarding cyber security with two vendors. Apparently, that was the week for vendor interactions, as I had one too. My interaction was with a control system component vendor, attempting to explain the premise of my upcoming S4 presentation.

I’ve have been downloading as much automation software as I can over the past few weeks, and running Microsoft’s Attack Surface Analyzer against all of them looking for common vulnerabilities and insecure changes. I plan to present the findings at S4, along with some directions for improvement. Please note, this is much different than attempting to find exploits in the software, my work is to see how the software itself can change the underlying OS to make it less secure. I’ve done ~16 pieces of software thus far, and I’m hoping to include a few more as well.

The control system vendor I ran into made a zip file containing the software available on their website, but required an email to get the password to the zip file. Thinking this was just a formality, I sent in an email explaining the premise of my study. To my surprise, the president of the company responded that they “do not see any value in such a study”, and that their software “is as secure, or as insecure, as others that support OPC Data Access V2.0”.

[Read more…]

Oversold? Field Security Devices

by 8 Comments

SCADA FirewallI have a problem with field security devices. Well, not really A problem, but multiple problems.

1. Avoiding The Root Cause of Insecurity

There is a tendency in the ICS community, and even among those considered ICS security gurus, to promote building higher walls around insecure by design products and protocols rather than adding even minimal security controls to those ICS products and protocols.

Many loyal blog readers are shouting at their browser — Field Security Devices Are Compensating Controls! True, and they have their place as an interim strategy. However every critical infrastructure owner/operator who is considering buying and deploying field security devices should be simultaneously demanding their vendor provide a plan (features, cost and timetable) to add security to their PLC or RTU. If the vendor says no, go to another vendor until you find one willing to produce and sell a secure PLC.

By the way, this avoiding the root cause of insecurity goes well beyond proponents of field security devices. Take a look at NERC CIP, ISA99, ISASecure certification and other standards and community efforts and you will see the higher wall approach.

2. PLC’s Still Expose Vulnerable Services (by necessity)

In most SCADA and DCS there is still a need to access the PLC in a way that a field security device can not prevent or protect. Write function codes typically must be allowed through. If you want to use the multitude of capabilities in a Modicon Quantum you need to let function code 90 through the field security device. Since these protocols lack even basic authentication, an attacker with a spoofed IP address will be able to attack through the security device.

Similarly, many owner/operators, particularly those with geographically dispersed SCADA systems, have operational requirements for insecure PLC management protocols such as Telnet, FTP and TFTP. The activities that are required for remote management are the activities an attacker would use.

3. Overselling The Benefits

I don’t blame a field security device vendor for promoting their product and every possible benefit, it’s their job, but others should be a bit more skeptical. In a recent blog entry: SCADA Security: Tofino Provides an Alternative to Patching, Eric Byres discusses how security profiles in Tofino can detect and prevent attacks on known vulnerabilities in the PLC it is protecting. For example, it could detect an attempt to login to the FTP with default or hard coded credentials. Or if a known string causes the PLC to crash, it could be blocked.

There is some benefit to this, but if the ICS protocol or other required protocols allow writes, ladder logic changes, firmware updates, etc., how much risk have we actually reduced. In addition, these same restricting feature could be configured in infrastructure routers or switches in a much more efficient manner. Or some of the PLC’s allow you to turn off vulnerable management services such as the http or telnet service.

I guess if you have deployed Tofino’s it is another way to take advantage of a paid for and deployed device, but it hardly would be a reason to recommend a field security device. To be fair to Eric and Tofino, he flat out states it is not a panacea for all PLC security faults.

The Place For Field Security Devices

After that rant, you may think I don’t believe in field security devices. Not true. They have their place including:

  1. The read only models are useful for segmenting DCS from Safety Systems (SIS). For example, the Tofino Read Only Modbus firewall allows the DCS to pass properly formatted read requests packets to the SIS, but it blocks writes, diagnostics and all other function codes. We have seen a few owner/operators use this with great success.
  2. In front of fragile PLC’s that crash with malformed or unexpected traffic. An attacker can still compromise the PLC, but the vast majority of PLC failures have been from non-malicious but unexpected traffic hitting the PLC interface. I have wondered if vendors such as Honeywell chose to sell a firewall in front of their controllers rather than fix fragile protocol stacks.
  3. As more granular ICS protocol support is added, the potential benefits increase. Everyone starts with Modbus TCP because it is such a simple protocol, but you could come up with a set of non-invasive capabilities (an extension of the read only idea)
  4. Integrated into the Ethernet card of the PLC as one of many embedded security features. This is the future.

Finally another Weissian comment. Reid Wightman is testing a number of field security devices and will present the positive and negative results at S4 2013.

Unsolicited Response Podcast #1 – Brian Ahern, Industrial Defender

by 2 Comments

SCADA Security Podcast

Yes, it’s a new podcast. The Unsolicited Response podcast will be similar to This Month In Control System Security podcast in format and content, but I have given up the idea of doing it on a regular schedule.

The inaugural episode is an interview with Brian Ahern, CEO and President of Industrial Defender. Industrial Defender is one of the oldest companies dedicated to ICS security and has moved emphasis back and forth between products, professional services and managed services. In recent years the focus has clearly been on the monitoring and management technologies and partnering with large ICS vendors.

I talked with Brian about two main issues:

  1. What do these announced partnerships with ABB, Elster, GE, Itron, and Telvent really mean. Are they more than marketing buzz? How does a company the size of ID support all those partners? What level of integration is happening with ID security agents and technology with the vendor products? How important is the partner business to ID today and what is expected in the future?
  2. How does ID balance the monitoring and recovery benefits of remote access to a SCADA/DCS with the risks of this remote access, especially in light of the recent cyber attack on Telvent?

Subscribe to Unsolicited Response Podcast in iTunes

I selected the new title “Unsolicited Response” for two reasons. First, it is not regularly scheduled and will occur as events, topics and interviewees warrant. And second, we are increasingly being told to be quiet and not push so hard, so a lot of our views are unsolicited and issues that many would be prefer be kept quiet so they can continue to be ignored and kept at the status quo.

We will not be accepting sponsors for the podcast until we get a feel for how active the podcast will be. I always enjoy the interviews so hopefully it will be a frequent occurrence. Stay tuned.

Image by Five Wun O Clothing

 

Quick Review: AlertEnterprise ComplianceExpress for NERC CIP

by Leave a Comment

AlertEnterprise

AlertEnterpriseWe recently reviewed Industrial Defender’s ASM that had a large NERC CIP compliance automation component. Now  AlertEnterprise enters that market with their ComplianceExpress. This review is limited to price and spreadsheet information.

The first interesting data point is AlertEnterprise is specifically targeting this at the smaller utilities, municipals and cooperatives. While they may appreciate the automation more because of limited manpower, they also are likely to have fewer critical cyber assets.

The main product features are:

  • Document management for the documents that must be created and maintained for CIP
  • Workflow management to guide the company through the required processes
  • Notification monitoring of key feeds for NERC CIP information
  • Compliance event tracking

Pricing starts at US$99,000 and can go up based on what modules are selected and the size of the system. It is less than the ASM pricing, but it also has significantly less features and capabilities.

A big question with all NERC CIP compliance products is future proofing. If you buy it today, does the support come with the upgrades for NERC CIP Version 5? This will be important to safeguard not only the investment in buying the product, but also the manpower invested in entering the data.

Image by squeaks2569

Product Review Part II – Industrial Defender ASM Online Demo

by 1 Comment

In Part II reviewed Industrial Defender’s Automation Systems Manager (ASM) based on interview and some limited detail documents. Today I had the opportunity to get an online demo of the ASM interface and ask a lot of questions for just over an hour. You can see in the diagram below that the ASM has a number of software applications, more than can be covered in an hour, but here are my thoughts pro and con.

Industrial Defender

Asset Management

ASM really begins with the Asset Management module. Minimal information is entered into the ASM, and then the ASM gets the rest of the information through either agent, Industrial Defenders IT and ICS agents, or agentless technology, such as WMI for Windows systems. Information on the ports, services, software, users, etc. are all pulled into the ASM where it can be monitored for change and used for other purposes, such as the security patching program.

What about assets that are not entered into the ASM? An ARP Watch feature on either the Network IDS sensor or ASA collector appliance looks for any MAC or IP addresses not in the ASM and generates an alert that an unknown device is on the network.

North American electric utilities probably already understand the NERC CIP value this ports, services, user information can provide from a compliance standpoint, but it is valuable for any sector’s security monitoring and management. Alerts can be generated when new ports, services or software are on a system (and yes they have ways to deal with dynamic ports and services that start and stop).

The Asset Management module has the information and management component of patch management, but it does not actually apply any patches. Assets can be put into groups, and there should be some thought put into the groups. You can have OS groups, device type groups (eg HMI, EWS, Historian, PLC, router), or anything else you can think of. Your groups will affect the security patch management workflow because the ASM user needs to designate what new patches apply to the groups.

One of the most interesting futures is the capability to import security patch information from the ICS vendors. For example, GE or Siemens could provide a list of the approved and required OS, database, and ABB security patches tested and approved for deployment in a file that the ASM could import and then apply to the appropriate assets. The ASM user would then see all the security patches that need to be applied by working with the asset through the agent or agentless connection.

Configuration Change Management

Read the title carefully. This module does not provide the ability to change the configuration of a firewall, router or ICS device. Rather it provides the ability to identify changes.

A simple example — the IT Department has the skills to manage the Control Center / Enterprise firewall, but the Operations Group is worried that changes will be made without their approval. ASM could identify and generate an alert for all firewall changes. This is not a replacement for a Tripwire-type product, but it can identify changes in any configuration file.

[Read more…]

Product Review – Industrial Defender’s ASM

by Leave a Comment

Industrial Defender ASMIndustrial Defender announced “its flagship product”, the Automated Systems Manager (ASM) last week at the SANS SCADA Security Summit. On Tuesday I had the opportunity to talk with ID’s CEO Brian Ahern and VP of Marketing Kim Legelis to learn more about the product.

Industrial Defender believes, backed by some recently commissioned Pike Research Reports, that owner/operators would like a unified management platform that handles virtually everything: log managment, security device management, security monitoring, compliance management, change management … This is largely true with the exception of some very technical types who want best of breed of everything and will cobble it all together themselves.

So if we accept the premise that most organizations want one manager to rule them all, why isn’t this prevalent in the IT space or the ICS space? Easy answer – it is very hard to do at all and even harder to do well. Network Associates tried this by buying a number of security products with the hope of unified security management, failed miserably and eventually dumped the effort. Cisco struggled to make a useable manager for firewalls and routers, and what Industrial Defender is trying to do here is even bolder.

When I raised this challenge to Brian he felt that their ownership of all of the management code will make this easier. Most of the code has been developed by ID over the past ten years, and when they use a third part product they buy IP (intellectual property) rights to use the code. A recent example of this is the CoreTrace application whitelisting product. ID has the right to modify or fork the code if they want, and use it as they choose in ASM.

I haven’t had the opportunity to sit in front of the manager and give it a test run yet, so that will be Part 2 of the review at a later date. Here are some of the main points of the ASM offering:

  • Initially ASM is being sold as a 2U appliance, but in the near future it will be available as a virtual machine loadable on customer hardware.
  • There is a plan for ASM modules that would be collectors (or jump servers) placed throughout the network. This is very similar to the managed security service provider (MSSP) model to distribute the collection and processing of data.
  • The ICS specific portion of the ASM offering are related to the agents ID has had for years that collect data from SCADA and DCS, better understanding of ICS systems and protocols, and compliance modules and reports for things like NERC CIP and CFATS.

The pricing of the ASM is very complex. First it is based on the choice of Monitor, Manage or Protect – three packages of capabilities as shown in the figure below (click on it to enlarge). With the capabilities selected, the pricing is then based on the number of endpoints, clients, servers, licenses and other factors related to the size and complexity of the system. Pricing is difficult to state even when you’re not trying to duck the question. When pressed, Brian said pricing could range from $35K to a couple of million dollars.

[Read more…]

About Us

Digital Bond was founded in 1998 and performed our first control system security assessment in the year 2000. Over the last sixteen years we have helped many asset owners and vendors improve the security and reliability of their ICS, and our S4 events are an opportunity for technical experts and thought leaders to connect and move the ICS community forward.

Recent Comments