I’m a week or two late on this, but I think that the community as a whole has paid far too little attention to the advisory released a few weeks ago by the folks at C4/CERT, and the response to them by Rockwell. Full disclosure, I have not personally verified these findings as we don’t have a Micrologix device in our lab, but trusting CERT for basic details is usually fine.

Let me start off by saying that I really don’t have a problem with poor security if you aren’t trying to be secure.  If a trade off has been made for usability, safety requirements, or something else, then that is the prerogative of the business and those in charge of keeping its processes going. While I believe its usually a misguided choice, at least its a choice. What does bother me are security features that provide a false sense of security.

Which brings us back to the advisory.  Specifically the line:

“During the authentication process, the PLC transmits the management password in plain text to the client.”

The user attempting to authenticate to the device is sent the password, in plain text no less, that they need to respond with to authenticate.  That’s bad.  The authentication model of the device relies on the users being nice enough to use the provided client software. This is client side authentication at its worst, and I would have thought that a professional developer with any sort of training, or experience, would have realized that that is poor design and something that should never be used.

But why should we even care about this when most devices like this don’t have any sort of authentication built in at all? Because it misleads the end user as to what their security expectations should be for the device. It could lead to opening up a hole in their firewall, allowing access from anywhere on the corporate or the control network. It requires authentication so that would be fine, right?

But it doesn’t end there, the gap between any sort of security understanding and this vendor is even more evident in their response to these vulnerabilities, contending that it would take a “highly skilled, unauthorized person, using specific tools to intercept the controller password.”  Highly skilled to examine the network traffic directed at their system? And use specific tools? What does that even mean? Isn’t any given tool a specific tool? Sounds like someones PR/marketing department is a little too involved in the disclosure process.  And I’m still scratching my head over the workaround advice to change the password frequently.

Vulnerabilities like this give a glimpse into the security education and development process inside the company walls, and what we’re all seeing isn’t pretty. We’ve been pushing for a while that operators and vendors alike raise the bar of security expectations in control systems, but issues like this, which are poorly thought out at best and intentionally misleading at worst, is not the way to do it.

A popular week for cyber security in the press this week including a big article on cyber security jobs in the USA Today.

A few thoughts after the intelligent comments, additional info, sound and fury:

• Microsoft is in the very rare top tier of companies spending time and money on security. In gross $ and time probably number 1 and very high on a percentage of security to software development time. They are also among the most attacked. So the information they provide on fuzz testing effectiveness and other parts of the SDL is an important data point.
• Microsoft’s approach with white box testing and their SAGE tool is useful info for the right people in a vendor’s development team. Again, Daniel will blog on this next week.
• In case I didn’t emphasize it enough, one of Matt’s important points was, “One of my conclusions (which I was pleased to hear echoed in the Microsoft talk) is that no single tool is best, no single approach is adequate–and that there are different types of fuzzing users that will require different feature sets.” Read his full blog entry on this that includes a challenge to a young researcher to do a fuzzer bake-off project rather than develop another fuzzer.
• I would really like to see a bake-off of fuzz testing solutions.
• I buried the lead and put the most important point last, vendor’s need to be fuzz testing their products. So whether it is Mu, Wurldtech, a collage of open source, or home grown tool is still not the most important issue in the control system community, unfortunately. Many more vendors have added fuzz testing to the SDL than five years ago so the trend is positive, and the fuzz testing solution vendors have helped this happen. Hopefully ISCI will help even more. Asset owner’s should be asking for the SDL of their vendors. If it is not readily available, yellow or perhaps red flag. If it can not be explained consistently, red flag. If it does not include fuzz testing, red flag. As has been pointed out in almost every presentation, even those that were not fans of my post, dumb fuzz testing finds exploitable vulns in many products that have not been fuzzed by the vendor.

Some vendors have reached out to provide more information on their approach, and I’ll have our offensive security guys follow up on this.

Read Best Way to Fuzz Part 1 and comments

Today’s press release from an unnamed company (to protect the innocent of course) has driven me to zombify the tired “all you base” internet meme. In our ever growing drive to trade security for ease of use and convenience you can purchase micro devices that plug into your RS-485 lines and on one end, encapsulate your previously somewhat protected serial comms into nice TCP/IP packets, and send them across your ethernet network to the other end where they are decapsulated and passed back serially to your field device. Sounds nice at first glance, allowing you to use existing twisted pair wiring, until you look at it with a security perspective.

These communications which were previously somewhat protected from the majority of IT based attack vectors are now exposed to the same packet tampering, replay and data spoofing attacks of any other TCP/IP communication. The protocols are not unknown, serial Modbus being an example where documentation is available and an educated attacker could now readily tamper with the data. From a hacker’s perspective, the more perviously tough to get at serial back plane communications now encapsulated in TCP/IP the better.

Upon reading the press release I dug up more info on the product hoping to find some assurance of packet hashing or encryption.

At S4, one of the break discussions I engaged in was how much would it cost manufacturers to upgrade CPUs in forthcoming devices to CPUs that would support full packet encryption, without adding latency into the communications pathway. The consensus was from a pure manufacturing stand point, less than $5.00 a device. While this doesn’t account for the additional R&D and testing such a product would require it does beg the question:

When will we stop trading ease of use and the saving of a few dollars in product development for actually producing products where security is one of the driving design principles?

Last week McAfee and CSIS released a report titled In the Crossfire: Critical Infrastructure in the Age of Cyber War. Honestly, I dismissed it at first as marketing hype and even took some shots at it on Twitter because of the lack of real data. But they are actually very clear that it is a survey, and not even one that uses valid statistical sampling and error margins. They describe it as a “rough measure of executive opinion” which includes “600 IT and security executives from critical infrastructure enterprises across seven sectors in 14 countries”.  As much as I tried, I couldn’t ignore it. Even if you believe it’s just an attempt to keep the budget faucet flowing, the report does offer some interesting observation points.

Most of the survey is focused on the critical infrastructure organization as a whole. There are some charts that show a measure of perception or belief about things like which foreign governments are attacking them for example. Let’s save that for a future discussion — tie it into APT perhaps.

The subset of the survey respondents I want to focus on are those who indicated they have control system responsibilities (143 out of the 600, a notably smaller group). They were asked questions about measures deployed on their SCADA / ICS networks. Working in the field, we develop our own observations of different sectors based on what we see. The organizations that hire us are generally proactive about security so our data set tends to be somewhat biased. (Although I’m sure there is some level of bias for those who respond to a McAfee survey as well.) Keeping that in mind, here are a few of the more interesting data points from the survey with my comments:

Application whitelisting is more widely adopted in control system networks than IT networks

I’m actually not that surprised — it just seems like a natural fit. But then I’ve already outed myself as an application whitelisting fan. We had an interesting paper at S4 this year discussing the topic as well.

65% report using firewalls as a security measure on their control system networks

To get the full picture on this, I’d like to see it broken down by sector which wasn’t done in the report. Just another stark reminder that there’s still a lot of 101 level work to be done out there.

43% report SIEM/SEM technology is employed on their control system networks

I was surprised by this number. It’s a higher percentage than I would have guessed. Given the current status of control device security monitoring capability, however, the effectiveness of this measure is questionable.

Over 75% say their systems are connected to the Internet or other IP networks

That sounds about right. I wonder how much this will change because of security concerns raised by report like this and the possibility of dodging potential compliance requirements such as NERC CIP. Could the rate of Ethernet/IP adoption flatten out? Maybe the real question revolves around the networks that have been IP-connected for 10-12 years or longer. Will the same business drivers that pushed them into IP also push other “newer” technology – wireless, SCADA as a Service, etc…?

Beyond these points, there is a lot of confirmation of things we probably already knew. There were more attacks and greater impact reported in the Oil and Gas sector than others. The simple explanation: more connectivity equals more exposure to attack. On the other end of that continuum is the Water sector, which reports lower attack and security measure adoption numbers but also the lowest degree of IP connectivity. Again, not much explanation required there.

If you haven’t seen the report yet, here are some direct links for your convenience:

Direct link to the report

Audio commentary

Presentation slides

I leave you with this question: Given the opportunity, what would you ask 600 (or 143) IT and security executives from critical infrastructure organizations around the world?

There was an interesting discussion and information on what is the “best way from an ROI measure” to fuzz test at the CERT sponsored Vulnerablity Disclosure Workshop in DC this week. It led to some tweets back and forth between Digital Bond alumni Matt Franz and myself. First some background:

Fuzz testing is used by vendors, I hope, to look for common coding errors that can lead to vulnerabilities, such as buffer overflows. Consultants, researchers and hackers of all hat colors use fuzzing to look for exploitable vulnerabilities. Steve Lipner of Microsoft and co-author of the Security Development Lifecycle [SDL] said in his 2008 S4 keynote that fuzz testing and threat modeling proved to be the most effective ways to reduce exploitable vulnerabilities. Asset owner should be asking their vendors in RFP’s and User Group Meetings to explain their SDL and insure fuzz testing is part of it.

We have two security vendors that are trying to sell products to the control system market: Wurldtech with their Achilles platform and Mu Dynamics with their Mu Test Suite. [FD: Wurldtech is a past Digital Bond client and advertiser] One of the features of these products is they both send a large number of malformed packets at an interface – - typically crashing protocol stacks that have ignored negative testing.

While this greatly simplifies the issue, the two vendors have taken different approaches on how to create those negative packets. Wurldtech touts the use of a structured grammar to create the malformed packets, and MU takes more of an expert systems approach where there security engineers determine what would be the most effective malformed packets to send. There is certainly some overlap between the two approaches, but the question has always been what is more effective at identifying protocol stack errors.

So with that as background, Matt’s tweet “At CERT vuln discovery workshop. Interesting MSFT says grammar based fuzzing has lower ROI than dumb fuzzing” caught my attention. Matt was kind of enough to expand on his summary in an email:

The consensus of the talks was that you can’t rely on a single tool or technique but the ROI was higher for dumb “mutation based” fuzzers and white box approaches like SAGE than the time and effort to develop grammar based approaches, model the target, etc.

The direct comment was they still used “smart fuzzers” for highly critical code, Office, IE but that it wasn’t practical for other platforms like Exchange due to the way that it would hold up development and release cycles. Even/Especially in MSFT and CSCO devtest resources are precious and finite. Relatively poorly skill devtesters were able to achieve good enough results.

So if you are a vendor, or even an asset owner, starting from scratch you will have a low ROI on developing a grammar based fuzzer. But what if the grammar based solution already exists, such as in the case of Achilles, and you can buy it? This makes the ROI decision more interesting because you could compare the Achilles and the Mu Test Suite head to head and take into account any cost differences. So actually the question still remains if you are looking to purchase a control system fuzzer.

In a future blog post we will have Daniel cover Microsoft’s fuzzing efforts in the form of their SAGE tool which does “white box fuzzing” using symbolic execution and negative constraints. SAGE is still an internal Microsoft tool, but the approach is public.

Earlier in the week I came across a very interesting article regarding control systems that we normally do not discuss but has a similar issue that we experience in other control system implementations.

The FAA recently published a “special conditions” document within the Federal Register (FR DOC 2010-661) discussing concerns over Boeing’s 747-8/-8F models network architecture.  Specifically, they discuss the concern of the potential of having critical systems (flight safety control, nav systems, etc.)  networked together and susceptible to attack from an external wired/wireless source.  The paper goes on to say, that a means of ensuring network security needs to be provided by Boeing for these models and any future models with this network design.  Lastly, the paper notes that it was issued because “14 CFR regulations and current system safety assessment policy and techniques do not address potential security vulnerabilities.”  Many of you may note, that the FAA had issued a similar document in January 2008 regarding the Boeing 787 and its network security.  The 2008 document listed several responses, mostly by Airbus, regarding the FAA’s guidance.

While I give the FAA kudos for documenting this concern, why haven’t the regulations been changed or guidance provided to manufacturers regarding network security?  As new planes and models roll out they will inevitably take advantage of the improvements in networking technologies.

• Wurldtech’s Achilles appears to have won the battle over Mu Dynamic’s MUSIC in control system protocol stack certification. This week they announced that Honeywell’s Experion PKS C300 Process Controller achieved Achilles certification. This is noteworthy because Honeywell first went with MUSIC certs three years ago. This nod for the need for Achilles cert in the oil/gas sector combined with the list of Achilles certified products dwarfing Music certified products, at least 4 x 1 even with conservative counting, gives the TKO to Wurldtech on the certification front. I don’t have visibility on product sales so Mu could be winning in other areas, and I know the Mu products has some custom protocol features that have won some business. [Full disclosure: Wurldtech is a past Digital Bond client]
• Byres Security announced an extension of their Tofino field device security line. “The Tofino Argon Product Series, a suite of five rugged Plug-n-Protect™ security appliances with varying connectivity, temperature and power supply capabilities”.

If you administer, manage or run an Energy Management System (EMS) odds are good that you employ OSIsoft’s PI historian to record and archive the point data of your control system. Portaledge leverages the Advanced Computational Engine of PI to provide a Security Event Monitor (SEM) for the control system.

Portaledge plays two important roles in a control system. First it monitors the network for various data points that are indicative of and attack and second, it will (shortly) provide logging information that helps to meet regulatory requirements.

Portaledge monitors the network in various ways. Currently modules exist that watch for enumeration activities and for changes/degradation in the amount of resources on a system. The enumeration techniques are the standard tools by which the majority of attackers will determine what exists on a control system. The availability modules monitor PC based systems, field devices and network components.

There is also a nice Traffic Monitor module that watches and alerts for “out of the ordinary” network sessions. The ordinary traffic on a control system is somewhat easy to define. The Traffic Monitor simply alerts on any network session out of the “allowed” list defined by the administrator. This is a useful tools as most attack and exploit traffic will be targeted at port ip duplets outside of the normal ranges. The traffic monitor is part of the Enumeration Module.

Our forthcoming work on Portaledge is to provide modules that will meet NERC logging requirements. In a previous blog post I delineated some of the NERC requirements that Portaledge can help meet. There is more information also available about NERC CIP requirements and how Portaledge can aid in acheiving these requirements in the SCADApedia. The NERC CIP modules will be our primary development goal after our soon to be released Meta Event package.

If you already have PI deployed on your EMS odds are you have the majority of the licenses required to run Portaledge’s Availability and Enumeration Modules. Portaledge is available to Digital Bond’s digital content subscribers, said subscriptions are available for $100 a year. All S4 attendees also receive a one year subscription (another good reason to come to S4).

There is an interesting story from the Christian Science Monitor regarding attacks on some US oil companies. According to the article, the attackers used the same techniques described at S4 2010 in the keynote speech on Advanced Persistent Threat (APT), given by Kris Harms of Mandiant. The attackers used multiple teams to gain access, maintain access and extract data from the companies. While the article doesn’t state exactly what was taken, it does mention that the attackers may have been interested in oil deposit data. There was no mention of penetration into the control network but with the APT hacker’s ability to gain access to a oil company’s enterprise network, improper separation between control and enterprise could allow these highly skilled attackers could gain access to a control network.

One thing about Kris’ speech that intrigued me was the fact the APT hackers monitored their install base.  With enough power, water and sewage companies being infiltrated and monitored by these APT groups, serious damage could be inflicted when the time arose. While this is definitely a worse case scenario, the groups seem to have the skills to gain access to sensitive areas on a network and remain in those areas for a period of time. I cannot estimate their ability to interact with a control network but the fact that these groups use separate teams for different parts of the attack means they could find people with control expertise to perform unwelcome actions on a control system. Lucky for us the APT groups seem to be focusing on financial institutions and trade secrets.