Tridium Fails and ICS-CERT Flails

Tridium FailThe Billy Rios / Terry McCorkle article about the vulnerability handling of Tridium and ICS-CERT is a must read. I started to pull quotes from it and found I wanted to include almost everything. It’s clear that Tridium was unresponsive not only to Rios/McCorkle report of the vulnerability on January 30th, but also to a Pentagon customer in 2011. Billy and Terry write:

We don’t understand Tridium’s claims that, “The firm also is doing more to train customers about security” when the root cause of these issues is poor design and coding practices from Tridium itself. Maybe Tridium should invest in training their developers about security first.

There are some equally tough words for ICS-CERT and the US Government. While giving ICS-CERT credit for diligent coordination. They wonder out loud why the priority seems to be protecting the vendor reputation.

However, when a vendor is unresponsive or refuses to accept responsibility for an issue, ICS-CERT should have the authority to inform those customers who are vulnerable in a timely manner. DHS and ICS-CERT work for us, the American people… they do not work for the PR departments of ICS companies. ICS-CERT should be able to take the appropriate actions to ensure that we’re safe and to ensure ICS customers have the right information to mitigate and control risk. The PR damage done to any individual company should never be part of this equation.

Amen. We have seen this repeatedly with Siemens and other vendors driving the what and when of the bulletin unless some “irresponsible” researcher discloses the problem. From a personal standpoint, we are still waiting for the US Government to tell owner/operators their PLC’s are insecure and need to be replaced or upgraded.

ICS-CERT Alert

The Rios/McCorkle article came out prior to the ICS-CERT Alert, but I want to focus on the flailing in this alert.

What would an owner/operator be looking for from an alert? A clear understanding of what the vulnerability is and how it can be addressed. This and many other ICS-CERT alerts fail badly on this criteria when you consider the ICS owner/operator audience that ICS-CERT is speaking to. The two most vivid examples in this alert are:

1. Lack of clarity on the vulnerability

The vulnerability is written for a security professional and even a bit vague for that audience. The only ICS-CERT text describing the vulnerability is:

Independent security researchers Billy Rios and Terry McCorkle notified ICS-CERT of a directory traversal and weak credential storage vulnerability with proof-of-concept (PoC) exploit code for Tridium Niagara AX Framework a software. According to their research, the vulnerabilities are exploitable by downloading and decrypting the file containing the user credentials from the server.

Why not try keep it simple such as:

If an attacker is able to login to the Tridium Niagara AX Server with any account he can perform a directory traversal attack and download a file with all user account names and passwords. The Niagara installation automatically installs a guest with no password and a demo account with default password that can be easily found in an Internet search. If these accounts are not disabled, or if an attacker has recovered another set of account credentials, then the system can be completely compromised. The attacker would recover the account name and password for an administrator and use these credentials to attack the system.

I’d like to see one more sentence indicating the bad things an administrator can do to really drive the point home. ICS-CERT should not be coy about the impact of a vulnerability. The bulletin could add something about the protection of the credentials being weak, but that doesn’t really matter to the owner/operator.

2. Fuzzy Mitigations

The ICS-CERT Alerts and Advisories consistently include good security practice information in the mitigation section. While this is useful information, it doesn’t belong here where a customer is trying to figure out how to address the problem. How is using DHS’s CSET tool going to help with this problem? The firewall and VPN advice is classic don’t let the bad guys get to the vulnerable product advice. It represents a third of the Alert and distracts from the critical information.

Perhaps the biggest problem is the Tridium mitigation advice. It’s actually good advice, but since the vulnerability was never explained properly to the target audience, they may not realize that preventing the bad guys from logging in with any account is the key to preventing this attack until Tridium fixes the actual vulnerabilities. The Tridium advice sounds like good practice security advice, but so is the ICS-CERT advice that follows. The difference is the disabling Guest and Demo users is essential.

The Alert should be very blatant in the mitigation section. It should lead with some straightforward sentences such as:

The Directory Traversal vulnerability is still present in the Niagara software and can be exploited by anyone who can login to the Niagara web interface. The most essential step to prevent this attack is to disable the Guest and Demo users. Niagara software users should also take the following steps to make it more difficult for an attacker to login to the Niagara web interface …

ICS-CERT action is so consistently bad that a number of people I respect in the community have told me to stop bothering with ICS-CERT. They say ICS-CERT is worthless and a distraction. This is regrettable because ICS-CERT and DHS still are the best opportunity in the US to get ICS security information out to owner operators, and they have the biggest bully pulpit to instigate a change in culture and attitude.

Image by Asiatic League

14 comments to Tridium Fails and ICS-CERT Flails

  • I see ICS-CERT trying to develop phraseology such that people can reason this out for themselves. I have criticized their phraseology in the past where confusion could result, and they have been very good about changing the wording. Clearly, there is much work to be done here.

    The missing link here is a document explaining the phraseology, and what the implications are. So, for example, a weakly hashed signature is an attack vector. What does that mean from the perspective of a customer. Some may understand this right away, others need guidelines.

    The process of teaching people what these guidelines are is a major undertaking. DHS would practically have to write a significant handbook on SCADA system operations and design to describe all this.

    That’s why I think the note reads the way it does. I agree that it could be better. But the real problem is not writing a better note, but educating the public and evangelizing the whole process. Most don’t even know these notes are out there.

    Jake Brodsky

  • Great blog Post.

    Jake has a very good point. While there are some owner/operators that are very aware of the ICS-CERT work, most users have never heard of them. I would suspect that the vast majority of the folks that own/operate the Niagara system are not aware of ICS-CERT at all. The Washington Post article is much more likely to catch their attention (and that will miss most of the techies) than the ICS-CERT Alert.

  • This is a classic mistake of assuming attackers won’t be able to work out the coded phrases, but defenders will. In reality, defenders won’t be motivated to figure it out, because they’ll assume they’re covered by “best practices.” Attackers, on the other hand, WILL take the time to discover the vulnerability because they can be reasonably sure defenders won’t, hence it’s worth their time to do so.

    ICS-CERT and any other CERT needs to be very blatant about what the specific issues are so concerned defenders can immediately go straight to the problem and fix what’s wrong. Attackers will figure it out any way. As soon as the exploit is indexed by Google, every attacker has it. The same is not true for defenders.

  • bryan owen

    “The process of teaching people what these guidelines are is a major undertaking.”

    Perhaps webinars or another interactive formats could help with this. As I recall EnergySec hosted one to follow Basecamp disclosures.

    Is a monthly ICS-CERT roundup a lame idea or maybe this approach is already done when needed by the ISACs?

  • Jake – The effort you discuss of educating the ICS community is probably worthwhile and perhaps should have been done long ago.

    That said, why not use plain language?

    I would be more understanding and appreciative of the use of infosecurity lingo if it was providing more information. This is after all why various fields of study develop a set of terminology so that the topic can be discussed precisely. In the case of the ICS-CERT Alerts, this one being a prime example, the terminology is more in line with the “coded phrases” that Chort mentioned in his comment.

    Dale

  • From all the excuses I have heard for ICS-CERT bulletins being irrelevant the most funny one is that they would be coded in some kind of crypto, hiding hardcore information from hackers.

  • Dale, I have spent enough time on teleconferences and in face to face meetings reading and writing documents with various people from all around the world to know that “Plain Language” is a sick joke. You can think you’re making perfect sense and yet others can read what you wrote and get it horribly wrong.

    That’s why I am a big advocate of standardized phraseology. They use this technique for documents from the FAA when describing airframe and powerplant service bulletins. They rely heavily on this technique when communicating with air traffic control.

    I really do not care where the phraseology comes from, as long as it is consistent, unambiguous, terse, and reasonably descriptive of the issue. There will be tweaks. Even these days, the FAA changes phraseology decades after writing up the first generation of standardized terms.

  • Jake:
    I agree that a set of standardized terms is an invaluable aid to communication, especially when people come from a wide variety of backgrounds. It also makes the preparation of the communications like these alerts a much easier process.
    Having said that this only works when everyone knows and agrees upon the meaning of the phrases and words. If ICS-CERT is going to use standardized phraseology (and that has yet to be established) they need to publish a guide that can be used by the various people who need to read and understand the language.

  • Regarding Patrick’s recent comment: That was the point I tried to make in my first post.

    Once phraseology has been standardized, there has to be a document outlining the implications of saying such things in a report. I give ICS-CERT poor marks for not doing anything to

    1. Broadcast that these reports exist in utility forums.
    2. Discuss the phraseology and what it can imply.
    3. Authenticate these reports, particularly the TLP AMBER preliminary reports. The US-CERT e-mails are digitally signed, but for whatever reason the ICS-CERT ones are not. If anything, it ought to be the other way around.

    In reality, I’m not disagreeing with Dale very much. He gives ICS-CERT poor marks for not using “plain English”, and for not clearly defining the mitigation efforts.

    I think I took things a step further. I pointed out that such things probably shouldn’t be in these reports, given the diversity of systems out there. The issues in the report may range from serious to irrelevant. I point out that standard phraseology can do better than “plain English”. And finally, they are not doing so well at advertising what these reports, or how to interpret the severity.

  • Bryan Owen

    @Jake

    Any opinion on ICS security phraseology fit with existing IEC/ISA terminology or Mitre standards?

    Is common phraseology a valid initiative for the ICSJWG working groups?

  • I would like encourage readers to join and participate the LinkedIn group entitled S.N.A (Securing Niagara AX – http://tinyurl.com/cj4laeh). This group was established today and has already has a broad range of members including vendors and implementers.

    We are coming together to address security vulnerabilities of the Niagara platform.
    McKENNEY’S, INC.
    Fred Gordy
    Technology Evangelist
    fred.gordy@mckenneys.com

  • I have been working the Tridium product for over a decade and have seen it evolve and grow to become a very substantial and robust platform. Just like with any product we have to accept part of the responsibility of how the product implemented and how we secure. There are built in features that allow you to secure the platform. The Tridium document entitled NetworkingITGuide.pdf is fairly comprehensive and outlines best practices that if followed can secure the system. Also some of the responsibility fall on the IT staff that the system is to be installed on. Because they are network devices (like any other device on a network), the IT staff needs to make sure the network is secure physically and firewalled correctly.

    Tridium has not do a good job of promoting what is already built in but they have listened over the years and put in things such as LDAP integration, SSL, the ability to change ports, the ability to create security groups. The company that I work for makes it a priority to spend time with the IT staff prior to a job installation to make sure security requirements are met. We are able to use the documents supplied by Tridium to meet and exceed the security needs for any given job. My job specifically puts me in the position of working with IT managers, government agencies, etc. and this gives me an insight of what is expected and needed in order to secure an installation.

    I believe what is lacking is education on what is built into the Tridium/Niagara platform and how to implement the best security strategy.

    McKENNEY’S, INC.
    Building A Higher Standard
    Fred Gordy
    Technology Evangelist

  • Dale G Peterson

    Fred, I have to call bullshit on that comment.

    A serious set of vulnerabilities that were not difficult to find and likely to result in the compromise of many customers were ignored at least twice by Tridium. The first time, according to the Washington Post, was when a US Government identified the vulnerability. Tridium didn’t fix it or warn customers.

    The second time was when McCorkle & Rios found it and coordinated through ICS-CERT. The vendor refused to fix it until the Washington Post reported on it.

    Then Tridium had the chutzpah to say customers needed to be better trained when their developers made Secure Development 101 errors and refused to fix them.

    Does this mean Tridium or their product is universally bad? No. But there is no positive spin on these development and vulnerability response issues.

    Many vendors do an equally poor job the first time they are faced with this. Hopefully Tridium will learn and follow the ICSJWG or other guidance.

    Dale Peterson
    Digital Bond

  • Dale, I am in agreement with you and hope my point got a across. I do not want to absolve any responsibility on the part of Tridium. My point is that we as implementers need to do a better job of securing the systems and not use “out of box” setup.

    We are thankful that McCorkle & Rios brought this to forefront because it spotlighted that we, the suppliers and programmers of these systems need to move from the background to the foreground of leading the effort to make sure our customers are protected.

    Currently we(McKenney’s, Inc.) are in talks with Tridium to aid in the process of addressing these vulnerabilities. We are also establishing and documenting an independent security audit process for not only the Tridium product but other control systems as well. We began this several months ago and would invite anyone to participate with us.

    We also started a group on LinkedIn specifically for the purpose of drawing together others in the industry to participate in making the Tridium product secure. I invite you to join us as a viewer or even a participant into what we are doing. We started this group on July 24 and already have members from the US, United Kingdom, Australia, Japan, Switzerland, Singapore, Canada to name a few.

    Fred Gordy
    McKenney’s, Inc.
    Technology Evangelist

Leave a Reply