How DHS Can Best Help ICS Security

DHSInformation sharing is a key component of President Obama’s Executive Order with emphasis on sharing information on threats and incidents. With the current insecure state of ICS, the value of this information is limited to the question IF we should start to do something about the problem OR NOT; are we really still answering that question?

And quite frankly the critical infrastructure ICS community is not ready for this information. ICS are lacking basic security controls, are insecure by design and quite fragile. Threat and incident information is only important and helpful when you have a functioning and effective security program — unless you are arguing that basic security controls are not urgently required in critical infrastructure ICS.

The best and quickest way the US Government, and other governments around the world, can help improve critical infrastructure ICS security are:

  1. A focused and very public information sharing campaign showing how insecure by design and fragile ICS are today and
  2. Highlighting vendor and owner/operator efforts, or lack of efforts, to upgrade or replace these systems with secure and robust ICS

The good news is this information sharing does not require an Executive Order, legislation or even cooperation by the vendors and owner/operators to implement.

It’s axiomatic that information sharing and public discussions too date have not convinced owner/operators that they should spend money and effort securing critical infrastructure ICS. It’s also self-evident that ICS vendors, with few exceptions, have not found adding basic security and robustness to their products is a good business decision. Therefore DHS and other elements of the US Government should focus information sharing on convincing all involved in allocating these resources

We have heard often, and agree, that most participants in ICS security know (the “everyone knows” response to Project Basecamp) that ICS are missing basic security features and fraught with vulns due to poor design and coding practices. Trying to convince the ICS community to spend time and money on ICS security has and will continue to fail. There are a small number of owner/operator exceptions, but our experience is this is most often instigated by a C-level executive who is concerned, not the operations group. And the number of these ICS security early adopters are not large enough to drive the product vendors to secure their offerings.

Pick One Key Security Control

The key is focus. We are not going to be able to break through to the people who can fund and drive ICS security with a complete security program. Identify a few security controls, and focus on them one at a time. Controls that are so critical that little else matters if these are not in place, and controls that will make a major difference in keeping the bad guys out or detect them. Controls that can be explained simply and powerfully to a non-technical audience.

Here are three examples that I would start with:

  1. All critical commands and management functions must be secured – at a minimum authenticate the source and data before accepting the request.
  2. Eliminate all remote connections to the ICS except for emergency situations. The ability to remotely access the ICS must be disabled by default, and all remote access connections must be explicitly approved and reviewed periodically to verify they were emergency situations.
  3. The ICS must have an attack detection capability that would detect new or unusual communication as well as any communication that is indicative of a cyber attack.

However long the list is, the highlight should be on one item for six to twelve months before introducing the next. Build an entire marketing campaign around that item. Yes, I wrote marketing. Not only does the information need to be shared, but it also needs to be sold.

Bring the Focus and Pain to C-Level Executives, Public Utility Commissions, Stockholders, Vendors, …

Let’s say the first item is “All critical commands and management functions must be secured – at a minimum authenticate the source and data before accepting the request.” (Side Note – most executives, Senate staffers, and anyone else outside of ops is surprised when I explain to them the ICS lack any effective security if an attacker can get to them. It truly seems impossible this is still the fact 11.5 years after 9/11.)

The basic question for any owner/operator is have you started a program to implement authentication for critical commands and when will it be completed?

The US Government should now know what ICS systems and components are used in the critical infrastructure — we could quickly create a list with 90% accuracy. Then:

  • Create a public list of the devices lacking this basic security control from the critical infrastructure product list
  • Create demonstrations showing how this lack of authentication allows an attacker to take complete control with a variety of products. Put some teeth and detail behind the Cyber 9/11 and Cyber Pearl Harbor terms.
  • Be creative with videos, social media, presentations, etc. Show real physical damage like Aurora, show a case study of teens causing the damage with a couple of days of effort, show displays that are inaccurate, show documentation that causes the problem, show statements from the protocol committees on insecure by design, … Highlight this item in messaging at all levels of technical detail.
  • Show how ineffective “the keep the bad guys out” strategy is. Document and anonymize pen tests with different approaches to getting in, including physical, cyber and blended.
  • Have meetings with critical infrastructure CEO’s and discuss the risk they are passively accepting. Ask them what their plan is to authenticate critical commands? When will it be completed?
  • Create a set of questions, related only to this one issue, that every Public Utility Commission should ask their regulated utilities.
  • Create a set of questions that any shareholder or stock analyst should ask companies running the critical infrastructure. Meet and hold joint press conferences with the SEC on this issue. Work with the SEC to add each issue to the disclosure process.
  • Encourage Congressional Committees to call critical infrastructure owner/operator and vendors who sell to them and ask them if they have started a program to address this and when it will be completed.
  • Have the carrot as well. Highlight vendors who have added authentication of critical commands to their products, particularly those that have provided a simple upgrade path.
  • Track progress on this issue with percentage of vendor solutions with this control and deployed system percentages by critical infrastructure sector

Reluctance to put out this information was understandable pre-Stuxnet, both for hiding offensive efforts and not wanting to start an offensive cyber weapon frenzy. These reasons are no longer valid, and governments and organizations around the world understand the value of and how to compromise insecure and fragile ICS. Not sharing this information is only harming the effort to secure these systems.

After the message from the first item is accepted by the C-level executives and other decision makers … that this is mandatory and a program is started, move on to the next item. After even one item you will have engaged decision makers in the process and subsequent items will be easier, and eventually lead to an ICS security program.

Again the key is to have something that will make a big difference and is easily understood. For example, a PUC, stockholder, CEO or anyone else could easily ask, “Do we have any remote connections to our critical infrastructure ICS?”. Develop a marketing campaign showing all the different types of remote connections, how they can be exploited, and an accountability program to remove them. (I’d argue the S4x13 spear phishing real world test and making the Digital Bond spear phishing email and malware public was better than the generic ICS-CERT warnings) Go to centralized sites that have a tremendous number of remote connections to critical infrastructure ICS, such as ICS vendors and support sites, and show them alternatives to address the questions they should be getting from multiple customers.

It’s also important to identify what ICS-CERT, DHS and the USG should stop doing.  Accessibility from the Internet and Shodan are a great example. Many ICS are at risk because they can be reached from the Internet, but this is not a major issue for critical infrastructure ICS. This should not be part of the critical infrastructure messaging, but it could be appropriate for general cyber security messaging to individuals and businesses … which leads to ICS-CERT efforts today and in the future.


ICS-CERT is the wrong entity to perform vulnerability coordination. They should go back to the original plan for ICS-CERT, providing Level 2 support to US-CERT. US-CERT and CERT/CC do a fine job of vulnerability coordination. ICS-CERT should set a criteria for when the devote more time than a token phone call, such as:

  • the vulnerability affects a system on their critical infrastructure list AND
  • the vulnerability affects the security posture of the system

The second requirement is important and often leads to misallocation of ICS-CERT effort. Does it really matter if there is a CSRF or buffer overflow vulnerability in a device that you can connect to and take complete control using a feature?

Removing all the coordination and effort on vulns with little impact would allow ICS-CERT to dive deeper into a smaller number of vulnerabilities, not insecure by design but true vulnerabilities, that could have a major impact on the critical infrastructure. For example, a vulnerability in the PI Server protocol would jump up high on the list, as would a vulnerability in DNP3 SA stack or WirelessHART stack. They could get these systems into the lab, verify the full extent of the vulnerability, and test any proposed fixes. The resulting Advisory would be not based on trusting the vendor supplied information.

Similar filtering needs to take place in ICS-CERT’s incident handling and fly away teams. An attack on a company with an ICS doesn’t warrant ICS-CERT involvement. ICS-CERT highlighted cases they had worked related to SSH password cracking attempts on corporate network systems. This probably happens quite often to an Exxon or FPL.

Filtering needs to take place on the impact to critical infrastructure. The infamous Springfield Water Hack, that was a false alarm, shouldn’t have been an ICS-CERT issue even if it was real. A minor pump outage in a small municipal utility cannot be chased down by our elite ICSsec forces. When DHS offers the elite services at no charge, everything becomes important. Would these organizations pay for a response by hiring someone like Mandiant? This is a good measure of the true impact of the incident.

My contention is ICS-CERT has focused on these huge number of vulns and anything that can remotely qualify as an ICS incident to look like they are busy and making a difference. It has worked with the press, Congress and even many owner/operators. However, if the measure is the improvement of the ICS security posture over the last ten years it must be considered a failure.

ICS-CERT and DHS are not alone in this failure. The ICS security community including vendors, owner/operators, industry organizations, consultants, gurus and anyone else involved has failed as well. The difference is ICS-CERT, DHS and other elements of the US and other governments have the loudest and most respected voice. Their silence on actually fixing the security problem has a huge negative impact.

As much as I would like this site, my voice and the voice of other ICS security gurus to instigate a much delayed serious effort to make critical infrastructure ICS secure and robust, it is clear that the US Government is the big voice that can make a difference in the US. The average asset owner will never take action as long as the government is completely silent on an issue which is only stressed by a handful of consultants. We need DHS, FERC, DoE, DoD, and the Executive Branch to or a similar organization to actually say out loud that critical infrastructure ICS must be secured now, not decades later. I can only speculate why they have been silent or SCADA Apologists for the last 11.5 years.

Mature Information Sharing

The type of information sharing envisioned by the Executive Order could be very helpful once critical infrastructure ICS has basic security. Today it is just not that important. The information that needs to be shared now is “Your ICS systems and procedures are insanely vulnerable. Now do something about it.”

Next … The Cybersecurity Framework

3 comments to How DHS Can Best Help ICS Security

  • Bob Huber

    Couldn’t agree more. CERT/CC-US-CERT can handle the vulnerability disclosure process quite well, and they in fact did before ICS-CERT came along. It’s unfortunate that the government directed additional funds to ICS-CERT to perform a duplicative process. INL (as the primary contract resource for ICS-CERT) did not have vulnerability handling experience, they have ICS experience; hence, the reason it was envisioned as second level support.

    Regarding the fly-aways, it’s a whack-a-mole game. Addressing one incident at a time for an entire country doesn’t make much sense. Do you see US-CERT sending fly-away teams out left and right? No. Try scaling that model. Use those gov’t resources to drive investment in ICS security programs as Dale pointed out. Most companies are making limited to no investment in building out their security programs (save those affected by NERC CIP). DHS is in a unique position to have a large impact on ICS security. Direct and/or mandate minimal levels of investment. Yes, I know the argument of the government not imposing regulations on private industry, BUT, unlike financial institutions and other critical infrastructure organizations, most people can’t (really) pick their water/wastewater/electric/natural gas providers. If your water utility is hacked a few times causing you to lose service, you generally can’t just decide to switch to a new one (something tells me Jake B. has a well). A lot of this does come down to your state PUC. If they are informed of the issues (hello DHS), then they can also drive improvements, and approve rate cases requesting additional ICS security funding.

    Regarding information sharing, Dale’s assessment is spot on. I’m in the business of providing situational awareness and threat intelligence to ICS owner/operators. I have to agree, that the majority of organizations are not far enough along to even begin to ingest this information and action it, unfortunately for me. They indeed need to have a base level of processes, procedures and technology in place to address security before they receive information. There’s been this constant request from industry to “give us data”. Great, Mandiant just provided 3,000+ indicators. Let me know what you are going to do with them. Search your IDS/SIEM/DNS logs/SMTP logs/firewall logs/host based logs/AV logs/syslogs/Encase etc? Do you have data going back two years? 1 year? 90 days? Mandiant did more in a single report than the US gov’t will do in the next 10 years. I’ve read the classified reports, and produce that type of information myself. Let me be the first to tell you, if the Mandiant report didn’t prove it. The information the government has is no better than Mandiant just provided. Sure, if you want to know who specifically was behind it, the US gov’t report….wait….Mandiant covered that too!

    The US gov’t has thousands upon thousands of indicators. You will never get those lists. At best, ICS-CERT and the rest just release the few that are relevant to current issues, OR, they just pass on information that affected asset owners/operators passed them.

  • Chris

    Very good post, Dale. I hope that upper management in all industries with ICS learn how important security is…so that they put it in the budget as a line item.

    Don’t forget that many electric utilities and probably some oil/gas operations in the US still rely heavily on serial traffic in their SCADA systems. Serial-based systems will still be around for a long time. Even with the issues of Ma-Bell pulling the plug on analog leased-lines, parts obsolescence and the upcoming NERC/CIP v5 (where some serial-based cyber assets will become critical cyber assets), utilities are stretching their budgets by installing hybrid RTU systems and using serial/ethernet terminal servers.

    I hope that those utilities who are using serial-based systems that are looking to upgrade soon will look at security as a major decision factor (instead of just being NERC/CIP compliant, etc).

  • Hey Dale,

    The solution is not an either/or choice of what information to share how and with who, but nonetheless your primary point is correct: the maximum value returned from information sharing at this point will be found where it moves C-Level actions.

    In many ways the very act of issuing the Executive Order is just that. It is also the kind of Small Data information sharing that is appropriate for C-Level use. Executives don’t process information like “you need to develop a comprehensive identification and remediation strategy for vulnerable devices on your process control system” (much less anything deeper). Conversely, they are very good at processing information such as:

    “The nation’s CEO is looking at you (and he’s pissed).”

    The President and his bully-pulpit can perform that sort of single-packet information sharing very well. Much more effectively than those of us with more granular opinions and smaller megaphones. The purpose, though, is not to make those same C-Level executives then listen to the detailed solution – which they won’t do and wouldn’t understand – but rather to cause them to turn to their subordinates and share a similarly simple packet of information which junior executives are well versed in processing:

    “The President of the United States is pissed off at me. Make it stop.”

    This “Public Sector Executive to Private Sector Executive” information sharing needs to support the more detailed information sharing that folks like ourselves can accomplish by encouraging more of the right people to engage. We see this effect already both in the activity of the ICS-ISAC as well as other conversations we engage in. The increase in ICS-ISAC membership and attendance at the center’s public briefings since the Executive Order is mirrored in other groups’ briefings we have attended since.

    Executive-to-Executive information sharing needs to support the more detailed information sharing that you mention above, though, not replace it. Inasmuch as the Executive-to-Executive information sharing is successful there needs to be not less but rather more opportunities for sharing of the pointillistic information that will satisfy the simple mandates facility executives roll downhill to their staffs.

    The details of that granular information sharing among operational peers is the subject of other conversations, but the fact that those conversations are now getting more involvement is an indication that the Executive-to-Executive efforts are having the desired impact. What folks like your team and our team and the various groups and individuals who have comprised the information sharing activity to date need to do is continue to escalate those efforts, leveraging the seismic shifts caused by the Administration.

    Any lack of success to date is well-shared among all involved. The combined efforts of all involved have not been completely unsuccessful, though. In the twenty years since these issues first crossed my own bow there has been a consistent growth in understanding among all involved. It may have been a long, low slope for much of that time, but the curve has continued to steepen and is following a predictable path leading to a foreseeable future.

    The work that you and others have done has created the environment that leads to the President of the United States taking his recent action, an accomplishment worthy of a measurable amount of praise. Laurels are not for resting on, though. The general shape of what we all need to do going forward is clear, and a large part of it will include sharing more information with more people with more precise aim and craftsmanship.



Leave a Reply