INL ICS SecurityLet’s take a closer look at DHS since this is the week of DHS’s ICSJWG Spring Conference. Like many, I’m guilty of treating ICS-CERT as if they are THE DHS sponsored organization responsible for ICS security in the US Government. ICS-CERT is part of the DHS Control System Security Program (CSSP) and should be treated and evaluated as a CERT.

ICS-CERT does a fine job coordinating activities between researchers and companies. They are balanced and try to reach a compromise that satisfies both parties. It is a huge benefit for a researcher to be able to turn over findings to ICS-CERT and let them deal with the coordination. Just ask McCorkle and Rios who turned over a huge amount of HMI vulns. There have also been many cases where ICS-CERT knocking on a vendor’s door has gotten a response after the researcher was ignored.

While ICS-CERT has done well on coordination when the researcher and company cooperate, their products, alerts and advisories, are based and biased towards whatever the vendor has admitted or released. It’s not surprising that the vendor panel at ICSJWG had high praise for ICS-CERT. It’s also not surprising they support the vendor’s point of view since many are customers paying INL top dollar for ICS security services.

ICS-CERT has failed to use their ICS expertise or wealth of lab equipment in the ICS-CERT Alerts and Advisories. They have been a clipping service, reporting whatever information others have chose to make public and no more. The best example of this is the Beresford vulnerabilities where ICS-CERT had the Siemens equipment, must have known Dillon was right, and still went with the Siemens party line until it was no longer tenable. That ICS-CERT did not suffer a massive black eye when they completely missed the PLC attack portion of Stuxnet is still baffling. It’s so easy to get me ranting on this topic … and the point is that a CERT is just a portion of what DHS is responsible for in ICS security.

While ICS-CERT ≠ DHS CSSP, it is credible to say that ICS resources at Idaho National Labs (INL) is DHS CSSP. Marty Edwards, the DHS Director of CSSP was formerly in a similar role at INL. Marty still lives in Idaho and has his office at INL. ICS-CERT resources are from INL. The DHS CSSP program office has been staffed by INL contractors. The DHS training courses were developed and are delivered by INL. The fly away teams come from INL. It goes on and on.

PNNL and Sandia play bit roles in DHS ICS security compared with INL, and actual DHS employees unconnected to INL are anomalies and tend to only last a year or two.

A fair characterization is DHS has outsourced ICS security to INL.

Many people mistakenly believe that national labs and non-profits operate for the public good and not as businesses. Most try to maximize revenue like other companies and find ways to allocate and spend that money so it is not considered profit. The national labs are no different. In fact the rates at INL for things like training and support are much higher than commercial industry, both small and large commercial organizations, because the labs have played the game for many years and know how to establish and support huge rates without making a “profit”.

Sometime buy a manager at a national lab a beer and ask them how the operating company, Battelle Energy Alliance (BEA) for INL, makes money. You will hear a load of stories about all the tricks and restrictions preventing reasonable use of knowledge and resources available from the lab. They are a business, and the INL 10-year management contract was valued at $5B. This is not bad, but thinking of them as an altruistic organization is deeply flawed. In fact, INL has more, and gets away with more, conflicts of interest than any other organization in the ICS security space.

There is ICS security talent at INL. This is not the issue. They should be a resource available to the US Government, vendors, and owner/operators can consider to help with ICS security. INL shouldn’t be the DHS CSSP.

DHS is now almost 10 years old. Was the expectation of Congress and the various Administrations that DHS would outsource critical infrastructure ICS security? Is this going to continue? Should this continue based on the progress over the last ten years?

14 comments to ICS-CERT ≠ DHS CSSP; INL = DHS CSSP

  • Bryan Owen

    Dale, hope you don’t mind a contrary view.

    Other than collaboration from within our software supply chain, specifically Microsoft, there isn’t any other organization responsible for as much positive impact on our cyber security initiatives as INL.

    I am very appreciative and respectful of the lasting partnership and trust. In my opinion both DoE and DHS should be commended for this program.

  • Dale G Peterson

    Bryan, we welcome all views.

    I read you saying that OSIsoft hired INL for their ICS security expertise, and it has been very beneficial for OSIsoft products and OSIsoft customers. I noted that INL has talent, and I’m not surprised at the positive results of assessments of vendor products if the vendor chooses to follow their recommendations.

    My point on INL is different. It’s not questioning their skill. It’s questioning their effectiveness. And more importantly it’s questioning whether DHS should outsource the ICS security program to INL. How would you feel if they outsourced this to Lockheed or Raytheon? Lockheed, Raytheon, INL, and for that matter, Digital Bond, all have commercial interests that supersede the mission of DHS.


  • anonymous

    Good article – interesting to say that the three companies mentioned all have a CTF wargame winning teams (INL = Acme Pharm ( with some IOActive), Raytheon (Hates Irony), and Lockheed (b33rM3).

    So talent of any the three wouldn’t be in question what is in question is their objectivity.

  • Meh

    Interesting in that while the majority of the ICS-CERT, and most of the CSSP functions are outsourced to INL, there was NEVER a bidding process for that. Since INL was already the DHS CSSP program steward, and already had the money from DHS, it was the easy route, versus putting out an RFI/RFP and having a competition, which would benefit everyone. That makes those connections between INL and DHS all the more interesting. Yes, the lab does do good work in this space, but most of it can be done by quite a few other vendors. There are very few things that make the lab unique from a control systems standpoint.

  • Bryan Owen

    On the contrary industry could be staring down another lost decade had DoE and DHS not bootstrapped the program at INL.

    The DigitalBond DoE research is another example of positive contribution. I’m certain you don’t mean to discount other programs that have been successful.

    That said effectiveness and efficiency are needed for any kind of sustainable cyber initiative. Furthermore, paraphrasing Mark Weatherford, Cyber IQ does not appear overnight.

  • Kizzer

    Well, they were pretty useful for me during the Curran-Gardner Public Water District hacking farce around last (US) Thanksgiving (ICSB-11-327-01). As for the news clipping stuff, well, who doesn’t do that these days? My inbox is typically pretty full of these. At least their’s is solicited…

  • Alan Morris

    Today there was stated on your website that DHS has no new approaches to ICS defense. We have emailed our new approach, being technology, not software, to everyone in DHS who has “cyber” associated with his or her name or title, with nary a response. Also INL, and NSA. The cyber titles would fill the numbers of a battalion.

  • Alan Morris

    Today there was stated on your website that DHS has no new approaches to ICS defense. We have emailed our new approach, being technology, not software, to everyone in DHS who has “cyber” associated with his or her name or title, with nary a response. Also INL, and NSA. The cyber titles would fill the numbers of a battalion.

    We never said this on your comment.

  • anonymous

    You probably could extend you equation to:

    DHS CSSP = INL = Lofty Perch

    To the best of my knowledge Lofty Perch personnel are the sole instructors for the DHS CSSP free training classes. Just last week, the Introduction to Control Systems Cybersecurity (101) course had about 100 students all being instructed by a Lofty Perch instructor while the DHS CSSP = INL person sat in the back and watched.

    Don’t get me wrong. The Lofty Perch instructor was excellent. But why/who decided that Lofty should not only get the contract but the unbelievable promotion they get by teaching all the DHS CSSP classes. And why, after all these years, isn’t the DHS CSSP / INL guy capable of teaching the Introductory course? Does this really have to be outsourced?

    At what point will the DHS CSSP free training be shutdown and allow private industry offer training? There are plenty of private sector control system security courses but they all have to compete with FREE training offered by DHS. That’s pretty hard to compete with. Plus, DHS seems to have an unlimited marketing budget to show up at every tradeshow, symposium and user conference to offer their FREE training. Of course, I’m sure for every student trained a good chunk of US taxpayer dollars get sent to INL and Lofty. And many students aren’t even US citizens!!! DHS goes out of their way to invite international students to their free training. Why? Shouldn’t we at least charge international students?

    BTW – Do people know that Marty Edwards, head of DHS CSSP, is Canadian? So is Lofty Perch. Were there really no Americans qualified for the job or American security companies available to contract to?

    Sorry for the rant but I’m glad people are finally starting to talk about the unfair practices of the DHS CSSP to funnel money to their cronies in INL and Lofty Perch. The situation is really sickening.

  • Meh

    Interesting to note that the Lofty contract is sole source. It does not go out for competition. At one time DHS talked about getting out of the training business and letting industry take it over. That’s fine, but it’s not in the best interest of INL to do that. INL makes money by providing the training. There is nothing in any of the training that industry couldn’t do. Heck, extend that, and there’s nothing with the control system pentests (assessments test as the lab calls them, they aren’t allowed to call them pentests) that the lab does that industry couldn’t do. It is impossible to compete with the lab when the training, and the pentests are subsidized by the gov.

    Regarding training foreign students, International Training is going on this week! So, the training is free for international students as well. I could see training some of our allies, good will and all. But I know there are folks from many countries other than allies at the training, and in fact some of them could be considered outright adversaries. Again, all funded by the US taxpayer.

    (Marty is a US citizen btw).

  • Dale G Peterson

    Agreed. Mark Fabro does a great job teaching the DHS/INL courses. He knows the topic well and is very entertaining.

    The training is the most blatant area where INL has been and is competing with industry like Jonathan Pollet/Red Tiger, SCADAhacker, Infosec Institute, SANS and even Digital Bond from time to time. The fact that INL hires an outsider a private consultant to perform the training is clear evidence that this is something industry can do.

    I’m a bit uncomfortable with the non-US / Canada comments, but we allow on topic comments here.

    Marty is a very nice guy who knows the ICS security space. As soon as DHS selected him for the post, it was likely the already tight linkage between INL and DHS CSSP would only increase. Again it is not a question of whether INL has talent or well intentioned people, they do. For whatever reason or reasons, they have been unable to perform when it has mattered most.

    Bryan Owen raises the interesting question if momentum on solving the ICS security problem would be lost if DHS moved to someone else besides INL. Perhaps, but in my opinion INL has made so little progress in pushing this issue that a different approach needs to be tried. This doesn’t mean that INL needs to abandon the space. They can continue to perform their services for vendors that value and pay for this work.

    In a way it is similar to the PLC security problem. As a community we made virtually no progress because we never started. It was a hard problem that would take years so we never started. Now are we saying we can’t switch from INL because it would take years for some other group or groups to ramp up?


  • Anonymous

    I’d like to retract the comment about Marty. Someone once told me that he was Canadian. I’m sure he is a US citizen now. My point was that DHS is inappropriately competing with private industry by offering free training and that they follow non-competitive practices when outsourcing the work.

  • Anonymous

    Now they are offering 3 days of training in Atlanta – All at no cost. Why doesn’t DHS just start operating all of the critical infrastructure in the US, and developing their own control systems and providing free system integration services. Who needs private industry?


Leave a Reply