Digital Bond

For Secure & Robust ICS

Unsolicited Response Podcast: Interview with Steve Bitar and 10-minute Rant

by Leave a Comment

This episode begins with a 10 minute monologue from Dale Peterson on why demonstrations of insecure by design, no SDL and modifying physical processes is not particularly interesting for the advanced ICS security audience … and why it is still important.

Then we play Dale Peterson’s interview with Steve Bitar of ExxonMobil on the Open Process Automation (OPA) initiative.

 

If the OPA is new to you, you will learn how OPA is trying to get away from single vector lock in and be able to choose the best of breed and have it work together.

The more interesting part of the podcast for most will be Steve’s addressing the hard questions that everyone is thinking but don’t come up when you get the OPA presentation. For example:

  • Why would a DCS vendor who has control of the account / market every want to support this?
  • Speaking of support … who would be responsible for the rigorous global support that ExxonMobil requires when the solution comes from many vendors?
  • What business model changes would be required for OPA to work?

Steve took on all the tough questions, because clearly they have come up behind closed doors before.

As always I welcome comments and suggestions, send them to s4@digitalbond.com.

Subscribe to the Unsolicited Response Podcast in iTunes

The ICS Security Stories We Tell And Love

by 1 Comment

We, the ICS community, have some mantras:

  • It will take decades to fix the ICS security problem
  • Operations Technology (OT) is different than Information Technology (IT)
  • You can’t do X, Y or Z in ICS because … which is followed by a variety of reasons such as the system can’t go down, we can’t introduce any change, it might cause a catastrophe, the vendor won’t support it, your reason here, …
  • Management will not pay to secure the ICS

Those mantras lead to a very depressing story, and it is natural to wonder why anyone would want to be in the ICS security community and play a role in that story. Life is short, and why waste it on this doomed endeavor?

The truth is people get something out of the stories they tell themselves. I was reminded of this listening to a series of Brian Koppelman interviews with Seth Godin on The Moment podcast. Seth describes the comfort an unsuccessful screenwriter gets from the story of being a struggling artist working against a system that doesn’t appreciate his commitment to art and the need for purity in his craft.

“the comfort is fabulous because the story you get to carry around with you is bulletproof. It’s insulation. It’s the outside world doesn’t understand me. The outside world is against me. …   As long as you carry that around you are safe. It has completely transferred all the responsibility to someone who is not you.”

Now think back to the story we tell ourselves in the ICS security community if we buy into those mantras:

If ICS do not get more secure it is not our fault. It’s a problem that cannot be addressed for decades, and even if we came up with solutions the company or industry won’t spend the money to fix it. Success is not possible, so failure cannot be our fault.

We are special. The knowledge and experience required to play a role in ICS security precludes anyone outside our small clique from participating. And when they try to participate they can be quickly cast aside as they are told all the reasons why their ideas will not work and their participation will actually harm the cause.

That’s a nice story, we are special and can do no wrong. Add to that all the attention the ICS security space is getting in the media and security events, and we can be even more special.

I contend these mantras, stories, and walls around ICS change are crumbling. It is still hard to see now if you look at the broad ICS community, but as I said in my optimistic S4x17 mini-keynote (see below) we are seeing the small start and non-linear growth in truly addressing the ICS security problem. Like most change it will seem impossible until all of sudden it has changed. The business drivers for changes to ICS combined with a better understanding of the long unknowingly accepted risk will break down the walls and force change.

Loyal readers may have already discarded these mantras and story, but it is so omnipresent that it is easy to fall back into this trend. I caught myself this week actually saying in a call that we are finally solving the Level 1 problem, but it will take a decade+ to solve Level 0.

If you are in the ICS security arena and believe and live those mantras, then it is time to look for another line of work. If you are wrong you will be passed by; if you are right you are wasting your talents and life on a doomed cause.

The seed for this article came from another article I’m writing about what S4 is all about, who it is for and who shouldn’t come. As you might guess, S4 is not for people that believe in the old, doomed story.

Unsolicited Response Podcast with Rob Lee

by Leave a Comment

Dale Peterson interviews Rob Lee, founder and CEO of Dragos – SANS 515 Creator – former SCADA Diva – Chief FUD Debunker – …, focusing on how an asset owner should select an advanced IDS detection solution from a crowded market of 25+ new offerings.

Here is a breakdown of the episode:

3:50 What surprised Rob most about the response to Crashoverride?

8:40 What should be in place before an asset owner considers an advanced threat detection solution, and how many and what type of people are required to gain the benefits of a sophisticated detection solution.

13:30 Rob’s controversial view that their should be a separate ICS Secure Operations Center (SOC) rather than integrating it into an existing Enterprise SOC.

Then we talk about Rob’s breakdown of four different classes of ICS detection solutions

15:55 Configuration Analysis Solutions

19:15 Statistical Analysis (Modeling, Baseline, Threshold and Time) Solutions

24:50 Indicator (signatures) Solutions

30:35 Behavioral Solutions compared to other three approaches

35:50 How does an asset owner choose between the 25+ offerings?

37:40 Rob’s view that vendors in this space are startups and can’t do a good job in multiple classes. They need to focus on one class and a small number of sectors to be credible.

39:35 Depth v. Breadth and the push to please VC’s by saying you cover the entire ICS space

43:50 You got to test it

Signup for the ICS Security: Friday News & Notes email

Check Out the S4 Events YouTube Channel

Unsolicited Response Podcast with Joel Langill

by Leave a Comment

Joel Langill, aka the SCADAHacker, joined me on the Unsolicited Response podcast to discuss ICSsec training and workforce development. Joel is the Director of ICS Cybersecurity at AECOM, see http://www.aecom.com/solutions/converged-resilience/. He also runs the popular ICS security website https://scadahacker.com/ , and details on the training he describes in the podcast is available at that site.

Subscribe to the Unsolicited Response Podcast in iTunes

Loyal followers of Digital Bond content know that Joel and I don’t agree (some may say vehemently disagree)  on a number of ICS and ICS security issues. Rather than rehash those arguments, I had two main goals in this episode. First, to dig into Joel’s background pre-ICS security because context is so important. We have the classic issue of a lot of heated disagreements in the ICSsec space that I believe are largely due to admirable passion and a lack of understanding of the others context.

Second, and what most of the episode is about, to get Joel’s thoughts on ICS security training and workforce development. How many people need to be trained, what type of training, lessons learned from his ~10 years of training, …

Here are some highlights and structure for the episode.

2:15 – Joel’s background in ICS/automation prior to getting involved in security.

13:05 – How Joel’s background has informed his approach to security.

16:50 – Started discussing ICSsec workforce development.

18:50 – What would be the ideal training for an ICS security professional?

23:25 – Using a pharma company as an example, how many people in that company would require ICSsec training. What type of training for what roles? .

26:05 – Where should the ICS security talent be located in a company?

27:40 – How to scale training, online training, and how he structures his online training

29:55 – Joel’s belief that an online class is more effective than an in person course and a discussion of the course Labs.

32:10 – What has Joel learned in training 500+ students.

35:20 – If 40 hours is not enough to get you where you need to be as an ICSsec professional, how do you or the market address the need for additional training? Joel notes most students did not meet the criteria/skill set to fully benefit from the class.

38:40 – The benefits of a hands on assessment to determine current skill level and needs to select required training.

41:20 How Joel’s joining AECOM will affect the SCADAhacker training.

Unsolicited Response Podcast Is Back … With John Matherly of Shodan

by 1 Comment

Rebooting the Unsolicited Response Podcast was one of my 2017 goals, and I didn’t want it to be one and done. So I recorded a number of them before issuing this first episode so you can expect at least one a month. (Episode 2 is with Joel Langill, aka SCADAhacker).

If you have any suggestions for guests or topics please send them to s4@digitalbond.com.

In Episode 1 I interviewed John Matherly, the creator of Shodan, in Kuwait. Lots of good content with the breakdown of highlights and times below.

3:10 What is Shodan?

4:45 John’s background and why he started Shodan

9:10 Adding ICS to Shodan … originally John  thought it was too risky

10:45 How fast he can add new ICS protocol support (less than a day)

13:00 Looking to add more support for medical devices

16:00 How are the customers using Shodan, external network monitoring is most common use case … but few ICS related customers … more ICS vendors

19:30 Does John see Shodan ever scanning an internal network?

21:00 Shodan does legitimate request scanning … a proper handshake

24:45 What does he do when someone doesn’t want Shodan to scan their address space?

27:30 What has been the industry impact of his Internet connected ICS map?

29:20 The number of Internet connected ICS has only increased since he has been tracking

32:15 The Omron example

35:00 What else are you going to do with all this data, the real value of Shodan’s database

38:15 John’s request of the ICS Community

And at the end we get a bit into the weeds about what Shodan can and should do with various ICS protocol examples.

Insanely Crowded ICS Anomaly Detection Market

by 7 Comments

Goal: Help Owner/Operators select the best anomaly detection solution for their ICS.

It sounds simple, but after getting numerous demos and pitches from vendors, the almost unanimous contention from each vendor was that their solution was the best. Why? Because they go deeper, understand the protocol, system or user better than the competition, who were often denigrated as smoke and mirrors. Most would follow up with a specific example of their detection capability, but none provided even the start of a way of comparing solutions short of installing multiple solutions and testing.

I describe the technical marketing to date in this space as emphatic assertion. Ours is better. No really ours is better.

At S4x17 we had two sessions on “How Deep is Your ICS Deep Packet Inspection” with the goal of getting to a consensus approach or at least examples of how to perform a product comparison. While the sessions gave good anecdotal examples, we were no closer to a methodology.

In two weeks, I’ll be at our S4xEurope event (June 1-2 in Vienna) and try again. There is a promising session from Sentryo on Detecting Cyber Attacks Through Machine Learning of Process Variable Tracking. It uses the automobile CAN network, and falls into the great examples category. I also will be moderating a panel with technical representatives from Claroty, Nozomi and Security Matters. While it’s not a hostile interview, I’ve warned the panelists that I’m going to push them on this evaluation issue and call BS if necessary. (Note: I welcome any suggested questions for the panel, and we video the event so you will get to see it late June on our S4 Events YouTube Channel)

As preparation for the panel I wanted to have a list of vendors offering what I’m calling an ICS Anomaly Detection solution and created a LinkedIn Post trying fill out my original list of 14. That list, see below, grew to 22 even with removing suggestions with the three following characteristics:

  1. Standard IDS/IPS Solutions – We are believers in IDS/IPS signatures, after all Digital Bond wrote the first basic ICS signatures and they are still widely used. Classic IDS/IPS solutions are not included because this new product category is focused on learning or knowing “normal” network, device, application or user activity and identifying variations from this norm that could be indicative of an attack.
  2. Perimeter Security Solutions – There are a number of ICS gateway solutions with some impressive ICS protocol intelligence or effective one-way technology. Perimeter security products are essential to an ICS security program, but not in this new anomaly detection category.
  3. Primarily IT Security Solutions – Most of the mainstream IT security products are adding some ICS intelligence, and at some point they could be competitive with ICS focused products.

All three of these are judgment calls, and I welcome any comments where you think I’ve missed the mark.

As the LinkedIn article comments came in I decided to add two columns to the list:

  • Country of Origin … this is interesting to identify where the startups are coming from and also becomes important with the increasing cyber nationalism
  • Funding … you have 20+ vendors competing for a very new, and some would say unproven, market. Having enough runway to survive until the market grows will be key, although burn rate is just as important. And yes, there will be carnage.

So here is the list, and I expect it will require updating this week as more companies and better information comes in.

It’s hard to miss that a large number of the companies are Israeli (9 of 22), and that most have raised money in the last 12 months. However what I want to focus on is the difficult situation facing any ICS Owner/Operator who is considering buying an ICS anomaly detection solution, especially when they all are saying close to the same thing about why they are the best.

Evaluating ICS Anomaly Detection Solutions

by 2 Comments

It’s not getting better, and the number of vendors offering ICS anomaly detection solutions continues to grow in numbers and angel/venture funding.

How is an asset owner to determine what anomaly detection approach, if any, is right for them?

The first decision points are simple:

  1. Are you ready for ICS anomaly detection?
    If your ICS security protection program is not mature and under control, then you’re not ready. If you are not doing basic detection, such as monitoring firewall logs and endpoint protection, you are not ready. If you don’t have the detection and incident response team to assign to anomaly detection, you are not ready.
  2. Does the ICS anomaly detection support your deployed products and protocols?
    All of the vendors clearly state what they support, but some are a bit vague on when the support will be available. The protocol work is fast and furious.
  3. Is the solution passive-only or a combination of active scanning and passive monitoring?
    I made the case for the active/passive hybrid in a recent LinkedIn article, but there are many asset owners who will only consider passive.

After those three questions the evaluation runs into significant difficultly. I have had numerous demos, conference calls and discussions with ICS anomaly detection vendors, but I must say the arguments the vendors give as to why their solution is better is typically emphatic assertion (we support and alert on more of the protocol than the competition) and identical to what I hear from their competitors.

We tried to make progress on evaluation methodologies with two “How Deep Is Your DPI” sessions at S4x17 on Stage 2. The sessions gave good and specific examples on how ICS anomaly detection can detect cyber attacks and incidents, but really didn’t move the evaluation challenge forward much.

So we are trying a different approach at S4xEurope, June 1 – 2 in Vienna with two sessions.

First, I’ll be interviewing a panel of technical vendors on stage including Damiano Bolzoni from Security Matters, Andrea Carcano from Nozomi Networks, and a third panelist to be announced. I’m working on my pointed questions and followups in an attempt to get past the generalities, and welcome any suggestions. My focus is going to be on the evaluation criteria, and how they are using machine learning or other techniques to identify potential cyber incidents.

Second, we have a very promising session from Jean-Cristophe Testud of Sentryo Security entitled Detecting Cyber Attacks Through Machine Learning of Process Variable Tracking. Much of the work today in ICS anomaly detection is related first to communication pairs and patterns, and second to identifying high impact requests (something we did poc with the DHS funded ICS signatures in 2006).

Since this is a vendor session, we required early submission of the presentation to check for commercialism and content. It’s great. It shows modeling/learning of automobiles via CAN traffic, and detecting false data and commands. This session shows the power of the structured machine learning and also shows how a vendor could potentially provide a listing of capabilities per protocol in this area.

RSA Conference Report

by 4 Comments

ICS Industry Pioneer and Expert Eric Byres of ICS-Secure reports on the RSA Conference last week.

I just returned from the RSA Conference 2017 in San Francisco, after a five year hiatus. If you are not familiar with the RSA Conferences, they are one of the largest cyber security events in the world, with a reported 40,000 attendees last year.

The last time I was at the RSA Conference, I didn’t think anyone there was taking ICS security seriously, so I decided to stop attending the event. This time I was hoping that with the interest in the Industrial Internet of Things (IIoT) and all the news of proven cyber impacts on critical infrastructures (like those in the Ukraine), things would be different.

Sadly nothing has changed.  The RSA Conference (RSAC) is still pretty much a waste of time for anyone concerned with either IIoT or ICS security.

RSAC 2017 looked promising at the start. Mike Assante was on the annual keynote panel “The Seven Most Dangerous New Attack Techniques, and What’s Coming Next”. You can see video of the talk here: https://youtu.be/45_ciRquXBE

But it went down hill after that with only one mainstream talk on ICS security. No tracks, no panels, no sessions… And while I didn’t attend that one talk*, the description told me enough:

IoT and SCADA: Lessons Learned and Case Studies    

Law | Mobile & IoT Security | Technology Infrastructure & Operations | Peer2Peer

This session will review key lessons learned about SCADA and IoT breaches and attacks such as Stuxnet and Mirai. We will look at the consequences of SCADA breaches and potential legal fallout, analyze two case studies, and discuss best legal and security practices. Case studies will feature two different potential attackers: a hostile nation state and aggrieved employees.

Facilitator: Lawrence Dietz, General Counsel and Managing Director of Information Security, TAL Global Corporation

Stuxnet?? Haven’t we heard enough of that old worm? Has nothing else happened since 2010? And why is this talk in a law session run by a lawyer? I’m sure that Mr. Dietz is a fine lawyer, but if he is the only ICS expert that RSAC can find, what are they saying? Maybe the message is “Time to lawyer up boys, ‘cause there’s nothing else we can do to secure ICS”.

In fairness there were four decent ICS sessions in the RSA Sandbox, all presented by familiar faces like Clint Bodungen, Andrey Nikishin and Tom VanNorman. Unfortunately the RSA SandBox is advertised as “Full of hands-on interactive experiences to test your infosec skills…This year the Sandbox opens with RSAC’s third annual craft beer tasting event, CyBEER Ops”.

Perhaps RSAC should add; “Hey all you ICS kids… come play in the sand box where you can’t hurt yourself. Just remember don’t run and don’t throw sand at each other”. I’m sorry, but the term “sandbox” and the setup of the sandbox area just did not instill any feeling that RSAC managment team thinks ICS security is important.

On the show floor the situation was equally sad. Out of the several hundred booths I visited, I only found four with staff that could talk intelligently about ICS or IIoT issues. Now I’m sure I missed a few booths, but RSAC didn’t make it easy to find ICS security vendors. On the RSA 2017 web site there are over 100 possible search keywords and 20 core topics, covering everything from ”Access Control” and “Anti-Spam” to “Zero Day Vulnerability”. Everything that is except the terms ICS, SCADA, or IIoT security. Those terms were conspicuously absent from the 120 search choices RSAC offered.

So would I recommend the RSA Conference to the ICS security community? Maybe if you want to meet up with colleagues – I had many productive face-to-face meetings with clients, potential partners and old friends. But if you want to see new ICS security technologies or listen to talks on the state of the art in IIoT security, go somewhere else. Dale Peterson’s S4 events  the SANS ICS Summit,  the ICSJWG meetings and the ARC Forum are far better ways to spend your time and money. And you can still meet your friends there too.

Will I be back in 2018? Maybe, but only for the face-to-face meetings. Instead I will be heading to the SANS ICS Summit in Orlando, March 19-21. Hope to see you there.

*In the interests of full disclosure, I didn’t buy a full conference pass, so there might have been things I missed – but I doubt it. And I didn’t manage to visit every booth in the show.  With over 650 exhibitors this year, I doubt anyone did. But I did struggle through every line of the exhibitors list and didn’t see any ICS-related vendors, except arguably my old friends at Belden. Unfortunately they were advertised only as TripWire, with no mention of ICS security in their show description.

Attack On Ukraine Power Grid Added To S4x17 Agenda

by Leave a Comment

Learn More and Register For S4x17, Jan 10-12 in Miami South Beach

We have learned in recent years to leave a slot or two for late breaking attacks on ICS or hot research in the S4 agenda. Ukraine has helped fill this spot now for the second year in a row. We know that something happened again to the Ukrainian Power Grid, and there is still much that is not known or not yet public as researchers and analysts are once again working hard over the holidays.

So who is best to put on stage to reveal and discuss the latest information and analysis?

Answer: The people closest to the information and problem/challenge.

So we will have Marina Krotofil, who hails from Ukraine and now is working for Honeywell, on stage and
Oleksiy Yasinskiy from ISSP in Ukraine on a live video feed. Marina and Oleksiy may choose to add additional people on stage or via video from Ukraine.

I’d like to be able to give you more of a description or feel to what you will learn, but it likely would be out of date in the next day or two. What I can say is you will get the latest and most detailed information known on January 10th.

Image by Oran Viriyincy

Secure ICS Protocols at S4

by Leave a Comment

2016 was a turning point with secure ICS protocols. For a while it was limited primarily to OPC UA and DNP3 SA, but 2016 brought us a secure version of CIP / Ethernet/IP, Secure Modbus and a couple of others that will soon be unveiled. This should be enough critical mass to force the other protocol bodies to do the same in 2017 – 2018.

We have two secure ICS protocol sessions at S4x17:

Secure Modbus with Role Based Authorization with Daniel Clarke

Schneider Electric has developed a Secure Modbus protocol that they are proposing to the Modbus organization. It will support authentication and encryption of course, and Daniel will explain how. What I found most interesting is the use of certificates to enforce roles at the PLC/RTU itself. This delves into a PKI which can be a morass. So I’m looking forward to hearing how this will be implemented and managed.

Secure SCADA Protocol for the 21st Century (SSP21) with Adam Crain and Rich Corrigan

After beating on DNP3 and other protocol applications as part of Project Robus, Adam decided to work with Rich to come up with a more secure protocol. SSP21 is intended to fill a technology gap where existing technologies like TLS are not applicable, namely for serial communication channels and endpoints with limited bandwidth and/or processing capabilities

FYI … At S4xEurope we had a panel discussion with four major PLC vendors on the NextGen Secure PLC.

About Us

Digital Bond was founded in 1998 and performed our first control system security assessment in the year 2000. Over the last sixteen years we have helped many asset owners and vendors improve the security and reliability of their ICS, and our S4 events are an opportunity for technical experts and thought leaders to connect and move the ICS community forward.

Recent Comments