I had the pleasure of interviewing Dan Geer on the S4x18 Main Stage for 30 minutes. He typically speaks from prepared papers, so an interview is a bit unique, and his papers provided plenty of topics and questions.
We covered a wide range of issues including:
Risk: The impact of complexity and dependencies. How redundancy can increase risk against a sentient opponent. The trade off between preventing random faults and protecting targeted faults.
The importance of eliminating silent failures. Even so far as raising the probability of failure if it eliminates or reduces silent failure.
Business risk acceptance when society would not make the same risk decision.
The need for “different” redundancy, two systems with no common mode failures. Manual is an obvious different redundancy, but can two cyber systems have no common mode failures?
The growing importance of integrity.
The value of patching or otherwise reducing vulnerabilities based on whether vulnerabilities are sparse or dense. The density of medical device vulnerabilities was discussed as an example.
Are we going to take the path of proof of correctness and rigid change control or almost constant change?
This episode was sponsored by CyberX. Founded by military cyber experts, CyberX has developed a platform that helps organizations continuously reduce ICS risk.
Check out the CyberX Global ICS and IIoT Risk Report and my podcast from last year on the report with Phil Neray.
Update: If you are watching closely, you will see we pushed it back from Feb 1st to 8th.
S4x18 was a big success, in my eyes, and a huge amount of work. A major web site update planned for December 1st shifted to January 1st and then post S4. We will have the new site up on January 31st.
This is why you are not seeing new articles here, but I’d encourage you to subscribe to the S4 Events YouTube channel and follow me on LinkedIn and Twitter to get the latest content. We are still putting out a lot, just not bothering to put it up on this site that will fade away shortly.
This episode begins with a 10 minute monologue from Dale Peterson on why demonstrations of insecure by design, no SDL and modifying physical processes is not particularly interesting for the advanced ICS security audience … and why it is still important.
Then we play Dale Peterson’s interview with Steve Bitar of ExxonMobil on the Open Process Automation (OPA) initiative.
If the OPA is new to you, you will learn how OPA is trying to get away from single vector lock in and be able to choose the best of breed and have it work together.
The more interesting part of the podcast for most will be Steve’s addressing the hard questions that everyone is thinking but don’t come up when you get the OPA presentation. For example:
Steve took on all the tough questions, because clearly they have come up behind closed doors before.
As always I welcome comments and suggestions, send them to s4@digitalbond.com.
We, the ICS community, have some mantras:
Those mantras lead to a very depressing story, and it is natural to wonder why anyone would want to be in the ICS security community and play a role in that story. Life is short, and why waste it on this doomed endeavor?
The truth is people get something out of the stories they tell themselves. I was reminded of this listening to a series of Brian Koppelman interviews with Seth Godin on The Moment podcast. Seth describes the comfort an unsuccessful screenwriter gets from the story of being a struggling artist working against a system that doesn’t appreciate his commitment to art and the need for purity in his craft.
“the comfort is fabulous because the story you get to carry around with you is bulletproof. It’s insulation. It’s the outside world doesn’t understand me. The outside world is against me. … As long as you carry that around you are safe. It has completely transferred all the responsibility to someone who is not you.”
Now think back to the story we tell ourselves in the ICS security community if we buy into those mantras:
If ICS do not get more secure it is not our fault. It’s a problem that cannot be addressed for decades, and even if we came up with solutions the company or industry won’t spend the money to fix it. Success is not possible, so failure cannot be our fault.
We are special. The knowledge and experience required to play a role in ICS security precludes anyone outside our small clique from participating. And when they try to participate they can be quickly cast aside as they are told all the reasons why their ideas will not work and their participation will actually harm the cause.
That’s a nice story, we are special and can do no wrong. Add to that all the attention the ICS security space is getting in the media and security events, and we can be even more special.
I contend these mantras, stories, and walls around ICS change are crumbling. It is still hard to see now if you look at the broad ICS community, but as I said in my optimistic S4x17 mini-keynote (see below) we are seeing the small start and non-linear growth in truly addressing the ICS security problem. Like most change it will seem impossible until all of sudden it has changed. The business drivers for changes to ICS combined with a better understanding of the long unknowingly accepted risk will break down the walls and force change.
Loyal readers may have already discarded these mantras and story, but it is so omnipresent that it is easy to fall back into this trend. I caught myself this week actually saying in a call that we are finally solving the Level 1 problem, but it will take a decade+ to solve Level 0.
If you are in the ICS security arena and believe and live those mantras, then it is time to look for another line of work. If you are wrong you will be passed by; if you are right you are wasting your talents and life on a doomed cause.
The seed for this article came from another article I’m writing about what S4 is all about, who it is for and who shouldn’t come. As you might guess, S4 is not for people that believe in the old, doomed story.
Dale Peterson interviews Rob Lee, founder and CEO of Dragos – SANS 515 Creator – former SCADA Diva – Chief FUD Debunker – …, focusing on how an asset owner should select an advanced IDS detection solution from a crowded market of 25+ new offerings.
Here is a breakdown of the episode:
3:50 What surprised Rob most about the response to Crashoverride?
8:40 What should be in place before an asset owner considers an advanced threat detection solution, and how many and what type of people are required to gain the benefits of a sophisticated detection solution.
13:30 Rob’s controversial view that their should be a separate ICS Secure Operations Center (SOC) rather than integrating it into an existing Enterprise SOC.
Then we talk about Rob’s breakdown of four different classes of ICS detection solutions
15:55 Configuration Analysis Solutions
19:15 Statistical Analysis (Modeling, Baseline, Threshold and Time) Solutions
24:50 Indicator (signatures) Solutions
30:35 Behavioral Solutions compared to other three approaches
35:50 How does an asset owner choose between the 25+ offerings?
37:40 Rob’s view that vendors in this space are startups and can’t do a good job in multiple classes. They need to focus on one class and a small number of sectors to be credible.
39:35 Depth v. Breadth and the push to please VC’s by saying you cover the entire ICS space
43:50 You got to test it
Signup for the ICS Security: Friday News & Notes email
Check Out the S4 Events YouTube Channel
Joel Langill, aka the SCADAHacker, joined me on the Unsolicited Response podcast to discuss ICSsec training and workforce development. Joel is the Director of ICS Cybersecurity at AECOM, see http://www.aecom.com/
Subscribe to the Unsolicited Response Podcast in iTunes
Loyal followers of Digital Bond content know that Joel and I don’t agree (some may say vehemently disagree) on a number of ICS and ICS security issues. Rather than rehash those arguments, I had two main goals in this episode. First, to dig into Joel’s background pre-ICS security because context is so important. We have the classic issue of a lot of heated disagreements in the ICSsec space that I believe are largely due to admirable passion and a lack of understanding of the others context.
Second, and what most of the episode is about, to get Joel’s thoughts on ICS security training and workforce development. How many people need to be trained, what type of training, lessons learned from his ~10 years of training, …
Here are some highlights and structure for the episode.
2:15 – Joel’s background in ICS/automation prior to getting involved in security.
13:05 – How Joel’s background has informed his approach to security.
16:50 – Started discussing ICSsec workforce development.
18:50 – What would be the ideal training for an ICS security professional?
23:25 – Using a pharma company as an example, how many people in that company would require ICSsec training. What type of training for what roles? .
26:05 – Where should the ICS security talent be located in a company?
27:40 – How to scale training, online training, and how he structures his online training
29:55 – Joel’s belief that an online class is more effective than an in person course and a discussion of the course Labs.
32:10 – What has Joel learned in training 500+ students.
35:20 – If 40 hours is not enough to get you where you need to be as an ICSsec professional, how do you or the market address the need for additional training? Joel notes most students did not meet the criteria/skill set to fully benefit from the class.
38:40 – The benefits of a hands on assessment to determine current skill level and needs to select required training.
41:20 How Joel’s joining AECOM will affect the SCADAhacker training.
Rebooting the Unsolicited Response Podcast was one of my 2017 goals, and I didn’t want it to be one and done. So I recorded a number of them before issuing this first episode so you can expect at least one a month. (Episode 2 is with Joel Langill, aka SCADAhacker).
If you have any suggestions for guests or topics please send them to s4@digitalbond.com.
In Episode 1 I interviewed John Matherly, the creator of Shodan, in Kuwait. Lots of good content with the breakdown of highlights and times below.
3:10 What is Shodan?
4:45 John’s background and why he started Shodan
9:10 Adding ICS to Shodan … originally John thought it was too risky
10:45 How fast he can add new ICS protocol support (less than a day)
13:00 Looking to add more support for medical devices
16:00 How are the customers using Shodan, external network monitoring is most common use case … but few ICS related customers … more ICS vendors
19:30 Does John see Shodan ever scanning an internal network?
21:00 Shodan does legitimate request scanning … a proper handshake
24:45 What does he do when someone doesn’t want Shodan to scan their address space?
27:30 What has been the industry impact of his Internet connected ICS map?
29:20 The number of Internet connected ICS has only increased since he has been tracking
32:15 The Omron example
35:00 What else are you going to do with all this data, the real value of Shodan’s database
38:15 John’s request of the ICS Community
And at the end we get a bit into the weeds about what Shodan can and should do with various ICS protocol examples.
Goal: Help Owner/Operators select the best anomaly detection solution for their ICS.
It sounds simple, but after getting numerous demos and pitches from vendors, the almost unanimous contention from each vendor was that their solution was the best. Why? Because they go deeper, understand the protocol, system or user better than the competition, who were often denigrated as smoke and mirrors. Most would follow up with a specific example of their detection capability, but none provided even the start of a way of comparing solutions short of installing multiple solutions and testing.
I describe the technical marketing to date in this space as emphatic assertion. Ours is better. No really ours is better.
At S4x17 we had two sessions on “How Deep is Your ICS Deep Packet Inspection” with the goal of getting to a consensus approach or at least examples of how to perform a product comparison. While the sessions gave good anecdotal examples, we were no closer to a methodology.
In two weeks, I’ll be at our S4xEurope event (June 1-2 in Vienna) and try again. There is a promising session from Sentryo on Detecting Cyber Attacks Through Machine Learning of Process Variable Tracking. It uses the automobile CAN network, and falls into the great examples category. I also will be moderating a panel with technical representatives from Claroty, Nozomi and Security Matters. While it’s not a hostile interview, I’ve warned the panelists that I’m going to push them on this evaluation issue and call BS if necessary. (Note: I welcome any suggested questions for the panel, and we video the event so you will get to see it late June on our S4 Events YouTube Channel)
As preparation for the panel I wanted to have a list of vendors offering what I’m calling an ICS Anomaly Detection solution and created a LinkedIn Post trying fill out my original list of 14. That list, see below, grew to 22 even with removing suggestions with the three following characteristics:
All three of these are judgment calls, and I welcome any comments where you think I’ve missed the mark.
As the LinkedIn article comments came in I decided to add two columns to the list:
So here is the list, and I expect it will require updating this week as more companies and better information comes in.
It’s hard to miss that a large number of the companies are Israeli (9 of 22), and that most have raised money in the last 12 months. However what I want to focus on is the difficult situation facing any ICS Owner/Operator who is considering buying an ICS anomaly detection solution, especially when they all are saying close to the same thing about why they are the best.
It’s not getting better, and the number of vendors offering ICS anomaly detection solutions continues to grow in numbers and angel/venture funding.
How is an asset owner to determine what anomaly detection approach, if any, is right for them?
The first decision points are simple:
After those three questions the evaluation runs into significant difficultly. I have had numerous demos, conference calls and discussions with ICS anomaly detection vendors, but I must say the arguments the vendors give as to why their solution is better is typically emphatic assertion (we support and alert on more of the protocol than the competition) and identical to what I hear from their competitors.
We tried to make progress on evaluation methodologies with two “How Deep Is Your DPI” sessions at S4x17 on Stage 2. The sessions gave good and specific examples on how ICS anomaly detection can detect cyber attacks and incidents, but really didn’t move the evaluation challenge forward much.
So we are trying a different approach at S4xEurope, June 1 – 2 in Vienna with two sessions.
First, I’ll be interviewing a panel of technical vendors on stage including Damiano Bolzoni from Security Matters, Andrea Carcano from Nozomi Networks, and a third panelist to be announced. I’m working on my pointed questions and followups in an attempt to get past the generalities, and welcome any suggestions. My focus is going to be on the evaluation criteria, and how they are using machine learning or other techniques to identify potential cyber incidents.
Second, we have a very promising session from Jean-Cristophe Testud of Sentryo Security entitled Detecting Cyber Attacks Through Machine Learning of Process Variable Tracking. Much of the work today in ICS anomaly detection is related first to communication pairs and patterns, and second to identifying high impact requests (something we did poc with the DHS funded ICS signatures in 2006).
Since this is a vendor session, we required early submission of the presentation to check for commercialism and content. It’s great. It shows modeling/learning of automobiles via CAN traffic, and detecting false data and commands. This session shows the power of the structured machine learning and also shows how a vendor could potentially provide a listing of capabilities per protocol in this area.
Recent Comments