Unsolicited Response Podcast Is Back … With John Matherly of Shodan

May 23, 2017 by Dale Peterson 1 Comment

Rebooting the Unsolicited Response Podcast was one of my 2017 goals, and I didn’t want it to be one and done. So I recorded a number of them before issuing this first episode so you can expect at least one a month. (Episode 2 is with Joel Langill, aka SCADAhacker).

If you have any suggestions for guests or topics please send them to s4@digitalbond.com.

In Episode 1 I interviewed John Matherly, the creator of Shodan, in Kuwait. Lots of good content with the breakdown of highlights and times below.

3:10 What is Shodan?

4:45 John’s background and why he started Shodan

9:10 Adding ICS to Shodan … originally John thought it was too risky

10:45 How fast he can add new ICS protocol support (less than a day)

13:00 Looking to add more support for medical devices

16:00 How are the customers using Shodan, external network monitoring is most common use case … but few ICS related customers … more ICS vendors

19:30 Does John see Shodan ever scanning an internal network?

21:00 Shodan does legitimate request scanning … a proper handshake

24:45 What does he do when someone doesn’t want Shodan to scan their address space?

27:30 What has been the industry impact of his Internet connected ICS map?

29:20 The number of Internet connected ICS has only increased since he has been tracking

32:15 The Omron example

35:00 What else are you going to do with all this data, the real value of Shodan’s database

38:15 John’s request of the ICS Community

And at the end we get a bit into the weeds about what Shodan can and should do with various ICS protocol examples.

Insanely Crowded ICS Anomaly Detection Market

May 22, 2017 by Dale Peterson 7 Comments

Goal: Help Owner/Operators select the best anomaly detection solution for their ICS.

It sounds simple, but after getting numerous demos and pitches from vendors, the almost unanimous contention from each vendor was that their solution was the best. Why? Because they go deeper, understand the protocol, system or user better than the competition, who were often denigrated as smoke and mirrors. Most would follow up with a specific example of their detection capability, but none provided even the start of a way of comparing solutions short of installing multiple solutions and testing.

I describe the technical marketing to date in this space as emphatic assertion. Ours is better. No really ours is better.

At S4x17 we had two sessions on “How Deep is Your ICS Deep Packet Inspection” with the goal of getting to a consensus approach or at least examples of how to perform a product comparison. While the sessions gave good anecdotal examples, we were no closer to a methodology.

In two weeks, I’ll be at our S4xEurope event (June 1-2 in Vienna) and try again. There is a promising session from Sentryo on Detecting Cyber Attacks Through Machine Learning of Process Variable Tracking. It uses the automobile CAN network, and falls into the great examples category. I also will be moderating a panel with technical representatives from Claroty, Nozomi and Security Matters. While it’s not a hostile interview, I’ve warned the panelists that I’m going to push them on this evaluation issue and call BS if necessary. (Note: I welcome any suggested questions for the panel, and we video the event so you will get to see it late June on our S4 Events YouTube Channel)

As preparation for the panel I wanted to have a list of vendors offering what I’m calling an ICS Anomaly Detection solution and created a LinkedIn Post trying fill out my original list of 14. That list, see below, grew to 22 even with removing suggestions with the three following characteristics:

Standard IDS/IPS Solutions – We are believers in IDS/IPS signatures, after all Digital Bond wrote the first basic ICS signatures and they are still widely used. Classic IDS/IPS solutions are not included because this new product category is focused on learning or knowing “normal” network, device, application or user activity and identifying variations from this norm that could be indicative of an attack.
Perimeter Security Solutions – There are a number of ICS gateway solutions with some impressive ICS protocol intelligence or effective one-way technology. Perimeter security products are essential to an ICS security program, but not in this new anomaly detection category.
Primarily IT Security Solutions – Most of the mainstream IT security products are adding some ICS intelligence, and at some point they could be competitive with ICS focused products.

All three of these are judgment calls, and I welcome any comments where you think I’ve missed the mark.

As the LinkedIn article comments came in I decided to add two columns to the list:

Country of Origin … this is interesting to identify where the startups are coming from and also becomes important with the increasing cyber nationalism
Funding … you have 20+ vendors competing for a very new, and some would say unproven, market. Having enough runway to survive until the market grows will be key, although burn rate is just as important. And yes, there will be carnage.

So here is the list, and I expect it will require updating this week as more companies and better information comes in.

It’s hard to miss that a large number of the companies are Israeli (9 of 22), and that most have raised money in the last 12 months. However what I want to focus on is the difficult situation facing any ICS Owner/Operator who is considering buying an ICS anomaly detection solution, especially when they all are saying close to the same thing about why they are the best.

Evaluating ICS Anomaly Detection Solutions

May 8, 2017 by Dale Peterson 2 Comments

It’s not getting better, and the number of vendors offering ICS anomaly detection solutions continues to grow in numbers and angel/venture funding.

How is an asset owner to determine what anomaly detection approach, if any, is right for them?

The first decision points are simple:

Are you ready for ICS anomaly detection?
If your ICS security protection program is not mature and under control, then you’re not ready. If you are not doing basic detection, such as monitoring firewall logs and endpoint protection, you are not ready. If you don’t have the detection and incident response team to assign to anomaly detection, you are not ready.
Does the ICS anomaly detection support your deployed products and protocols?
All of the vendors clearly state what they support, but some are a bit vague on when the support will be available. The protocol work is fast and furious.
Is the solution passive-only or a combination of active scanning and passive monitoring?
I made the case for the active/passive hybrid in a recent LinkedIn article, but there are many asset owners who will only consider passive.

After those three questions the evaluation runs into significant difficultly. I have had numerous demos, conference calls and discussions with ICS anomaly detection vendors, but I must say the arguments the vendors give as to why their solution is better is typically emphatic assertion (we support and alert on more of the protocol than the competition) and identical to what I hear from their competitors.

We tried to make progress on evaluation methodologies with two “How Deep Is Your DPI” sessions at S4x17 on Stage 2. The sessions gave good and specific examples on how ICS anomaly detection can detect cyber attacks and incidents, but really didn’t move the evaluation challenge forward much.

So we are trying a different approach at S4xEurope, June 1 – 2 in Vienna with two sessions.

First, I’ll be interviewing a panel of technical vendors on stage including Damiano Bolzoni from Security Matters, Andrea Carcano from Nozomi Networks, and a third panelist to be announced. I’m working on my pointed questions and followups in an attempt to get past the generalities, and welcome any suggestions. My focus is going to be on the evaluation criteria, and how they are using machine learning or other techniques to identify potential cyber incidents.

Second, we have a very promising session from Jean-Cristophe Testud of Sentryo Security entitled Detecting Cyber Attacks Through Machine Learning of Process Variable Tracking. Much of the work today in ICS anomaly detection is related first to communication pairs and patterns, and second to identifying high impact requests (something we did poc with the DHS funded ICS signatures in 2006).

Since this is a vendor session, we required early submission of the presentation to check for commercialism and content. It’s great. It shows modeling/learning of automobiles via CAN traffic, and detecting false data and commands. This session shows the power of the structured machine learning and also shows how a vendor could potentially provide a listing of capabilities per protocol in this area.

RSA Conference Report

February 20, 2017 by Dale Peterson 4 Comments

ICS Industry Pioneer and Expert Eric Byres of ICS-Secure reports on the RSA Conference last week.

I just returned from the RSA Conference 2017 in San Francisco, after a five year hiatus. If you are not familiar with the RSA Conferences, they are one of the largest cyber security events in the world, with a reported 40,000 attendees last year.

The last time I was at the RSA Conference, I didn’t think anyone there was taking ICS security seriously, so I decided to stop attending the event. This time I was hoping that with the interest in the Industrial Internet of Things (IIoT) and all the news of proven cyber impacts on critical infrastructures (like those in the Ukraine), things would be different.

Sadly nothing has changed. The RSA Conference (RSAC) is still pretty much a waste of time for anyone concerned with either IIoT or ICS security.

RSAC 2017 looked promising at the start. Mike Assante was on the annual keynote panel “The Seven Most Dangerous New Attack Techniques, and What’s Coming Next”. You can see video of the talk here: https://youtu.be/45_ciRquXBE

But it went down hill after that with only one mainstream talk on ICS security. No tracks, no panels, no sessions… And while I didn’t attend that one talk*, the description told me enough:

IoT and SCADA: Lessons Learned and Case Studies

Law | Mobile & IoT Security | Technology Infrastructure & Operations | Peer2Peer

This session will review key lessons learned about SCADA and IoT breaches and attacks such as Stuxnet and Mirai. We will look at the consequences of SCADA breaches and potential legal fallout, analyze two case studies, and discuss best legal and security practices. Case studies will feature two different potential attackers: a hostile nation state and aggrieved employees.

Facilitator: Lawrence Dietz, General Counsel and Managing Director of Information Security, TAL Global Corporation

Stuxnet?? Haven’t we heard enough of that old worm? Has nothing else happened since 2010? And why is this talk in a law session run by a lawyer? I’m sure that Mr. Dietz is a fine lawyer, but if he is the only ICS expert that RSAC can find, what are they saying? Maybe the message is “Time to lawyer up boys, ‘cause there’s nothing else we can do to secure ICS”.

In fairness there were four decent ICS sessions in the RSA Sandbox, all presented by familiar faces like Clint Bodungen, Andrey Nikishin and Tom VanNorman. Unfortunately the RSA SandBox is advertised as “Full of hands-on interactive experiences to test your infosec skills…This year the Sandbox opens with RSAC’s third annual craft beer tasting event, CyBEER Ops”.

Perhaps RSAC should add; “Hey all you ICS kids… come play in the sand box where you can’t hurt yourself. Just remember don’t run and don’t throw sand at each other”. I’m sorry, but the term “sandbox” and the setup of the sandbox area just did not instill any feeling that RSAC managment team thinks ICS security is important.

On the show floor the situation was equally sad. Out of the several hundred booths I visited, I only found four with staff that could talk intelligently about ICS or IIoT issues. Now I’m sure I missed a few booths, but RSAC didn’t make it easy to find ICS security vendors. On the RSA 2017 web site there are over 100 possible search keywords and 20 core topics, covering everything from ”Access Control” and “Anti-Spam” to “Zero Day Vulnerability”. Everything that is except the terms ICS, SCADA, or IIoT security. Those terms were conspicuously absent from the 120 search choices RSAC offered.

So would I recommend the RSA Conference to the ICS security community? Maybe if you want to meet up with colleagues – I had many productive face-to-face meetings with clients, potential partners and old friends. But if you want to see new ICS security technologies or listen to talks on the state of the art in IIoT security, go somewhere else. Dale Peterson’s S4 events the SANS ICS Summit, the ICSJWG meetings and the ARC Forum are far better ways to spend your time and money. And you can still meet your friends there too.

Will I be back in 2018? Maybe, but only for the face-to-face meetings. Instead I will be heading to the SANS ICS Summit in Orlando, March 19-21. Hope to see you there.

*In the interests of full disclosure, I didn’t buy a full conference pass, so there might have been things I missed – but I doubt it. And I didn’t manage to visit every booth in the show. With over 650 exhibitors this year, I doubt anyone did. But I did struggle through every line of the exhibitors list and didn’t see any ICS-related vendors, except arguably my old friends at Belden. Unfortunately they were advertised only as TripWire, with no mention of ICS security in their show description.

Attack On Ukraine Power Grid Added To S4x17 Agenda

December 28, 2016 by Dale Peterson Leave a Comment

Learn More and Register For S4x17, Jan 10-12 in Miami South Beach

We have learned in recent years to leave a slot or two for late breaking attacks on ICS or hot research in the S4 agenda. Ukraine has helped fill this spot now for the second year in a row. We know that something happened again to the Ukrainian Power Grid, and there is still much that is not known or not yet public as researchers and analysts are once again working hard over the holidays.

So who is best to put on stage to reveal and discuss the latest information and analysis?

Answer: The people closest to the information and problem/challenge.

So we will have Marina Krotofil, who hails from Ukraine and now is working for Honeywell, on stage and
Oleksiy Yasinskiy from ISSP in Ukraine on a live video feed. Marina and Oleksiy may choose to add additional people on stage or via video from Ukraine.

I’d like to be able to give you more of a description or feel to what you will learn, but it likely would be out of date in the next day or two. What I can say is you will get the latest and most detailed information known on January 10th.

Image by Oran Viriyincy

Secure ICS Protocols at S4

December 12, 2016 by Dale Peterson Leave a Comment

2016 was a turning point with secure ICS protocols. For a while it was limited primarily to OPC UA and DNP3 SA, but 2016 brought us a secure version of CIP / Ethernet/IP, Secure Modbus and a couple of others that will soon be unveiled. This should be enough critical mass to force the other protocol bodies to do the same in 2017 – 2018.

We have two secure ICS protocol sessions at S4x17:

Secure Modbus with Role Based Authorization with Daniel Clarke

Schneider Electric has developed a Secure Modbus protocol that they are proposing to the Modbus organization. It will support authentication and encryption of course, and Daniel will explain how. What I found most interesting is the use of certificates to enforce roles at the PLC/RTU itself. This delves into a PKI which can be a morass. So I’m looking forward to hearing how this will be implemented and managed.

Secure SCADA Protocol for the 21st Century (SSP21) with Adam Crain and Rich Corrigan

After beating on DNP3 and other protocol applications as part of Project Robus, Adam decided to work with Rich to come up with a more secure protocol. SSP21 is intended to fill a technology gap where existing technologies like TLS are not applicable, namely for serial communication channels and endpoints with limited bandwidth and/or processing capabilities

FYI … At S4xEurope we had a panel discussion with four major PLC vendors on the NextGen Secure PLC.

More S4 CTF Tips and Info

December 7, 2016 by Dale Peterson Leave a Comment

Register for S4x17 now! Ticket Block 151 – 200 on sale now for $1,395.

First – Reid provided me with the official Killer Robots, Inc logo.

Second – My thoughts on who should consider participating in the S4 ICS CTF.

A person with hacking skills, but little experience in ICS. The flags will give you guidance on what an attacker would actually try to do once they can get to an ICS.
A person responsible for defending an ICS. Even if you just spend time understanding the flags you will learn many of the end goals and techniques that will be used against your ICS if an attacker can gain access to it.
A person with great ICS hacking skills. You will find this a challenge and perhaps you can win the S4 Black Badge.

Third – Some tips from Reid for CTF participants:

A successful team will need a variety of skills, including the ability to analyze industrial controls, to basic network scanning, to lockpicking, as well as solving more traditional CTF problems.

Some challenges are purely control systems focused, such as identifying configuration items in controllers or analyzing oddities in ICS protocols. Some of these control systems challenges will have a cyberphysical element — as teams solve the problems, they may want to watch process control equipment to see how their finding helped attack a process. A few of these will involve ICS Foreverday vulnerabilities.

Other challenges involve incident response: analyzing traffic from compromised systems. Bring your traffic analysis hats for these. We even have RF analysis flags. We will have a handful of SDR receivers and will provide hints for how to search for these flags; players want to familiarize themselves with the RTL-SDR prior to coming.

There will also be lockpicking challenges. We will have a few lockpick sets onsite as well as a practice lock. Players might want to see this tweet ( https://twitter.com/S4CTF/status/796779068628828160 ) for a hint about one of the lock challenges.

Finally there will be a few more traditional CTF challenges that have a basis on real ICS implementation issues. For example, custom cryptographic implementations that have some interesting issues.

Ransomware Hitting ICS

December 7, 2016 by Dale Peterson 1 Comment

There are two sessions at S4x17, Jan 10-12 in Miami South Beach, covering actual ransomware incidents in ICS. Marcelo Branquinho of TI Safe will go over two case studies that occurred in South America on the Main Stage, and RSA will discuss an ICS ransomware case in the US that also involved the FBI. All three cases will be anonymized, but there is some very interesting detail on how the companies dealt with the incidents.

This article comes on the heals of a ransomware incident on San Francisco’s Muni Train and Bus ticketing system, and likely a large number of other ransomware attacks that are never made public. I don’t think it is a bold prediction that ransomware in ICS will increase.

Given that change is minimal in ICS, even a quarterly high confidence, off network backup is likely to be sufficient for recovery without unacceptable loss of information. High confidence and off network are key. We often find in assessments that the hot standby system is used as the “backup”, and interview and inspection shows more of an occasional good effort backup spread over servers, laptops and USB drives.

The bigger issue with ransomware in an ICS may be around the time to recover and the confidence in the ability to recover. Is the Recovery Time Objective (RTO) truly an acceptable outage time and is the asset owner certain it can be met? This also has ramifications for the attacker. They will need to shorten the time they give for payment, which means the asset owner will have a shorter time to decide to pay or not … another good scenario for a tabletop incident response exercise.

Should be two interesting sessions and lots of good discussion at S4x17.

Image by portal gda

Developing Next Generation of ICS Security Talent

December 5, 2016 by Dale Peterson Leave a Comment

We wanted to do it at S4x16, but couldn’t get it done. It’s going to happen at S4x17.

A South Florida High School Class will go through two days of hands on automation and security training with Matthew Luallen and the CybatiWorks kit, and then 12 of the students and their teacher will come to the Main Stage on Thursday to discuss the experience. They will hang around at lunchtime if you want to meet and talk to them.

Matthew Luallen deserves big thanks for first putting together the CybatiWorks kit and program, second working with the school we connected him with in Palm Beach, and third volunteering his time to perform the training. Digital Bond is purchasing six of the CybatiWorks kits for the course and will donate them to the high school after S4.

Our hope is that some of the S4x17 audience will be inspired by this and look for ways to improve on this effort and potentially develop something that is scalable. One of the larger challenges is if something like this is successful, meaning students learn from it and get excited about doing more of this type of learning and work, how do you make something like this available to 1,000 students? 1,000 students in each state? It’s more than just raising the money to buy the kits. It’s training and supporting the teachers. Developing courseware and likely a host of other items.

Marc Blackmer of Cisco, a Cabana Session Sponsor, is giving another important and related talk on the Sponsor Stage entitled: Mentoring for Fun and Non-Profit. He will talk about his experience in creating 1NTERRUPT, a free, non-profit cyber security program for students ages 14-22 that emphasizes creativity, community, and meritocracy. 1NTERRUPT is based in Worcester, MA, and now has chapters in San Francisco, Atlanta, and Portland.

Killer Robots, Inc. at S4xCTF

December 2, 2016 by Dale Peterson Leave a Comment

OSIsoft is back again as a S4xCTF sponsor, and they are bringing back Killer Robots, Inc. with new and unsolved flags from last year. Enter Harry Paul of OSIsoft to give you some information and hints to help you get some of the PI System related flags in the S4x17 CTF.

The S4x17 Killer Robots CTF environment is designed to be an interactive, fun source of industrial security challenges. After all, CTF is a great way to explore and defeat ‘forever’ day configuration issues. This year the OSIsoft team has improved and expanded the PI System environment, planting flags inspired by case studies, new security features and threat models.

Below we have a summary of the PI challenges from last year. OSIsoft provided 11 of the 43 total flags for the competition. There were 5 flags left standing at the end of the competition and 4 flags that were only solved by one team. The most successful competitor captured 450 of the possible 2025 points from the PI challenges.

Flag	1	2	3	4	5	6	7	8	9	10	11
Points	25	50	100	100	125	50	125	300	300	500	400
Successes	16	6	1	1	1	1	0	0	0	0	0

Reviewing the logs in our environment revealed that many teams did perform reconnaissance, but did not progress. Perhaps the low success rate of the competitors has gone to our heads, so this year we are upping the ante. The first (if any) team that captures the mysterious, illustrious “Golden PI” flag, will win the opportunity to deliver ~3.14 pies to the faces of the OSIsoft security advisory team in attendance. You heard right, this is your opportunity to exact sweet revenge on a vendor!

Want to learn more? Every Wednesday in December we’ll give an inside look at the CTF environment on the PI Square Security Forum, providing background and perhaps even a few hints along the way. Search for the S4x17 tag to get all posts related to the event.

Image by Gerd Leonard

Digital Bond

For Secure & Robust ICS