Comment from Anonymous
Time: May 11, 2007, 11:38 am

I’m an IT security professional who is skilled at intrusion detection, reverse engineering, incident response, and policy development. I’m employed in the energy sector, for an organization that is in scope for NERC CIP standards compliance, and we have numerous highly qualified SCADA professionals.

Our IT group has been tasked with taking the lead in CIP compliance as it has more financial resources that can be brought to bear. We are definitely not SCADA experts and our SCADA experts are not IT security experts. As such, the two groups have historically been at odds when it comes to situations where the two disciplines intersect. I suspect this is extremely prevalent in the industry, though perhaps not as visceral as the situation here.

My vision would be a training program geared towards owner/operators and not towards specific industry professionals – SCADA or IT. The goal of the program would be three-fold: provide a general level of security training to both sides of the house, provide a common language that allows barriers to be broken, and provide training organizations valuable insights that can then be input into the training curriculum for a more robust program industry-wide.

Week 1) Provide basic SCADA and SCADA security training to IT security professionals.

Week 2) Provide basic IT security training to SCADA operations and engineering professionals.

Week 3) Put the IT secuirty and SCADA professionals through a melded SCADA/IT security session that would be designed to facilitate communications between both groups who should now be able to at least use the same taxonomy when communicating.

It’s an ambitious program that takes a significant investment of time on the part of owner/operators. It should be semi-frequently offered so as to loop in all relevant personnel for an organization as no operation can go unmanned for the length of even one session.

Not even considering operational issues that would result from a “cyber” security breach, the alternative cost of an ill-conceived and poorly implemented NERC CIP compliance program would far outweigh the investment in time. Additionally, if the training is of a high technical quality, the lessons learned should prove valuable for any owner/operator on a long-term basis.

Any quality training should discuss and embrace differences between regulatory compliance requirements (aka bottom-of-the-barrel operational security) and real-world excellence.

Comment from Ron Southworth
Time: May 11, 2007, 6:30 pm

Hi Dale and Anonymous!

A subject very near and deer to my heart and current thoughts Dale! Many thanks to you both for raising the subject.

Having attended a one day course at INL Just last week as a part of the inaugural U5 conference I can definitely see there is a need to provide a “Convergence SCADA security course” to the marketplace. The course I attended had the range of attendee disciplines from computing to process control and the course was equally beneficial to all attendees. The course was tailored to the mean average of the experience of all the participants and was a one day, eight hour throttle down course presented by someone we all respect and admire – Mark Farbro!

We do need to try and bring together the gap we speak of so often and I know upsets both you and Matt when it is mentioned and rightly so. May I suggest at the point of starting a debate over an us and them situation that it is all really part of the scope of engineering disciplines anyhow so one course would be suitable to both groups. Last week confirmed that this is valid and is in line with Anon’s thoughts as well.

The duration of the course identified by Anonymous would take some reasonable resources and material to be brought into play. Like you Dale I run a two week program a year and some ad hoc training on Protocols & Practical Radio Telemetry and I can agree that teaching is certainly a taxing but rewarding, requiring a fair amount of energy for a small number of instructors to have to provide and sustain especially for a two week structure such as has been proposed. I think that if there was sufficient interest in the subject that the training provider I run the course for would be interested in rising to the challenge of providing such a course as I am certain other providers would be.

(I have viewed the scope of the course you offer Dale and it is a good three day course for certain FYI.)

I can say that INL would be very capable of providing a very good quality course of the scope that is being mentioned but I don’t know if they have sufficient dedicated teaching resources at present without detracting from their primary organisational goals and objectives to be running it to the extents that have been suggested so far. Perhaps some fresh dialogue with them Dale would be beneficial and they are a very approachable organisation.

I think this is worthy of open discussion so I have put forward some of the questions I would be asking before commencing on creating such a course.

Here is a few to perhaps start the ball rolling?

What level of accreditation are you talking about with respect to running such a course.

Do participants want to learn from the experts or from someone that is a professional trainer (practitioner to practitioner is a more successful model IMHO)

What depth of knowledge or technical or practical level is desireable to be achieved as an baseline outcome for the participants.

Do you try to make the course global in nature and have the regulation modules of the course segregated so that the course can be reused globally? ( I would suggest that this would be a very good and achievable idea to implement)

Do you really require to understand how to exploit vulnerabilities and to perform penetration testing to be capable of implementing mitigation strategies and therefore should the course focus on defender capabilities and aspects only.

We need to include and create Operator focussed courses as well!

What do you guys or others think about this topic?

southworthrg@bigpond.com

Comment from Ron Southworth
Time: May 11, 2007, 6:35 pm

Dale I forgot to put this in abouve. This topic is also part of the PCSF charter as well so some cross pollination from you blog may be worthwhile?

Comment from Ralph Langner
Time: May 14, 2007, 4:28 am

Ok guys, come over to Germany and attend our 3 day seminars. We got good beer, too.

As for a “SCADA Security for the IT Security Professional” seminar — we did offer this once, but got ZERO interest.

Comment from Jake Brodsky
Time: May 14, 2007, 7:07 am

Ralph’s comment is dead nuts on. The IT departments of most utilities don’t care about this subject either. They merely want to extend their staff to yet another application they think they understand.

This is the gulf that separates Control Engineers and IT. The control engineers don’t take this security stuff seriously. IT doesn’t take control engineer concerns seriously either. We all discuss common ground, but few actually get there.

As the anonymous post suggests, there is a great deal of cultural hate going back and forth. And if he’s reading, he should understand that his situation is all too commonplace.

Here are my suggestions for bridging the gap: First, threaten the Control Engineers. Show them all sorts of misapplications of IT security products. Fill them with scare stories about how safety might be compromized. Then, offer training. They’ll snap it up. On the IT side, threaten them with stories of safety violations and lawsuits for operating without a professional engineer’s certification. Show them what happens when an untested patch is deployed. Show them what happens when bandwidth is “temporarily” clogged with broadcast storms. Then offer them courses in real time computing security. They’ll snap it up.

Finally, after both sides have been through these courses, offer a coordination course.

Why wouldn’t this work? It’s what we’re doign already, right? Oh, Wait…

Comment from Jake Brodsky
Time: May 14, 2007, 8:09 am

The anonymous poster and Ralph’s experiences are entirely too typical. This gulf between IT and Control Engineers might as well be the Grand Canyon: Getting across it can take weeks of careful navigation and negotiation.

Most Engineers know very little about IT security. Conversely IT security people have very little understanding of what a real time system is and the safety issues entangled with them. Sprinkle some mediocrity on either side and you have the situation in most companies today.

I hate using FUD for a sales tactic. However, at the moment I’m at a loss to see how anyone could use another. The regulatory process is slow. It’s very incomplete at the moment. And in fact, I’m not sure I ever want to see what a regulatory framework for administering and configuring a security system would look like.

Maybe the solution is to throw a few well targeted and researched FUD bombs in to the industry. Observe who gets splattered and who cleans up. Highlight the clean-up cases and make sure everyone sees how dirty the other guys are (without mentioning names, of course).

I feel slimy just typing this sort of thing, but I’m really at a loss to find anything better.

Comment from Clint Bodungen
Time: May 14, 2007, 1:29 pm

Our institute has offered a complete CIP and SCADA security course for about 2 years now. If you are just looking for SCADA security, then I would definately go for the more straight to the point SCADA security course created by Dale and offered by Infosec Institute. It’s an excellent course. If you are looking for a complete CIP course from management to technical to applied skill, you may want to check out our course. It’s a 3 class program. Upon completing the program successfully, you are awarded the PCIP certification (Professional in Critical Infrastructure Protection). For more information, visit http://www.ci-institute.org

Comment from Ron Southworth
Time: May 14, 2007, 4:24 pm

Thanks Clint, I will check it out as you can tell this is something I see being of a real need here in Australia.

Jake I can understand your perspective on the void for certain. there are a couple of chasms I can name off the top of my head. I think the only way to stop making it a void is to fill the hole in.

I don’t know that good engineers would know nothing about IT security, I think it is more a case of it not being considered to be on their radar perhaps, and alternately we need to make IT people aware of not just the different Availability requirements but also the different configuration, zone and information flow requirements for control systems.

It is a shame you missed U5 Jake. This may have changed your mind somewhat. I guess the two ingredients are definitely training and experience to fill the convergence gap. I know that this can be filled so what training do we specify to fill the gap?

Comment from cnioperator
Time: May 15, 2007, 7:29 am

Hi guys, I’m the “large asset owner” who asked the question of Dale. What I really want is a training course to give our IT Security folks a grounding in SCADA security. The audience are IT security professionals not run of the mill IT guys. Our company is big enough that we have a security function quite separate from IT. The point being that these guys don’t need training in encryption firewalls, etc. They do need training in what control system are, what security technologies work in our world and, more importantly, what doesn’t!

Incidentally, I’ve been running SCADA security courses for control engineers in my company for 3 years now. We developed our own training in house and it turned out to be way more work than I imagined. So, now that I want to train my IT security folks, I was hoping to “buy not build”

Comment from Ralph Langner
Time: May 15, 2007, 8:39 am

“One of the challenges in the course was some students had vast control system experience and knowledge but no IT or IT Security background. Conversely, other students were from IT Security departments but knew nothing about control systems.” said Dale. This is also true for attendees of our seminars, although the IT folks usually are the minority.

Here’s how we try to use this in a positive sense. Many security flaws are simply due to the fact that IT and operations don’t talk to each other. They speak different languages, dress differently, cultivate different habits, drive different brands of cars etc. We try to give our attendees a sense that things that are perfectly clear for one group of people may be voodoo for another group of people, even if both groups are working for the same company, in the same building, and for the same objective. We try to involve these groups into discussions which might help them in their real life environments to establish team play across departments. I believe this is worth the effort as there are few problems in SCADA security that can be solved by technology alone.

Comment from Ron Southworth
Time: May 15, 2007, 9:32 am

Hi the CSSP website has it’s training system link up and running A pretty good first attempt not a hard course and the games are a bit of fun as well! A good awareness session.

http://cssptraining.labworks.org/training/lms/cgi-bin/login

Hi cnioperator I had a feeling it was U ! The need to find a bridging course is definitely there

Hi Ralph – Beer and you guys well I have drunk beer with your countrymen before! they put vodka shots into our beer to give it more umpfh! No competition from here!

Communications and culture are part of the big picture for certain.

The course syllabus from Dales course material from what I have read would be a good 3 day course for IT people for certain. Dale may be able to say if it could be extended to add more practical sessions to the course material. I would have to say that I am looking for a very similar course for my workplace as well there is still the problem of the old dog not wanting to learn new tricks however – This is something Jake can relate to for certain!

Comment from Jake Brodsky
Time: May 16, 2007, 7:00 am

Sadly, like so much other security training, this particular concern is the convergence of many disparate fields of study. It’s hard to get someone excited about this sort of thing. The knowledge base is very broad and it can be fairly deep in places.

Finding people who are actively interested and willing to learn such a broad field of study is not easy. I don’t mean to be crass about this, but in the end, what do those who make this effort get? Do they earn more money? Do they get more respect? Maybe a bit more of both, but is it worth the effort? I suspect that many are saying no.

Of those of us who are interested in SCADA security, I suspect many of us stumbled in to it. I know that’s what happened to me. Ralph? Ron? Dale? Any comments?

Comment from Ron Southworth
Time: May 16, 2007, 9:59 am

Hi Jake,

Stumbling across it all is a good analogy for me too. Willingness to learn well that is an interesting phrase I would also add and open mind. To be balanced this problem you speak of is something that you find regardless of the industry you work in. I have been finding that SCADA security has some of the most open and approachable people around – maybe because of the “rapid” pace compared to other areas of engineering!

I still don’t quite get why there is such a disparity of the technology or the culture. The broad scope of knowledge is what I find the most interesting Jake and I suspect that may be the common glue?

Comment from Joe Weiss
Time: May 16, 2007, 11:35 am

There is a significant difference between corporate IT and control systems and the neceassry training for each. Consequently, I was asked to develop a training course for the IEEE Training Department- Cyber Security of Substation Control and Diagnostic Systems. It is part of the Expert Now series. The url for the training course is http://www.ieee.org/web/education/Expert_Now_IEEE/modules.html#power

Additionally, Gary Seifert from INL, Jeff Dagle from PNNL, and myself have provided 4 hour short courses on cyber security of industrial control systems at various IEEE, Distributech, and ISA conferences.

I would be happy to provide more details if interested.
Joe

Comment from Ron Southworth
Time: May 16, 2007, 6:36 pm

Hi Joe,

Thanks for the link – you know me I am the human sponge so whatever you are prepaired to share I will certainly read.

I dont disagree at all about the the differences between system requirements and cultural differences – The need to maintain non-deterministic systems such that they can reliably supoport deterministic modes of operation. (any other key differentiators).

That gap we speak of is very much still a reality Joe. The but is…. We have to do something to bring everyone all together. A key lesson from the U5 gathering! IT and Process is all within the scope of Engineering disciplines so the difference is all perception and empire related anyway!

I still cannot quite understand why there is a need for two different programs as this does not do anything to bridge the gap. The thing that we are talking about is convergance. Mark Farbro and the guys at INL used this term at U5 and it is the fist time I have heard that word used in a security course to describe what this Security stuff is all about. Such a great word Mark!
(Apologies if someone else holds the title of the first to use this word for this context)

Every teacher trying to acheive knowledge sharing on any subject has an outcome driver to bring every participant up to the same level of understanding (at least by the end of the course). If a training program has sufficient scope and depth of information you must be able to then use one program?

It is good to see there are a number of short courses available for certain and this helps raise awareness.

cnioperator was looking for something more substantive than even a three day or 4 hour or 8 hour course ( I guess I am too) I have looked at the Masters syllibus being suggested on the PCSF training site and this is an excellent document and would make an excellent course.

The course described looked to be not so much for a practicioner and more for a manager so no joy for me in this instance I am afraid to say.

Am I trying to find what has not yet been developed Joe?

Comment from anonymousII
Time: June 12, 2007, 7:26 pm

Are you aware of any studies that project compliance costs for NERC CIP standards? Obviously there will be a cost projection for CIP 002 where all will have to identify critical assets and then a smaller universe for those who will have to proceed with CIP 003 through CIP 009. Majority of the costs will be labor intensive but there is also the acquisition of hardware and software to implement various documentation procedures. For those entities that started with Urgent Action 1200 the costs should be less. Welcome your feedback.

Comment from Ben
Time: June 16, 2007, 7:25 am

Which training is recommended for security professionals looking to gain SCADA expertise?

1. http://www.ci-institute.org
2. http://www.infosecinstitute.com/courses/scada_security_training.html
3. or other ?

Comment from Anonymous
Time: July 25, 2007, 3:39 pm

Anyone know of universities that teach courses (or have a whole curriculum) centered on cyber security for control systems?