Comment from Jake Brodsky
Time: August 22, 2007, 10:56 am

Dale, people are lazy about this until the disaster happens. Only then do they demand protection.

One of the big beefs with Vista is that it actually tries to use its security. People don’t like it. It rubs everyone the wrong way. The reason? Nobody understands the security model.

In my opinion, that’s what is going on here. Users don’t understand the security. We need to sell the model, not just the products and how they fit in. And try as we might, we can’t sell it through procurement boilerplate. That presumes people understand enough to want it.

It’s like trying to sell airbags and seat belts to a bunch of Model T owners. They don’t even know they have a problem, and we’re already trying to sell a solution. Meanwhile, even if we did sell these products to them, the entire car is a death trap. The seatbelts and airbags might help some, but they’re not going to fix the more serious crashworthiness and handling characteristics.

Once again, this problem isn’t just at the lower levels, it goes right to the top of the organization. As Walt says, we need to sell to the corner office folks. We need to get to the CxO crowd on this. Unless and until they understand, we’re doomed to this holding pattern until we crash and burn.

Comment from Anonymous
Time: August 22, 2007, 11:00 am

Working for what is likely a typical asset owner, I can tell you that you hit the nail on the head by stating that owners do not want secure by default enabled. While I am not of the same mindset, the folks running our process control systems get immense influence with management given the ten figure revenue generated by said systems.

Like most organizations in this space, we have this prevailing notion that the process controls are on isolated networks that have no common logical connections - let alone physical. Also like most organizations, we have applications that report on the real-time and historical operations of these systems so that business decisions can be made.

Yet somehow, through some magical incantation of the phrase, “We are air-gapped”, the problem of security magically disappears - even when presented with traceroutes no less that show otherwise.

The fundamental problem is that the old guard is running process control systems and has not been exposed to the nuances of how systems can be compromised. When shown such things they dismiss them as being “IT” issues and impertinent to process controls with all these “obscure” and “exotic” technologies that “no hacker could possibly comprehend, let alone attack effectively”.

There is also the belief that because a system may not be actually critical to the process control, it is not a system that could pose a threat. Those that are critical “can be removed from service before a real problem arises should they be attacked”. My favorite is, “We let our guys at the plants walk around unsupervised and they can cause much more damage by flipping a physical switch. Why on earth would I care about a hacker?”.

Like it or not, the asset owners will only change their attitude towards “cyber” security once a major event happens - one that will unfortunately likely involve loss of life before the industry pulls it’s collective head out of the sand.

Don’t give up, and keep pushing the industry for reasonable security. Hopefully engineering schools can start pushing students towards an understanding of what can happen if they design electronic controls in an insecure manner, though the instructors also hail from the old guard.

In the IT world it took Melissa, Code Red, Nimda, Slammer, and Sasser for the industry to wake up to security. Has that awakening helped? Absolutely, which is why most attacks occur over application vectors (web browsers, IM, etc.) these days.

If folks like you who are in a position to lead or influence government policy and industry standards continue to push, we can get PCS/SCADA environments to that level before we have to start yelling BOHICA every two weeks as we did in IT with the worms. If we can get even there before focus is applied to PCS/SCADA by “the bad guys”, the situation will be orders of magnitude better and not nearly so dire.

Comment from Dale Peterson
Time: August 22, 2007, 11:16 am

Your “Airgap” comment is funny and true. I resisted, and readers would be surprised at the amount we resist blogging on for a variety of reasons, a presentation at the Weiss event.

A major vendor was showing a recommended “Airgap” architecture and then proceeded to show multiple connections bridging the “Airgap”.

Comment from Ralph Langner
Time: August 22, 2007, 11:24 am

Yeah, it would be unfair to put all the blame on the vendors. For the situation to improve, asset owners must wake up. This will likely be caused by some major disaster or by regulation.

Comment from Eric Cosman
Time: August 22, 2007, 12:48 pm

Interesting dialog, but I think that there may be an element of the picture that has been overlooked. In some cases, the resistance to security (or secure by default) may be rooted in the perception (or fact) that the security related features are simply too difficult for the lay user to understand.

Like any technology, security can be intimidating to the average person. As long as tools continue to present themselves using concepts and terminology that is foreign to people who have to use it, there will be a push to simply turn it off as a means of avoiding more complexity in an already complicated world.

Yes, power and communications cables have to be connected when a system is installed, but these are tasks that people are familiar with. Let’s strive to make security just as easy and commonly accepted.

Comment from Anonymous
Time: August 22, 2007, 12:59 pm

Regulation is there in some industries, such as the one in which I work - energy. The main issue with regulations as they stand today is that they are designed by the industry they are meant to regulate.

Don’t get me wrong, industry input is valuable and vital to the process. When it comes to being the authority on the regulations - both from a development and audit standpoint - it simply doesn’t pass the red face test. Involvement should end at SME input.

For example, see the FERC NOPR on the NERC CIP standards. NERC, comprised of industry players, created a documentation driven set of standards that left too much wiggle room. FERC has taken issue with many of them and is recommending closing of some loopholes such as the “reasonable business justification” and “technical feasibility” out clauses.

One potential outcome is they will “grade to the curve” on the first pass of the standards, attempt to make an example of one or two entities, then succumb to a great wailing and gnashing of teeth begging them to lessen the restrictiveness of the standards, such as it is.

Will FERC have the fortitude to hold the industry’s feet to the fire?

Will the industry thumb it’s collective nose at the feds and refuse to cooperate knowing full well the industry can’t shut down?

Regulations in some sectors (critical infrastructure) can only go as far as the industry allows them to.

I’ve always held the belief that SCADA/PCS security is not a technical challenge - never has been. It’s us vs. them and damned be all others. This attitude prevails in IT, in the process controls space, in the very organizations themselves. This is a people problem pure and simple and it’s about who has the power to control and organization’s direction.

Challenge the stonewall or let it stand. Which is more effective at inducing change? Do we, the security-minded folks, consciously stand idly by and watch the wall fall, taking with it it’s caretakers?

Comment from Jake Brodsky
Time: August 22, 2007, 11:11 pm

Preachers, allow me to introduce you to the choir!

The other half of this challenge is educating a generation of Operators that they need this stuff too, and how to look at the big picture. The use of IDS, firewalls, and the like is not simple to most old hands. I don’t expect I’ll ever convince those guys that they need this stuff. But the new kids won’t have nearly as much trouble with the concepts.

I need to sit down and read the FERC NOPR on NERC CIP…

Comment from Ralph Langner
Time: August 23, 2007, 5:40 am

Wait a second, Eric and Jake. Security as we see it in today’s products IS way to complex for the everage engineer, but it wouldn’t HAVE to be. If customers put enough emphasis on security features, vendors could come up with smart solutions. However as long as customers favor a mindset that is centered around “industry standards” (i.e. do what all the others do just that you can’t be blamed when it doesn’t work out), security will be an add-on that appears to be quite complicated by nature. Just look at OPC. The combination Modbus/TCP + OPC is, from a contemporary technology standpoint, about the most bizarre construction that one could come up with. Anyway, it continues to be installed about every day because it is “standard”. Now if you want to secure it you determine that all the DCOM stuff is way beyond your knowledge, capability, AND SPARE TIME. So you just leave the doors open and think, well, these guys in security really don’t know about our needs. Which brings us back to Anonymus’ statement: Security is not a technical challenge.

Comment from cnioperator
Time: August 23, 2007, 7:36 am

How curious that you obseve “the vast majority of asset owners do not want security”

I don’t think I’ve met this majority!

As a multinational asset owner I guess I (& most everyone I know) must be in the minority.

Or maybe I’ve missed the point completely

Comment from Dale Peterson
Time: August 23, 2007, 9:14 am

cnioperator - I think the balance of the blog entry explains this conclusion, and there is a difference between talk and action. Vendors are saying:

- when they offer basic and important security options in their products they are rarely used

- when they implement Secure By Default or even minor security configuration customers object in mass

There are a minority of asset owners that care, and we are fortunate that we get to work with some of them. They are the ones willing to spend time and money on security, attend control system security events, and realize security will involve some changes in workflow.

If the majority wanted security, the market would respond. The epiphany for me was that not only are asset owners not demanding security from their vendors, but asset owners are telling vendors trying to add security that they don’t want it turned on by default (and subsequently most are not turning it on at installation).

I should add this gives me no joy. Personally I thought we were further along than this as a community and found the results from many interviews quite surprising and as I said in the blog, disheartening.

Comment from Fred Loveless
Time: August 23, 2007, 9:43 am

Dale, As a Support Manager for a company that makes OPC Products I can say that 80% or our calls are DCOM Security related this with all of our documentation showing how to make DCOM Security as loose as possible. That said, i panic every time i hear of our software being used in some high profile process and no security.

I think all vendors do as much as possible to protect their customers but as the saying goes “You can lead a horse to water but you cannot make him drink”. I think the worst thing about this is that the minority of users that want more security left out to dry as it were.

I look forward the day when Secure data collection and processing by default is as common place as the no-security by defult is today.

Comment from Kevin McGrath
Time: August 23, 2007, 10:57 am

Well back in the day none of the asset owners wanted to pay for safety either and then the Feds created OSHA. I keep saying I’m not a big proponent of imposed regulations but in the control system cyber security area I think it’s time for the government to start hitting these people between the eyes with a 2 x 4 until they finally get it.

Comment from Ralph Langner
Time: August 23, 2007, 11:45 am

Fred, I would like to show respect for your stated position. Fact is, every vendor of OPC products is faced with support inquiries on how to poke holes in DCOM security — at least since XP SP2. Inquiries that have no relation to the software product itself and that you can spend several hours on for a single customer — hours that the customer doesn’t pay for. Been there, done that, got that t-shirt. Another fact is that quite a bunch of OPC vendors have concluded (some by simple economics) to get rid of such support inquiries by using an install process that opens the doors automatically. Now the customer is happy because everything works right out of the box, and the vendor is happy, too, because he doesn’t have to continue what would have been Microsoft’s responsibility in the first place. It’s good to know that your company is responsible enough to discourage users from creating insecure environments. Let’s hope that sometimes it pays out for you.

Comment from anonymous manager
Time: August 23, 2007, 5:33 pm

This problem is strongly related to the evolution of field devices and the general inertia in utility organizations. The solutions are changing faster than the staff and organization. Most organizations resist change that affects the span of control of an established group. Yet the changes in technology require different skills than they did in the past.

Engineers and technicians that are responsible for field equipment are sold new technology that replaces the old. They are the buyers that don’t care about security. They are not IT experts, but they have always been responsible for the field equipment. They naturally want to keep control and don’t want to have to be IT guys.

Engineers that used to just plan out wiring for relays, and transducers are now buying electronic devices. Devices that are incorporating Windows or Unix operating systems. They never needed cyber security for their relay wiring, so why is it needed for a protective relay device?

Telecommunications staff has traditionally been responsible for layer 1 only. Now they are buying layer 3+ devices and still wanting to use them as layer 1 media converters! They never needed security for media converter, so why is it needed for a router?

The engineers and technicians really don’t want to learn how to manage IT security, because it is not what they are skilled at. They do not want to give up responsibility for their work, and they don’t want to complicate their work process by putting an IT analyst that never sees outside a server room in the field. IT staff may know about CYBER security, but they don’t have experience in substations. IT staff us usually sized for centralized management, and would need more staff to support a remote substation and communications facilities.

Organizations need to blend in IT expertise with their engineering design and communications field crews. Utilities need to add IT staff to field work teams. Engineering and telecommunications staff needs to accept the IT involvement, and not get too bent out of shape when security restrictions make it harder to do their job. They need management that is committed to get these “cross-divisional” teams to work together through the full lifecycle of field deployments.

When the proper team is involved in the work of buying and implementing the equipment, then the expectations for the vendors will reflect the proper requirements for a control system security posture. But getting this done in an organization will likely be more difficult than the technical work of securing a control system.

Comment from Rob Lewis
Time: August 28, 2007, 11:42 am

I hope that it does not boil down to a digital Pearl Harbor to get asset owners and governments to act on this, as Dale suggested. As part of a vendor that deals with high security environments and trusted systems, my experience has been to receive a lot of skepticism, not that we can not offer data and system assurance, but that we can deliver that without impeding business functionality and in a user-friendly package.

That is the bottom line, really. Secure by default will be accepted when it is the easier option. What asset owners need to realize is that if done properly, security decisions can become more intuitive, and the cost of security can decline, since unauthorized access attempts fall off the system as they are denied, business data flows can be optimized and stabilized, and disasters, clean-up, and fall-out can be avoided.