Comment from Ralph Langner
Time: November 14, 2007, 5:51 am

“Anything with software running on your control system network needs to be patched.”

I tend to disagree, Dale. Even if a software vendor has approved a certain patch for his application, this doesn’t mean that the patch doesn’t disturb anything else in the complex environment given. Think of some vintage driver software, some specific PLC module with funny side effects etc. Lots of things can go wrong. While many engineers don’t know what exactly will go wrong, they know that there are lots of details they don’t know about and that could cause trouble. I like Rumsfeld’s phrase of the “known unknowns” in this context. Besides that, patching is often associated with downtime, i.e. cost.

The bottom line is, patching tends to be quite expensive. At the end of the day, it may well be more expensive than the damage probably prevented. I rather favor Markus Ranum’s approach: You can almost forget about patching if you have hardened your systems and have set up your perimeter properly. Funny enough, while hardening is one of the least costly controls for SCADA systems, it is seldomly done.

Comment from Jake Brodsky
Time: November 14, 2007, 7:57 am

I’ll second Ralph’s comment. If industrial control system users patched as many times and as frequently as their IT department did, we’d all lose our jobs.

The criteria I use is very simple: If the patch is one for safety reasons, I’ll do it right away. If the patch is for reliability, I’ll consider it if looks like it already is a problem. If it’s for a security issue, such as an internal escalation of privilige problem, I will carefully consider where the device is, and what the threat might be. In most cases, I’ll limit access of that sort if I can. If the security problem is one of public interface, I’ll probably pull it out of service and fix it. Process reliability doesn’t extend to the public’s ability to look over my shoulder.

Comment from Dale Peterson
Time: November 14, 2007, 8:27 am

Three comments on Jake and Ralph’s comments.

1) The “applicable security patches” phrase in my post is very important. The application and device vendors should first be doing the analysis if the patch even applies. We have seen excellent analysis of this by some vendors for Microsoft patches, but nothing else. Sometimes disabling a service or uninstalling an application, like the Compaq Web Server mentioned in the blog, is the best approach. If you have deployed a hardened configuration you will have the minimal attack surface possible.

2) I don’t buy the Ranum argument at all. Asset owners are responsible for managing risk. If an exploit is out there that will allow a remote user to run arbitrary code on your system, and the vendor says there is a patch available that has been tested in the lab, it is hard to imagine a case where it would not be prudent to test, using a lab and/or system redundancy, and deploy this patch. This is not instantaneous like on a corporate desktop; NERC CIP sets a 30 day window after patch availability.

Is it expensive. Yes. But the asset owner should have realized this when they purchased and deployed a complex IT application. Maintenance on physical assets in these systems has costs as well.

3) Jake - it sounds like you are willing to live with known vulnerabilities where solutions are available that would allow attackers who have penetrated the perimeter to own your devices and process. Am I misreading that? I would hope that there would be nothing on the control system network with a public interface with the exception of your perimeter security devices. Perhaps a few systems on a DMZ with an interface to the corporate network or third party partner.

Comment from Jake Brodsky
Time: November 14, 2007, 12:54 pm

I know you’re aware of this Dale, but this is one of those cultural issues of control systems design versus IT management.

I’m willing to live with privilige escalation vulnerabilities as long as 1) The machine is behind a DMZ, 2) the authorized users are unlikely to be able to exploit that vulnerability, and 3) It is their own damned plant –if they wanted to harm someone they can walk right over to the controls and manually cause harm. In short, the disruption from the patch may not make things worth pursuing.

Now for border machines, facing or inside a DMZ, I will not hold anything back. If there is a patch, I’ll apply it. Those machines are sacrificial. If the Chief of Operations doesn’t get his data e-mailed to him, he can walk down the hall and ask the Control Center to print a report.

This does not preclude the use of an IDS. It does not preclude the use of a firewall in various parts of the system. Basically, what I’m saying is that I’m willing to risk a potential vulnerability as long as the rest of the defense in depth is in reasonable shape.

Dale, I have personal experience where supposedly well tested application patches caused more problems than they fixed. Every patch is a crap shoot. We can ask for regression testing. However, at the end of the day, only a real site with an honest real time performance requirement and a database with real updates can effectively determine whether something is truly stable.

The NERC CIP 30 day patch window seems quite simplistic and silly in this context. How will anyone know if the patch itself is any good?

As for system redundancy, it’s there to help us deal with routine hardware and software issues, such as backup and restore operations. If we want to try out a patch, one should have a third server for patch testing and application software development. We often beta test our patches on the development station first. I refuse to mess around with a working pair of redundant servers until I have some reasonable assurance that the patch can work without leaking memory, or breaking some other part of the application.

Comment from Dale Peterson
Time: November 14, 2007, 1:25 pm

Jake - sounds like we are just going to have to disagree on this one.

My strong recommendation is you and anyone else taking this approach make sure you get approval for not patching and accepting that risk from someone at the appropriate level in the organization.

You probably have done this, but risk acceptance by those without the authority to accept risk is a common problem we see in assessments.

Comment from Ron Southworth
Time: November 14, 2007, 1:40 pm

Hi Guys,

Thanks for the dialog. Jake and Ralph, I could not have articulated it any better.

With respect to the comment regarding expenses Dale ( It must be Pick on Dale Day but it is not really).

Many decision makers don’t get it on the real costs - quite often, especially in the industry Jake and I work in. Effective listening is not a high priority!

We have a project that I know is going to double - even potentially treble the capital outlay presently budgeted for and it is a big budget for the organizations size.

The operating cost and maintenance - costs associated to the extra staff that will be needed to maintain the system have not been allowed for.

Whole of life costs - not even on the horizon!

Not all places are like this but a high number are!

Possibly your regular customer base does not have these issues but across the industrial base they are very real and are an upward moving trend!

Comment from Stephan Beirer
Time: November 15, 2007, 6:51 am

If you have a well secured perimeter you might get away with
not patching and - as Dale pointed out - having a formal approval for this approach.

fact is, that most of the perimeters I have seen are not well secured. especially in the industrial environment the scope of the inner “secure” network is much too broad (e.g ‘we trust the whole plant’), the firewall has a ruleset with many undocumented exceptions and/or too many inernal or external parties are accessing the ’secure’ network.

While the control networks in the critical infrastructure sector are often better isolated than in the industrial world, the number of systems exchanging data over a control center DMZ is steadily increasing (and we will not stop this trend). Accordingly the probability of a breach of the perimeter increases - especially if the DMZ is facing a corporate network with thousand of users. Thus, to obey the principle of defense in depth one should not have unpatched security vulnerabilities behind the secure perimeter.

We are working on a project where our client will get a brand new system. Luckily, the whole system is from a single vendor. This vendor will be responsible for patching the whole system - from the base operating system up to the application. Of course, this will increase the costs of the maintenace contract enormously. The upper managment will have to learn: if you take advantage of the benefits of standard IT technology you’ll also have to face the consequences: these things have to be patched. very often.

Comment from cnioperator
Time: November 15, 2007, 8:40 am

We have a slightly different perspective. Our approach is to patch unless there’s a really good reason not to.
We do wait for our control system vendors to accredit the patches but once they tell us the patch won’t affect their apps, we install.
Dale is right, many of these vendors accredit in a really timely manner. One of our vendors averages less than a day to accredit MS patches for their system!

My take is that it’s too much effort to analyse each patch to determine whether to install or not. Just installing them isn’t difficult and isn’t really expensive. Once you have the infrastructure in place, it’s virtually free.

So in summary: why bother analysing, just install. We haven’t had any issues with patches affecting our operations anywhere in the world.

I suppose the only caveat to this would be a scenario where you have rolled your own control system. In that scenario you have to decide which patches affect your systems and balance the risk of install V’s the vulnerability. We don’t roll our own so this doesn’t apply to us.

Comment from Ron Southworth
Time: November 15, 2007, 9:44 am

Good procedural change management and sign off is absolutely important. Risk appetites and perceived risk appetites are very different between utilities and verticals. The bottom line values create different levels of vendor knowledge and degrees of bundled components in a solution.

The risk associated with patching is still largely pushed back down to the enterprise support people in the lower end of the market place or in the more resource poor verticals if you prefer.

I am really hoping that I see vendors in the entry to mid levels of the market place providing more of the support you speak of cnioperator as it is just not there right now across the board.

End users are paying the same sort of percentage for support arrangements so why is there such a difference in what the end users receive for their dues.

Comment from Rob Lewis
Time: November 15, 2007, 1:28 pm

This discussion brings up the need for a security model that does not have patching as its basis , but one that prevents systems/apps with vulnerabilities from being exploited.

This requires a technology that enforces behaviors, in a deterministic sense. Besides preventing escalation of privileges that lead to attacks, such an approach could also prevent damage due to unintentional errors (i.e config errors) and deliberate sabotage by authorized users.

It’s value would be even greater when dealing with legacy systems that are no longer supported, or for trying to protect non-standardized or home-grown systems/apps that may be broken by the usual patching process. It could at least buy time so that patching could be deferred until a chosen time of greatest convenience/effiency.

Comment from Matthew Franz
Time: November 16, 2007, 12:17 pm

Rob,

I’m not sure what you are talking about (I’m not sure “patching” and “security model” can even be used in the same sentence) but it sounds like the “IPS to buy time for patching” argument or the naive “develop secure software that doesn’t have vulnerabilities”. But it is a shell game. As the OS level attack surface is reduced (ideally, and vuln disclosures back this trend) the threats/vulns move up the stack into application space or down the stack into firmware, device drivers or into new technologies (virtualization, or these “behavior enforcing” technologies” you describe)

Comment from Ron Southworth
Time: November 16, 2007, 9:22 pm

Matt I love the analogy (and a pun to boot).

I think we are heading down the road of integrated performance monitoring systems - describing nominal operation signatures for our SCADA systems and then having “clever filtering” systems to indicate out of nominal conditions from all the data compiled.

Comment from Rob Lewis
Time: November 18, 2007, 9:23 pm

@Matt,

When you say:

“As the OS level attack surface is reduced… the threats/vulns move up the stack into application space… or down the stack into firmware, device drivers or into new technologies”.

That conventional thinking is commonly expressed, and based on the status quo, is quite correct; it is a shell game.

For a behaviour enforcer to be able to stop even previously unseen executables, unauthorized use of data (insider theft) or system resources (sabotage), a technology would indeed have to deliver a very comprehensive security model and deliver several layers of security simultaneously. Those layers would be:
Rules based access and audit control; Mandatory access control; Domain separation; RBAC and Multi-level security; and privilege separation, including separation of root from system.

Such a model would have to control access at the OS level while using parameters from the OS layer to the application layers for its rules enforcement.

The technology that I am thinking about does so, and it happens to protect against attacks down the stack as well.

I am just presenting it as an alternative to the circular discussions that appear on the board, and others.

@ Ron,

This same technology can be used to filter anomolous behavior in a non-intrusive way, using a kernel level policy enforcer that operates in real-time based on policies that map directly to business rules; no need to look at compiled data for anomolies, no pattern matching or signatures, just a whitelist of allowed executables, at the data, system and network level, on a per user basis. In other words, a behavior enforcer does not allow any out of nominal conditions.