ISA99 on Security Patching in ICS

ICS PatchingISA99 continues to churn out quality security documents. Some are written to be ISA/ANSI/IEC standards and others are Technical Reports for guidance. Recently a draft of ISA-TR62443-2-3: Patch Management in the IACS Environment was released for review.

Loyal readers likely don’t need this information because the topic has been discussed for years, but the average ICS owner/operator who is beginning to work on security is struggling with security patching and looking for guidance and good security practices.

Section 5 lists the asset owner requirements in security patching. It does an excellent job in describing and requiring an inventory of “updateable devices”. Documenting what software is running on each device and feeding this into the security patch process is key. Security patching is much more than simply applying Microsoft patches. There are component patches (JRE), 3rd party app patches (web server, database), and patches for the ICS application itself.

Where Section 5 falls down is on prioritization of security patches and phased deployment. There is more information in Appendix B4.3.3, but even that is lacking at this point in the draft. I could write a long section on this, but two important points that weren’t there:

  1. Prioritized Patching List – It’s a sad fact that many ICS organizations are just beginning security patching efforts and trying to accomplish this across all systems tends to fail. We recommend owner/operators determine what systems are most at risk and focus the initial security patching effort on those systems. For example systems in the ICS DMZ and ICS network systems that are accessible from any external network or the ICS DMZ. If that is too much for a first bite, the prioritized patching list can be further tightened to the services accessible from external networks. This is a lot more practical than evaluating patch by patch.
  2. Consider Insecure By Design – If the component is insecure by design, patching even a serious issue will do little to improve the security posture. Why do vendors even bother patching systems that can be compromised using a feature.

Section 6 covers the Product Supplier Requirements. It is brief, and the most interesting part is it sets a 30-day window for vendor action on applicable patches.

Section 7 and Appendix A define an XML Schema Definition for vendors to provide Vendor Patch Compatibility information. Will this catch on and is it worth the effort? At least it gives vendors guidance on what information should be provided to their customers.

Most readers should skip straight to Appendix B – IACS Asset Owner Guidance on Patching. Free from the IEC formatting restrictions, this Appendix provides practical advice from start to finish on this process.

The standards making process tends to prevent any strong statements or controls. For example from Section 4:

Some extremely critical systems may have no allowed outage windows available, and can therefore not be patched.

Applying patches is a risk management issue. If the cost of shutdown to apply patches is greater than the risk evaluated cost, then the patch may be delayed, especially if there are other security controls in place that mitigate the risk (such as no internet access from the system).

The idea that it is acceptable for “extremely critical systems” to have no outage window for patching or other component maintenance is flawed. If bringing a component (server/workstation) down for maintenance, not the process, puts an ICS at risk then there is a lack of redundancy and robustness in the system. The danger is an owner/operator not wanting to apply security patches now has an out from a reputable ICS security standards organization — and “such as no internet access” as a security control that mitigates the risk … we would be in for a professional liability claim if we gave that advice.

ISA99 is soliciting comments on this and other drafts. It’s an open process, and they want your input.

Image by ceci un matt

3 comments to ISA99 on Security Patching in ICS

  • Unfortunately, the problem is not as “black and white” as Dale would like it to be. Having been responsible for the design, integration, deployment and maintenance of many high-integrity ICS solutions in a variety of critical industries, the issue of patching extends far beyond any lack of redundancy/robustness.

    Any one who has worked in the automation field in critical industries knows that redundancy is deployed not as a convenience, but rather as something that you would like to only have to depend on when needed. Outside of safety systems that must be tested on regular intervals (which is not part of this discussion), people just do not like to disable redundancy (a high risk event) to support the deployment of a patch (a lower-risk event) in the unlikely event that the redundancy is required during the upgrade and is not available causing an immediate event that has far greater consequences than that from not deploying the patch until a more “appropriate” and “safe” time.

    ISA99 obtains inputs from a large audience of end-users, and these statements are included to provide the practical and rational solution that balances not only the risks associated with a “cyber event”, but also those that encompass the much broader business “business impact” resulting from loss of availability that is typically the primary priority in any ICS security program.

    No matter how much testing is performed prior to patch deployment … you can never be 100% sure until you deploy it on the actual target system. Consider the impact of a failed patch that brings down the ICS controlling a major process unit (immediate and real), versus the impact of a potential cyber event that must first penetrate numerous other security controls as part of any solid defense-in-depth strategy as defined by ISA and other industry documents.

    I believe that patching is a strong security control, but like those controls that worked 5, 10 or 15 years ago, they are no longer strong enough to be considered a primary security control necessary to protect an ICS from current and emerging threats. Sometime I need to show Dale how easy it is to compromise a fully patched ICS when you have the tools and knowledge of the target system. I recnelty had the opportunity to show this to some foreign government entities and they realized that in a changing landscape, the controls need to be progressive and resilient to stand the test of time!

  • bryan owen

    @Joel – “…easy it is to compromise a fully patched ICS” sounds consistent with Dale’s point #2 above.

    Perhaps the worse kind of security theater is prioritizing ICS security updates when a device will simply accept anonymous commands or code by design.

    Ironically, the demand for patches is in part driven by awareness created by mainstream security research. Arguably industry would not be too far along without this research but there are unintended consequences such as misaligned priorities.

    Kudos to NERC CIP-007 R3 for program requirement on timely assessment of security updates for assets within the security perimeter. And optimistically TR62443-2-3 will help the industry by reducing the effort to manage such a program.

    Meanwhile, unsafe ICS protocols seen to be the elephant in the room.

  • Dale Peterson

    Joel – I was wondering if you were just trolling me for an argument while agreeing on some of the major points, but there are ideas in the post that appear to be conflicting while actually they are addressing different points. The two biggest items are:

    1. The criticality of a system (“extremely critical systems”) is an important factor in the risk assessment, and the security posture, including security patching, should be given a higher priority for extremely critical systems. The ISA99 document essentially takes the opposite approach and suggests that a system may be too critical to provide true defense in depth by hardening the endpoint OS and applications in addition to the perimeters.

    2. I think Bryan stated it well that focusing on security patching is often an example of “misaligned priorities”. The effort to apply security patches is likely better applied elsewhere if the component is insecure by design and easily compromised by a feature. We have been in the odd position of having security patching of the bulk of components in an ICS rather low on the priority list of recommended actions for efficient risk reduction.

    You mention in this and other writings about things being impossible or difficult in real world critical infrastructure ICS. We are fortunate to work with early adopter / security conscious asset owner clients that have been doing these impossible or difficult things for years now.

    Dale

Leave a Reply