Comment from Jake Brodsky
Time: October 9, 2007, 6:47 pm

Dale, you hit a glancing blow on the real reason many control engineers don’t like firewalls: It’s yet another damned point of failure. This is one reason why I liked the emergency bypass switch concept on the Tofino device.

With network complexity comes a myriad of new failure modes. THIS is what scares many controls engineers. Throw a few process safety requirements in to the mix and you can see why so many long for the isolation days. The reliability computations get unwieldly. The opportunity for misconfiguration is greater.

At the end of the day, the real problem isn’t firewalls. It’s the idiots who think they need the information in real time, when in reality, they can do just as well with 15 minute summary data. We have this CxO “need” to see the plant floor mentality which I think is very ill advised. If that kind of talk didn’t exist, we wouldn’t need so many firewalls and our systems would be simpler, and less expensive to operate.

Comment from Ron Southworth
Time: October 9, 2007, 8:13 pm

Hi Dale and Jake.

I agree that the rule-set configuration for a corporate firewall is indeed a very time consuming activity and therefore ends up being very complex.

How well documented are the rule-sets for a corporate site typically?

Corporate business has dictated almost globally the associated risks demand the philosophy of only permitting the end users access to what port / service / device and application has to be undertaken to perform their activities hopefully in a balanced and user friendly approach.

I do agree on the perspective that Jake has, in principal, with interconnected systems and it is certainly in the majority of cases valid for the water industry context. The final risk assessment in many health and safety assessments can be that a particular activity is just too risky and therefore elimination is identified as the measure. Why does this not happen in the information systems world? Probably through a lack of real understanding and acknowledgment of the consequences for the actions or activities.

The reality is however that SCADA systems are connected to business systems. We have to move forward and reduce significantly & minimise, if possible eliminate the risks, and bridge the knowledge and understanding gap associated to this “point of transfer”.

Differing SCADA systems (E.g. EMS, DMS) need near real time or real time systems info to be available to and by other expert systems, that sit in a layer conceptually above or to one side of our traditional operator HMI SCADA layer to assist and predict and control interdependencies.

Where these systems sit or what compartment they sit still resides in what I would describe as needing to be in a highly structured and highly isolated from the outside world environments.

Minimising risk certainly seems to be understood well in the corporate sphere when assessing other business activities but when the time comes to look at the “plant floor” to desktop requirements this understanding of risk and need for effective mitigation seems to be diluted in many instances.

Where and when the need for plant to floor data and the time frame it needs to be propagated into the corporate systems is also a factor that seems to be discounted when risk assessments are performed.

The delineation point / single point of entry of the control systems -The firewall rule-set for control systems as we all agree is generally a much easier rule set to create. If not, than as Jake has pointed out the control systems design has left the principal of KISS well behind. Multiple points of failure is something that needs to be reduce wherever possible to maintain availability.

In the future will it or should it be a firewall at this delineation point (change of focus from operational to business process needs?)

“Data diode” firewall appliances are becoming available for control systems.

Hopefully these devices will provide a higher degree of ” true isolation” both physical and logical & with the sort of separation that is necessary to provide a “safe” control system that also ends up meeting the hunger for near real time data for business process appetites.

Even with the cyber threat from the business system / outside world “gateway” eliminated Jake, we still have in geographically distributed SCADA systems at least, a range of communications layer entry points that also need to be dealt with and effective risk analysis and mitigations put in place for.

I think the effort guy’s like you Jake are putting into DNP3 for example are the most important area to be working on in managing the communications layer. And I commend you all for the efforts to date.

The PHY layer has plenty of mitigation tools available in shrink wrappers and at the end of the day there are limitations to what extent we can eliminate these risk vectors. More work needs to be done on effective key management for the SCADA communications suite of protocols.

With all of this stated we still need to work on the biggest threat to our systems, our fellow work colleagues or trusted insiders.

Comment from Matthew Franz
Time: October 9, 2007, 9:11 pm

Dale,

I’m not sure if this blog entry is meant to be deliberately provocative, but I think you are comparing apples and oranges and would argue that a simple 10 line ruleset in a firewall/ACL documentation is hardly more complex than something similar in a PLC manual. It is possible to compare a PLC with a Firewall.

But a large complex enterprise security infrastructure with IDS/IPS, Firewalls (multi-vendor if you follow best practices), SIM Products, NAC, Network & Application Monitors, Log servers, Routers, Switches, Load Balancers, WAN acceleration appliances, and God knows what else etc. is closer to the order of magnitude of complexity (to design, build, and manage) as a control system. In fact some enterprise operations centers *look* like control centers. Why? Well because they are control centers.

And as Ron hinted at, documentation of rule changes (and request for rules changes) across tens to hundreds of devices can be tedious, time consuming, and more or less difficult given the firewall feature set and the capabilities (or lack thereof) of the firewall management software.

And lo and behold (to beat the same dead horse I’ve been riding for the last 5 years) many IT application owners have the same fear of a single point of failure and the tendency to blame the firewall for outages, performance slowdowns, etc.

Comment from Ralph Langner
Time: October 10, 2007, 4:40 am

All is easy if you know your stuff. Few process engineers know enough about IT to master the networks they’re actually using. Even worse, most guys on the plant floor don’t have a clear picture of the actual data flow in the network. In the average risk assessment, it takes us several days to figure out and document which systems are connected to each other using which protocol for which purpose. Some clients don’t even think this is possible from start. With this mindset, you certainly think firewalls are complex. And with no past incidents, you may well think, why bother with all this security stuff.

Comment from Dale Peterson
Time: October 10, 2007, 7:35 am

Matt et al, my purpose in the post was two-fold:

1) Raise the point that the rulesets for most firewalls in control system environments are simple, especially a field firewall. This is true even if you look at the virtual rulesets that Innominate and Byres simplify through the use of device definition and templetes respectively. Even the corporate / SCADA firewall rulebase should not be too large. If it is the asset owner is likely letting way too much through the perimeter.

2) Point out that control system engineers can certainly complex deployments.

Finally, Ralph - - you are going to be really pleased with a modeling, anomaly detection paper at S4.

Comment from Eric Byres
Time: October 20, 2007, 6:53 pm

I agree that a small firewall can be simple (and so can a small PLC) - that is why home firewalls are successful deployed - the rule complexity is low and the protocols well known and simple.

But life on the plant floor is not as simple and the rule sets get messy, often because the protocols like OPC, FF HSE and so on do not fall into the nice “source IP, destination IP and port” category. Then we get firewalls with either really conflicted rule sets or shortcut rules (e.g. Allow ANY, ANY, ANY on outbound) so that people don’t have to spend a lot of time sorting out their network traffic patterns.

Now Dale makes a good point - How simple is [a firewall] compared to monitoring and controlling a complex process distributed over a plant or large part of the country with 5000 points or 100,000 points”? But the difference is that the control system world has evolved to create a lot of tools that simplify the management of these large systems. When I started in 1980, a PLC2 with 400 rungs of ladder logic would be nightmare to debug - it was spaghetti code at its worse. Add configuring an analog input card on a PLC - it was terribly complex.

Today technologies in all the control system vendors products help automate the programming and management of these systems to a degree that a 5000 point system is manageable. But a firewall with even 500 rules is not and the opportunity to create logically inconsistent rule sets is high. Thus I think there is a lot to learn from the control system world about building better configuration tools that make a firewall as simple as a PLC.