Virtualization in the SCADA World: Part 3 – Testing
For part 3 of this series, let’s dive into one of the major uses of virtualization – testing – and see if we can sort through how and where it applies to control systems. As we discuss this topic, we are assuming that the majority of servers connected to control systems are physical machines. If you’ve already embraced this concept and are running your production servers in a 100% virtualized environment, some of the conversation may not apply. For the rest of us, let’s assume that we want to use virtual machines to introduce or enhance testing ability in a system made up primarily of physical servers.
Let’s take a fairly typical system with a mixture of *nix and Windows machines, performing a variety of functions – HMI, Historian, SCADA server, ICCP server, etc… Using a tool like P2V (physical to virtual) from VMware, a virtual copy of each of these machines can be created and hosted, potentially, on a single server. We’ll save vendor support and licensing for a later discussion. Having this type of lab environment is something that has been a traditional use of VM in enterprise IT but is drastically underused in the control system world. You can use your imagination for the possibilities that exist when you have a virtualized replica of your production system, but for this discussion let’s focus on how we can use it for testing. For those in the electric industry, VM has the potential to provide some pain relief for CIP-007 R1 and others.
We discussed the snapshot capability in an earlier post – this one small feature is a huge value in any testing process. Here are some more specific examples of test scenarios:
1.) Patch and upgrade testing
This is perhaps the most obvious use of a virtualized test environment. Integrating this into your change control process can yield the ability to test to a level that may not have been previously available because of the quick rollback functionality. I am aware that this type of testing has limitations but contend that at least 95% of potential issues can be identified in the VM environment, and any others subsequently identified in a development or backup environment. The patch management process, for example, might include testing in the VM environment, a subsequent test on development or backup servers, and finally, production server implementation.
2.) Simulation
Virtualized environments are also ideal for simulation of various types of events – this can be used for a variety of functions including operator training or load testing. In the case of load testing, a new VM can be introduced that simulates a large volume of SCADA protocol or other traffic.
3.) Security testing
Because it offers a certain degree of isolation, a virtualized environment affords security administrators the ability to be much more liberal with security testing. Even in development and backup environments, there is often resistance to vulnerability scanning, let alone a full bore penetration test. A virtualized environment can provide an “air-gapped” place to do this more comfortably, without fear of production problems.
These are just a few examples of testing scenarios. Virtualization is not a panacea and there are certainly things that cannot be tested well in a VM environment. That said, I believe that any negative trade-off is minimal compared to the advancements that are possible.
I’m not naïve about the security implications, either. Like many things, increased functionality can be a double-edged sword. But sometimes increased functionality can have a serendipitous benefit to security – such is the case with virtualization, in my opinion. Stay tuned for a future post dedicated to security issues as well as a discussion of how the recovery benefits of VM can be used in a production implementation.
Author: Jason Holcomb
Posted: February 7th, 2008 under VM.
Comments: 5
Comments
Comment from amino world
Time: February 7, 2008, 7:32 pm
this is, perhaps intentionally, an oversimplification of the problem of patching and testing a copy of a “live” system. let’s consider an HMI that is cloned as jason recommends (and skipping licensing issues):
1) the clone _cannot_ be allowed to talk to the existing network, at least to write to any existing tags on that network. it cannot be talk to other nodes, as it will have the same name and address as the real node and may confuse the network. (in fact, the existing network should probably shun it’s communications as from an unauthorized source!)
2) if the clone is separated from the real network (as it should be), when it starts up it will try to find the DCS, PLC, RTU, and other nodes on the network and report scads or (and probably constant) errors. portions of the control system software will probably not run or run in some impaired mode… is this a suitable test environment for “proving” that a patch doesn’t adversely effect the application? (hint: no.)
3) most control system nodes talk to other nodes – a lot. again, this needs to be approximated in some measure to insure that the patch (or whatever) being tested is really being tested. booting without errors and services starting up is a necessary but not sufficient test. some testing under load is needed. i don’t know how the ‘virtualized servers’ test their apps, but something like this is surely done before putting the new server online, isn’t it?
4) ok, so some (or nearly all) of these things can be overcome by simulation, but now we’ve raised the work and/or costs involved considerably beyond that suggested by the article. not that this isn’t worth the effort and an attractive option to staging a full duplication of the system, but it’s hardly as ‘drag and drop’ as the article makes this sound.
that said, i’d like to see some comments from the SCADA and DCS suppliers about the validity of virtualized testing for their applications. when they embrace this technology, then we end users should, too.
Comment from Ron Southworth
Time: February 7, 2008, 8:27 pm
I don’t have a problem with utilising a virtual environment for testing and proving systems changes. I think amino world has identified one key motivatior – when vendors support the virtualisation environment then we can consider this more seriously for production.
The world virtual does it not mean “nearly the same” but not exactly. There is still some element of risk in porting from a virtual environent into the production environment it is less than operating change management on the production world and in the woater industry this is quite common. It still comes back to your risk appetiite!
Comment from Jake Brodsky
Time: February 8, 2008, 10:17 am
There are risks to virtualization, and there are risks continuing without it.
There are more flexible backup options with virtualized servers. One could switch contexts and then make mirrors of the partition for a full performance backup.
Freezing the image and writing it to an evidence disk could become an essential part of a forensic effort to determine what happened. Today, when something ugly happens in a company, law enforcement types want to impound all computer hardware immediately. However, we can’t do that in a live 24/7 environment. Giving them a dump from the live context might be a happy compromise.
That said, Ron has a point. We need to be sure the applications are written so that this kind of switching is supported. Real time systems don’t react well when they’ve been out of context for much more than a couple seconds. Without support from vendors, I don’t see how this can safely go forward.
Comment from Marty Edwards
Time: February 8, 2008, 1:36 pm
I agree with ‘amino’ on the challenges of virtualizing an entire system, but let’s not forget that we can have a hybrid type of approach where the ‘wintel’ boxes are virtualized and then we have a ‘representative sample’ of real controllers, PLC’s, etc. on your test environment.
Not necessarily a complete duplicate of your production system, but should be able to test critical comms and functions.
After all, it is the ‘wintel’ boxes that need the most frequent patching so it makes sense to virtualize them in the test environment.
I have successfully used virtualization in production systems for everything in the DMZ. Gives good flexibility to change on the fly.
Comment from Ron Southworth
Time: February 8, 2008, 11:05 pm
Hi Marty,
I think we are all on the same page and simply describing the benefits and risks associated from different perspectives. I think with some time passing validating the environments as being suitable it can be a workable prospect. The litmus is in persuading our vendors that this is a worthwhile endeavour. is it too big an ask for them to support what is really another OS – a special case OS but never the less another OS. I tried unsuccessfully with a couple of vendors about 24 months ago here to look at an alternate to Windows without too much success. Without the demand or captured interest not every vendor will invest. I have discussed this with one vendor representative that agreed with the concept in principal so this is not a lost cause. I can see an aditional benefit with virtualisation with virus signature detection and management and with change management & testing environs. One subject that crops up from time to time is a standard operating environment for SCADA maybe a virtualised environment if it was adopted couldbecome a very good option with time and some collaboration. It would be a bit spooky to have everyone have exactly the same environment!
Write a comment