Another Look at Application Whitelisting in Control Systems
Someone needs to tell me where the downside is with products like CoreTrace Bouncer. I’ve tried to be skeptical of application whitelisting but the more I see, the more I like it. Recently I had the opportunity to see Bouncer demonstrated on a Yokogowa Centum DCS. I’ve seen lab demo’s before but this was the first time I had seen it in the context of control system servers. My overall impression: this is an elegant and effective solution to some of the security challenges we face with Windows servers and workstations in control systems.
Perhaps the simplest definition for application whitelisting is that you allow the known good programs to run and nothing else. Blacklisting, on the other hand, is the traditional approach where you allow everything to run and then identify known bad things using anti-malware software. The trouble with blacklisting is that the list is moving and ever-expanding. For some additional perspective on this, see #2 in Marcus Ranum’s “The Six Dumbest Ideas in Computer Security“. One of my favorite quotes from that article:
Why is “Enumerating Badness” a dumb idea? It’s a dumb idea because sometime around 1992 the amount of Badness in the Internet began to vastly outweigh the amount of Goodness.
Whitelisting skepticism has always centered around management and performance. How do I allow people to do their job and is having to check the whitelist every time I click on myprogram.exe going to slow things down? With Bouncer, there are a number of ways to allow people or programs to make changes by designating them as trusted entities. This is critical in enterprise networks but because of the deterministic nature of most control systems, I think it’s less of an issue. As far as performance, the magic is that the core whitelisting part of Bouncer loads in the kernel. It’s not, for example, modifying the actual ACLs of hundreds of binaries in the system – it controls their execution from a lower level.
So let’s break it down: we have Windows machines in control environments that are difficult to patch. Delivery of AV signatures, not to mention the overhead of running AV to begin with, is also painful. Finally, the long life cycle of these systems means we’re dealing with old OS versions in many cases. Introduce Bouncer, a solution that defeats malware to the point that patching may be irrelevant and it works with Windows versions back to NT 4.0. CoreTrace admits they didn’t go out targeting our market but it’s easy to see why over half their new clients this year are utilities and other organizations trying to solve control system security issues.
If you have NERC CIP responsibility, some light bulbs are probably going off about now. Can I deploy a product like Bouncer and not have to do AV updates and patches? The CEO of Encari (Matthew Luallen) and the Midwest-ISO chairman (Paul Feldman) make a case for meeting “both the spirit and letter of the law” in this whitepaper: Malicious Software Prevention for NERC CIP-007 Compliance. The case is pretty clear for anti-malware. For patching it may at least buy you some time as a compensating control. The Luallen/Felman paper says this regarding CIP-007 R3:
By preventing the execution of malware — including those that are deposited via vulnerabilities that haven’t been patched or via memory-based attacks like DLL injections — application whitelisting is a compensating control until the PCS vendor approved security patches are installed during regular maintenance windows.
Incidentally, the Emerson Process Management group put their vote in for whitelisting by including Bouncer as the anti-malware component of their Ovation Security Center product.
Between doubts about the effectiveness of anti-virus and the current security and compliance challenges faced in control systems, there are some compelling reasons to have a look at application whitelisting.
(Full Disclosure: CoreTrace has been an advertiser on the Digital Bond site)
Author: Jason Holcomb
Posted: September 21st, 2009 under Security Tools.
Comments: 5
Comments
Comment from Michael Toecker
Time: September 21, 2009, 9:54 am
Personally, I’m a big fan of what I’ve seen so far from application whitelisting programs. They lock the system down exceptionally well, much better than any of the standard configurations. It’s relatively simple, and requires much less overhead than other processes doing similiar work. Plus, this method speaks to process engineers, they like the idea of restricting a ‘process’ to only the most essentially elements.
But since you asked for some of the downsides/tradeoffs:
First, you have to pay attention to exploits that inject code into running processes. Since these processes are already running in memory, the whitelister will not pick up on the overflow. So, patching of those high risk network services still needs to be a priority. However, you will see if the exploit is blocked while attempting to create new files or processes, which helps in determining if you are infected.
Second, never ever never ever put the whitelist on unless you are absolutely sure you are finished with the box. Otherwise, you will have to back out and reapply, which can take a good deal of time. Lack of proper planning will hurt you with whitelisting. Since most owners are concerned about time spent on security controls, they should be requiring a detailed game plan when implementing and managing a whitelisted set of systems. And “reapply whitelist” should always be the very last step.
Third, unintended consequences. I’ve personally had an experience where we set up a trusted account to apply patches, and the whitelister flaked. It refused to apply the patches because of the way the patches invoked permissions from other (non protected) accounts. Very annoying, and since we didn’t want to add those accounts to the ‘allowed’ list, we had to back out of the whitelist, patch, regenerate, and reapply. Proper testing should pick out this issue before it hits production, but there are many ways to install a patch that could cause this type of situation.
Fourth, in order to run certain kinds of vulnerability assessments, especially third party ones with custom tools, the whitelister must be disabled to allow tools to be loaded on the machine. This means that you are opening the system up to an outside party, so make absolutely sure they aren’t infected. With the whitelister turned off, owners need to realize they are at the mercy of the 3rd party’s security. And when the 3rd party is finished, you may have to regenerate a whitelist, bringing the newly loaded viruses into the ‘allowed’ list. So, owners should put procedures in place for 3rd party interaction with the system to minimize this very real risk.
Michael Toecker
Burns & McDonnell
Comment from Wes Miller
Time: September 21, 2009, 4:16 pm
Thanks Jason, for your kind words, and thanks Michael, for your insightful feedback.
Michael – the downsides as you elaborated are indeed true of whitelisting as it has classically been designed – that is, as “lockdown”. Microsoft’s Software Restriction Policies in Windows since Windows 2000, for example, demonstrate the downsides you highlight quite well when implemented as a whitelisting solution.
When we designed BOUNCER by CoreTrace, these were exactly the problems we worked to negate, by doing our best to integrate in with your existing management solution, and building a whitelisting product that has as close to zero-friction as possible. When you think about it, antivirus hasn’t been with us for two decades because it works particularly well. No, it’s with us because it is somewhat efficacious (was for some time, anyway) and it -generally- won’t cause a system to go belly-up if it is overly aggressive (unless it false-positives an OS binary, as many have in the past).
Specifically, we’ve looked at all of the consequences you mentioned. Exploits in whitelisted code always seem to hit the fore, even though most AV products aren’t good at blocking memory-only exploits. This is something we intentionally designed into BOUNCER, since it often becomes a first objection point about whitelisting (again, even though most AV products can’t stop them). Your second point is indeed true of most other whitelisting products I’ve ever worked with – thus the reason why we build our whitelist based on the executables that are on a system as of the time we secure it. The third is indeed one of the key points of building what we call Trusted Change, a mechanism that allows us to install patches and updates in as seamless of a manner as possible. Finally, your fourth point, though more of an issue for those of us trying to actually secure systems via vuln assessments, can be handled via the use of virtual machines – my preferred methodology for isolating questionable code of any kind (which often brings us back to a circular reference with regard to your first objection, blocking exploits in trusted code).
Indeed, you’ve brought up some good points when analyzing traditional whitelisting solutions. Which is one of the reasons why we’ve tried to be non-traditional in our approach to whitelisting.
Comment from Anonymous
Time: September 21, 2009, 4:54 pm
Application whitelisting is clearly not the silver bullet for information security; however, I consider it the only future for anti-virus/anti-malware. It must be implemented in this light, coupled with defense-in-depth principles. You are only as secure as your weakest link (e.g. if you lack physical security, anyone with an O/S CD can rip Bouncer out of a system in 10 minutes…)
I don’t currently have Bouncer, but I have been watching it for over a year, and I plan to implement it in at least a limited pilot project on a control system in the near future.
As to Mr. Toecker’s comments:
1) I believe that Bouncer does in fact have a feature that protects against buffer overflows, and they are talking of SQL injection protection for the next version. However, I personally think that this goes beyond their stated intent, and isn’t a necessary change.
2) I concur with this for sure; if your system isn’t secure to begin with, all hope is lost. This argument shouldn’t prevent anyone from moving to whitelist technology, since it also applies to blacklist solutions.
I have been preaching to the CoreTrace salesmen that Bouncer shouldn’t be sold as an alternative to anti-virus — IT IS ANTI-VIRUS. We’ve just been brainwashed for so long to believe that Anti-Virus means blacklist…
Pingback from Digital Bond » NERC CIP and Application Whitelisting Redux
Time: October 23, 2009, 10:11 am
[...] My recent blog post on application whitelisting, and specifically the Bouncer solution, sparked a lot of offline discussion. One of those conversations was with someone who has a significant stake in NERC CIP and agreed to let me post his comments. I try not to get too involved in hair-splitting discussions about standards compliance but I thought this was well said. It is also potentially useful information for organizations working on CIP and interested in whitelisting. [...]
Pingback from Digital Bond » S4 Preview: An Analysis of White Listing Security Solutions and Their Applicability In Control Systems
Time: November 19, 2009, 1:20 pm
[...] White listing solutions are being targeted at the control system space as both an augmentation to and replacement for anti-virus. The fit is intriguing because of the relatively static or unchanging nature of control system workstations and servers as compared to traditional corporate or business networks. Adding fuel to this discussion is the growing disillusionment with anti-virus effectiveness due to things like the race to zero. [...]
Write a comment