Guest author Michael Toecker is active in ICS security and tweets as @mtoecker. The opinions expressed in this post are Michael’s, and not necessarily those of his employer.
While Googling some NERC CIP research activities, I came across several forum posts regarding cyber security where details of systems, systems that had automation software installed. The posts involve HijackThis logs, which are tools that dump configuration information for posting on a support site, and it’s one of many such tools. On those support sites, users can get crowd-sourced help in identifying and removing viruses, trojans, and other malware.
The details revealed by the HijackThis logs are illuminating. For instance, the log posted here in 2008 is the worst of the 11 I identified in my @mtoecker twitter posts.
The HijackThis log clearly shows (and the moderator quickly points out) that the laptop is riddled with malware. At first glance, here are two problems I’ve noted just from inspection:
- The computer’s DNS queries have been redirected to two Ukrainian DNS servers that were used by Zlob.DNSChanger from mid-2006 to redirect users to nasty sites.
- There is a key in WinLogon for xxywwwx.dll. Using WinLogon keeps certain programs from being removed in run time, and the DLL is clearly from Adware.Virtumonde.GFH from 2007.
The HijackThis log also shows the automation equipment the laptop was most likely interfacing with. Installed are several GE software brands (Proficy, Intellution, and FANUC) and the Alstom controller interface and programming software, Alspa Pilot. The computer also shows the OPCEnum interface from the OPC foundation as well, needed for interfacing with OPC servers.
Using the HijackThis log, it’s also possible to infer details of ownership and use. The system is obviously a laptop, it has Intel ProSet Wireless drivers and services installed. The system was owned by Alstom UK, and is part of the former Power Conversion division, namely because of the “rugbr.uk.powerconv.alstom.com” domain name shown several times. There is automation software in use, a cell phone productivity suite for a Nokia 6600 (an old phone for 2008), at least some token IT security apps (AV, VPN, a Domain Architecture), and some office apps. All evidence points to this being a technician laptop, and is a good model for nearly any vendor technician laptop anywhere in the world.
Some basic conclusions that can be drawn from this model laptop.
- The Alstom (or whoever) IT department treated this like any office system, and gave it no protections outside of the norm. At some point, this user moved past the IT department and went online to solve his virus problems. Not good, as leaving them out of the loop means you take a good deal of risk management out of the process.
- The ZLob trojan shown in my inspection was preventable by 2008, when the post was made. Its possible that the bulletin board has an incorrect date. But if the date is correct, then there was a serious lapse in anti-virus signature updates for two years. Catching malware as old as Zlob with updated AV would have been very unlikely. I would guess that the adware was too, but I’m less confident.
- Having a capable wireless adapter on a laptop that connects directly to the automation network is a frightening prospect, it could bridge the automation network to the Internet quite easily if improperly configured. Those of you under NERC CIP, pay attention to the ‘access point’ section of CIP-005.
So, what lesson or action can I take away from this information? Well, I add another reason to not let third party systems connect into the control system. Vendors and their techs won’t be happy with this conclusion, considering the level of work each one puts into obtaining and loading all their tools and apps. This post, and the others I saw, show the issue with vendors bringing their own computers in, and reasons for requiring that vendors use designated systems for controls work.
To be perfectly honest, the fact that automation system information can be found on the Internet doesn’t worry me. The secrecy argument, the ‘closed-system’ argument, the ‘required specified knowledge and testing conditions” argument don’t hold much water without demonstrable evidence. Secure your systems like the details are posted on the Internet, then you are on the right track.
Image by Search Antigua
Great blog post Michael!
I agree that much need to be done in order to limit third party access to control systems, both on site as well as remote access. I’d also suggest that asset owners start monitoring their control system networks and remote access links properly. By monitoring I mean bringing in IDS’s, full packet capture of network traffic and of course retaining firewall logs. This would enable asset owners to detect if a contractor brings in a “dirty” laptop. The fact that the network access is being monitored might also put some extra pressure on contractors to make sure they only bring in clean computers.
The contractor laptop issue is a very serious Achilles heel to control systems. If there was ever a reason to use zone and conduit systems with firewalls everywhere, this is it.
It is also a very good case for keeping in-house engineering staff dedicated to building the control systems. You never know where the contractor laptop has been. I’m a big fan of keeping contractors off of the live control systems.
Note to plants and distribution system managers: cultivate your engineers and don’t let contractors connect directly to your control systems. The ultimate costs will make you wish you hadn’t –once you realize what hit you.
@Jake – of course it’s not just laptops from otherwise well intended contractors. While I agree this is a serious vector remote access to control systems is also very common and potentially risky.
I do have a follow up question regarding your experience with wireless media. Where do you rank threat from such an open media? Do the same defensive policies work for laptops?
Erik, thanks and long time no see!
There has been considerable action already on the firewall logging piece, I’m actually working on regular (D/W/M/OnEvent) monitoring reports for various components of the architecture, including remote access. There is some external guidance on exactly what needs to be monitored and how often it needs to be, but very very little, most is determined by guys like me.
The laptop issue has been simple for me: Don’t allow them to interact with our control networks directly, require contractors to use systems we own where we have verified security controls active. If we don’t have the tools to do what contractors need to do, we work through change management to get them verified and installed.
Mike Toecker
//Private Citizen
Mike,
We have run into owner/operator clients with the same policy, and it usually doesn’t go well. Things like 4 year old version of Nessus lacking professional feed. Systems they don’t have the admin credentials to modify. A huge lack of tools, plus unless you are going to load a #$@load of tools it is hard to specify all you will need until the first round of recon and enumeration has taken place.
We have been a lot more exacting in our requirements, but when you arrive and it isn’t there you have to do your best.
I appreciate the approach, but so far we have think the analysis/testing of the laptop before it is attached along with contractual agreements and professional liability insurance has been more effective in protecting the asset owner and getting them quality results from the consulting.
Dale
Dale,
I guess the security piece is a little different, but security vendors typically aren’t conducting work at the site 40-60% of year. I was talking more about the guys we have coming out to tune turbines, or make control changes under contract.
Most of these control technicians have a laptop with every tool they need to do work on any system they are typically sent to. We saw this on the vendor laptops from my twitter posts. That violates ‘only necessary for job function’ in my opinion, when it relates to specific site needs. What can be done is question them on what tools they need to do their job *for us*, and install those tools on computers we control and monitor.
So, if the vendor “Normal Dynamics” comes in to tune a turbine, we ask them what applications they need installed to complete the task. We then go through the process of validating and installing the tuning applications on our system, provisioning a user account for the technician, ensuring they have access to data and control files, etc. This whole operation can then be tied into our existing security infrastructure, rather than someone elses. Work can take several weeks, requiring access to runtime data, historical data, startup, shutdown, the works.
For this type of normal service, it’s perfectly reasonable. However, for a specialty service like security, there may be no way to install the applications ourselves, and that a system may need to interact directly with a real-time feed, etc that we can’t support. Then the measures you describe become more important. But honestly, security is not yet a service in the industry that has reached parity with turbine tuning and control logic changes.
Mike Toecker