- The details on the CAL ISO incident last week in the LA Times. “He broke a glass seal and pushed an emergency electricity shut-off button, plunging the Cal-ISO building in Folsom, a Sacramento suburb, into darkness and crashing computers used to communicate with the power market.”
- Citgo and InTech magazine just started a project blog on Citgo’s effort to design and implement security in the process control network at the Lake Charles Manufacturing Complex. The first entry shows the RFI Citgo issued in February. Subscribe to the blog at http://www.isa.org/intech/blog-citgo/atom.xml.
- I’m up in Vancouver at Wurldtech today and saw the list of products that have passed Achilles Controller Level 1 Certification. These controllers, and any other pass in the next two weeks, will be announced on May 15th. It is very cool that there are multiple controllers from multiple vendors that will be certified right out of the gate.
Archives for April 2007
We released two new Modbus TCP IDS signatures and some improvements and updates today. The download of the entire new SCADA IDS package and links to the documentation are available on our IDS research page.
The new signatures identify Modbus scanners in two different ways.
- SID 1111013, Modbus TCP – Function Code Scan, identifies a scanner attempting to determine what function codes are implemented. This is common in the reconnaissance phase of an attack and also can identify the controller vendor if they use proprietary function codes.
- SID 1111014, Modbus TCP – Points List Scan, identifies when a scanner is “walking” all valid coils, discrete inputs and registers to see which are being used. This could be the precursor to a detailed analysis of the process or simply a way of identifying points that could be written to create chaos.
A few other improvements and comments on this latest release:
- We have created a SCADApedia page for the Modbus TCP Signatures that includes a list of the signatures and brief description. The Snort rules and detailed documentation pages are still subscriber only content.
- The rules are still provided in a Snort format. Most IDS/IPS vendors and many MSSP’s have imported our SCADA IDS signatures into their systems in the past. Contact your IDS/IPS vendor to determine if and when they support these new signatures. The vendors release signatures at least quarterly so this should not be a long delay if they choose to add these.
- All applicable Modbus TCP rules have an additional content check for the 0x0000 in the protocol ID field of the MBAP. This will slightly reduce the chance of a false positive.
- The pcap file we provided was create from the University of Tulsa Modbus Scanner developed under the I3P program. It performs Modbus TCP function code scanning and walks the points list among other things. It is an interesting and helpful tool. However, we found different results from scans on the same PLC with the same points list. We are still looking at why that happened.
There is a long running discussion/argument on whether Mac’s are more secure than PC’s. Actually – that is not quite correct. I believe the argument is: are the number of Mac exploits relatively small, as compared to Windows, due to better software security engineering practices at Apple or because the motivation to find Mac exploits is dramatically less?
So the CanSecWest conference out if Vancouver provided one data point last week by sponsoring a contest. Hack a fully patched MacBook Pro and you get the laptop. Well, after day one there was little activity and no success. Did this prove hacking a Mac was very difficult, because there was a lot of talent at the event, or was the prize was insufficient motivation?
On day two, “motivation” was increased by a $10,000 bounty put up by 3com/Tipping Point. Shortly after that Dino Dai Zovi of Matasano developed an exploit and Shane Macaulay performed it on site to win the prize. A quote on the Matasano blog has Dino saying, “I think I may have set a land-speed record”.
If you are interested in the Apple / Mac discussion there is a lot of hot blog entries by the always interesting, funny and acerbic Thomas Ptacek, see here, here and here, and he includes links to the other side’s arguments. The Matasano blog is a must read for me.
I draw two terrifying SCADA security conclusions from this incident.
1. The dearth of control system exploits is likely to continue because there is little motivation for the vast majority of the white, gray or black hat community to expend effort on control systems. The footprint is minuscule compared even to the Mac. Many will have a false impression that the control system software has superior security to IT applications and OS.
2. A state sponsor or terrorist organization could easily find multiple exploitable control system vulnerabilities with a little cash. A few $50K bounties would be more than enough. They would not get the folks at Matasano or other reputable people to work for them, at least knowlingly, but there are plenty of others with talent and less scruples.
- For our Australian readers, the TISN is putting on a series of free SCADA Security executive and practitioner briefings in Brisbane, Sydney, Melbourne, Perth and Adelaide in June. TISN is similar to the US PCSF. More information and registration here. (Hat tip: Ron Southworth)
- Dark Reading, an IT Security resource, made a SCADA scare article their lead story on Tuesday. Nothing blogworthy in the article – very formulaic. Talk to a few experts, pick out the most sensational quotes … What to note is not the content in the article but the increased intention SCADA is getting in the IT press. Are we going to see an article in Forbes in 2007?
- Eric Murphy of Matrikon weighs in on the Part I of the OPC Security Whitepaper we wrote along with Eric Byres on his OPC Exchange blog.
- Rockwell Automation and Cisco announced they are developing a “reference architectures and detailed design guidelines for the use of common networking technologies across the production and enterprise network”. These documents will be helpful to maximize existing security features if you are a Cisco shop and even more so if you are a Cisco shop and have RA on the plant floor.
- Bruce Schneier has a blog entry and column in Wired that laments the lack of knowledge and information to make sound security product decisions and makes the case for security standards and independent testing. It is just as true in control system security, see our RA AssetCentre blog for an example. This is why we are big proponents (full disclosure: also paid consultants) on the Achilles Controller Certification. Asset owners should not and most can not assess the security of a protocol stack implementation. A related concern is the scrutiny of the security being designed into control system protocols such as OPC UA and DNP3. The process is an open standards effort, but there is a lack of crypto expertise on these working groups – – but that is a topic worthy of a separate blog entry.
- Finally, we probably would be remiss in not linking to the California ISO incident. Information is minimal and not very interesting, but power was lost to five computers apparently in the control center. They have arrested a suspect who was a former contractor. Watch out for those disgruntled insiders, especially security consultants. Fortunately this guy was not very clever.
Many SCADA and DCS vendors are integrating their applications with Microsoft’s Active Directory. There are some benefits to this:
- Control system vendors no longer need to develop and maintain user management system and other directory services (typically not a core competency)
- Support for strong, two-factor authentication
- Group policy to harden OS platforms
- Single sign-on
However one of the benefits we often hear – – using the enterprise Active Directory eliminates the need to reenter users or maintain two sets of accounts – – requires at a minimum domain controller to domain controller communication between the enterprise and control center security zones. This communication can be exploited and a new Microsoft DNS vulnerability vividly points this out.
With this new vulnerability, that to date does not have a patch, an attacker who has gained access to the enterprise would be able to compromise an Active Directory domain controller on the enterprise. Once the attacker owns this box, he could launch attacks at Active Directory domain controllers on the control system because this communication must be allowed through the firewall. Since the domain controller on the control system is also vulnerable, the attacker would own this box as well and have a launching point for attacks on all systems in the control center.
Sometimes a vulnerability is not even required to cause this enterprise to control system problem. Many of the control systems that rely on Active Directory also rely on Active Directory’s DNS rather than hardcoding or otherwise providing name to address resolution. An innocent DNS change on the enterprise could cause control system devices to no longer know how to find each other. Similarly, a Group Policy change on the enterprise could have a negative impact on control system computers.
Digital Bond and many others in the SCADA security community have recommended for years a completely separate domain for control systems, when a domain is required. This control system domain should also not be in the same Active Directory tree or forest as the enterprise domain.
a href=”http://www.scadapedia.com” _mce_href=”http://www.scadapedia.com” target=”_blank”–>SCADApedia – – subscribers can write; all can view.
New SCADApedia entries from April 1 – 15.
- Energy Sector Roadmap
- FactoryTalk AssetCentre
- FactoryTalk Security
- Interactive Energy Roadmap
- Innominate mGuard
- LiveData ICCP Server heap buffer overflow vulnerability
- Programmable Automation Controller (PAC)
- Rockwell Automation
- SISCO OSI stack fails to properly handle malformed packets
- SISCO OSI stack fails to properly validate packets
- Style Guide
- Vulnerability Notes
Updated SCADApedia entries
- IEEE P1689 (key management details)
There are now 20 Scadapedia entries. See a list of all entries.
We will list all new SCADApedia entries on or about the 1st and 16th of each month.
Have a suggestion for a SCADApedia page? Send an email to email@example.com
We added two SCADApedia entries on the security features of Rockwell Software Management: FactoryTalk Security (formerly RSAsset Security) and FactoryTalk AssetCentre (formerly RSMACC). The naming is still confusing with much of the documentation, website content, and RA customer and employee base still using the old names.
There is a lot to like about the security controls in these controller management solutions. In fact, there is little we don’t like “inside” these applications except perhaps the complexity, especially with AssetCentre, which may not be worth the time for organizations without large numbers of PAC’s. Some of our favorite features:
- FactoryTalk Security is a central repository of authentication and authorization controls. Users are entered and managed in one location rather than trying to enforce the same controls on each PC with the RSLogix application.
- The authorization is quite granular and can be based on 50+ actions individually settable for each PAC. The application supports groups for role based access control which makes life easier. Just because you can create an extremely complex authorization environment does not mean you should. Complexity makes periodic review and changes difficult and prone to errors with a security impact.
- FactoryTalk Security allows users to be entered locally in the application or it will proxy authentication to a Windows domain controller. If you have deployed a separate domain (in its own tree/forest unrelated to the enterprise), you can leverage that existing infrastructure. However, you are not forced to deploy a domain controller or, even worse, use the enterprise domain controllers and subject the control system to all the related risks. While Windows Active Directory supports numerous two-factor authentication solutions, we have not yet verified if these can be used with FactoryTalk Security.
- The change management system in AssetCentre is a huge benefit, especially if you have found your separate change management process to be ineffective or cumbersome. Achiving each change and knowing who made each change is great for forensics. Rolling back for recovery is simple. AssetCentre leverages FactoryTalk security authentication and authorization to limit who can do what.
- A periodic backup of each controller can be automated and scheduled.
- The Verification Compare action will verify that the master record in the AssetCentre matches what is running in the field. It will identify when someone has circumvented AssetCentre and made a change locally or by some other means (see below).
- AssetCentre supports an integrated RSLinx Gateway so all communication can be sent from the AssetCentre server to the controllers. This simplifies access control lists and firewall rules.
The Bad and Ugly
and it does get ugly . . .
The authentication and authorization features stop at management software.
Consider a top of the line FactoryTalk Security and AssetCentre example. A user with an AssetCentre client will login and be authorized at AssetCentre, make changes, and send those changes to one or more Logix controllers. The communication between AssetCentre and the Logix controller requires no authentication and has no security.
In fact, many of the great automated and scheduled features such as Verification Compare and Backup require the controllers to remain “unlocked”, not password protected.
Of course, many of you have leaped to the really big problem, why would an attacker bother going through the AssetCentre? An attacker would get a copy of RSLinx or RSLogix and go directly at the unprotected controllers as shown in the figure below.
The figure above shows the recommend compensating control, limiting access via a router ACL or firewall. This is recommended anyway for defense-in-depth, but let’s be honest, it will be a long time, if ever, until most of the community gets to this point. It is a lot faster and cheaper to provide some rudimentary access control features on the controllers.
Another compensating control is to use the AssetCentre Verification Compare feature to identify unauthorized changes, but the time between Verification Compares will be a time window when one or more controllers may be easily compromised and undetected.
What makes this even uglier is almost everyone we talked to assumed AssetCentre/RSMACC to controller communication was authenticated. The documentation is silent on this, not giving false information, but all the benefits listed in AssetCentre and FactoryTalk imply the authorization and authentication restricts who can manage/exploit the controllers.
To prove the perception point further, we talked to asset owners who purchased RSMACC and who assumed the authentication was required prior to controller access. We talked to RA systems engineers and third party integrators who assumed this was true, but never could explain how it could work when a CPU lock password was not ever entered by the user or in AssetCentre. After a few weeks we finally found someone at RA who knew the details.
Suggestions for RA
These were the unsolicited suggestions we made to RA last year. We do not know if any have been implemented (anyone in the know please let us know so we can update this and the SCADApedia) or are planned for a future release. To the best of our knowledge there are no announced plans to secure the AssetCentre to contoller communications.
1. Immediate Action
There is a relatively simple solution that is far from perfect but would be a dramatic security improvement. Integrate CPU Lock in AssetCentre. When AssetCentre is communicating with a controller it would unlock the controller, perform all actions, relock the controller.
This would not require any changes to the controller code. The CPU Lock password could be entered only in the AssetCentre, and users would be authenticated to AssetCentre with unique username and passwords via FactoryTalk. User and attackers could no longer circumvent AssetCentre.
Of course the controller would be vulnerable for other attackers during the time AssetCentre had it unlocked and the CPU Lock messaging is not secure. So it is not a long term solution.
2. Better Solution
Update the CPU lock function in the controller and AssetCentre to a secure authentication protocol and restrict unlocked access to the session that sent the valid unlock message.
This would still only support one account and have limited logging, but asset owners that require this could get those controls by deploying AssetCentre.
3. Best Solution
The standards bodies are working on this. It would include support for multiple user accounts on the controller, strong authentication, better error message logging, etc.
Of course, the other immediate solution from an asset owner perspective is to continue to use RSLogix with the CPU Lock utility and avoid the FactoryTalk solutions until this security issue is addressed.
Slow week on the SCADA security front.
- The Procedings of SCADA Scientific Security Symposium (S4) is now available for purchase on Amazon.com in addition to on our web site.
- Dick Caro has a good review of the latest SP100 meeting that took place in Germany at the ControlGlobal site.
- Walt Boyes from Control Magazine has moved his blog. More importantly, it now has an RSS feed.
We just finished a series of SCADApedia entries on security in Rockwell Automation (RA) controllers and software applications. Remember the SCADApedia is a place for facts, so I’ll lay out some opinions and conclusions in this two part blog.
- The ControlLogix PAC (powerful PLC) is a prime example of why we are fans of the simple, little IEEE P1686 standard effort. The Logix family only supports a single password that locks and unlocks the PAC through a CPU Lock utility, that quite frankly was not integrated with any management software at least through Version 15. P1686, which is for much simpler IED’s, requires support for ten UserID and Password combinations. Maybe P1686 is a low bar, but it would be a dramatic improvement in the security in most controllers including the RA family.
- If anyone can confirm the CPU Lock feature was integrated into RSLogix in Version 16 it would be appreciated. Screen shots and documentation hints at this. Also, how about RSLinx?
- In our informal survey of RA users, RA personnel and integrators, it was clear that even the limited security features in the Logix PAC’s (CPU Lock, Source Protection, Priority and CIP limits) were unknown and unused by most. One of the most common recommendations was to turn the physical key to Run and remove the key to prevent any remote changes. Great control, if practical in your environment. Of course, this is not possible or prohibitively expensive in many larger or geographically dispersed systems.
- Does previous point mean security awareness efforts are failing? Are Joe Weiss and others correct to continue to hammer the basics at PCSF and other events? Perhaps awareness has been achieved in a narrow percentage of the community – – the ones that attend events, contribute to standards, read SCADA security blogs, etc. The challenge is building security awareness amongst those who are not part of the control system security community, which would be the majority of control system users.
- I’m uncertain what to recommend for limiting CIP connections. The ControlLogix supports up to 128, and has a feature that can set a lower maximum. So if an asset owner knows their should only be ten valid CIP connections at most, a maximum could be set. However would this make it more or less susceptible to denial of service attacks? If forced to choose, I would recommend setting the limit because it would be so easy for an attacker to exceed the 128 maximum if denial of service was the goal. Setting the limit may identify an attacker who is trying to go low and slow.
- RA should be commended for making huge amounts of documentation available. In fact, I cannot think of any other control vendor where we can read huge manuals on all of the products and options. Often this level of detail is not even written let alone available. RA does need to retire superseded documents or mark them obsolete because they conflict with current information. For example, we read that CPU Lock was not supported on the Ethernet interface in multiple documents when in fact that was old news.
Part II will cover opinions on security in the Rockwall Software management applications. We have some very strong and important pro and con opinions on this.
Note: This series of RA blogs and SCADApedia entries was the original impetus for the SCADApedia. We were frustrated on how difficult to get accurate basic security guidance. Once we had the information, we had a lot of opinions in addition to the facts. We didn’t to mix the two, and we didn’t want the facts in the blog entry to age off and be of little use after all this work. Hopefully this helps our current and future readers.
Pauldotcom’s latest security weekly (episode 66) elaborates on the usage of bluetooth in devices other then mobile phones. Apparently some vendors have integrated bluetooth into pole-top devices like transformers for monitoring purposes in the UK.
I’m not all that surprised about it being used for monitoring, but what about programming? Hopefully the companies purchasing and deploying these devices are doing their security homework!