The folders that ICS applications are installed in are usually configured as exclusions to anti-virus scanning.
In some cases, the almost constant updating of the ICS data files leads to unacceptable performance if subjected to anti-virus protection. In other cases the vendor chose to avoid a potential, yet unseen problem.
To make this problem worse, the permissions on the ICS application folders are typically far from least privilege. Full Control for Everyone is not unusual for a default install. Folder permissions was an area we spent a lot of time with ICS vendors in developing the Bandolier Security Audit Files. They can be locked down, but rarely are.
We have not yet seen mass market malware seek out ICS application folders that were typically excluded from anti-virus scanning. However, a directed attacker might put his malware in these folders to prevent getting detected by a future signature.
Our Stephen Hilt released another Project Redpoint script as part of his DerbyCon presentation on Sunday. Modicon-info.nse will identify PLC’s and other Schneider Electric/Modicon devices on the network and then enumerates the device.
The script pulls information that would be helpful in maintaining an inventory as well as assessing the security status of the device, such as types of Ethernet, CPU modules and memory cards as well as the firmware version.
My favorite part of the script is the Project Information field. Here you will see the name of the Project, the version number of Unity Pro (the engineering workstation software) that was used to load the Project, and often where on the engineering workstation the Project file is stored. Below is sample output; click on the picture to see a larger version.
This script is possible due to Schneider Electric’s proprietary and magic function code 90. This is the same function code that Reid used in the modicon_stux_transfer metasploit module to upload and download logic to the PLC a la Stuxnet. You can do almost anything you want to the Schneider PLC’s through this unauthenticated, insecure by design function code 90.
The script begins using function code 43 to identify if there is a Modbus device at the targeted IP address. Schneider, unlike many vendors, supports function code 43 and will return some variant of Schneider in the response message. I should note that even if the Modbus device being queried does not support function code 43 the response can confirm it is a Modbus device.
We had an internal discussion on whether the script should include all identified Modbus devices, but decided to only report on Schneider/Modicon devices since there were already Modbus detection capabilities in Nmap. If we can pull additional useful information there may be a generic Modbus script in the Redpoint rack in the future.
There is a truism in information security, and it is that everything will eventually be found to be vulnerable.
I believe the lesson here should be, ‘plan to patch.’ It is tragically common in the embedded device space that vendors don’t take this advice. There is still an awful lot of embedded industrial control systems equipment being manufactured today which has no way to even apply update.
Today’s big news story in the infosec space is the ‘Bash bug’. In a nutshell, the bash bug is a mistake in command-line processing. A lot of embedded industrial control components will end up being affected. Basically, any industrial control system that runs embedded Linux, and which features a protocol that ends up calling GNU utilities will likely be vulnerable. Primarily the vulnerability will affect webservers that allow configuring and reading interesting information from a device, and protocols such as potentially CoDeSys which may end up calling other applications by using a shell for some vendor’s products.
Legally speaking, any control system vendor which sells a device running GNU software has to provide a notice with the device informing the end user what software is in use (and that the source code to said software is available from the vendor). Unfortunately not all vendors play nice by providing this notice. The only real way to know what is vulnerable is to test it.
Digital Bond Labs has a nice test environment with a variety of equipment in various forms of hackedness. One such device is Wago’s 758-870 series PLC. The product runs Linux and includes a version of bash that is vulnerable, as demonstrated above. It also runs an embedded webserver which executes cgi scripts (even calling execve() during some webapp command executions), so we will likely find a way to exploit the bash bug on these systems. Although, this system already has documented backdoor accounts, and Wago has already decided that they will not produce firmware updates for this product, so exploiting the bug here really has no point.
I think that the lessons we can draw from both the Bash Bug and Heartbleed is simply that vendors need to consider security upgrades in their product design. Bugs such as the Bash Bug provide a potential way to gain command-line access to some of these embedded systems. This access may be the only thing preventing unauthorized access to or even unauthorized cloning of a vendor’s product. Vendors owe it to themselves to protect their intellectual property, and owe it to customers to provide patches when the inevitable happens.
Be sure that whatever product you are rolling out to your control systems environment can at least have upgrade applied. Worrying about when you can apply a patch is a much better problem than worrying whether your IDS/IPS rule can be evaded because the patch will never come.
Pumpkin image by kams_world
The clock is ticking to get your session proposal in for S4x15 Week. Take a look at the full CFP and get it in by October 1.
We don’t just wait for the CFP responses. We actively chase down researchers and topics. So if you see something that is S4-worthy please send us an email.
I’ll take it a step further. If you have any idea for a S4 session, a Great Debate topic, onstage interviewee or proven good practice for OTDay, send us the idea and we will find the right speaker.
One other S4x15 Week note … we will have a slight increase in prices, our first since 2007. So the best way to get in for free is to present a great session. We also will have group pricing and the first 50 registrants will see no price increase. More on registration on October 1 after we finalize the agenda.
David Perera of Politico released a good article yesterday on the difficulty of taking out the electric grid. Unfortunately the headline writers missed the mark, “US Grid Safe From Large Scale Attack, Experts Say“, and it is difficult to write two very different points in one mainstream press article. Let me try with our ICS security focused audience.
Point 1 – Taking Down An ICS Doesn’t Necessarily Cause A Catastrophe
The article did a good job of capturing this point, but it is broader than the electric grid.
- Some ICS will continue to run just fine if large portions of the control systems are lost, particularly servers and workstations.
- There are often safety systems to prevent really bad things from happening. Admittedly the quality of implementation of these safety systems vary a great deal.
- Some of the safety measures cannot be changed over the network or even serial connections.
The skilled offensive cyber adversary / hacker will likely take control of the insecure by design and fragile ICS if he has network access, and he will be able to take all or part of the ICS down. The Operations Group will not be able to use the ICS to monitor and control the physical system. The impact of this will vary by sector and system.
Take down some electric distribution SCADA systems and there will be a delay in knowing about an outage. Take down a pipeline leak detection system, and they will likely shut down the pipeline in a few hours. Take down a gas or electric meter reading SCADA, and they will estimate bills and perhaps send people out for a manual read. Take down a turbine control system and that unit in the plant will likely not generate power until it is fixed. Take down a food manufacturing plant control system and some will run on manual operations, while others will be shut down.
The key point that David’s article captured is just because the ICS that run generation and transmission in the power grid are insecure by design and fragile it does not mean that even a skilled hacker or researcher can cause a widespread blackout.
Point 2 – The US Grid & Other Critical Infrastructure Are Definitely Not Safe From The Right Team Of Attackers
With the addition of ICSage: ICS Cyber Weapons to S4 Week we have been thinking a lot about nation state or well funded offensive security teams going after critical infrastructure ICS. We believe it would consist of:
- A “Hacker”. Actually the easiest job as Dillon Beresford, Project Basecamp and others have demonstrated.
- An Engineer. They need to understand the process or system that is being attacked, and determine what would cause the damage they desire. This could be expensive, hard to replace physical equipment damage that would cause a long term outage. Release of materials harmful to people and the environment. Damage to reputation. Or something subtle like Stuxnet that causes a maintenance or equipment failure issue that is costly, difficult to diagnose and saps confidence in the process.
- An Automation Expert. Once the Engineer has determined what should be done, and the Hacker has provided access to the ICS, the Automation Expert has to write the logic to implement the attack. This could be logic in a PLC, changed displays, database changes, or a variety of other ICS modifications. This is a real challenge since the Automation Expert likely cannot simulate the process completely. This may have been the most impressive aspect of Stuxnet.
I’m seeing a major shift that started at S4x14 and is continuing at S4x15 to the engineering and automation aspects of attacking and defending ICS. S4x13 showed exploit after exploit of vulnerable ICS components. The leading researchers have moved beyond that and are now looking at what to do with the owned ICS and how to defend against the really bad things a skilled attack team would want to do.
What David’s article probably couldn’t tackle is the somewhat conflicting ideas that while a highly skilled hacker or researcher likely couldn’t cause a catastrophic impact to a critical infrastructure ICS, the electric grid and other critical infrastructure is highly vulnerable to a talented and motivated team with the right mix of skills.
The vaunted safety systems often have holes in them, and the people on sites can usually tell you how they would cause long term damage to the physical system. Just a couple of examples:
- Safety systems are often implemented in safety PLC’s. These are your typical insecure by design PLC’s with extra redundancy. And there has been a push for years now by some vendors to integrate the control and safety systems. Change the safety logic and it will either stop the process when it shouldn’t or fail to stop the really bad things from happening.
- One of my favorite examples is vibration monitoring. This is often a separate system or application, such as Bently Nevada. It can be configured to trip a turbine or some other physical system if vibration reaches a certain level. Simply change the trip value, set it to a constant value, change the scale, … and it doesn’t provide the proper safety function.
- Or the safety system was designed to stop problems that have seen by equipment failure or human error, but they never considered what an active attacker would do. This is why efforts to take the ICS Safety Approach with ICS Security has never worked.
All that said, David did his homework and wrote a good article. Perhaps a better title would have been “Hackers Would Have A Very Difficult Time Taking Out US Power Grid”.
We have been working with author Rob Lee and the very helpful Richard Stiennon to translate SCADA and Me – a book for children and management into Japanese. Attendees at our S4xJapan, Oct 14-15 in Tokyo, will receive a free copy of this fun book. It’s being printed now, so enjoy a few of the galleys below (click on a picture to see full page in more detail).
We also have the full agenda translated into Japanese now with the very kind help of Mai Kiuchi of the Cyber Defense Institute. Kiuchi-san will be assisting with the Q&A portion of S4xJapan as she is fluent in English, Japanese and ICS security.
I spoke at the inaugural ArchC0n in St. Louis this Saturday. The main reason I chose to go to this IT security event was they had Richard Bejtlich, Bruce Schneier and Charlie Miller as keynotes. Quite a haul for the first run. Here are some of the items that I wrote down:
- Richard Bejtlich’s talk had an interesting factoid. When Mandiant goes in and looks at compromised networks they often find multiple, unrelated attackers who have compromised the organization’s systems. The record so far is seven independent attackers on the same network.
- Kyle Wilhoit gave the IT version of Malware Incubation, although it showed how he used it to learn more about Havex. He will give the ICS version at S4xJapan next month. Kyle is working on an incubation system he calls ChickenHawk. I can see a lot of applications for having a malware incubation environment for researchers and asset owners. More about this as an S4xJapan preview, but suffice it to say this is a highly interesting project.
- Liam Randall of Critical Stack talked about using Bro to identify and react to attacks on ICS. He is just scratching the surface of what is possible here, and as a Bro-master he should be able to do some great things with that platform. We are hoping to see him at OTDay S4x15, and are considering if this should be one of the deep dive classes on the Friday of S4 week.
- Charlie Miller gave a funny but depressing talk on 2007 vs 2014, and how difficult it is to tell them apart from a cyber security standpoint. I did leave with a positive feeling about the impact Digital Bond Labs can make finding and fixing vulns before they get deployed. Also had a chance to talk with Charlie about his car hacking … that is an expensive and difficult to create and maintain test environment.
- Bruce Schneier had, as usual, a couple of thought provoking data points. One was from a TED talk, skip to 10:50. In brief, you have $1000. You can keep it or flip a coin for double or nothing (heads=$2000, tails=$0), about 75% play it safe and keep the $1000. Test 2 is you owe $1000. You can pay the $1000 or flip a coin (heads= you owe $2000, tails=you owe $0). 75% take the risk and flip the coin because they have a chance to loss $0. This is called Loss Aversion, and it could be related to why people choose not to spend money on security. If they spend $0, then they have not lost anything and there is a chance they will not get hacked or compromised.
The agenda is up and registration is open for the first S4xJapan, Oct 14-15 in Tokyo. There is space for 100 people so register now to get your spot.
Tuesday, October 14th is Operations Technology day (OTDay). Attendees will learn proven techniques to run a reliable and secure ICS. There will be sessions on virtualization in ICS, unidirectional gateways, wireless on the plant floor and more. I will have a session that shows how to use assessment tools on an ICS in production without causing an operational impact and obtaining maximum information.
We are proud to announce that the Kaspersky Industrial Protection Simulation (KIPS) will take place as the last session of OTDay. Kaspersky graciously has translated KIPS into Japanese and will have a team of native Japanese speakers to lead everyone through the simulation. KIPS has received great reviews at ICS events, and we are pleased to bring it to Japan for the first time.
After KIPS is open stay and enjoy food and drink and connect with fellow ICS security professionals at the S4xJapan social event. Also enjoy a great view of Tokyo at night from the 49th Floor of the Mori Building in Roppongi where S4xJapan is being held.
Wednesday is the main day of S4xJapan where we move from good security practice to leading and bleeding edge ICS security research. They include a variety of perspectives:
- Offensive – Reid Wightman’s session on Vulnerability Inheritance where he will show examples of third party software integrations leading to compromise in Japanese ICS
- Defensive - Wataru Machii’s session on dynamic zoning based on operational mode
- Intelligence – Chris Sistrunk and Kyle Wilhoit showing new data from observed ICS attack techniques
- Education – Learn in detail what Havex does to ICS applications and devices
- Tragedy – See the survey of Internet accessible ICS applications and devices in Japan
We will be previewing many of the sessions and other S4xJapan information on the digitalbond.com and digitalbond.jp sites.
Some of the delay in opening the registration is we have been working hard to make this a Japanese event. Of course there will be simultaneous translation English/Japanese or Japanese/English as necessary, and we searched hard to find the best individual translators with security and technical knowledge. But is more than just that:
- approximately half of the sessions are in Japanese, and the other half are in English.
- the presentation slides will have key content translated. I saw this at one JPCERT session, and it was even more helpful than the simultaneous translations.
- an ICS security expert fluent in Japanese and English will handle the Q&A. Q&A translation of technical questions is a common failure.
- we have an Internet based Q&A engine so attendees can ask questions anonymously if desired.
- The Kaspersky KIPS is in Japanese.
- more surprises to come.
If you have any questions or difficulties registering contact us at email@example.com.
The S4xJapan registration, Oct 14-15, opens on Monday morning, Tokyo time. We have been working hard to make this a Japanese event in terms of session focus, language and fun. For example, Kaspersky generously translated their KIPS experience into Japanese for the event. Only 100 seats, so be ready.
The big news of the week is from Norway where 50 companies in the oil sector were hacked. Not a lot of info on this yet, but on the heels of Havex there appears to be some serious targeting of the European energy sector.
According to Dave Aitel, the new VulnDisco Pack for Canvas has a number of new ICS exploit modules. This is in addition to the Gleg SCADA+ pack.
Matthew Luallen of Cybati opened a Kickstarter campaign for an ICS security training kit. The goal is $30K, and it is about 10% there after the first week.
Some of Digital Bond’s Redpoint scripts are in Nmap release 6.47. There is typically a three to six month lag from when we release them on Github until they get integrated into Nmap. Stephen is busy working on the next protocol script.
In case you missed it, there are two quality ICS security events in Europe this fall. The SANS European ICS Summit is Sept 21-22 in Amsterdam, and the inaugural 4SICS is Oct 22-23 in Stockholm.
Last week Stephen made a minor, but very helpful, update to the Redpoint script that identifies and enumerates BACnet gateways and devices. All publicly available Redpoint scripts are on our GitHub, and some of the scripts have been integrated into the nmap download.
The latest version has the option to pull the Foreign Device Table (FDT) and the Broadcast Distribution Table (BDT). Both are helpful in enumerating BACnet devices on different, and possibly inaccessible to scanning, subnets.
Imagine the case where you have a BACnet device on the corporate network that is used by the team to view the status of an otherwise segmented building management system from their corporate computers. The BDT and FDT may help you identify those non-accessible devices.
Any time a BACnet network consists of more than one subnet, each subnet must have a BACnet Broadcast Management Device (BBMD). Each BBMD in the BACnet network has an identical Broadcast Distribution Table (BDT) that lists all of the BBMD’s in the network. So by recovering the BDT you will learn all the subnets that have BACnet devices in the BACnet network.
Well, not quite. There is another way for a BACnet device on a different subnet to join a BACnet network … by registering as a foreign device. To fully participate in the BACnet network the foreign device should register with a BBMD. However the foreign device can register with any device that supports foreign devices, and most BACnet gateways do.
So the Redpoint script can also pull the Foreign Device Table (FDT), which is useful in identifying BACnet devices and possibly even attackers.
Each entry in the FDT is suppose to have a Time-To-Live for each registered foreign device, and then erase foreign devices that don’t re-register in that time period. In practice we have found that many foreign device entries never time out.
Let’s look at a practical example from Redpoint output: Read More