CoDeSys Publicly Responds, Honest but Sad

It is hard for me to write it any better than 3S, from their site:

In general, we do not offer any standard tools in CODESYS which are to protect the controller from a serious cyber attack. Should the offered password functionality suggest such a protection, this was definitely not our intention. The implementation of standard security mechanisms (firewall, VPN access) is an absolute must when operating a PLC runtime system on a controller accessible through the internet.

This statement is accurate, but a completely foolish position to take as a software vendor. So an attacker that has network access to a device with your software has the ability to get application command shell, upload and run any files they want, compromise the integrity and availability of the process, and pivot from the PLC/controller and attack the rest of the ICS.

Equally ridiculous is the fact that 200+ ICS vendors implemented the CoDeSys runtime and 3S never felt any obligation to point out that the authentication you see in the application is only client side and not passed to the server. BTW, this is common in the ICS world. Another pertinent example is Rockwell Automation’s FactoryTalk. Or that a company selling a library never felt any compunction to provide security related application guide.

The brief 3S statement also alludes to this problem affecting Version 2.3 with the unwritten implication that it does not affect Version 3. The system in our lab is Version 2.3, but we had access to a few other systems and all were vulnerable save one vendor solution that implemented their own security envelope to require authentication. Since 3S did not “offer any standard tools in CODESYS to protect the controller” we believe all versions lack authentication to access the application command shell, transfer files, change logic, … And remember that the application usually runs as a highly privileged account directly on the OS.

Reid pointed out the big endian / little endian issue with his tool. So even if the tool does not work on your particular CoDeSys implementation, it still likely is vulnerable. Every vendor may implement it slightly different, so a universal tool is unlikely to work on every product. You should assume your version is vulnerable unless the vendor has specifically implemented their own authentication or other security measure to limit access to the CoDeSys ladder logic runtime software on the PLC/controller.

It’s an ugly problem now with 200+ vendor implementations and a large and diverse deployed base. Even if 3S offers a patch, they need to go the extra mile and start tracking what vendors have implemented the fix.

Tenable Network Security put out a Nessus plugin to detect this vulnerability. Again given the large number of vendor implementations I would expect a significant number of false negatives. Still they did what they could and deserve kudos.

Oh … its nice to see on the CoDeSys page that Sandvik Mining and Construction Drill Rigs are running CoDeSys. Hit refresh and you will see a bunch of other companies using the product that does not “offer any standard tools … to protect the controller”.

And if you can take one final comment – it shows the power of threat modeling. The vendor we know that is not vulnerable (or at least significantly less vulnerable) to this threat put in security controls because they found this problem during threat modeling. I wasn’t there for the process, but just finished a detailed threat model project. This would have jumped out when the entry points were listed.

23 comments to CoDeSys Publicly Responds, Honest but Sad

  • I am a bit disappointed that Digital Bond publicly discloses vulnerabilities like these, and while making an issue that these are such serious vulnerabilities, they fail to even offer basic compensating controls like updates to the QuickDraw IDS signatures.

    For the same reason that I believe these tools are useless if not maintained, doesn’t it make sense to offer users of these devices more guidance than just “your device is vulnerable and you should replace it”???

    I am finalizing a white paper on this topic with Eric Byres at Tofino Security where we will provide not only IDS signatures to detect both discovery and attempted attacks, but also a Security Policy that can be used with the Tofino Security Appliance to protect these devices from such an exploit attempt without touching the field device! … all while still allowing standard engineering programming and maintenance functions through to the device.

    Stay secure …

  • Ahh Joel to the rescue. We’re glad you’re here to stroke Eric’s Tofino Security Appliance.

    Without public disclosures, how would you and Eric have signatures for your ‘appliance’, you certainly aren’t finding this vulns.

    I’m disappointed too. Disappointed that you feel the only way to hock your training and your buddies appliance is to bash on others.

    Stay classy…

  • Bill G

    @Joel – I don’t see the benefit in not making these vulnerabilities public. As with vulnerabilities in general, publication is common practice. Why not for ISC products? I think ICS (and embedded device manufacturers in general) have been given a lot of leeway in the past – often with little or no response. Disclosing to the vendor and expecting dissemination to their user base is foolhardy. I’m glad Digital Bond is making the ICS security issue more public.

  • I can confirm that their runtime version 2.4 (which might be CoDeSys v3?) is affected.

    In my opinion 3S software has offered a lot of misdirection on the vulnerability. The vulnerability was first announced in April, and six months later the 3S solution was to enable a (useless) password. Once the tool was released, 3S confessed that the password didn’t do anything.

    The rumor mill says that we should expect an update to the CoDeSys runtime in November-December. The update might even fix the password mechanism so that it does something useful. This won’t help most of the current PLCs, though…My WAGO for example has no feature for performing a firmware update.

  • Dale Peterson

    Joel – I’m disappointed that you and many other respected ICS security gurus refuse to push ICS vendors to provide basic security controls like source and data authentication. Instead you focus on third party, add on solutions. But I guess we will both have to live with our disappointment.

    IDS/IPS signatures, firewalls, etc won’t help with this type of vulnerability unless you are going to block functionality for all – good guys and bad. It would be like the physical program switch of old. All those controls would do is detect when a dangerous, rare but occasionally required request/command/action took place or limit those request/command/action to a set of source IP addresses.

    There are values in those controls and products, but our focus is getting the ICS community to recognize and address the root problem. We are now 11 years with little or no progress.

    Dale Peterson
    Digital Bond, Inc.

  • AnotherAnon

    The authentication is not only client side! It is actaully sent to the device in normal operation. But it’s a pretty poor implementation.

  • For the record … this has nothing to do with pushing training or a product, but rather dealing with the hard fact of life that ICS systems will have vulnerabilities that will have to be managed with compensating controls until such time that they are deemed “secure” by those that like to place such labels on them. It appears everyone wants to make this personal, but it is not.

    So I ask Reid and Dale and everyone else who wants to exchange meaningful dialogue on this subject… “What should an end-user do know that you have told the world that there devices are vulnerable to attack?” Many of us are still waiting for an answer to this.

    That is why security needs to focus as much on Defense (direct and compensating controls) as much as Offense (vulnerability disclosures and exploit code).

    This is why I am equally critical at organizations like ICS-CERT that refuse to publish meaningful “suggestions” regarding compensating controls that users who are so desperately seeking guidance can use. Sure, we all know that these devices have problems.

  • Anon E Mouse

    “What should an end-user do know that you have told the world that there devices are vulnerable to attack?”

    Push their vendor to fix the problem. Ask the vendor why it took a third party to discover/notify on the problem. Evaluate their system from a security operations standpoint to determine if this has an impact. Apply a mitigation strategy.

    Or we could just keep these nasty little secrets to ourselves.

    Think like a hacker…™

  • Dale Peterson

    Joel – I wrote a four part blog series back in February on what asset owners should do with new projects, what asset owners should do with deployed systems, what vendors should do, and what government/industry organizations/standards bodies should do.

    http://www.digitalbond.com/2012/02/21/what-should-you-do-part-4-gov-stds-orgs/

    Dale

  • Gentlemen, we can criticize, oblige, shame, insult, scream, and throw tantrums until we’re blue in the face –but it doesn’t change the reality that this stuff is already deployed in very large numbers, and nobody even knows that the problem exists.

    It isn’t worth the effort to fix this product. We know it is lacking. People are designing better ones, but the market for this sort of thing is slow to take hold, just as the market for seat belts in cars was slow to take hold. Nothing we say can change it. All we can do is to study what broke and design better things.

    Publishing a vulnerability like this does very little to fix the problem. It would have been better had you consulted with ICS-CERT and attempted to work this from the inside. As things stand now, you have given world a giant flashing sign that says here is how you can break in to the middleware built in to more than 200 different OEM products. And most end-users have no clue what is going on, and very little way to learn any of this.

    Least you think we’re slacking, I need to remind you that these people do have plants to run, things to fix, costs and personnel to manage, regulations to comply with, and many many other daily tasks. This is but one of many things that can rise up and bite asset owners.

    Let’s keep this problem in perspective and remember that the goal is to improve security for industry; not to annoy vendors, asset owners, government, and the like.

  • That blog back in February was well-written, but considering the industries that I am closest to including refining, oil & gas, pulp/paper, chemical, and pharmaceutical … it really does not reflect how asset owners typically perform ICS migrations.

    Let’s look at many of those that have implemented ICS in the past … it is not common for companies with a large installed base to include controllers in their migration strategy until vendor support is no longer available. I walk into sites all the time that have controllers installed in the 70′s and 80′s that are still working to peak performance – and in many cases better than today’s replacements! Sure, many have replaced the “top end” used for the visualization and application framework, but until parts replacement and support becomes excessive, there is typically no financial incentive to replace the field devices. Unless you perform a complete process re-engineering activity, little ROI is gained from migrating a control loop from one older platform to a newer one. In fact, many TCO calculations actually show this as a “negative ROI”!

    So, though I understand what you are trying to say, I just don’t see it based what is being done in the field.

  • I’m all for compensating controls as a stop-gap solution. What I really want to see though is defense in depth: good perimeter security *and* equally good host security. Even ‘not equally good host security,’ would be better than we’ve got.

    All I’m asking for is authentication of privileged operations. I don’t care if the authentication is plaintext…let’s get *some* security and *then* we can start beating up on it. I think where Joel and I differ is that he views IDSes and firewalls as *the* solution, while I view them as fine and dandy but ultimately they’re not the whole picture.

    I’ll just lay out my arguments for disclosing here in as logical a format as I can muster in between firmware reversing sessions. Feel free to disagree with my statements (and let me know which you disagree with, they’re numbered for convenience :)).

    0 [Premise] All security can be defeated, but it’s still good to give security your best shot.

    1 [Premise] Proper defense in depth means both perimeter security and basic host security.

    2 [Premise] Hosts include anything with an IP address.

    3 [From my PLC User Manuals] PLCs have IP addresses.

    4 (From 2,3) PLCs are hosts.

    5 (From 1,4) PLCs should have basic security as a means of implementing proper defense in depth.

    6 [From Basecamp] Most PLCs don’t currently have basic security.

    7 (From 1,5,6) Most PLCs currently restrict us from implementing proper defense in depth.

    8 [From History Books] “The industry” (aka guys and gals like me, Dale) has tried for a decade to convince PLC makers to add basic security with little effect.

    9 [From History Books] “The industry” has historically not publicly disclosed vulnerabilities for fear of damage or negative reputation.

    10 [From Common Sense] Public disclosure is embarrassing to vendors.

    11 [Induction] Public embarrassment, or fear of public embarrassment, convinces vendors to implement basic security (as an aside: at the cost of a bit of reputation for the discloser).

    12 (From 1,10,11) Public embarrassment may convince vendors to give their customers the tools to implement proper defense in depth.

    13 [Induction, History] There is nothing so permanent as a temporary compensating control.

    14 “The industry” has viewed compensating controls as “a solution” to its PLC problem.

    15 A solution to PLC problems decreases vendor embarrassment.

    16 (From 11,13,14,15) Providing compensating controls disincentivizes (that’s a word, honest) vendors from adding basic security to their products.

    17 (From 1,17) Compensating controls hinder development of proper defense-in-depth.

    That’s pretty much my argument for why I do public disclosure without compensating control information. I think that it’s healthy for the industry to sweat a little, and that it’s going to help everybody in the long-term.

    When I read Joel’s comments I see things like, “My detection cant be evaded,” which is an impossible statement given Premise 0 above but I’ll let it slide as excitement at a neat detection system.

    My challenge still stands, Joel: give me your security system and I’ll produce a tool to evade it, or at least force you to detect legitimate traffic as malicious, in far less time than it took to implement the security add-on.

    I’ll issue my own challenge in return: refute my proof above and show us that either (1) adding basic security features to PLCs weakens the overall security of an ICS network somehow or (2) that providing compensating controls along with the vulnerability will more strongly motivate vendors to implement security in their PLCs. All I’m asking is to prove that what we’re doing is antithetical to our desired outcome (who knows, maybe I’ve got some nasty logic errors hard-coded in my MPU).

  • I always enjoy a good dialogue, but do not appreciate when text is taken out of context … much like most of our politicians are quite good at doing.

    What I said was “given your exploit, I have a vulnerability that will detect and block it without impacting normal engineering activity”. Considering that you have not seen it, nor worked with the equipment I am using, don’t I deserve any credit or benefit of the doubt here?

    The point is to protect our endpoints as best we can based on what we known regarding the current threat landscape. I believe that I have done that … but again, Reid nor Dale has ever actually seen this implemented and working on actual equipment.

    I never said that firewalls and IDS were THE answer, but rather a very good strategy that is part of defense in depth, and when DiD is implemented using solid risk management principles, it is very very hard to evade.

    Everyone seems to think authentication is the answer … well … think again. Few authentication mechanisms have not been cracked, and now try to implement anything advanced on ICS field hardware and you have another game all together. Authentication may be A SOLUTION but it too is not THE SOLUTION. This is the whole idea behind DiD which always seems to drop from the responses that discredit what I am trying to get across.

    It just would be nice if those that claim to be ICS Security Experts could offer actual ICS Users some advice other than rip and replace all that physical and intellectual property that you have invested for something that may be secure today but may not stand up against tomorrow’s threats.

  • Let me amplify a point that Joel is making: Authentication is merely an opportunity for a denial of service.

    True, it prevents potentially toxic things from being done, but it also makes denial of control attacks much easier. Before using authentication one should consider what to do when authentication fails. For real time systems (DCS, not SCADA), this is a serious problem. They are generally built with the notion that there WILL be a message every so many milliseconds.

    The whole control strategy will need to be rethought as to how these zones break apart when the conduit fails. The SCADA systems out there will have a slightly easier time with this exercise. Communications are not automatically considered to be reliable, so there generally are lost communications situations embedded in those processes, though some scenarios are subtly different in the case of failed authentication.

    The implicit assumption behind authentication is that you know what to do when it fails. But many processes are not developed in that area. Unlike the office, when authentication fails, doing NOTHING can be very dangerous.

  • Joel -

    Give my comment a read. It’s long but worth the investment. In the end I feel that you’re taking a lot of what I say out of context (particularly your use of ‘a’ and ‘the’).

    What part of my argument do you disagree with, specifically? Point to a number, and then we can start talking about it.

    Reid

  • Thanks jumping in Jake – it is refreshing to have a voice from the field in this dialogue!

    I think that many of these discussions are more caused by not considering the same baseline versus the technical merits included. I am a DCS guy, and have always said that. What we do in the DCS side of the ICS spectrum is very different than the SCADA side, and is probably why this is a problem. I tried to point out yesterday in other Digital Bond post criticizing Achilles certification that many of the DCS devices already use authentication, especially those that have Achilles certification – yet the certification is then put down. The example I provided actually showed where the vendor went way above and beyond that required to just meet the cert. Hats off to Honeywell and their C300, FIM and SM!

    Also, I want to specifically point out the exceptional job that Siemens has done in addressing this very point – authentication – and their new enhanced communication processors for the S7-300 and S7-400 PLCs. I have reviewed these products personally, and believe that they are a step in the right direction … yet no one seems to give them any credit for learning from the past, investing in their products, and delivering on this issue all in less than 2 years.

  • Dale Peterson

    Joel – this is related to your 15:29 comment (sorry was out chopping wood for the winter). Our disagreement is actually simple. You don’t believe that critical infrastructure owner/operators should replace PLCs and other controllers with models that are not insecure by design. I believe they should because the risk is too great to live with until they stop working.

    Actually it makes financial sense to replace the insecure by design PLCs when you look at the risk. The cost of the physical infrastructure in the plant, pipeline, refinery is huge compared to the cost of replacing a PLC. Even the cost of centrifuge or turbine or large pump, substation, etc. is large compared to replacing or upgrading the PLCs. This does not even include the economic damage to the region, potential loss of life, and environmental damage – which is where the risk based incentive breaks down when the impact exceeds the value of the company.

    Where we probably have some common ground is new installations. Isn’t it pathetic that an owner/operator putting in a new system still has to buy an insecure by design PLC? 11 years after we supposedly started caring about critical infrastructure cyber security? You can’t buy a PLC with basic security features from GE, Schneider, Rockwell Automation and most others. Not even there top of the line model. I get huge heartburn when my customers buy new products that you say will be around until they are obsolete (20 years? 30 years?) that lack even basic security controls.

    Two other things before I stop the rant:

    1) We, the ICS community, know how to build secure controller. Just look at a WirelessHART or ISA100.11a device. It has authentication and optional encryption (mandatory in WirelessHART).

    2) The “you don’t understand the field or ICS” is a lame excuse that we need to shelve. First of all where do you think Digital Bond made our money the past 12 years. It has been with owner/operators and vendors helping them secure real systems. But even if we hadn’t, it is a lame excuse that is too often used to dismiss an argument. We have people coming into the ICS security community from a variety of fields. We need them in this community because it is way too small. If you think I or anyone else is wrong, explain why.

    Dale Peterson
    Digital Bond, Inc.

  • I guess we can just disagree again and since this has gotten personal … I will end it here. Unfortunately … our relationship has probably ended here as well.

    Having been involved in the capital justification and funding of many migrations, I don’t think you are looking at the total cost inclusive of risk in migrating field controllers. When looking at the risk exposure in industries that require high uptime and require hot cutovers, the actual risk of such a migration can be far greater than the risk reduction that can be introduced with acceptable security controls specifically designed to mitigate threats to vulnerable endpoints. You continue to mention PLCs, where I am specifically talking about BPCS and SIS controllers in most integrated control systems like DCS’s that are used in so many industries. It involves much more than just replacing some hardware, but re-engineering and re-applying “thousands” and “tens of thousands” of manhours of intellectual property for what … a vulnerability that can be addressed with dozens of compensating controls that are cheaper, easier to implement at lower risk and without exposing the business to unnecessary risk!

    This must just be the view of someone who has spent his entire life in automation and control.

    As for ISA100 and WirelessHART … you should reconsider your position. Have you actually done any deep diving on the actual equipment? If you have, you would know that it has vulnerabilities as well.

    I guess our differences in opinion is why you have your clients and I have mine. You suggest to your clients to replace their endpoints and “hope” they are secure tomorrow … I will suggest a more holistic approach to ICS security that includes controls that will offer a bit more longevity.

  • Dale Peterson

    Joel – Sorry you think this has gotten personal. I don’t see it that way, just a disagreement on whether insecure by design PLCs should be replaced with more secure models or not. Or at least how urgent it is for the critical infrastructure ICS. As the disagreement is clearly laid out in the comments, which is a good thing, there is no need to continue. People can chew on this topic and decide for themselves or chime in.

    I don’t think anyone at Digital Bond has ever questioned your skill or knowledge or ability to assist owner/operators. We frequently recommend your training class and offered you an opportunity to train at S4 the last two years. We wouldn’t do that if we didn’t respect your ability to train on this topic.

    Dale Peterson
    Digital Bond, Inc.

  • Takeaway from this discussion:

    1. A software (!) company completely fucks up security for their product

    2. ICS security experts discuss the case like it was 2002

    Fellows, things will never change if you don’t realize which strategies did NOT work. Other than that, the discussion gives the few decision makers that might see it another indication that “the whole proposition of ICS security is highly debated even among experts (so why give a shit)”. Given that this must be one of the most extensive discussions ever in this blog, I cannot but feel sorry.

  • Take it easy guys… Joel isn’t saying that that 3S is doing the right thing. He isn’t defending them at all.

    All Joel is saying is that if for whatever reason, an end user can’t rip and replace their controllers, what other actions can they take to be more secure? Since 3S is not offering any guidance at all to their users, what Joel is suggesting (with no benefit to himself) is a big step up. It might not be ideal, but it is far better than doing nothing and waiting years for 3S.

    Now certainly some of Joel’s suggestions involve Tofinos and for me, that is good. But some of his ideas involve solutions (like SNORT) that I do not sell and that is great too. They are real solutions and when the vendor is being useless, any help is appreciated.

    Joel should be thanked, not criticized for trying to help.

  • Just exchanged some dialogue with 3S and other vendors who are “implied” to be vulnerable, and they all confirm that Version 3 is NOT vulnerable to these exploits. If you have documented proof of specific vendors/models running CoDeSys v3 that are vulnerable, please share.

  • Dale Peterson

    Joel – we do not have Version 3 in the lab as it has never been sent to us for testing. We have tested others listed as runtime port v3 (2.4.7.33) for owner/operators and those were vulnerable. This may be version 3, as ours is listed as runtime port v2, but it may not be. We don’t know how 3S version numbering is set up.

    We can’t say anything definitive because the information sharing has been a one way street with one exception. We were sent one potential fix in October that did not fix anything.

    Since 3S did not see this as a vulnerability in Version 2, it is hard to see why they would have changed it for Version 3. Remember Siemens insisted the Beresford vulnerabilities did not affect the S7-300 or 400 until Dillon got his hands on a S7-300 and quickly proved it. We are assuming at this point that short of some explanation from 3S on how the insecure by design issue was fixed that it is vulnerable.

    The only other data point we have is CoDeSys customers saying that 3S released a patch on November 6th. You don’t patch something that is not vulnerable or insecure by design. Given that it needs to be integrated into the product it will take a while before we see products with this patch.

    A researcher colleague is getting his hands on a true V3 PLC in his lab and this may settle the issue.

    Dale Peterson
    Digital Bond, Inc.

Leave a Reply