Comment from Jake Brodsky
Time: March 30, 2007, 10:31 am

Concerning Working Group 6, I think one of the key problems facing those who would like to patch SCADA software and firmware is the lack of any document which might outline responsibilities.

To wit: Do you patch the firmware for the microprocessors in your car? How do you know if or when you need to do so? How would you do it?

Consider the situation where an instrument manufacturer has firmware which includes a communications module which uses a common protocol stack, such as MODBUS TCP. They didn’t write the implementation. They bought someone else’s. Now supposed they are steadily improving their product line and they no longer support MODBUS TCP. And further, suppose a common vulnerability is found with similar implementations of that MODBUS TCP stack . What do the manufacturers owe the customer?

How “patch-able” should embedded systems be? Should all firmware include some version display methods? What about modules within the firmware? What should a responsible company do when it can’t support the existing base of firmware?

These aren’t idle questions. I’m not amused at the thought of CERT vulnerability notices for embedded systems that few if any may know how to patch, let alone that they need to patch them.

If the only thing that comes of this group are a set of recommendations or policies for who should be responsible for what, then I think it will have done a lot.

Comment from Dale Peterson
Time: March 30, 2007, 10:42 am

Jake - you may have a point. It might be more interesting if the ISA effort focused on vendor responsibilities. The issue of vulnerable third-party components is one we see in our white box testing. Vendors start with the current version of the component, but over a development cycle that may span years it is not unusual to see security vulnerabilities patches come out.

And yes - - I patch my car firmware but not to stop potential adversaries.

Comment from Ron Southworth
Time: March 30, 2007, 1:23 pm

HI Jake & Dale.

Jake with what you were just mentioning would not a governance framework be more effective than a document on change management or do you see the need for more ganular or detailed description or prescription being required than a philossophy based governance framework?

On the change management part of your notes, Dale.
There is an AU standard on change management that is usually not implimented particularily well for contol systems. (It is usually applied in a buisness systems context). I can see some advantage in a document that would at the very least be a means to facillitate such a process.

I opened the scope a bit more from just patch management in my response in saying change management as this is frequently identified as being an area of diifficency in control systems security presentations and it is also a subject that involves the convergance of engineering focus, IT and Process.( I cannot think of the standard number off the top of my head but I will look it up on Monday if you like. I am certain it is a British standard or has an equivalent)

Dale your comment re vendor responsabilities is quite timely as this was somthing I was discussing with our present vendor support people today and it is something that I have had some email discussions with fairly recently. The vendor rep was illuminatng me as to the issues they were having in trying to keep up with testing MS updates and the impact each one could have on their product and I think this is somewhat complex an issue.

The evaluation process will be dependant on the OS and other applications that are loaded on a particular machine and the complexity that this brings is a very high burden. Is it going to be acheivable to place this burden all on to the vendor, I don’t think so. I think it is going to be something that requires a more evolved approach in sharing the load and responsability to be effective and will require end users to have the necessary non production based testbed’s and resources if they dont already have so or require during plant down time to include testing of patches.

Risk assessment is going to be a big factor with how each end user handles this and it is tied back into how connected or inter connected the system is. If the system is essentially an isolated system if a change requirement is not for availability or reliability.
If the system is heavily interconnected then the need for more vigorous changemanagement and testing is going to be required.

Comment from Jake Brodsky
Time: March 31, 2007, 11:31 am

Ron, your perspective got me thinking.

This is really a chicken/egg problem. Manufacturers don’t disclose or even deal with security problems in embedded systems because there are no change management policies for industry. And there aren’t any change management policies for industry because there really isn’t any support for them from manufacturers or regulators.

We need to start somewhere. Dale, I think we’re looking at flip sides of the same coin. We need to assign responsibilities somehow, and perhaps the best way to start with this is to mandate a change management policy for securing embedded systems. Once that happens, manufacturers would know how to communicate the patches and bug fixes to the users, and the users would know what to do with them. Finally, when all that is in place, then we can discuss governance and regulation of responsibilities.

Comment from Matt Franz
Time: March 31, 2007, 8:27 pm

Jake,

Believe it or not other vendors have had to deal with security vulnerabilities and “firmware updates” in “embedded systems” in fact even in some of the same RTOS’s used in controllers. Once again, not new territory or even especially unique territory.

Google http://www.google.com/search?q=psirt+vxworks for yourself.

And lo and beyond CERTs have even had to deal with issues and advisories in them before.

Imagine that

- mdf

- mdf

Comment from Ron Southworth
Time: April 1, 2007, 7:09 pm

Hi Matt,

Glad to see you are still about in the space and mainstream IT has not completely taken over your time.

With the high end of town whay you say may hold true but with this marketplace, in the global scene there are a lot of Vendors that are not as prepaired for any of this as you may think but they produce a cost effective product. I think that you have be fortunate to work with some of the more prepaired organisations.

Jake I do think a lot of what we talk about with security is a “chicken and egg” analogy. One reason why there is a push from AU for the shared responsability model. If you recall Bob Landman’s comments re smaller vendors and real innovation we need to strike a very delicate balance between regulation and overhead burdens so we don’t styme the development of new technologies. I think this is why we need to as much as possable seek an industry sharing process as a preferred model. I am not convinced on regulation being the answer ( The recent example SOX compliance loss of data as a valid excuse etc). I do understand your thoughts behind the need for regulation Jake as I agree there are a number of cultural issues in managment that there are wide spread examples of. It may be a slower road to travel but if we can acheive it through “peer pressure” and industry co-operation (cultural change) it will prodiuce a more desireable outcome.

I think we need to encourage the sorts of processes you are speaking of in essence what this is all about is not so much regulation but more of “a code of conduct”?

(P.S. Present research indicates the answer is the egg)

Comment from Ralph Langner
Time: April 2, 2007, 5:41 am

Good point, Jake. Here’s my solution. Don’t wait for regulations. As an asset owner, just use the respective parts of INL’s cyber procurement language in your buying terms. Thereby you create a case for first keeping lawyers in business, and second improving your cyber security in the long term.

In my opinion, some direct pressure from the customer is definitely required. We have to face the situation that thousands of controllers are out there with buggy firmware, the vendor has a fix, the fix can be installed easily, but the vendor doesn’t tell anybody in fear of bad press (”we can’t afford to let anybody know that there are bugs in our products”). Don’t wait for standards bodies or industry organizations to change this.

Comment from Jake Brodsky
Time: April 2, 2007, 7:24 am

Matt, Ralph, and Ron, while there are many out there who will be waiting for industry regulations to force them in to action, I think writing a standard is a prerequisite. The regulations will be easier to craft if they have some industry standard to point to. And please understand, there are plenty of us among the utilities out there who won’t wait for the regulation to happen because we can’t afford to play catch-up.

In a perfect world, Matt, your scenario would work pretty well. However, one of the problems often encountered among any priesthood of profession is that they forget their products need to be used by people less literate in their field. I really don’t know what real time OS lurks in the process controllers we use. The manufacturers don’t typically reveal such information to us. I know they probably are using someone else’s OS. I suspect they have some interesting tweaks and adjustments on that OS as well. However, if I see some news regarding an embedded OS, I have no idea whether it’s relevant to the PLC gear I use or not. Nobody reveals this information because too few ask for it.

A standard, and regulation to come later, will help to enforce such behavior. Until then, we’re all stumbling around in the dark.

Comment from Matthew Franz
Time: April 2, 2007, 9:52 am

Jake/Ron

I actually don’t have any opinion on regulation of this matter, either for or against but standardization (within the IETF or whatever) isn’t what caused network device vendors to do what Cisco does [in advisories.] There are no real standards on this matter. Does Dell (or other 2nd/3rd tier vendors) reveal the RTOS they use in their switches? Probably not. If they received enough pressure from customers? I just don’t understand the [we can't do that, no one does that] defeatest attitude that is pervasive in the responses to recent blogs. Now that I’m an end-user (albeit non-SCADA) I’m on the other (and sometimes less comfortable) side of the vendor/consultant relationship, but I can say if end users don’t push your vendors/consultants, well then that is your (my) fault.

Comment from Ralph Langner
Time: April 2, 2007, 10:02 am

Jake, I agree that a standard is a good thing, and as a matter of fact I am participating in standard bodies myself.

I disagree that we’re stumbling around in the dark. In almost every real life scenario, security experts can make suggestions how to reduce a given risk that is unacceptable high, and can help to push vendors. My concern is that vendors just get another excuse for doing nothing by claiming that they are waiting for a standard to materialize. I predict what will happen after we have such standard(s). Vendors — probably asset owners as well — will wait several other years to check how the standard works, how it is accepted by the community etc. etc. In the meantime, many more million $ will be been lost due to security incidents.

Comment from Matt Franz
Time: April 2, 2007, 10:06 am

Another reason you aren’t stumbling in the dark

franz-macbook:~/Desktop mdfranz$ strings PowerConnect_34XX_boot-10101.rfb | head
CRER
4Copyright 1984-2003 Wind River Systems, Inc.;

Comment from Jake Brodsky
Time: April 2, 2007, 11:28 am

Okay Matt, perhaps we aren’t stumbling around in the dark. Let’s call it moonlight. There is still an awful lot we can’t see. For example, whose DNP stack is lurking in the MVI Module, hanging on the ControlLogix PLC backplane? I can take an educated guess. I may even get a good answer if I were to call the vendor and ask them. But I can guarantee that the module doesn’t give that information when I request version information.

I’m not bothered by that approach. This is an embedded system, not a laptop. However, if a vendor is going to act like an OEM, they owe me notification of every security and safety related patch that they become aware of. And they need to do this in a timely manner. I don’t want to be given an urgent patch for some buffer overflow exploit from ten months ago.

Ralph, I think the efforts of WG6 are aimed exactly at what you’re describing. The problem as I stated before is that the roles and responsibilities of both the vendor and the users are not well defined for embedded systems. We can borrow a lot from many IT practices; however, at the end of the day, we have to recognize that we’re dealing with an embedded system, not a PDA.

As a motivated asset-owner myself, I’m not making excuses for anyone. The problem exists at both ends. Most of my colleagues in the industry wouldn’t know what to do with an embedded system patch even if manufacturers were sending them as you say a responsible manufacturer should.

Yes, this issue is coming up very late in the game. And as Matt points out, it isn’t even universal among IT equipment manufacturers. This problem is one of perception as much as anything else. I’m not suggesting that security researchers shouldn’t be dealing with this issue. The problem is that they’ve been beating your heads against this problem for some time, yet most of us in the marketplace don’t know about it. Managers don’t know it is even an issue. We need a shorthand way to refer to this problem. We need a standard.

That’s the reality of this situation. This stuff isn’t likely to have an effect on procurement contracts, let alone boardrooms, unless they have a document that everyone can point to and say “make it happen like this…”

Comment from Bryan Singer
Time: April 2, 2007, 3:28 pm

Its actually interesting, the response has been overwhelming in support of at least a technical report in this area. And, I think it is far beyond just basic patching procedures, which I agree with Dale usually are easy to at least determine. More difficult to implement and maintain once burned once or twice, but conceptually patch management programs should be straightforward to put into place. Building on that concept, many are asking for straightforward language and standards from which to work with IT (to keep them off of operations backs while patches are tested), and work with vendors to ensure suitable testing and notification protocols. As a software developer, I have all to often seen “surprises” in patches such as memory leaks or other issues that might take 36 hours of runtime to manifest. Often, in the lab, machines are rebooted multiple times per day, clearing buffers, or they are not given enough runtime or stringent enough testing to ensure there are no problems under an operational system load. I have also seen far too often where patches come out that touch more than they advertise, and thus when a cursory review may say, “go ahead and apply, its safe” ends up breaking something else.

Enough said, I hope… and I also hope that a number of vendors and asset owners continue to show the interest they have now…

Comment from Ron Southworth
Time: April 2, 2007, 3:45 pm

Hi everyone, some interesting words all round.

I was interested to read your comments on vendors not revealing the info on the OS behind the imbedded systems Jake as I have been finding that largely when I ask this sort of question I get some very straight answers or someone gets back to me in short order. I found it most encouraging. as you say quite often you can see hints of the underlying OS or if you open the lid and the chipset numbers have not been rubbed off you can usually work it out. I have had quite a few vendors very willing to show the quality of the product.

WIth respect to WG6 I mentioned a code of practice rather than a standard as a possable alternative “label” on the basis of reducing cycle times and adoption hurdles - case in point the resolution of IEC101 and DNP3 security standard? Something everyone wants to see up and running but going thru the process is taking longer to go thru than what you would like to see happen. Aus standards suggested a fast track method for standard cycles but I think even this shortened method did not lend itself to rapid development.

I think that with respect to changing the situation we as end users have to start asking the right questions. I went to a presentation on some SCADA software yesterday not one mention on security apart from oh it has passwords. This was an intergated system quite scary to think that an operator given the right password can look at logical code and “delete” a protection device out of the ladder logic to force a process back into life !

I had to ask the questions to prompt the people. I know the answers but the poeple selling the product and maintaining the product did not! I know the product has the sorts of features but their own people don’t and don’t sell it as part of the presnetation.

Comment from Rob Lewis
Time: April 2, 2007, 10:01 pm

Great discussion, I’m one of those vendors trying to learn more about your industry’s security concerns.

We do something different, and in light of Bryan’s comment might be of interest to you all. We are creating secure environments that remain secure, even without available patches. I am talking about a kernel level policy enforcer that plugs into existing IT infrastructures and prevents privilege escalation. This would buy you enough time to do thorough patch testing and then apply in a scheduled fashion. It might even provide security in cases where patching is altogether problematic and impossible.

Comment from Bryan Singer
Time: April 2, 2007, 11:01 pm

I don’t think that vendors actually “try” to hide things. Speaking from experience as a developer and patch tester in a previous life, its easy to miss things. Test scripts and procedures typically cover unit tests, system tests, and basic integration tests. The assumption is that with most patches, the problem is manifested quite quickly in most IT systems, which for the most part is largely true. And while I do *not* want to start a conspiracy theory, there is such as thing as degrading returns when looking at the operational costs of testing. The other challenge to remember is that hardware and software vendors still base their products upon other people’s OS’s and software, and are often at the hands of their disclosure, patching procedures, etc.

The only way to be truly certain of what has happened to a system from a patch is to run a full file based integrity checker, registry checker, etc. both pre and post patch to know what was modified. Then, as most are precompiled binaries, the only way to know for SURE if the file was modified or not is to use a numeric hashing algorithm (md5sum or checksum). I have one more than one occasion seen two files that were the same size, same last modified time, etc. All the properties were the same (they are just file system flags, afterall)… And then see that md5sum returns back two different values. Scary stuff. Then, while IANAL (I am not a lawyer), I imagine that further reverse engineering would be a violation of many of the new Copyright and info-terrorism laws that are out there, so sometimes vendors or integrators are limited to “notification and question” only when they see something amiss.

Don’t get me wrong, that kind of test takes DAYS to do properly, and, just as most vendors would choose, they would rely on their full system tests to hopefully reveal any problems. One would hope that vendors (like many are doing now), can at least provide early warning on whether or not to use certain patches, and then perhaps later reveal if something more difficult is going on. To be sure, you don’t have to do a full analysis on every patch, but having worked in a telecom field where patching was serious business 24×7, there were quite a few times we had to fire up the binary analysis tools, debuggers, and disassemblers to understand what had happened and then to set out to prove it. Sounds a lot like vulnerability analysis, doesn’t it?

I guess the only challenge I see is that Ring 0 protection against escalation of privileges is not really a main concern to most asset owners. Perhaps I am wrong, but usually we want to guard against any system impact or disruption, given the extremely sensitive timing of many of our applications. While I still am an advocate of hardware abstraction and kernel mode protection from any user processes (if the performance were better, I LOVE the Java sandbox concept), those prevent more in the realm of rootkit, backdoors, and exploitation attacks. Those are nasty, to be sure, but I can think of lots of ways to exploit weak services on most boxes (NTP or SNMP, anyone? I don’t believe either runs with any protected access to kernel mode), that can take a component down due to disruption in no time..

Ok, Thoughts?

Comment from Ralph Langner
Time: April 3, 2007, 5:20 am

Bryan, in my experience, a modern Java plattform would be potent enough to be useful for the majority of automation tasks. I doubt though that any vendor will jump on the wagon in the next ten years, because Java is “too new”. Look at OPC. We see industry acceptance after the technology is around for ten years (and DCOM is said to be discontinued).

For the hiding thing… well… let’s take an example. There is this one big European vendor. They have an open TCP port 1010 in one of their products. Nobody knows what for, and probably it isn’t even used for anything. We asked several of their employees from engineering what this undocumented port is actually good for. They simply don’t know. The ****ing port is so secret that even employees must not know about it.

My point is: Several nasty issues in this area are not technical problems. They are social problems.

Comment from Jake Brodsky
Time: April 3, 2007, 11:22 pm

Gentlemen (and any Ladies who happen to be reading this), I think we all understand each other. And I think we all see the need for each of us to be pursuing the courses of action that we’re discussing here.

Ralph, I believe you and I are in agreement that the social end of this is the driving factor. A standard will help, but it will take time to craft. As you pointed out, these problems exist right now, so we’re all on the backside of the power curve. Even a standard isn’t the whole answer. After all, there are issues of warranty service and long term committment.

Asset owners are the ones left holding the bag. This is one reason why it doesn’t always pay to purchase inexpensive equipment. Service after the sale, including security patches and the like are a serious consideration.

That said, I think the Microsoft extreme of “release now and patch later” will hurt busines very badly if we allow it to permeate the embedded systems we use in industry. This is not just an OS we boot up each morning. It’s an embedded product with an expected functionality. Rebooting, restarting, or even allowing the watchdog timer to perform a reset is a luxury we frequently do not have.

My suspicion is that we’re going to discover ranges of needs for various types of equipment. and that any specification for patch management and the responsibility for testing or deployment is going to have to be broken up according to functionality of that system.

Regarding Bryan’s Ring 0 protection issues, I’ve been following Joanna Rutkowska’s research. This stuff scares me. But thankfully, it is still reasearch. Right now, my efforts are focused toward deploying better defense in depth strategies. I’ll watch the research, but I’m not going to make Better the victim of Best. I know I need improvements right now. I’ll do what I can to implement them. Then, I’ll see what the defenses for this stuff might look like and I’ll evaluate whether the overhead for the defense is worth the likelihood and the cost of failure.