Response Fuzzing

fuzzyFuzzing, as a practice, has been around for a while. Throw garbage at an input to a program and see what falls apart. Analyze the crashes and dumps, and see if any involve commonly exploitable issues, such as buffer overflows, off by one errors, etc.

I’ve seen SCADA protocols brought low by a simple repetition of ‘AAA…’; and I’ve seen much more complex fuzzing efforts (such as those brought up by Terry McCorkle and Billy Rios in some of their advanced training). In normal IT, fuzzing often focuses effort on the ‘server’ or ‘input’ part of the communication, where the user can influence input sent to the server. This is because the user is determined to be the larger threat, and the server is considered the system to defend. There are exceptions to this, one of the notable ones are browser based bugs designed to infect users based on input received from a web server.

But most ICS deployments have the exact opposite use case. Yes, we use client-server type architectures, but the systems that require the most protection are often not the server ones (the PLCs), they are the client systems (the controlling HMIs, etc). For example, take a Modbus communication: The client requests point data directly from a PLC, which then responds back to the client with the point data requested.  Simple right?

However, when the client is requesting data it is also vulnerable to a malicious response. In most research I’ve read, this malicious response has usually been ‘bad data’, data designed to provide upstream operators with false readouts and influence decisions. Easy to do when your protocols lack integrity. But, what if there were buffer overflow conditions in code that processes a response? An attacker might be able to use these conditions to crash the upstream process, and potentially insert their own code.

This request-response type architecture is present everywhere in automation, and runs over both TCP/IP and bare serial protocols. The more interesting condition is that a control center (the client) communicates to potentially hundreds of end devices (the servers), and only a small portion of those end devices may have physical and cyber protections. I use this example because I see it in NERC CIP implementations everywhere, where you have a Control Center with a dozen critical substations and 4 dozen that are non-critical.

I had an opportunity to do some very basic fuzzing for clients in the past, and recently looked at this condition. While I lacked time (this was limited to about 4 hours of  hacking, including recording the requests and responses) to put together something truly magnificent, I did write-up some basic Python code to send back malformed data when a SCADA system requested it. I didn’t find anything with this, maybe you will. Use your best judgement in implementing this code, as it is intentionally designed to provide data intended to crash systems and process.

This code was written to pretend to be a Modbus device in a few very simple cases, allowing the Control Center to request data from it. The responses from the program were intended to be scripted, but you could build more intelligence in if necessary with the PyModbus project. Specifically, this code pretended to be a serial Modbus device, accessible through a Ethernet to serial converter. This made interception and monitoring of the communication much simpler than going through a bare serial interface.

The intent of the code was simple:

  1. Listen for a specific requests from the control center
  2. Queue up a valid response to that request
  3. Randomly alter some of the parameters from a valid response
  4. Respond to the control center
  5. Log all information regarding the communications to flat file for analysis

The major limitation in this code is lack of CRC generation, which would ensure that the malicious data is accepted by the subsystem as valid, and then processed. I’ve also removed the payloads I worked with, you’ll need to generate your own valid requests and responses.

Link to code

title image by oskay

8 comments to Response Fuzzing

  • It should be noted that this process, “defensive fuzzing” is the exact patented process that HoneyPoint Security Server (our honeypot products for ICS/SCADA) uses. It is our “HornetPoint” behavior and goes back to the mid-2000s when we first created the product. It is designed to be a self-defending capability for using honeypots to proactively defend environments. We have had extended success with the technique across a variety of implementations.
    This technique is built directly into HoneyPoint Security Server and HoneyPoint Personal Edition.

  • Dale Peterson

    Brent – A couple of questions:

    1) Does the HoneyPoint support response fuzzing for any of the ICS protocols?

    2) Assuming your comments on the patent are correct – has any other vendor licensed the patent? It would seem to be an important part of vendor fuzz testing in a SDL.

    Dale

  • Chris

    I don’t know of any commercial solutions that fuzz DNP3 master, IEC 61850 MMS master or ICCP master etc.

  • Chris is absolutely right. They don’t exist yet, commercially…

    The interesting thing about some of these ICS protocols is that they have outstation to master “push” capability…. i.e. dnp3 unsolicited responses. This allows you to “speed fuzz” master stations.

  • I’ve always wondered about the unsolicited response functionality of DNP3, and similar protocols.

    Fundamentally, do they process and then discard the data, or do they simply block it unless there is a request sent? From a design perspective, I can’t imagine unsolicited and solicited responses should interact in the same way, or it could get confusing.

    So, by extension, would the speed fuzzing using unsolicited responses be the same as response fuzzing with solicited? Is there extra logic involved that would give more protection to one or the other?

    Questions I have, answers I have not.

    mike

  • Mike,

    In my experience, masters process unsolicited responses using the same stubs of codes as solicited responses… so each has the same set of possible vulnerabilities. In most implementations, I don’t think there’s much difference in terms of attack surface area. Some implementations ignore unsolicited data if the master isn’t configured to expect it, others process it anyway.

    In general they process the data in the same way, because duplicate data is not a problem when your only concern is static measurement database synchronization.

    Chris Sistrunk and I are currently doing extensive research on this, much of which will be public in the coming weeks.

    Happy to talk with you in more detail via email. You can find my email address via the about tab on my blog:

    http://www.automatak.net

    -Adam

  • No one has yet licensed the functionality, though we are in current discussions with a variety of vendors, including those outside of ICS.
    Yes, the functionality can be used to fuzz ICS protocols. We are in the process of extending the capability into our ProtoPredator line of products as well, which are dedicated to lab testing efforts. It’s just a matter of development cycles from my team. We added some of this capability into the Smart Meter testing product we released last year and more is coming as we release ProtoPredator for Raw Serial and ProtoPredator IP.
    I’m happy to engage more deeply if you would like to set up a call to discuss it.

  • @lbhuston

    Please provide a link to the patent you mention. My review of your product literature suggests that what Michael and I are talking about is in no way related to “defensive fuzzing”. We’re talking about offensive fuzzing, just with the attack data injected to the master, not the embedded controller. We’re not trying to ward off attacks. We’re talking about detecting 0days in masters the same way that you do it in outstations: mutated inputs.

    The process we describe really shouldn’t be patent worthy. It’s not novel. Anything that parses inputs can be fuzzed and that’s been known for a long time in the public domain.

    If your company does indeed have a patent that covers this aspect of fuzzing I will be extremely disappointed in the patent system, although it wouldn’t surprise me. The system gets gamed there all the time.

Leave a Reply