Part 2 - Testing Methodology and Coverage

As a developer, you look at the requirements and design specification and this dictates what the product or device must do. In the case of protocols, these specifications are in the form of standards issued by standards bodies or industry groups recognized as authoritative for that protocol.

Once the product is complete, it goes through QA testing to verify the implementation is secure and reliable. The question is of the total population of possible test cases, how many and which test cases should be performed?

The simplified diagram below shows the possible test coverage area. The small circle in the middle represents all possible test cases that verify the protocol works as specified. This is often called positive testing and most vendors perform positive testing.

One caveat on the state of current positive testing in the controller market. Many controllers only implement the subset of the protocol that they need for proper operation. So a portion of valid protocol messages are not supported or tested. The simplest and most common example in our experience is a controller that does not support broadcast traffic hangs and requires a reboot when it receives broadcast packets. There is nothing saying a controller needs to support broadcast traffic, but it should continue to operate properly when it receives broadcast traffic which leads us to negative testing.

The larger rectangle around the valid data circle represents all possible negative test cases. Negative testing sends data to the device that violates the protocol, and each test determines if the device properly processes (usually discards) the data and continues to operate properly. Mishandling of malformed packets and protocol fields is one of the most common cause of vulnerabilties in IT and control systems. For example, the recent ICCP vulnerabilities identified by Digital Bond and disclosed by US CERT where the result of malformed protocol messages.

The challenge is the population of potential negative testing is very large. There are a few common ways to approach this. Researchers like Digital Bond will typically analyze the protocol and device and estimate where design and implementation mistakes were likely to be made. A classic low-hanging fruit black box testing approach. This may be one of the best approaches to identifying ‘a vulnerability’ and mimics the hacker approach so perhaps it identifies the most likely found vulnerability. The problem is we get a very small amount of coverage.

As an example, lets look at the approach Lluis Mora from Neutralbit took related to OPC testing as presented at S4. Lluis identified and coded up 24 different test cases. These tests cases found implementation errors leading to security vulnerabilities in one-third of the OPC servers tested, so it was certainly effective from an awareness standpoint. Since he responsibly disclosed these to US CERT, we should see patches this year and a higher level of security.

But let’s look at the test coverage of Luis’s approach in the diagram below. The dots represent an individual test cases and the lines indicate paths where the researcher is trying a large number of illegal values for one or more fields. There is not much coverage, or therefore assurance, in this type of black box testing. In fact, these simplified diagrams grossly overstate the coverage provided by this ad hoc testing because the rectangle would be much larger.

The other problem besides limited coverage with expert or ad hoc black box testing is it requires a great deal of work to develop the test cases, and it must be automated to be effective in comparing products.

Wurldtech’s Achilles platform takes a different and unique mathematical approach using grammars. Grammar is a computer science term for a formalized and complete description of a protocol. The formal and complete description provided by the Achilles protocol grammars allow for provable claims of test and assurance coverage.

In fact, the Achilles grammar test case components use a special type of grammar called an attribute grammar. Achilles’ attribute grammar use attribute values to describe the protocol and create protocol tests based on the attribute grammar. The test cases produced from the attribute grammar can be proven to cover a defined portion of the protocol attack space.

I’m not going to try to explain the mathematics behind attribute grammars or the Chomsky hierarchy, but one of the items I am working with Wurldtech on is to have this explained in a series of white papers at different technical levels.

The important point is attribute grammars create large numbers of test cases, over 30 million in the entry level certification, with quantifiable and much larger coverage than other techniques as shown in the diagram below.

The test cases now cover area, everything within that polygon, rather than points or lines, and the attribute grammar is a concise and mathematical way of describing this coverage area. This large and quantifiable coverage is ideally suited for a certification effort as opposed to a set of ad hoc ‘expert determined’ tests.

Indepth testing of new protocols can be added to the tool in a matter of weeks rather than months because all that is required is a new grammar file and the Achilles engine generates the test cases.

In the next blog entry, we will discuss the protocols covered in the Achilles Controller Certification Levels and the timetable to certified products.

Read Other Achilles Controller Certification details:

Part 1 - Why Protocol Stack Testing