This is an interesting topic, and I want to throw in my less technical take.

Network security assessments are appropriate for owners and operators of control systems. Methodologies vary slightly by firm, but network security assessments will typically:

• scan the operating systems, common IT applications, and infrastructure systems for known vulnerabilities and missing patches. Most assessors use Nessus and a set of specialized tools for scanning databases, web servers, wireless access points, routers, firewalls, etc.
• analysis the security architecture
• inspect and analyze system and application configuration. This includes routers, domain controllers, firewalls, … and SCADA application configuration. There are often tremendous opportunities to improve security simply by implementing existing, paid for security features.
• review the security policy and compliance with the policy
• review the administrative controls such as backup, disaster recovery/bcp, change control, security awareness program, etc.
• review the physical security of cyber assets

Caveat: There are always a few consultants who will run a Nessus scan, customize the Nessus report a bit, and call this an assessment. In addition to being incomplete, this actually causes more problems because the asset owner needs to explain why all the false positives are not being addressed to management or an auditor.

A network security assessment will identify where an existing, installed SCADA or DCS is not following recommended practices. A vendor may choose to get a network security assessment to verify and document the secure deployment of their system. We are still seeing a lot of vendors and system integrators deploying their solutions in a highly insecure manner that does not even take advantage of the security features they tout - - but that is a topic for another entry.

Application security assessments are most appropriate for the vendors that developed the application. An independent third party should perform the assessment, and there are Common Criteria requirements that address this.

If an asset owner develops an application in-house, they should consider an application assessment. If your vendor refuses to perform an assessment or share the results, then you may need to get an application assessment to understand the risk. The problem is many vendors still refuse to fix identified security vulnerabilities. RFP language can include requirements to provide application assessment results.

I’m not going to preempt Matt’s paper on application security assessment methodology planned for ISA, but there is one very important point. Application security assessment focus on identifying zero-day vulnerabilities and vulnerabilities in the underlying components that would not be identified in common scanning tools. The application assessment evaluates how the application was implemented - - how it handled parameters that exceed boundaries, data that does not fit the expected format, large amounts of data, out-of-sequence events (timing attacks), etc.

This is not to say that Nessus and other tools are not run in an application assessment, but these tools are used more as a baseline and are not the real work and value of an application security assessment. Again, there are people out in the community calling Nessus and other scans application assessments, be warned.

Simply stated:

A network security assessment will identify known vulnerabilities and deviations from recommended practices. It is appropriate for owners and operators of SCADA and DCS.

An application security assessment will identify new, unknown vulnerabilities in an application that the application vendor needs to patch. It is appropriate for vendors and anyone else who develops an application.