I had no intention of blogging this (well, at least not tonight) but an email newsletter from Network World (Phil Hochmuth’s weekly column, “Linux, Unix safer than Windows? Take a look at figs from U.S. CERT”) sent me over the edge with this observation:

According to US-CERT, a combined 2,328 security vulnerabilities were reported last year affecting Linux and Unix systems, while 812 vulnerabilities were reported for Microsoft Windows-based systems. The safest operating system? Apple’s MacOS, which only registered 25 vulnerabilities on U.S. CERT’s list.

Observers say that the numbers attributed to Linux and Unix systems may seem high since U.S. CERT counts every update made after an initial vulnerability is reported towards that respective operating system’s score. Also, lumping together Unix and Linux system vulnerabilities is like saying there were more problems with transmissions in German and Japanese cars last year than Korean cars; whereas Germany and Japan have almost a dozen automakers and Korea has only Kia.

Still, the notion that Linux or Unix-based systems are inherently safer that Microsoft is challenged by the U.S. CERT findings.

Since US-CERT probably didn’t register any control system vulnerabilities, they must be the safest of all!

Apparently, Phil “didn’t get the memo” (meaning the Open Letter on the Interpretation of Vulnerability Statistics) last week from Steve Christey, who is the editor of CVE which systematically picks apart the statistical conclusions that can be drawn based refined vulnerability sources (aka RVIs) such as SecurityFocus, OSDB, and even CVE.

I hinted at some of these in My Thanksgiving Critique of the SANS Top 20 but he nails it, citing the following factors that make vulnerability statistics difficult to interpret:

• variations in editorial policy
• fractured vulnerability information
• lack of complete cross-referencing among RVI sources
• unmeasurable research community bias
• unmeasurable disclosure bias

Control systems vulnerabilities are especially impacted by the last two: most independent security researchers couldn’t spell PLC, and when flaws are discovered (by end users or vendors), they typically aren’t disclosed.

But there is more, the letter elaborates on the reasons for the variations of editorial policy which include completeness (further broken down into severity, veracity, product space, other), abstraction (vulnerability type and replication), timeliness, and reality.

These are good to remember the next time a vendor a security pundit makes a claim based this sort of data.