Vulnerability Scoring Metrics
Last week at PCSF there were a few issues that seemed to work their way into every presentation and discussion. It seems that both vendors and asset owners are looking hard for the government or some other entity to provide vulnerabilities with some sort of risk equation, but as of yet no one has really stepped up to the task and there’s a good reason for that. Any measurements/scores that come from sources outside of the organization affected are going to be less than perfect, and often much less than perfect.
Let’s take a look at a scoring system that’s used extensively, CVSS. The base score is based on a few seemingly simple variables, the access vector, attack complexity, level of authentication needed and the confidentiality, integrity, and availability impact. The problem with these scores is that so much of the information used to generate them is that they’re bound to the common case, and while that may account for 9 out of the 10 asset owners using it the 10th one is working from incorrect starting information that can lead to false conclusions that may negatively impact the business. This could come in the form of patching when it’s unnecessary, not patching when a clear threat was present, or any number of other things. And let’s not forget that these base metrics are in a state of flux and often deal with incomplete information, a vulnerability that is unexploitable one day may be made trivially exploitable the next depending on additional information or understanding of the technology, these scoring systems are valid for exactly the moment that it is calculated and grows stale quickly.
The feeling I got from a lot of the crowd at PCSF is that they’re looking for someone (government?) to do the scoring for them and that’s just asking for trouble. For these metrics to be of any real value they’re going to have to be analyzed, tweaked, and recalculated by the asset owners anyways, so let’s not put a middle man in place to begin with.
Author: Daniel Peck
Posted: September 2nd, 2008 under Calculating Risk.
Comments: 6
Comments
Comment from Kevin Lackey
Time: September 2, 2008, 12:10 pm
Industry wants some one to come in and give their networks the “stamp of approval” and if it is from the Feds even better. That way when something happens they can say, not our fault so and so certified that we were golden. And then…. lawyers.
Comment from Jake Brodsky
Time: September 2, 2008, 1:51 pm
Following up on what Kevin wrote, too many people view safety as a product, a design feature, or a service –not as policies, practices, and a way of thinking.
Those who do not understand this problem, it appears to be an endless commitment. And it is, though it doesn’t have to be particularly expensive.
That’s why they’re looking for a government stamp of approval. It gives them a way out, a checkbox that says they’ve done all they need to do.
We’ve been there before with safety concerns, though. And it doesn’t stop the lawyers from winning cases. I don’t know what that government approval stamp could possible do to improve the situation.
Comment from Ron Southworth
Time: September 2, 2008, 11:06 pm
Hi Jake I think you are bang on as per usual. Bryan Singer and others have been putting out the message about the need to view security and safety as one and the same in terms of life cycle and in cultural uptake and it has tremendous merrit.
I personally am looking towards government to provide credible threat information in order to assess the risk to my buisness processes and systems I maintain.
I am certain people are looking for their get out of jail card by receiving a certification to some notional target, however while there is a gap between this target and the public’s expectations the blame game will continue to be a part of it all.
The CVSS system when looking at control systems from an information system perspective gives you some figures to work on but will it acheive a granular enough differentiator or will you be looking at the decimal point differences in order to quantify differeces?
Comment from Éireann Leverett
Time: September 3, 2008, 9:01 am
Teaching anyone STRIDE and DREAD models, and making them work it out for themselves is a valuable exercise. More discussion and debate that actually results in consensus is what we need, not a new model, or middle man. We need to teach people to use and adapt the tools we have instead of indulging in analysis paralysis.
Comment from Ron Southworth
Time: September 3, 2008, 3:19 pm
Hi Éireann I think the debate is worthwhile and even looking at different models as part of the debate.
Settling on something that is internationally acceptable is important in the long run.
I am certain you have performed numerous risk assessments and in the SCADA realm relative to every other risk you assess the differentiators can be very small largely because the consequences are all very extreme and this would apply equally with any mission or business critical system. I used all seven information security terms recently to describe differentiators between different system types and I can see some benefit in maybe using all seven.
Comment from Éireann Leverett
Time: September 5, 2008, 8:26 am
I must have misrepresented myself. I think the debate is valuable because it teaches use to think critically, BUT too many debates in the control system security space attempt to re-invent the wheel. Vulnerability management and scoring are fairly well explored fields in both practice and theory, studying up on the work already done before defining a new process would benefit us all.
I agree the risk management is different, but the differences do not invalidate the model. This is the heart of the point I am trying to make. Computer security has models. The values will change to suit the domain, but the basic models are valid.
Also, returning to Dale’s point, these scores evolve rapidly and the scoring of vulnerabilities is only as useful as it is relevant. To keep it most relevant to your business, it should be done within you business. At the very least within a group of industry representatives.
Returning to your question:
“The CVSS system when looking at control systems from an information system perspective gives you some figures to work on but will it acheive a granular enough differentiator or will you be looking at the decimal point differences in order to quantify differeces?”
I think the biggest difference here is that we are protecting the availability of functionality more often than valuable information. So we change some of the scoring mechanisms but continue to use the basic models in house.
We need to teach our communities to do this work, and then explain to our relevant authorities how we want it done, not ask them to create metrics for us that lack relevance to our businesses, industries, protocols, hardware, and configurations.
I completely agree with you though on this point:
“I personally am looking towards government to provide credible threat information in order to assess the risk to my buisness processes and systems I maintain.”
Write a comment