FPL - - Whatever Happened at Browns Ferry?
While I live in South Florida, I was in California during the short FPL blackout yesterday. At dinner with some other control system security professionals the talk obviously went to the FPL event. A few interesting points:
- Since this affected the Turkey Point nuclear plants we may get a NRC report on the incident. So often the details of events are never made public, and that is why the Browns Ferry Report was so interesting and instructive.
- The true cause of the Browns Ferry scram is still not known. My guess is the controller protocol stack. Others believe it was an overwhelmed switch or network infrastructure. We may never know.
- Then we revisited that last statement. Why don’t we know? Sure it may be impossible to see the actual data, but can’t we put that type of controller and switch in a test environment and send some malformed data or excessive data? You could go full out and do a Mu or Wurldtech test, but it likely could be identified with much simpler, open source tools in a matter of minutes. It should be simple to determine with a high probability what the cause was and focus on what needs to be fixed in the offending device.
Author: Dale Peterson
Posted: February 27th, 2008 under Calculating Risk, US Government.
Comments: 6
Comments
Comment from Jake Brodsky
Time: February 28, 2008, 8:00 am
Dale, you’re assuming that someone cares enough to bother doing this. I don’t know why we keep butting our heads against this, but we do: The biggest problem with security is apathy.
It is goodl that you bring up the case of Browns Ferry every so often, because the official reports on the incident are either deliberately obtuse, or incredibly ignorant. And nobody seems to care that it doesn’t say much.
Like you, I’m disturbed by the questions it raises, and that nobody has bothered to investigate.
Comment from Ralph Langner
Time: February 28, 2008, 9:22 am
Good point, Jake. Some people / organizations just don’t care about security, no matter which arguments you throw at them. In my experience, however, I wouldn’t attribute such behavior to apathy. They just don’t care — because they claim they don’t have the time, they don’t have the budget, they don’t have the people, they don’t have this and that. I think it essentially boils down to corporate culture and ethics. There are organizations where security is part of the culture, and there are those where it’s not.
Comment from John Saunders
Time: February 28, 2008, 3:56 pm
Perhaps this was a good thing. It provides another case that demonstrates the fragility of the current system. I work hard not to be a FUDder. But good grief if these cascading events occur due to a tree branch falling or failure of a single component in a substation, what could happen through a concerted attack on a number of nodes? Now the South Floridians have a taste, although I suspect experience with outages through hurricanes leaves them less bothered.
Comment from Jake Brodsky
Time: February 28, 2008, 10:54 pm
The cascading events happen I suspect because we SCADAteers are doing our jobs a bit too well. With good management, we can keep our existing distribution system going with very few upgrades. However, there comes a time when, despite the excellent management, we simply can’t avoid building new transmission infrastructure. The shocking thing is that nobody really wants to contemplate the costs of that kind of business.
They’d rather build massive Rube Goldbergian monstrosities than address the real problems at hand. I’ve seen these compromises several times in my career, and though each case can be justified by those with sharp pencils, I’m still let wondering if anyone was looking at the big picture.
Despite all that, the Browns Ferry incident is one of several interesting cases in which I’ve heard rumors of VFD malfunctions in the presence of excessive network traffic. I really don’t care about the specifics, but I would like to know more about how the problem came about and what we can do to mitigate it. If fixes are available, I would like to know where they are and what they do.
Comment from CallBEFOREYouDig
Time: February 29, 2008, 10:33 pm
I don’t want to start another Differences discussion (SMACK!), but I do think that the “yet another remarkable contraption saves the day” mindset (first one wins!) can become a serious SCADA safety/security/reliability problem as the overall complexity becomes unmanageable.
It seems to me that other parts of the organization (like the computer guys, say - OUCH!) typically have better controls on productionizing architecturally non-aligned solutions, possibly because the inventives (oops, incentives) are not quite so perverse.
Comment from Dale Peterson
Time: March 1, 2008, 9:34 am
CBYD - the “yet another remarkable contraption” argument has been a long standing problem in security. The percentage of resources spent on products is a triumph to the salesmanship and marketing of the security products industry. I could take this comment in about five different directions:
- the idea that asset owners don’t need to worry about control center or field security because they have a firewall as a prime example of a remarkable contraption
- Protocol developers like PROFINET who view security as an external issue as opposed to DNP3 and OPC UA that are integrating it into the protocol.
- the security consequences of proponents to integrating Safety and Control systems. Some of the whitepapers are truly frightening.
- security increasing the attack surface
- and of course my favorite discussion of secure by default in control systems
-
Write a comment