So You Want to Be A CIP Consultant….

68847866_934e68cbcd[1]Between confusing standards, odd implementations, lack of security capability from control systems, and the craziness of The Audit, NERC CIP is not a field for the faint of heart. I’ve been doing work in this space for 8 years now, and I don’t pretend to have the complete picture from beginning to end. But, I have made mistakes along the way that have sat with me, mistakes that I’ll go ahead and share with the aspiring CIP’er that might chance across this blog.  So, here are some basic rules I have for CIP consulting, rules that I have made because various headaches and rules that I have broken (that caused various headaches).

Rule #1 – Don’t Show Up Without CIP-004 Access
CIP-004 Personnel & Training is the bane of any CIP consultant’s existence, namely because it takes forever to complete. It doesn’t matter if you’ve have been hired to solve problems, improve programs, configure devices, or install equipment, you need the initial access from CIP-004 to start that work. I can’t tell you how many times I’ve spent the first day on-site patiently waiting to “clear NERC CIP customs”.

The main issue is that the CIP-004 process involves several finicky pieces of documentation and lengthy time requirements. Sometimes, large organizations require that you go through their background check process, which is often done by another department entirely. Large organizations being what they are, you could get stuck in the background check process. Add in a  lengthy training program, unfamiliarity with the access process (most employees have gone through it themselves only once), and a ‘special’ CIP-004 for contractors and consultants, and you’ll spend the first day twiddling thumbs.

Get your plan in gear two weeks in advance to avoid this.  Have all background and criminal check  information ready to go in a CIP packet for your client, and prepare to walk through the process with multiple people over the phone and email.

Rule #2 – Don’t Forget Devices
One of the consistent causes of a violation I see everywhere comes from forgetting devices. CIP is not just for PCs and Servers, it is intended to cover all applicable Cyber Asset.  This means entities need evidence of protections for network switches, routers, PLCs, firewalls… Basically every cyber asset needs something. Many a time I’ve asked for the backup configuration of a serial to Ethernet converter, and received blank stares in response.

Rule #3 – Have a Process to Report Deficiencies to the Client
As a compliance consultant, you know what violates the NERC CIP and what is in line with NERC CIP. While on site, you may see deficiencies in the process that need to be addressed so that your client may avoid a fine or finding. Without a process to report these to the client, there is no definite way to ensure that your observation could be tracked and acted upon. This is important for your client to avoid glaring violations, but also to ensure you don’t get a “why didn’t you catch this” phone call.

Set this up at the start of the project, but be prepared: Some clients are not interested in hearing about their deficiencies.

Rule #4 – Make a Clear Distinction Between NERC Required, and a Recommendation
I have an opinion, you have an opinion, and then there is what is required.  Make sure to draw bold lines between what is a definite NERC Required item, and your interpretation of a NERC requirement. While the standards can have a lot of room to interpret, there are some definitive required/not required items. Separate these from your interpretations by specifically stating “this is NERC required, due to the following evidence”.

One of the best ways to illustrate this difference is with the Backup requirements in NERC CIP-009. The NERC Required element is that a backup be done annually. However, I’ve usually recommended that a backup be done after each significant change.  I’ve supported this by discussing the level of effort necessary to bring a 9 month old backup up to full operation and compliance will often outweigh a benefit from only doing a backup annually. Because I’ve separated the NERC Required element from my recommendation, there can be discussion on the points, rather than a “you must do this”. Plus, I always recommend the use of an automated backup system, which removes a lot of manual actions.

I’ll be the first to admit that I’ve broken a few of these rules from time to time, and I generally regret doing so.  If you have any other CIP rules you’d like to share, feel free to leave them in the comments for discussion.

5 comments to So You Want to Be A CIP Consultant….

  • Well done. This is a good initial set of rules for CIP consultants.

    As an attorney that often helps clients interface with Regional Entities, NERC and FERC, I’ll add the following fundamental rule for anyone that engages in audit preparation and/or gap analyses:

    Rule #5 – Do not make findings that indicate “violations” or “non-compliance”

    It is certainly the purview of the CIP consultant to express expert opinions and to identify shortcomings and areas that need strengthening and improvement, and also to point out what the reliability standards require. And it is also the job of the CIP Consultant to know and understand all the ancillary supporting guidance from NERC and industry best practices for CIP compliance… HOWEVER, please refrain from making findings or recommendations which state that you have found “violations” or “non-compliance.”

    While you may believe that you “know what violates the NERC CIP and what is in line with NERC CIP”, as stated above, you are not the ultimate fact-finder. Ultimately, the decision of what constitutes a “violation” is a legal conclusion. Thus, leave it to the Regional Entities, NERC, FERC and reviewing courts to make such conclusions. Your client may have reasons why such acts or omissions were not violations. As stated above, there are lots of grey areas.

    The hazard here is this: a CIP consultant that reports to his client findings of “violations” or “non-compliance” prematurely reaches the ultimate legal determination. This may severely prejudice your client in a subsequent administrative or judicial proceeding to adjudicate whether or not there were, in fact, violations… and your report may very well be an exhibit in such a proceeding, or one that may occur years later.

  • Great post Michael!

    I would echo the previous commenter about including findings of non-compliance or possible violations in any reports. Call them something else (almost anything but a violation or non-compliance). I prefer to hold a special “pencils down” meeting with the compliance officer and their legal staff for a “preview” of their final report. I *verbally* (not in writing) convey the potential gravity of the findings from a non-compliance perspective and discuss the matter in that forum.

    Rule #6: If you don’t know, ask.

    Simply put, don’t make stuff up. If you don’t know the answer, get it from another qualified source. As a former WECC Auditor and Investigator for the NERC CIP standards, I could tell which consulting firms made stuff up and which didn’t. I’ve actually seen utilities that were compliant before the consultant came in to “help” and non-compliant after the consultant left.

  • bryan owen

    Sorry in advance for a rant on Rule #1.
    CIP stakeholders need both an effective and efficient CIP-004 implementation. Sure we don’t want to be too prescriptive but couldn’t NERC centralize some of the CIP-004 requirements?

    All I can think of is that the drafting team didn’t really feel empowered to propose a requirement that NERC itself would need to implement.

    Every $ wasted waiting for access or spent on duplicity issues related to CIP-004 training boilerplate for contractor/vendor access is money that could be spent on improving security.

    /rant

  • bryan, no apologies necessary. Considering the amount of 3rd parties that were required to run a generation plant, a hefty chunk of time could be spent tracking all those contractors down for PRA, Training, etc.

    But, NERC does not have jurisdiction over contractors and vendors, they have jurisdiction over owners. So far, there has been no banding together of owners and vendors/contractors to develop some kind of shared resources for PRAs, Training, etc, so the condition is likely to continue.

    And, thanks to all the commentators for their valuable insight on NERC CIP as consultants!

  • Cyber Sec Turned SCADA Sec

    @Patrick and @Andrew… Although I understand the sticky area of legal compliance and legal non-compliance, do those “pencils down” meetings get asset owners to actually do something later? After all, the whole point of the CIP is to ensure safety and security of the asset and general population surrounding the facility. The stakeholders want to see COMPLIANT because their profits remain intact… but the neighbors of the facility and customers want to see SAFE/SECURE/RELIABLE. If that translates into Compliant great. But if there are some things that are greyish in the standards and they squeaked by on the borders of SAFETY, wouldn’t you hope that you are not a facility neighbor or customer.

Leave a Reply