The security policy rollout survival guide

This blog post is a written version of a talk that our cofounder Maya gave at BSidesSeattle in February. You can also watch the recording and get the slides.

Security teams are constantly defining and rolling out new policies. Defining the policy is the easy part. The hard part is knowing what's actually possible to implement, and then implementing it.

Here's the map for how to navigate everything that needs to happen between defining a policy and it going ‘live’ in your environment.

Get input

There's something driving this policy forward, whether it's a compliance requirement, a recent incident, or just a security best practice, but there's usually room for interpretation between the policy (what you want to be true, like peer review on production changes) and the control (how you enforce it, like branch protection rules). The first move is to get input from the people who'll have to live with your controls, because they control your policy’s destiny.

For prod policies, your stakeholders are engineering, product, and SRE. They want to know, “how long will this take to implement?” and more importantly, “who gets paged when it breaks?”. For corp policies, your primary stakeholder is IT. Too often security throws a policy over the wall to IT to figure out how to implement, plus handle frontline response for user issues. What IT asks is, “how do we resolve issues that come up?”, but what they mean is, “are we responsible for this?”. Just like getting buy-in for any other change in a large organization, be proactive, and answer questions before they come up.

Communicate the change

When you’re making a change, communicate way more than you think you need to, and in multiple ways: email, Slack, and all-hands meetings. Before the rollout, during the rollout, and after it's a success. Tell people what's changing, when, and, most importantly, why (compliance counts as a why). People will put up with the friction if they understand the reason, and may be more tempted to circumvent it when they don’t.

Run a pilot

You wouldn't roll out a change to prod without testing it, would you? Don't roll out a change to your users without testing it either. There will be edge cases, controls that fail in unexpected situations — and you really need user feedback.

A good pilot is made up of a few elements:

• A representative set of participants: A diverse group of real users, who have time to engage with you. Don’t pick the team next to you because it’s easy, and definitely don’t pick the team shipping a feature next quarter because they’ll be too stressed to give you good feedback. Look for a range of roles, devices, and workflows: macOS, Linux, and Windows; employees, interns, and contractors; engineering, support, and sales.
• An appropriate timeline: Enough time to roll out a change and get feedback on it, but importantly, long enough that the control is actually tested. A policy requiring access approvals every 90 days can’t be tested in two weeks.
• A support channel: A way for users to report issues, escalate bugs, and give feedback.
• Clear success criteria: Success for a pilot is a go/no-go decision. You can usually ‘go’ once you know your rollout will achieve the desired outcome, blocking bugs are addressed, user feedback isn’t overwhelmingly negative, and the support burden is manageable.

Iterate

It’s possible (in fact, highly likely) that you will get feedback during the pilot that changes your implementation. That’s the point.

Use the pilot to find bugs, identify edge cases, and — most importantly — test that your controls actually work. If users find a way around your control, your policy doesn't work.

When you triage feedback, ultimately, the question is: is this blocking rollout? Is this affecting all or only some users or platforms? Is this a bug or an improvement? Can we handle this case differently, and still get the same outcome?

Roll out policy

I have a few pet peeves on doing policy enforcement ‘well’.

First, make enforcement of a policy visible. Measuring whether people comply changes if they do. If you want users to take action, nothing beats showing them how they compare to their peers.

Second, the person who sets the policy should be the one enforcing it. They need to feel the pain (that’s why you did a pilot!). Any ongoing user friction should hurt the policy owner more than the user. (The one exception is compliance requirements… you can’t make your auditor feel the pain for you. They are, however, very good at making you feel pain.)

Lastly, exceptions. There will always be exceptions. Always! What matters is knowing this ahead of time, so that you can build these in and have a strategy for managing exceptions over time from the start. In addition to the obvious ones (specific departments, employee types, platforms), you'll also likely have exceptions for users with disabilities, users in certain countries, demo devices, and automated workflows. There is no one size fits all. What’s important is deciding if these are out of scope, or just have an exception. Either way, managing them is never done.

After all that work, you've rolled out a policy. Did the policy do what you said it would? Probably, because you got input, piloted it, iterated on feedback, and then managed the inevitable exceptions. The harder question is whether you kept your users' goodwill while doing it. Because there's always a next one.