If the strategic goal of a security awareness programme is to change user behaviour on an organisation-wide scale – essentially, making good cyber hygiene behaviour a self-regulating requirement for belonging to the organisation’s culture – how do we know when we’ve achieved our objective? This is a vexing challenge for those of us in the field. It’s far too easy to miss the forest for trees.
When your programme is new, your goals are tactical and limited. You might track the percentage of users who complete their on-boarding security training. Something that’s simple to measure and roughly indicative of programme effectiveness. As your programme matures, you flirt with metrics and statistics, via one-off reports and comprehensive dashboards. How many users “fell” for a phishing simulation?
How many departments have a “security ambassador” appointed (and do they show up for meetings)? What percentage of company laptops can be validated in a quarterly inventory? Most awareness pros will grab hold of whatever concrete numbers they can assemble to convince upper management that they’re doing something well … but those arbitrary “success criteria” graphs don’t provide any insight into the cultural impact of the awareness programme. In short, are people doing what’s needed because they choose to?
I suggest that the best way to measure programme-level impact can’t be tracked with mathematics; it can only be observed on the office floor. To recognize wilful compliance, you need to silently observe to how users act when their only motivation for doing “the right thing” (security wise) comes entirely from within themselves.
As an example, I was gobsmacked week-before-last during a lousy breakfast. I was out-of-town, attending some family medical drama. The details aren’t important to this story; what matters is that my wife and I had stopped at a popular restaurant before meeting our relatives. We’d chosen a place that we’d frequented in the 1990s on long trips (but hadn’t been to in years).
We were fatigued from too many early mornings, late nights, missed meals, and unrelenting stress. With no small children to entertain, we each tended our beverages in companionly silence. My wife was concentrating on steeping her tea. I was focused on getting coffee into my muzzy head as swiftly as possible. While I drank, I scanned the room, curious to observe the locals.
Most of the diners were, like us, staring quietly into the middle distance while they waited for their coffee to kick in.
Suddenly, an older woman sitting near the windows rose in alarm and made a beeline towards us. She was moving with intent but seemed anxious rather than aggressive. Perplexed, I made eye contact with her right as she came to an abrupt halt a metre from our table.
“It’s not what you think,” she insisted. “I had a procedure.”
Being only half awake and completely clueless as to what “it” was that I must have thought, I pulled a delaying trick from my staff officer’s handbook and said only “Oh?” while raising an eyebrow. That trick always bought me time to catch up when ambushed in a meeting.
It worked here, too. The woman seemed relieved, as if I’d confirmed her suspicions. She let out a held breath, smiled, and visibly relaxed.
“I realized you saw it,” she explained. “It’s totally not that. I had surgery and the dentist wound up taking more bone out of my cheek and jaw than he’d planned on. That’s why this happened. It wasn’t” she paused and pointed back at her still-seated male companion, “anything he did.”
The pieces clicked. Now that she was close, I could see blue-black bruising on the side of her jaw. I hadn’t noticed it while she was seated halfway across the dining room. Once she drew my attention to it, I had to agree that her bruise looked like a left-hander had clocked her with a punch to the face.
My father was still teaching martial arts into his eighties, so my first illogical assumption was that this woman had gotten suckered by a low left hook while sparring. My father once clocked me with a new Jujitsu kick that gave me a wicked black eye.
From her perspective, a strange couple had looked at her, “noticed” her bruising, and had made the logical deduction: she was a battered spouse. This stranger had come over to us to set things straight before we called the police to rescue her from her abuser … or intervened directly, causing a scene in the restaurant. Her conclusion was perfectly logical … if you assumed that strangers would take immediate action to intervene in a stranger’s domestic violence situation.
When I was growing up, that never would’ve happened. Most people back then would have “politely” pretended to not notice. They certainly wouldn’t have gotten involved. Even ten years back, it was far more likely that strangers would rationalize their inaction rather than take a risk on being wrong or causing a scene.
Time, though, seem to have changed. What this woman’s emotionally charged demonstrated was that the people of this town had evolved for the better. Intervention on behalf of a suspected battered spouse was now the expected norm – so much so that she couldn’t take a risk on strangers drawing the wrong conclusion from the visible evidence.
This was … fantastic! I was thrilled to see it. Once I realized around what had happened, I smiled back at the stranger and thanked her for explaining. I glanced back at her partner and started to ask a question, but she knew where I was headed and said “It’s not like that. He doesn’t beat me. Trust me – we’re fine.”
I thanked the woman for her candour and for setting us straight. She returned to her table just in time for her meal to be presented. My wife and I discussed the surreality of the moment and agreed that this marked an excellent change for the community. Hopefully fewer victims of domestic violence will be left to suffer alone in this town now that their culture has changed to compel acknowledgement of and intervention in manifestations of partner abuse rather than meekly consider it. Good on them!
There’s a popular misconception in American media that people in small communities never evolve or mature. That’s nonsense. Cultures constantly change.
That’s what I meant when I suggested the best way to know that your security awareness programme is achieving its strategic objectives is to observe your users when they think they’re not being watched. When users do the right thing because the right thing is compelled as a requirement of membership, that manifestation can be considered reliable evidence of effective culture change.
These examples might start with users holding one another accountable to lock their workstations but that peer enforcement constitutes constructive cultural change. When your messages resonate, you’ll see those messages manifest as pressure to conform from one’s peers.
Is such meaningful evidence going to be difficult to track? Sure it is. It’s the sort of social proof that can’t be easily quantified and documented in a spreadsheet. Metrics nerds will tear their hair out because they can’t measure behaviour in simple statistics. That’s fine. We’re in the people business and people are inherently difficult to quantify. Height, weight, inseam, hat size, and BMI are all easy to measure, but don’t tell anyone a damned thing about what a person will or won’t do, especially when left alone. Behaviour, reasoning, and motivation are complicated, illogical, and often contradictory.
Still, that’s the business we’re in. People are fascinating but they’re also messy. The security awareness function is – at its heart – to influence both individual and collective behaviour to make unnecessary dangerous actions so unacceptable that people won’t just choose to avoid engaging in them … they’ll pre-emptively intervene to prevent anyone else from engaging in them either. That’s how you know you’re doing things right and your programme is succeeding at its ultimate strategic objectives.
