Of all the questions we get from clients, we fear only one.
“So, how do we rank against other companies you work with?”
Ok – fear is a bit exaggerated, but I couldn’t have gotten away with my click-bait headline without a little flare for the dramatic, right?
All kidding aside, it is a fair question that we appreciate at Pathfynder. We all want to know how we measure up in this data-driven age. Are my programs working? How well is the dev team securing our application? Do we have the appropriate visibility to detect and quarantine attacks as they occur?
Is what we are doing making a difference?
For my part, I have floated different ideas internally on capturing this, but nothing has landed right. Graphs. Quadrants. Scoring. Relative risk calculations. None of these met the intent – at least from Pathfynder’s perspective.
Why? Because comparing performances across engagements against different clients is highly nuanced, and trying to give a client a concrete answer is wishy-washy at best. At worst, it is borderline deceptive.
This is not me throwing stones at glass houses. Pathfynder competitors are doing their best to answer the same question. I have seen some fancy graphics. They look great, but I am curious how they derived the answer. What difference does that content make?
The Operator’s Dilemma
Here is the challenge. I can say – anecdotally, of course – regarding insights from our internal penetration testing that we gain access to a company’s crown jewels – whether that be highly sensitive data or administrative rights within the network – in the vast majority of our engagements. We often accomplish our attacker objectives with basic attacks such as password brute forcing against a lack of multi-factor authentication (MFA). This often-seen oversight facilitates access, lateral movement, and, eventually, privilege escalation.
Typically, these do not require advanced tactics. This happens because basic security controls are not in place or not effectively enforced. With that as the backdrop, how can we answer the question “how did we do” when so many organizations are vulerable to the same basic attack techniques?
And please – do not mistake me here – we know that cybersecurity is really hard. I do not envy the challenge the defender has. They have to answer to executives that demand perfection and do this in a time when budgets are tight, and the adversaries only need to find a single hole in an otherwise buttoned-down defense.
With all that in mind, I have a few ideas regarding measuring your company’s performance with cybersecurity engagements.
Attacker LOE vs. Defender Visibility
One of our operators uses this quote when explaining his view from the attacker’s seat, and I love it: "Cybersecurity is a statistics game."
The upshot here is that given a long enough horizon and adversarial resources, your defenses will fail in some way.
That does not mean you should turn fatalistic and not invest in strengthening those defenses. Instead, you should strive to make the enemy’s job hard enough that they do not see your company as a target worth pursuing. Or, perhaps more realistically, strive to develop sufficient defender visibility in detecting attacker actions so you can close holes before attacker objectives are met. As the cliché in bear country goes, “you don’t have to be the fastest of your friends – just don’t be the slowest.”
So how do we suggest measuring your cybersecurity assessment performances? The best answer is some function of the adversarial level of effort against the defender’s level of visibility.
First, ask operators how long it took to achieve their attacker objectives, and how sophisticated their attacks had to be. Did you own us in a few hours? Or did it take a whole week? How long did it take to brute force passwords? Were these basic dictionary attacks (i.e., company name + season + year)? How many accounts did you get? Why did MFA fail (if it is being leveraged)? When you then kerberoasted service accounts and dumped all domain hashes, what percentage of passwords could you crack? How long did that take?
Be prepared for an honest answer, and try not to take it personally. The baseline here is the key.
Second, what was your company’s level of visibility? What did you detect? How much noise did our operators have to make for you to realize that an attack was underway? What do you need to do to tune your detection capabilities better?
If you are one of the lucky few that managed to throw a no-hitter, great. You survived this time. But don’t rest on your laurels. The nature of defense is fluid and ever-changing. What works today may not in a span of just a few months. The game is constantly in flux. The enemy is always on the move.
What Really Matters
We believe that cybersecurity is very much nascent in its maturity. Companies around the globe – large and small – struggle with the same issues. While industries are responding appropriately, the progress is still slow.
Great tools are out there, and they are getting better every year. Companies are doing a better job educating their staff on emerging cybersecurity issues. But the enemy innovates too, and they are everywhere. Shared knowledge among adversaries produces new ways to defeat mature defenses all the time.
The key to a strong defense is not just buying the latest security tools and relying on the same pen test compliance may demand. If you are selecting your offensive security provider based solely on price, you are likely not getting a realistic assessment of your current risk.
Seek out cybersecurity companies with the talent and experience to evade and, in all likelihood, defeat your defenses. Be selective in working with an organization that will also work with you to develop a strategy to remediate your vulnerabilities, improve your knowledge, and make you better. There aren’t many of these folks out there.
Partnerships in this space should be more than trite marketing catchphrases hung on home pages. You should want someone to reveal the unvarnished truths about your cyber defenses. And that team should be there to help retest as you close the gaps they discovered.
The keys here are simple. Seek out a baseline. Assess the operator’s level of effort delta from engagement to engagement. Steadfastly improve your visibility. Aggressively initiate remediation. And then do it again.
Remember: you do not have to be the fastest. Just do not be the slowest. The bears get hungry too.