The angriest I have ever been in a CISO chair had nothing to do with a breach. It was a letter grade.
A ratings platform had dropped our score. A customer saw it, escalated, and I spent the better part of two weeks proving a negative to people who had a dashboard and no context. The finding was an IP block that had not been ours in years. We were right. We got the correction. And none of that time went to anything that made a single customer safer.
That’s the business model. A company that has never been inside your environment or laid eyes on a single control sells your customers a reason to interrogate you. Better yet, then they try to sell you a subscription so you can see what they said.
It grades the lawn and calls it a home inspection
An outside-in scan sees expired certificates, open ports, unpatched internet-facing services, and email authentication records. Those are real signals and I want them. Nobody should be shipping an expired cert.
Now look at what the scan can’t see. Whether production data is segmented. Whether the backups restore, which is a different question from whether the backups run. How fast an incident gets escalated on a Saturday, and to whom. Whether offboarding revokes credentials or just files a ticket that dies in a queue. Every control that decides whether your customer data is actually protected is invisible to the scanner, but the scanner still prints a number to one decimal place.
The scope problem is worse than the staleness, and it is the reason I stopped taking these numbers seriously at all. In most environments I have run, the platform was grading our marketing website, not the products. The products lived on different domains, and the platform never connected them to us. The thing our customers actually logged into and put their data in was never in the score. We could have run the product on a hollowed-out server in a closet and the grade would not have moved. Whoever renewed the TLS certificate on the brochure site was, as far as the rating was concerned, running our security program.
That’s the attribution problem. These platforms map domains and IP ranges to companies at industrial scale, and at that scale, the map is frequently wrong. It misses what matters and it blames you for things that were never yours.
In cloud, the premise falls apart completely. I do not own that IP space. Nobody does, not in the way the rating model assumes. An AWS Elastic IP was mine on Tuesday and belongs to a stranger on Thursday, and the reverse is just as true, which means the address I picked up this morning arrives carrying whatever the last tenant did with it. Scoring a company by the addresses it appears to hold made a kind of sense when companies held addresses. That world is gone and the model never noticed.
So the number is precise, confident, and built on a partial view with known error modes. A credit score at least draws on your actual payment history. This one is computed from your mailbox and the state of your lawn.
The tax nobody puts in the budget
The part that makes me furious is the math.
I have rarely seen a ratings dispute make an organization more secure. The hours go to screenshots and a support portal. Meanwhile the real work sits there untouched. The vendor holding unmasked customer data on a contract with no breach-notification clock. The access review nobody has run since spring.
Multiply it. Your vendors are burning those same hours on you. Somewhere out there a two-person security team at a vendor is assembling evidence about a decommissioned subnet instead of patching something. The whole industry is paying that tax to correct a product’s errors, and the product gets paid either way.
Then comes the upsell, and this is the part that, honestly, feels like a shakedown. You are graded whether you asked to be or not. The grade is shown to your customers whether you asked or not. If you want to see the finding behind it, or push a correction through without waiting in line, there is a subscription for that. Free to be scored, paid to defend yourself. And it’s not just one of these platforms, there are multiple. Nobody in a regulated industry would tolerate that arrangement from a credit bureau. We tolerate it here because the dashboard is pretty.
Then it quietly replaces your judgment
I understand why these platforms sell. One number that ranks four hundred vendors is irresistible when your risk function is two people and a spreadsheet, and “the dashboard is green” is a comfortable thing to say out loud to leadership.
But watch what happens next. Vendors get onboarded or escalated on the score. Contract negotiations cite it. Some cyber insurers factor it in. A community bank with a lean risk team ends up outsourcing its vendor judgment to an algorithm that has never seen a control at any of those vendors, and calls that a program.
The stale attribution punishes vendors who did nothing wrong. That’s annoying. The reverse is what actually hurts: a vendor with an immaculate external surface and a disaster inside scores well, right up until disclosure day. Go look at the last few years of large breaches and check what those companies were rated the week before. They were fine. They were fine because the thing that failed was never visible from the parking lot.
Treat it like a smoke alarm
I have never spent a dollar of security budget on one of these platforms, and I am not going to. You will still have to deal with one, because your customers buy them and your board reads them.
So use the thing for what it is worth. A sudden score drop is a decent reason to send a vendor an email. That is the whole of it. A tripwire that goes off on squirrels.
A high score is not permission to skip diligence on the vendor holding unmasked customer records. A low score is not evidence of anything until you ask and hear the answer.
For the vendors that matter, do the work the scan cannot do. Get a breach-notification clock in the contract, in hours, with a named recipient. Ask for evidence of controls instead of an attestation cover page. Negotiate data minimization terms so the vendor holds less customer information in the first place, which is the only vendor control that keeps working after they fail. Then rank your vendors by what data they touch and what happens to your customers if that vendor has a bad month. That ranking decides where your diligence hours go, not a number from a company that has never met either of you.
If your vendor risk program has quietly become a ratings dashboard with a policy stapled to it, that is worth a conversation. Book a call: https://cal.com/vaughn-cyber-group
