Tony Martin-Vegue has been posting lately about the need for "nutrition labels" or quality ratings on cybersecurity industry reports. I wholeheartedly agree with the need for this -- especially in our current time when "thought leadership" reports abound. Many of them look pretty, so it's hard to distinguish which ones you should take into consideration and which ones you should ignore. This was a major reason Cyentia released the Industry Research Library almost 10 years ago. John Hoffoss commented on this in response to one of Tony's posts, and it reminded me that a public statement was in order, since that library no longer exists as of earlier this year. We're a small company with a need to focus our limited resources on activities that drive revenue and/or have industry-wide impact. We hoped the library might do both. F5 graciously sponsored it for a few years, but overall, we simply couldn't justify allocating the time and energy required to sustain and grow it. And while some used the library, it fell far short of our goals for impacting industry knowledge and practice. I'm convinced that one of the major reasons it fell short is that we never got around to doing what Tony is calling for - rating the quality of reports. We intended to do that, and even created and tested assessment criteria. I'd like to say more about that, but I will leave it to a follow-up post. For now, thanks to those of you who used the industry library over the years. I hope we didn't leave you hanging when it became a thing of the past.
I get the motivation and I like the push to report methodological details. I dislike collapsing quality onto a discrete 5-point scale as it'll create bad incentives, along the lines of Goodhart's law. Vendors will reverse engineer how to produce "5 star" but misleading reports, which people will blindly trust because of the quality label. The response "it's better than nothing" is somewhat true, but it mischaracterises the status quo. Imo the community doesn't blindly trust vendor reports at the moment. There's a more informal reputation system where some reports are perceived to be good and believable (e.g. DBIR) for good reasons, and other reports are largely ignored by reasonable people. Imo informal reputation is the worst option, apart from all the others.
This sounds like something akin to cybersecurity labels for IoT products, which I don't think it is practical or will be relevant. The idea derives from a mindset that cybersecurity is purely technical, while ignoring the process, behaviors, and adversaries. But if we are talking labels for industry reports, then that falls into a better framework classification, like fiction vs non-fiction. If this is about applying critical thinking monikers to reports, such as sponsor bias, marketing/sales pitch, survey vs actual data, small sample size warning, elongated history assumptions, assumptions based upon low accuracy models, etc., then I am all in. For forward looking conclusions, how about grading the authors previous analysis and predictions to see if their methodologies and expertise actually measure up? Far too often people confuse marketing and aspirational articles for proper research and analysis.
The first thing I look at is the source of data in the report. If it’s survey data, I’m less inclined to put credence in the report. If the data is from “real” transactions, breaches, incidents, etc. I believe it’s a much more meaningful report. Findings from these reports should allow a focus on risks for Cyber Managers.
Wade, the library was useful and it's a real loss. Looking forward to the follow-up on rating criteria. The report quality problem cuts both ways. Most of us treat "published by a credible firm" as the rating, because there isn't a better one. A real rubric would change what gets produced, not just what gets read.
It was a great resource and I hope it is revived one day
Relevant post from Tony: https://www.linkedin.com/posts/tonymartinvegue_if-i-were-in-charge-of-things-every-security-share-7459634580543782912-6sfk/