Gene Spafford

There's always been talk of how academic and real-world security research are far apart. For me personally, this has become quite clear in the LLM/AI security research space. As an example data point, let's examine the problem of system prompt extraction. On the academic side, I still see many papers that are claiming this to be a security problem where the system prompt must be protected from extraction. On the real-world side, system prompts generally do not have any strong protections against extraction anymore. Most frontier LLM systems do not have any specific protections in place to prevent extraction. Anthropic (noted for being safety-first) even proactively publishes these prompts (https://lnkd.in/gu6k6eqa). Additionally, there are many trivial ways to get the model to output the system prompt verbatim. Now, the system-prompt-protectors might argue that a frontier LLM might make it public, but there's a whole set of wrapper-systems whose only financial value/intellectual property is the system prompt. They will give evidence of prompt marketplaces to support this. In my opinion, this is fundamentally incorrect because of the logical contradiction that system prompt protection introduces. A system prompt by definition is supposed to affect the LLM's responses on a global level. Thus, it always has an effect on the model's outputs, and this is observable to the attacker. From a security perspective, if the system prompt is to be provably protected, it should not have any effect on the model's outputs. Otherwise, the prompt can be approximated from the observable effects, and you get an arms race that doesn't lead anywhere good. This is the contradiction. Either you get security and you negate the point of the system prompt, or you get the system prompt's utility and you have no way making a logical or formal argument about what security you are providing.

46

4 Comments

The ATARC (Advanced Technology Academic Research Center) cATO Working Group has published a new paper: Achieving ATOs Through Declarative Architecture and Shared Common Platforms. It outlines how agencies can move away from manual compliance work and toward architectures that are transparent, consistent, and repeatable. Shared configurations, Infrastructure as Code, and tiered service models help teams deliver secure systems faster and with less friction. Appreciate the work from everyone involved. Link to paper: https://lnkd.in/epZi4qKw #ATO #cATO #Cybersecurity #Technology #Software

32

5 Comments

The National Security Agency just released the first two Zero Trust Implementation Guideline (ZIG) documents in their new series: a Primer, and the Discovery phase document. These releases are both substantive documents, building on the National Institute of Standards and Technology (NIST) Cybersecurity and Infrastructure Security Agency and United States Department of War Zero Trust foundations. We will be reviewing these, and talking about them in a live session next week. Look for information on this soon! Links to source documents: https://lnkd.in/ecx-MY9Z #ZeroTrust

18

1 Comment

After DHS did not renew its funding contract for reasons unspecified, MITRE’s 25-year-old Common Vulnerabilities and Exposures (CVE) program was slated for an abrupt shutdown on April 16, which would have left security flaw tracking in limbo. CISA stepped in to provide a bridge. A CISA spokesperson sent CSO a statement saying, “The CVE Program is invaluable to cyber community and a priority of CISA. Last night, CISA executed the option period on the contract to ensure there will be no lapse in critical CVE services. We appreciate our partners’ and stakeholders’ patience.” Sources say the contract extension will last 11 months. Yosry Barsoum, vice president and director of the Center for Securing the Homeland at MITRE, commented: “Thanks to actions taken by the government, a break in service for the Common Vulnerabilities and Exposures (CVE®) Program and the Common Weakness Enumeration (CWE™) Program has been avoided. As of Wednesday morning, April 16, 2025, CISA identified incremental funding to keep the Programs operational. We appreciate the overwhelming support for these programs that have been expressed by the global cyber community, industry, and government over the last 24 hours. The government continues to make considerable efforts to support MITRE’s role in the program and MITRE remains committed to CVE and CWE as global resources.” https://lnkd.in/gPhB8vVQ

170

12 Comments

Very pleased to share that the paper "Quantifying costs of enhanced security in multifactor authentication", co-authored with Seth Hastings, PhD, Neil Gandal, and Noa Barnir, was published in Information Systems Frontiers: https://lnkd.in/g5EFhKvU. We often hear about the importance of multi-factor authentication (MFA) to protect people and organizations. We completely agree! Nonetheless, there are real costs to adopting MFA. In addition to paying for the technology itself, MFA also introduces opportunity costs, from the extra time spent authenticating to the time legitimate users spend locked out if they cannot complete authentication. Through a collaboration with The University of Tulsa Office of IT, we inspected several years' worth of authentication logs. When the university switched over to a more onerous form of MFA (requiring a two-digit code through the mobile app in addition to biometrics), these opportunity costs shot way up, especially when using mobile devices. For full details, check out the paper! Pre-publication version available in case the article isn't available through your institution: https://lnkd.in/gtfm7MAb.

97

6 Comments

🚨 The regulatory chaos in cybersecurity is staggering: 45 different incident reporting requirements across 22 federal agencies, with 49-79% of parameters in direct conflict. We need urgent reform and CISA modernization for today's AI-enabled threat landscape. My thoughts in @TechNewsWorld 👇 https://bit.ly/45V77km #Cybersecurity #CISA #RegulatoryReform #AIThreats

20

1 Comment

Excited to share that I was quoted in Evan Schuman’s latest piece for CSO Online reacting to NIST’s new AI concept paper.   👉 “You can’t patch what you don’t know is running. That applies to AI, too. National Institute of Standards and Technology (NIST) should make AI model inventories step one. If companies don’t even know which models employees are using, the rest of the controls don’t matter.”   Grateful for the opportunity to weigh in on this important discussion around AI security and risk management. You can read the full article here: https://lnkd.in/gg5TMfTt

53

5 Comments