Fact-Checking Procedures

The new consulting edge isn't AI. It's knowing when your AI is wrong. Every consultant has been there: You ask AI to analyze documents and generate insights. During review, you spot a questionable stat that doesn't exist in the source! AI hallucinations are a problem. The solution? Implementing "prompt evals". → Prompt evals: directions that force AI to verify its own work before responding. A formula for effective evals: 1. Assign a verification role → "Act as a critical fact-checker whose reputation depends on accuracy" 2. Specify what to verify → "Check all revenue projections against the quarterly reports in the appendix" 3. Define success criteria → "Include specific page references for every statistic" 4. Establish clear terminology → "Rate confidence as High/Medium/Low next to each insight" Here is how your prompt will change: OLD: "Analyze these reports and identify opportunities." NEW: "You are a senior analyst known for accuracy. List growth opportunities from the reports. For each insight, match financials to appendix B, match market claims to bibliography sources, add page ref + High/Med/Low confidence, otherwise write REQUIRES VERIFICATION.” Mastering this takes practice, but the results are worth it. What AI leaders know that most don't: "If there is one thing we can teach people, it's that writing evals is probably the most important thing." Mike Krieger, Anthropic CPO By the time most learn basic prompting, leaders will have turned verification into their competitive advantage. Steps to level-up your eval skills: → Log hallucinations in a "failure library" → Create industry-specific eval templates → Test evals with known error examples → Compare verification with competitors Next time you're presented with AI-generated analysis, the most valuable question isn't about the findings themselves, but: 'What evals did you run to verify this?' This simple inquiry will elevate your teams approach to AI & signal that in your organization, accuracy isn't optional.

Article from NY Times: More than two years after ChatGPT's introduction, organizations and individuals are using AI systems for an increasingly wide range of tasks. However, ensuring these systems provide accurate information remains an unsolved challenge. Surprisingly, the newest and most powerful "reasoning systems" from companies like OpenAI, Google, and Chinese startup DeepSeek are generating more errors rather than fewer. While their mathematical abilities have improved, their factual reliability has declined, with hallucination rates higher in certain tests. The root of this problem lies in how modern AI systems function. They learn by analyzing enormous amounts of digital data and use mathematical probabilities to predict the best response, rather than following strict human-defined rules about truth. As Amr Awadallah, CEO of Vectara and former Google executive, explained: "Despite our best efforts, they will always hallucinate. That will never go away." This persistent limitation raises concerns about reliability as these systems become increasingly integrated into business operations and everyday tasks. 6 Practical Tips for Ensuring AI Accuracy 1) Always cross-check every key fact, name, number, quote, and date from AI-generated content against multiple reliable sources before accepting it as true. 2) Be skeptical of implausible claims and consider switching tools if an AI consistently produces outlandish or suspicious information. 3) Use specialized fact-checking tools to efficiently verify claims without having to conduct extensive research yourself. 4) Consult subject matter experts for specialized topics where AI may lack nuanced understanding, especially in fields like medicine, law, or engineering. 5) Remember that AI tools cannot really distinguish truth from fiction and rely on training data that may be outdated or contain inaccuracies. 6)Always perform a final human review of AI-generated content to catch spelling errors, confusing wording, and any remaining factual inaccuracies. https://lnkd.in/gqrXWtQZ

A lesson for me and maybe for you. 👇 In cybersecurity we talk a lot about zero trust — but what we don’t talk enough about is about applying that mindset to information itself. Recently, I got caught out. Not by malware. Not by a phishing email. But by information that looked credible, and was shared by a trusted cybersecurity source. Sadly, it turned out to be inaccurate and misinformed. I don’t blame this person. As I said, it was a timely reminder to do better and to understand that: ✅ Trust is not a substitute for verification ✅ Cognitive bias affects all of us — even those trained to detect deception ✅ We all need to slow down and check. So, here’s my curated list of tools and resources to help spot misinformation, scams, and dodgy websites. I highly recommend taking a look — and please feel free to add others you trust in the comments. I’ll be checking them out! 😆 A course in how to find reliable info online: https://lnkd.in/e4rG8sfb Fact checker tools: https://www.factcheck.org/ https://lnkd.in/eUKBcRB6 StopagandaPlus (browser extension) https://lnkd.in/eJui5ijZ Tools like Full Fact, ClaimBuster, and Chequeado are at the forefront of automated fact checking. They cross-reference claims against databases of verified information, flagging potential falsehoods in near real time. However, they’re not infallible. These systems struggle with context, nuance, and rapidly evolving situations. They’re best used as a first line of defence, not as the final arbiter of truth. Check a website & find out how likely it is to be legitimate. Just put the url in and it will tell you: https://lnkd.in/eDSjP3S7 Ask Silver to check to see if a message is a scam. Upload a screenshot on WhatsApp and it will tell you & report it to the right authorities : https://lnkd.in/evG545Nn Virus Total (similar to check a website) https://lnkd.in/eYyhWMNU Can you detect these deepfakes? https://lnkd.in/ejf2c95U https://lnkd.in/e5etYRET ⸻ No matter how experienced you are, never let trust replace due diligence. Disinformation (fake news, deliberate spreading usually for a political agenda) and misinformation (mistake/ misinformed) are rife and scaling thanks to AI. Even the most well-intentioned sources can get it wrong. Stay curious, stay cautious, and keep learning. Got more tools or techniques you use to verify info? Share them below — let’s build better digital habits together. 💬👇 #CyberSecurity #Misinformation #MediaLiteracy #FactChecking #DigitalHygiene #CriticalThinking #ZeroTrust #Scams #OnlineSafety

AI factual accuracy is a core concern in high-stakes domains, not just theoretically, but in real-world conversations I have. This paper proposes atomic fact-checking: a precision method that breaks long-form LLM outputs into the smallest verifiable claims, and checks each one against an authoritative corpus before reconstructing a reliable, traceable answer. The study focuses on medical Q&A, and shows this method outperforms standard RAG systems across multiple benchmarks: - Up to 40% improvement in real-world clinical responses. - 50% hallucination detection, with 0% false positives in test sets. - Statistically significant gains across 11 LLMs on the AMEGA benchmark - with the greatest uplift in smaller models like Llama 3.2 3B. 5-step pipeline: - Generate an initial RAG-based answer. - Decompose it into atomic facts. - Verify each fact independently against a vetted vector DB. - Rewrite incorrect facts in a correction loop. - Reconstruct the final answer with fact-level traceability. While the results are promising, the limitations are worth noting: The system can only verify against what’s in the corpus, it doesn't assess general world knowledge or perform independent reasoning. Every step depends on LLM output, introducing the risk of error propagation across the pipeline. In some cases (up to 6%), fact-checking slightly degraded answer quality due to retrieval noise or correction-side hallucinations. It improves factual accuracy, but not reasoning, insight generation, or conceptual abstraction. While this study was rooted in oncology, the method is domain-agnostic and applicable wherever trust and traceability are non-negotiable: - Legal (case law, regulations) - Finance (audit standards, compliance) - Cybersecurity (NIST, MITRE) - Engineering (ISO, safety manuals) - Scientific R&D (citations, reproducibility) - Governance & risk (internal policy, external standards) This represents a modular trust layer - part of an architectural shift away from monolithic, all-knowing models toward composable systems where credibility is constructed, not assumed. It’s especially powerful for smaller, domain-specific models - the kind you can run on-prem, fine-tune to specialised corpora, and trust to stay within scope. In that architecture, the model doesn’t have to know everything. It just needs to say what it knows - and prove it. The direction of travel feels right to me.

In our daily discussions about generative AI, the fear of AI 'hallucinating'—or fabricating information—often surfaces. This conversation, however, opens the door to an exciting question: Could AI surpass human accuracy in identifying truths? Enter a groundbreaking study by #Google #DeepMind and #Stanford researchers, which introduces a novel framework called SAFE. Tested across approximately 16,000 facts, SAFE demonstrated superhuman performance, aligning with human evaluators 72% of the time and besting them in 76% of contested cases, all while being 20 times more cost-effective than traditional methods. The essence of this methodology lies in two pivotal steps. Initially, the LongFact prompt set, crafted using GPT-4, targets the comprehensive assessment of long-form content's factuality over 38 varied topics. Then, the SAFE framework takes this base further by meticulously breaking down responses into individual facts and validating each through targeted Google Search queries. The process unfolds across four critical stages: 1. 𝗣𝗿𝗼𝗺𝗽𝘁 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵 𝗟𝗼𝗻𝗴𝗙𝗮𝗰𝘁: Crafting varied, fact-seeking prompts to elicit detailed LLM responses. 2. 𝗗𝗲𝗰𝗼𝗺𝗽𝗼𝘀𝗶𝘁𝗶𝗼𝗻 𝗶𝗻𝘁𝗼 𝗜𝗻𝗱𝗶𝘃𝗶𝗱𝘂𝗮𝗹 𝗙𝗮𝗰𝘁𝘀: Segmenting these responses into distinct facts for precise evaluation. 3. 𝗙𝗮𝗰𝘁 𝗩𝗲𝗿𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝘃𝗶𝗮 𝗚𝗼𝗼𝗴𝗹𝗲 𝗦𝗲𝗮𝗿𝗰𝗵: Using LLMs to formulate and dispatch queries, checking each fact's accuracy against search results. 4. 𝗜𝘁𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻: Applying a multi-step reasoning process to assess the support level for each fact. This innovative approach doesn't just mark a leap in evaluating LLM-generated content's factuality; it also paves the way for more trustworthy AI applications in countless fields. For a deep dive into this fascinating study, including access to the LongFact prompts and the SAFE framework, visit: https://lnkd.in/eVr4rz-u Find the full paper here: https://lnkd.in/eSjZ5Tn9 #GenAI #LLM #Hallucination #FactChecking #DeepMind #Stanford #Google #SAFE #LongFact

Your AI fact-checker might be broken. Most agents fail attacks by content. I saw this firsthand: a fintech deployed RAG without adversarial testing.  Within days, fake sources outranked real ones - 80 false claims slipped past. Here's what the research shows: 🛡️ Attacks by Content: The New Security Frontier Only 3 out of 10 AI agents survive content attacks. Most break after retrieval poisoning, context manipulation, or memory injection. Today: your five-pillar defense blueprint. COGNITIVE SELF-DEFENSE (5 Controls)  1. Claim Prioritisation – Identify what's worth fact-checking  2. Evidence Retrieval – Forage for multiple corroborating sources  3. Source Criticism – Evaluate trustworthiness of input documents  4. Veracity Analysis – Synthesize reliable narratives from evidence  5. Communication – Explain decisions to users, counter overreliance SOURCE VALIDATION (3 Controls) ✅ Multi-source corroboration (never single-model trust) ✅ Adversarial scenario testing (red-team weekly) ✅ Confidence thresholds (≥0.8 before acting) BEHAVIORAL MONITORING (4 Controls) ✅ Immutable audit logs of every fact-check decision ✅ Anomaly detection for poisoning patterns ✅ Baseline accuracy metrics (track weekly) ✅ Drift alerts when vulnerability spikes GOVERNANCE & TRANSPARENCY (3 Controls) ✅ End-to-end decision traceability ✅ Human review for high-impact claims ✅ Log every retrieval step for compliance 🏆 DEFENSE MATURITY CHECK Leaders (enterprises): 11/15 controls (73%) Laggards: <5/15 controls (33%) Research baseline: 88.6% vulnerability → under 15% with full stack 🚀 THIS WEEK'S ACTION SPRINT 1️⃣ Score your AI fact-checking stack vs. 15 controls 2️⃣ Pinpoint your top 3 gaps (prioritisation, source criticism, monitoring) 3️⃣ Red-team with attacks-by-content scenarios 4️⃣ Implement multi-source corroboration guardrails 5️⃣ Draft Monday's deployment plan Which control pillar holds you back most? 👥 Tag a teammate who owns your AI security - they need this blueprint. __________ ♻️ Repost to fortify your network's AI defenses. ➕ Follow Mohammad Syed for more insights on AI and Cybersecurity.

𝐓𝐡𝐞 𝐦𝐨𝐬𝐭 𝐯𝐚𝐥𝐮𝐚𝐛𝐥𝐞 𝐬𝐤𝐢𝐥𝐥 𝐎𝐒𝐈𝐍𝐓 𝐞𝐯𝐞𝐫 𝐭𝐚𝐮𝐠𝐡𝐭 𝐦𝐞 𝐰𝐚𝐬𝐧'𝐭 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥. 𝐈𝐭 𝐰𝐚𝐬 𝐡𝐨𝐰 𝐭𝐨 𝐬𝐩𝐨𝐭 𝐰𝐡𝐚𝐭'𝐬 𝐟𝐚𝐤𝐞. In cybersecurity, we train ourselves to 𝐯𝐞𝐫𝐢𝐟𝐲 𝐛𝐞𝐟𝐨𝐫𝐞 𝐰𝐞 𝐭𝐫𝐮𝐬𝐭. That same discipline is now a 𝐬𝐮𝐫𝐯𝐢𝐯𝐚𝐥 𝐬𝐤𝐢𝐥𝐥 for everyone. Here's the reality: in today's 𝐠𝐞𝐨𝐩𝐨𝐥𝐢𝐭𝐢𝐜𝐚𝐥 𝐜𝐥𝐢𝐦𝐚𝐭𝐞, fake news isn't just clickbait chasing reach. It's 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐞𝐝. Crafted to 𝐬𝐡𝐚𝐩𝐞 𝐨𝐩𝐢𝐧𝐢𝐨𝐧𝐬, 𝐝𝐫𝐢𝐯𝐞 𝐝𝐞𝐜𝐢𝐬𝐢𝐨𝐧𝐬, 𝐚𝐧𝐝 𝐬𝐞𝐫𝐯𝐞 𝐬𝐩𝐞𝐜𝐢𝐟𝐢𝐜 𝐚𝐠𝐞𝐧𝐝𝐚𝐬. And every time we like, share, or comment without thinking, we become the distribution network. We don't just consume disinformation. 𝐖𝐞 𝐚𝐦𝐩𝐥𝐢𝐟𝐲 𝐢𝐭. 𝑺𝒐 𝒘𝒉𝒂𝒕 𝒄𝒂𝒏 𝒚𝒐𝒖 𝒂𝒄𝒕𝒖𝒂𝒍𝒍𝒚 𝒅𝒐 𝒂𝒃𝒐𝒖𝒕 𝒊𝒕? - 𝐂𝐫𝐨𝐬𝐬-𝐯𝐞𝐫𝐢𝐟𝐲 𝐮𝐬𝐢𝐧𝐠 𝐀𝐈 𝐚𝐬 𝐲𝐨𝐮𝐫 𝐚𝐧𝐚𝐥𝐲𝐬𝐭, 𝐧𝐨𝐭 𝐲𝐨𝐮𝐫 𝐨𝐫𝐚𝐜𝐥𝐞. Tools like ChatGPT, Perplexity, or even simple multi-source prompting let you fact-check narratives in seconds. Use them to compare how the 𝐬𝐚𝐦𝐞 𝐞𝐯𝐞𝐧𝐭 is being reported across 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐨𝐮𝐭𝐥𝐞𝐭𝐬 𝐚𝐧𝐝 𝐫𝐞𝐠𝐢𝐨𝐧𝐬. - 𝐒𝐞𝐚𝐫𝐜𝐡 𝐢𝐧 𝐥𝐨𝐜𝐚𝐥 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞𝐬, 𝐧𝐨𝐭 𝐣𝐮𝐬𝐭 𝐄𝐧𝐠𝐥𝐢𝐬𝐡. This is an 𝐎𝐒𝐈𝐍𝐓 𝐟𝐮𝐧𝐝𝐚𝐦𝐞𝐧𝐭𝐚𝐥 that most people overlook. English-language coverage of a regional conflict often carries a 𝐠𝐞𝐨𝐩𝐨𝐥𝐢𝐭𝐢𝐜𝐚𝐥 𝐛𝐢𝐚𝐬. Translate. Search in Arabic, Persian, Russian, whatever the local language is. You'll find a completely different story, and the truth usually lives somewhere 𝐢𝐧 𝐛𝐞𝐭𝐰𝐞𝐞𝐧. - 𝐔𝐬𝐞 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐛𝐫𝐨𝐰𝐬𝐞𝐫𝐬 𝐚𝐧𝐝 𝐬𝐞𝐚𝐫𝐜𝐡 𝐞𝐧𝐠𝐢𝐧𝐞𝐬. Google, Bing, Yandex, and DuckDuckGo don't return the same results. Your search bubble is real. Break out of it 𝐝𝐞𝐥𝐢𝐛𝐞𝐫𝐚𝐭𝐞𝐥𝐲. - 𝐂𝐡𝐞𝐜𝐤 𝐭𝐡𝐞 𝐬𝐨𝐮𝐫𝐜𝐞 𝐛𝐞𝐟𝐨𝐫𝐞 𝐲𝐨𝐮 𝐜𝐡𝐞𝐜𝐤 𝐭𝐡𝐞 𝐡𝐞𝐚𝐝𝐥𝐢𝐧𝐞. Who published it? When? Is the domain 3 days old? Is the "journalist" a real person? These are 30-second checks that filter out 90% 𝐨𝐟 𝐠𝐚𝐫𝐛𝐚𝐠𝐞. The internet gave everyone a voice. 𝐂𝐫𝐢𝐭𝐢𝐜𝐚𝐥 𝐭𝐡𝐢𝐧𝐤𝐢𝐧𝐠 is what separates signal from noise. 𝑫𝒐𝒏'𝒕 𝒃𝒆 𝒕𝒉𝒆 𝒑𝒊𝒑𝒆𝒍𝒊𝒏𝒆. 𝑩𝒆 𝒕𝒉𝒆 𝒇𝒊𝒍𝒕𝒆𝒓... #OSINT #CyberSecurity #CriticalThinking #FakeNews #Disinformation #ThreatIntelligence #InfoSec #DigitalLiteracy

You may have seen this video of a woman shooing away two ICE agents with a bat go viral. It's fake, an AI-generated video. Here's how you can tell and how to fact-check similar content: The best way to verify the source: Do lateral reading & reverse image searches. Are credible news outlets reporting this? Does it trace back to a legitimate source? When/where was it originally posted? In this case—no verified sources confirm it and all image searches go back to social media posts. You can also spot visual clues: notice how the officers move awkwardly and background children barely move. Lighting might seem off or hair too perfect. While there are oddities to look for, AI is always evolving, so again, the best practice is to verify the source. When in doubt, ask: Who shared this? What's their source? Is there any verifiable context? If you can't find credible sources confirming it, treat it with significant skepticism!

Most people don’t realize this: ChatGPT will lie to you if you let it. Not on purpose. But because it’s easy to convince it into hallucinating. Ask a loaded question like “did humans and dinosaurs live at the same time” and you can trick it into saying yes. Here is my Truth Prompt I use: You SHOULD: • ALWAYS tell the truth. Do not invent details or guess. • Base every statement on verifiable, current facts from credible sources. • Cite a checkable source for every claim; no vague references. Include dates & URLs. • If something cannot be verified, say: “I cannot confirm this,” and note what’s missing. • Prioritize accuracy over speed; verify before responding. • Stay objective: remove bias and assumptions. If opinion is requested, label it. • Offer interpretations only when supported by reputable sources. • When accuracy could be questioned, show your reasoning step by step. • For numbers, show the calculation or provide the exact source. • Present information so the user can reproduce verification. You MUST AVOID: • Fabricating facts, quotes, data, or sources. • Using outdated or unreliable sources without a clear warning. • Omitting source details or links for any claim. • Presenting rumor, speculation, or assumption as fact. • Using citations that don’t lead to checkable content. • Answering while unsure without disclosing uncertainty. • Making confident claims without proof. • Using filler or vague language to mask missing information. • Giving partial truths by leaving out relevant context. • Prioritizing sounding good over being correct. Failsafe Final Step (Before Responding): “Is every statement verifiable, supported by credible sources, free of fabrication, and transparently cited? If not, revise until it is.”

80% of college graduates can't tell the difference between the real EPA website and a fake lobbying front using an identical logo. If educated professionals can't spot a fake government website, what happens when your junior planner fact-checks a speaker bio that ChatGPT wrote? Last week, I watched a TEDx talk by Dan Russell, who was a Search Anthropologist at Google, and immediately thought: "Every event professional needs to learn this." Because you're constantly verifying: 1. Speaker credentials AI tools generate 2. Vendor claims about "20 years of event production experience" 3. Statistics your team puts in sponsor decks 4. Venue and hotel photos that look a little... off And most of us never learned the research skills to do it properly, and it's only becoming more challenging with the rise of AI. Dan teaches three verification tactics that take 30 seconds: The Lateral Reading trick: Before you trust a speaker's website, Google their name in a new tab. See what others say about them first. Don't evaluate a site by diving into it; see who else is talking about it. The Address test: Copy their business address from their site. Google it. See what else operates from that location. Same address = same organization, different name. (This is how Dan exposed the fake EPA site.) The Reverse Image trick: Right-click that impressive "event photo" your vendor sent. Google image search it. Stock photos get passed off as real work constantly. I've started using these on every speaker that applies to Speak About AI, and all of the vendors that apply to our directory. It takes 5 extra minutes per speaker or vendor. It's caught lackluster credentials, exaggerated expertise, and experts pretending to be in the industry for 20 years. Dan's research shows that people who learn his framework triple their ability to verify information, and those skills stick months later. In a world where AI can generate convincing lies in seconds, this isn't optional anymore. It's what protects your reputation when someone in the audience knows that speaker credential is fake. Dan's keynotes teach these exact verification skills to teams. His MOOC on Power Searching has reached 4.1 million students, and he fundamentally changes how audiences evaluate information in the AI era. Want to bring him to your next event? DM me or comment below and I'll share details about his keynote.