How to Ensure AI Accuracy

LLMs are great for data processing, but using new techniques doesn't mean you get to abandon old best practices. The precision and accuracy of LLMs still need to be monitored and maintained, just like with any other AI model. Tips for maintaining accuracy and precision with LLMs: • Define within your team EXACTLY what the desired output looks like. Any area of ambiguity should be resolved with a concrete answer. Even if the business "doesn't care," you should define a behavior. Letting the LLM make these decisions for you leads to high variance/low precision models that are difficult to monitor. • Understand that the most gorgeously-written, seemingly clear and concise prompts can still produce trash. LLMs are not people and don't follow directions like people do. You have to test your prompts over and over and over, no matter how good they look. • Make small prompt changes and carefully monitor each change. Changes should be version tracked and vetted by other developers. • A small change in one part of the prompt can cause seemingly-unrelated regressions (again, LLMs are not people). Regression tests are essential for EVERY change. Organize a list of test case inputs, including those that demonstrate previously-fixed bugs and test your prompt against them. • Test cases should include "controls" where the prompt has historically performed well. Any change to the control output should be studied and any incorrect change is a test failure. • Regression tests should have a single documented bug and clearly-defined success/failure metrics. "If the output contains A, then pass. If output contains B, then fail." This makes it easy to quickly mark regression tests as pass/fail (ideally, automating this process). If a different failure/bug is noted, then it should still be fixed, but separately, and pulled out into a separate test. Any other tips for working with LLMs and data processing?

Founders getting 10x better AI outputs aren't using Secret Tools. They're using Structured Frameworks. I spent 18 months working with AI solutions. The difference between good and bad outputs? It's never the AI. It's always the prompt. The immigrant mathematics of prompting: Random questions + No structure = Generic garbage Right framework + Clear context = Outputs you can actually use Here are 9 prompting frameworks that will make you dangerous: LEVEL 1: EVERYDAY TASKS C-L-E-A-R → Context: Why you need this → Logic: What you're trying to accomplish → Expectations: What the answer should look like → Action: The specific task → Restrictions: Limits on tone, length, sources S-M-A-R-T → Specific: No vague questions → Measurable: Define success criteria → Achievable: Keep it realistic → Relevant: Aligned with your goal → Time-bound: Include deadlines if applicable Q-U-E-S-T → Question: Start with a clear problem → Understanding: What you already know → Expectation: What a good answer looks like → Scope: What to include or exclude → Time: Relevance timeframe LEVEL 2: DETAILED WORK G-U-I-D-E → Goal: What you're trying to achieve → Understanding: Your prior knowledge → Information: Data you need → Direction: How to structure the response → Evaluation: How you'll assess quality F-O-C-U-S → Function: Role the AI should play → Outcome: What the ideal response includes → Criteria: Key quality factors → Underlying assumptions: Biases to acknowledge → Strategy: Research method you prefer I-D-E-A → Intent: Purpose behind your research → Details: Background information → Examples: References to shape the response → Adjustments: Room for refinement LEVEL 3: COMPLEX PROJECTS R-I-S-E-N → Requirement: What information you need → Information: Supporting data required → Strategy: Approach the AI should take → Evaluation: How to measure accuracy → Negotiation: Flexibility in the response R-H-O-D-E-S → Research: Topic and key aspects → Hypothesis: Testable statement → Objectives: Specific goals → Development: Plan to test strategies → Execution: Analyze and collect data → Synthesis: Summarize findings C-R-E-A-T-E → Conceptualize: Define the challenge → Research: Gather background info → Experiment: Test different approaches → Analyze: Evaluate results → Transform: Refine into a plan → Evaluate: Measure success You don't need all 9. Pick one from each level. Master those three. 💭 Which framework are you trying first? ♻️ Repost to help someone stop blaming AI for bad outputs ➕ Follow Anna Bilan ⚡️ for AI insights you can use today

The new consulting edge isn't AI. It's knowing when your AI is wrong. Every consultant has been there: You ask AI to analyze documents and generate insights. During review, you spot a questionable stat that doesn't exist in the source! AI hallucinations are a problem. The solution? Implementing "prompt evals". → Prompt evals: directions that force AI to verify its own work before responding. A formula for effective evals: 1. Assign a verification role → "Act as a critical fact-checker whose reputation depends on accuracy" 2. Specify what to verify → "Check all revenue projections against the quarterly reports in the appendix" 3. Define success criteria → "Include specific page references for every statistic" 4. Establish clear terminology → "Rate confidence as High/Medium/Low next to each insight" Here is how your prompt will change: OLD: "Analyze these reports and identify opportunities." NEW: "You are a senior analyst known for accuracy. List growth opportunities from the reports. For each insight, match financials to appendix B, match market claims to bibliography sources, add page ref + High/Med/Low confidence, otherwise write REQUIRES VERIFICATION.” Mastering this takes practice, but the results are worth it. What AI leaders know that most don't: "If there is one thing we can teach people, it's that writing evals is probably the most important thing." Mike Krieger, Anthropic CPO By the time most learn basic prompting, leaders will have turned verification into their competitive advantage. Steps to level-up your eval skills: → Log hallucinations in a "failure library" → Create industry-specific eval templates → Test evals with known error examples → Compare verification with competitors Next time you're presented with AI-generated analysis, the most valuable question isn't about the findings themselves, but: 'What evals did you run to verify this?' This simple inquiry will elevate your teams approach to AI & signal that in your organization, accuracy isn't optional.

I have fallen for the “facts” AI has presented me. Several times. And I’m not the only one. You hear about it in the news, notice subtle mistakes yourself, or start double-checking sources. The fact is, most people suck at understanding the risks of AI hallucinations. And imagine the impact when hallucinations impact healthcare. If you avoid these 5 common mistakes, the risk should be reduced. Ignoring the potential for AI hallucinations → Assuming AI systems are infallible and will always provide accurate information. Do this instead ↳ Recognize that AI tools can produce incorrect or misleading information ↳ Critically evaluate AI outputs before relying on them for healthcare decisions Underestimating the consequences of AI hallucinations → Failing to recognize the serious impact AI errors can have on patient care and outcomes. Do this instead ↳ Understand that AI hallucinations in healthcare can lead to unnecessary treatments, patient anxiety, and even harm ↳ Prioritize the development of trustworthy and reliable AI systems Lack of transparency and explainability → Using AI tools without understanding how they work or how they arrived at their conclusions. Do this instead ↳ Demand transparency from AI providers about their models' inner workings and training data ↳ Prioritize the development of explainable AI systems that can justify their outputs Inadequate testing and validation → Deploying AI tools without rigorous testing to ensure their accuracy and reliability. Do this instead ↳ Implement robust testing and validation processes for AI systems before using them in healthcare settings ↳ Continuously monitor AI performance and update models as needed Indifference about AI risks → Assuming that AI hallucinations are a minor issue that can be easily managed. Do this instead ↳ Stay informed about the latest research and developments related to AI hallucinations in healthcare ↳ Advocate for the development of appropriate safeguards and regulations to manage AI risks Though AI tools have great potential to improve healthcare, we need to be aware about the risks of AI hallucinations. By understanding these risks and taking proactive steps to mitigate them, we can harness the power of AI while ensuring patient safety and high-quality care. What plan does your organization have to reduce AI hallucinations?

Should you blindly trust AI? Most teams make a critical mistake with AI - we accept its answers without question, especially when it seems so sure. But AI confidence ≠ human confidence. Here’s what happened: The AI system flagged a case of a rare autoimmune disorder. The doctor, trusting the result, recommended an aggressive treatment plan. But something felt off. When I was called in to review, we discovered the AI had misinterpreted an MRI anomaly. The patient had a completely different condition - one that didn't require that aggressive treatment. One wrong decision, based on misplaced trust, could’ve caused real harm. To prevent this amid the integration of AI into the workforce, I built the “acceptability threshold” framework. Here’s how it works: This framework is copyrighted: © 2025 Sol Rashidi. All rights reserved. 1. Measure how accurate humans are at a task (our doctors were 93% accurate on CT scans) 2. Use that as our minimum threshold for AI. 3. If AI's confidence falls below this human benchmark, a person reviews it. This approach transformed our implementation and prevented future mistakes. The best AI systems don't replace humans - they know when to ask for human help. What assumptions about AI might be putting your projects at risk?

We have to internalize the probabilistic nature of AI. There’s always a confidence threshold somewhere under the hood for every generated answer and it's important to know that AI doesn’t always have reasonable answers. In fact, occasional "off-the-rails" moments are part of the process. If you're an AI PM Builder (as per my 3 AI PM types framework from last week) - my advice: 1. Design for Uncertainty: ✨Human-in-the-loop systems: Incorporate human oversight and intervention where necessary, especially for critical decisions or sensitive tasks. ✨Error handling: Implement robust error handling mechanisms and fallback strategies to gracefully manage AI failures (and keep users happy). ✨User feedback: Provide users with clear feedback on the confidence level of AI outputs and allow them to provide feedback on errors or unexpected results. 2. Embrace an experimental culture & Iteration / Learning: ✨Continuous monitoring: Track the AI system's performance over time, identify areas for improvement, and retrain models as needed. ✨A/B testing: Experiment with different AI models and approaches to optimize accuracy and reliability. ✨Feedback loops: Encourage feedback from users and stakeholders to continuously refine the AI product and address its limitations. 3. Set Realistic Expectations: ✨Educate users: Clearly communicate the potential for AI errors and the inherent uncertainty involved about accuracy and reliability i.e. you may experience hallucinations.. ✨Transparency: Be upfront about the limitations of the system and even better, the confidence levels associated with its outputs.

AI can generate information that sounds accurate but is completely wrong. AI hallucinations can undermine trust in reporting, introduce compliance exposure, and create financial or operational losses. They can also surface sensitive data or misinform decisions that affect capital allocation, investor communication, and audit readiness. AI hallucinations are not a signal to slow down innovation. They are a signal to strengthen your governance and controls. With a thoughtful risk management approach, leaders can understand uncertainty and build a more confident, resilient AI strategy. Considerations for leaders to reduce AI hallucination risk: 1. Create a validation and review process for AI generated financial outputs. Leaders must ensure that any AI generated forecasts, variance analyses, reconciliations, or narrative summaries have structured validation for source accuracy and logic. 2. Strengthen compliance and regulatory controls within AI workflows. AI hallucinations can create errors that lead to noncompliance and regulatory exposure. Leaders can embed compliance checkpoints into AI driven processes to avoid misstatements, inaccurate filings, or unintended disclosure. 3. Prioritize data governance using high quality, company specific data to reduce the risk of fabricated or inaccurate outputs. This is critical for forecasting, scenario modeling, and automated reporting. 4. Use retrieval augmented generation and automated reasoning for workflows. Pairing these methods anchors AI generated analysis in verified data sources rather than probability-based guesses. 5. Enable filtering and moderation tools to block misleading or irrelevant results. Teams cannot work from flawed or unverified outputs. Filters help prevent misleading content from entering critical workflows or influencing decisions. AI is gaining traction. Now is the time to formalize your AI risk mitigation approach. Start the discussion within your leadership team today. Identify where AI is already influencing decision-making, assess your current controls, and define the safeguards you need next. #RiskManagement #AI #Leaders

𝐌𝐨𝐬𝐭 𝐀𝐈 𝐟𝐚𝐢𝐥𝐮𝐫𝐞𝐬 𝐚𝐫𝐞 𝐧𝐨𝐭 𝐓𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥, 𝐓𝐡𝐞𝐲 𝐚𝐫𝐞 𝐀𝐈 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞 𝐅𝐚𝐢𝐥𝐮𝐫𝐞𝐬. Here are the 10 Principles that prevent costly Production Disasters: 𝟏.𝐏𝐑𝐎𝐌𝐏𝐓 𝐚𝐧𝐝 𝐌𝐎𝐃𝐄𝐋 𝐋𝐈𝐍𝐄𝐀𝐆𝐄 & 𝐕𝐄𝐑𝐒𝐈𝐎𝐍𝐈𝐍𝐆 • Version data, prompt code, and models (MLflow, DVC) • Track data sources and transformations • Support rollback and A/B testing 𝟐. 𝐂𝐋𝐄𝐀𝐑 𝐀𝐂𝐂𝐎𝐔𝐍𝐓𝐀𝐁𝐈𝐋𝐈𝐓𝐘 • Define RACI with escalation paths • Log decisions tied to model versions • Add approval gates at deploy and monitor stages 𝟑. 𝐑𝐄𝐀𝐋-𝐓𝐈𝐌𝐄 𝐎𝐁𝐒𝐄𝐑𝐕𝐀𝐁𝐈𝐋𝐈𝐓𝐘 • Monitor data, prediction, and concept drift • Set SLO alerts before performance drops • Maintain feedback loops with full lineage 𝟒. 𝐂𝐑𝐎𝐒𝐒-𝐅𝐔𝐍𝐂𝐓𝐈𝐎𝐍𝐀𝐋 𝐆𝐎𝐕𝐄𝐑𝐍𝐀𝐍𝐂𝐄 • Run regular AI risk and ethics reviews • Use NIST AI RMF for risk assessment • Gate high-risk models before launch 𝟓. 𝐅𝐀𝐈𝐑𝐍𝐄𝐒𝐒 & 𝐁𝐈𝐀𝐒 𝐓𝐄𝐒𝐓𝐈𝐍𝐆 • Audit fairness across protected groups • Monitor subgroup performance drift • Define acceptable parity metrics 𝟔. 𝐒𝐀𝐅𝐄 𝐅𝐀𝐈𝐋𝐔𝐑𝐄 𝐃𝐄𝐒𝐈𝐆𝐍 • Use circuit breakers and rule-based fallbacks • Enable fast rollout and rollback • Keep human workflows for edge cases 𝟕. 𝐒𝐋𝐎𝐬 𝐋𝐈𝐍𝐊𝐄𝐃 𝐓𝐎 𝐁𝐔𝐒𝐈𝐍𝐄𝐒𝐒 𝐊𝐏𝐈𝐬 • Define SLOs for latency, accuracy, and cost • Trigger alerts based on business impact • Automate fixes when limits are breached 𝟖. 𝐃𝐀𝐓𝐀 𝐆𝐎𝐕𝐄𝐑𝐍𝐀𝐍𝐂𝐄 & 𝐐𝐔𝐀𝐋𝐈𝐓𝐘 • Enforce data contracts and quality checks • Validate data before inference • Retrain when drift crosses limits 𝟗. 𝐄𝐗𝐏𝐋𝐀𝐈𝐍𝐀𝐁𝐈𝐋𝐈𝐓𝐘 & 𝐀𝐔𝐃𝐈𝐓𝐀𝐁𝐈𝐋𝐈𝐓𝐘 • Use SHAP/LIME for black-box models • Maintain model cards and datasheets • Log feature attributions for audits 𝟏𝟎. 𝐇𝐔𝐌𝐀𝐍-𝐈𝐍-𝐓𝐇𝐄-𝐋𝐎𝐎𝐏 • Provide ranked outputs with confidence • Capture human corrections • Define SLAs for manual review 𝐖𝐇𝐀𝐓 𝐓𝐄𝐀𝐌𝐒 𝐆𝐄𝐓 𝐖𝐑𝐎𝐍𝐆 They treat Governance as Paperwork, not Infrastructure. 𝐑𝐞𝐬𝐮𝐥𝐭: • Models deployed without lineage tracking • No Accountability when Failures occur • Drift goes undetected for Months • Bias discovered after impact • No rollback capability 𝐌𝐘 𝐑𝐄𝐂𝐎𝐌𝐌𝐄𝐍𝐃𝐀𝐓𝐈𝐎𝐍 Before production, verify: ✓ Lineage tracked? ✓ Accountability defined? ✓ Observability configured? ✓ Cross-functional review complete? ✓ Fairness tested? ✓ Failure modes designed? ✓ SLOs linked to KPIs? ✓ Data governance enforced? ✓ Explainability implemented? ✓ Human review defined? 𝐖𝐡𝐢𝐜𝐡 𝐩𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞 𝐚𝐫𝐞 𝐲𝐨𝐮 𝐦𝐢𝐬𝐬𝐢𝐧𝐠? ♻️ Repost this to help your network get started ➕ Follow Anurag(Anu) Karuparti for more PS: If you found this valuable, join my weekly newsletter where I document the real-world journey of AI transformation. ✉️ Free subscription: https://lnkd.in/exc4upeq #GenAI #AIAgents #AIGovernance

Reliable AI comes from calmer systems when things go wrong. Not from bigger models. Not from clever prompts. From architecture that expects failure and stays stable anyway. This is what reliable AI actually looks like in production: ‣ Fail-safe by design Assume the model will fail. Build graceful degradation, fallbacks, and safe defaults so users aren’t punished when AI misfires. ‣ Explicit error handling Validate inputs, catch failures, retry safely, and switch paths when needed. Silent failures are the fastest way to lose trust. ‣ Redundant execution paths Never bet critical workflows on a single model or service. Primary routes need backups, health checks, and traffic switches. ‣ Observability first Logs, metrics, traces, latency, and anomalies must be visible end to end. If you can’t see it, you can’t fix it. ‣ Continuous evaluation Production AI needs constant testing for accuracy, relevance, and safety. Shipping once is easy - staying correct is hard. ‣ Drift detection Data changes quietly. Behavior shifts slowly. Drift monitoring is how you catch decay before users do. ‣ Human-in-the-loop High-risk decisions need escalation paths. Automation earns autonomy only after trust is proven. ‣ Cost & performance controls Latency, tokens, caching, routing, and spend all need guardrails. Reliability without cost control doesn’t scale. ‣ Secure by default Treat AI like production software - permissions, validation, encryption, audit trails, and access controls included. ‣ Version everything Models, prompts, datasets, and pipelines must be versioned. Reliability depends on reproducibility and safe rollback. AI reliability is an architectural discipline, not a model upgrade. Most failures happen outside the model - in workflows, monitoring, and controls. If your AI feels impressive but fragile, don’t ask “Which model should we use?” Ask “Which of these principles are we missing in production?” Follow Vaibhav Aggarwal For More Such AI Insights!!