Planning Feedback Loops

Feedback loops are AI’s compound interest engine.. if you skip them and your AI performance will just erode over time. Too many roadmaps punt on serious evals because “models don’t hallucinate as much anymore” or “we’ll tighten it up later.” Be wary of those that say this, they really aren't serious practitioners. Here is the gold standard we run for production AI implementation at Bottega8: 1. Offline evals (CI gatekeeper): A lightweight suite of prompt unit tests, RAGAS faithfulness checks, latency, and cost thresholds runs on every PR. If anything regresses, the build fails. 2. RLHF, internal sandbox: A staging environment where we hammer the model with synthetic edge cases and adversarial red team probes. 3. RLHF, dogfood: Real users and real tasks. We expose a feedback widget that decomposes each output into groundedness, completeness, and tone so our labelers can triage in minutes. 4. RLHF, virtual assistants: Contract VAs replay the week’s top workflows nightly, score them with an LLM as judge, and surface drift long before customers notice. 5. Shadow traffic and A/B canaries: Ten percent of live queries route to the new model, and we ship only when conversion, CSAT, and error budgets clear the bar. The result is continuous quality and predictable budgets.. no one wants mystery spikes in spend nor surprise policy violations. If your AI pipeline does not fail fast in code review and learn faster in production, it is not an engineering practice, it is a gamble. There's enough eng industry best practice now with nearly three years of mainstream LLM/GenAI adoption. Happy building and let's build AI systems that audit themselves and compound insight daily.

HR doesn’t need more dashboards. It needs better listening. Most people teams measure what’s easy…like engagement scores or turnover. But the best teams? They build feedback loops that help them predict problems, not just react to them. This post gives you 11 of the most useful, often-overlooked loops you can implement across the employee lifecycle: 🟢 Week 2 new hire check-ins (capture early impressions) 🟠 Post-interview surveys (from both sides) 🔵 Onboarding reviews (day 90 is your goldmine) 🟡 Skip-level 1:1s (cross-level truth-telling) 🟣 Quarterly team health check-ins (lightweight, manager-led) …and 7 more. 📌 Save this if: • You’re building a modern HR function • You want fewer “We should’ve seen this coming” moments • You believe listening is strategy Which feedback loop is missing in your company?

🌀 User journey maps often capture “perfect” journeys users never take. We need to stop designing paths, and start designing loops, especially in AI products ↓ We use journey maps to capture, understand and refine user's experience. However, these maps are merely an idealistic view of what users SHOULD be doing, rather than what they actually ARE doing. Linear paths don't consider detours, circling back and forth, abandonments and returns and shortcuts. In fact, our interactions with reality rarely follow a well-defined, structured script; they’re a series of adjustments and feedback loops — depending on environment, disturbances, decision-making and actions. Workflows shouldn’t be perceived as a rigid cage, but as an orchestrated loop. Matt Fick and Max Peterschmidt suggest to rethink the idea of designing paths and design loops instead, especially with AI products in play. We start with a goal, make decisions, sense what’s going on, study environment, take action and then keep checking again, and again, and again. It follows a simple structure: 🎯 1. Setting a goal First, we establish a goal: what is the user trying to achieve? Desired outcome is the foundation on which the product will ground all its actions and adjustments. We must help people articulate their goal — with slow prompting and better calibration (knobs, pre-prompts, buttons, sliders). 🌡️ 2. Studying the current state (Sensors, Environment) To improve something, we must understand its current state. We find the right sources and collect the right inputs to get a snapshot of the current state. Often there are many meaningful inputs, and often they are very difficult to predict ahead of time. 🧠 3. Making decisions (Controller) Next, we evaluate the data and compare it against the goal. We come up with meaningful actions and get recommendations, grounded in trusted sources. Mapping the reasons for recommendations is critical for building trust and confidence — with AI, but not necessarily with LLMs. 🚀 4. Taking actions (Actuator) Once we decide that an adjustment is necessary, we take an action, or we ask agents to take an action — directly manipulating the environment closer to the desired outcome. The actions are typically initiated or approved by humans, and that’s what we mean with “human in the loop”. 🧲 5. Studying and refining the new state We gather data about the changed environment, and then use these inputs to suggest the next batch of changes as output. With nested loops, when many people or AI agents are involved, output in one loop becomes an input in another and informs next decisions and actions there. An interesting and realistic model in AI world, matching the complexities of the real world better than journey maps often do. Indeed, workflows aren’t rigid cages — they are non-linear, cyclic and must be highly adaptive to be meaningful. They must sense, respond and learn — and loops do just that.

Closing the Loop Between Planning and People Most planning starts with good intentions. Too much of it ends as a document the neighborhood never feels. We’ve all seen it: a glossy plan, a community meeting, a final report. Then the block stays the same. Sidewalk gaps. Vacant lots. “Coming soon” signs that never come. That’s the gap I keep coming back to. Not a gap in ideas. A gap in connection. Cities plan because they have to: growth, housing, infrastructure, climate risk. Communities show up because they care and because they know things no spreadsheet can capture. So why do we still end up with plans that don’t reach the people they’re supposed to serve? Because engagement gets treated like an event instead of a feedback loop. Implementation gets treated like “later” instead of the whole point. And planning stops at permission. Policy creates permission. Delivery creates belief. Here’s the question: What would change if we measured planning success by what residents can actually see, touch, and use? A few moves that close the loop: -Write a “Block Version” of the plan. Plain language: what’s changing, when, who owns the next step, and where the money comes from. If people can’t understand it, they can’t hold anyone accountable. -Put execution next to vision. Every major recommendation needs an owner, a timeline, a funding path, and a first 90-day action. This is how plans stop becoming shelf documents. -Build a standing feedback rhythm. Quarterly check-ins. Resident advisory groups with stipends. Public updates that track what got done and what didn’t. Trust doesn’t survive silence. -Fund the people work. Translation, childcare, stipends, door knocking, relationship-building. We budget for reports, then act surprised when the plan doesn’t land. Community trust is infrastructure too. -Deliver one proof project. A safer crossing. A small storefront rehab. A pop-up third place. A small-scale housing pilot. Something neighbors can point to and say, “That came from the plan.” Belief through delivery. This is also where r.plan fits. We help connect the dots between city planning, community vision, and real projects on the ground by pairing analysis with lived experience and strategy with implementation. Clear owners. Clear sequencing. Clear accountability. Not just what we build, but how we build. Your turn: Where have you seen planning lose the thread between the document and the block, and what’s one step your city could take this year to close that loop?

User Feedback Loops: the missing piece in AI success? AI is only as good as the data it learns from -- but what happens after deployment? Many businesses focus on building AI products but miss a critical step: ensuring their outputs continue to improve with real-world use. Without a structured feedback loop, AI risks stagnating, delivering outdated insights, or losing relevance quickly. Instead of treating AI as a one-and-done solution, companies need workflows that continuously refine and adapt based on actual usage. That means capturing how users interact with AI outputs, where it succeeds, and where it fails. At Human Managed, we’ve embedded real-time feedback loops into our products, allowing customers to rate and review AI-generated intelligence. Users can flag insights as: 🔘Irrelevant 🔘Inaccurate 🔘Not Useful 🔘Others Every input is fed back into our system to fine-tune recommendations, improve accuracy, and enhance relevance over time. This is more than a quality check -- it’s a competitive advantage. - for CEOs & Product Leaders: AI-powered services that evolve with user behavior create stickier, high-retention experiences. - for Data Leaders: Dynamic feedback loops ensure AI systems stay aligned with shifting business realities. - for Cybersecurity & Compliance Teams: User validation enhances AI-driven threat detection, reducing false positives and improving response accuracy. An AI model that never learns from its users is already outdated. The best AI isn’t just trained -- it continuously evolves.

Picture two nonprofit boards: 𝗕𝗼𝗮𝗿𝗱 𝗔 reviews reports, approves budgets, and ensures compliance. 𝗕𝗼𝗮𝗿𝗱 𝗕 does all that 𝘱𝘭𝘶𝘴 examines the thinking before and behind every major decision. Which one do you think is more impactful, long-term? Enter mental models. Investor Charlie Munger popularized the idea, urging leaders to “build a latticework of models” and see problems from multiple angles. Thinkers like Annie Duke and the Farnam Street community have shown how powerful this can be for anyone making high-stakes decisions in complex systems. And our nonprofit sector is definitely a complex system. If your model is “𝘔𝘰𝘳𝘦 𝘢𝘤𝘵𝘪𝘷𝘪𝘵𝘺 = 𝘮𝘰𝘳𝘦 𝘪𝘮𝘱𝘢𝘤𝘵,” you’ll measure success by busyness. If your model is “𝘍𝘰𝘤𝘶𝘴 𝘤𝘳𝘦𝘢𝘵𝘦𝘴 𝘳𝘦𝘴𝘶𝘭𝘵𝘴,” you’ll measure by alignment and progress. Yesterday, I mentioned reflection questions in Step 2 of a post I wrote on how nonprofit EDs can create conditions for better board meetings. (1st comment). Here's some to get you started: 𝗪𝗵𝗮𝘁 𝗰𝗵𝗮𝗻𝗴𝗲 𝗮𝗿𝗲 𝘄𝗲 𝗰𝗵𝗮𝘀𝗶𝗻𝗴 𝘁𝗵𝗮𝘁 𝘄𝗲 𝗺𝗮𝘆 𝗻𝗲𝘃𝗲𝗿 𝘀𝗲𝗲, 𝗯𝘂𝘁 𝗺𝘂𝘀𝘁 𝘀𝘁��𝗹𝗹 𝗯𝘂𝗶𝗹𝗱 𝘁𝗼𝘄𝗮𝗿𝗱? This is long-term, 𝗰𝗼𝗺𝗽𝗼𝘂𝗻𝗱𝗶𝗻𝗴 𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴. It forces perspective toward systems-level change. 𝗜𝗳 𝘄𝗲 𝗵𝗮𝗱 𝘁𝗼 𝗽𝗿𝗼𝘃𝗲 𝗼𝘂𝗿 𝗶𝗺𝗽𝗮𝗰𝘁 𝘁𝗼 𝗮 𝘀𝗸𝗲𝗽𝘁𝗶𝗰𝗮𝗹 𝗳𝘂𝗻𝗱𝗲𝗿 𝘂𝘀𝗶𝗻𝗴 𝗼𝗻𝗹𝘆 𝘁𝗵𝗿𝗲𝗲 𝗺𝗲𝘁𝗿𝗶𝗰𝘀, 𝘄𝗵𝗮𝘁 𝘄𝗼𝘂𝗹𝗱 𝘁𝗵𝗲𝘆 𝗯𝗲? This reflects the 𝟴𝟬/𝟮𝟬 𝗽𝗿𝗶𝗻𝗰𝗶𝗽𝗹𝗲. It focuses on what truly signals progress. 𝗜𝗺𝗮𝗴𝗶𝗻𝗲 𝗼𝘂𝗿 𝗯𝗶𝗴𝗴𝗲𝘀𝘁 𝗶𝗻𝗶𝘁𝗶𝗮𝘁𝗶𝘃𝗲 𝗳𝗮𝗶𝗹𝗲𝗱 𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲𝗹𝘆. 𝗪𝗵𝗮𝘁 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗻𝗴 𝗳𝗮𝗰𝘁𝗼𝗿𝘀 𝘄𝗼𝘂𝗹𝗱 𝗵𝗮𝘃𝗲 𝗹𝗲𝗱 𝘁𝗼 𝘁𝗵𝗮𝘁 𝗼𝘂𝘁𝗰𝗼𝗺𝗲? I𝗻𝘃𝗲𝗿𝘀𝗶𝗼𝗻, a pre-mortem helps prevent failure before it happens. 𝗪𝗵𝗮𝘁 𝗮𝗿𝗲 𝘆𝗼𝘂 (𝗯𝗼𝗮𝗿𝗱 𝗺𝗲𝗺𝗯𝗲𝗿) 𝗺𝗼𝘀𝘁 𝘄𝗼𝗿𝗿𝗶𝗲𝗱 𝗮𝗯𝗼𝘂𝘁 𝘁𝗵𝗮𝘁 𝘄𝗲’𝗿𝗲 𝗻𝗼𝘁 𝗹𝗼𝗼𝗸𝗶𝗻𝗴 𝗮𝘁? 𝗦𝗲𝗰𝗼𝗻𝗱-𝗼𝗿𝗱𝗲𝗿 𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴 surfaces hidden risks and unintended consequences. “𝗪𝗵𝗼 𝗮𝗿𝗲 𝘄𝗲 𝗻𝗼𝘁 𝘁𝗮𝗹𝗸𝗶𝗻𝗴 𝘁𝗼 𝘁𝗼 𝗯𝗲𝘁𝘁𝗲𝗿 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘁𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 𝘄𝗲 𝗵𝗲𝗹𝗽 𝘀𝗼𝗹𝘃𝗲?” 𝗠𝗮𝗽 𝘃𝘀. 𝘁𝗲𝗿𝗿𝗶𝘁𝗼𝗿𝘆 model- test assumptions and widen your lens. “𝗪𝗵𝗮𝘁 𝗱𝗼𝗲𝘀 𝗼𝘂𝗿 𝗰𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝘆 𝗹𝗼𝗼𝗸 𝗹𝗶𝗸𝗲 𝗶𝗳 𝘁𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 𝘄𝗲 𝘄𝗼𝗿𝗸 𝗼𝗻 𝗶𝘀 𝘀𝗼𝗹𝘃𝗲𝗱? 𝗜𝗳 𝘄𝗲 𝗹𝗼𝗼𝗸 𝗮𝘁 𝗼𝘂𝗿 𝘄𝗼𝗿𝗸, 𝗱𝗼𝗲𝘀 𝗶𝘁 𝗴𝗲𝘁 𝘂𝘀 𝘁𝗵𝗲𝗿𝗲? 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴. Aligns board discussions with the ultimate outcome, not the next milestone. Opportunity Cost: 𝗪𝗵𝗮𝘁 𝗮𝗿𝗲 𝘄𝗲 𝘀𝗮𝘆𝗶𝗻𝗴 𝗻𝗼 𝘁𝗼 𝗯𝘆 𝘀𝗮𝘆𝗶𝗻𝗴 𝘆𝗲𝘀 𝘁𝗼 𝘁𝗵𝗶𝘀? Every decision carries a tradeoff. Clarifying it keeps strategy honest. Feedback Loops: 𝗪𝗵𝗲𝗿𝗲 𝗮𝗿𝗲 𝘄𝗲 𝗴𝗲𝘁𝘁𝗶𝗻𝗴 𝘀𝗶𝗴𝗻𝗮𝗹𝘀 𝘁𝗵𝗮𝘁 𝗼𝘂𝗿 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝘆 𝗶𝘀—𝗼𝗿 𝗶𝘀𝗻’𝘁—𝘄𝗼𝗿𝗸𝗶𝗻𝗴? Builds learning into oversight. Try it and I bet you'll develop even stronger questions.

Years ago, when we shipped one of our first containers of shoes overseas, I thought we had everything figured out. Everything looked great on paper. Only after our partner received the container did the feedback not go so well. It’s easy for leaders to lean into dashboards and what I call EKG reports with lots of lines showing performance. But that alone isn’t essential. So are rapid feedback cycles for fast decision-to-action timelines. When our partner received the shipment, everything was right, with solid packaging and tight systems. Still, our partners told us that packaging wasn’t working due to the country’s humidity, and the unloading conditions were much harsher. I knew they wanted to continue to work with us, and they weren’t complaining. They were informing. I didn’t defend the system, I simply turned to our team and said since they’re the experts, so listen and adapt to our partner needs. Within a week, the team redesigned how shoes were sorted and packed, and soon it became the global standard for us. Execution doesn’t happen in a boardroom. It happens in real places, with real people who see what leaders miss. Here’s what I learned about a fast feedback loop: ✅ Listen early and often. Feedback loops can’t wait for scheduled meetings. Stay tuned in. ✅ Empower your team. When a challenge arises, allow your team to speak up and do the work. ✅ Adjust rapidly. A strong feedback loop allows you to get critical feedback. Use it to innovate and execute faster. Listening at all times. Feedback loops are essential—make sure you become a master. Always: listen, listen, listen. It’ll allow you to fix problems, adjust faster, and scale your business.

𝗚𝘂𝗮𝗿𝗱𝗿𝗮𝗶𝗹𝘀 aren't an afterthought or extra credit anymore - they're core architectural patterns that determine whether your agentic system is safe to deploy. So here are four different workflow patterns that we've seen implemented in production systems: 1️⃣ 𝗔𝗱𝗮𝗽𝘁𝗶𝘃𝗲 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸 𝗟𝗼𝗼𝗽𝘀 Worker agents execute tasks → Supervisor evaluates → Rewards Service updates policies → Guidelines adjust → Workers improve over time. This creates a continuous learning cycle where the system reinforces effective behaviors and discourages risky ones. It's reward-driven learning that improves with iteration. 2️⃣ 𝗖𝗼𝗿𝗿𝗲𝗰𝘁𝗶𝘃𝗲 𝗔𝗰𝘁𝗶𝗼𝗻 The centralized Supervisor assigns tasks, compares outputs against application guidelines, and if errors are detected, engages alternative workers. The best validated result gets returned. This prevents bad outputs from ever reaching users. 3️⃣ 𝗛𝘂𝗺𝗮𝗻 𝗶𝗻 𝘁𝗵𝗲 𝗟𝗼𝗼𝗽 For sensitive domains (medical diagnosis, legal review, financial approvals), agents generate preliminary responses but humans validate before execution. The workflow automatically pauses for expert review, then resumes once approved. 4️⃣ 𝗘𝗺𝗲𝗿𝗴𝗲𝗻𝗰𝘆 𝗦𝘁𝗼𝗽 Critical for high-risk environments like trading systems. Agent 1 collects market data → LLM processes signals → Agent 2 evaluates conditions → if anomalies or risks detected, execution halts immediately. Consider a trading bot with access to a volatility API showing VIX at 42 (extreme market stress). Even if the bot generates an aggressive trade recommendation, the evaluator independently verifies: "Given current volatility, does this make sense?" If not, it blocks the action entirely. 𝗕𝗲𝗵𝗮𝘃𝗶𝗼𝗿 𝗦𝗵𝗮𝗽𝗶𝗻𝗴 is the underlying philosophy here - a three-step loop of scoring, feedback, and correction. The evaluator doesn't just measure performance after the fact. It actively intervenes: triggering rollbacks for bad transactions, halting workflows propagating incorrect data, or routing edge cases to human reviewers. This is especially important when agents interact with volatile external states - market conditions, API health, system load. The evaluator provides a sanity check to ensure the model correctly interpreted the signals it was given, not just that it generated understandable text. The goal isn't catching every possible failure upfront (impossible). It's building systems that detect problems as they happen, understand what went wrong, and automatically correct course before damage propagates. Inspired by our most recent ebook we did with StackAI and Weaviate: https://lnkd.in/dKt9SVya

TOC Jedi Insights: On Planning Loops�� “When you need multiple iterations to stabilize a plan, you’re not managing reality, you’re chasing it.” Planning is meant to create clarity. One plan. One direction. One coordinated execution. Yet in many organizations, the first plan never survives contact with reality. So we revise it. Then revise the revision. Then adjust again. Each loop promises stability. Each loop reveals that the last one was detached from how the system actually behaves. What do we see? ▪️A first plan built on averages and optimistic assumptions. ▪️A second pass to “fix” capacity conflicts. ▪️A third to adjust for delays, shortages, or firefighting. ▪️A fourth to align with what really happened in execution. By the time the plan finally looks feasible, reality has already moved on. So the loop starts again. Multiple planning iterations are a sign of excessive sophistication. And excessive sophistication is a proxy for lack of clarity about reality. Multiple planning iterations are a symptom of a model that ignores variability, constraints, and real execution dynamics. Every new loop is an attempt to force certainty onto an uncertain system. Instead of absorbing reality, the plan resists it. Instead of stabilizing execution, it destabilizes priorities. And so we chase the plan instead of managing flow. True stability does not come from tighter calculations. It comes from designing the system to live with uncertainty: ▪️Planning around the constraint, not ignoring it. ▪️Using buffers to absorb inevitable variability. ▪️Executing with fast feedback, not long replanning cycles. When execution is synchronized to flow, planning does not need endless correction. The system becomes stable because it is responsive. 💡 The TOC Jedi knows: when plans need constant recalculation, the problem is not the math—it is the model of reality behind it. Plan with the constraint, protect with buffers, execute with flow. Flow is the force. May the flow be with you. #toc #goldratt #onebeat

Great strategy needs stars. But it only works when the whole team runs the system. This is Phil. Before he arrived, one part of the team dominated the rest of the team and the team had modest success. Then he instituted the triangle offense. It forced the sharing of the ball, putting the skillsets of the players around their best player in the best position to succeed, and integration over self-reliance (the one-on-one mentality) in order to win championships. Phil won 11 championships. When Finance, Ops, and RevOps aren’t truly part of the planning process, strategy becomes siloed, and execution gets political. People follow plans they help create. Here’s how you can get collaboration to show up in the planning cadence in practice: Weekly: Ground-Level Insights Each department logs weekly learnings - what’s working, what’s bottlenecked, what’s forecasted. These mini feedback loops feed the broader plan over time. Planning is no longer an annual fire drill. It’s iterative. Monthly: Rolling Planning Updates Monthly working sessions keep the plan alive. Pipeline changes. Delivery capacity shifts. CAC jumps or drops. Every department shares what’s changing in their world so the plan flexes with reality, not fantasy. Quarterly: Strategic Recalibration This is where leadership + department heads evaluate risk, investment areas, and team capacity. Finance brings cash modeling. RevOps brings revenue forecasts. Ops brings fulfillment feasibility. Everyone has a seat, and everyone speaks up. Annual: Joint Planning Workshops Budgeting. Hiring. Pricing strategy. Tooling. All on the table. But here’s the catch: planning doesn’t start in Finance. It starts cross-functionally. Each function informs the plan from their vantage point. No hidden agendas. Just shared direction. Strategic planning doesn’t live in a spreadsheet. It lives in the conversations you have before the spreadsheet is built.