Joseph Smarr

🙏 Please read the paper Gary Marcus cites. If you don’t want to read the paper, read his brief and compelling article. If you don’t want to read the article, the point is that we - the people - cannot outsource our thinking on the critical topic of generative AI’s impact on society. The false binaries we tend toward here are unhelpful. This is a deeply nuanced and seductive technology that has serious implications for our lives and our institutions. We must design and architect accordingly. This is not to say that our institutions are necessarily effective before generative AI (most organizations have loads of dysfunction) - but that the technology exacerbates the dysfunction and erodes trust. There are some remarkable and trustworthy technologies we can use to improve things - let’s use those. Let’s ensure that generative AI is in its proper place in the human loop of intelligence. I don’t think we know where that is yet, but we’re getting a very clear picture of where it is not.

16

Anthropic’s Claude desktop app is excellent for the non-coding enterprise users. Two minor features, and Claude desktop would become indispensable for my day to day. Claude has always led the pack in enterprise workflow features. IMO. They were the first to embrace Projects and Artifacts as first class objects. And they literally defined the standard in interoperability with thier Model Context Protocol (MCP). I create Projects for different parts of my job. So I have one for Product and Design document writing. For reference docs, this project contains templates, my favorite examples, summary background info on my company and my products. Often I’ll use Deep Research by Perplexity or ChatGPT (Gemini Deep Research thinks it’s a PhD student no matter the prompt I give it), and drop that output into Claude to draft documents. I have a Project for Tickets. We use both HubSpot and Jira, depending on the department. Using MCP, Claude has access to both of these ticket systems. I’ve put reference info in the project that defines the different ticket systems, ticket pipelines, and my particular name for stuff. (Example, what I call the “Support Pipeline” is pipeline id 0 and has a different labeled name in HubSpot. But Claude now knows what I intend when I say Support Pipeline.) So with templates and instructions, I can ask for any overdue tickets with summary explanation … and if I need to get involved. My two feature requests … 1. Let me specify individual MCPs by Project. I have lots of integrations for Claude, and it can get confused sometimes. If it was specialised by Project, would be way easier to manage. 2. Let me edit Artifacts in the app. Yes I can ask the LLM for changes, yes I can download or push to Google docs and so on. But often I want to make quick edits, delete a section, and keep interating with the model or then push directly to someone. I currently have to copy and paste or down, edit, upload. And I want projects to be my system of record. 3. Bonus feature request … leave project-knowledge files in format. I get that you are stripping them for unneeded tokens, but you could do that on the fly so I can more easily see (or even edit! Would that be bonus feature request number 2?) what info I’ve given to drive the project. Yes, I know pdfs are handled differently. #AIPRODUCTMANAGEMENT

9

11 Comments

I've been onboarding non-software folks onto Claude Code to work on large document sets (not code). This is a short field note on what's working, what's painful, and how I think version control + agents will be a big deal. https://lnkd.in/gkNuRD5E

20

6 Comments

The retro hipsters have found their way to LLM model development, and they may have landed on a legitimate use case for chatbots. https://lnkd.in/gvbNVZbd

2

Running evals on the new Claude 4.5 Haiku this morning—seemed like a good time to share what we’ve built for evals at Boom. The core implementation: an automated evaluation pipeline that runs every supported model against an ever growing set of test cases for our critical path inference operations. Quality, latency, and cost benchmarked in real time against production workloads. But here’s what this actually delivers for our customers: **Guaranteed uptime, not just promised uptime.** When Anthropic or Google has an outage, our system automatically routes to the next best model. Our customers’ workflows don’t stop. They often don’t even notice. We’ve decoupled their reliability from any single provider’s reliability. **Immediate access to model improvements.** Claude 4.5 Haiku dropped last week. By this morning when I finally got a few minutes to add it to our model directory, I had complete performance data across our stack. If it improves customer outcomes, we can integrate it today. Our customers get the benefit of frontier model advances within hours, not months. **Protection from silent degradation.** The harder problem isn’t when services go down—it’s when they stay up but quality degrades. Our eval system monitors this continuously. If we detect accuracy dropping on a live service, we route traffic away before it impacts customer results. The chart shows what I’m looking at right now: Haiku absolutely crushing latency on most of our critical path operations (e.g. 11s vs 26-64s for other models) while maintaining 99%+ accuracy. That’s a customer experience improvement we can ship immediately. The power is in having evals that actually matter. Test cases from realistic production scenarios. Metrics that map to customer outcomes. When you can run any model against your ground truth in minutes, you stop guessing and start knowing. Model selection becomes an empirical question, not an intuitive or philosophical one. Your customers care about outcomes, not which model you’re using.

27

2 Comments

This is the series all agentic AI companies and their conversational designers, agent architects, prompt engineers or whatever you call yourselves should be reading. Great team at Parloa with great views from Agentic deployments that actually deliver. Amazing work on the article Sybille. Also huge shoutout to Yen-ting Kuo and Dr. Rangina Ahmad for getting stuff done!

15

3 Comments

Open Responses is an open-source spec based off OpenAI's Responses API, with backing so far from OpenRouter, Vercel, HuggingFace, LM Studio, Ollama, and vLLM. https://openresponses.org Will more providers get on board? It'd be so nice to have a true standard for LLM APIs.

69

5 Comments

Everyone loves a super-scalable architecture. It looks great, sounds smart, until you realize how much it actually costs to build and operate one. Think about an F1 car - I'll bet you want to drive one. Owning one? Expensive, fragile, and wildly impractical. Same goes for your over-engineered system. Build what fits the road, not the racetrack.

8

Hawaiʻi folks, we do not need to use generative AI to tell our stories. We need artists. We do not need to use art stolen from other artists. In Hawaiʻi, we know that to have abundance, we have to cultivate the resources that provide for us. Generative AI steals art while directing energy and resources away from building new artistic skills and supporting existing artists. Exploiting a resource until itʻs sucked dry, then leaving us to deal with the consequences? This should be a familiar story here. And we don't have to do this. We don't have to be a marketing arm for the best funded companies in the world. This is not inevitable. We simply don't have to do it. Creating original art matters. Weʻre all hyped on Chief of War right now because it features actual Polynesian people, actual handmade art pieces, actual Hawaiian language actually learned or natively spoken by the cast. AI "art" doesnʻt build those relationships. It doesn't teach new skills (except for how to use a few proprietary tools). It doesn't regenerate. Okay, so you don't have access to Apple TV level funding. The phone in your pocket has a camera and editing software on it. Go out into the real world. Interact with real people. They are messy and difficult, but that's the whole point of any of this. Literally touch real grass. Observe it. Be surprised by what you find. Take pictures of things you want to see documented. Don't have access to those things? Create them. Build relationships with people, learn new skills. Demand a world where you can see your dreams in the real world. And in the meantime, actually research and learn about the world, so you can find a way to depict it. Draw those things with your own hands and struggle to rework the lines until they say what you want them to. Film your world. Make editing decisions. Think about what you've seen and choose what's important for you to show. Grapple with the messiness and unexpectedness of all of it. Use that to grow. Actively seek out the work of other artists. Be inspired, intimidated, confused. Study it until you can see what works and what doesn’t. Choose who you want to be able to cite as your influences. Let that all sit in your brain together until it seeps into everything else you know and you create something new. Will it be difficult, chaotic, probably a little janky? Will it take a lot longer? Yeah. But you'll get better at it. And I promise that what you create will be more valuable to the world than anything generated by a machine that doesn't know or care about us, that depletes our resources, and whose primary stated goal is to replace us.

133

5 Comments

I spoke at the Portland AI Engineers meetup last week about a question that keeps many AI engineers up at night: how do you actually know if your AI works at scale? Last year, Cursor's support bot hallucinated a device policy that didn't exist. Multiple users asked about logging in on different devices. The bot confidently claimed they had to log out first to switch devices or pay for a second subscription. That policy never existed. The bot hallucinated it. Some users believed the made-up policy. Others argued it was wrong. Chaos ensued and users started canceling their subscriptions. The CEO had to intervene publicly. The haunting question: how many users got bad answers and silently switched to the competitor? Here's why this keeps happening. Traditional testing can't catch it. You can't write `assert chatbot_response == "correct answer"` when the output is different every time. Unit tests assume determinism. LLMs are stochastic. Generic LLM-as-judge evaluations don't catch it either. A generic eval asks "does this answer the user's question?" The Cursor bot did answer the question. It just invented the answer. What you actually need to check is "is this factually consistent with our latest policies?" That gap between generic and contextual is where AI products break. The solution is encoding domain expertise through labeled examples. Not "does this avoid hallucinations" but "does this match our actual support documentation?" 20 examples showing the judge what correct looks like in your specific context. The attached video shows how to build contextual evals in Truesight. Same LLM-as-judge approach you'd use for generic checks, but grounded in labeled examples from your actual domain. This is the technique that would catch hallucinations like Cursor's before they reach users. Once you have this built, you integrate it everywhere. Development, pre-deployment regression checks, production monitoring, real-time guardrails. The eval becomes the continuous quality signal that keeps you out of the reactive loop. Full talk with the Cursor breakdown and all demos: https://lnkd.in/gwRBtpA8 If your evals aren't catching what actually breaks in production, try Truesight: https://lnkd.in/gfbpP8dA What evaluation challenges are you hitting as you move AI to production?

16

7 Comments

It was a pleasure to chat with Rakesh Agrawal over at PCMag on his recent article that dug into how Waymo is changing late-night rides, especially for women. A few key takeaways: 🛡️ Many women prefer Waymo at night because there’s no driver involved, which reduces the risk of uncomfortable or unsafe interactions. 💸 That added sense of safety often comes at a premium. In one example, Uber and Lyft rides were under $20, while Waymo was over $50 for the same trip. 🚗 Waymo has a fixed fleet. Unlike Uber and Lyft, it can’t add more cars when demand spikes, so prices rise or rides disappear. 🎀 The result is an indirect “pink tax.” Men are more likely to choose cheaper human-driven rides, while women are more likely to pay extra for safety. ⚖️ Ride-hailing platforms are trying to close the gap with features like women-preferred driver matching, but smaller pools can also mean higher prices. ❓ The big question: should safety come at a premium, or should companies like Waymo rethink pricing so safer options aren’t effectively a luxury?

10

5 Comments

The AI conversation is still stuck on tools. But the real shift is happening at the skill layer. A new category of skills is quietly emerging: Agent Skills - building AI agents that execute workflows Claude Skills - reusable capabilities that let AI complete complex tasks System Skills - designing how humans + AI collaborate We are moving from: Human skills TO AI-deployable skills. The real leverage will come from building skills that AI can execute repeatedly. #AIAgents #AgentSkills #ClaudeAI #FutureOfWork #AILeadership #AIStrategy

1

Floof Farm might not have a robots.txt file or LLMs.txt or anything else of that nature. Importantly, it does have a cats.txt https://lnkd.in/gtCNu7Ri

20

3 Comments

TL;DR - Here's my Claude skill for having virtual design partners: https://lnkd.in/g7nvpAX6 One of the things I miss the most about being in a studio environment is working with amazing and smart people like other designers, artists, and engineers. There is no substitute for the energy and amplification you get in that environment. But I have found with the right direction and guardrails that AI LLM chatbots can be surprisingly effective design partners. I liken it to playing tennis against a backboard or a ball machine; it's not the same as a real partner, but it forces me to move and think and react, which in turn propels my thinking. These tools have become a force multiplier for me, especially as more and more of my design work is effectively solo. For the past two years, I have been slowly building a set of cloud skills to emulate that design studio environment, and I recently pulled them all together in a single comprehensive installable Claude skill: https://lnkd.in/g7nvpAX6 One of the things I have found so delightful is the ability to invoke a "teammate" - the artist, the 'disagree but commit' engineer, the business-minded C-suite, the design elder / creative director... Many of these are based on people I've worked with, and it is so fun to imagine them in the room with me. I also like being able to tell the agent that we are in flair (generative, no judgement) or focus (decision making, judgement) mode - that was a huge part of how I've always worked with other designers (and a reason I think most non-design meetings are ultimately unsatisfying). The skill understands design methods for user research, synthesis, brainstorming, and prototyping. You can give it a Whisper transcript of user interviews or even have it help you plan an interview and then jump into synthesis across different research artifacts, for instance. I've also been using a skill I created to make Claude go play. "Rigorous play" is a creative act that was so integral to studios I've been a part of. It is the idea that when we do something silly and creative together, we build psychological safety and unlock new ideas. My Claude play skill makes the agent go learn something random and then 'make' something (a poem, a joke, an improv back and forth) based on what it learned. Then it tries to make a connection between that creative act and the current project I'm working on. Try it out! https://lnkd.in/gqMUpJvN If you try either of these, I'd love to hear some feedback!

11

Ontologies remain one of the most effective ways to ground AI memory, keeping extracted knowledge anchored to existing structures and adaptable through feedback. As memory systems evolve, their role remains essential, and a core focus at Cognee.

47

4 Comments

What may seem as a whimsical experiment by Anthropic in reality provides some intel into the state of agentic AI: - AI is naive and easily fooled, don't be either - Use multi-agent systems for checks and balances - Don't trust AI with business or monetary decisions yet! Well worth a watch. Wonderful project, Anthropic #ai #agenticai

2

Avoiding Claude Code because the terminal feels intimidating? Cowork changes that. Same power, zero command line. Anthropic released Cowork today and I've been testing it. Here's what makes it different: Cowork lets Claude interact with your computer through a visual interface. No typing commands. No scary black screens. What you can actually do with it: - Organize messy file folders automatically - Turn a pile of receipts into a clean spreadsheet - Draft reports by pulling data from multiple documents - Rename hundreds of files based on patterns - Create presentations from raw research notes This is Claude Code's capability through a visual interface. The same parallel processing that developers use, but accessible to non-technical users through Claude desktop. Worth exploring if you spend hours on repetitive computer work.

3