Jeff Savin

See the fastest LLM serving engine, 100% open source!!! If you’re running multi-round QA, RAG, or any long-context workload, you know the pain: slow time-to-first-token and expensive bills. You should take a look LMCache, a next-gen LLM serving engine extension designed to slash TTFT (time to first token) by 3–10x and boost throughput. How? LMCache stores KV caches from any reusable text (not just prefixes) across GPU, CPU DRAM, and disk, then shares them across serving instances. That means fewer recomputations, faster responses, and huge GPU savings. Key highlights: 🔥 Seamless integration with vLLM v1 ⚡ High-performance CPU KV offloading 🔗 P2P KVCache sharing across servers 💾 Storage on CPU, Disk, or NIXL ✅ Supported in vLLM production stack, llm-d, and KServe 📦 One-liner install: pip install lmcache If you’re scaling LLM inference, you should check it! 👉 GitHub link in the comments. #LLM #AI #Inference #OpenSource #RAG #AgenticAI

378

27 Comments

I don’t think there’s ever been a more understandable time to feel exhausted by change in this field. I’ve seen a few posts lately from programmers saying they feel behind. Not behind on one tool or one framework, but behind in a broader, harder-to-name way. New layers keep showing up. Agents, prompts, memory, workflows, integrations. It can start to feel like there’s some configuration you’re supposed to have figured out by now. I saw Andrej Karpathy describe this recently, and I get what he’s pointing at. The thing I’d push back on slightly is the idea that there is a clear version of “keeping up” happening somewhere else. As far as I can tell, nobody really has this mapped yet. Even the people closest to the models are still discovering what works, what breaks, and what turns out not to matter as much as it first seemed. A lot of this is being figured out in real time. That changes how I hear the whole “falling behind” thing. It doesn’t feel like lagging a stable baseline. It feels like standing in the middle of something that’s still forming. I didn’t start using every new tool the moment it appeared. I wanted to see how it played out. Some people jump in early and help uncover the shape of things. Others wait until there’s a bit more signal than noise. That split isn’t new, and it doesn’t say much about who’s ahead. Most of this still feels unsettled. I suspect it will for a while.

154

58 Comments

Question: What is the most important shift in software development since the cloud? It’s not Copilot. It’s not Microsofts Clippy. Its not when AOL sent software updates by CD in the mail. You guessed it… It’s Agent-Native Development and lucky for us our company Factory is leading the charge. Today, Factory announced a $50M Series B led by New Enterprise Associates (NEA), with participation from Sequoia Capital, NVIDIA, JPMorganChase, and some absolute legends like Frank Slootman, Nikesh Arora, and Aaron Levie Droids are here and they’re not just code monkeys (whatever those are, just read this on X once). These agents smashed the #1 spot on Terminal Bench, beating out everything from Claude Code to Cursor. And they don’t just write code, Factory offers a few things which seem very differentiated, notably their scope includes all the other areas of the development lifecycle (collaboration, documentation etc. ) which I truly believe give them a meaningful edge. But what really sets Factory apart? → LLM-agnostic → IDE-agnostic → Interface-agnostic → Context-aware across the org Slack, GitHub, Jira, Sentry, Linear, Datadog. The Droids plug into everything. You can delegate from Terminal, IDE, Browser, or headless automations. No friction, no retraining. And yes if you’re wondering, it took me months to wrap my head around this all… BUT This is what the next era looks like: Not autocomplete. Not isolated agents. Orchestrated, programmable co-pilots that evolve with your stack. Teams at Nvidia, MongoDB, Bayer, Zapier, Clari, and EY are already using Factory to 30x feature delivery, cut migrations by 96%, and reduce on-call pain by nearly 96%. That’s the real unlock when agents are actually usable in production. Shout out to Matan Grinberg, Natan, and the whole Factory team. What you’ve built isn’t just futuristic. It’s practical, it’s scalable, and it’s already changing the way great teams build. Since you guys stole our Ai investor Francesca LaBianca a year ago, I can finally let go of the pain, as you are all kicking butt! We’re proud to be in the trenches with you at Mantis. Let’s f*cking go. @Factory.ai Mantis Venture Capital @NEA @Sequoia @NVIDIA @JPMorgan @FrankSlootman @NikeshArora @AaronLevie

164

22 Comments

“Developers are a resource. They don’t talk to product people.” I overheard that on Seattle’s light rail, and it stuck with me. I make it a point to talk directly with the people who use the software — delivery drivers, paralegals, and others doing the real work. Every conversation saves me hours of coding the wrong thing. When developers are treated as “resources,” they’re cut off from the feedback that makes their work valuable. Talking to users isn’t a luxury — it’s how you build the right thing faster. You don’t need a big product team or AI tools for that. Just curiosity and a conversation.

247

42 Comments

Screw vibecoding. You know what LLM coding agents are *really* useful for? Reverse-engineering legacy code! "Hey can you research and explain how the flux capacitor settings are loaded, specifically where are the margadon-level overrides applied exactly?" It's a lifesaver.

389

101 Comments

If you don't have "taste", then your LLM usage will generally output slop. I have a pretty good taste for code, so I'm able to quickly evaluate LLM generated code to make sure the parts I care about aren't slop. I have a decent taste for writing, so I write most things myself and just use LLMs for editing and critique. I'm OK at evaluating design, but no where near as experienced as I am with code. So I'm actively leaning into building my taste in that area. What other areas are important to develop taste in as a product builder? Business acumen taste? Architecture taste? Any tips for developing taste so you can turn slop into something tasty? #slop

13

2 Comments

If you're looking for a new docs framework for your dev tool, I can't recommend Docusaurus enough. 1. You can change the appearance of any component—no matter how big or small—by ejecting its raw React code into your source tree. I've ejected almost all of its major components and restyled them with Tailwind. 2. It supports content written with MDX, which is like Markdown but better. MDX lets you write JSX tags so you can render React components (like badges and banners) directly inline your content. No more messy templating. 3. It bundles Prism for code snippet highlighting. So you if you want to change how each and every token of your language gets styled—which I do—you can override it with a simple centralized map—which I have. 4. You can swap out its on-site search engine with Algolia, now widely considered the best option for developer sites. It is, in other words, the perfect tool for people like me who are obsessed with every pixel.

118

16 Comments

"Now that code is free..." Any assertion that follows this is predicated on a falsehood. Plausible-looking code might now be cheap. But *working*, reliable code that can be changed at low cost without breaking it? For most teams, that's even more expensive now.

38

11 Comments

People keep calling Clawdbot / Moltbot an overnight success. That framing misses the point completely. I didn’t look at the launch tweets. I looked at the builder’s GitHub. What stood out immediately wasn’t Moltbot it was the years of groundwork behind it. Before Moltbot ever existed, Peter Steinberger had already built and shipped dozens of tools focused on one thing: 👉 making AI agents actually usable in the real world. Not flashy demos. Not prompt tricks. Infrastructure. Things like: token accounting MCP servers browser + terminal automation CLI tooling agent rules and guardrails file and system orchestration Each project solved a narrow, unglamorous problem. Together, they formed a system. So when Moltbot appeared, it didn’t come from nowhere. It clicked into place. Why Moltbot caught fire Because it crossed a line most “AI assistants” don’t. It doesn’t just respond. It operates. • Runs continuously on your own machine • Integrates where work already happens (mail, chat, calendar) • Executes tasks without being nudged every time • Feels closer to a teammate than a chatbot That’s why developers went out and bought dedicated machines just to run it. That’s why it exploded on GitHub in weeks. But let’s be honest about the downsides This is not consumer software. Misconfigured instances leaked credentials. Prompt injection is a real attack vector. You need to understand systems, permissions, and isolation. This is early-stage, builder-grade tooling. Powerful but sharp. The takeaway isn’t “build Moltbot” The real takeaway is simpler: Breakthrough products are rarely breakthroughs. They’re compilations. Years of shipping small, useful things. Learning where systems break. Building boring parts no one tweets about. Then one day, the pieces finally line up. Everyone sees the moment. Almost no one sees the mileage. That’s how it usually works. 🔖 Save this if you’re building AI agents ♻️ Repost if “overnight success” is a myth you’re done believing

94

42 Comments

NVIDIA just made one of their smartest acquisitions and almost nobody noticed. SchedMD. 40 people in Lehi, Utah. Undisclosed price. They maintain Slurm, the workload manager running on 65% of TOP500 supercomputers. More than half of the top 10. More than half of the top 100. Every time a researcher submits a training job, every time an ML engineer queues batch inference, every time a national lab allocates compute for a simulation - there's a decent chance Slurm is deciding which GPUs actually run it. Combined with the Run:ai acquisition earlier this year ($700M for Kubernetes-based GPU orchestration), NVIDIA now owns the scheduling layer for both traditional HPC AND cloud-native AI workloads. Two doors into GPU computing. NVIDIA controls traffic on both. Everyone's focused on escaping CUDA. AMD's ROCm. Google's TPUs. Intel's Gaudi. OpenAI's Triton. All valid competitive threats. But programming models can be abstracted. Compilers translate. PyTorch already runs on AMD GPUs. Orchestration is stickier. Your Slurm configs encode years of institutional tuning. Your batch scripts define workflows that took months to get right. Your accounting systems tie GPU usage back to grants and budgets. Your users know the commands without having to think. Your cluster topology is optimized for Slurm's scheduling algorithms. Switching programming models is a recompile. Switching schedulers is a reorganization. NVIDIA looked one layer up from the chip wars and realized: whoever owns the orchestration layer owns the decision about which chips run which workloads. They bought the bouncer, not the dance floor.

339

37 Comments

If you're at #AutoCon4, you'll have an opportunity to chat with the folks at Forward Networks, Inc.--and you should. If you don't know Forward Networks, they are the original digital twin. I don't mean "we stand up some images in ContainerLab and try to replicate your network" sort of digital twin. I mean they model your network mathematically and bring your actual network state into that model as often as you need to facilitate network operations. From an automation perspective, run your proposed change against the Forward model to make sure that you aren’t breaking prod, and do it from two perspectives. 1️⃣ Use Forward not only for "Will this work?" questions, but also... 2️⃣ "Did this work?" questions--both pre- and post-change. Really useful for your network automation platform. There is of course a lovely Forward Networks GUI, but for the network automation folks walking the halls of AutoCon, you'll be more interested in the API. If you swing by the booth, they can show off what they can do, and answer your technical questions.

65

3 Comments

Something I said in a thread elsewhere, which bears repeating here: I think the reason the C programming language excited people back around 1980-85 is that, for the first time, we had a programming language that really did replace assembler -- it gave you an efficient way to do basically anything the computer could do. Previous programming languages, especially FORTRAN and BASIC, didn't give you the full capabilities of the computer directly. They were designed to put a layer of separation between the programmer and the CPU, the way R, for instance, does today. FORTRAN in particular had a strong emphasis on portability. BASIC had a strong emphasis on the things a beginner would want to do. C made it easy to program down to the bare metal, so to speak.

43

16 Comments

Tired of hearing excuses from your dev team? (Here’s what high-performing teams do differently.) Every time you ask why a project isn’t done, there’s a new story: ⤷ “We ran into unexpected challenges.” ⤷ “The requirements changed.” ⤷ “It turned out bigger than we thought.” It’s like playing whack-a-mole with your own roadmap. But here’s the truth: Good dev teams don’t hide behind blockers. They build with boundaries. What the Best Teams Do: 1/ Set unmovable deadlines → Not vague estimates → Real dates that drive accountability 2/ Define scope clearly → What’s in? What’s not? Put it in writing → Scope creep dies in documentation 3/ Protect momentum → No shifting goalposts → No sprinting in circles ⤷ Why It Matters: A dev team that avoids hard lines doesn’t just delay launches. They build a wall between you and your goals. If that’s what you’re up against, it’s not a people problem, is a standards one. Ready to work with a team that delivers? Start here: https://lnkd.in/gBr3wFRC 👉 Follow for more no-BS advice on building faster, accountable dev teams.

33

29 Comments