Burke Holland

Cool recent paper from our team (Instagram Ads Ranking) on customizing the learning process of attention layers to preserve mutual information with minority cohorts while improving global performance. This could be quite relevant to other industry groups working on large scale recommendation systems. https://lnkd.in/gyfVWbS5

115

1 Comment

For anyone curious about the third thing I wanted to try implementing in Sguaba, I ended up writing it myself, and then having Claude suggest the tests. Worked quite well! Given the nuances involved in designing this mechanism (and I discovered more nuance as I went), I suspect it would have struggled quite a lot with implementing it from scratch 😅 https://lnkd.in/dSRm6Kxm

71

3 Comments

→ 𝐓𝐡𝐞 𝐇𝐢𝐝𝐝𝐞𝐧 𝐏𝐨𝐰𝐞𝐫 𝐨𝐟 𝐏𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠: 𝐀𝐫𝐞 𝐘𝐨𝐮 𝐔𝐬𝐢𝐧𝐠 𝐈𝐭 𝐑𝐢𝐠𝐡𝐭? Prompt engineering remains one of the most effective AI alignment strategies. It lets developers steer large language model behavior without changing model weights. This means fast, low-cost iteration and control through natural language. But there’s a catch - prompts can be fragile. Changing just one word can lead to very different results. Plus, long, complex reasoning can be hard to scale with simply prompting. Different tasks call for different prompting methods. Knowing when and how to use each can make or break your AI project - especially if you care about budget and outcome. 𝐋𝐞𝐭’𝐬 𝐝𝐢𝐯𝐞 𝐢𝐧𝐭𝐨 𝐭𝐡𝐞 𝐭𝐨𝐩 8 𝐩𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 𝐬𝐭𝐫𝐚𝐭𝐞𝐠𝐢𝐞𝐬 𝐲𝐨𝐮 𝐦𝐮𝐬𝐭 𝐤𝐧𝐨𝐰: • 𝐂𝐡𝐚𝐢𝐧 𝐨𝐟 𝐓𝐡𝐨𝐮𝐠𝐡𝐭 (𝐂𝐨𝐓) Teach your AI to break problems into logical steps for clearer, more accurate answers. • 𝐑𝐞𝐀𝐜𝐭 (𝐑𝐞𝐚𝐬𝐨𝐧 + 𝐀𝐜𝐭) Make the AI alternate between thinking and doing - reason, act, evaluate, and adjust dynamically. • 𝐓𝐫𝐞𝐞 𝐨𝐟 𝐓𝐡𝐨𝐮𝐠𝐡𝐭 (𝐓𝐨𝐓) Explore multiple reasoning paths before choosing the best solution - great for complex, ambiguous problems. • 𝐃𝐢𝐯𝐢𝐝𝐞 𝐚𝐧𝐝 𝐂𝐨𝐧𝐪𝐮𝐞𝐫 (𝐃𝐧𝐂) Break big problems into smaller parts to solve them in parallel, then combine for a final result. • 𝐒𝐞𝐥𝐟-𝐂𝐨𝐧𝐬𝐢𝐬𝐭𝐞𝐧𝐜𝐲 𝐏𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 Have the AI answer multiple times, then select the most consistent response for reliability. • 𝐑𝐨𝐥𝐞 𝐏𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 Assign a persona (lawyer, doctor, etc.) to shape tone, knowledge, and context of replies. • 𝐅𝐞𝐰-𝐒𝐡𝐨𝐭 𝐏𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 Show a few strong examples. The AI learns the pattern and replicates it, ideal for structured tasks. • 𝐙𝐞𝐫𝐨-𝐒𝐡𝐨𝐭 𝐂𝐡𝐚𝐢𝐧 𝐨𝐟 𝐓𝐡𝐨𝐮𝐠𝐡𝐭 Prompt the AI to “think step-by-step” without examples - perfect for spontaneous reasoning. Prompt engineering is an art and science. Master these methods to harness LLMs with precision and confidence. 🌟 Follow the AIKaDoctor (Free AI & Data Science Resources)  channel on WhatsApp: https://lnkd.in/dCTCEKKc 📌Follow Habib Shaikh For more such content.

129

20 Comments

Happy Friday! This week in #learnwithmz, I’m building on my recent post about running LLMs/SLMs locally: https://lnkd.in/gpz3kXhD Since sharing that, the landscape has rapidly evolved, local LLM tooling is more capable and deployment-ready than ever. In fact, at a conference last week, I was asked twice about private model hosting. Clearly, the demand is real. So let's dive deeper into the frameworks making local inference faster, easier, and more scalable. Ollama (Most User-Friendly) Run models like llama3, phi-3, and deepseek with one command. https://ollama.com/ llama.cpp (Lightweight & C++-based) Fast inference engine for quantized models. https://lnkd.in/ghxrSnY3 MLC LLM (Cross-Platform Compiler Stack) Runs LLMs on iOS, Android, and Web via TVM. https://mlc.ai/mlc-llm/ ONNX Runtime (Enterprise-Ready) Cross-platform, hardware-accelerated inference from Microsoft. https://onnxruntime.ai/ LocalAI (OpenAI API-Compatible Local Inference) Self-hosted server with model conversion, whisper integration, and multi-backend support. https://lnkd.in/gi4N8v5H LM Studio (Best UI for Desktop) A polished desktop interface to chat with local models. https://lmstudio.ai/ Qualcomm AI Hub (For Snapdragon-powered Devices) Deploy LLMs optimized for mobile and edge hardware. https://lnkd.in/geDVwRb7 LiteRT (short for Lite Runtime), formerly known as TensorFlow Lite Still solid for embedded and mobile deployments. https://lnkd.in/g2QGSt9H CoreML (Apple) Optimized for deploying LLMs on Apple devices using Apple Silicon + Neural Engine. https://lnkd.in/gBvkj_CP MediaPipe (Google) Optimized for LLM inference on Android devices. https://lnkd.in/gZJzTcrq Nexa AI SDK (Nexa AI) Cross-platform SDK for integrating LLMs directly into mobile apps. https://lnkd.in/gaVwv7-5 Why Local LLMs Matter? - Edge AI and privacy-first features are rising - Cost, latency, and sovereignty concerns are real - Mobile + Desktop + Web apps need on-device capabilities - Developers + PMs: This is your edge. Building products with LLMs doesn't always need the cloud. Start testing local-first workflows. What stack are you using or exploring? #AI #LLMs #EdgeAI #OnDeviceAI #AIInfra #ProductManagement #Privacy #AItools #learnwithmz

66

2 Comments

Large language models and agentic tooling are rewriting the rules of engineering leadership. I’ll explore how the executive playbook must evolve to lead effectively in this new landscape. I’m excited to be speaking at ELC Annual 2025 on "How the Engineering Executive Playbook Must Change When Code is No Longer the Source of Truth" https://lnkd.in/gcd-x-Zy #ELCAnnual2025 #EngineeringLeadership #AI

81

10 Comments

I keep hearing that folks are running out of tokens with Claude Code. I use the $20 tier and while I get close to the limit at the five-hour point, I usually don't run out. And if I do, I just take a short break and wait for them to reset. I suspect people are not using Claude Code optimally. The trick is to deliberately and consciously manage the context Claude operates on. Take a look at my flow to see what I mean: 🆕 Start fresh: This means a blank slate. No context. Claude knows nothing about your project. ⬆️ Load essentials: Ask it to read CLAUDE.md to get up to speed on the project. This loads information about your codebase into the context without having to load all the code. 🔄 Stay focused: Work with Claude on a specific task. Tell Claude to look at code that is relevant to that task. Tell it where that code is at. Don't deviate from that task. This keeps Claude from loading irrelevant code into the context. ⬇️ Summarize the changes: When you're done with that task, ask Claude to update CLAUDE.md with what changed so that future versions have an accurate, and more importantly summarized, view of the software. ⏮️ Commit, clear, and repeat: Commit all these changes to Git, clear the context with the /clear command, and start on the next task. This flow results in the minimum viable context for the task at hand. Keeping the task focused results in shorter conversations and thus smaller contexts. Small contexts have fewer tokens. And fewer tokens means you don't run out. But, it's not just about the tokens. If the context is loaded with a variety of things, Claude can get confused about which parts of the context are relevant to the current task and which aren't. Keeping the context tidy keeps Claude focused on the right things. So, that's what I do. Got any tips of your own? Lemme know.

101

25 Comments

A recent K8s podcast with Clayton Coleman and Rob Shaw is a fascinating listen: https://lnkd.in/gJmpX_DB Why do standard K8s approaches don't work with LLM inference? Unlike a normal web app, LLM generates text one token at a time, each one depending on the one that came before. The technique to remember the context of the conversation is called KV caching. Now regular K8s tools for balancing traffic, adding servers and managing resources at huge scale work really well, because the amount of required varies depending on the prompt. The tools the community created to solve this problem: 🛠️  vLLM - popular inference server that is very good at KV cache management. 🛠️  Inference Gateway - smarter load balancer that enables efficient traffic distribution. 🧩 The llm-d project brings these together in a ready-to-use stack, tested and benchmarked well-lit paths, smarter routing, intelligent work splitting to optimize time-to-first token, methods to run "Mixture-of-Experts" models across multiple servers

50

One year ago, Philip Walsh, PhD and Dave Micko predicted that AI will lead to more software engineers overall. It was a contrarian take, and still is. Right now the trends are hiring freezes and layoffs as a kind of "AI productivity signaling", rather than hiring AI engineers who can create software effectively using AI. So it's good to hear Greg Brockman, co-founder and president of OpenAI, make the same point as Philip and Dave on the Latent Space podcast this week. Here is what he said: "The productivity impacts of people being able to do more means we actually want more people. We are limited by the ability to build more software, to refactor, to deal with tech debt, and if we have tools that make that 10x easier, we're going to be able to do 100x more things. There is an incredible opportunity where the models are not just a driver of 'Do the same stuff more efficiently' but instead 'Do way more'. " [Minute 53 of the podcast] https://lnkd.in/ef9K-8q3 Here is our prediction last year [preview available, full access for Gartner subscribers] : https://lnkd.in/gEmeYkCB

117

3 Comments