Shishir Suman

If DeepSeek's claim that it trained its model with just $5.5M is true, it proves a fantastic theory again. So, what did DeepSeek do? 👇 Forced to use H800 GPUs - a sanctioned variant with 30% less memory bandwidth - DeepSeek engineers achieved remarkable optimizations through low-level PTX programming. 🚀 They trained 671B parameter MoE model (DeepSeek-V3) using 2,048 H800s for $5.58M - 11X more efficient than Meta's comparable Llama 3 training. What's the theory that they prove? 👇 More funding does not necessarily result in better technology. 😂 Startups with less funding and fewer resources can run in a highly optimized manner with a singular focus - customer success. With less than 10% of funding compared to some of our competitors, Lyzr AI has been a disruptive startup in the enterprise AI space. - first among the enterprise agent platforms to launch a self-serve platform for the enterprise devs and business teams - first among agent frameworks to launch a safe AI and responsive AI focused agent architecture  - first among the agent frameworks to launch Auto-prompt Enhancer, even before DSPy - first among the general-purpose agent platforms to launch a voice agent and a SQL agent - first among the agent platforms to launch dynamic app building like Bolt and Lovable Yes, we did see many failures in the last 18 months. But we continue to learn from these lessons. - The first version of AI SDR agent that we launched completely bombed. - We lost almost 4 months to the AI SDR hype cycle.  - We had to rewrite our core framework twice to get to the desired state.  - We failed at developer marketing. So, what actually matters? 🤔 With OpenAI's MOAT being questioned since the DeepSeek revolution, the start-ups building on GPT have a better MOAT. Papers, Ph.Ds, Stanfords, FAANGs, Network - none of these matter if you are not customer obsessed. And as a startup, if you are customer obsessed, none of the above matters. 🤷‍♂️

83

1 Comment

In a new blog by Smallest AI, we talk about issues in large fused kernels, particularly problems with: 1. Dynamic Input Shapes 2. Synchronization and Deadlocks 3. L1 cache latency compounding 4. Cross SM coordination 5. Warp context bottlenecks Further, we segway into what would it take to build a sub 50ms(and lower) speech model. Link and author name in comments, let us know what you think!

135

8 Comments

Introducing "Building with Llama 4." This short course is created with Meta, and taught by Amit Sangani, Director of Partner Engineering for Meta’s AI team. Meta’s new Llama 4 has added three new models and introduced a Mixture-of-Experts (MoE) architecture to its family of open-weight models, making them more efficient to serve. In this course, you’ll work with two of the three new models introduced in Llama 4. First is Maverick, a 400B parameter model, with 128 experts and 17B active parameters. Second is Scout, a 109B parameter model with 16 experts and 17B active parameters. Maverick and Scout support long context windows of up to a million tokens and 10M tokens, respectively. The latter is enough to support directly inputting even fairly large GitHub repos for analysis! In hands-on lessons, you’ll build apps using Llama 4’s new multimodal capabilities including reasoning across multiple images and image grounding, in which you can identify elements in images. You’ll also use the official Llama API, work with Llama 4’s long-context abilities, and learn about Llama’s newest open-source tools: its prompt optimization tool that automatically improves system prompts and synthetic data kit that generates high-quality datasets for fine-tuning. If you need an open model, Llama is a great option, and the Llama 4 family is an important part of any GenAI developer's toolkit. Through this course, you’ll learn to call Llama 4 via API, use its optimization tools, and build features that span text, images, and large context. Please sign up here: https://lnkd.in/gXKeipht

2,834

111 Comments

Earlier today we saw significant market reactions, including a drop in tech shares of major AI chip and tech companies, as DeepSeek-R1 rivaling with OpenAI-o1 model. 𝐇𝐞𝐫𝐞’𝐬 𝐰𝐡𝐲 𝐃𝐞𝐞𝐩𝐒𝐞𝐞𝐤 𝐢𝐬 𝐭𝐫𝐞𝐧𝐝𝐢𝐧𝐠 𝐚𝐧𝐝 𝐡𝐚𝐬 𝐪𝐮𝐢𝐜𝐤𝐥𝐲 𝐫𝐢𝐬𝐞𝐧 𝐭𝐨 𝐭𝐡𝐞 𝐭𝐨𝐩 𝐢𝐧 𝐭𝐡𝐞 𝐀𝐈 𝐦𝐚𝐫𝐤𝐞𝐭: 1. Rapid Ascent: DeepSeek grabbed the top spot on the App Store, challenging established player OpenAI. It's R1 model was built in just 2 months, showcasing its capabilities to bring advanced AI solutions to market. 2. Top Performance: Matches the capabilities of leading AI models with fewer resources. 3. Cost-Efficient: Developed for under $6 million using Nvidia's less-advanced H800 chips, making it significantly cheaper than its competitors. 4. Pricing Strategy: Highly competitive, with models priced up to 40 times lower than comparable models from OpenAI. 5. Energy Efficient: Uses a fraction of the GPUs, reducing energy consumption and environmental impact, and challenging the notion that scaling AI requires more computing power and investment. 6. Open Source: Allowing developers to test and improve the models. 7. Architecture: Efficient use of computational resources through Mixture-of-Experts (MoE) architecture. 8. Reasoning Capabilities: Advanced reasoning, logical inference, and problem-solving - model repeatedly learns through Reinforcement Learning (RL) approach 9. Market Disruption: Forcing major players to lower their prices and rethink on their strategies. 10. Accessibility: On web, app, and API for free 𝐖𝐡𝐲 𝐢𝐭 𝐢𝐬 𝐚 𝐛𝐢𝐠 𝐝𝐞𝐚𝐥 𝐢𝐧 𝐭𝐡𝐞 𝐭𝐞𝐜𝐡 𝐢𝐧𝐝𝐮𝐬𝐭𝐫𝐲?  - DeepSeek's innovative approach not only challenges the status quo but also sets new standards for efficiency and sustainability in AI development. - We see market shifting with tech stocks and this could lead to a reevaluation of AI investment strategies and evaluating how to do sustainable AI development practices. 𝐀 𝐧𝐨𝐭𝐞 𝐚𝐛𝐨𝐮𝐭 𝐩𝐫𝐢𝐯𝐚𝐜𝐲 𝐜𝐡𝐞𝐜𝐤: Deepseek stores user information on servers in China, raising concerns on data privacy and security. Please review their privacy policy on what all information is gathered and learn from their site how they handle personal info. Image Credit: REUTERS #AI #Innovation #DataScience #MachineLearning #Algorithms #RiskManagement #Technology #Business #Markets

47

1 Comment

👉 Read here: https://lnkd.in/gAsk6N6G 🚀 ANNOUNCEMENT - Amidst all the hype around “AI breakthroughs,” one thing often gets overlooked — rigorous evaluation. It’s easy to build another model or agent and harder to measure ROI. We laser focussed on 2 specific metrics, Precision and Recall. Today we are releasing the complete implementation of the Semantic Model generation framework and evals of one of our core IP. While we are unable to disclose the ground-truth and evals in customer implementations, be rest assured we have achieved higher values for both the metrics across 5 enterprise customers who are live in Production. Hope the community will benefit from this! Author Credits - Jaskaran Singh Narotsit Karki Qamar Ali Ankit Baliyan Raphael Tony #Opensource #DataEngineering #DataScience #GenAI #SemanticModel

39

1 Comment

When I worked at Google, I was lucky to collaborate with some of the brightest machine-learning (ML) engineers. They worked on feature engineering. By picking the factors to guide the ML model, their advances could generate tens to hundreds of millions of additional revenue. Imagine an Excel spreadsheet with hundreds of columns of data. Add two columns, multiply two, divide by another, and subtract a fourth. Each of these is a feature. ML models used features to predict the best ad to show. It started as a craft, reflecting the vibes of the era. Over time, we’ve mechanized this art into a machine called AutoML that massively accelerates the discovery of the right features. Today, reinforcement learning (RL) is in the same place as feature engineering 15 years ago. What is RL? It’s a technique of teaching AI to accomplish goals. Consider a brave Roomba. It presses into a dirty room. Then it must make a cleaning plan and execute it. Creating the plan is step 1. To complete the plan, like any good worker, it will reward itself, not with a foosball break, but with some points. Its reward function might be: +0.1 for each new square foot cleaned, -5 for bumping into a wall, and +100 for returning to its dock with a full dustbin. The tireless vacuum’s behavior is shaped by this simple arithmetic. (NB : I’m simplifying quite a bit here.) Today, AI can create the plan, but isn’t yet able to develop the reward functions. People do this, much as we developed features 15 years ago. Will we see an AutoRL? Not for a while. The techniques for RL are still up for debate. Andrej Karpathy highlighted the debate in a recent podcast. This current wave of AI improvement could hinge on RL success. Today, it’s very much a craft. The potential to automate it—to a degree or fully—will transform the way we build agentic systems.

38

1 Comment

25 types OF rag-architectures Rule-Based RAG – Uses predefined rules to filter and retrieve data. Conversational RAG – Optimized for dialogue systems to maintain context. Iterative RAG – Repeatedly refines responses by re-querying databases. HybridAI RAG – Combines neural and symbolic AI for enhanced retrieval. Generative AI RAG – Leverages generative models to craft more coherent outputs. Explainable AI (XAI) RAG – Ensures transparent and interpretable retrieval processes. Context Cache LLM RAG – Stores previous context for faster access and accuracy. Grokking RAG – Enhances performance by deeply understanding patterns. Replug Retrieval Feedback – Iteratively improves by plugging refined data back into the model. Attention Unet RAG – Uses attention layers with UNet structures for precise retrieval. Corrective RAG – Adjusts responses based on user feedback. Speculative RAG – Generates speculative answers while awaiting retrieval. Agenetic RAG – Applies agent-based systems for better data interaction. Self-RAG – Auto-corrects and improves using internal feedback. Adaptive RAG – Dynamically adjusts retrieval strategies based on context. Refeed Retrieval Feedback – Refeeds retrieved data to enhance iterative learning. Realm RAG – Language models enhanced by specialized retrieval modules. Raptor RAG – Uses tree-structured retrieval paths for complex queries. Memo RAG – Memorizes key information for rapid recall. Attention-Based RAG – Focuses on attention mechanisms for selective data retrieval. RETRO RAG – Integrates retrieval with transformers for superior long-context handling. Auto RAG – Automates retrieval and response generation workflows. Cost-Constrained RAG – Optimizes retrieval by minimizing computational costs. ECO RAG – Environmentally conscious RAG with energy-efficient processes. Replug (Retrieval Plugin) RAG – Modular plugin for external retrieval enhancements. Follow the AIKaDoctor (Free AI & Data Science Resources)  channel on WhatsApp: https://lnkd.in/dCTCEKKc Follow Habib Shaikh For more such content.

384

53 Comments

I enjoyed speaking at #PyTorchCon today. Wanted to share one slide from my talk about open source AI infra. This is about how Ray and vLLM work together. LLM inference is growing more and more complex, and doing a good job with LLM inference means working across layers and solving challenges both at the inference engine layer (vLLM) as well as the process orchestration / coordination layer (Ray). The Ray and vLLM open source communities have collaborated closely on this to enable fine-grained control over placement of vLLM workers. This is important for enabling - Cross-node parallelism (e.g., pipeline and tensor) - Prefill-decode disaggregation - Prefix-aware routing - Wide EP

151

6 Comments

Today's guest, Vaibhav Gupta, has developed BAML, the programming language for AI. If you are calling LLMs, you've gotta check BAML out for instant accuracy improvements and big (20-30%) cost savings. More on charming and terrifically brilliant Vaibhav:  • Founder & CEO of Boundary (YC W23), a Y Combinator-backed startup that has developed a new programming language (BAML) that makes working with LLMs easier and more efficient for developers. • Across his decade of experience as a software engineer, he built predictive pipelines and real-time computer vision solutions at Google, Microsoft and the renowned hedge fund The D. E. Shaw Group. • Holds a degree in Computer Science and Electrical Engineering from The University of Texas at Austin. This is a relatively technical episode. The majority of it will appeal to folks who interact with LLMs or other model APIs hands-on with code. In today’s information-dense episode, Vaibhav details:  • How his company pivoted 13 times before settling upon developing a programming language for A.I. • Why creating a programming language was "really dumb" but why it’s turning out to be brilliant, including by BAML already saving companies 20-30% on their AI costs. • Fascinating parallels between today's A.I. tools and the early days of web development. • His unconventional hiring process (I’ve never heard of anything remotely close to it) and the psychology behind why it works. Thanks to Domo and the Open Data Science Conference (ODSC) East 2025 for supporting today's episode of the "Super Data Science Podcast with Jon Krohn", enabling the show to be freely available on all major podcasting platforms as well as the video version we publish on YouTube. This is Episode #859! #superdatascience #machinelearning #llm #largelanguagemodel #ai

49

2 Comments

Inference with large language models has two very different phases, each imposing different demands on hardware. The prefill phase consumes the user’s input and runs it through the transformer stack to build up key-value (KV) caches: dense matrix multiplications dominate. This phase thrives on raw FLOPs and large memory capacity to hold KV cache as context length grows. Memory bandwidth is less of a bottleneck here. The decode (generation) phase reverses that profile: emitting one token at a time, reusing cached KV-pairs, and appending new entries. Arithmetic intensity drops; memory bandwidth dominates, since every new token generation must stream in data from an ever-growing cache. This duality motivates disaggregated inference, where specialized hardware can be used for each stage: compute-rich devices for prefill, bandwidth-rich devices for token generation. The payoff is lower cost, higher efficiency - but it requires complex SW orchestration. Nvidia’s #Rubin CPX GPU is an embodiment of this idea. It is optimized for the prefill/context phase, offering up to 30 petaFLOPs of NVFP4 compute and 128GB of GDDR7 memory. It is monolithic, avoiding the yield and package complexities of massive HBM chiplets. It becomes the “context builder” in a rack-scale design, feeding its results into HBM4-laden Rubin GPUs for decode. #Nvidia is formalizing this mix with the Vera Rubin NVL144 CPX rack: 144 CPX and 144 Rubin GPUs (+36 Vera CPUs) in a single system, each doing what they do best. They also mention CPX only racks with 144 CPX GPUs. CPX does not speak NVLink; it rides on PCIe 6.0 x16, ~128 GB/s per direction, typically via ConnectX-9 SuperNICs that mediate CPX↔Rubin data movement. This is intentional to keep the cost lower: make decode devices fast to each other, and let prefill devices deliver results over cheaper pipes. It also sets an upper bound on how often one can ping-pong between phases; This approach is not without trade-offs. Splitting workloads across heterogeneous GPUs increases orchestration complexity: KV caches must move across slow interconnects, scheduling has to keep both phases balanced, and latency can creep in if software isn’t mature. Not all inference needs million-token contexts, so the benefits are workload-specific! The shape of the rack-scale systems is changing. On one side are GPUs with massive HBM designed to carry both inference phases, on the other are heterogeneous racks with compute-rich and bandwidth-rich devices working in tandem. There is yet another model: composable fabrics - systems where GPUs, CPUs, and memory nodes are not fixed into one rack/tray but are pooled into a shared resource fabric, typically connected over #UALink, #NVLink, or #Ethernet. Software can then dynamically compose a “virtual system” from this pool... As AI pushes toward even larger scales with dynamic workloads, these evolving hw designs promise to redefine efficiency... but only if the software can keep pace. Any thoughts? 🤔

457

29 Comments

A prospective.... From dashboards to LLMs ? did the industry just switch vehicles without rethinking the destination? Remember when the goal of data-driven decision-making was clarity? Dashboards weren’t perfect, but they kinda of worked. They had: - Clear KPIs - Drilldowns and correlations - A shared understanding - Low friction once set up ( heavy engineering efforts sure !!) They told a story. And everyone understood the plot. Now we’ve jumped to LLMs with that comes: Hallucinations, token management, prompt tuning, RAG, caching, AIOps etc.. Sure, LLMs let us ask questions in plain language and it reducing time to insight drastically.. but.. Are we solving the same problem in a shinier way or just transferring complexity from one vehicle to another? do they know your business context? Your data logic? Your governance rules? What is the impact of that to "time to trusted insight" ? Maybe the real challenge isn’t dashboards or LLMs. It’s designing experiences that align with human trust, business logic, and real decisions not just new tech for tech’s sake. Analytics is about making decisions with confidence and clarity. Let’s not forget that. #data #ai #prospective

40

15 Comments

Meta's AI Retreat Reveals The Paradox of Scale No One's Talking About A $1T company just admitted something shocking about AI that changes everything we thought about enterprise deployment. 💥 The Counter-Intuitive Truth: → Meta is deliberately DEGRADING its AI capabilities → They're moving from 40,000 moderators to crowd-sourcing → Their AI accuracy was too high... and that was the problem 🧠 The Hidden Framework: When AI reaches 95%+ accuracy, something fascinating happens: ↳ False positives become more politically costly than false negatives ↳ Scale amplifies precision errors exponentially ↳ The "last mile" of accuracy becomes economically irrational This reveals a profound truth about enterprise AI that no one's discussing... 🔍 The Meta Paradox: 1. Better AI → More Edge Cases 2. More Edge Cases → Higher Business Risk 3. Higher Risk → Political Pressure 4. Political Pressure → Technical Compromise The implications are staggering... 💎 Four Non-Obvious Strategic Insights: 1. The Accuracy Trap → Past 95% accuracy, each 1% improvement doubles operational complexity → Most enterprises are overinvesting in the wrong metrics → The real battle isn't technical - it's perceptual 2. The Scale Inversion Law → At massive scale, precision becomes more important than recall → This inverts traditional ML optimization strategies → Explains why smaller companies can often deploy more aggressive AI 3. The Governance Arbitrage → Meta's move creates a new market gap → Watch for startups exploiting the space between high-accuracy AI and acceptable business risk → This is where the next billion-dollar opportunities lie 4. The Regional Paradox → Global AI systems are becoming impossible to optimize uniformly → The future is "AI federalism" - regional models with local governance → First spotted in Meta's US vs International performance delta 🌋 Market Signals Most Missed: → Anthropic's $60B valuation isn't about tech - it's about governance frameworks → Microsoft's $3B India investment signals regional AI fragmentation → OpenAI's agent delays reveal the new economics of AI risk 💡 The Game-Changing Realization: We've been building AI systems for technical excellence when we should have been building them for governance elasticity. The next wave of enterprise AI won't be won by better models, but by better frameworks for managing their imperfections. 🔥 Want more breakdowns like this? Follow along for insights on: → Building with AI at scale → AI go-to-market playbooks → AI growth tactics that convert → AI product strategy that actually works → Large Language Model implementation #EnterpriseAI #AIStrategy #TechNews #Innovation #digitaltransformation

199

195 Comments

Interesting new blog by Vishnu Sarukkai from Stanford. He suggests we are all making a fundamental mistake instructing LLMs through prompt engineering. Vs using data-driven learning. Looking forward to the details in subsequent posts https://lnkd.in/gec9y5TD

37

2 Comments