Bhaskarjit Sarmah

💰 𝗧𝗵𝗲 𝗔𝗜 𝗮𝗿𝗺𝘀 𝗿𝗮𝗰𝗲 𝗶𝘀 𝗵𝗲𝗮𝘁𝗶𝗻𝗴 𝘂𝗽. Big Tech is pouring billions into AI infrastructure. FY25 AI CapEx estimates: 🟠 𝗔𝗺𝗮𝘇𝗼𝗻: $100B+ ⚫ 𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁: $80B 🔴 𝗔𝗹𝗽𝗵𝗮𝗯𝗲𝘁: $75B 🔵 𝗠𝗲𝘁𝗮: $63B Despite record spending, supply is still catching up with demand. AI is already paying off for these companies. But who’s best positioned to win?

91

2 Comments

***** Algorithm of Thought (AoT) ***** Most of us have tried clever prompts to get better answers from large language models. But there is a new way of thinking called Algorithm of Thought (AoT). Instead of only giving instructions and hoping the model figures it out, AoT teaches the model how to explore a problem space. It does not settle for the first answer; it searches, tests, and refines, similar to how a person reasons through a tough question. Why this matters: More accurate and reliable results Handles complex, multi-step problems better Makes LLMs feel smarter and more useful in real work This feels like the shift from guessing to thinking. The prompting may not be about writing the perfect question but about guiding the model’s reasoning itself. #AI #PromptEngineering #LLM #AlgorithmOfThought #MachineLearning

28

8 Comments

What things in Gen AI a Data Engineer should know! LLM - Large Language Model (act's like a brain) This LLM has 2 main stages - Pre training - Fine tuning although there are more.. In Pre training Big Companies will take a big chunk of data (100's of TB) available on internet, blogs, wikipedia etc and get a gist of it in a few GB's. Lets consider 70 Billion parameter model, each parameter here will be a float of 2 bytes and will take 140 GB. This will take 10's of days and 1000's of heavy machines, and 10's or 100's of million dollars to do this and we get a base/foundation model. This base model cannot answer question, rather if you give it a question it might give more questions. Now, we can leverage the foundation model and develop something. As a Data Engineer, => Prompt Engineering is the easiest thing to do, but when creating applications you need to write it in a programatic way. writing better prompts with more context can help us get better results. => Retrieval Augmented Generation (RAG) - As a Data Engineer you can index your company documents, here you would require to clean the data and index it in a vector database. Later when a prompt is given, it will try to do a simarity search and will search in your vector database. whatever results come will be augmented to your prompt and it's like more context is passed as part of the prompt for better results. So, setting up a vector DB, cleaning documents, and storing in vector DB can be directly in a Data Engineers portfolio. => Model Fine Tuning - This is where the world is moving towards. Instead of striving for bigger and bigger base model, it has been seen that a decent size base model with a good fine tuning done for specific tasks can perform better. So, we will take a foundation mode, and we might have 10's or 100's of thousands of quality questions and answers (high quality training data) on which we fine tune the model. There can be different fine tuned models specific to each task. Do not try using a single LLM for everything. A better way is to use specialised LLM's fine tuned differently to do different stuff's. A model might be good at summarisation, other might be good at judging which summaries are better. Also the LLM's can be given access to various tools like google search, libraries to plot a graph, calculator etc in order to achieve a task. => Now when you develop complex application, it might happen you use a LLM for summarisation, and the output of this LLM goes to other LLM for judging. So there can be a big pipeline, this can be developed using langchain. So I feel - A basic understanding of terminologies around Pre training - Vector Database - RAG (Augmenting the prompt) - Model Fine tuning - Lang Chain - Agentic AI are the things which at the minimum a Data Engineer should know. But there are many more things within this, which I will cover in extreme depth in my "AI for Data Engineers Program" I hope you learnt something new!

50

3 Comments

"How deep should I go in ML?" This question has always bothered me, where there is so much to learn. -New models emerge weekly. -LinkedIn is flooded with AI breakthroughs. -Research papers are published at an insane pace. So as an ML enthusiast, what should I do? Should I dive deep into every new model? Is it worth it? What if it becomes obsolete next month? A couple of weeks ago the entire LinkedIn was about DeepSeek. It is impossible to keep up with everything deeply. I hate doing superficial work, but I can’t master every new development. That is why I have decided to eliminate noise. I want to focus on what has long-term value, not short-term hype (not saying DeepSeek is noise - it is fascinating). But distinguishing signal from noise is hard. The one thing I know for sure: is that foundational knowledge is invaluable. It builds confidence. It is timeless. But even foundational knowledge is an endless ocean. I could spend: -A month on Lasso regularization. -Three weeks on bias-variance trade-off. -Eight months on Graph Neural Networks. How much is too much? What’s the right depth? I developed a technique to answer this: I ask myself, “Do I truly understand this, or am I fooling myself?” It is easy to believe we know something- until we explain it. So here is my method (since the last few months): I record a quick lecture on an interesting topic. As I speak, I pinpoint my weak areas. If I realize gaps, I trash the recording, learn more, and try again. I repeat until I am confident. This has been the most effective way I have learned. I retain concepts longer, and the clarity I gain is unmatched. The downside? Teaching takes time- time I could spend building AI products at Vizuara. But the satisfaction of simplifying and sharing knowledge is worth it. Am I saying ignore new AI trends? No. Stay aware, but don’t get distracted by hype or FOMO. Have a vision, stick to it. There are people who know a lot of information- the latest news, models, gadgets-but nothing deeply. Don’t confuse information with knowledge. ***** If you wish to learn ML and LLMs from scratch check this out: https://lnkd.in/gua3g9cn

134

4 Comments

Byte Pair Encoding (BPE) is one of the key algorithms which modern LLMs use for tokenisation. BPE was originally invented in the 1990s and was meant to be an information compression algorithm. How did it enter the LLM space? The reason was a 2015 paper titled “Neural Machine Translation of Rare Words with Subword Units” In this paper, researchers showed that the BPE algorithm can be used to improve the efficiency of language translation. BPE solves issues associated with word based tokenisation. If your vocabulary consists of all words in the English language, that is word based tokenisation. The main issue with word based tokenisation is “Out of Vocabulary (OOV) words”. Let’s say you receive a new piece of text which has a spelling mistake, there is no way to tokenise it, since it does not exist in your vocabulary. That’s the OOV problem. BPE allows sub-words to be tokens. Even characters can be individual tokens. If a spelling mistake is made in a word, BPE will just split it into characters and store the characters as tokens. The OOV problem is solved. The way BPE works is that it goes on merging pairs of frequently occurring characters, and adds them to your vocabulary. In this way, common prefixes and common suffixes will be individual tokens. For example, if you consider “tokenisation”: “token” will be one token and “isation” will be another token. That’s how sub-word tokens are created. Commonly occurring words will be retained as full tokens. Rare words will be split into sub-words. We have made 2 videos regarding BPE on our YouTube channel: (1) BPE Theory + Coding: https://lnkd.in/gwg8SucV (2) BPE explained visually (made by Mayank Pratap Singh): https://lnkd.in/gAez3WEU Thanks Mayank for the brilliant visual video and also for the GIF below!

84

Mythology meets machine learning: India's first case study in AI-powered storytelling at scale. There's been legitimate skepticism about AI in entertainment—can technology truly capture the emotional weight of something as sacred as the Mahabharat? What Collective Artist Network & JioHotstar have executed with Mahabharat: Ek Dharmayudh offers a compelling answer. This is production infrastructure, not just production assistance: ✅ 100 episodes with cinema-grade VFX ✅AI-driven character modeling for expressive consistency across the series ✅Workflows that compress multi-year timelines into months ✅Visual ambition that would've been cost-prohibitive a decade ago Today's AI pipelines enable higher visual standards (as compared to the ones we have grown up with) within scalable budgets—turning what was once a financial barrier into competitive advantage. This marks India's evolution from AI as a tool → AI as core infrastructure for entertainment. 💡The real question: If AI can industrialize the Mahabharat while maintaining cultural depth, what's the next frontier? Original Indian epics designed for global audiences? I would suggest to embrace rather than fighting it, change can be scary but it the only constant #AIInnovation #CulturalTech #ThoughtLeadership #StreamingStrategy #Mahabharat #AI

95

20 Comments

Large language models (LLMs) have crushed NLP on many fronts, but Indian languages face a significant hurdle where. The diverse scripts, complex morphology, and rich linguistic nuances of these languages often make traditional tokenization methods inadequate. To better understand this challenge, I've compiled research from several publications into an article that explores the impact of different tokenization strategies on LLMs for Indian languages. It delves into the unique challenges these languages present and examines the latest research pushing the boundaries of efficiency and accuracy in LLMs. Curious to know more? Let's connect and discuss! https://lnkd.in/g57HU983 #NLP #LLMs #IndianLanguages #AI #MachineLearning

21

3 Comments

Eight years ago, nearly no one foresaw what deep learning would become today. When I funded Eightfold in 2017, I was persuaded by Ashutosh Garg's core conviction - “Deep learning is going to completely change the way we think about search today,” he told me. He immediately tied that belief to a clear use case: reinventing how companies match people to jobs. The story of Cohesity was similar. A decade ago with the boom in data, businesses were generating 100s of duplicate copies of the same information. Founder Mohit Aron warned that “this duplicative data is going to kill companies at scale.” At the time, only Google had a globally distributed file system capable of managing data at that level. Mohit’s vision was to bring that capability to every enterprise - and that became Cohesity. Many factors contributed to these startups’ success, but a common thread ran through both: the founders anticipated the future in specific, actionable ways. Ashutosh saw that deep learning could reinvent how companies match people to jobs. Mohit saw that duplicate data would cripple enterprises as they scaled. That kind of foresight gave them a first-mover advantage. By the time others caught on, they were already far ahead of any competition. When I talk to founders that's what I look for: Can they see beyond what's obvious? Do they have prescient insights into where the world is headed? When I meet such people, I can feel the enthusiasm in their vision. It's contagious, and I can't help but rally behind them.

137