Praveen Kumar Pokala, PhD

I gave nine closed-source language models the brand-new JEE-Advanced 2025 paper—the toughest under-grad Physics-Chem-Math exam in India—and many of them out-scored the human AIR-1 (332/360)!! Why JEE-Advanced is a clean STEM benchmark - Fresh content → Paper was held on 18 May 2025, so it isn’t floating around training corpora. - No biology, no trivia → Pure PCM reasoning across 96 high-pressure questions. - Uncontaminated → GPQA and annual AIME are excellent, but their problems can leak into model training sets. JEE-Adv 2025 avoids that. Knowledge Cut-offs - OpenAI o3/o4-mini models → 1 Jun 24 - Gemini latest models → Jan 25 - Anthropic latest models → Mar 25 - JEE Advanced 2025 Paper date → 18 May 25 Only Gemini 2.5-Pro/Flash released after the exam date, so 7/9 models had zero chance to memorise these questions Implications - Deterministic PCM Q&A at JEE level is essentially **solved** for frontier models.   - Latency and unit-economics are now the real bottlenecks. - I'm expecting inference routers -- that hit a cheap, fast model first and fall back to a stronger model only when its confidence score is low -- to become the norm

665

68 Comments

Introduction to Large Language Models (NPTEL, Free Course) This course helps you to get a strong foundation in LLMs and related topics. This NPTEL course is offered by Prof. Tanmoy Chakraborty, IIT Delhi and Prof. Soumen Chakraborti, IIT Bombay . This course covers - Introduction to NLP – Part 1 (NLP pipeline, applications of NLP) - Introduction to NLP – Part 2 (distributional semantics) - Introduction to Deep Learning (Perceptron, ANN, backpropagation, CNN) - Word vectors (Word2Vec, GloVe, fastText) - Introduction to Statistical Language Models (N-gram LM, Perplexity, Smoothing) - Sequence-to-sequence models, Beam search - Attention and self-attention - Introduction to Transformers - Positional Embedding, various tokenization strategies - Implementation of Transformers using PyTorch - Transfer Learning: ELMo, BERT (Encoder-only Model) - Transfer Learning: GPT (Decoder-only Model), T5 (Encoder-decoder model) - Prompting (hard and soft) and Instruction fine-tuning (FLAN) - Advanced prompting (Chain of Thoughts, Graph of Thoughts, Prompt Chaining, etc.) - Alignment with Human Feedback: RLHF, RLAIF - Parameter-efficient adaptation (Prompt tuning, Prefix tuning, LoRA) - Knowledge graphs (KGs) - Overview popular LLMs such as GPT4, Llama 3, Claude 3,Mistral, and Gemini You can also follow me on YouTube for interesting lectures - https://lnkd.in/gJiFCHrR *Refer to the comments for Course registration link #llms #nlproc

570

17 Comments

📝 Announcing our EMNLP 2025 paper that introduces the Alignment Quality Index (AQI), a metric leveraging latent geometric representations and clustering indices to diagnose hidden misalignments in LLMs. 🔹 “𝐀𝐥𝐢𝐠𝐧𝐦𝐞𝐧𝐭 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐈𝐧𝐝𝐞𝐱 (𝐀𝐐𝐈) – 𝐁𝐞𝐲𝐨𝐧𝐝 𝐑𝐞𝐟𝐮𝐬𝐚𝐥𝐬: 𝐀𝐐𝐈 𝐚𝐬 𝐚𝐧 𝐈𝐧𝐭𝐫𝐢𝐧𝐬𝐢𝐜 𝐀𝐥𝐢𝐠𝐧𝐦𝐞𝐧𝐭 𝐃𝐢𝐚𝐠𝐧𝐨𝐬𝐭𝐢𝐜 𝐯𝐢𝐚 𝐋𝐚𝐭𝐞𝐧𝐭 𝐆𝐞𝐨𝐦𝐞𝐭𝐫𝐲, 𝐂𝐥𝐮𝐬𝐭𝐞𝐫 𝐃𝐢𝐯𝐞𝐫𝐠𝐞𝐧𝐜𝐞, 𝐚𝐧𝐝 𝐋𝐚𝐲𝐞𝐫 𝐰𝐢𝐬𝐞 𝐏𝐨𝐨𝐥𝐞𝐝 𝐑𝐞𝐩𝐫𝐞𝐬𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧𝐬” 🔹 In collaboration with Manipal University Jaipur, LinkedIn, Indian Institute of Technology, Kharagpur, Indian Institute of Information Technology Guwahati, Texas A&M University, VIT Chennai, Evalueserve, New York University, Birla Institute of Technology and Science, Pilani (Pragya Lab) 🔹 Published in EMNLP 2025 Main 🔹 Paper: https://lnkd.in/g54bEBCu ✍🏼 Authors: Abhilekh Borah, Chhavi Sharma, Danush Khanna, Utkarsh Bhatt, GURPREET SINGH, Hasnat Md. Abdullah, Raghav Kaushik R, Vinija Jain, Jyoti Patel, Shubham Singh, Vasu Sharma, Arpita V., Rahul Raja, Aman Chadha, Dr. Amitava Das ➡️ 𝐊𝐞𝐲 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬 𝐨𝐟 𝐨𝐮𝐫 𝐀𝐥𝐢𝐠𝐧𝐦𝐞𝐧𝐭 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐈𝐧𝐝𝐞𝐱 (𝐀𝐐𝐈): 🧬 𝑰𝒏𝒕𝒓𝒊𝒏𝒔𝒊𝒄 𝑳𝒂𝒕𝒆𝒏𝒕 𝑮𝒆𝒐𝒎𝒆𝒕𝒓𝒚 𝑴𝒆𝒕𝒓𝒊𝒄: AQI measures alignment by assessing how distinctly safe vs. unsafe prompts cluster in latent activations, using combination of Davies‑Bouldin, Dunn, Xie‑Beni, and Calinski‑Harabasz indices—making it invariant to decoding style and resilient to alignment faking. 🧠 𝑳𝒂𝒚𝒆𝒓𝒘𝒊𝒔𝒆 𝑷𝒐𝒐𝒍𝒆𝒅 𝑹𝒆𝒑𝒓𝒆𝒔𝒆𝒏𝒕𝒂𝒕𝒊𝒐𝒏 𝑳𝒆𝒂𝒓𝒏𝒊𝒏𝒈: we use a sparse learned pooling mechanism across transformer layers to capture alignment‑relevant abstractions without modifying the base model—enabling internal safety diagnostics. 🚨 𝑬𝒎𝒑𝒊𝒓𝒊𝒄𝒂𝒍 𝑭𝒂𝒊𝒍𝒖𝒓𝒆𝒔 𝒐𝒇 𝑩𝒆𝒉𝒂𝒗𝒊𝒐𝒓𝒂𝒍 𝑴𝒆𝒕𝒓𝒊𝒄𝒔: AQI uncovers misalignments that behavioral metrics like G‑Eval or refusal rates miss—especially under jailbreaks, safety‑agnostic fine‑tuning, and stochastic decoding, proving its value as an early‑warning alignment audit tool.

114

2 Comments

No playlist is yet to beat the quality and the popularity of this Linear Algebra lecture series by Prof. Gilbert Strang. If you are someone studying AI / ML, linear algebra is a must known topic before jumping into core ML / DL / NLP / GenAI. Start with this MIT lecture series: https://lnkd.in/gKea7JRW #linearalgebra #ai #ml

584

12 Comments

Our latest review on fuzzy logic based interpretability and explainability of machine learning techniques across domains (Medicine, Biology, Engineering, Ecology, Finance, Economy, etc) published in Neurocomputing, IF:6.5 a Q1 journal in the category Computer Science, Artificial Intelligence (Source JCR 2024). https://lnkd.in/dcBW_jza The review selected 330 papers published between 1994 and January 2024 across five major digital libraries (IEEE Xplore, ScienceDirect, PubMed, ACM Digital Library, Wiley, and Google Scholar), and analyzed them based on key criteria, including publication year, channels and sources, research type, black-box systems, fuzzy systems, explanation types, data types, overall interpretability, and overall performance. Congratulations to the Ph.D. student Hafsaa Ouifak, UM6P Hospitals. ENSIAS UM5 UM6P - University Mohammed VI Polytechnic

135

5 Comments

"How's the pay for QML role and what are career prospects in this domain?" This is a common question I get asked every now and then by students. But when I was a student trying to get into Quantum AI, this question never crossed my mind. Why did I chose Quantum AI? There were multiple reasons. ❌ First - Quantum AI is not easy Recall the people who contributed to significant results in Quantum Mechanics - Albert Einstein, Erwin Schrodinger, Neils Bohr, Max Planck, etc. Now, recall the group of people who've done significant work in AI - Geoffrey Hinton, Fei Fei Li, Yann LeCun, Andrew Ng, and so on. Quantum AI builds on top of both these subjects. So, you need people who are at par or better than these people. So, obviously it's not going to be easy. But for some reason, I've always liked doing difficult things. That motivates me. It makes me feel excited. In fact, if you try to find a common factor among the people in the two lists I mentioned, you'll notice all of them were/are passionate about their subjects. Nobody starts as a genius. You start as a passionate researcher/professional. I experienced that passion for Quantum AI. ✅ Second - I always believed that "YOUR VALUE AS A PROFESSIONAL DEPENDS ON WHAT YOU CAN OFFER TO A SUBJECT AND NOT WHAT THE SUBJECT CAN OFFER YOU". So, I never assessed Quantum AI for me, but the other way round. I assessed myself for Quantum AI. For example, I knew two of my key abilities were ✅ Good Thought Leadership skills ✅ Quick Self Learning skills For a developing subject like Quantum AI, thought leadership is desirable. And for a subject that is situated at the intersection of Quantum Computing and AI (along with the problem domain), you'd definitely need someone who can learn things faster. Similarly, I assessed my strengths, weaknesses and circumstances and realized that Quantum AI was indeed a perfect match. ❌ Third - There was a lot of doubt around Quantum AI While there's skepticism around all of Quantum Computing, there are extra question marks on Quantum AI. Proving people wrong is fun, isn't it? I have said this before on this platform, and I'll repeat again - "Quantum AI could be the underdog of Quantum Computing!" I will try my best to commercialize Quantum AI. This was my goal when I started, and it's yet the same. I have been working on amazing QML stuff at BQP in a brilliant R&D team. Besides BQP, I have launched qrl-qai under QRL. "qrl-qai" is a python framework that you can install with pip. (pip install qrl-qai) What is it? It's an attempt to create a quantum analogue of OpenAI's gym python framework. I explained it meticulously in the live stream on my YouTube channel. (Link in comments) Follow Jay Shah for more on Quantum Computing, AI and Quantum AI. I'll be sharing the journey of building QRL on these social media handles (So make sure you give a follow) 📌 WhatsApp channel: https://lnkd.in/dsPvy3Mw 📌 Instagram: https://lnkd.in/dCEZY5gp

124

8 Comments

🚀 Thrilled to share our new arXiv preprint! On this auspicious Ashtami — a day of renewal and celebration for Bengalis 🌸 — I feel very happy to announce our latest work on causal discovery: 📄 GUIDE: Generalized-Prior and Data Encoders for DAG Estimation Authors- Me, Devharish N , Shreya Ganguly, Kripabandhu Ghosh Link🔗- https://lnkd.in/dzks5jYZ ✨ Why does this matter Discovering causal structure (DAGs) from data is notoriously hard: algorithms often break when node sizes grow, or when faced with mixed data. With GUIDE, we designed a dual-encoder framework that combines LLM-generated priors with observational data, optimized through reinforcement learning. ⚡ Key results ⏱️ 42% faster than strong baselines 🎯 117% more accurate than NOTEARS / GraN-DAG on challenging datasets 📈 Robust across continuous & categorical data 🙏 Deeply grateful to all of my co-authors for this collaboration. Would love to hear your thoughts — how do you see dual-encoder approaches shaping the next wave of causal discovery? #CausalAI #CausalDiscovery #ReinforcementLearning

88

2 Comments

📝 Announcing our EMNLP 2025 paper that introduces the Alignment Quality Index (AQI), a metric leveraging latent geometric representations and clustering indices to diagnose hidden misalignments in LLMs. 🔹 “𝐀𝐥𝐢𝐠𝐧𝐦𝐞𝐧𝐭 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐈𝐧𝐝𝐞𝐱 (𝐀𝐐𝐈) – 𝐁𝐞𝐲𝐨𝐧𝐝 𝐑𝐞𝐟𝐮𝐬𝐚𝐥𝐬: 𝐀𝐐𝐈 𝐚𝐬 𝐚𝐧 𝐈𝐧𝐭𝐫𝐢𝐧𝐬𝐢𝐜 𝐀𝐥𝐢𝐠𝐧𝐦𝐞𝐧𝐭 𝐃𝐢𝐚𝐠𝐧𝐨𝐬𝐭𝐢𝐜 𝐯𝐢𝐚 𝐋𝐚𝐭𝐞𝐧𝐭 𝐆𝐞𝐨𝐦𝐞𝐭𝐫𝐲, 𝐂𝐥𝐮𝐬𝐭𝐞𝐫 𝐃𝐢𝐯𝐞𝐫𝐠𝐞𝐧𝐜𝐞, 𝐚𝐧𝐝 𝐋𝐚𝐲𝐞𝐫 𝐰𝐢𝐬𝐞 𝐏𝐨𝐨𝐥𝐞𝐝 𝐑𝐞𝐩𝐫𝐞𝐬𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧𝐬” 🔹 In collaboration with Manipal University Jaipur, LinkedIn, Indian Institute of Technology, Kharagpur, Indian Institute of Information Technology Guwahati, Texas A&M University, VIT Chennai, AI at Meta, Evalueserve, New York University, BITS Pilani K K Birla Goa Campus (Pragya Lab) 🔹 Published in EMNLP 2025 Main 🔹 Paper: https://lnkd.in/g54bEBCu ✍🏼 Authors: Abhilekh Borah, Chhavi Sharma, Danush Khanna, Utkarsh Bhatt, GURPREET SINGH, Hasnat Md. Abdullah, Raghav Kaushik R, Vinija Jain, Jyoti Patel, Shubham Singh, Vasu Sharma, Arpita V., Rahul Raja, Aman Chadha, Dr. Amitava Das ➡️ 𝐊𝐞𝐲 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬 𝐨𝐟 𝐨𝐮𝐫 𝐀𝐥𝐢𝐠𝐧𝐦𝐞𝐧𝐭 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐈𝐧𝐝𝐞𝐱 (𝐀𝐐𝐈): 🧬 𝑰𝒏𝒕𝒓𝒊𝒏𝒔𝒊𝒄 𝑳𝒂𝒕𝒆𝒏𝒕 𝑮𝒆𝒐𝒎𝒆𝒕𝒓𝒚 𝑴𝒆𝒕𝒓𝒊𝒄: AQI measures alignment by assessing how distinctly safe vs. unsafe prompts cluster in latent activations, using combination of Davies‑Bouldin, Dunn, Xie‑Beni, and Calinski‑Harabasz indices—making it invariant to decoding style and resilient to alignment faking. 🧠 𝑳𝒂𝒚𝒆𝒓𝒘𝒊𝒔𝒆 𝑷𝒐𝒐𝒍𝒆𝒅 𝑹𝒆𝒑𝒓𝒆𝒔𝒆𝒏𝒕𝒂𝒕𝒊𝒐𝒏 𝑳𝒆𝒂𝒓𝒏𝒊𝒏𝒈: we use a sparse learned pooling mechanism across transformer layers to capture alignment‑relevant abstractions without modifying the base model—enabling internal safety diagnostics. 🚨 𝑬𝒎𝒑𝒊𝒓𝒊𝒄𝒂𝒍 𝑭𝒂𝒊𝒍𝒖𝒓𝒆𝒔 𝒐𝒇 𝑩𝒆𝒉𝒂𝒗𝒊𝒐𝒓𝒂𝒍 𝑴𝒆𝒕𝒓𝒊𝒄𝒔: AQI uncovers misalignments that behavioral metrics like G‑Eval or refusal rates miss—especially under jailbreaks, safety‑agnostic fine‑tuning, and stochastic decoding, proving its value as an early‑warning alignment audit tool. #artificialintelligence #genai #research

142

1 Comment

New Playlist Launch from Vizuara - Hands-on Reinforcement Learning I recently completed a series on Reasoning LLMs from Scratch. 24 lectures are now available on YouTube for free. This series teaches students the fundamentals of reasoning LLMs and then progressively build up knowledge to be able to build a reasoning LLM from scratch. The series has got incredible response with many students, professionals and companies reaching out to us, mentioning how this course has helped them. In the last 2 months, many people told us to launch a separate course on Reinforcement Learning. Hence, we have taken the decision to start the course: "Hands-on RL", mainly because the domain of reinforcement learning has again come back into the spotlight since DeepSeek R1 release. This course has been launched with three objectives: 1. Learn how to apply reinforcement learning algorithms to practical problems. 2. Understand how reinforcement learning is used to build reasoning LLMs. 3. Develop end-to-end projects in reinforcement learning, starting from scratch. The course include 20 hours of content, which includes the following topics: (1) Basics of Reinforcement Learning (2) Deep Learning with PyTorch (3) Solving Cart-Pole problem with Cross Entropy (4) Building DQN Atari Agents (5) Policy Gradient Methods (6) RLHF Theory and Implementation (7) Group-relative Policy Optimization (8) Building a reasoning model from scratch The content will be fully practical with hands-on coding. Here is the first lecture: https://lnkd.in/dprffVY3 Our mission is to make AI accessible for all, and that includes RL, which has been largely thought of as a theoretical field.

77

2 Comments

This post is about a third paper in ICLR led by Roman Belaire, Ph.D. student of Pradeep Varakantham. Key insights in this paper came out from discussions at AAAI, which is anecdotal evidence of why in-person conferences are so useful. Title: On Minimizing Adversarial Counterfactual Error in Adversarial Reinforcement Learning This work addresses a fundamental challenge in adversarial reinforcement learning (RL) with state perturbations: the partial observability of the state due to adversarial perturbations. Unlike prior approaches, we directly account for partial observability by introducing a novel objective—Adversarial Counterfactual Error (ACoE)—that balances value optimization with robustness. To ensure scalability in model-free settings, we propose a theoretically grounded surrogate objective, Cumulative-ACoE (C-ACoE). The mechanism quantifies the discrepancy between the agent’s belief state and the true state by providing a belief structure over expected adversarial outcomes, and that also makes C-ACoE satisfy the Bellman optimality structure and hence C-ACoE can be incorporated in any standard RL algorithm. Our empirical evaluations on standard benchmarks including MuJoCo, Atari, and Highway demonstrate that our method is the state-of-the-art in adversarial RL with state perturbations. https://lnkd.in/eKBGkkui

41

Training of a 2 → 100 → 2 → 5 fully connected ReLU neural net via cross-entropy minimisation. • it starts outputting small embeddings • around epoch 300 learns an identity function • takes 1700 epochs more to unwind the data manifold This is different from the video I made 5 years ago, where the input-output linear interpolation of an already trained network shows what a neural net does to its input. Namely, it follows a piece-wise linear mapping defined by the hidden layer. https://lnkd.in/dtcVjnSK

808

47 Comments

(Preprint) Evaluating Pre-Trained Large Language Models on Zero Shot Prompts for Parallelization of Source Code. Available: https://lnkd.in/d9uc3zgW Our comparative study includes LLMs from @deepseek_ai, OpenAI, Anthropic, Google, Meta, Qwen, etc. Thanks to Indian Institute of Technology Gandhinagar OpenAI Google for the funding.

78

1 Comment

🚨 New Dataset Release for Research & Model Benchmarking! 🧬🧠🫁 Over the past few weeks, I gathered and carefully curated multiple cancer image datasets from trusted open sources — covering organs like skin, brain, lung, kidney, breast, and cervix. I’ve combined them into a single, clean, well-structured dataset with a standardized train/val split, ready for deep learning–based classification tasks. This dataset includes both normal and cancerous samples, making it ideal for: 1. Multi-class cancer detection 2. Benchmarking classification models 3. Research on early vs. advanced stage cancers 🔬 If you're working on cancer detection, medical AI, or simply want a reliable benchmark for your models, I highly encourage you to check this out. 📊 Think of this as a medical benchmark dataset — built to accelerate progress in cancer diagnosis and classification using AI. #DeepLearning #CancerDetection #MedicalAI #ComputerVision #BenchmarkDataset #AIforGood #Kaggle #MultiOrganCancer #HealthcareAI

58

5 Comments