Stream’s Post

Update on Cognivise: Building the English AI Coach A short time ago, I was dealing with personal issues and depression that completely blocked me from writing code. I had stopped developing entirely. But thankfully, being introduced to the Stream(Vision_AI_SDK(https://lnkd.in/gwdp7yjz)) by Kunal Kushwaha during the WeMakeDevs hackathon reignited my spark. I am finally building again. My ultimate goal remains the same: to create the most accurate real-time cognitive monitoring system for students and learners. Today, I’m excited to share my latest major addition to Cognivise. 🗣 New Feature: The English AI Coach I wanted to see if I could build a conversational AI that doesn't just listen to your pronunciation, but literally watches your face as you speak. Here is what the English Coach is doing under the hood: 🔹 Dual-AI Architecture: Combining local browser processing (MediaPipe FaceMesh) for 30fps physical tracking, synced with a cloud LLM (Gemini/Groq) for speech parsing and reasoning. 🔹 Canvas Layer Multiplexing: Built 4 distinct visual feedback modes (Composite, Landmarks-only, Full HUD Analysis, Raw Frame) by manipulating HTML5 canvas layers behind the live WebRTC stream. 🔹 Real-time Synchronicity: Overcame strict browser autoplay policies and synchronous call-stack limits to ensure the AI's Text-To-Speech (TTS) responds naturally to the user's analyzed audio. ___________ 🛠 What Was Actually Hard This Time Getting the Vision SDK to accurately track full movement is incredible. But translating that raw physical movement—every micro-expression and blink—into an accurate emotional state is intensely difficult. Right now, the system catches the physical layout perfectly, but the semantic "emotion tracking" is still not exactly where I expect it to be. Sending frame-by-frame snapshot analysis over WebSockets without colliding with the synchronous audio pipelines required deep debugging of React state refs and Canvas requestAnimationFrame loops. Testing this in a remote location with an unstable internet connection makes debugging real-time video+WebSocket systems extremely painful. I am learning exactly why tracking every single frame matters, and how quickly it scales in complexity. _________________________________________________________________ ⚠️ Honest Status & What's Next Cognivise is not perfect yet. Capturing true "emotion" is a work in progress. But getting out of my slump was step one. I’m heading back to my hometown in Delhi soon, and I now have 3 brand new AI project ideas I plan to start sprinting on this month. ----------------------------------------------------------------------------- Massive thank you to Kunal Kushwaha and the WeMakeDevs community for creating environments that push developers back into motion. The journey continues! 🔗 GitHub: https://lnkd.in/gZvJnCfa #AI #VisionAgents #VisionPossible #MachineLearning #WebRTC #FastAPI #ReactJS #BuildInPublic #OpenSource #algsoch #MentalHealthInTech