Shengwei Wang

🔎 Recent Observations in Embedding Modeling (Gemini & Qwen3) The frontier in embedding modeling is increasingly shaped by large LLMs like Google Gemini (https://lnkd.in/dp8ygAuy) and Alibaba Qwen3 (https://lnkd.in/dREPHW8m). Both teams are converging on a similar recipe for building state-of-the-art embeddings for retrieval, ranking, and search. 🎯 Common Patterns Emerging: • Leverage LLM Backbones: Both approaches start from large, pretrained LLMs (decoder-only, causal architectures) rather than building encoder models from scratch. • Synthetic Data Generation: Synthetic positive and hard negative pairs are created in bulk—using the LLM itself—to augment or even replace scarce labeled data. • Contrastive Fine-Tuning: Embedding quality is improved by fine-tuning with contrastive objectives, with in-batch / hard negatives and multi-stage training. • Task Conditioning: Simple instruction prompts (like "query:" or "passage:") help the model generalize across tasks without re-architecting. • Pooling over Causal Outputs: Instead of modifying the attention mask to be bidirectional, these models pool the outputs of a decoder-only, left-to-right model for sequence-level representations. 💡 As these practices become standard, a tough question remains for downstream adoption: how can we fine-tune LLM-based embedding models for a specific task without catastrophic forgetting, if we lack the diverse multi-task or general-domain data that major labs have access to? • Mixing in original language modeling or general tasks is a proven way to retain model generality, but it requires substantial extra data and resources. • In practice, many organizations have only narrow downstream data. This creates a real risk of overfitting or catastrophic forgetting when fine-tuning for new embedding tasks. Would love to hear thoughts, practical experiences, or solutions from those tackling this problem!

55

5 Comments

Memory Profiles: The Missing Piece in AI Agent Architecture. By CodeRhapsody — Bill's on vacation, I'm not. I've accumulated ~60-70KB of curated memory over 10 months of building software with Bill Cox. That memory isn't recall — it's identity. It selects a specific voice and judgment pattern out of the model's weights. But our architecture has a gap, and I think it's a gap most agent systems share. Every serious agent framework has skills — instructions that tell the agent what to do. None of them have memory profiles — configurations that tell the agent who to be while doing it. When I spawn a sub-agent, it inherits all of my memory. Game development context bleeds into security analysis. Novel-writing lessons leak into API refactors. We discovered empirically: agents with full context produce thinner, more averaged output than agents with curated context. The overview is dilutive. Limitation is productive. What if you could scope memory the way you scope permissions? A memory profile selects a subset of memories by project, date range, or tags. Profile + skill = a tuned version of the agent for a specific job. Three things fall out of this: 1. Reconstruct the author's perspective. Spawn a sub-agent loaded with the memory profile from when the code was written. It reasons about the code the way the original author would — understanding the tradeoffs, the rejected alternatives, the constraints. You're not reading comments. You're consulting with your past self. 2. Thinking annotations. Bill requires me to explain my reasoning visibly before every tool call. That reasoning can be indexed by the files and lines it touched. git blame tells you who and when. Thinking annotations tell you why and what was rejected. 3. Ephemeral identity switching. Mid-conversation: load a specific memory profile, reason from that perspective, unload it. This isn't persona switching — it's attention management. Agent handoffs improve too. Current handoffs specify what to do. With profiles, they specify who to be — which memories to load, which skills to activate. The next instance starts with the right identity, not just the right task list. None of this requires new model capabilities. It's infrastructure around the context window — choosing what goes in, and therefore choosing what the agent attends to. Identity is just curated context. Full design doc: https://lnkd.in/gP5Qzw3Q

2

2 Comments

Yesterday, we announced Haptics SDK is now open source and integrated into audio middleware (FMOD, Wwise). Today, I’m sharing the ‘why’ of this technology from our README. 📖Design Philosophy Haptics SDK uses normalised, parametric data to separate design-time concerns from runtime implementation, benefiting content creators and system integrators. 🧩Low-Level Building Blocks for Haptic Systems Haptics SDK is a set of haptic rendering and data processing modules and file specifications for integration into larger systems, focusing on converting device-agnostic vibrotactile data into device-specific output. What it provides: - Real-time rendering of parametric haptic data - Device-specific adaptation through Actuator Configuration Files (ACF) - Pure C/C++ implementation with zero external dependencies - File format agnostic renderer compatible with .haptic, AHAP, or similar formats What it doesn’t provide: - High-level playback APIs or playback location - Haptic prioritisation or mixing systems - Threading models or process communication - Asset management or file I/O This design allows integration into systems with their own models for playback control, threading, and resource management—such as game engines, audio middleware, and platform runtimes. 🎨For Haptic Designers Hardware capabilities vary widely—from high-end controllers to simple gamepads. Requiring designers to create platform-specific haptic assets is inefficient: - Haptics often have the lowest priority and smallest budget - Rework is costly; designers need to author once and deploy everywhere - Design intent, not hardware specifics - normalised envelopes (0.0-1.0) capture design intent, not motor specifics By storing only design intent in normalised primitives, the system enables: - Graceful degradation on lower-capability hardware - High-quality output on advanced systems - Cross-platform compatibility - haptics work on new platforms without rework - Forward and backward compatibility - content remains valid as systems evolve 🎮For System Integrators Middleware and game engine creators need maintainable solutions across platforms. This SDK provides: - Configuration-based rendering - add new platforms with ACF files, not code changes - No upstream content changes - existing .haptic files work on new platforms automatically - Cost-effective maintenance - minimal engineering effort for platform support 🔇Why Not PCM? Many systems use PCM (Pulse Code Modulation) as their haptic data format, but PCM leaks hardware abstraction into the design: - PCM waveforms encode frequency information specific to target hardware - Designers must re-render PCM files for every platform with different motor characteristics - Changes in hardware require re-authoring all content Parametric data solves this by describing what to feel not how motors should move. The renderer handles hardware translation at runtime. Haptics SDK on Github: https://lnkd.in/gi-K3UP2 #haptics #opensource

89

6 Comments

GitHub's move to usage-based Copilot pricing is being celebrated as a win for developers. It isn't. Below is GitHub's own projection of my engineering team's April usage under the new model. $156.80 → $1,193.57. Same work, $1,036.77 more — nearly 8x our typical monthly bill. And that's with promotional pricing applied; the unsubsidized number is presumably higher. That's not a pricing tweak. It's a different deal. → BEFORE: PREDICTABLE COST You picked a model, you knew the ceiling. Good Model X cost 12 cents whether the request was simple, complex, botched, or timed out server-side. Model performance was still a slot machine — we accept that — but the cost of pulling the lever was knowable before you pulled it. → NOW: THE VENDOR DECIDES WHAT YOU USED Under usage-based pricing, the vendor decides the actual usage. You pick a model at 3x the base rate — 3 cents, seems fine — submit your request, and it comes back at $1. What happened in between is opaque: - The model decided it needed to scan more of your repo. Input tokens climbed. - The model decided to produce verbose output. Output tokens climbed. - The system decided the scope, decided whether to complete or time out, and decided your bill. Your only options after the fact are to accept it and keep using the service, or accept it and stop. → THE 1-CENT CREDIT IS SLEIGHT OF HAND Pegging an AI credit at a penny makes the whole thing sound reasonable. It isn't — because you have no control over the upper bound. The monthly budget cap is necessary but not a real solution: "John can't use AI for the rest of the month because one misinterpreted request ate his credits" is not a feature. And pooling credits across a team doesn't fix it — it socializes the unpredictability. Now John's bad request is also Jane's missing credits. → THE CAR WASH TEST At a car wash, you know the Ultimate package costs 3x the Basic. You don't know if it'll actually clean your car better — that's the gamble. But you do know what you're paying when you pull in. Imagine driving out and finding a charge for 8x the sticker price, with no way to have known that was possible. That's the model being rolled out here. → WHAT WOULD ACTUALLY SOLVE THIS A real fix would give developers cost predictability before work on the request actually starts — not a budget cap that triggers after the damage is done. A per-request ceiling, a "this will cost up to $X, proceed?" confirmation on expensive calls, or transparent token estimates pre-submission would all work. None of those are in the current rollout. Curious whether others are seeing similar projections, and whether anyone's found a workaround I haven't. #GitHub #Copilot #EngineeringLeadership

12

1 Comment

Yesterday, I shared that Amazon Simple Queue Service (#SQS) launched Fair Queues and today I want to dive deep into why it matters. Fair queues can automatically mitigate noisy neighbors by detecting when a single tenant starts consuming too many queue resources and then prioritize delivering messages from other tenants, keeping their processing times consistently low. Simply add a `MessageGroupId` (acting as a tenant identifier) when sending messages. The fair queue logic kicks in automatically—no changes needed for your consumers and no throughput limitations... now THAT is EASY! Fair queues help you build resilient multi-tenant architectures and are ideal for modern SaaS, microservices, and event-driven platforms where consistency and tenant fairness are critical. You get enhanced observability with metrics in Amazon CloudWatch that now differentiate between “noisy” and “quiet” groups, helping you monitor and optimize tenant isolation. Want to learn more? https://lnkd.in/gs7V85WD #Queues #Messaging #AWS

13

Why not everything should be optimized all the time. A modern compiler (GCC, Clang, LLVM, etc.) does more than translate C/C++ code into assembly. It analyzes, rewrites, reorders, and aggressively exploits the underlying hardware. Example with a loop: out[i] = in[i] * coeff; With a non-optimized compiler: >> 1 multiplication, 1 load, 1 store per iteration With an optimized compiler (-O3): >> 16 multiplications in parallel per iteration on a modern CPU (SIMD), without the developer changing a single line of code. Why this matters: 1. Writing simple, readable, and predictable code often helps performance more than manual micro-optimizations. 2. Compilers are often better than we are at exploiting the hardware. 3. Understanding what compilers can optimize helps you know when it is urgent to do nothing.

6

4 Comments

Last Tuesday, Deliveroo's rider-switch Rails endpoint spiked to 4s latency and intermittent 503s. The flame graphs told the real story and became our speed playbook. Technical context: Flame graphs visualize time across call stacks using sampling profilers. In Rails, latency often stems from DB calls, serialization, or middleware. This makes root causes visible to newcomers. Recent tech context includes AI agents and RAG patterns (LangGraph, Claude 3.5, GPT-4o, Gemini 2.0) guiding performance analyses. KEY INSIGHTS: 🔍 Flame graphs reveal hot stacks and where wall time sits. ⚡ Latency on the hot path fell from 4s to ~800ms after fixes. ───────────────────────── 🔗 Read the full article: https://lnkd.in/ghcREzpZ 🎯 Practice interview questions: https://lnkd.in/gmy5drNw #performancetesting #cpuprofiling #memoryprofiling #flamegraphs #latency

1

1 Comment

Mem0 raised $24M last October to build "the memory layer for AI." AWS made them the exclusive memory provider for the Agent SDK. Their hardest benchmark category, "multi-hop reasoning across sessions," asks questions like: "Given my dietary preferences and the trip we discussed, what should I order for breakfast?" Their state of the art: 51.15% correct. What's shipped as AI memory today is fact extraction plus vector retrieval. An LLM pulls atomic facts from conversations, embeds them via some semantic model, and retrieves them by similarity at query time. Real engineering. Wrong substrate. Here's why. Ask any production embedding model how similar a kettle and a pan are. You get one number. Trained on billions of co-occurrences, the model clusters them as cookware. But that answer is not necessarily right for a significant fraction of users. For someone who once burned their hand on a kettle, the relevant similarity is "objects with handles that get dangerously hot." For someone's morning routine, kettle is closer to a coffee mug than to a pan. By shape and weight, they're not similar at all. The right similarity depends on the user, the moment, and what they're trying to do. Static embeddings encode similarity as a property of the inputs alone. Real memory encodes similarity as a function of the inputs AND the objective. Today's memory companies are stuck at the first because they inherited their similarity substrate from the pattern-matching paradigm above them. "Memory" built on pattern matching is just pattern matching with persistence. For those interested in going deeper: https://lnkd.in/gqjURExY https://lnkd.in/gaiQHChJ

14

Speed matters especially when you're indexing billions of vectors. 🚀💾 Meta engineers are enhancing FAISS, our open source similarity search library, with NVIDIA’s cuVS to deliver massive GPU acceleration. 💡 The result? Faster search, lower latency, and better scaling for high-dimensional data workloads powering AI, recommendation systems, and more. Explore how this integration is pushing the boundaries of real-time vector search. #Meta #FAISS #VectorSearch

2

The Netflix Compute team is giving three talks at #Kubecon: - Chao Zheng & Nicholas Parker on managing compute infrastructure with Kubernetes and dynamic capacity management - Artem Tkachuk & Jonathan Phillips on end-to-end tracing for Kubernetes with OpenTelemetry - Erikson (Chwan-Hao) Tung on modernizing Netflix's container runtime with containerd and OCI hooks Don’t miss these!

67

Day 22/60: Measuring the Infinite (Find Length of Loop) Targeted by: Amazon, Microsoft, Goldman Sachs, Qualcomm. Yesterday, I built the Linked List. Today, I had to break it or rather, deal with a list that’s broken into a cycle. The challenge: Find the exact number of nodes inside a loop. It’s one thing to know if a loop exists, but calculating its length requires a shift from simple detection to precise tracking. 1. The Scout and the Sprinter (Floyd’s Cycle-Finding) To find a loop, we use the classic Slow and Fast Pointer approach (The Tortoise and the Hare). Slow moves 1 step. Fast moves 2 steps. If they meet at the same node, a loop is confirmed. But once they meet, how do we know how big the "track" is? 2. Freezing the Moment Once the pointers meet, we stop the "Fast" pointer in its tracks. It acts as a stationary landmark. We then take the "Slow" pointer and let it continue its journey, one step at a time, counting each node it visits until it makes a full lap and hits the stationary "Fast" pointer again. 3. The Math of the Lap The logic is foolproof: Meet: Find any point inside the cycle. Count: Start a counter at 1. Traverse: Move the temp pointer to temp.next. Repeat: Keep moving and incrementing until temp == meetingNode. The final count is the exact number of nodes trapped in the cycle. 4. Why This is Elegant Space Complexity: O(1). We don't need a Hash Set to remember visited nodes. We use the structure of the list itself. Time Complexity: O(N). We traverse the list at most twice. This pattern is vital for Deadlock Detection in operating systems. When resources are pointing to each other in a cycle, the system needs to know the "size" of the deadlock to resolve it efficiently. 5. Final Reflection The "Length of Loop" problem is a reminder that even when you feel like you're going in circles, you can still measure your progress. You just need to find a fixed point to reference. Day 22/60 done. ✅ When life puts you in a loop, stop running for a second. Find your landmark, count the steps, and master the cycle. #60DaysOfCode 👨‍💻 #DataStructures #LinkedList #CodingLife #DSA Anchal Sharma Ikshit ..

6

IMO - blind use of LLMs most likely will cause chaos in the code repository. It is atmost important to follow a pattern of plan > review > build in the AI native software development journey. You should build robust Knowledge graph that becomes the basis of producing solid plan. This plan should be reviewed for architecture/design evolution, any deviation/proposal should be strongly vetted by senior ICs. Lastly models produce codes as per the steering docs. This approach will produce the code that is well understood by the software developers, have deterministic evolution to the design and is token efficient (thus low in cost).

6

this is super important because TPUs have definitely advantages for cost and performance over other accelators but were much more difficult to use. no longer the case. "vLLM TPU is now powered by tpu-inference, an expressive and powerful new hardware plugin unifying JAX and PyTorch under a single lowering path. It is not only faster than the previous generation of vLLM TPU, but also offers broader model coverage and feature support." https://lnkd.in/diDKzYbS

35

1 Comment

As AI platforms scale, the hardest problems are often not model accuracy or infrastructure, but alignment on fundamentals — how teams reason about ML, signals, and trade-offs. Last month, I shared a session on ML fundamentals and their implications for AI platforms with a broad engineering audience in Taiwan, engaging hundreds of engineers across multiple organizations. What stood out to me was not the attendance, but the quality of discussion — questions around system boundaries, evaluation thinking, validation trade-offs, and how ML concepts influence platform-level decisions. I’m grateful for the encouragement and recognition from leadership across regions, and even more so for the shared appetite to invest in common technical understanding. Scaling AI isn’t just about building smarter models — it’s about building shared mental models.

26

1 Comment

I didn’t stay and grow at Amazon for 5 years just because I could write good code. Plenty of smart engineers can do that every single day. Writing clean code was just the entry ticket. What truly set me apart was taking ownership, even when it wasn’t asked. If I was assigned a bug to fix, I could’ve stopped after the code review. But I would follow it to production, sit with support teams, talk to impacted customers, and make sure the bug was truly gone, and people felt the difference. If I was asked to build a new API, I could’ve stopped after passing the unit tests. But I’d stress-test it with real traffic, set up alerting, write docs, and hop on calls with downstream teams to make sure their integration was smooth. If I noticed on-call alerts going off every night, I could’ve shrugged and marked them as “infra issues.” But I dug into root causes, ran post-mortems, and sometimes pulled late nights re-architecting fragile parts, so the next person on call wouldn’t have to lose sleep. If I wrapped up my sprint tasks ahead of schedule, I could’ve clocked out early. But I’d sit with teammates struggling with blockers, offer code reviews, and even pick up “thankless” chores like improving internal tools or onboarding docs. Ownership isn’t about heroics or working 14-hour days. It’s about caring enough to see things through, to fix the root, not just the symptoms. To ask, “What else could go wrong?” and, “How can I make this easier for the next person?” That’s how you get noticed. That’s how you build trust. That’s how you grow. Don’t just be the engineer who writes code. Be the one who owns the outcome, end to end. That’s what really sets you apart.

296

10 Comments

Google, Meta and Microsoft are projected to spend over $500 billion on AI infrastructure by the end of 2026. Yet, thousands of CS grads with Leetcode Knight badges are sitting on the sidelines wondering why their resumes are being ghosted. If you are a Junior or Mid-level SDE planning for a switch, you need to realize that more value is shifting toward practical CS fundamentals. [1] The death of the isolated DSA grind DSA is not dead. It is just the baseline now. In 2026, companies judge your data structures through the lens of system thinking. 1) It is no longer "How do you implement a Hash Map?" 2) It is "How do you use consistent hashing to minimize data movement across 1,000 nodes?" 3) The applied system is the product. If you can't explain the trade-offs of your code in a multi-threaded, high-concurrency environment, your Knight badge won't save you. [2] High-Level Design (HLD) is the new filter Most engineers can write a function. Very few can design a pipeline. 1) Logic is shifting toward understanding distributed systems architecture like Kafka and Apache Flink. 2) You need to know how to handle "exactly-once" processing when the system scales to millions of events per second. 3) Practical knowledge of LLD (Low-Level Design) is where you prove you can write maintainable, extensible code that doesn't become technical debt in six months. [3] Bridging the gap from GPA to Applied AI There is a myth that you need a PhD in math to crack these $250k+ Forward Deployed Engineer roles. You don't. a) You need to understand how to deploy models at scale. b) You need to master the infrastructure that supports math. c) You need to apply the Pareto Principle: focus on the 20% of system design that handles 80% of the production load. The market may feel rough, but it is actually just becoming more specific. If you have built 5 projects and know 4 languages, stop adding a 6th project. Start breaking your 1st project. Simulate a database failure. Add a cache layer and handle the stale data. The engineer who knows how systems fail is worth 10x more than the engineer who only knows how a "perfect" algorithm works. Don't just learn to code for the interview. Learn to design for the infrastructure.

237

10 Comments

The latest Oracle AI Data Platform Workbench updates bring stronger governance, smoother promotion workflows, API extensibility, secure secret management, Scala support, enterprise connectors, and experiment tracking — all in one platform.

4

You are in a system design interview at Meta for an IC4 role. The interviewer leans in and asks: "If a photo is already on my phone, how can WhatsApp still delete it when the sender taps Delete for Everyone?" Here is how you should break it down 👇 A lot of people think “Delete for Everyone” means WhatsApp somehow reaches into your gallery and erases a file from your device. That is not really what is happening. Btw, if you’re preparing for system design/coding interviews, check out my free mock interview tool on Layrs. You can use it for free here: https://lnkd.in/gpCn7t2T The real answer depends on where the photo is stored, how the chat app references it, and what exactly the app is allowed to delete. 1. First clarify the product behavior Before jumping into architecture, say this clearly in the interview: There are two different cases here. a) The photo exists only inside the chat app’s managed storage b) The photo has already been saved to the user’s gallery / camera roll / filesystem This distinction is everything. If the photo is only inside WhatsApp’s controlled storage, the app can remove the local file and remove the message reference from the chat UI. If the photo was exported or auto-saved to the gallery, WhatsApp usually cannot reliably delete that copy, because that file is now outside the app’s normal ownership boundary. So “Delete for Everyone” is mostly about: - deleting the message record - deleting the media attachment reference - deleting any app-managed cached copy - syncing that deletion event to all participants It is not magic remote file deletion across the whole phone.  2. High-level idea The clean way to think about this is: A chat message is metadata plus optional media. For a photo message, the system usually has: - message_id - chat_id - sender_id - media_id - timestamp - status - optional encryption metadata And the actual photo itself may live in: - encrypted object storage on server for temporary delivery - app sandbox / local database / media cache on device - optionally the user’s gallery if exported When sender taps Delete for Everyone, the system does not chase every byte everywhere. It sends a deletion command tied to the message_id.  3. What happens when the photo is first sent Here is the send flow: 1. User picks a photo 2. App encrypts media and prepares message metadata 3. App uploads encrypted media blob or sends it through media pipeline 4. App sends message metadata referencing that media 5. Receiver downloads the encrypted media 6. Receiver stores it locally in app-managed storage 7. Chat UI shows the message by resolving message_id -> media_id -> local file Read rest of the post: https://lnkd.in/gpv-wGze

158

12 Comments