Linux Kernel's Secret to Fast Reads: RCU

********. Slurm vs. PBS .********* The High-Performance Computing Showdown Choosing the right Workload Manager (WLM) is critical for any serious High-Performance Computing (HPC) environment. For years, PBS (Portable Batch System) and its derivatives like Torque and PBS Pro have been staples. But now, Slurm (Simple Linux Utility for Resource Management) is widely adopted and often the default for new clusters. Why the Shift to Slurm? While both manage resources, schedule jobs, and enforce policies, there are notable differences: 1. Open Source & Community: Slurm is purely open-source with a highly active community, which appeals to many research and academic institutions. PBS Pro, while robust and commercially supported, is proprietary. 2. Syntax & Workflow: Slurm often streamlines the process. For example, batch jobs start in the submission directory automatically, unlike older PBS implementations where you might need cd $PBS_O_WORKDIR. Also, Slurm tends to leverage the kernel's control groups (cgroups) for resource limits more directly. 3. Resource Specification: There are differences in how resources are requested. PBS often focuses on nodes and processors per node (ppn), while Slurm emphasizes tasks and CPUs per task, which can offer more flexibility. The Migration Challenge For teams moving from a legacy PBS/Torque system to Slurm, the real work lies in converting years of submission scripts and getting users accustomed to new commands (e.g., sbatch instead of qsub, squeue instead of qstat). Question for the HPC community: If your cluster has recently migrated from PBS to Slurm, what was the single biggest lesson learned or the most useful conversion script/tool you found? #HPC #HighPerformanceComputing #Slurm #PBS #SystemAdministration #WorkloadManagement

Do you know when you press a key, open a browser, or run a command who actually makes it all work behind the scenes? It’s not your application. It’s not even the operating system you see. It’s the Kernel. The Operating System is the mind of your computer, the kernel is the heart. It’s the first part of the OS that wakes up when your system boots, and it never stops working until you shut it down. Its job is simple to describe yet incredibly vital: To connect software to hardware. Every program you run eventually needs to talk to your machine’s CPU, memory, or disk. But user programs can’t just reach into hardware directly. So they make a request instead, called a system call. The kernel receives it, decides how to handle it, and safely executes it on the hardware’s behalf. For example, when you type "cat" in Linux, the shell doesn’t read files itself, it asks the kernel to do it. The kernel talks to your storage, retrieves the data, and hands it back in a clean, safe way. This separation ensures stability, security, and fairness among programs. In short, the kernel manages everything that keeps your system alive, the CPU’s time, memory space, file access, and device communication. It allows dozens of apps to run side by side without crashing into each other. Think of it as a translator between human intent and machine language. Without it, your computer would still be a silent box of circuits waiting for someone to flip switches manually. Every tap, click, or command passes through this middle layer -> the kernel, turning human actions into machine execution. #kernel #operatingsystem #linux #systemcalls #techvisualization #softwareengineering #computingfundamentals #learntech #developereducation #programmingconcepts #computerarchitecture #RushikeshShelar

🚨 Kernel Panic Survival Guide: Common Bugs & How to Fix Them 💻🔥 If you’ve ever faced a “Kernel Panic”, you know that heart-stopping moment when the system freezes and spits out cryptic logs 😅. But each panic tells a story if you know how to read it. Here’s a quick survival guide 🧠👇 🔍 1. Null Pointer Dereference 💬 Log looks like: Unable to handle kernel NULL pointer dereference at 0000000000000008 ✅ Fix: Always validate pointers before dereferencing (if (!ptr) return -EINVAL;). ⚙️ 2. Stack Overflow 💬 Log looks like: kernel stack overflow (double-fault) on CPU#0, current process: kworker/0:1 ✅ Fix: Avoid deep recursion or large local arrays; use heap (kmalloc) or static buffers instead. 🧩 3. Use-After-Free / Double Free 💬 Log looks like: BUG: unable to handle page fault for address 0xdead0000 ✅ Fix: Set pointers to NULL after kfree(); enable KASAN to detect invalid memory access. 🔐 4. Race Condition 💬 Log looks like: BUG: spinlock already unlocked (state=0) in process xyz ✅ Fix: Use proper synchronization (spin_lock, mutex, atomic_t), and review shared data handling. ⚡ 5. Invalid Access in ISR (Interrupt Context) 💬 Log looks like: BUG: sleeping function called from invalid context at kernel/sched/core.c:xxx ✅ Fix: Never call blocking functions inside ISR; use workqueues or tasklets for deferred work. 🧾 Bonus Tip: Enable kdump or pstore to capture crash dumps — your kernel’s black box 📦. It helps trace the exact line that triggered the panic. 💬 Kernel panics aren’t just crashes they’re feedback from the system. Each one helps you become a better kernel engineer. #Linux #KernelPanic #KernelDebugging #LinuxInternals #EmbeddedSystems #CProgramming #Debugging #KernelDevelopment

🧠 Kernel Source Tree Navigation – Visual Breakdown This infographic explains the Linux Kernel Source Tree located under /usr/src/linux/, showing how the kernel’s source code is structured internally for developers and system engineers. At the top, the root directory /usr/src/linux/ forms the base of the Linux kernel, from which multiple essential subdirectories branch out — each handling a specific part of the kernel’s functionality. 📂 Directory Overview 🔹 arch/ – Architecture-specific code → Handles CPU-specific implementations like x86, ARM, ARM64 🔹 drivers/ – Device driver source codes → Includes character, USB, and network drivers 🔹 kernel/ – Core kernel code → Manages processes, scheduling, and system calls 🔹 mm/ – Memory management subsystem → Allocates pages, handles virtual memory & page faults 🔹 fs/ – File system layer → Supports ext4, nfs, fuse and Virtual File System (VFS) 🔹 net/ – Networking stack → Implements TCP/IP, UDP, and socket communication 🔹 init/ – Boot-up initialization routines 🔹 scripts/ – Build helper scripts used during kernel compilation 🔹 Documentation/ – Developer documentation & kernel APIs 🧩 Why It Matters Understanding this structure is essential for: Kernel developers Device driver engineers System programmers Anyone debugging or building custom kernel modules It gives you a clear map of where every major subsystem lives — making kernel exploration easier and more organized. ✨ #LinuxKernel #DeviceDrivers #EmbeddedLinux #KernelDevelopment #LinuxInternals #OpenSource #EmbeddedSystems #SystemProgramming #OperatingSystems #FirmwareDevelopment #IoTDevelopers #CProgramming #LinuxForDevelopers #EmbeddedEngineer #TechEducation

Michael Hennecke, Hewlett Packard Enterprise, will be presenting “DAOS on 400 Gbps Fabric," on November 17th at the Parallel Data Systems Workshop (PDSW), adjacent to SC’25. Session Overview: The talk will present early performance numbers of using DAOS for a checkpoint and restart mechanism in a classic HPC application: PALM - an large-eddy simulation code written in Fortran. Different methods are supported: Fortran IO with a file-per-process scheme and two MPI IO based methods that use a single shared file and where one method can aggregate IO in a single process per node. The presentation will reveal early performance numbers of the Fortran IO and the two MPI IO variants in PALM using 9.216 MPI processes and both MPICHs native DAOS support as well as mounted DAOS containers in the Linux filesystem. The DAOS system used in the study provides approx. 0.5 PB of storage using Optane memory technology distributed across 19 storage nodes and is connected to the HPC system via an Omni-Path interconnect. Finally, the numbers are compared to Lustre and GPFS filesystems in production. Learn more and plan to attend | https://lnkd.in/g5Q888P8 #DAOS #DAOSIO #HPC #SC25

The Unsung Hero of Modern Computing: How Linux SMP Scales Performance 📈 Dive into the core of high-performance computing, and you'll find the Linux Symmetric Multiprocessing (SMP) architecture. This isn't just about throwing more cores at a problem; it's a sophisticated design challenge within the kernel itself. What is Linux SMP? It's the architecture where two or more identical CPU cores share the same memory and are controlled by a single instance of the Linux OS. In an SMP system, every core is a peer, capable of executing any task—including kernel code. The Engineering Challenge: The true complexity lies in ensuring efficiency and data integrity: Dynamic Load Balancing: The Completely Fair Scheduler (CFS) constantly balances execution across all cores for optimal utilization. Synchronization: Mechanisms like spinlocks and atomic operations prevent race conditions when multiple cores access shared kernel data structures. Cache Coherence: The kernel and hardware must work together (e.g., via the MESI protocol) to ensure consistent data views across all core caches. Understanding these kernel-level concepts is fundamental for anyone working on high-throughput servers, data centers, or robust embedded Linux systems. The efficiency of your infrastructure often boils down to how well the kernel manages its SMP environment. What kernel-level challenge do you find most fascinating? Share your insights! #LinuxKernel #SymmetricMultiprocessing #KernelDevelopment #HighPerformanceComputing #EmbeddedSystems #OperatingSystems #TechTalk

Most monitoring tools tell you what broke. None tell you why. I got tired of waking up at 3am to alerts that said "CPU at 95%" and spending the next hour SSH-ing into servers, running top, grepping logs. The problem isn't the alert. It's that it has zero context. So I built Linnix. eBPF watches every process fork, exec, and exit in the kernel. When it sees a pattern - fork storm, CPU spin, runaway process - it doesn't just alert you. It explains what happened and what to do about it. Not "CPU high." "Fork storm in bash pid 3921. Spawned 240 children in 5 seconds. Likely a runaway cron job. Kill pid 3921. Add rate limiting. Check /etc/cron.d/" An LLM reasons over the telemetry. You can use OpenAI, local llama.cpp, or any OpenAI-compatible API. <1% CPU overhead. Zero instrumentation. Works with any process. Built in Rust. Open source. Apache 2.0. I built this nights and weekends because I wanted monitoring that treats me like an adult. https://lnkd.in/dEXpy2p6 Demo: https://lnkd.in/dA_f9KcD What would you want your monitoring to explain? #DevOps #SRE #OpenSource #eBPF #Linux #Rust

Built My First Linux Kernel USB Driver in C! After diving deep into kernel development for the first time, I successfully created a minimal USB driver that monitors SanDisk devices. Key Features: ✅ Detects when USB devices are connected/disconnected ✅ Retrieves device information (Vendor ID, Product ID, Speed, Serial Number) ✅ Real-time kernel logging ✅ Proper memory management & cleanup Repo Link https://lnkd.in/ddrq6zTq Technical Stack: - Language: C - Target: Linux Kernel Modules - APIs: USB Core Driver Framework, URB handling - Tools: GCC, Make, dmesg The Challenge: As a beginner in kernel programming, this was incredibly challenging. Resources are scarce—mostly fragmented documentation and complex source code. I had to piece together information from kernel docs, header files, and AI assistance. This project gave me deep insights into how operating systems interact with hardware at the lowest level. Check out the code and demo video! 👇

Testing my console-based MCAP player and visualization plugin for remote control. What’s not visible in the video: • The broker (the system can also run without it, but it simplifies coordination under light load) • The node converting Velodyne HDL-32 point clouds The console player supports a wide set of commands - and you can see its output in the terminal - but controlling it from the visualization UI is much more convenient. In this demo, two V4L webcam streams and a serialized point cloud (recorded in MCAP) are being replayed. Playback speed can be adjusted, and the file position is controlled with a timeline slider. YODPS (Yet One Distributed Processing System) is a modular pub/sub framework, lightweight cross-platform distributed high-performance open source alternative to ROS/LCM/YARP/ADTF. https://lnkd.in/eNQKjaQQ #yodps #Cplusplus #cpp #rbfx #Urho3D #ZeroMQ #zmq #ZeroMQ #LCM #YARP #adtf #LiDAR #MCAP #PCAP #Realtime #velodyne #mobileye #msgpack #httplib #openstreetmap #osm #gps #gis #dbc_parser_cpp #xml #tinyxml #adtf #ros #ros2 #plotjuggler #archlinux #buildroot #embedded #rpi #android