Gregor Beyerle

I constantly share open datasets for medical imaging. Then the nice folks here start sharing other open datasets in the comments. So I thought I keep compiling all of them in new posts. Here it goes! Amazing datasets for everyone in medical imaging (and more at the end of the post!) 1. SAROS dataset - This is a goldmine of 900 CT scans with detailed notes about different body parts and organs. It's perfect for anyone working on medical image analysis and it's totally free to use. Paper: https://lnkd.in/ekQnxpNb Dataset: https://lnkd.in/eZAUxRwf 2. TotalSegmentator from the NCI Imaging Data Commons - They've added features and segmentations for over 126,000 CT scans from the NLST collection. This is huge for AI research in medicine! Discussion: https://lnkd.in/e3-wGehc Dataset: https://lnkd.in/ebaE-dbC 3. Healthy whole body CT scans from the cancer imaging archive. It contains 30 CT scans of full body. Dataset: https://lnkd.in/egVyuuGH 4. Histopathology and spatial transcriptomics. Repo: https://lnkd.in/enWfgtBh Dataset: https://lnkd.in/ev__8tbU 5. BRACS: BReAst Carcinoma Subtyping Dataset download page (requires registration): https://lnkd.in/e4FZQ7TY 6. BCBM-RadioGenomics | MRI Dataset of Metastatic Breast Cancer to the Brain with Expert-reviewed Segmentations and Tumor-derived Radiomic Features. Dataset download page: https://lnkd.in/eaekwCrK 7. TOMPEI-CMMD Dataset | The TOMPEI-CMMD dataset adds segmentations (masses, calcifications, etc) and corrections to labeling errors in the original Chinese Mammography Database (CMMD) previously published on TCIA. Dataset download page: https://lnkd.in/ejKSwZe5 8. Stroke dataset From Teknofest AI Healthcare competition. Dataset download page: https://lnkd.in/ezCRDY3T 9. A bunch of datasets from Osirix: Cardiac and coronary study, CTA abdomen and lower extremities,... Datasets download page: https://lnkd.in/eJcAy5Dv 10. HaN-Seg dataset which includes CT and MR image pairs with manual segmentations of 30 organs. It’s well-suited for segmentation and image-to-image translation tasks. Dataset download page: https://lnkd.in/eYs8gsTQ 11. Automated Segmentation of Coronary Arteries. Dataset download page: https://lnkd.in/eY64_6qU Other datasets shared in the comments but I couldn't find their download links: https://lnkd.in/ec72r3ny https://lnkd.in/eQichS-Q If you know of other interesting open datasets for medical imaging then please share them below in the comments! With more free data available, we can build more tools and solutions to help doctors and save lives! More folks keep sharing new datasets but I have reached the limit of my linkedIn post's number of characters. To solve this, I've put these datasets on our website. Check out the link in the first comment below. ➡️Btw, we help companies build medical imaging web platforms powered by advanced DICOM viewing capabilities. Interested? DM me!

881

40 Kommentare

If you are working with Information Retrieval - RAG, Search, Recommendations You should know about 2-Tower architecture. 👇 ⚙️ 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗶𝘁? It’s the foundational setup where two separate towers (neural networks) are used to encode queries and documents/items independently into the same embedding space. ⚙️ 𝗪𝗵𝘆 𝗶𝘁 𝗺𝗮𝘁𝘁𝗲𝗿𝘀? It’s the backbone of multi-stage retrieval systems - enabling scalable, low-latency retrieval across millions of candidates based on vector similarities. ✅ Used heavily in production retrieval systems: Search engines, recommendation engines, ad platforms & now RAG - you name it! ✅ Shows up in many forms - ColBERT, DSSM, Bi-Encoder… but the core idea stays the same! ✅ A natural fit with VectorDBs and ANN search. ✅ You’ll find it in almost every scalable retrieval system in one form or another. Linking some good references in the comments.👇 Have you worked with this architecture? Curious to hear your experience. -- Found it insightful? ♻️ Repost 💬 Let’s discuss in the comments👇 👀 Follow me - Shantanu for AI - ML - MLOps content and Career tips!

94

21 Kommentare

The `cancensus` R package by Jens von Bergmann and myself has had 81k downloads since 2018 - it's the best way to access StatsCan census data for R users with great adoption in government and academia, but less so in industry. We've had a number of requests over the years to build a python version but never really got around to it. The good news is that agentic cli tools like Claude Code are perfect for the task. So here's `pycancensus`, a fully equivalent python port built for about $10 in tokens. This kind of problem is ideal for agentic CLI because the models is given clear examples and asked to work against a very well-defined reward function (the unit tests and example code in the source R package). I provided the R source as reference and built automated cross-validation tests that directly compared R vs Python outputs programmatically. This gave Claude objective feedback to iterate and self-correct function-by-function. I'm not actively working with Canadian Census data so while the package has been tested extensively and replicates my old cancensus workflows, it hasn't hasn't been battle-tested yet. If you're an existing cancensus user working in python or anybody that works with Canadian Census data in general, please check it out -- feedback and issues welcome on the GitHub repo. Check it out: https://lnkd.in/gdfCVC3A And if anyone is interested in the original R package, you can find it here: https://lnkd.in/g78t--Kv

70

4 Kommentare

{r5r} good news: now it's possible to adjust car speeds in {r5r} to simulate different scenarios of traffic congestion and road closures in R. This feature is available in the dev version and will be included in the next release. In case you'd like to test it, here's a quick demonstration with a reprex. We would love some feedback! https://lnkd.in/d-a7H5Vr #rstats #rspatial #r5rstats

408

18 Kommentare

Plotnine, my favorite Python plotting library, now has built in support for combining multiple plots together! This plot composition feature released in v0.15.0. You can now use the / and | operators to put plots side-by-side, or above each other. ggplot2 users might recognize this as similar to patchwork 😁. I'm such a big fan of Plotnine that I maintain its user guide, and learned a ton from its author Hassan while working on the plot composition page. Hope it's helpful to folks! https://lnkd.in/eCwYy9Zs

517

26 Kommentare

Offset pagination works OK for small datasets. For large datasets, use this instead: It's called 𝗞𝗲𝘆𝘀𝗲𝘁 𝗽𝗮𝗴𝗶𝗻𝗮𝘁𝗶𝗼𝗻 or 𝘀𝗲𝗲𝗸-𝗯𝗮𝘀𝗲𝗱 𝗽𝗮𝗴𝗶𝗻𝗮𝘁𝗶𝗼𝗻 and is a much more efficient way to paginate through a large dataset. The problem with Offset pagination (skip and take) is that the DB must still process the skipped entries, even if they aren’t used. But with Keyset pagination, you don’t need to skip any rows. You just remember the last key you fetched and use it to fetch the next rows. As long as your ID column is indexed, queries that use Keyset pagination will be very efficient, even for large datasets. What if you are using GUIDs for your IDs? Then use UUID Version 7, which are designed to be globally unique and sortable. Get my Free .NET Developer Roadmap 👇 https://lnkd.in/gmb6rQUR

1.249

81 Kommentare

🚀 Nextflow 25.04 was recently released, and it brings some amazing updates! If you're working with reproducible pipelines or large-scale data workflows, this version introduces features that really improve the development experience. A few highlights I found especially useful: 🔹 A new syntax parser integrated into the CLI   🔹 Auto-formatting and a stronger `lint` command   🔹 Native data lineage support for better provenance tracking   🔹 Cleaner and more intuitive syntax for declaring outputs These improvements make workflows more transparent, easier to maintain, and better suited for collaboration—especially for teams working in #bioinformatics or #datascience. 🔍 You can check out the full release notes here: https://hubs.la/Q03mPv4k0 #Nextflow #NextflowRelease #nfcore #ReproducibleResearch #WorkflowAutomation

34

Just over a year ago, I published the #GIST method and embedding models after exploring whether I could improve the retrieval of data mentions in sentences and make semantic search more accessible—especially in low-resource environments. Since then, these models have collectively been downloaded over #15million times. That number might not reflect unique users, but here’s what does matter: People I’ve never met are using GIST in ways I could never have imagined. It’s been incredible to see the reach—from semantic search engines to automated software refactoring. Just recently, I received a message from a researcher using GIST for code search. I paused and thought: this was a product of my imagination, now being used by someone I’ve never pitched it to. That made me genuinely happy. To me, this is the true value of open-source #AI. Putting a product of your imagination in the wild and letting others learn and build on top of it, unconstrained! I imagine a budding engineer in a low-income country choosing GIST-small-Embedding-v0 or GIST-all-MiniLM-L6-v2 because they run efficiently without a GPU. AI models for low-resource settings ensure no one is left behind. That’s the real win—not just downloads, but accessibility, impact, and possibility. Thank you to everyone who’s used, shared, or built on GIST! GISTEmbed collection: https://lnkd.in/ere3-aVM P.S. I am cooking something that truly demonstrates how these low-resource AI embedding models can be practically applied in something so meaningful, watch out! #AIforData #GISTEmbed #AI #DataforAI #OpenSource

61

2 Kommentare

🚨 Small number of poisoned samples can seriously compromise LLMs! I just read an eye-opening study by Anthropic, UK AISI, and The Alan Turing Institute titled “A Small Number of Samples Can Poison LLMs of Any Size.” Their core finding is astonishing: with as few as ~250 malicious documents, an attacker can successfully insert a “backdoor” into large language models (ranging from 600M to 13B parameters) — causing them to misbehave on specific trigger phrases — regardless of the total training data size or model scale. This marks a major shift from the earlier belief that an attacker must control a proportion of the dataset. Instead, this research shows that it’s the absolute number of poisoned samples that truly matters. The full technical paper — “Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples” — dives deeper into the experiments and implications. 📄 Read here: https://lnkd.in/gje8V-76 🎥 Watch a great explanation: https://lnkd.in/gMk68vj3 🙏 Thanks to Krish Naik for breaking it down so clearly!

34

2 Kommentare

The tl;dr of the brilliant blog post from Horace He/Thinking Machines (https://lnkd.in/gZAj7hZJ) is that the low level code that is heavily optimized to make full use of GPUs sometimes switches strategies of how to chunk up the computation based on how big the batch size is. And those different strategies can give different answers because adding a list of numbers together that are of vastly different scales is sensitive to the order of that list. E.g., (0.1 + 1e20) - 1e20 = 0. But 0.1 + (1e20 - 1e20) = 0.1. So the answer you get from your model provider will have a dependence on how many other queries are in the batch, which is outside of your control. There are ways to fix this, but it comes with not taking advantage of some of the GPU optimizations and therefore have a performance penalty.

41

5 Kommentare