Data Imputation Techniques

#Missing_data is a critical issue in statistical analysis and machine learning (#ML), often leading to #bias and reduced reliability if not properly addressed. This review paper categorizes imputation methods into deterministic, probabilistic, and machine learning approaches. Traditional techniques like mean/median imputation and regression are compared with advanced methods such as multiple imputation, expectation-maximization, and deep learning (including autoencoders and GANs). It is important to understand missing data mechanisms—MCAR, MAR, and MNAR—as these influence method selection. Key considerations include handling various data types, computational efficiency, and preserving data distributions. Evaluation metrics (e.g., MSE, RMSE, F1-score) and challenges like bias, complexity, and the need for domain knowledge are discussed. Careful method selection and evaluation are essential for robust research, with future directions including federated learning and ethical considerations. Citation: Alwateer, M., Atlam, E.-S., Abd El-Raouf, M.M., Ghoneim, O.A., & Gad, I. (2024). Missing Data Imputation: A Comprehensive Review. Journal of Computer and Communications, 12, 53-75. https://lnkd.in/eZwnFvsQ

Missing data is one of those things that every UX or behavioral researcher has to deal with, but few of us are taught how to handle it well. We often default to deleting rows or filling in averages, just to get the analysis moving. But the method you choose to handle missing data isn’t just a technical decision. It can make or break the validity of your insights. Let’s say your dataset is mostly clean, with just a few missing points scattered randomly. In that case, listwise deletion or simple mean imputation might do the trick. But the moment your missingness is related to user characteristics- say, older users skipping certain questions - you’ve entered trickier territory, where those basic methods fall apart. For survey-style data, especially when variables relate to each other (like age, satisfaction, and ease of use), MICE - Multiple Imputation by Chained Equations - remains a rock-solid choice. It doesn’t just guess once; it builds multiple complete datasets based on patterns in your data, then combines the results so that your uncertainty is preserved. This is great for drawing valid conclusions without pretending the missingness never happened. If you're looking for something faster or easier to implement, K-Nearest Neighbors can be a good option. It finds similar users and borrows their values to fill in gaps. It’s intuitive and can work surprisingly well - especially with smaller datasets. Random Forest imputation is another great option when your dataset has both numerical and categorical data. It uses decision trees to learn patterns and works well even when your data is messy or non-linear. Once you start working with larger or more complex datasets - think logs, sensor data, or time-series interactions - AI-based methods start to shine. Autoencoders are neural networks that learn how to compress and reconstruct your data. If trained correctly, they can guess missing values in a way that reflects deep structure in your dataset. GANs, like GAIN, go a step further by learning to create data that’s indistinguishable from the real thing. They’re especially useful when missing data isn’t random and follows hidden patterns. Recently, transformer-based models like SAITS and ReMasker have pushed things even further. These models use attention mechanisms to find long-range patterns across time or across features. They’re particularly useful in behavioral and biometric research where data is sequential, noisy, and full of gaps. Still, none of these methods are perfect in isolation. That’s why ensemble methods are becoming more common - combining multiple imputation models into one workflow that balances their strengths. This is especially helpful in mixed-method UX studies where you might have numerical data, categorical survey responses, and even behavioral logs in the same dataset.

When imputing data with means (medians, modes, etc) everything that can be spoiled and go wrong is spoiled and goes wrong: 👎 it affects the variance (lowers it) for no reason 👎 it assumes symmetric unimodal distribution of data which automatically means that data were generated by an additive process - often we don't know this 👎 may be sensitive to outliers (mean) or frequencies (median) or multiple local modes (mode), so the imputations may bias it upward (right skewness) or downward (left) 👎in case of multidimensional data (more than single variables) it entirely ignores the dependency between them. In other words, it silently assumes ➡️ the MCAR pattern, just like the complete-case analysis. 👎 Since no conditioning takes place, this will be visible especially in presence categorical data, which levels render sub-groups. In all sub-groups the same mean will be used, which silently assumes that all group means are the same. WHAT!? 👎 ...plus in such case, the entire conditional distributions are disregarded - it may be not aligned with the target analysis, e.g. via medians or other quantiles. 👎 uncongeniality, which occurs if the imputation is specified as more restrictive than the complete-data model (which happens there) In clinical trials, one of the most suitable methods, in daily use, are: ✔️ FCS (fully conditional specification), a principle represented by MICE (multiple imputation via chained equations), allowing one to use the context taken from multiple other endpoints through chains of equations defined by the researchers. Applicable to any kind of data: numerical continuous and discrete, ordinal, nominal and binary, survival (censored) - via dozens of available different univariate "imputers" employed to impute subsequent endpoints. One great example: the Midas Touch PMM (predictive mean matching). 👍 MICE handles also interactions! Some of the univariate "imputers" handle repeated or clustered observations as well. 👉 Enables sensitivity analyses, e.g. against MNAR! ✔️ RBMI (reference based multiple imputation) dedicated for certain intercurrent event policies in the estimand framework. Enables sensitivity analyses, like tipping point analysis for certain patterns (pattern mixture models) in exploring the MNAR scenarios. In R, it's implemented so far only for Gaussian data, but binary responses could be handled via MICE and the latent multivariate normal model. ✔️ (M) JM ([multilevel] joint modeling). A principle of multivariate imputation for data of single kind (e.g. numeric), imputes data from a joint distribution of the endpoints. There's a possibility of handling binary and categorical data through latent normal variable. ⚠️ with GREAT CARE, methods like LOCF (last observation carried forward) and NOCB, still allowed by the FDA in cases where this gives conservativeness. So LOCF and related methods are definitely not dead (and openly mentioned in the regulatory guidelines), BUT mostly discouraged. 

Ever struggled with missing data during prediction? Elena Albu has developed an adaptation of the missForest algorithm for realtime missing-data imputation during prediction—an area where many traditional methods fall short. She benchmarked the approach against a wide range of classic and modern imputers, including: • mean/mode imputation • linear regression imputation • MICE • k-nearest neighbours • bagging • miceRanger • IterativeImputer The extensive evaluation covered simulated datasets with different missingness mechanisms and eight real-world datasets using various statistical and tree-based modeling techniques. Even better—she’s released the code for you to experiment and compare imputation methods on your own data. #DataScience #MachineLearning #MissingData #Imputation #Rstats #Python #AI https://lnkd.in/etWJWvgA

df.dropna() Easy to use. Easy to misuse. One of the fastest ways to quietly bias your analysis. It does not break your code. It breaks your conclusions. There is a common rule: If less than 5% of your data is missing, just drop it. That only works if the data is Missing Completely At Random (MCAR). In real-world datasets? It rarely is. When high earners skip salary questions, you are not losing random rows. You are removing a specific group of people. Your model still trains. Your metrics still look fine. But your insights are now biased. And you may never notice. This shows up everywhere: 📌 Dissatisfied customers skip surveys 📌 Sick patients miss appointments 📌 Highly engaged users complete more fields Missing data is not noise. It is signal. Before you drop or fill anything, ask: 📌 Why is this data missing? 📌 How much is missing? 📌 How critical is accuracy here? Then choose your approach: 📌 Exploration? Mean or median imputation 📌 Production? KNN or model-based imputation 📌 High-stakes decisions? MICE 📌 Time series? Forward or backward fill 📌 Categorical? Sometimes missing is its own category These are starting points. Your use case makes the final call. The biggest mistake is treating missing data as a cleaning problem. Sometimes it is the most important insight in your dataset. Think before you drop. Think before you fill. What is the worst data quality issue you have seen in a real project? Drop it below. PS: I am Anthonette, Lead Data Scientist. I share insights on data, careers, and the truths nobody tells you about breaking into this field. Follow along if that is useful.

Every blank cell in your dataset is telling you something. Most researchers aren't listening. Poor methods are the most common reason for desk rejection. And failure to handle missing data well is a common problem. 3-step framework for handling missing data the right way: 1️⃣ CLASSIFY the missingness The why behind the gap matters more than the gap itself. ↳ MCAR: Missingness is unrelated to any data. A random machine failure causes BP readings to go unrecorded. ↳ MAR: Missingness depends on observed variables. LDL values missing because fast-track clinics didn't draw labs. ↳ MNAR: Missingness depends on the unobserved values themselves. Patients with severe hypertension drop out of follow-up. Get this wrong and everything downstream is biased. 2️⃣ Choose the right STRATEGY First question: Does missing mean zero? In EHR studies, if a diagnosis code for heart failure or CKD is absent, that may or may not mean "no disease." It might just mean it wasn't recorded. → Always spot-check a sample of charts before assuming missing = 0. Then pick your method: ↳ Listwise deletion: Only valid if MCAR. Otherwise you're introducing selection bias. ↳ Mean/median imputation: Simple but distorts variance. Avoid unless missingness is minimal. ↳ Multiple imputation (MICE): Preferred for MAR. Creates several complete datasets, pools results, maintains uncertainty. 3️⃣ Protect your CONCLUSIONS ↳ Sensitivity analysis: Test how results change across different imputation approaches. ↳ Transparency: Document what you did, why, and how it might have influenced results. ↳ Prevention beats correction: Use structured fields. Define variables clearly. Train data collectors on what "zero" actually means. Blindly filling in blanks is a shortcut to misleading results. Think clinically. Think critically. And when in doubt: check the chart. 💬 What's the trickiest missing data scenario you've dealt with? — If this resonated, repost to your network ♻️ and follow Paras Karmacharya, MD MS for more. — 📌 P.S. Join my next live masterclass FREE: Academic Writing with AI on May 30, 07:00 am CDT. Register here: https://lnkd.in/gy24eGuw

Regression imputation is a powerful method for handling missing data by predicting missing values based on relationships with other variables. Two widely used approaches, deterministic regression imputation and stochastic regression imputation, differ in how they handle variability in the imputed values. Understanding these differences is essential for choosing the appropriate method for your analysis. 🔹 Deterministic regression imputation replaces missing values with the predicted values from a regression model. It is simple and easy to implement, ensuring consistency with the regression model's predictions. However, it fails to account for the natural variability in the data, as all imputed values lie directly on the regression line. This lack of variability can distort relationships in the data, reducing variability and underestimating standard errors. 🔹 Stochastic regression imputation builds on deterministic regression by adding random noise (from the regression model's residuals) to the predicted values. This approach preserves the natural variability in the data, introducing randomness that better reflects the true distribution of the variable with missing values. While it offers more realistic imputation results, it is slightly more complex to implement than deterministic regression. A Visual Comparison The attached image highlights the differences between deterministic and stochastic regression imputation. In the left panel, deterministic regression imputation is depicted, where all imputed values (red points) lie directly on the regression line. This lack of variability can result in unrealistic patterns in the data. In contrast, the right panel illustrates stochastic regression imputation, where imputed values include added noise from the regression model’s residuals. This approach better captures the natural variability observed in the data (black points), resulting in a more realistic representation. Which Method Should You Use? If your analysis requires preserving variability and maintaining accurate relationships between variables, stochastic regression imputation is generally the better choice. By introducing randomness that reflects the natural variability in the data, stochastic regression produces more realistic results and avoids the potential biases introduced by deterministic regression. For a step-by-step explanation of both methods, check out my full tutorial here: https://lnkd.in/e7zvYZe I’m thrilled to announce my online workshop on Missing Data Imputation in R, beginning February 20, limited to 15 participants. Learn more by visiting this link: https://lnkd.in/eXApSEBw #data #analysisskills #dataanalytic #database

Handling Missing Data? Here's a Guide! Dealing with missing data can be a challenging task, but it's crucial for reliable analysis. This document provides a step-by-step example of multiple imputation using the mice package in R. 💡 What you’ll find inside: How to simulate data and inspect missing data patterns Defining imputation models and generating imputed datasets Visualization and validation of imputed values Tips for pooling results and performing sensitivity analysis 📖 Whether you're new to data imputation or looking to refine your skills, this resource is a great starting point. 🔗 MICE package https://lnkd.in/eDqSXD6i #RStats #DataScience #Imputation #MissingData

To account for intercurrent event information when handling missing PRO data, the missing data model should be separated from the analysis model as ICEs may occur later than the missed measurement. Therefore, MI and IPW methods are preferred over direct likelihood-based methods to handle missing data. The choice between MI and IPW depends on whether one believes to have an appropriate model for the missing outcomes themselves or for the probability of missingness, respectively. In an example dataset, the authors have illustrated how information about intercurrent events can be incorporated in the multiple imputation of a continuous, longitudinal PRO. Further research is required to formally compare imputation methods that include ICE information. Whichever method is applied, it is important that information about the timing and occurrence of ICEs is taken into account when handling missing PRO data. #Patientreportedoutcomes #diseaseprogression #researchmethodology #epidemiology #datascience #dataquality #biostatistics #estimand #missingdata #multipleimputation #dataanalytics #medicalresearch #data #realworlddata #rwd #clinicaltrials International Society for Pharmacoepidemiology