Feature Selection Methods

🤖 Too many features, not enough signal? In high-dimensional data, variable selection isn’t just helpful—it’s essential. But picking the right method can feel like choosing a tool in the dark. Penalized regression? PCA? Knockoffs? Forward selection? LAR? In my latest post, I walk through: 🧲 Shrinkage methods like Lasso and Ridge 🎯 Geometric tricks like PCA 🎲 Randomization-based tools like Model-X Knockoffs 🚶 Greedy search techniques like Forward Selection and LAR https://lnkd.in/g5ZFAuaQ What unites them? Each method tries to answer the same core question: which variables actually matter? This isn’t a cookbook—it’s a field guide. I focus on intuition, use cases, and hidden pitfalls to help you match the method to your problem. Code included in both R and Python. Would love your thoughts: which method do you reach for first, and why? #DataScience #MachineLearning #VariableSelection #AdvancedStats #BlogPost #Lasso #CausalInference

Features are crucial in machine learning models and greatly impact a model's performance. However, as time passes, the number of features can grow exponentially, leading to a "feature explosion." To ensure maintainability and improve performance, it becomes necessary to perform feature selection. This blog, written by a data scientist from JustEatTakeaway.com (the parent company of Grubhub), presents a practical approach to utilizing a generic algorithm for feature selection. The team first highlights that standard feature selection methods, such as Shapley values, may encounter challenges when dealing with highly correlated features. In such cases, a generic algorithm offers a more systematic and rigorous approach to feature reduction. The team then introduces the key concept of the generic algorithm, which mimics the process of natural selection. In this algorithm, the fittest individuals in a population survive and reproduce, passing on their genes (in this case, a list of features) to the next generation. Survival means that one feature set is selected for reproduction, while reproduction involves randomly exchanging non-overlapping features between two feature sets to create new "individuals" for the next round of evaluation. After several cycles of reproduction, the best "individual" can be identified as the optimal feature set for selection. What's great about this blog is that the author not only explains the concept but also provides concrete code to assist readers in implementing it. It's a valuable toolkit for anyone interested in this topic. #datascience #machinelearning #featureselection #generic #algorithms – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gj6aPBBY    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gT6btHQg

Suppose you have 10,000 features, how would you perform feature selection? Here's my recipe you can steal for model building👇 Newbies typically use a univariate approach (also called filtering) to choose the top K signals for the model. But, often, this is quite naive. The univariate approach leads to collinearity among top signals. Another approach is using variable importance from Random Forest, but that still doesn't address collinearity. Place 2 signals that are collinear, the variable importance of those two signals will be deflated than if it was just one signal. A more ideal solution is this combination - (1) systematize, (2) cluster, and (3) test multiple feature sets, not just one. (1) Systematize - Start by removing signals in poor condition - no-variance, high missingness, deprecated. From 10K, let's say that this shrinks signals to 8K. Then, calculate your variable importance using a reliable statistical measure - I often use mutual information, or Spearman with clipped+scaled for both X and Y - I want to minimize the impact of outliers (often seen in financial data) on importance measure. (2) Cluster - This part helps address collinearity. Create a correlation matrix among the signals, using MI or Spearman. If you have 8K signals, you would have 8K x 8K matrix. Then, cluster along the rows using hierarchical clustering. The idea is that two signals that are similar (or collinear) would be lumped together. Then, choose the highest importance per each cluster from step 1. This helps reduce variable importance. (3) Test Feature Sets Don't just test 1 particular feature set, and X number of models with parameters. Rather, think of creating multiple feature sets to test - think of it like a parameter in hyper-parameter tuning. Ultimately, this helps remove heuristics, and helps you find the best combination of algorithm + parameter + features Oftentimes, from step 1 and 2 - I tinker with different importance measure, and parameters of clustering algo, which results in 15 to 20 different feature combination sets. There are, of course, other feature selection part I also test as well including L1 model, PCA, ICA, which further adds feature sets to 30 to 40. 👉 So, that's my usual personal recipe on feature selection. What about you? What would you do in this case? Drop a comment 👇 Now go upskill in data on👇 📕 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗣𝗿𝗲𝗽 𝗖𝗼𝘂𝗿𝘀𝗲𝘀: https://lnkd.in/gzgB-dHT 📘 𝗝𝗼𝗶𝗻 𝗗𝗦 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/eiA5Ntdp 📙 𝗝𝗼𝗶𝗻 𝗠𝗟𝗘 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/e5VaYyTz 📗 𝗟𝗲𝗮𝗿𝗻 𝗠𝗼𝗱𝗲𝗹 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻: datascienceschool.com 👉 Smash 👍 and follow Daniel Lee for data & AI

Feature Importance vs. Feature Selection: How are they related? In machine learning, feature selection and feature importance play pivotal roles in constructing accurate and efficient predictive models. These concepts are essential for optimizing model performance, reducing dimensionality, enhancing interpretability, and improving generalization. 👉 Feature importance refers to the degree of influence each feature has on the output or prediction made by a classifier or regression model. It quantifies the relevance or contribution of each feature to the predictive power of the algorithm. 👉 Feature selection, on the other hand, is the process of choosing a subset of features from the original set of input features. By reducing the number of inputs to a machine learning model, we reduce training and scoring times, which are essential when putting models into production. ❓ So, how are these terms related? 👉 Feature importance plays a crucial role in guiding the feature selection process by providing insights into which features have the greatest influence on the target variable. 👉 Many feature selection algorithms involve deriving the feature's importance, ranking the features based on their importance, and finally selecting the top-ranking features. ❓ How can we derive feature importance? ▶️ Statistical tests assign importance through their p-values. ▶️ The variance could be used as a rudimentary importance metric. ▶️ Linear and logistic regression assign importance through their coefficients. ▶️ Decision tree based models assign importance based on the number of times a feature is used to make a split across the trees and the reduction in impurity. ▶️ For models that do not assign importance natively, we can infer feature importance by randomly shuffling the variables and obtaining a measure of the performance degradation. ▶️ Training single feature classifiers or regressors and then obtaining a performance metric like the ROC-AUC or the mean squared error, is an alternative way of inferring how important a feature is to predict a certain outcome. You can find more about feature selection in our course (https://buff.ly/3TTf9mS ) and book (https://buff.ly/3BallhE ). More on feature importance is coming soon in my new course (https://buff.ly/45qukK3 ). For a discussion on feature importance vs. feature selection, check out this article (https://buff.ly/3BS7nBS ). And now: how do you understand which features contribute the most toward your model outputs? Let's discuss in the comments 👇

*** Variable Selection in Regression: A Critical Component of Modeling *** Selecting the most appropriate subset of variables to include in a regression model is complex and often challenging. Statisticians employ a variety of variable selection techniques, each with its own strengths and weaknesses. It’s essential to recognize that not all methods produce effective outcomes. Poor variable selection can severely impact the model's performance, potentially compromising solid statistical theory and contributing to an inaccurate theoretical framework. In this overview, I delve into five widely used methods for variable selection, each serving a distinct purpose in the modeling process: 1. **Enter Method**: This approach simultaneously includes all independent variables in the model from the outset. It is straightforward but can lead to overfitting if many irrelevant variables are included. 2. **Remove Method**: Similar to the Enter Method, this technique starts with all independent variables. However, it focuses on eliminating those that do not significantly contribute to the model, thereby refining the variable pool based on their statistical significance. 3. **Stepwise Method**: This method is more dynamic, as it introduces variables incrementally into the model. At each step, the model evaluates whether to keep or discard a variable based on its significance levels, allowing for a more tailored selection of predictors. 4. **Backward Method**: This technique commences with all available variables in the model and systematically removes the least significant ones. This allows for gradually pruning redundant or irrelevant variables, refining the model iteratively. 5. **Forward Method**: Unlike the Backward Method, the Forward Method begins with a basic model with only a constant term. From there, it sequentially adds variables, each chosen based on its potential to improve the model fit, allowing for a focused and stepwise enhancement of the predictive capabilities. This comprehensive exploration highlights the fundamental principles of each method, emphasizing the importance of careful variable selection in constructing reliable regression models while preserving the integrity of the original concepts. --- B. Noted