feature engineering
feature engineering on Beyond Market Intelligence: a running collection of 6 stories we have gathered and hand-picked because they are worth your time. Every post here touches on feature engineering in some way — the news, the analysis, the deep dives, and the occasional surprise find. Acme AI is the next-generation, AI-powered spreadsheet platform built to replace Excel and redefine how analysts, data scientists, and enterprise teams work with data. New stories are added to this page as we find them, so check back if you want to keep up with what is happening around feature engineering, or subscribe to the RSS feed to get them as soon as they are published. Browse the collection below, or head back to the homepage to see everything Beyond Market Intelligence is covering right now.
Swiggy Uses 350+ Features and Multi-Task MLP to Predict Customer Lifetime Value
Swiggy has developed an innovative, in-house predictive lifetime value (pLTV) model, leveraging over 350 pre-order features and a multi-tasking MLP architecture for both its Food and Instamart services. This approach, incorporating order count as an auxiliary task, significantly streamlined the model—reducing parameters by 63% while simultaneously boosting predictive accuracy. Now, Swiggy utilizes this refined pLTV signal with Google Target ROAS bidding, optimizing customer acquisition strategies with data-driven precision. For further exploration of related methodologies, consider our article, "Are HMMs still used for unsupervised tasks? [D]".
Quantifying User Behavior Patterns to Build Better Predictive Features
Simply knowing a user’s clicks—like a 35-year-old male in Seattle clicking 12 times last month—reveals little about their intent. Quantifying user behavior patterns, however, unlocks powerful predictive capabilities. We move beyond superficial metrics to analyze sequences, durations, and interactions, building features that genuinely anticipate user needs. This approach transforms raw data into actionable insights, driving more effective product development and personalized experiences. For a deeper dive into understanding data assumptions, explore “What We Miss About Missing Values.”
What We Miss About Missing Values
Missing values are a ubiquitous challenge in data science, yet their implications often go unexamined. "What We Miss About Missing Values" explores the hidden assumptions embedded within the data we *do* observe—recognizing that what's absent can be just as informative as what's present. This post delves into the biases introduced by missingness and offers a framework for more thoughtful analysis. For a related perspective on navigating complexity in data systems, see "Why RAG Complexity Should Be Earned."
py-evoFE: Automated Evolutionary Feature Engineering for Tabular ML in Python (Genetic Algorithms + Scikit-Learn + Polars) [P]
Announcing py-evoFE (v0.3.0), an open-source Python library designed to automate and optimize feature engineering for tabular machine learning. Leveraging genetic algorithms alongside Scikit-Learn and Polars, py-evoFE intelligently discovers and combines feature transformations, addressing a critical bottleneck in model development. Unlike brute-force methods that generate excessive, noisy features, py-evoFE employs evolutionary selection to produce compact, high-impact recipes.
Introduction to Semi-Supervised Learning
## Introduction to Semi-Supervised Learning Semi-supervised learning offers a powerful bridge between supervised and unsupervised techniques, leveraging both labeled and unlabeled data to build more robust models. This primer explores the core concepts, detailing common algorithmic approaches—from self-training to graph-based methods—and their practical applications. While utilizing unlabeled data can significantly enhance performance, it's crucial to acknowledge inherent limitations; biases in the unlabeled set can propagate, impacting model accuracy.
Why Your Betas Explode: The Hidden Geometry of Multicollinearity
Regression coefficients behaving unexpectedly? The phenomenon of “exploding” betas often stems from a less-discussed culprit: multicollinearity. This post unveils the hidden geometry behind this statistical challenge, explaining why highly correlated predictors destabilize your models. Discover how understanding the underlying geometric relationships – specifically, the angle between variables – can illuminate coefficient volatility and guide effective feature selection. Explore practical strategies to diagnose and mitigate multicollinearity, ensuring stable and reliable regression results.