py-evoFE: Automated Evolutionary Feature Engineering for Tabular ML in Python (Genetic Algorithms + Scikit-Learn + Polars) [P]
Our take
The emergence of automated feature engineering tools like py-evoFE marks a significant step forward in the democratization of machine learning. For too long, effective feature engineering has been a bottleneck, requiring deep domain expertise and considerable manual effort. While libraries like Scikit-Learn provide the building blocks, crafting truly impactful features often demands a level of intuition and experimentation that’s simply unsustainable at scale. The challenge of orchestration across various AI tools, as highlighted in Orchestration is the new challenge for CX in the age of AI agents, underscores the need for intelligent automation that streamlines the entire ML pipeline, and py-evoFE directly addresses a key component of that pipeline. Similarly, the complexities revealed when models like LightGBM struggle with interaction terms, as explored in [Why does lightgbm not fit my toy example but catboost does? (2 order interactions) [D]](/post/why-does-lightgbm-not-fit-my-toy-example-but-catboost-does-2-cmt76sp7r0nyvmi9zjrrxn1da), highlight the limitations of relying solely on algorithms to uncover intricate relationships within data.
py-evoFE’s approach, leveraging genetic algorithms and incorporating a robust ecosystem of transformers and optimization techniques, is particularly compelling. The library’s ability to search the feature space intelligently, avoiding the pitfalls of brute-force methods and prioritizing parsimonious, generalizable recipes, is a crucial differentiator. The use of Polars and PyArrow for vectorized computation speaks to a performance-focused design, enabling users to tackle larger datasets without prohibitive memory constraints. The inclusion of features like multi-fidelity screening and the island model with Caruana ensembling further demonstrates a sophisticated understanding of evolutionary algorithms and their application to machine learning. The interactive replay viewer is a brilliant touch, facilitating understanding and debugging of the evolutionary process – a feature that dramatically lowers the barrier to entry for those less familiar with genetic programming. The seamless integration with Scikit-Learn’s pipeline and GridSearchCV further enhances its usability, allowing for easy adoption within existing workflows.
The core strength of py-evoFE lies in its ability to bridge the gap between raw data and model performance, effectively augmenting the capabilities of even powerful algorithms like LightGBM and XGBoost. The author’s emphasis on avoiding overfitting and promoting generalization is particularly noteworthy, addressing a common challenge in machine learning. While enterprises are increasingly recognizing the importance of limiting agent autonomy as discussed in Enterprises winning with AI agents are limiting how much the agents can do alone, automating feature engineering provides a complementary avenue for maximizing model performance while retaining human oversight. This library isn’t about replacing data scientists; it's about empowering them to focus on higher-level strategic tasks, freeing them from the tedious and often time-consuming process of manual feature engineering.
Looking ahead, the evolution of automated feature engineering tools like py-evoFE will be critical to unlocking the full potential of AI, particularly in domains where data complexity and feature interactions are high. One intriguing question is how these tools can be further integrated with automated machine learning (AutoML) platforms to create truly end-to-end AI solutions. Will we see a future where feature engineering becomes a fully automated process, driven by AI to discover and optimize features beyond human intuition? The open-source nature of py-evoFE and the invitation for community feedback suggest a vibrant future for this project, and the broader field of automated feature engineering, poised to reshape how we build and deploy machine learning models.
Hey everyone!
I’m excited to announce the release of py-evoFE (v0.3.0) — an open-source Python library that uses genetic algorithms to automatically discover, combine, and optimize feature transformations for tabular datasets.
- GitHub: https://github.com/tanopereira/py-evoFE
- PyPI:
pip install py-evoFE - License: MIT
The Problem It Solves
Feature engineering is still where most tabular ML competitions and production models are won or lost. While GBDTs like LightGBM and XGBoost excel on raw tabular data, they struggle to discover complex ratios, nested group-by aggregations, nonlinear dimensional projections, and interaction graphs on their own.
Manual feature engineering is either tedious or constrained by human intuition, while brute-force feature generation explodes the feature space exponentially with colinear noise and high memory usage.
What py-evoFE Does
py-evoFE searches the space of possible feature recipes using genetic programming: 1. Hierarchical Chaining: Evolved features become building blocks for future generations (e.g., log(ratio(groupby_mean(x1, by=x2), x3))). 2. 40+ Built-in Transformers: - Non-linear arithmetic & log-ratios - Target encoding (multiclass, pooled, WoE, quantile target encodings) - String similarity (MinHash, Gap encodings) - Manifold & Dimensionality Reduction (PCA, UMAP, MCA, FAMD, Between-Group PCA) - Graph & Density Clustering (Genie, Lumbermark, MST anomaly scoring) 3. Performance & Speed: - Vectorized computation powered by Polars and PyArrow. - Matrix Hashing & Nearest-Neighbor Caching: Stateful projections (like UMAP and $K$-NN lookups) are cached via byte-hashing to eliminate redundant computation across CV folds. - Multi-Fidelity Screening: Fast low-fidelity CV screens initial populations; only promising candidates proceed to full-fidelity evaluation. 4. Island Model & Caruana Ensembling: - Multi-population parallel search across Ring, Torus, Grid, Hypercube, and Tiered topologies with Gibbs migration. - Post-search greedy Caruana ensembling over island winners' out-of-fold predictions. 5. Interactive Replay Viewer: - Run view(evo.get_recipe()) to generate a self-contained, zero-dependency HTML dashboard replaying the evolutionary search over time. 6. 100% Scikit-Learn Compatible: - Implements fit, transform, predict, and predict_proba. Plugs directly into standard sklearn.pipeline.Pipeline and GridSearchCV.
Quick Example
```python import polars as pl from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from evofe import EvoFE
Load data
bc = load_breast_cancer(as_frame=True) df = pl.from_pandas(bc.frame) X, y = df.drop("target"), df["target"].to_numpy()
X_train, X_test, y_train, y_test = train_test_split( X.to_numpy(), y, test_size=0.2, random_state=42, stratify=y ) X_train_df = pl.DataFrame(X_train, schema=X.columns) X_test_df = pl.DataFrame(X_test, schema=X.columns)
1. Initialize EvoFE
evo = EvoFE( task="classification", evaluator="lightgbm", # "lightgbm" | "xgboost" pop_size=15, n_generations=10, cv_folds=3, verbose=True, random_state=42 )
2. Fit: Runs evolutionary search
evo.fit(X_train_df, y_train)
3. Inspect evolved recipe
recipe = evo.get_recipe() print(f"Discovered {len(recipe.genes)} high-impact features:") for gene in recipe.genes: print(f" • {gene.to_formula()} -> {gene.output_col}")
4. Transform & Predict
preds = evo.predict(X_test_df) proba = evo.predict_proba(X_test_df) ```
Why not just brute-force feature generation?
Brute-force libraries generate thousands of features upfront, leading to severe overfitting, massive memory usage, and colinear noise that degrades tree-based models. py-evoFE uses evolutionary selection pressures with complexity penalties to discover compact, parsimonious recipes that actually improve generalization.
I’d love for the community to try it out on your datasets or Kaggle benchmarks! Feedback, issues, and feature requests are very welcome on GitHub.
[link] [comments]
Read on the original site
Open the publisher's page for the full experience