March 30, 2026•1 min read•from Machine Learning

[P] Unix philosophy for ML pipelines: modular, swappable stages with typed contracts

Our take

Introducing an innovative open-source prototype that applies Unix philosophy to machine learning pipelines, we have designed a modular system where each stage—such as PII redaction, chunking, deduplication, embeddings, and evaluation—functions independently with defined typed contracts. This architecture allows for easy swapping of components, enabling users to efficiently test and compare the impact of individual changes on retrieval performance. As we refine this prototype, we invite feedback on our design assumptions. Explore our repository for further details: [GitHub Repo](https://github.com/mloda-ai/rag_integration).

We built an open-source prototype that applies Unix philosophy to retrieval pipelines. Each stage (PII redaction, chunking, dedup, embeddings, eval) is its own plugin with a typed contract, like pipes between Unix tools. The motivation: we swapped a chunker and retrieval got worse, but could not isolate whether it was the chunking or something breaking downstream. With each stage independently swappable, you change one option, re-run eval, and compare precision/recall directly. ```python Feature("docs__pii_redacted__chunked__deduped__embedded__evaluated", options={ "redaction_method": "presidio", "chunking_method": "sentence", "embedding_method": "tfidf", }) ``` Each `__` is a stage boundary. Swap any piece, the rest stays the same. Still a prototype, not production. Looking for feedback on whether the design assumptions hold up. Repo: [https://github.com/mloda-ai/rag_integration](https://github.com/mloda-ai/rag_integration)

submitted by /u/coldoven
[link] [comments]

Read on the original site

Open the publisher's page for the full experience

View original article →

LLMs for data pipelines without losing control (API → DuckDB in ~10 mins)Hey folks, I’ve been doing data engineering long enough to believe that “real” pipelines meant writing every parser by hand, dealing with pagination myself, and debugging nested JSON until it finally stopped exploding. I’ve also been pretty skeptical of the “just prompt it” approach. Lately though, I’ve been experimenting with a workflow that feels less like hype and more like controlled engineering, instead of starting with a blank pipeline.py, I: start from a scaffold (template already wired for pagination, config patterns, etc.) feed the LLM structured docs run it, let it fail paste the error back fix in one tight loop validate using metadata (so I’m checking what actually loaded) LLM does the mechanical work, I stay in charge of structure + validation AI-assisted data ingestion We’re doing a live session on Feb 17 to test this in real time, going from empty folder → github commits dashboard (duckdb + dlt + marimo) and walking through the full loop live if you’ve got an annoying API (weird pagination, nested structures, bad docs), bring it, that’s more interesting than the happy path. we wrote up the full workflow with examples here Curious, what’s the dealbreaker for you using LLMs in pipelines? submitted by /u/Thinker_Assignment [link] [comments]

[P] Unix philosophy for ML pipelines: modular, swappable stages with typed contracts

Related Articles