•2 min read•from Data Science
Thoughts on how to validate Data Insights while leveraging LLMs
Our take
In my latest blog post, I explore how to validate data insights while utilizing large language models (LLMs). While LLMs can generate code to facilitate data science tasks, it’s crucial to employ additional tools to verify the validity of their inferences. Drawing on my 12 years of experience in statistics, data science, and machine learning, I propose a framework that emphasizes the multiplicative nature of data science—valid outputs require all input steps to be sound.
I wrote up a blog post on a framework to think about that even though we can use LLMs to generate code to DO Data Science we need additional tools to verify that the inferences generated are valid. I'm sure a lot of other members of this subreddit are having similar thoughts and concerns so I am sharing in case it helps process how to work with LLMs. Maybe this is obvious but I'm trying to write more to help my own thinking. Let me know if you disagree!
Data Science is a multiplicative process, not an additive one
I’ve worked in Statistics, Data Science, and Machine Learning for 12 years and like most other Data Scientists I’ve been thinking about how LLMs impact my workflow and my career. The more my job becomes asking an AI to accomplish tasks, the more I worry about getting called in to see The Bobs. I’ve been struggling with how to leverage these tools, which are certainly increasing my capabilities and productivity, to produce more output while also verifying the result. And I think I’ve figured out a framework to think about it. Like a logical AND operation, Data Science is a multiplicative process; the output is only valid if all the input steps are also valid. I think this separates Data Science from other software-dependent tasks.
[link] [comments]
Read on the original site
Open the publisher's page for the full experience
Related Articles
- LLMs for data pipelines without losing control (API → DuckDB in ~10 mins)Hey folks, I’ve been doing data engineering long enough to believe that “real” pipelines meant writing every parser by hand, dealing with pagination myself, and debugging nested JSON until it finally stopped exploding. I’ve also been pretty skeptical of the “just prompt it” approach. Lately though, I’ve been experimenting with a workflow that feels less like hype and more like controlled engineering, instead of starting with a blank pipeline.py, I: start from a scaffold (template already wired for pagination, config patterns, etc.) feed the LLM structured docs run it, let it fail paste the error back fix in one tight loop validate using metadata (so I’m checking what actually loaded) LLM does the mechanical work, I stay in charge of structure + validation AI-assisted data ingestion We’re doing a live session on Feb 17 to test this in real time, going from empty folder → github commits dashboard (duckdb + dlt + marimo) and walking through the full loop live if you’ve got an annoying API (weird pagination, nested structures, bad docs), bring it, that’s more interesting than the happy path. we wrote up the full workflow with examples here Curious, what’s the dealbreaker for you using LLMs in pipelines? submitted by /u/Thinker_Assignment [link] [comments]
- Is the ds/ml slowly being morphed into an AI engineer? [D]Agents are amazing. Harnesses are cool. But the fundamental role of a data scientist is not to use a generalist model in an existing workflow; it's a completely different field. AI engineering is the body of the vehicle, whereas the actual brain/engine behind it is the data scientist's playground. I feel like I am not alone in this realisation that my role somehow got silently morphed into that of an AI engineer, with the engine's development becoming a complete afterthought. Based on industry requirements and ongoing research, most of the work has quietly shifted from building the engine to refining the body around it. Economically, this makes sense, as working with LLMs or other Deep Learning models is a capital-intensive task that not everyone can afford, but the fact that very little of a role's identity is preserved is concerning. Most of the time, when I speak to data scientists, the core reply I get is that they are fine-tuning models to preserve their "muscles". But fine-tuning is a very small part of a data scientist's role; heck, after a point, it's not even the most important part. Fine-tuning is a tool. Understanding, I believe, should be the fundamental block of the role. Realising that there are things other than "transformers" and finding where they fit into the picture. And don't even get me started on the lack of understanding of how important the data is for their systems. A data scientist's primary role is not the model itself. It's about developing the model, the data quality at hand, the appropriate problem framing, efficiency concerns, architectural literacy, evaluation design, and error analysis. Amid the AI hype, many have overlooked that much of their role is static and not considered important. AI engineering is an amazing field. The folks who love doing amazing things with the models always inspire me. But somehow, the same attention and respect are no longer paid to the foundational, scientific side of data and modeling in the current industry. I realise it's not always black and white, but it's kind of interesting how the grey is slowly becoming darker by the day. Do you feel the same way? Or is it just my own internal crisis bells ringing unnecessarily? For those of you who have recognized this shift, how are you handling your careers? Are you leaning into the engineering/systems side and abandoning traditional model development? Or have you found niche roles/companies that still value the fundamental data scientist role (data quality, architectural literacy, statistical rigor)? I'd love to hear how you are adapting submitted by /u/The-Silvervein [link] [comments]
- Anyone else paranoid using AI for analysis?I'm a data scientist by training with my own process for AI-assisted analysis, SOPs, asserts, sanity checks. Just want to see if others feel what I feel. Claude Code for products: incredible, tight feedback loop, works or it doesn't. Claude Code for analysis: paranoid every time. Wrong analysis looks identical to right analysis, silently dropped rows, miscoded variables, a slightly wrong groupby, the code runs, the number has decimals, and you have no idea if it's real unless you read every line. And I feel one step removed from the data now. I used to write every line myself and notice the weird distribution, the unexpected category, the row that didn't belong. That peripheral awareness is where real insight comes from. With the LLM in the loop, I touch the data less, and I catch less. Do you also feel one step removed from the data compared to before these tools existed? What are you doing to safeguard and double-check AI-assisted analysis? Has AI-assisted analysis ever caused you to ship a wrong number to a stakeholder? What happened? submitted by /u/Ghost-Rider_117 [link] [comments]
- Do MLEs actually reduce your workload in your job?Maybe I’m wrong, but I feel like in the bigger companies I have worked for, the “client - provider” kind of setup for MLEs / MLOps people and Data Scientists is broken. Not having an MLE in the pod for a new model means that invariably when something is off with the serving, I end up debugging it because they have no context on what’s happening and if it is something that challenges the current stack, the update to account for it will only come months down the road when eventually our roadmaps align. I don’t feel like they take a lot of weight off my shoulders. The best relationship I ever had with MLEs was in a small company where I basically handed off the trained model to them for deployment and monitoring, and I would advise only on what features were used and where they come from (to prevent a distribution mismatch in their feature serving pipelines online). Discuss submitted by /u/yaksnowball [link] [comments]
Tagged with
#data visualization tools#data analysis tools#generative AI for data analysis#Excel alternatives for data analysis#big data management in spreadsheets#conversational data analysis#real-time data collaboration#intelligent data visualization#enterprise data management#big data performance#data cleaning solutions#self-service analytics tools#machine learning in spreadsheet applications#business intelligence tools#rows.com#collaborative spreadsheet tools#financial modeling with spreadsheets#natural language processing for spreadsheets#digital transformation in spreadsheet software#no-code spreadsheet solutions