Is the ds/ml slowly being morphed into an AI engineer? [D]
Our take
Agents are amazing. Harnesses are cool. But the fundamental role of a data scientist is not to use a generalist model in an existing workflow; it's a completely different field.
AI engineering is the body of the vehicle, whereas the actual brain/engine behind it is the data scientist's playground.
I feel like I am not alone in this realisation that my role somehow got silently morphed into that of an AI engineer, with the engine's development becoming a complete afterthought. Based on industry requirements and ongoing research, most of the work has quietly shifted from building the engine to refining the body around it.
Economically, this makes sense, as working with LLMs or other Deep Learning models is a capital-intensive task that not everyone can afford, but the fact that very little of a role's identity is preserved is concerning.
Most of the time, when I speak to data scientists, the core reply I get is that they are fine-tuning models to preserve their "muscles". But fine-tuning is a very small part of a data scientist's role; heck, after a point, it's not even the most important part. Fine-tuning is a tool. Understanding, I believe, should be the fundamental block of the role.
Realising that there are things other than "transformers" and finding where they fit into the picture. And don't even get me started on the lack of understanding of how important the data is for their systems.
A data scientist's primary role is not the model itself. It's about developing the model, the data quality at hand, the appropriate problem framing, efficiency concerns, architectural literacy, evaluation design, and error analysis. Amid the AI hype, many have overlooked that much of their role is static and not considered important.
AI engineering is an amazing field. The folks who love doing amazing things with the models always inspire me. But somehow, the same attention and respect are no longer paid to the foundational, scientific side of data and modeling in the current industry. I realise it's not always black and white, but it's kind of interesting how the grey is slowly becoming darker by the day.
Do you feel the same way? Or is it just my own internal crisis bells ringing unnecessarily?
For those of you who have recognized this shift, how are you handling your careers? Are you leaning into the engineering/systems side and abandoning traditional model development? Or have you found niche roles/companies that still value the fundamental data scientist role (data quality, architectural literacy, statistical rigor)? I'd love to hear how you are adapting
[link] [comments]
Read on the original site
Open the publisher's page for the full experience
Related Articles
- What has been people's experience with "full-stack" data roles?I started my career being a jack of all trades - hired as a data analyst but I had to extract, clean, and then analyze data and even sometimes train models for simple predictions and categorization. That actually led me to become a data engineer but I've spent most of my career working closely with data scientists and trying my best to make their jobs easier by taking away all the preprocessing tasks away from them so they can focus on training, inference MLops, etc. While I claim to have helped them, to be honest DE teams often become a bottleneck and an obstacle. Everything from not being able to provide the data needed to train on time, or how we processed the data was wrong and led to bad performance, or they went live with a model blindly because we couldn't get them the observation data on time for them to analyze accuracy. I'm wondering how much of the data engineering tasks can be automated/vibed away by data scientists. My guess is that in larger companies this won't be the case but I think startups and SMBs want to move fast so they'd rather have data scientists own the whole pipeline. What has been other's experience with this and where is it heading? submitted by /u/uncertainschrodinger [link] [comments]
- Beyond Jupyter Notebooks: The real work behind Production ML systems [D]One day, someone asked me about my day-to-day work and what ML Platform Engineering entails. They wanted to know how I excelled in this field despite coming from a Software and Data Engineering background. How did I manage to break into ML Platform Engineering and lead an ML platform initiative at my current company, one of the top tech-savvy startups in India? This question made me pause and reflect for a few moments 😄 I realised there is no single answer; it involves a lot of context and background. Firstly, I have never limited myself to the roles of a Data or Software Engineer in my various positions. Instead, I have focused on creating products that meet the needs of the moment, handling everything from start to finish. Interestingly, I've enjoyed DevOps more than traditional coding! In my daily work, I have consistently engaged with tools such as Kubernetes, Docker, CI/CD, Open Table format, compute and query engines, and messaging queues. Often, I have been responsible for designing high-level system architectures for the problem statements I encountered. These high-level and low-level designs have been instrumental in helping me understand products in depth. One key factor that has contributed to my success is my close collaboration with Data Scientists and Product Managers. They have always been my stakeholders, and I feel fortunate to have worked with many exceptional individuals. I have a habit of asking questions until I fully grasp my requirements. While it’s true that I have never worked as a pure Data Scientist and have never trained an ML model, I believe that Production ML is not solely about the model itself. In fact, ML model training and development are just small parts of the entire ML lifecycle. Let me elaborate on what an ML Platform Engineer actually does. When people start learning Machine Learning, most of the attention goes to the model. Which algorithm should we use? XGBoost or Neural Network? How do we improve accuracy? Can we tune hyperparameters better? All of that matters. But once you move from notebooks to production, you quickly realise something: The model is only one part of the system. A production ML system has many more questions: Where is the training data coming from? Who owns the feature pipelines? Are the same features available during real-time inference? How do we deploy the model safely? What happens if the model starts drifting? Who gets alerted when predictions become wrong, but the API is still returning 200 OK? This is where ML becomes platform engineering. In real production systems, a model needs: A reliable data pipeline. A feature store. A training pipeline. A model registry. A deployment workflow. A low-latency inference path. Monitoring. Drift detection. Retraining. Rollback strategy. Without these, even a great model can fail silently. This is one of the biggest mindset shifts in ML Platform Engineering: A model is not a product. A model becomes useful only when it is wrapped inside a reliable system that can train it, serve it, monitor it, and improve it continuously. That is why production ML is not just about Data Science. It is also about Data Engineering, Distributed Systems, APIs, Infrastructure, Observability, DevOps, and Software Engineering. The best ML teams are not always the ones with the most complex models. They are the ones who can reliably ship models to production and keep them working when data, users, and business conditions change. That is the real work behind Production ML. submitted by /u/thebigdatashow-ankur [link] [comments]
- How do you keep up without burnout?DS sometimes feels like there's infinite amount of things to learn. Most recent trend has been AI engineering And it's not like AI came in so you can deprioritize something else, but instead it just gets added to the heap. So you already had this massive amount of content to know from stats & product, trad. ML, deployment, ops, engineering, cloud, etc. and then you add the new thing on and the new thing. And when you read the job descriptions they literally list of all of this. I just had an interview for a random gaming company that wanted cloud, snowflake, stats, ML, ops, and AI experience in 1 person and it was for like 3-5 years of experience. And I wish that this was a one off thing but it seems to get more common. It actually feels like FAANG is easier to interview for because they silo people and not expect you to know and do everything What is your strategy for learning these skills without getting exhausted, or do you feel companies expectations are overflated? Is this a by product of AI where people are expected to do a lot more with less? submitted by /u/LeaguePrototype [link] [comments]
- A decade of being an average Data Scientist! My personal experience.Hello! I know there's people here with PhDs, working in FAANG, on top of the newest tech, and are absolutely brilliant Data Scientists. I'm not one of them. I've worked in medium to small companies with outdated technology, companies where I'm the only Analyst/Scientist, and places you've most likely never heard of. I don't do anything extraordinary, don't consider myself smart/brilliant, and I wouldn't pass a current day FAANG interview. But I have still had an amazing experience being a Data Scientist, and I have made real impact with companies I've worked in. I still interview at companies and have no issues getting job offers (although it's much more difficult right now). I've always had a hunger and drive to learn new things, but I found that I have had a knack for translating complicated information into a way anyone can understand. I make sure I'm kind, compassionate, and show anyone that data can be interesting and fun. I don't live to make myself look smarter, especially at the expense of other people, so I love breaking down complicated concepts in a way anyone can understand! I love showing insight from data and directions we can go. I enjoy building models - even if a lot of them go nowhere. Some of the biggest impacts and decisions companies have made have come from bar charts and basic KPIs. And I plan to keep doing it. I'm so average, maybe even below average, but I love what I do and I lean into what I'm good with. I have seen such a drastic change in the field, especially with AI, and I'm currently adapting to those changes too. Anyway, I just wanted to share my positive experience from someone who is painfully average lol!! I wanted to show people, especially new grads and/or people pivoting into the field, that you don't have to be the smartest person in the room to get hired. You need to drill into the solid foundations and a have a drive to make change/bring value to a company. submitted by /u/tits_mcgee_92 [link] [comments]