•2 min read•from Data Science
What has been people's experience with "full-stack" data roles?
Our take
Many professionals in "full-stack" data roles have navigated the complexities of multiple disciplines, from data extraction to model training. This experience often leads to a deep understanding of the data lifecycle but can also create bottlenecks, especially when data engineering teams struggle to meet the demands of data scientists. As companies evolve, particularly in startups and SMBs, there’s an increasing interest in whether data scientists can take on more of the pipeline responsibilities.
I started my career being a jack of all trades - hired as a data analyst but I had to extract, clean, and then analyze data and even sometimes train models for simple predictions and categorization.
That actually led me to become a data engineer but I've spent most of my career working closely with data scientists and trying my best to make their jobs easier by taking away all the preprocessing tasks away from them so they can focus on training, inference MLops, etc.
While I claim to have helped them, to be honest DE teams often become a bottleneck and an obstacle. Everything from not being able to provide the data needed to train on time, or how we processed the data was wrong and led to bad performance, or they went live with a model blindly because we couldn't get them the observation data on time for them to analyze accuracy.
I'm wondering how much of the data engineering tasks can be automated/vibed away by data scientists. My guess is that in larger companies this won't be the case but I think startups and SMBs want to move fast so they'd rather have data scientists own the whole pipeline.
What has been other's experience with this and where is it heading?
[link] [comments]
Read on the original site
Open the publisher's page for the full experience
Related Articles
- 'Full stack' data scienceI'm noticing more and more roles require end-to-end production skills. Previously a DS role seemed to involve training a model to solve a problem, or creating a POC, then passing it to engineers to put into production. Now jobs want you to own the whole life cycle from training, to deployment, to monitoring, with knowledge of scalability, compute and engineering best practices. The problem is outside of start ups or small companies where the role has a large scope, it is difficult to develop these skills. Is this similar to others experience and what do they recommended? submitted by /u/likescroutons [link] [comments]
- Interview Experience: Big teams look for potential, smaller teams look for how fast you can instantly come add valueMy interview experience has been a massively varied at this point, but what I've noticed is the massive difference between big companies like FAANG and smaller orgs like DS in banking or random small companies At FAANG it's kind of like an IQ + knowledge test (what google calls Role related knowledge) and smaller companies do assessments for very specific types of modeling or use cases, like build a model being evaluated on a certain metric. So at FAANG I was asked questions like "why is the formula for s.d. different for pop. vs sample', or 'what happens to the bias/variance in x,y,z situation' mean while at companies that are smaller and pay less they sent me a random 30-60 minute assessment and asked me to directly clean data and code up a model with sklearn/pandas. Is this what everyone else has experienced? It does seem like at smaller or traditional companies test if you will be a good code monkey while others look for actual understanding. submitted by /u/LeaguePrototype [link] [comments]
- How do you keep up without burnout?DS sometimes feels like there's infinite amount of things to learn. Most recent trend has been AI engineering And it's not like AI came in so you can deprioritize something else, but instead it just gets added to the heap. So you already had this massive amount of content to know from stats & product, trad. ML, deployment, ops, engineering, cloud, etc. and then you add the new thing on and the new thing. And when you read the job descriptions they literally list of all of this. I just had an interview for a random gaming company that wanted cloud, snowflake, stats, ML, ops, and AI experience in 1 person and it was for like 3-5 years of experience. And I wish that this was a one off thing but it seems to get more common. It actually feels like FAANG is easier to interview for because they silo people and not expect you to know and do everything What is your strategy for learning these skills without getting exhausted, or do you feel companies expectations are overflated? Is this a by product of AI where people are expected to do a lot more with less? submitted by /u/LeaguePrototype [link] [comments]
- How are you helping your company understanding the limitations of AI derived data?From my perspective, one of the biggest challenges of data science as a field right now is the tension between: A) AI can give "pretty good" answers extremely fast and democratizes it B) Those answers are often decent, but could be nontrivially "wrong" C) That "wrongness" is often not exposed for months or years That is, AI fully democratizes "getting a number" to our biz stakeholders across just about any business problem. A lot of times that number is off some but still pretty good and useful, but we all know sometimes it's catastrophically wrong. However, even in those worse cases though, there's a pressure to move fast, and so the consequences of that wrong number are not eaten or discovered until a good while later (when you find out a prediction was wrong retro-actively, when flaws in a matching process are discovered, when it turns out to have been the wrong "data-informed" decision, etc etc). This is exacerbated by seemingly a lot of biz users either not understanding, or simply not caring, that "number could be wrong". That's not helped by perverse incentive structures either. So my questions is - what, if anything, are you doing at your company to help stakeholders understand that? Or more importantly, to help build a culture that takes the scenario more responsibly? (yes yes, there's maybe not much we can do about it. CEO whims and all that. But interested in what steps people are taking pro-actively) submitted by /u/jshkk [link] [comments]