Full stack software engineering vs. data engineering?
I'm currently a full stack software engineer working with React/Typescript for front end, Scala/Java for API design, and AWS. I love what I work with, but my team has had significant changes and I am looking to switch. There is an open role at my company for a data engineering role using python and airflow for ETL and they want someone to automate their pipelines. It sounds like the role is parsing/manipulating data to be sent to LLM models and the role works closely with ML engineers. I'll be honest that automation/data parsing doesn't sound as fun to me as full stack software engineering, but I was wondering if it sounds like a good opportunity to learn some ML for someone with zero ML experience. Does anyone have advice on if data engineering is a good career or will I be siloed away from software engineering forever?
![Senior Software Engineer [E5] at Meta company logo](/_next/image/?url=https%3A%2F%2Ffirebasestorage.googleapis.com%2Fv0%2Fb%2Ftech-career-growth.appspot.com%2Fo%2Fcompany_logos%252Fmeta.jpeg%3Falt%3Dmedia%26token%3Da7a5bcc0-e178-4b5b-b71e-09bf6c6aefae&w=48&q=75){kind=small}
![Entry-Level Software Engineer [L3] at Google company logo](/_next/image/?url=https%3A%2F%2Ffirebasestorage.googleapis.com%2Fv0%2Fb%2Ftech-career-growth.appspot.com%2Fo%2Fcompany_logos%252Fgoogle.jpeg%3Falt%3Dmedia%26token%3D780356ef-3efc-4728-9eb4-d909b75bb4a6&w=48&q=75){kind=small}
Discussion
(4 comments)- 0
4 months agoFounding ML Engineer @ Lancey (YC S22)ML roles vary widely between companies. Data Engineer can be anything from Database Admin/"SQL monkey" to creating pipelines using spark to writing scala/java code working on the data infrastructure.
The classic quintessential bay area data engineer role is one where you're working on large scale data pipelines using (py)spark.
Im curious what the size of the company is. This doesnt sound like a dedicated "Data Engineering" role to me and sounds more like an ML Engineering/ backend engineer job especially given there is no SQL based work.
And most of the work of an ML Engineer is cleaning/parsing the data. This is more true for smaller companies where you dont need a dedicated engineer for handling the data.
Honestly I wouldnt take this if youre looking to learn ML. LLM stuff is really just software and people treat LLMs like a blackbox.
I will also say frontend -> MLE is a huge difference in verticals and quite honestly there is not much overlap other than meta skills you get with writing software in general
- 04 months agoSenior Software Engineer [OP]Taro Community
Do you have recommendations on how to learn ML?
- 04 months agoFounding ML Engineer @ Lancey (YC S22)
Yes, but it typically depends on what are you trying to do with ML. I dont suggest learning ML from first principles typically because 1) there's so many verticals within ML its impractical and inefficient and 2) most of the time you actually dont need to know ML if you're working with LLMs.
For LLMs in specific I suggest just building stuff with them. You really dont need to know any classic ML to fine tune LLMs. Most of the times it will depend on the model and the platform you're using and following the docs. Think of a simple project and learn as you go to build that project out.
But if you do want to learn classic ML Engineering I suggest this crash course: https://github.com/serodriguez68/designing-ml-systems-summary/tree/main
It has everything you need to know about real world ML without the ML theory.
Should you want to learn classic ML theory I suggest this book: https://d2l.ai/
- 0
4 months agoMentor Coach for SWEs | Former Staff Software engineerI personally think data engineering experience is worth having as long as it's legit DE, like Sai said.
Unlike full stack (which comprises of the vast field of backend engineering), DE is more "contained". It becomes increasingly hard to develop breadth in BE as you progress in your career because of the vastness of the field.
DE is easier to be an expert in. :) The complexity in DE is also lower since a lot of cloud-provider services are available as "lego blocks" to orchestrate pipelines. There are fewer technical concerns such as memory management, threading, etc.
DE is here to stay and it's actively evolving. A lot of DE engineers today are not necessarily well-versed in strong "software engineering" skills, so you can bring a lot of technical excellence to the DE team.
Together with DE and ML, you develop a rare understanding of software engineering and how software at the company-level works—I have personally benefitted from being in this position!
If you try out DE only for a while, you won't be siloed away from SE. Depending on the lay of the land at your workplace, you might position yourself as an asset! Going forward, you'll likely have a choice to interview for either role and regardless of which you take, you will have a clear edge over those who know only one or the other.
Check out https://www.deeplearning.ai as well.
