How to leverage side projects as a Data Engineer?

+1: Just getting better scope at your current job would be way better than trying to figure out a meaningful side project

You don't need your manager for this either. Go around your organization to data engineers you trust and ask them if they have anything meaningful on their backlog. If they're senior/staff level, they almost certainly do. From there, just pick up those tasks and do them. This situation is the best as there's mutual benefit:

1. You get meaningful work
2. They get a worker bee to delegate to and help them scale (they can claim some portion of your impact as they were the one who gave you the task)

Unfortunately I cant. I work for a pure classic finance company (not hedge fund) where I do mostly analytics/engineering. Most of the work is doing sql (+ managing DB) -> extracting insights -> making powerpoint slides/dashboards

We don't have the volume of data to be doing spark and we also don't have any production models so we dont need airflow so it's not something I can push within the org. It's also not a tech company and I'm part of an analytics division

The ceiling for this analytics/engineer job is pretty low which is why I want to upskill to get the higher paying data engineering roles

I see, the answer here is pretty obvious then hehe: [Course] Ace Your Tech Interview And Get A Job As A Software Engineer

I wouldn't gatekeep your job search on lacking skills - This will lead to you studying and building side projects forever. I recommend just applying a lot and seeing where that takes you. If you have spare time, you can study/build in your remaining free time.

The important part about applying (especially if you apply a lot) is that you'll get data. If you're noticing common themes across rejections, you can target your outside studying to fill that specific gap. Many engineers don't realize that interviewing is actually a data collection exercise in disguise 🕵️

Thanks Alex, it helps. I was curious if you have any general advice for people who are looking to get more interviews where users as a measure of impact is not super applicable? aka roles in infra/backend/data?

Also I get that getting users means the project is impressive which makes it a better project which would lead to more interviews. But why are projects with users better? Isn't a project with users just giving me signal about your marketing/sales/product skills and not engineering skills? Especially if you're building a utility app/one time use app e.g. background remover

Thanks alex. The issue with open source is that (from my limited experience) most projects are at the infra layer while most jobs expect you to be a pro at the application layer. That is building pyspark pipelines and using airflow and stuff. I feel like the overlap between infra layer and application layer is not that much?

I've briefly worked on open source for ml and a lot of it needed me to know stuff at the compiler layer or the pytorch low level stuff but when I try to train an ml model that level of depth and the problems faced doesn't translate much?

In other words working on ml/data infra is very different application of ml/data infra and I'm not sure if working on open source is the best path to gain better data engineering skills

Hmm, then maybe you can build a side project just for the learning as you mentioned? It won't help you get interviews as it likely won't get users, but learning is still valuable (and it can help you pass interviews once you get them).

The other option is to find additional scope at your current role as I talked about in my reply to Rahul on this thread.