Just to follow up, Joseph Machado's new course on Spark is out and I think it's a great way of learning more about Spark!

<https://josephmachado.podia.com/efficient-data-processing-in-spark?coupon=SUBSPECIAL524>

Data Engineer @ CI Financial

Gideon Blinick

Don't have any ideas for a good project, but I can recommend, as a baby step, to find someone who has done a project with the stack you're interested in and walk-through/copy their project to get the project-building juices flowing. Should be easy enough to find on YouTube/Google/LinkedIn.

It's better to deviate from the exact project you're following and add your own twist to things. 

But if you're having trouble getting started or finding an idea, seeing other people's ideas in action could point you in the right direction.

For ideas, I'd recommend asking friends and family for things they find interesting or problems they want solved, so that way you have a user/user-base off the bat.

I'd also recommend doing some brainstorming sessions with ChatGPT/Bard.

Hope that helps!

I'm no expert in Spark, and Spark has been something that I've been trying to learn more about recently, so I can't answer your question based on my own experience that well. 

Still, I follow a lot of great Data Engineers in the field, and one of them happens to have written about this literally today! I'll quote from him:

>  Many high-paying data engineering jobs require expertise with distributed data processing, usually Apache Spark. Distributed data processing systems are inherently complex; add to the fact that Spark provides us with multiple optimization features (knobs to use), and it becomes tricky to know what the right approach is.
>
> You realize there are too many moving parts (configs, metadata, partitioning, clustering, bucketing, transformation types, join strategies, lazy loading, testing, etc.) to learn to become a Spark expert. On top of all the moving parts, you also need to know how they interact and influence each other during data processing. Trying to understand all of the components of Spark feels like fighting an uphill battle with no end in sight; there is always something else to learn or know about.
>
> What if you knew precisely how Apache Spark works internally and the optimization techniques that you can use? Distributed data processing system's optimization techniques (partitioning, clustering, sorting, data shuffling, join strategies, task parallelism, etc.) are like knobs, each with its tradeoffs.\
> \
> When it comes to gaining Spark (& most distributed data processing system) mastery, the fundamental ideas are:
>
> 1. Reduce the amount of data (think raw size) to be processed.
> 2. Reduce the amount of data that needs to be moved between executors in the Spark cluster (data shuffle).
>
> I recommend thinking about reducing data to be processed and shuffled in the following ways:
>
> 1. **Data Storage**: How you store your data dictates how much it needs to be processed. Does your query often use a column in its filter? Partition your data by that column. Ensure that your data uses file encoding (e.g., Parquet) to store and use metadata when processing. Co-locate data with bucketing to reduce data shuffle. If you need advanced features like time travel, schema evolution, etc., use table format (such as Delta Lake).
> 2. **Data Processing:** Filter before processing (Spark automatically does this with Lazy loading), analyze resource usage (with UI) to ensure maximum parallelism, know the type of code that will result in data shuffle, and identify how Spark performs joins internally to optimize its data shuffle.
> 3. **Data Model:** Know how to model your data for the types of queries to expect in a data warehouse. Analyze tradeoffs between pre-processing and data freshness to store data as one big table.
> 4. **Query Planner:** Use the query plan to check how Spark plans to process the data. Ensure metadata is up to date with statistical information about your data to help Spark choose the optimal way to process it.
> 5. **Writing efficient queries:** While Spark performs many optimizations under the hood, writing efficient queries is a key skill. Learn how to write code that is easily readable and able to perform necessary computations.

He also has a great diagram showing these concepts together, but since I don't think I can share that in this answer, feel free to message me on Taro Slack or LinkedIn and I'll share it there!\
\
Also, if you haven't already, I recommend asking ChatGPT or Bard for tips, as they can  point you in the right direction.

Hope this helps!

I have a different take from Alex, as I'm more positive on the BootCamp. Big disclaimer upfront: I haven't taken it, so you should read what I write with a big grain of salt and also take to folks who actually have taken in (and get both the good and the bad from them).

Here are a few things to think about:

One, the cost of the bootcamp is steep at $2K. But compared to other bootcamps that charge north of $10K, it's not so steep. I myself took one of those expensive bootcamps.

Two, a big factor to me is whether or not you're currently working. If not, I think the BootCamp makes more sense. From folks I've spoken to who have done it, the commitment is min 10 hours/week, and can be 20 or 30 depending on how "well" you want to do. Zach gives a "Superb" certification and that means you have to attend every lecture, do all the homework, and have a great project. I imagine that takes a good deal more than 10 hours per week to get. Balancing that on top of a full-time job is doable but difficult. Whereas if you're not currently employed, the structure of the bootcamp can be a good supplement to the applying, networking, and learning you're doing on your own.

Three, the mental-model I have for time spent outside of a job has 6 buckets to it:

1) applying to jobs

2) networking

3) interview prep (e.g. Leetcode)

4) projects

5) courses/learning

Bonus: 6) Open source contributions

I think the BootCamp will help you with all 5 of those. You'll get a capstone project or two, so should hopefully come away with a good resume/portfolio project. You'll learn a ton of DE material that's relevant for Big Tech. I believe Zach has an interview prep module. You'll get to network with and meet other Data Engineers (and other folks taking the BootCamp). And you'll be part of Zach's network so that when recruiters come to him, you'll be a name on his list. And if you do a great job in the BootCamp, you'll be towards the top of that list.\
\
Zach also is marketing the camp towards people who have pre-existing DE knowledge, so the quality of your networking should be higher.\
\
Of course, the decision also depends on your financial situation.

But on the whole, if you're not currently working, I'm leaning towards taking it.

I was (and still am) in the same boat as you viz-a-viz the course and would be happy to chat about it with you. Hit me up!

Just want to +1 what Alex and Charlie said and say that ChatGPT/Bard/Copilot has personally helped me avoid courses. I now have an aid always available to give me feedback about tactics (coding) or strategy (design/architecture). 

In pre-historic times before ChatGPT, when I had a question, I'd Google it and have to sort through the links. If my answer wasn't easy to find in those links, I might throw up the question on Stack Overflow and wait a day or two (or more) until someone responded, if they did at all. That might kill my momentum. Now, AI is always there to give me a hand (cue the creepy music).

Of course, you will still need to Google things sometimes, sort through links and StackOverflow answers, and perhaps ask a few yourself, but AI can do a great job dealing with low-hanging fruit and help you see more clearly where your gaps are and where to invest.

Hope that's helpful!

Just thought I'd add [roadmap.sh](http://roadmap.sh) has roadmaps for both Docker (<https://roadmap.sh/docker>) and Kubernetes (<https://roadmap.sh/kubernetes>). It's not a practical project like you're looking for, but they at least give you a bird's eye view of both tools.

Sorry to hear about your mental + physical health not being well :(

In my opinion, those should be prioritized and somewhat non-negotiable. For me I work out at least 3x a week and usually right after my workday ends at 5. It helps in 2 ways:

1. Working out is super important for both mental and physical health. See [here](https://www.jointaro.com/lesson/RXEca9zfyZG0pl015att/2-ways-to-100x-your-productivity-that-nobody-does/) for Alex and Rahul talking about that
2. It marks the end of my workday and prevents me from staying at my desk longer than I should. If you need help with motivation to exercise, I recommend joining a class or working out with a buddy. Sports also count :) Looking into getting into a pickleball league myself right now.

Here are 2 good questions that address much of what you're talking about (one of which I asked):

1. [**How to balance interview prep with a job?**](https://www.jointaro.com/question/fob1V9LjohGiwRIn8Uvd/how-to-balance-interview-prep-with-a-job/)
2. [**How can I have less burnout and better work-life balance?**](https://www.jointaro.com/question/yvEFVA9n1wEcBU3fHNKU/how-can-i-have-less-burnout-and-better-work-life-balance/)

As a Data Engineer currently applying to different places, I can second Alex's thought about being cautious. The programs are expensive and you want to make sure you can genuinely get a lot out of them.

I did a lot of DSA prep last year on my own with AlgoExpert and found I largely haven't needed it (that could change) for a few reasons:

1. The companies that ask DSA questions aren't hiring as much currently and haven't been since at least fall 2022.
2. DE in general is a lot less heavy on DSA than SWE. The kinds of questions you get asked for a DE role will probably be medium-level at most, and more straightforward regarding the algos required.
3. I'm in Canada (Toronto) which is not dominated by tech companies but instead by financial companies and other companies that have more individualized processes.

I spoke to a Formation recruiter recently and think it's a good program if you're a SWE looking for a SWE role in the Bay area during a good economy (to use the opposite of me). But unless you know your problem is passing DSA interviews and you think you can get a steady stream of more of them, be wary. Also, try to find an alum who's working as a DE and in the same city as you so you can get their thoughts.

Lastly, [interviewing.io](http://interviewing.io) specializes in mocks, so they might also be worth checking out if you want to focus on that. I'm not affiliated with them in any way and haven't actually used them myself. But they do charge a lot per mock, and as Alex mentioned, you can find great people on Taro to do mocks with.

Hope that helps!

See Alex and Seed's [amazing and detailed answer here](https://www.jointaro.com/question/3VNPK7wol76s89EkgFeW/what-are-software-engineering-fundamentals/).

I think [this question](https://www.jointaro.com/question/fhV2SCn56aYF4gfPVLmV/how-do-i-balance-my-time-between-studying-dsa-and-fleshing-out-my-portfolio/) on DSA vs. side projects addresses your thoughts.

From my own perspective, I'd say you have 5 main ways to spend time outside of work:

1. Applying to jobs
2. Networking
3. Side projects
4. Interview Prep (Leetcode, Systems Design, behavioural, etc.)
5. Online Courses (Udemy, Coursera, etc.)

Online Courses are the lowest leverage, but if you're learning something totally new, they are probably somewhat valuable (e.g. Rahul's [Android tutorial](https://www.youtube.com/watch?v=KLuiX1oHBII&t=3s) for tip calculation).

Yes, although I have never actually done a FAANG interview, so take this with a grain of salt.

From what I know, SWE interviews are made of 1) DSA 2) Systems Design 3) Behavioural

DE interviews tend to have these rounds: 1) a SQL round; 2) Python coding (DSA light I think); 3) Data Modelling; 4) Data Architecture. Source: Zach Wilson <https://www.linkedin.com/posts/eczachly_dataengineering-activity-7016121971394437120-HJpA/?utm_source=share&utm_medium=member_android>\
\
There's also a behavioural round. \
\
For prep, I would guess they won't care too much in the behavioural rounds if you've got substantial SWE experience. You could probably just talk about that without any issues, particularly if it's backend, which is similar to DE. But also depends on the type of DE role, some are more backend-like and some are more data analyst-like.\
\
Hope that helps!

I can recommend some resources as a DE looking for jobs. Although I'm curious why you're looking to go from SWE to DE since I'm going in the opposite direction.

But here are some good ones:

1) Ace The Data Job Hunt by Nick Singh (just bought this, haven't gone through it)

2) Start Data Engineering: <https://www.startdataengineering.com/post/10-skills-to-ace-your-data-engineering-interview/>

3) Ben Rogojan (aka the Seattle Data Guy): <https://betterprogramming.pub/the-data-engineering-interview-study-guide-6f09420dd972>

Let me know what you think!

4+ hours a day on side projects! How much of that was at work versus outside of it?

Financial Company

Data Engineer

Great question and answer! I'm in the same boat as the OP, so appreciate the feedback :)

[Link to my question](https://link.jointaro.com/HKBbmwEMkF4ua4Mk6)

Masterclasses

If I told you that you can make $10,000+ just sending a couple emails, would you do it? Definitely yes right? This is why you should always negotiate your offers. Use this collection to learn how you can navigate these awkward, high-intensity conversations to get the pay you deserve.