Finding more scope internally vs. swapping company

Question

I've been a Data Engineer for most of my career and my observation is that scope as a Data Engineer can plateau and therefore I see a lot more L4/5 DE's than L6+. I think it is because you don't impact the bottom line directly and regularly.

At FAANG's I've worked at so far, finding new scope can be difficult even when you are working with stakeholders: it is "easier" to scope/build a product (i.e. SWE work) and show metrics of success to add value vs building a data pipeline which may be limited to them having a reporting need for example which often isn't the case especially in a more established firm.

I moved into a partner facing DE role to help more with scope/stakeholder exposure. The highest impact project I worked on so far is influencing an internal team to change the way we measure a particular metric. This involved mostly stakeholder management and nothing more complex than SQL queries from a technical standpoint. While it was fulfilling, this is also something I 'stumbled' upon and is rare due to challenges like partner scope/vision is limited/slow (their leadership can change and therefore you projects/ideas can), technical challenges of automating things because of larger concerns (e.g. privacy, lack of infra on their side which you have no control over) and so on (you generally have even less control than an internal DE).

In my current role, I am generally able to derive projects, but (in my opinion) they are limited in scope/value: i.e. build a pipeline, deliver an analysis. Therefore, even though the projects 'ticks the boxes' for an L5, it is not really driving a 'transformation' as an L6+ would. I also directly asked my manager what are some of the hardest problems we have, and have been told we have a lot, yet, I'm not hearing or seeing them.

Given the situation, would you:

Move to a SWE role internally at FAANG for a more established path 'up' (not sure this resolves the scope problem especially at FAANG as I think SWE-DE's can almost be even harder to get to L6+ on because they generally lack stakeholder visibility and focus on more top down work?).
Seek roles outside of FAANG where the scope of the work is already scoped to L6+ e.g. Airbnb so the 'heavy lifting' has been done in terms of scope.
Refine your scoping strategy within you own team, and if so, how?

Note: my motivation is to thrive at work, this isn't for a promo, just incase the post comes across as promo-focused. :)

Alex Chiou · Accepted Answer

Given the market, I'm leaning towards Option #3 of "Refine your scoping strategy within your own team":

Interviewing sucks right now and you're at Google, one of the greatest companies on Earth. As I'm writing this, Google is at ATH with $155/share. You're already in an incredible situation relative to almost everyone else.
Google is infamous for hard L4 -> L5 promotions so switching to SWE doesn't seem like a great solution, at least in the short and even medium-term. However, if you're really passionate about SWE and have fallen out of love with DE, you should 100% switch.

In terms of refining scoping strategy, I believe your manager is telling the truth when they mention that there's a lot of problems. However, there are 2 reasons why they aren't being more specific with you about these problems:

Not enough time - Distilling all the noise, chaos, and complaints you get as a manager into something actionable takes a lot of effort and managers are extremely busy people.
They are forcing you to grow - If an L5 project is handed to you, it ceases to be an L5 project. Being able to identify problems and create scope on your own is crucial for L5 and even more important at L6.

If you can't find a meaty problem to take on, I recommend going full on user researcher mode (UXR). Schedule 45 minute to 1 hour 1 on 1s with anyone on your team and across sister teams who is willing to talk to you and is connected to the data layer in some way. Tell them that you are here to help and want to hear all the complaints they have. Compile the notes into a master doc and see what patterns you can extract. There was a very high-performing L5 infra engineer I worked with at Instagram who did just that to great success as I describe in-depth here: "How can I come up with big initiatives, especially those at a Staff level?"

Here's a potential freebie on my end too: Are there any logging events you can delete? At a giant FAANG company like Meta or Google, I'm sure there's a lot of deprecated logs that aren't being used by anyone. By killing them, you will save the company tons of money. One of the big workstreams I created in partnership with an L5 Data Engineer at Instagram was identifying all the deprecated logs related to Instagram Stories. The savings from killing all those events was $25 million+ per year.

I'll be releasing my promotion course in the next week as well which will talk about these concepts in far more detail.

Finding more scope internally vs. swapping company

Discussion