As a software engineer with 5 years of varied experience, now in a tech lead role at a large company focused on business-oriented full-stack/fintech projects, how can I effectively pivot my career towards more technical domains, such as database engineering or developer tooling?
Should I transition to a different department? Different company?
What strategies can I use to develop domain-specific skills (e.g., compilers) that aren't directly related to my current job responsibilities?
Context:
I've bounced around a bit over the past 5 years of my career, and I'm in my first big tech role as a tech lead.
I've had to step up and laser focus my time and energy due to the demands of this job, needing to learn a lot about the business needs to deliver impact in my role.
I realize that my past roles I never really picked up how much the business domain and project domain should have mattered to me, because now I feel l'm working on the "wrong" domain and types of problems that inherently interest me.
One of the benefits of working at a large tech company is that you have access to people who are experts in all sorts of different domains.
I definitely would explore internally before leaving the company for a new tech area. Two benefits of doing this:
A related question that is specific to ML: How can I successfully transition my career into the data and ML side?
What strategies can I use to develop domain-specific skills (e.g., compilers) that aren't directly related to my current job responsibilities?
I'd first narrow down the areas of your interest and their longevity. IMO, databases, file systems, networking, and observability will always be in demand.
Then, I'd read research papers in that area. Most of the state-of-the-art software systems used in the industry today are based on research papers that are easily available online. Reading the relevant papers in sequence, starting with the oldest, will really help you understand the first-principles and build a strong foundation in that area. Taro has a research paper reading group. One mighty list is available here.
Looking up conference talks and blog articles is also a good strategy.
If you feel so driven, you can also try to implement a small portion of the paper yourself to gain a deeper understanding of the problem and appreciate the complexity involved.
Over time, you will know the under-the-hood workings of industry-standard software in your area. At that point, you can try to apply for a job (at your company using strategies shared by Rahul or elsewhere) on a core team. You may also be able to get referrals if you show up at a few conferences and chat with the speakers/attendees to build your network. IMO, this is a viable strategy to switch to a core domain without having to do a PhD, etc. Your software experience will also give you an edge over someone who's purely an academic.