On most projects that I worked it was enough to have a high-level of how the higher picture of the code works and for the most tasks it was enough to copy paste and adapt existing code and for debugging it was also enough to have grasp the high level image.
However, I worked only on maintenance projects, that weren't very innovative. In order to have the chance to work on a good project how well should I know the tech stack?
For example for the stack: java/spring/elasticsearch/kafka/AWS, should I know the newest annotations or how the framework works under the hood, and probably be able to extend the framework, how to debug machine on VM etc?
Is there any sort of metric with respect to how in depth you should know your tech stack? Because in order to write code in a tech stack you don't need a lot of time to learn if you know the basics, but to write state of the art code I feel like you need years.
Great question! The answer is "Whatever is necessary to write great, smoothly functioning code". Software is incredibly dangerous as it's easy to rabbit-hole and learn a bunch of stuff you don't need. Whatever your tech stack is, there are an infinite amount of abstractions to unravel. If you unravel them optimistically, you will be stuck doing that forever. Here's an excellent thread about this: "How to avoid going down the rabbit holes when learning new things?"
When it comes to building depth and being a good coder, my broader advice is to learn how to nimble and improve quickly as opposed to trying to hyper-prepare and "study" your way to mastery preemptively. Pay close attention to user complaints, treat code review feedback as a gift, and try to gracefully cover as many edge cases as you can. If you do that, you will become a rockstar coder in no time. I dive incredibly deep into that in our code quality course: [Course] Level Up Your Code Quality As A Software Engineer
In my experience, having a specific space or product in mind will help direct focus when it comes to depth.
The goal is to have enough technical tools in your belt to build in a specific space with both grace and speed. That requires a balance of system knowledge, language familiarity, how to implement certain algorithms and data structures in said languages, how to automate parts of your development flow, and the most useful libraries that exist within the ecosystem you're focused on. Debugging is a great callout too - anything to save you time while still maintaining good quality because in the end, that's what is valued by newly spun up product teams that launch truly great and innovative things.
From the topics you mentioned, certain aspects of depth matter, like how Kafka works under the hood, general latency numbers, how to be fault tolerant, load handling, and etc. These are important because they ultimately contribute to how well a good product will work. Good teams will evaluate you on these topics. If not, they'll at least evaluate how good you are at picking up the necessary amount of depth required to build something good.
I consider framework or language developments like news - I'll hear about the most important stuff if it is meaningful enough. I would only focus more on them if I were in language or framework development myself or if I have taken a personal interest.