Struggling with an underscoped refactoring task - What to do?

In this particular case, I would try to firstly spend some time only for the analysis phase. Basically, I try to analyse where all the lambda's are being invoked from.

I start scoping out the changes which need to be done in order to facilitate the requirements. Thereafter, we analyse the impact of those changes. I feel that an engineer who has been in the team a bit long enough might be able to help you assess all the business use-cases that are being impacted, or point you in the direction of how to evaluate the same.

Once you have the business usecases listed down, I suggest you to surface the same to the team, and try to understand more about what these business usecases entail at a high level. It is better to phase out a project that impacts alot of flows, especially when you are refactoring something. One way of doing it is to have some feature flags based on which you can enable the refactoring for a certain percentage of users. This way the scope is communicated much better to the entire team and it is a learning opportunity for the team as well.

Following things are very important in my opinion:

• Start it in phases, and a certain phase should contain the cluster of code pieces that are cohesive or as a whole form a certain business usecase, so that impact is minimal in terms of the changes, in case of any issues.
• Add tests, if they are not there already for the changes you are making, in order to make sure that whilst these changes are being done, the core functionality should still be served in the same manner. For example: a function intending to save data to a table should still do that and input - output mapping for that function should still remain the same.
• Telemetry: Push monitoring metrics around the flow you are refactoring. This helps you to assess issues if any, once you rollout the changes. In my opinion, post-production support and how quickly a system reacts to impact is testament of its resilience.
• Document, Document, Document! I cant stress this enough how important is it to document the things you are doing. Why did you chose to refactor it in a certain design pattern and not something else? For each business usecase, what all testing was done? What is the go to production checklist?
• Remove the dead code. Once you have rolled out your changes, and they are out in production serving all the business usecases correctly, then plan to remove the dead code. Do read this article I wrote on how important and underrated documentation is.

Please make sure to keep other team members in loop on the changes you are making and get it reviewed from time to time, in order to get feedback.

Some other references for a fun-time read:

1. https://martinfowler.com/books/refactoring.html
2. https://www.amazon.in/gp/product/0131177052

How do I communicate this with the team...

I think similar to the above advice, I recommend just creating a very nice write-up on why the refactor is much more complicated than initially thought. I'm assuming you have a task, so you can just put this context there and let whoever assigned the task to you (or the entire team if it makes sense) know.

Some tactics to use for the write-up:

• Have a 1-2 sentence "tldr" if possible that has the core point on why the refactor is complex.
• If it makes sense, make some diagrams illustrating the overall architecture as it is now.
• If possible, come up with multiple approaches on how to tackle the problem with pros/cons.

Something else I want to call out is to view this as an opportunity. Solving problems with gnarly technical scope is a huge part of growing from E3 -> E4, so this task gives you an opportunity to demonstrate this deeper technical skill.

...and not look incompetent?

Don't worry at all about this!

You're a fresh E3; of course you aren't going to be a super competent engineer at this point in time. Your team knows this as well, and they'll be supportive of whatever you ask or bring up as long as you do it politely and show that you did the legwork (i.e. doing the documentation as mentioned before). Embrace being a n00b and just put yourself out there! When you do that, you'll learn fast and stop being a n00b sooner than you thought.