Dealing with missed time estimates

First, you are not alone in this.

I have seen projects overrun and miss estimates many times.

When we were doing waterfall development, releases would get delayed frequently.

When you say open-ended, there seems to be an exploratory angle.
You are likely soliciting stakeholder feedback periodically. As they see what you deliver, requirements and deliverables might change.

In the Agile setup, with the frequent feedback cycle, I have also seen changes in scope or solutions taking longer than you expect.

The estimations are based on past predictors and/or what you believe needs to be done as per initial requirements.

When you start working on the project, you may find new information or experience unforeseen issues.

In another scenario, I have also seen some other critical issues come up and resources may need to be redirected to address them. As a result, in-flight projects might get delayed.

Communicating up your chain, as you have done, is the right step.

To meet deadlines, typical steps are:

• Reduce scope and break down the project into additional phases with stakeholder buy-in.
• Add more resources, if possible
• Add in buffers to ensure you have time to address the unexpected.

I'm glad your instinct was to be honest and report it as soon as possible. Many engineers will hide the problem in these scenarios, which leads to terrible downstream consequences (they either miss the deadline, burn themselves out trying to make up for the loss, or both).

In terms of how often missed estimates happen, it happens all the time and you shouldn't worry too much about it. The important thing is that you show people that you're diligently trying to learn from those mistakes (which should lead to you making less bad estimates in the future): "How to turn a string of silly mistakes into a mature positive outcome?"

Here's a course to help you make better estimates as well: [Course] System Design Masterclass: Shipping Real Features To Production

It is very common for software projects to miss estimates. In fact, perhaps even the majority. What's important is how you handled the delay:

• Did you communicate the miss early enough and share a remediation plan?
• What's the reason behind the delay? You will be judged based on how valid your explanation is.
  • "I forgot about my vacation" is a bad excuse.
  • "We discovered a legacy feature when looking at call sites for this API migration, so we're now working with the team to determine the path forward: X or Y" sounds more legit.

Speaking of efficiency, Mo Tiwari is giving a talk about exactly this Efficiency Tips for Engineers (From An OpenAI Researcher + Stanford PhD)