Best Practices for Retrofitting Documentation in a Growing Codebase

Okay, I have a lot of thoughts here, so I'll split things up.

Writing The Documentation

The key to communication is empathy, and documentation is a written form of communication. Your goal at a high-level with documentation is this:

This content will be so simple even the clueless intern who just joined the team can understand it.

In terms of how this manifests, it will depend on what you're writing about and the writing culture of the team. But here's some high-level pointers:

1. Include as many visuals as possible like diagrams (a picture is worth 1000 words!)
2. Don't rely too much on acronyms, just spell them out
3. Use basic words and analogies where you can

You can probably use AI to help with a lot of this, even generating the diagrams. ChatGPT integrated image generation natively fairly recently.

Here's a good thread about doing better writing: "How does Alex write so well?"

The Value Of Documentation

So here's my meta-point: I don't really believe in documentation that explains how code works. I have many reasons for this:

1. It conflicts with development - It is impossible to move fast and evolve the codebase quickly while simultaneously documenting it extensively. What you'll find is that writing the initial doc isn't so bad, it's maintaining the docs that completely destroys you - It just takes way too much time and cognitive load. This is why Meta had almost 0 documentation about how its codebase worked. If you need to learn a codebase or how some component works, just go talk to someone. It'll be more efficient, and you'll build up relationships with your peers through these interactions. That social capital will allow you to move even faster in the future.
2. Code should be self-documenting - At the base level, code should sort of explain itself if it's well-written. On top of that, code and systems should have natural artifacts that are generated as byproducts from working on them like pull request context sections, test plans, system design documents, well-written tasks, and even git commit/blame history.

Instead of making an explicit effort to create standalone documentation, it's way faster and smoother to create it organically along the way through existing avenues. I break this down more here: "Tips for someone with poor working memory?"

Here's my thoughts on your specific questions:

How do you prioritize and balance retroactive documentation with the need to keep up with new development within a Scrum framework?

I think just don't do it honestly.

But if you absolutely need to do it, your balance will come from working backwards from deadlines. So if the deadline is 2 weeks (10 business days) out and you can finish the project in 9 days, you have 1 day to work on the documentation. With that time constraint, you can figure out the level of thoroughness and quality you can achieve.

What strategies or modifications to the standard Scrum process have you found effective for embedding documentation tasks into sprints without hindering delivery?

With stuff like this, you need to weave it into the culture. If you get a project that takes 1 month for just the raw execution and the deadline is 1 month, then you have 0 time for extra stuff like documentation. Engineers just have to burn themselves out to write the documentation, which isn't sustainable.

If documentation is a first-class citizen at your company, you need to work with whoever is making the deadlines (i.e. leadership) so they know they have to add 10-25% buffer to every project for the documentation to be done. In other words, delivery always needs to be pushed back for this work, because it is considered part of the core work.

Are there specific tools, processes, or best practices you recommend for tracking documentation progress and managing technical debt in this context?

If you're a JIRA shop, Atlassian has Confluence. I haven't used Confluence in a long time, so I don't know what features it has, but it seems like you're creating tasks for documentation anyways. You can use JIRA to see what documentation tasks are open, which are on the backlog, which are blocked, etc.