Obviously, we need good UX, but I find myself obsessing over small details.
E.g. When the loading animation shows the display flexbox is slightly smaller than when the data loads because of scroll bars added once the data comes in. This makes it a bit jarring
There are other things like messing with colors, UX, animations, transitions, spacing.
Clearly you can spend all day tweaking UX. Where do you draw the line?
As with any project, work backyards from launch date and milestones. This is a critical technique called time-boxing where you tell yourself (and other other relevant stakeholders): "Whatever we have by Day X, we will ship it, no questions asked."
Simply having this mental model will often have a surprising amount of value. It puts a fire under your butt, and your brain will naturally start ignoring rabbit holes and ruthlessly prioritizing.
When it comes to side projects and building good products overall, the concrete line I draw is around efficiency. Getting the spacing or size exactly right is nice, but it's low ROI. If you can figure it out in 5 minutes, great, otherwise don't do it. But if you can take a core flow from taking 5 clicks across 1 minute to 3 clicks across 30 seconds, you should definitely do it, even if it takes a couple hours. I talk about this here: [Masterclass] How To Build And Grow Tech Products To 500k+ Users For Free
For additional inspiration, take this quote from Reid Hoffman (LinkedIn cofounder and PayPal mafia): "If you're not embarrassed by the first version of your product, you've launched too late."
How much you should care depends on your goals:
Alex's answer is also great -- it's a reminder that software is magical because it is malleable. The cost of improving and updating your app is very low (usually entirely free).