What are the best methods to debug and fix a bug in production?
These are tremendously tricky to deal with, so what are your best strategies to navigate these pesky bugs?
Under the assumption: it only occurs in the highest level prod environment.
![Senior Software Engineer [E5] at Meta company logo](/_next/image/?url=https%3A%2F%2Ffirebasestorage.googleapis.com%2Fv0%2Fb%2Ftech-career-growth.appspot.com%2Fo%2Fcompany_logos%252Fmeta.jpeg%3Falt%3Dmedia%26token%3Da7a5bcc0-e178-4b5b-b71e-09bf6c6aefae&w=48&q=75){kind=small}
![Entry-Level Software Engineer [L3] at Google company logo](/_next/image/?url=https%3A%2F%2Ffirebasestorage.googleapis.com%2Fv0%2Fb%2Ftech-career-growth.appspot.com%2Fo%2Fcompany_logos%252Fgoogle.jpeg%3Falt%3Dmedia%26token%3D780356ef-3efc-4728-9eb4-d909b75bb4a6&w=48&q=75){kind=small}
Discussion
(6 comments)- 37 months agoTech Lead/Manager at Meta, Pinterest, Kosei
How long does it take to deploy code, and what's the cost of the bug in production?
If the deployment speed is very fast and the bug is not very severe, one option is to just make (educated) guesses with blind fixes, e.g. guard different method calls or add checks.
The more methodical fix is to add logging and instrumentation so you can see where the bug is coming from. Use something like posthog or a log collection tool to figure out what the errors are.
See this case study from Meta: https://www.jointaro.com/lesson/xAVw3j6fAB1GR9LUnq8n/meta-case-study-debugging-a-massive-production-issue/
- 2
7 months agoTech Lead @ Robinhood, Meta, Course HeroHere's my overall process to fix any bug:
- Understand the end-to-end flow (break it down into steps)
- Go through each step and see if it works/breaks
- Once you find the breaking step, analyze the code and find the fix
If you can never find the exact breaking step, you probably have to decompose it down more into sub-steps.
In other words, follow the advice from here: [Masterclass] How To Become A Debugging Master And Fix Issues Faster
The very tricky part is when you have limited observability, so it's hard to figure out if a particular step is breaking or even what the exact steps are. In that case, I recommend sharing more exact context when you ask for support in Taro, and we can creatively jam on some ideas 😊
- 3
7 months agoFriendly TarodactylTaro CommunityI can vouch for the limited observability part. Sometimes mobile app problems are so much harder to fix than backend. We can't ssh into users phone, while we can easily have root control of backend server. Mobile app uses different libraries in different phones, while we have 100% control in backend
- 17 months agoTech Lead @ Robinhood, Meta, Course Hero
Mobile issues can indeed be very gnarly, especially on Android. If you have a more global app (like Instagram where I spent the biggest chunk of my career), there are going to be a lot of users on janky old phones that are on an ancient version of the Android OS and have a weird screen size. Pain.
- 0
7 months agoSoftware Engineer [OP]GovernmentThis particular issue;
- Intermittent
- On Mobile
- It can only be tested if built to the phone (no f5 debugging)
- Testing on the Google Play Store vs building directly to the phone results in various failure %s
So a very tricky issue.
I might have fixed it now but I ended up implementing:- Logging service that sends errors and stacks to a local API and displays physical toasts
- Code push to more quickly deploy the app to the Play Store
- Posthog
- Firebase crashlytics (this was a gamechanger as it was erroring in an abstraction layer via a firebase auth package whose error couldn't easily be observed)
- 16 months agoSoftware Engineer [OP]Government
Follow up: FIXED
The steps I followed:
Google sign-in wasn't working and (the package) would return a vague error.- After implementing custom error logging and forking the package I got an error that was associated with incorrect keystores.
- I implemented code push, tested in different envs etc.
- I then changed everything from Firebase to Supabase.
- Tried recreating it as a single project
- Would always work when you built it but never if downloaded through the store
- Used every single AI tool
- Tried implementing fb login - Facebook sign-in prod gives me an error about key hashes not matching
- 1 random Stackoverflow post mentioning Gplay signing
- I was using Google Play automatic signing and so the App on Gplay was using a different keystore
- I added that SHA1 and it worked
