What's the best way to carry out an end-to-end (E2E) testing effort?

• Core principle: Testing is an upfront time investment in the hopes of saving time later on by preventing a nasty regression from reaching production.
  • On a related note: All engineers eventually feel the pain of a broken test blocking their diff only to realize that the test is just buggy/flaky. Understand how often this occurs with your tests. When it shows signal, is it 100% right or crying wolf a non-trivial amount of the time?
• Deeply understand the bugs you're catching with your tests:
  • How many are you catching? 5 per half or 50 per half?
  • What's the severity of the issues you're catching? Are they just small UI regressions or SEVs?
• Meta-point: e2e tests are tricky as they're very time intensive, prone to flakiness, and hard to maintain as they span across the stack.
  • My org at Instagram had very few e2e tests, especially as we came to realize how flaky and annoying our e2e suite was. In fact, our end goal became to deprecate as much of our e2e testing suite as possible
  • Ask yourself if it's possible to get most, if not all, of the impact of your e2e tests by using lighter tests like unit tests and snapshot/screenshot tests, especially if you have found the e2e tests to be flaky

For more great resources around automated tests:

• "What do mobile testing strategies look like at top tech companies?"
• "Automated vs manual UI testing?"

• It's important to map E2E testing effort into the end user's experience and measure the exact coverage of it, especially for critical user's journeys. You can start from checking the percentage of regression tests that your E2E automation covers and set the goal to fully eliminate a need in manual regression testing. And the ultimate goal would be to eliminate manual testing completely. I would say this is an ambitious goal, but many companies have it in their plans. Moreover, some of them even ship their products without manual QA with varying degrees of success.
• I personally believe that having a manual QA team is also valuable, but all of the bugs they find during testing should ideally result in extra E2E tests to prevent them happening in the future. This could be another evaluation of your E2E testing effort's success.
• Also any crashes and production issues are obvious candidates (and measures as well) into the automation test suite.
• Moreover, usually the effort to make critical E2E tests land-blocking (preventing any pull request to be landed into master branch until all the automation tests pass) usually pays off in the long-term with higher engineering productivity due to a stability of the main development branch.