Automated vs manual UI testing?

There are a few questions you can ask yourself to inform your decision better:

1. How many users do you currently have?
2. How many features do you currently have?
3. What is the impact if anything breaks?

If there aren't a lot of users using your web app, then you should focus your efforts on either building out critical features or acquiring new users.

If there aren't a lot of features to test, then it should be quick to make sure you aren't accidentally breaking any existing features when you commit new code.

If the impact when something breaks is not severe, then you can tolerate some bugs if it means you are pushing the business forward without too many regressions.

It can be easy to underestimate how much work is involved with automated testing. It can be easy to set up your first few automated tests, and it even seems magical in the beginning. But, you end up having to develop and maintain your automated testing framework. There is also a maintenance burden of maintaining your automated tests (can be flaky or change often).

When you reach a stage where your product becomes critical and any downtime can really impact the customer, you can start building out a lean automated testing framework with tests for the critical paths.

I’m currently leading an UI automation effort in addition to my dev project. Here are some of my take aways :

1. It depends on how mature the product is, have they finalized the design, and only adding new features or are they still updating the UX Design. As with tools like playwright, if the UI devs have not properly identified the locators, they UI test will fail each time there is a design change.

2. Can you do a combination of API and UI tests, build out a basic UI setup, call this setup via an API, and UI test core functionalities.

3. What is the composition of the team? Are devs expected to write end to end tests or is there a dedicated SDET team for the same.

How do you choose whether to use automated UI testing vs manual UI testing for a new web application being developed for the first time?

With very few exceptions, it's not worth it having automated tests on a brand new front-end application (web, Android, iOS, PC/Mac app, etc):

• Front-end inherently has more testing complexity than back-end as there's pixels on screen while back-end has very clear contracts around what data it takes in and what data it returns (i.e. stuff that can be easily unit tested).
• These front-end pixels will vary based on their browser, device type, and a whole boatload of other factors.
• In the beginning, you should just ensure the quality by dogfooding a lot (i.e. using the app yourself) and proactively seeking feedback from your initial users - In other words, manual methods are more effective. You should be doing this to build up that user empathy anyways (super crucial with early-stage products).

That strategy around dogfooding and being customer obsessed can take you surprisingly far as well: Instagram got to 1 billion users with <10% automated test coverage across its mobile apps!

Any good practices or guidelines for when and how to make that choice?

As with many things in tech, it depends so much on your situation. I can't just say that you should just start doing it at 100k users, 1 million users, or 10 million users - It really depends. However, here's some questions to ask yourself:

1. How bad is it if stuff breaks? - Some bugs are harder to walk away from than others in that they erode user trust more. If you're working with something sensitive like finance or healthcare, the cost of failure is high. This means that being proactive with tests can be better.
2. How fast is the team growing? - New team members won't have that product intuition and flow knowledge that seasoned engineers have. This means it's much harder for them to catch bugs manually as they're testing their own code, presenting a case to have more automated tests if your team is rapidly onboarding new front-end engineers.
3. Are bugs occurring in complex flows? - Some flows can be very hard to test manually, usually very long and niche ones. In these cases, automated tests can make sense for the raw time-save. However, early-stage products shouldn't have these flows on the front-end ideally - The best products start out very simple with low-click, low-complexity flows.

I cover all this more in-depth in these 2 other discussions:

• "What do mobile testing strategies look like at top tech companies?"
• "What's the best way to carry out an end-to-end (E2E) testing effort?"

Agree with Charlie - it all depends on the life expectancy of the UI code you are writing. If it has no users and pivots are likely, there's no real value in adding UI tests. When you serve millions of users, tens and maybe hundreds of developers are working on the same code base, and regressions are harder to detect (for the simple reason that not all developers are experts at the product behavior anymore), automation is great and enables faster development because it allows developers to be confident in their changes.

That being said, there are actually several different types of UI automation (from simpler, unit-test-esque screenshot tests designed to test UI rendering and simple interactions to heavier end-to-end browser tests like Selenium). The heavier the tests are, the more cautious you should be about implementing them.

I fully agree with Charlie. He’s completely right.