How many tests shall we run?

As many as possible, of course. However, as I discussed in the previous post, we don’t have to run all the test to estimate the overall success rate. It’s useful especially for release scheduling. You can get the automation test success rate daily or even hourly by just running a part of the test suites.

So, how many test cases shall we run to get good estimation? Here is the plot of Agresti-Coull formula with 95% confidence level. X-axis is the number of test runs and Y-axis is the error range. Each line indicates the various success rate.

The graph shows that roughly 150 to 250 test cases results in $\pm5\%$ to $\pm10\%$ confidence interval. Even if you run more tests, the interval doesn’t shrink much.

Conclusively, in the development phase, pick 150 to 250 test cases randomly from the automation test suite. Then process the success rate through Agresti-Coull formula. It gives you the quick estimate of the product stability.