How many tests shall we run?

As many as possible, of course. However, as I discussed in the previous post, we don’t have to run all the test to estimate the overall success rate. It’s useful especially for release scheduling. You can get the automation test success rate daily or even hourly by just running a part of the test suites.

So, how many test cases shall we run to get good estimation? Here is the plot of Agresti-Coull formula with 95% confidence level. X-axis is the number of test runs and Y-axis is the error range. Each line indicates the various success rate.

The graph shows that roughly 150 to 250 test cases results in \pm5\% to \pm10\% confidence interval. Even if you run more tests, the interval doesn’t shrink much.

Conclusively, in the development phase, pick 150 to 250 test cases randomly from the automation test suite. Then process the success rate through Agresti-Coull formula. It gives you the quick estimate of the product stability.


About Moto

Engineer who likes coding
This entry was posted in Algorithm, Tips and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s