Last Wednesday, November 28th, we released Bonsai Authorized BETA. This is a significant milestone in the development of this product. It took a little longer to reach this date because we spent some extra time making sure that many of the issues surrounding Install were tested deeper. As part of Bonsai, we moved to the latest releases of our Install tools, InstallShield, DemoShield, etc. We are now on the latest releases for those products and we needed to make sure that upgrading and system creation tasks were tested in several different environments.
Speaking of testing - I wanted to share some of the statistics/metrics we used for this first Authorized BETA and discuss our expectations for final release.
Across all of our components (POA/MTA/GWIA/WebAccess, Windows/Linux/Mac Clients, etc), we ran all of our our P0 Levels test cases. All components passed at 100%, except Calendar Publisher and Monitor. They passed at 89% and 78% respectively. Our P1 test cases (1576 of them) were also run. Each component passed these tests at a 70% to 100% range, the lowest being the GWIA on Linux at 72%. The POA on Win and the POA on NetWare passed at 100%. We have over 4000 P2 level test cases - we were able to run about 3000 of these test cases with pass rates mostly in the 80% average range.
As we get to each milestone in the release cycle, we will expect the product to be able to run more test cases at a higher pass rate. We also know which areas we need to focus our efforts on and which areas need the most improvement before we ship. This also helps our BETA sites to know which components to pound on and spend some extra time with to shake out the issues.
As far as defect fixing ... we use a "Severity" model, and we have metrics around our threshold of defects with a particular severity. For Authorized BETA, we fixed 85% of the defects found in Bonsai with a "Critical" severity. We fixed 68% of the Majors, 65% of the Normals and 27% of the Minors.
Before we release, we expect these numbers to be the following: 100% of Criticals, 95% of Majors, 90% of Normals and 85% of Minors. We believe this will set a bench mark for a quality product. It also allows us to do better escape analysis when a defect is reported by a customer - did we not have a Test Case for the scenario, or did we not run it, or was it one that we knew about and shipped anyway?
As you can see, we are not ready for release yet - but it is with these metrics that we will measure ourselves to determine when the final ship date will be! In the end, we want to ship Bonsai with great quality, because we already know it has the features everyone wants. What a great combination!