Blog Entry

dlythgoe's picture

Bonsai Authorized BETA has SHIPPED!

Author Info

4 December 2007 - 10:07am
Submitted by: dlythgoe

Tags

blog
Reads:

1147

Score:
0
0
 
Comments:

12

Last Wednesday, November 28th, we released Bonsai Authorized BETA.   This is a significant milestone in the development of this product.   It took a little longer to reach this date because we spent some extra time making sure that many of the issues surrounding Install were tested deeper.   As part of Bonsai, we moved to the latest releases of our Install tools, InstallShield, DemoShield, etc.   We are now on the latest releases for those products and we needed to make sure that upgrading and system creation tasks were tested in several different environments.

Speaking of Testing - I wanted to share some of the statistics/metrics we used for this first Authorized BETA and discuss our expectations for final release.

Across all of our components (POA/MTA/GWIA/WebAccess, Windows/Linux/Mac Clients, etc),  we ran all of our our P0 Levels test cases.  All components passed at 100% except Calendar Publisher and Monitor.   They passed at 89% and 78% respectively.   Our P1 test cases (1576 of them) were also run.   Each component passed these tests at a 70% range to 100% range.   The lowest being the GWIA on Linux at 72%.  The POA on Win and the POA on NetWare passed at 100%.   We have over 4000 P2 level test cases - we were able to run about 3000 of these test cases with pass rates mostly in the 80% average range.

As we get to each milestone in the release cycle, we will expect the product to be able to run more test cases at a higher pass rate.   We also know which areas we need to focus our efforts on and which areas need the most improvement before we ship.   This also helps our BETA sites to know which components to pound on and spend some extra time with to shake out the issues.

As far as defect fixing....We use a 'Severity' model and we have metrics around our threshold of defects with a particular severity.   For Authorized BETA, we fixed 85% of the defects found in Bonsai with a 'Critical' severity.   We fixed 68% of the Majors, 65% of the Normals and 27% of the Minors.

Before we release, we expect these numbers to be the following:   100% of Criticals, 95% of Majors, 90% of Normals and 85% of Minors.   We believe this will set a bench mark for a quality product.   It also allows us to do better escape analysis when a defect is reported by a customer.   Did we not have a Test Case for the scenario, or did we not run it or was it one that we knew about and shipped anyway?
As you can see, we are not ready for release yet :) - but it is these metrics that we will measure ourselves with to determine when the final ship date will be!    In the end, we want to ship Bonsai with great quality because we already know it has the features everyone wants.   What a great combination!


Author Info

4 December 2007 - 10:07am
Submitted by: dlythgoe

Tags




User Comments

Good news and great work. I

Submitted by AJ (not verified) on 4 December 2007 - 2:47pm.

Good news and great work.

I notice that the POA on Linux did not reach 100%, is this an indicator of larger problems porting to Linux? Given Novell is more and more a Linux centric vendor (From an OS standpoint), it does concern me that its not passing 100% on Linux. Perhaps a bit of information on GW and SLES/OES2 future together would be helpful.

I have been running GW 7 on Linux with largely no issues, however there are some annoying bugs (Linux path in DSL dialogues that are not working for example) that just wreak your head from time to time.

Apart from that, well done, GW still rocks.

Sharing your concern :-(

Submitted by Eric (not verified) on 4 December 2007 - 4:15pm.

Sharing your concern :-(

AJ, don't read too much into

Submitted by Alex Evans (not verified) on 7 December 2007 - 2:48pm.

AJ, don't read too much into the differences in pass rates. The GW team are 100% behind linux. You may already know but GroupWise server components are single source; by that I mean there are not separate source code files for each platform, so any fixes we make are made for all platforms at the same time. There are, of course, platform specific source files (server toolkit) that handle file i/o and memory allocation, but those are in the minority. Some of the pass rate differences are down to some of the platform specific differences - things like path support.

We have not addressed OES2 as a separate entity as we assumed is was a given - we will, of course, support OES2, SLES11 etc - this is our corporate strategy and we see GroupWise as a major driving force in making that strategy a success.

The test cases that did not

Submitted by Dean Lythgoe (not verified) on 8 December 2007 - 8:30am.

The test cases that did not pass have to do with install and configuration. That is the one area that is not single sourced and which changes from platform to platform. The test cases that failed were all on OES Linux and making sure that the configuration files are written correctly upon install/upgrade and that merging happens properly with settings that already exist with new settings that are available in Bonsai. These issues have already been resolved and had simple work arounds - that is why they did not hold the BETA. Obviously, we still have a lot of deep expertise on NetWare and Windows and developing to a new platform always has a learning curve. Our Linux experience is growing and as development tools improve on Linux, this will help us develop better products. Its all about evolution....We are evolving to greater understanding :)

I've tried to get on the

Submitted by George Passantino (not verified) on 12 December 2007 - 11:05am.

I've tried to get on the GroupWise authorized betas in the past, but to no avail. I seem to meet all of the criteria, especially my ability to use the beta in a near daily capacity. What's the deal with getting on a Beta? I'd like to get ahead of the curve in learning this so I can keep us on a decent upgrade path. My company tends to use a product beyond end of life, and that's a cultural caveat that I would like to see go the way of the dodo.

George - I use a number of

Submitted by Alex Evans (not verified) on 12 December 2007 - 11:44am.

George - I use a number of criteria to select the customers that we invite to join the beta. Space is limited and the number of applications are always high. As an example we took the applications from the last year, of which there were over 1500, and I had to filter that to less that 100. We generally look at past betas that customers have been a part of and their involvement on the calls, forums and through support - we are looking for customers that will provide valuable feedback, rather than seeing this as a way to get a cool new toy. We also look at the vertical, size, name and where they want to target their testing. What often helps an application is providing more detail on what you want to achieve out of the beta program when applying online. And now I have said that I know that I will have a lot more reading to do for the next beta - but these really are the things we look at.
Hope that helps

Alex

Alex, thank you for a better

Submitted by George Passantino (not verified) on 13 December 2007 - 9:35am.

Alex, thank you for a better explanation. I'm looking to demonstrate the viability of SLES in our network, and also trying to have them hold off using ACT where Bonsai would probably do for them. Showing them on paper doesn't cut it. Like I said, they tend to use products to EOL and beyond. I came here in November 2004 and it took me a solid month of demonstrating in a test enviroment that GroupWise 6.5 would work better than the GroupWise 5.5.3 they were running on. I circumvented that battle by running GroupWise 7 from the moment I got the public beta. Tested it myself in a test PO, with most of our Tech Support staff to rule out bugs (which we did find, just not as fast as others). I was looking for leverage in that proverbial battle. And our perpetual "Why we should stay on GroupWise instead of MSEX".

But at least I've got a better understand of how the selection criteria works. I'll wait for the Public Beta. Again, thank you for the writeup.

George

I should also have pointed

Submitted by Alex Evans (not verified) on 13 December 2007 - 11:38am.

I should also have pointed out that we always do a public beta prior to shipping, so it will be available prior to actual release for people to try out in their labs and kick the tyres a bit.

Maybe AJ and I are just

Submitted by Eric (not verified) on 13 December 2007 - 3:53pm.

Maybe AJ and I are just reading too much into the numbers. Plus, Linux isn't really "new" so much anymore (hasn't it been a GW platform for 3 1/2 years now?). Maybe OES is more the issue? It would be interesting to know if the numbers would have been different if you separated Linux and OES-Linux out as separate platforms.

Hi there, Late to the

Submitted by Arian (not verified) on 18 December 2007 - 2:57am.

Hi there,

Late to the conversation (sorry), with a quick question: when might we see a public beta? I have a handful of Linux users (~20 out of 700+) for whom I'd love to have the Bonsai client rather than the current Linux client...

Thanks,

Arian

Sounds good to me. I'm just

Submitted by George Passantino (not verified) on 27 December 2007 - 1:08pm.

Sounds good to me. I'm just anxious (and sort of bored), that's all. :)

We are shooting for a public

Submitted by Dean Lythgoe (not verified) on 29 December 2007 - 5:50pm.

We are shooting for a public BETA sometime around March/April - but it all depends upon our meeting our next exit criteria and quality metrics. However, that date is a good ball park. We will keep you posted as the time gets closer.

© 2009 Novell, Inc. All Rights Reserved.