Novell Home

Product development at Novell is no longer done only in the United States. We have developers all over the world. But our geographically dispersed teams have faced some challenges.

One such challenge was consistently developing high-quality code in all our locations while using several different test case management systems. Refined metrics around test coverage and product quality is a crucial part of the development process as it informs decision making company-wide. To make those decisions, Novell executives and senior technical staff need unified and coherent test case data.

The problem was that each time data was required, people needed to manually gather information from all the various systems and combine it, which was a time-consuming and error-prone process. It was obvious that Novell developers needed to standardize on a single testcase management system. But which one? In this case study, I explain the challenges we faced and ultimately the evolution of our new open source test case management system: Testopia.

A big part of the problem was our largest test case management system (TCDB) was badly outdated. It was developed in-house years ago, and had become difficult to maintain. The TCDB tool was designed prior to Novell shifting its focus to Linux and open source, which caused even more problems. There were some client and platform dependencies that prohibited the use of this tool in some locations across the world. To solve this problem, we would have had to deploy additional instances in many remote offices. In addition to promoting the use of an outdated tool, all the remote instances would have worsened our synchronization problem. It became evident that a different approach was needed—and quickly!

The lot fell to the Engineering Tools team to resolve this problem. We started an in-depth analysis of Novell's needs and how well various commercial and open source offerings met those needs. We needed to take Novell to the next level. The right solution was not on the shelf and ready for our teams to use. Instead, it involved the evolution of an existing open source project. Led by Novell technologists, and helped by many members of the open source community, the work resulted in Testopia, our new open source test case management system.

I'll show you how you can use open source development, and the strengths of the global community of open source developers, to develop products to meet your particular needs.

Testopia is a test case management extension for Bugzilla. It's a generic tool for tracking test cases, allowing for testing organizations to integrate bug reporting with their test case run results. Though it is designed with software testing in mind, it can be used to track testing on virtually anything in the engineering process.

The front end is built in Perl and DOJO, and the back-end database is MySQL. Testopia is not an automated testing suite, and it does not invoke any automated testing; however, it does provide fields in which to store automated testing paths or references. It also has an API set that lets users access those fields. Testopia is simply a test case management system that stores test plans, including Test Plan and test case layers and provides users with metrics of test runs and their environments. (See Figures 1 and 2.)

Testopia is an enterprise-class open source project that engineering managers should consider for their organization's test case management needs. An in-depth article about the features of Testopia will be in an upcoming issue of Novell Connection. The scope of this article is how the Novell team approached this project, and with a little creativity and some work, wound up with a terrific resource that meets the company's needs.

> Analysis of Existing Tools
A mix of test case tools were already in use at Novell when this project began. These included some open source tools, a home-grown Java application (TCDB), documents or spreadsheets and a set of proprietary commercial tools. It became clear that none of the existing tools would be up to the task, and we'd need to find another solution. A driving factor was that most of the tools were Windowsbased, which would not be suitable given all the open source and Linux development going on within Novell. As for available open source offerings, there was only a very limited offering on the Linux platform. Taking a collaborative approach, the team met with the project sponsors and users of the tools to better understand their needs and desired outcome. From there, the project objectives were decided:

The solution would:

  1. be available on Linux or be platform-agnostic
  2. not be client-based, but preferably browser-based
  3. meet the requirements of the Novell testing organizations
  4. be an open source tool that Novell could give back to the open source community
  5. would tightly integrate with Bugzilla, the bug tracking system in use at Novell

With these objectives, we quickly excluded several options. Only one tool was being used in our Nuremberg office that met all of the requirements. It was an open source tool called Testlink. The team's research also uncovered an open source tool called Testrunner, an add-on to Bugzilla. As we evaluated all of the available options, we concluded that we couldn't just pick an existing tool and implement it. None of the tools were robust enough to meet our enterprise-grade requirements. Very few options would run on the Linux platform and most didn't come close to meeting the project objectives. Finally, we chose Testrunner to use as the basis for our project and planned to redesign it and enhance the functionality.


Bugzilla Testrunner was an open source project started in 2001. It acted as an extension of the Bugzilla bug tracking system from Mozilla by adding test case tracking capabilities and a link between bug records and test cases. When we first started investigating Testrunner, only one active developer, Ed Fuentetaja, was maintaining the program. He had taken it over for Maciej Maczynski, the original creator. Once we decided to pursue Testrunner, we went to work. We started refactoring the code to work with modern versions of Bugzilla and adding features that were missing. We worked closely with Ed to ensure the changes were acceptable. But it soon became apparent that we needed to redesign the tool from the ground up.

Shortly after starting work on the project, Ed became busy with a new job and no longer had the needed time to devote to Testrunner. He was working on the project on weekends and wanted to find some additional help from the community. When we explained that Novell could devote significant resources, Ed turned over full control of the project in January 2006 to Novell. Gregary Hendricks became the project administrator and full-time developer and Mozilla accepted it as an official project.

Meanwhile, we began an extensive redesign of the database structure and supporting code. Due to the complexities of Novell's enterprise needs, we had to work around a number of challenges. For example, Novell products can be many levels deep and often are grouped and sold to customers in bundles. Multiple product versions go into these bundles, and larger products such as SUSE Linux Enterprise 10 (novell.com/linux) are also bundled. All need to be tested at several levels, from the individual components to the whole bundle.

We also needed a way to track detailed testing environments across multiple builds. Also, Testrunner only had rudimentary searching capabilities and lacked the powerful reporting capabilities needed in enterprise applications. We began to add these features as well as some more advanced enhancement requests.

In addition, the code base for Testrunner was developed against old versions of Bugzilla. We brought the code up to par with the latest version of Bugzilla and moved it to an object-oriented approach, which makes the project more flexible as new features are added, and makes it easier to maintain.

> Rebranding
After redesigning the tool with such significant changes, we needed to rebrand the tool. This avoided confusion and alleviated trademark concerns. We held a naming contest for a month within Novell and the open source community, and the winning name was Testopia. We designed a new logo and printed some t-shirts for the winner, best bug entries, most bug entries, project developers and early adopters. (We still have some t-shirts if you want to get involved in the project mozilla.org/projects/testopia).

> The Catalyst
One of the best things we did was establish a focus group at the beginning of the project. It comprised individuals (25 in all) from each type of testing group located in different countries around the world, plus the five of us from the Engineering Tools team. Of these 30 people, two were dedicated to this project full time. The others contributed to the project in addition to maintaining their daily job functions.

We held regular meetings and were deeply involved in the design and validation process. The meeting frequency varied based on the phase of the project. This group became the catalyst that made this project a success. The members were the spring board for successful early adoption of the tool. The focus group also helped us gain acceptance in their organizations and helped their teams and locations begin using the tool. Everyone on the team worked for the benefit of the whole. It was an incredible experience to work on a team so willing to give and take to make the whole successful. Their participation in design, requirements gathering, gap analysis, testing and now the migration to Testopia was and is invaluable.

> Delivery Schedule
Our project began in January 2006 and the first phase rolled into production May 25, 2006. We release new features and functionality internally each month based on the priority set by the focus group and the needs of the testing organizations. The open source release follows shortly after internal releases. The agile development process works well when deploying an open source project that needs additional functionality added.

> Delivery Schedule
Our project began in January 2006 and the first phase rolled into production May 25, 2006. We release new features and functionality internally.

It's All About the Bennies

For Novell, the benefits of moving to a single test case management system were obvious:

  • generate a single list of project bugs quickly and easily
  • manage bug dependencies more effectively
  • save time and money through increased efficiencies
  • monitor test coverage and status worldwide
  • use the Novell Product Master database to collect product test coverage and product quality rates and leverage the complexities of our offerings
  • increase consistency and accuracy of reporting by using one tool and one process
  • demonstrate the viability of an open source tool for enterprise computing tasks
  • enable partners and customers to test in the same tool (coming soon)

> Working with the Open Source Community
We are starting to have a more active group in the community at mozilla.org/projects/testopia and some code has already been submitted back to us! We are grateful that Mozilla.org sponsored Testopia as a project on their site. We're excited that Mozilla.org is beginning to adopt Testopia as their test case management system for the Bugzilla bug tracking system. Find out all the details about Bugzilla and check out the latest code at bugzilla.org. Testopia runs only with Bugzilla. One of the critical pieces of our deployment at Novell is the security integration. Find out about our security integrations, hardware and system setup in a previous case study about our Bugzilla deployment at novell.com/connectionmagazine/2006/10/tech_talk_2.html. Because Testopia is an addon, it leverages all of this from the Bugzilla application.

> Migration into Testopia
In June of 2006, Novell development teams began using Testopia. Most of the early adopters were teams that did not have their test cases in another system. It really helped for people to use the tool. Their suggestions, based on their experiences, helped us refine the UI and ease-of-use enhancements. As we delivered more features each month, teams began to migrate based on their product delivery schedules and the particular features for which they were waiting. To this day there have been no mandated migrations, yet the project has grown exponentially. It has grown larger much sooner than the team ever anticipated. Five months after roll out, we have approximately 300 test plans with over 300,000 test cases in the system. Thousands of test cases come in daily. With our latest release enabling the edit/delete feature, robust environment tracking, expanded API set, and the ability to import XML or CSV files, the increase of adoption is astonishing.

> Conclusion
We continue to work on Testopia and other tightly integrated open source tools. For our Engineering Tools team, this is a project that turned out to be very successful. We learned how to work both to meet our company's needs and give back to the open source community as a project administrator.

If We Had To Do It All Over Again

Things That Went Well

  • establishment of the focus group
  • a seasoned team familiar with Bugzilla, development languages and working on open source projects (four developers, one manager, one business analyst)
  • talented developers (and manager) that work well together as a team with great vision and drive to be successful
  • supportive project sponsors committed to open source and giving back
  • four able students from Neumont College worked on the project with one of our senior developers to gain real world experience. The students worked on the "environment piece" of Testopia and did a fantastic job. More information about Neumont at: neumont.edu/.
  • agile delivery model

Things That Could Have Gone Better

  • Availability of more testers to run Testopia through its paces

Over time, usage grew exponentially. (See Figure 3.) Remember, Testopia required no additional hardware from our Bugzilla deployment, and there are no annual maintenance fees, because it's an open source project. We can customize or add functionality to meet the requirements of our enterprise. We currently have thousands of users hitting our Bugzilla/Testopia Web application and have had no issues. Both have become large-scale and enterprise-worthy deployments. Our main MySQL server has been up for about 55 days and was last taken down for upgrades. We send 1gb and receive 4gb in network traffic to and from the server. We've processed more than 1 billion queries. The slave server has 1gb sent, 1gb received in network traffic. It has been up for more than 188 days and has processed more than 950,000,000 queries. There are more than 95 tables holding more than 15 million records and using 19.1gb in space.

We've established ourselves as valuable contributors within the open source community. Better yet, we've provided a remarkable product that lets other companies deliver higher quality product into the market—on a more rapid delivery schedule. Testopia was designed to have the flexibility to facilitate test plans/test cases for even non-software testing scenarios. For Novell, we found that participation and contributions from the open source community comes back many fold. We are well on the way to solving our own internal test case management system needs as well as helping others. So take a look at Testopia as a viable test case management solution for your organization. red N



© 2014 Novell