For Novell, 2004 brought many changes. With thousands of employees and having gained a number of other companies through acquisitions in the previous years, we found ourselves in a very challenging position. With each acquisition the newly acquired company brought its own set of development and tracking tools which significantly increased the difficulty of integration into Novell processes and tools. Trying to run a large company at a high rate of speed with eight different bug-tracking systems was challenging and presented a high risk to the success of our products.
In August 2004 we, or the Novell IS&T team, received an executive mandate to switch to a single corporate bug-tracking system for employees and partners around the globe. It was further mandated that we would use the open source product, Bugzilla.
For those of you who are not familiar with Bugzilla, it's an open source application used for tracking bugs. It utilizes a Web client, a server component written in Perl and the #1 open source database, MySQL. You can learn more about Bugzilla and check out the latest code at bugzilla.org.
More than 500 companies have adopted and are using Bugzilla. Find a current list at: bugzilla.org/installation-list. The benefits of moving to a single bug-tracking system were obvious:
- ability to easily get a single list of project bugs easily
- ability to better manage bug dependencies
- increased cost efficiencies by managing one bug system (instead of eight)
- ability to monitor bug status outside of engineering
- ability to utilize the Novell Product Master database to allow for integration of product data
- ability to increase consistency by using one tool and process
- opportunity to prove that an open source tool is a viable option in a sizable enterprise
- allow partners and customers to work on selected bugs.
At the time of the mandate, Novell had little experience with implementing open source tools for enterprise use. The one advantage we did have was that we had recently acquired two companies, Ximian and SUSE, which were already using Bugzilla and very familiar with the open source arena; however, we could not just deploy either of these Bugzilla implementations, because both had been highly customized and could not easily be brought up to the current Bugzilla release. This fact highlighted for the project team the desire to use Bugzilla with as few customizations as possible, so that the implementation could always be brought to the current release without significant effort.
The project team agreed on the principle of minimal customizations, and also several other guiding principles for the Bugzilla project. The set of guiding principles included:
- Use vanilla Bugzilla: We will customize as little as possible to fill Novell's business needs, yet be able to efficiently upgrade to the latest Bugzilla release.
- Use the Novell Product Master (PM) to provide the product and component lists.
- Manage product and component creation with the Product Master application, not in Bugzilla.
- Use a Change Control Board to rule on requests for customizations.
The time line for the implementation of Bugzilla was aggressive and seemed completely impossible in the beginning; however, many factors contributed to the project that have made it incredibly successful. We'll share the details of our Bugzilla implementation and hope these details help you as you deploy Bugzilla or other open source applications in your enterprise. (See Figure 1.)
Formation of the Novell Bugzilla Project Team
One of the key factors that significantly impacted the success of this project was having the right project team. A representative of each of the business units and locations was on the team, which consisted of about 10 people. Each team member had the authority to represent their constituents and make decisions. This team was divided into several subteams:
- business process
As rough estimates, the number of hours needed for the project were:
- 100 hours: 10 Steering Team members
- 65 hours: 13 Business Process Team members
- 3,000 hours: 5 Implementation Team members (full-time approximately)
(Note: After the implementation and migrations, there are 1.5 full-time team members for ongoing support, which is much less than the number used on the previous proprietary system.)
High-Level Project Schedule for Our Bugzilla Implementation
- Mandate came in mid-July
- Requirements gathering Aug 1st - 23rd
- Requirements locked down Aug 27th
- Technical Blueprint, CRUD Analysis, Village Diagram, Application Map, UI Mockup/Screenshots and Implementation strategy Aug 27th - Sep 30th
- Testing & Deployment Plans complete Oct 8th
- Testing complete and approval for rolling into production Oct 22nd
- Working system available for internal use on Oct 31st
- Migration of the eight repositories based on product release schedules, finishing Apr 30th
Obviously, we didn't need time for vendor analysis and comparisons to select the application, because Bugzilla was mandated at the executive level. We analyzed such things as current fields (in the existing eight systems) and their amount of usage, scalability, global performance, architectural standards in relation to corporate guidelines and user needs.
The first big push was to gather and lock down the requirements. We did this in several ways. First, the Steering Team divided up the use case roles (Customer, Novell Technical Services, Tester, Developer, Project Manager, Product Manager and Engineering Management) and began interviewing to gather requirements. During the interviewing and analysis process, we used common methods for gathering the information, including visiting users in their offices to observe how they did their jobs and analyzing their needs. We asked this set of questions:
- What do you need and currently use in terms of reporting?
- How do you use your current application (and how often)?
- What types of things do you look up in the current system you are using?
- How do you deal with the concept of product suites?
- How do you identify the bugs assigned to you?
- How do you launch queries needed in the bug system?
- What are the top three things you use the bug system for?
- What are your top three dislikes with the current system?
- What are your top three requests for Bugzilla?
We also sent an e-mail to all employees as a general call to submit requirements. This allowed those who were interested and indirectly involved to participate and give needed requirements for this new tool. We published updates as needed to the full employee base. This also allowed for a way to communicate and train others, and it initiated the culture change to using an open source bug tracking system.
As requirements were received, they were categorized by the following attributes:
- Requirement number
- If it was included in Vanilla Bugzilla (Y=Exists, S=Substitute, N=Not exist, U=Unknown)
- Implementation Status (already there, how much was there, etc.)
- Implementation Priority ( M=Mandatory, D=Desired, F=Future, R=Rejected)
- Requirement Category
- GAP Comments
- Additional Notes
One of the members of this team was from our SUSE Engineering Tools team who had extensive experience with Bugzilla and the open source culture. This was very helpful to have him mentor us and possess the technical ability to answer many of the questions we had. He was also a great asset in helping us understand the open source community and culture. Another challenge was the fact that Bugzilla was a tool established around a process. We had to change our process to adapt to Bugzilla. We were able to do this because of the simplicity of the Bugzilla process model, even though it was painful to some teams who had used very sophisticated processes. (See Figure 2.)
The project team was challenged to accelerate the implementation of Bugzilla at Novell. To do this, the project manager scheduled a week-long project kickoff in early September, during which stakeholders from four different locations convened to hammer out the guiding principles, business processes, functional requirements and implementation plan for Bugzilla. The meeting accomplished more than just accelerating the project. In particular, good working relationships were established and solidified in the face-to-face experience that would have been more difficult to build at a distance.
Additionally, it was valuable for the concerns of the users of all the different bug tracking systems to be heard in person. Disagreements and conflicting needs could be discussed to closure in a focused, face-to-face setting. Several participants in the kickoff meeting gave credit to that experience as a significant contributing factor to the success of the project. One of the comments in the post-project analysis was, "The kickoff was key to the success of this project. The kickoff participants became the biggest advocates for moving to the new system."
Success Factors Determined
- It is critical that the Business Process Team define a common bug-tracking process that meets the needs of all bug-tracking stakeholders that is implementable within the Bugzilla tool functionality. This will require accommodation and compromise among all stakeholders, and some customization of the Bugzilla tool.
- It is critical that any customizations of the Bugzilla system be contributed back to the Bugzilla open source community and become part of the future releases of Bugzilla. Novell does not want to fork the code and then remain permanently diverged from the Bugzilla community. Novell wants to support the Bugzilla community rather than create its own divergent system. Novell Bugzilla customizations will be designed in such a way that they are de-coupled from the Bugzilla code so that Novell can easily migrate to new versions of Bugzilla even in the event that some Novell customizations are not accepted by the Bugzilla community.
- It is critical that the performance and scalability of the Bugzilla system meet the needs of the global Novell stakeholder community.
Integration with Novell's Authentication
When you visit bugzilla.novell.com or a protected page, it redirects you to the eLogin page that consumes credentials. When you have logged in, iChain forwards your trusted authentication. iChain is a proxy that stores user data such as user name, e-mail name and unique ID, and makes that data available to applications via the http header. By this data, the application decides, based on the application logic, what the user can and cannot access. In our implementation we use an attribute called the webid/syncid as the primary key so when a user changes e-mail addresses for example, we can still tell it is the same user. Also, Bugzilla servers only accept connections from the iChain server. Even if you are an employee and try to access this server directly from inside of the firewall, your access is routed through iChain.
So Why Did We Customize?
One of the most important reasons why we customized our implementation of Bugzilla was to integrate our Product Master tool. Product Master is the tool of record for the names of our products, components, milestones and versions. (See Figure 3.) It supplies that information to Bugzilla. Product Master modules take care of attachments, chart data sets, classifications, groups, products and information about users. PM gets array lists with all of the available information while leveraging the profiles tables in Bugzilla. In Product Master, we also store data for usage metrics.
We added two fields, Product ID and Version ID, to the version and milestone tables. So now we can leverage the hierarchical relationships from our Product Master tool. Many companies that use Bugzilla only develop a few products. Because of the vast number of products Novell develops and our "shared technologies" model, we had some issues with mapping the product portfolio to the Bugzilla schema. "Shared technologies" means we bundle certain software products into other Novell products. For example, the Novell Security Services code is bundled into nearly every product we release. So the complexities of managing the release levels and the dependencies that go into each of those releases was a real problem and a change in culture from the way we did it in our highly customized proprietary bug-tracking system.
The integration of Product Master with Bugzilla lets us leverage the hierarchical product relationships without managing those in Bugzilla and therefore, compromising our upgrade capability. In later releases, we added dashboard reporting external to Bugzilla that leverages the bug information in Bugzilla and the product bundling relationships in Product Master. Because of the Product ID, we can use any of the product data found within any Novell application that has the Product ID integrated. These types of integrations allow us to manage the complexities of the product portfolio, as well as the simple, single product releases. These are perfect examples of one of the benefits of using an open source tool such as Bugzilla: you have the code and can customize and integrate as you need without relying on specialized consultants.
Our hardware configuration consists of six servers:
- 2 for iChain
- 2 for MySQL
- 2 for Bugzilla
Two servers host iChain for authentication. They were existing servers used for authentication to other applications. We use one of the MySQL database servers for read/write transactions and the other as a read-only copy. These are HP ProLiant DL380R03 servers with Xeon 3.2GHz processors and 1 GB memory. With our implementation, the database alone has 92 tables, with a total of 10,487,007 records. It is 15.8 GB. A benefit of MySQL, beyond the fact that it is open source and has no cost for the software, is that it has become very stable and reliable for large corporate instances. Since the last server restart, 1 GB of data has been sent to and from the database server. Maximun connections at one time has been 277. We average about 1,500 connections to the database per hour. Our instance has been up for several months without interruption and has only been taken down to perform maintenance and upgrades.
The final two Bugzilla servers are HP ProLiant DL380 G4 servers with Dual Xeon 3.6GHz processors and 2 GB RAM. The Bugzilla servers act in a load balancing way. When a user connects, it connects to the least busy server of the two.
One set of the three servers are production servers and the other identical set of three servers are the staging and test instances. Our development model has each developer desktop machine set up with their own test system where they test code on their own development box first. Code is then checked in to our source code repository, Subversion, and deployed to the Bugzillatest server environment every 15 minutes so that the test server always has the latest code. We use the test server as a way to ensure that all our integrations work and the Web application wrappers are correct in the corporate networked environment. When the application has been tested and is stable enough for a release candidate, we deploy that copy to our Bugzillastage server for final regression testing and approval before deploying to our production system. Because we have a duplicate set of identical hardware/software, we always have a fail-over option should a critical need arise.
This project was a great learning opportunity for us at Novell. It was the first big implementation of an enterprise-class open source application. We overcame a number of hurdles. First, deploying this tool didn't fit the authorized architectural model for our data center. Perl was not an approved development language, and MySQL had not previously been used for this large of a global application. Despite these and other hurdles, we moved forward and implemented a very successful project. (To see which things went well and which could have gone better, see Lessons Learned.)
The entire project team felt that this was an ideal team and project to work on. The unique combination of talents and abilities coupled with the willingness of everyone to learn new things, made quite a synergy of efficiencies. Whenever one person had too much to handle, others stepped in to pick up the load.
We continue to work on Bugzilla and other tightly integrated open source tools. Our immediate roadmap of major directions for Bugzilla at Novell includes:
- Testopia plug-in: In June 2006, Novell released a new test case management system called Testopia into the open source community as a Bugzilla plug-in. This was the pilot release, and in a later article we will explore this new functionality in detail. In the next few months, we will close the loop between build management, bugs and most types of testing. (mozilla.org/projects/testopia/index.html)
- Requirements Management Plug-in: Another area we envision is the ability to track and manage requirements with complete traceability from requirement to code, builds, tests, bugs and delivery.
Obviously, one of the important considerations in the use of open source tools is the relative cost of open source versus proprietary products. Many people immediately think of open source as "free." This is seldom reality if you include internal costs of implementation. Prior to the Bugzilla implementation, Novell used a proprietary bug-tracking system for about six years. It's instructive to compare the costs of implementing the two different systems, including license and upgrade, support, consulting, training, internal development, integrations and internal management and control. Hardware costs are not included in the comparison.
The proprietary system cost during the first two years was approximately 2.5 times the cost of implementation of Bugzilla for the first two years. User satisfaction with Bugzilla after two years is somewhat higher than satisfaction with the proprietary tool was after two years. The steady state cost of maintaining the proprietary tool was 3.5 times the cost of maintaining Bugzilla, primarily because of the amortized licensing costs and the ongoing support costs.
Figure 4 shows the comparison of the costs of the two different implementations, from investigation and tool selection, planning, installation, customization, training, migration, and finally, into a steady running state. Licensing and support costs of the proprietary tool were amortized over the two year period for the purposes of this comparison. There were no licensing and support costs for Bugzilla. Each of the "Migration" and "Steady State" periods of time in the graph are approximately three months. (See Figure 4.)
For our Engineering Tools Team this was a pioneering project that turned out to be very successful. This was the open source project that we "cut our teeth on" so to speak. It was a catalyst for change not only with employees and culture, but in our architectural processes and exploration of open source tools. We took on the attitude that we would not rely on the community for all we needed; however, we would give everything back to the community and allow the community to leverage our work. We proved that open source is viable in an enterprise implementation, and that other open source tools, such as Bugzilla, Subversion and MySQL, can handle huge amounts of data transactions with reliability and have proven to be viable tools in a worldwide enterprise corporation.