Behind the Scenes with Novell Connecting Points,Part II
Novell Cool Solutions: Feature
By Karen L. Grant
Digg This -
Posted: 8 Nov 2000
After major tradeshows like Comdex and Brainshare, we always receive a lot of e-mail asking for details about how the Connecting Points network is set up. Our readers are very impressed with the tight security, reliability, and scalability of these temporary networks that take such a huge pounding throughout these events, and wonder what kinds of tricks are being used behind the scenes. Not that any of you have exactly this kind of network (knock on wood...), but you never know what you might be able to use.
Here's the second part of a three-part series that reveals how it's done. Part One was all about Planning the Tradeshow Networks. If you have further questions, let us know and we'll try to corner this frequent-flying team and get more info.
Setting Up Connecting Points Servers
Ah, it's that time of year when tradeshows begin to fill your calendar. Of course, you look forward to seeing the new technologies that will be touted and hearing the mind-boggling predictions made by keynote speakers, not to mention consuming lavish food and drink at fine restaurants (paid by your company of course).
Time to pack your bags. If you're planning to attend Comdex in Las Vegas, your packing list may include: swimsuit, bag of quarters, disco dance suit, and Elvis wig. But I assure you, Gary Norton, of the Novell tradeshow team, has an entirely different packing list!
Gary is the the tradeshow team member in charge of setting up the server portion of the network for Novell Connecting Points at tech tradeshows. In this article, I will describe the processes and procedures Gary and the other team members go through to set up and configure the server portion of Connecting Points.
- Gathering the Hardware
- Building the Core Servers
- Partitioning the NDS Tree
- Building the Secondary Servers
- Building the Vault
- Testing the Network
- Packing Up
- Bringing Up and Testing All Servers at the Show
- Importing Registered Users and Test System
- The post-mortem
- Stay Tuned...
- About the Author
In Part One of this series I described the planning process that the Novell tradeshow team undergoes before a major tradeshow. Once the planning is complete, it's time to make a list of what needs to be packed.
Once the list of required hardware for a show is made, the team begins to locate the needed hardware. This is tougher than it sounds since much of the hardware is scattered around the globe for various tradeshows. When the hardware is finally gathered in one place, it is set up and tested to ensure everything is functioning properly. In addition, Gary removes all the software from the hardware, so that each machine is clean and ready to start from scratch.
With the constant travel endured by the hardware, it is easy for the BIOS on each piece of hardware to become mismatched. Gary updates the BIOS and standardizes the hardware to one BIOS version.
Now we're ready to begin building the Connecting Points servers.
After the hardware is gathered, cleaned up, and standardized, it's time to build a base server. The base server is the first server to be configured and the first server in the core NDS ring. Gary calls this server the Master server.
To build the Master server, Gary installs NetWare, the latest NetWare service pack, and any necessary patches. At this point, the base server only has NetWare installed.
With the exception of some server monitoring software, none of the required applications or components are installed at this point. This allows Gary to ensure NetWare is working properly before installing anything else.
Since the Master server is the first server in the tree, it also contains the master NDS replica.
After the Master server is installed and configured, Gary checks the health of the server to ensure it is functioning properly. This preliminary health check involves:
- Ensuring time synchronization is working properly
- Ensuring the time zone is correct
- Ensuring NDS is properly functioning
- Monitoring NDS trace screens on the server console
- Running DSRepair
Now that the Master server is configured and running properly, it's time to configure the other servers in the core NDS ring. The core NDS ring usually consists of about three servers whose server objects are located in the root partition of the NDS tree. Gary installs and configures the other servers in the core NDS ring exactly like the Master server previously described. The servers in the core NDS ring are the servers that really determine the health of the tree because they contain replicas of the root partition.
Once the servers in the core NDS ring are installed and configured, Gary partitions the tree. Partitioning the tree provides fault tolerance and distributes problems by keeping the User objects in a separate partition from Admin or Server objects. It also keeps problems narrowed to one partition; thus, NDS problems that occur in the User partition will not affect the partitions where the Admin or Server objects reside.
Once the tree is partitioned, Gary checks the health of the servers again. This time, in addition to checking timesync, time zone, and NDS (as he did in the preliminary health check), he also ensures all the servers on the network are communicating with each other properly. (Go here in the online documentation and look under NetWare Server Documentation for more information on server communications.) This health check, as with all the other health checks, acts as an insurance policy against potential break downs. Even though Gary is the only person currently working on the servers, later in the process other team members will make changes to the system configuration. The tweaks and changes made to the system by various team members are not documented. So, performing frequent health checks can catch any potentially debilitating problems.
After performing the health check and repairing anything not working properly, Gary backs up the NDS database using DSRepair. It may seem useless to backup the NDS database this early in the process, considering the NDS tree is still fairly simple and no User objects have been imported into the tree. But, in reality, now is a great time to backup the database because it allows Gary to seal up the current favorable status of the network. At this point there are an Admin account, some login scripts, and a few setup parameters on the server. By backing up NDS now, Gary can restore NDS from the backup and be back in business within a short time should a problem occur. In the long run, frequent backups save time.
When backing up the NDS database, Gary backs up all servers currently installed. Even though there are NDS replicas on the other servers in the tree, those replicas can differ ever so slightly from one another. It is best to restore each server's NDS database to avoid potential problems and to get the tree back to proper health. As a matter of policy, the tradeshow team always backs up the NDS database before making any major changes to the tree.
If it becomes necessary to restore the database, Gary uses an internal tool. This tool is not available to the public; however, there are several commercial backup and restoration solutions available from companies like Cheyenne, Seagate and others that you can purchase to do the job.
Once the servers in the core NDS ring are built, functioning properly, and backed up, it's time to install and configure the secondary servers. Secondary servers are the remaining Connecting Points servers that are not part of the core NDS ring. Some secondary servers contain lower-level replicas like the User replicas. Other secondary servers contain no replica, like servers that house the GroupWise gateways or Web servers. Secondary servers do not have the same NDS dependencies as the other servers in the core NDS ring.
When all the secondary servers have been installed, Gary checks the health of the system and backs up the NDS database again.
If it is necessary to add another user replica to the tree, Gary creates a new partition or subpartition for the new user replica.
The next step is to create the GroupWise data volumes. These volumes are defined according to what is required for GroupWise, which was decided during the planning phase (see Part One of this series for more information). If Cluster Services is being used, these volumes reside on fiber-channel drive arrays in an external disk subsystem. Even if Cluster Services isn't being used, the volumes oftentimes reside on external storage, and some fault tolerance can still be obtained through disk mirroring or RAID.
The last server to be installed and configured is called the Vault. The Vault is the server where everything is stored in case the tradeshow team needs to rebuild the entire system. It contains things like images of servers and clients, service packs, installables, etc.
Installing the Software
Once all the Connecting Points servers are installed and configured, Gary installs the necessary software, such as Groupwise, ConsoleOne, DNS/DHCP, and any required snapins. The software installed is dependent upon the tradeshow requirements and the needs of the attendees. For example, Connecting Points for Comdex in Las Vegas this past spring looked something like this:
- Two servers had only NDS and NDPS
- One server had all GroupWise gateways including the GWIA and was used for GroupWise Web access
- One server was dedicated to the GroupWise Post Office Message Store
- One server ran all the GroupWise Post Office Agents
- Four servers ran Web access NLMs, the Web access gateways, and Novell Enterprise Web Server
- One server was the Vault
In addition to installing the software, Gary uses this time to create User accounts for the other tradeshow team members who will now start using the network to set up their pieces of the Connecting Points. Gary calls these User accounts Braintrust User Accounts. The Braintrust User accounts consist of five to ten users that have Admin rights to specific areas of the tree, depending on their individual responsibilities. In a way, the Braintrust User accounts provide protection for the network. From here on out several team members will be making changes to the system, and because there is no written account of all the changes being made, it is best to limit access to the Admin account to just one person, that being Gary. Gary is the only one to have total access to everything on the servers. Gary wants any team member to go to their BrainTrust User account first to touch the tree before coming to him to use the admin account.
After the Braintrust User accounts are created, Gary locks down the tree. This involves going through the tree and giving himself admin rights to the root partition. He creates a lot of organizational roles and groups to give individual rights to certain areas. This prevents backdoors from being created before other admins start working with the NDS tree.
With the tree locked down, it's time to check the health again.
At this point, once it looks like everything is working properly, the experts install and configure all necessary software on the servers. This includes things like the Web Servers and GroupWise.
After the software pieces have been installed and configured, Gary checks the health to ensure the servers are functioning properly. If everything is functioning properly, he backs up the NDS database and data sets. (Data sets are information that map to non-NDS processes, such as the GroupWise message store, contest questionnaires, and anything else on the tree that isn't protected by backing up the NDS database.) Gary now has what he calls the Clean Set. If he ever wants to roll back to a clean system, minus the real tradeshow users, this is what he restores from backup.
As with any network, everything has to be tested before it is put into a production environment. The Connecting Points network is no different. The first step in testing the system is to create a test user account containing one mock user. Shawn Bezzant, the team member who oversees the clients, uses this test user account with his client to test the network. He ensures the user can login and get the desired applications. He also checks to see if GroupWise works properly. If the test is passed with flying colors, Shawn adds a few more mock user accounts and tries the same thing.
If everything appears to be working, Shawn validates functionality and rights by ensuring the test users have access to certain patch files and can receive software updates pushed down with the NAL portion of ZENworks.
Now, it's time for the real stuff. Gary contacts the tradeshow registration company and asks for a pre-show import file listing the people currently registered. Attendees often wait until the last minute to register for a tradeshow, and some even wait until the first day of the tradeshow. So this list is not a final list of all the attendees. But it is a good start.
Once Gary receives the pre-show import file, he validates it by determining if the registration company provided the correct data in the proper fields. He looks for invalid characters and anything else that may be wrong. By asking for the data this early, Gary is able to iron out any potential problems he may have with the final registration file (which is downloaded at the tradeshow).
If this file appears to be okay, Gary uses his own utility to import a few of the users from the pre-show file into the NDS tree. Shawn then uses the data to test user accounts just like he did with the mock users created earlier.
After testing the data in the pre-show file, Shawn conducts a Lunch Time Testing session. About twenty employees in the Novell Tradeshows department receive test badges (like the ones used at the actual tradeshow). They use the Connecting Points like tradeshow attendees. However, they are unlike most tradeshow attendees--they have a mean and nasty streak in them and try with all their might to break the system. They also attempt to find any and all possible failures. After the test, the Lunch Time Testing group tells Gary and Shawn about things that may be wrong with the system. They list bugs and suggest how the system could be changed.
When all the issues found during the Lunch Time Testing are addressed, the group is reassembled, and the testing process begins again. Shawn continues conducting Lunch Time Testing until the tradeshow team feels confident the system is working properly.
After the Lunch Time Testing sessions are complete, Gary again uses his own utility to import the rest of the users in the pre-show file. The experts then test the system with several more user accounts from the import file.
When it looks like everything is working properly, Gary rolls back the DIBS and data sets. This involves getting rid of everything that was imported in the pre-show file, including user objects, GroupWise accounts, etc., and cleaning the system again. Once the tradeshow team arrives at the tradeshow, they restore the clean data sets and NDS database from the backups made prior to starting the test phase and then import the actual users from the registration file.
The experts then add to the system the list of changes that were identified during the testing phase, and at that point, they are ready to ship.
It's now time to pack up everything and send it off to the show.
It should be noted that sometimes all the preparatory set up work mentioned thus far has to be done at the show location due to close schedule dates for some tradeshows. For example, if there is a show in London immediately after a show in Paris, there is not enough time for the hardware to be shipped home to Orem, get configured, and the be shipped back to London.
When everything arrives at the tradeshow location (including the tradeshow team), the team gets right to work. First, they rack the servers, meaning they put all the hardware together and attempt to power it up. Often the power and external network connections are not working or are not available. When that happens the team has to track down tradeshow personnel to fix the problem.
Once all the hardware is put together and it looks like power and network connections are working properly, the team brings up the servers in the following order:
- Bring up Master Server
- Check health
- Bring up the other servers in the core NDS ring
- Check health
- Bring up secondary servers
- Check health
- Start processes--This includes all GroupWise components, Java servlets for questionaires, etc.
- Check health
If all the servers come up properly, Gary gets the actual registration data file from the registration company and imports a small number of users. He does this just to test the system to make sure everything works before importing the thousands of users contained in the file. Gary then backs up the NDS database and the datasets. He does this now so that he can rollback to this point in case he has problems with the rest of the registration data file once it is imported.
Gary then imports the rest of registration data using his own inhouse tool. This utility is not available outside Novell, but you can use Uimport or Bulkload to perform the same task. Once the entire registration file is imported, Gary backs up the NDS database and the data sets. Anyone who registers after this will be added to the system as needed during the show.
Gary now has a pretty close estimate of who is going to attend the show. Now he can set up mail distribution, which includes e-mail messages from Novell or the tradeshow planners to all users. For example, when users open their mailbox, they may see a message about a Novell product, or a message informing them about upcoming keynote addresses, or a welcome message. Mail distribution is automated so that specific messages can be sent to all users.
With mail distribution in place, the team is ready for the show to open its doors.
When the show is over, the team tears down everything and sends it home (or to the next tradeshow if shows are back-to-back). Once home, the tradeshow team sets up the Connecting Points again using the following process.
- Bring Up Master Server
- Check Health
- Bring up the other servers in the Core NDS Ring
- Check health
- Bring up secondary servers
- Check health
- Start processes
- Gather all necessary statistics
- Allow engineering teams to examine system, if necessary
Engineers check out bugs that were found at the show. The Novell engineers get a big benefit from this post mortem because they are able to see the effects of a system that was in in the real world, used by real people who beat on it for a week under heavy stress conditions. In addition, the Marketing people remove from the servers surveys, data and tracking information.
Now that you know how the Connecting Points servers are set up and configured, you'll want to know how the clients are set up. Well, be patient...we'll fill you in on the client side of things in Part Three.
Karen Grant is a technical writer with many years of experience documenting Novell products. She works for Write Tech, Inc, in Spanish Fork, Utah.
Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com