Novell Branch Office Q&A: An Open Discussion
Novell Cool Solutions: AppNote
By Gary Childers
Digg This -
Updated: 27 Apr 2006
Updated 27 Apr 2006 - New question - #18
Table of Contents:
- Q1: Is Nterprise Branch Office fully supported by Novell technical support?
- Q2: The user authentication scheme in NBO seems confusing. How do users get authenticated?
- Q3: How do I install an Nterprise Branch Office server?
- Q4: OK, then how do I configure the Nterprise Branch Office server?
- Q5: So what is the EASY way to setup an Nterprise Branch Office server?
- Q6: Why does Nterprise Branch Office use LDAP authentication?
- Q7: How does data replication work in Nterprise Branch Office?
- Q8: Why do I have to 'Prepare the Central Office' before installing Branch Office servers?
- Q9: What are the cost benefits of Nterprise Branch Office?
- Q10: What are the disaster recovery benefits of Nterprise Branch Office?
- Q11: Help! I made a change on my NBO server, and now it's no longer available. What can I do?
- Q12: I need to access the system console on my remote NBO server. How can I unlock the console remotely?
- Q13: I am repeatedly getting IO errors on the initial data synchronization for my remote NBO servers, but it seems to go OK after that. What could be happening?
- Q14: I think my remote Branch Office servers are successfully replicating, but it's hard to tell. Isn't there an easier way to monitor the data replication?
- Q15: Can I salvage deleted user files on my Nterprise Branch Office server?
- Q16: I deleted and re-created a corrupt user account on my NBO server, and now it looks like the user's home directory is completely empty. What happened?
- Q17: A user deleted a folder on the NBO server, and I need to restore that folder from the Central Office server. Exactly how do I do that?
- Q18: I need to test the disaster recovery feature of Nterprise Branch Office, so I want to stage a 'fire drill'. What steps do I have to take?
- What Works for You
My previous article on the RSYNC utility for NetWare (see "RSYNC Q&A") has spawned off a few questions about the "parent" product from Novell, that is Nterprise Branch Office. Perhaps this article could serve as an informal Branch Office "Q & A" to answer some of the most common questions that I've encountered.
Novell's Nterprise Branch Office product came about initially as a solution for one of Novell's own network management issues. What can you do when your organization sprouts numerous little satellite offices all around the world (or state, or county)?
Should you set up many little individual, independent networks? That's obviously stupid, inefficient, and impossible to manage. Do you set up one monolithic network spanning all sites, all under one eDirectory structure? While eDirectory certainly has the ability to scale for virtually any size enterprise, the cost to set up and maintain the monolithic network can be prohibitive, particularly for all the small office sites with just a few employees.
Novell had to find some middle ground - a way to leverage the advantages of a centralized eDirectory "identity vault" with all the corporate users and groups in order to have centralized management, and the advantages of the smaller, simpler, "cookie cutter" type of single-server environment for all the small offices out there. Hence we now have Nterprise Branch Office.
Nterprise Branch Office (NBO) accomplishes this feat by using specialized, but standardized servers at the Branch Offices which can provide the essential network services to users - file, print, DHCP, even email services - along with some extra benefits that the new Novell services provide: Web portal services (Virtual Office), and Web-based administration, and even ZENworks for Desktops, and ZENworks for Servers.
But one of the most appealing features provided by NBO is the data replication services, which can eliminate the pain of trying to manage traditional tape backup solutions at multiple remote sites, which usually don't have dedicated IT personnel. In fact, this advantage alone usually more than pays for the cost of the NBO product, when the cost of tape backup hardware, software, and labor (not to mention hassle) are all considered.
But let's move on to some commonly asked questions about Nterprise Branch Office:
Q1: "Is Nterprise Branch Office fully supported by Novell technical support?"
A1: That's a no-brainer. Of course it is. For those who are interested in a "roll your own" solution for just a piece of what NBO provides, many folks have implemented their own (unsupported) RSYNC solutions for data replication (see "RSYNC Q&A"). But for those who want the whole enchilada, the NBO solution comes with the full backing of Novell technical support.
Q2: "The user authentication scheme in NBO seems confusing. How do users get authenticated?"
A2: First off, let's clarify that NBO servers can run in four different modes, in terms of authentication: standalone, LDAP replicated to eDirectory, NT Domain, and NIS (Unix).
Standalone is simple enough to understand: user accounts are stored only on the local server, such as in a NAS device. This forces us into a "point management" situation, where all administration needs to be done in a distributed fashion, at each individual server for users at that site. This was the server-centric authentication model in NetWare 3.12, if you remember.
If you have a small business, with two or three small sites, this model might just work for you. But for the medium-to-enterprise organization with many remote sites, point management can be a nightmare.
The NT Domain method configures the NBO server to defer authentication to a Windows domain server, instead of local user accounts. Many NAS devices do the same thing. The NIS method is similar, only now deferring to an NIS server for NFS protocol authentication. Many NAS devices do this as well. Simple enough.
The confusion with NBO authentication usually comes with the LDAP authentication that is replicated to eDirectory. This requires a bit of explaining.
Users can access the NBO server through a variety of protocols, including AFP (Apple), NFS (Unix), HTTP and HTTPS (Web), NCP (Novell client) and CIFS (Windows native). But regardless of the means of access, the user is authenticated by one of the four authentication methods just described.
With LDAP authentication configured, the NBO server does not only defer authentication to a remote source (eDirectory, or even Active Directory), it will replicate the user and group accounts from that authentication source to the local server. Thus, users can authenticate to the local NBO server, even if the remote authentication source is unavailable.
The LDAP replication of user and group accounts does not occur continuously, as in eDirectory or AD synchronization, but is initiated, always from the NBO server, by a limited number of events, such as user login or password change. The NBO server configured for LDAP authentication will be configured for a specific eDirectory context, such as "Albany.Sales.Acme", which normally corresponds to a geographical location.
When a user logs in to the NBO server, the server checks the username against its local user database (which is eDirectory), and when LDAP authentication is configured, it then checks with the LDAP server for the user in the specific context in the corporate eDirectory tree for which it is configured (such as "ou=albany,ou=sales,o=acme").
If the user account does not exist on the local NBO server, but exists in the configured context in the Central Office tree, then the user account is automatically provisioned (replicated) on the NBO server (in its own eDirectory). A user home directory is automatically created, and the password is synchronized from the account in the Central Office tree.
If changes to the user account were made at the Central Office (corporate tree), such as a password change, group membership, or other account changes, then those changes will be replicated back to the NBO server's account database. The replication is triggered on the NBO server side by the login event, and only for that one user account.
LDAP is the protocol used to replicate the account information between the two trees. It is important to note that unlike Novell's Identity Management, no changes are "pushed" from the Central Office directory tree to the NBO tree, and changes do not occur continuously or periodically. Every replication occurs as a result of a specific event, such as a user login or password change, and the updates are then either "pulled" from the Central Office tree to the NBO server, or "pushed" from the NBO server to the Central Office tree, using LDAP.
The benefit here is that all user management for all branch offices, including password resets, can be done centrally from the Central Office, and then automatically replicated to the branch office servers via LDAP.
Q3: "How do I install an Nterprise Branch Office server?"
A3: Nothing could be simpler. NBO servers aren't "installed" in the sense of the traditional process of simultaneously installing and configuring an operating system - rather they are imaged with a standardized operating system, and then later configured for the specific operating environment in which they will be functioning.
All that is necessary to image the NBO server is to boot from the NBO installation CD, and then answer "yes" to the "Are you sure?" question that alerts you that you are about to image the hard drive, and lose whatever data is stored there. About 30 minutes later (on a newer computer), the server beeps at you, letting you know that it is ready to be configured - beginning with the IP address assignment.
Note that if you need to re-image the NBO operating system on an existing NBO server due to some kind of OS failure, you then are offered the Preserve (P) option at the beginning of the image process. This will preserve the data on the server (DATA volume), and the server configuration (ADMIN volume), and re-image only the NBO operating system (SYS volume).
Q4: OK, then how do I configure the Nterprise Branch Office server?"
A4: As we said, the imaging process lays down a standard server image, making this a "cookie cutter" server. At the end of the imaging process, when the server beeps repeatedly, you are prompted to give the server its basic IP configuration information, so that you can begin to communicate with and manage the new server. It's perfectly OK to build the NBO server on different network (LAN) than the one on which it will be deployed, allowing you to pre-build servers at the Central Office, perhaps in a lab, and then later deploy them out in the field.
Once the IP information is set on the server, and it cycles itself, you can then point a Web browser to the NBO server (https://serverIPaddress:2222). The server uses SSL for secure Web management, so accept its certificate, and login when prompted with the username "Supervisor" and a blank password.
The NBO Web Administration portal runs a setup wizard the first time you access it, to help you configure the basic settings you need to get the appliance up and running. The setup wizard prompts you to set the Supervisor password, the Appliance (NBO server) name, DNS name, domain, and DNS server address. It also walks you through setting up the automatic user access provisioning so that users (which have an account in an LDAP-enabled eDirectory tree) can access the appliance, and setting up data replication (if you have RSYNC configured on a server in the Central Office network).
NOTE: If you used a settings disk (see "the EASY way") to pre-configure the NBO server, you will not be prompted to set the IP address upon first boot, and you will not run the setup wizard upon first login to the NBO Web Administration portal.
Once this setup wizard has run (with correct information supplied), the NBO server is actually functional at this point. Users in the eDirectory context configured for the LDAP authentication can now login to the NBO server, and their user accounts will be automatically provisioned on the server, with home directory and all, and the user password will be synchronized with user account in the Central Office eDirectory tree.
It is recommended that you create a test user or two in the desired context, to test out the login capabilities at this point.
However, at this point essentially only file services are available to the user, as well as Web portal services (Virtual Office), which is enabled by default. To configure other services, such as DCHP, network printing, GroupWise email or ZENworks, you have to go back to the NBO Web Administration portal, and configure the correct parameters for each of these services.
Q5: "So what is the EASY way to setup an Nterprise Branch Office server?"A5: This answer only applies if you are planning to setup multiple NBO servers. If you are only installing one or two, you might as well skip this section.
Of course, you'll always want to setup your very first NBO server entirely in a lab environment, and I'd recommend that you go through every configuration step manually, using the Web Administration portal, just to better understand what is going on, and what configuration parameters are located where. But if it is your job to deploy, say, fifty of these servers, then even with the advantage of the speedy, hands-off server imaging process, the individual configuration of each of the servers might tend to be tedious.
As mentioned earlier, we can "pre-configure" the NBO servers through the use of a settings disk, that contains all of the server configuration settings in a text file. Then, in the server imaging process, the settings from the text file will be applied, and the server will boot up after imaging completely configured and ready to go - almost.
I say "almost", because although all the necessary settings can be pre-configured for the server - such as IP information, authentication sources, time servers and time zone, DCHP and data replication - some things just can't be gained from the text file. The SSL certificates for LDAP authentication and data replication have to be uploaded to the NBO server. Printer drivers also have to be uploaded to the server, when the NBO server will function as a print server.
But the use of the settings disk can get you 90% there by pre-configuring all the server settings that can be contained in text form. This can make it extremely easy to get a server up and running in literally half an hour.
The easiest way to get the settings file is to build and configure that first server the manual way, making all the configuration changes through the NBO Web Administration portal. Then capture a copy of the configuration file by using the "Import/Export Settings" link on the portal (lower left). You can direct the output of the file to a floppy (if the server has one), or to the ADMIN or DATA volumes on the server, and then retrieve the file over the network.
You can name the file anything you like when downloading, but when it is to be used in the imaging process it must be named "autoload.nbo". Once you've got a copy of this configuration file, then you can open it in a text editor, and comb through it to modify all the parameters necessary - IP information, server name, etc. - for the NEXT server you wish to image and pre-configure.
Imaging the NBO server with the configuration disk requires that the server have a floppy drive. The boot sequence in the server BIOS should be set to 1) CDROM, and 2) Hard Disk (since having the floppy as the primary boot device could cause a boot failure). The server should be booted with the NBO installation CD, with the settings disk with autoload.nbo in the floppy drive. The server will then image the hard disk, reboot, automatically apply the settings from the settings diskette, and then reboot again, this time with all the specified settings pre-configured.
Q6: "Why does Nterprise Branch Office use LDAP authentication?"
A6: As previously mentioned, NBO can handle user authentication in standalone mode (as a simple NAS device, with only local user accounts), or can utilize "automatic user provisioning" via LDAP authentication and replication of user accounts. What this means is that user accounts already exist and are managed in a central (typically) eDirectory tree, and the NBO server automatically replicates those user accounts to its local accounts database (which is a separate, "mini" eDirectory tree).
To answer the "why" question, we need to look at the alternatives: standalone (local user accounts), or eDirectory synchronization.
If yours is a small business, with just a couple of small offices, the standalone mode might be the most desirable for you. In this case, NBO server is merely a NAS appliance that can run on virtually any standard computer hardware.
However, if you are a typical larger organization, with many small remote offices, the "point management" of so many separate user account directories of the standalone model can become unwieldy (remember NetWare 3.12?). This is especially true if you already have the centralized eDirectory in place - the user accounts are already there, typically in geographically-oriented eDirectory containers (OUs), so the LDAP authentication merely piggybacks on this existing directory of user accounts.
The other alternative is regular eDirectory synchronization, which may be the traditional method for NetWare-based networks. Every geographic site has a NetWare server running eDirectory, and the eDirectory database is partitioned by geographic locale, and replicated for redundancy, and automatically synchronized.
This method of directory synchronization, now standard, is ideal for enterprise, medium, and even small-sized organizations, since it provides centralized management of network user accounts, of file, print, and many other network services. All is managed via a single, yet distributed and replicated, directory (eDirectory), which is reliably and automatically synchronized across the entire enterprise network.
The only disadvantage to the synchronized eDirectory model, when it comes to small branch offices, are the network costs associated with the synchronization. Typically, eDirectory works best when used on reliable, dedicated network connections between geographic sites. That usually means the leasing of dedicated lines, such as T1 links, between sites, so that the organization has its own private wide-area network.
For many organizations with small branch office sites, they could see real cost savings by using simple DSL or cable Internet connections from an ISP, rather than more expensive private WAN links. However, eDirectory was never designed to operate directly over the Internet (nor would we want it to, I suppose).
Bottom line is, LDAP authentication was introduced into Nterprise Branch Office to bridge the gap between the needs of the Central Office (eDirectory synchronization) and the needs of the small branch offices (semi-independent file and print services, using cost-effective Internet network connections).
Q7: "How does data replication work in Nterprise Branch Office?"
A7: One of the most attractive benefits of NBO is the data replication capability, which effectively replaces data backup at the remote sites. If you currently use the onerous method of individual servers at all your branch office sites, each having its own tape drive, tape backup software and backup schedule, tape media sets, and individuals who are responsible to rotate that tape media and check on the success/failure of the scheduled backup jobs, then you'd better give Nterprise Branch Office a serious look.
NBO data replication uses the RSYNC utility to periodically synchronize the data on the NBO server to a Central Office server. Like incremental backups, only new data on the NBO server is replicated during each replication cycle. Most administrators schedule the replication to occur during the night, effectively replacing the traditional tape backup solution. However, some administrators prefer to synchronize every 30 minutes - here is a capability that you won't typically find in a tape backup solution.
Now, the Branch Office data is effectively in two places - on the local NBO server, and replicated on the Central Office server. This provides data redundancy in the event of disk or server failure. Typically, the Central Office server employs the traditional tape backup solution - providing backups of the data backups. And this also provides what the replication solution does not provide: the ability to restore previous versions of files. Most network administrators have been called upon to restore (from tape backup) a version of a file or data set that was known to be good on, say, Thursday of last week.
For the small branch office with no trained IT personnel, the data replication solution means that we don't have to rely on the secretary or office manager to keep up with juggling tape sets or monitoring backup jobs. Both the data replication and the subsequent data backup can be managed and monitored by the Central Office IT staff.
Q8: "Why do I have to 'Prepare the Central Office' before installing Branch Office servers?"
A8: There are two main types of communication between the NBO server at the branch office site and the Central Office: LDAP authentication from NBO to eDirectory, and data replication from NBO to the central RSYNC server (these two functions can involve the same or different servers).
LDAP authentication occurs between the NBO server and a Central Office server that has access to eDirectory. This communication can happen over Internet links, so it needs to be secured using SSL. To configure the SSL security, we also need to have a functional Certificate Services infrastructure in the Central Office environment.
Likewise, data replication can also happen over Internet links, and thus also needs to be secured using SSL encryption. Also, the central RSYNC server must be configured to run the RSYNC utility in "daemon" (server) mode, and be configured to accept RSYNC communications from each individual branch office site.
For the LDAP authentication, if you are starting out with a spanking new NetWare 6.5 eDirectory tree at the Central Office, you probably will face very few challenges. However, I hardly know anyone in that situation. Most of us have NetWare environments that may have started with NetWare 4.x NDS, and subsequently upgraded to NetWare 5.x, 6.x and eDirectory. Have no fear - it will work out (with some work). But eDirectory is an absolute "must have" for the LDAP authentication.
The best recommendation is to setup a new, dedicated NetWare 6.5 server for the Central Office LDAP server. If you choose to use a NetWare 6.0 server, you will need to upgrade NICI and NMAS on the server (see "Steps to Configure the Central Office LDAP Server" in the NBO Documentation). This is because the NBO server utilizes Simple Passwords for authentication, and eDirectory utilizes Universal Passwords as a kind of "uber password" mechanism to synchronize all other types of passwords.
You will also need to extract the SSL certificates (also detailed in the documentation) from the Central Office server(s), in order to apply them to each NBO server for both LDAP authentication and data replication. This assumes that the PKI system is correct and functioning on the Central Office server. PKIDIAG is a handy utility to validate the PKI system (certificate chain) on a NetWare server.
Q9: "What are the cost benefits of Nterprise Branch Office?"
A9: First off, I'm not going to pretend to be a salesman. Any cost benefit for an NBO solution (or any other solution, for that matter) is going to be directly proportionate to how well that solution serves your specific needs, and then to how much the solution takes out of your budget in order to fill those needs. But here's an example:
For one of my customers, who needed to replace end-of-life NetWare servers at 46 branch office sites, he estimated the same-for-same replacement cost for each server at about $8,000 per server. This included servers with fully redundant disks and power supplies, each with its own tape drive, tape backup software and tape media sets.
However, a solution using Nterprise Branch Office utilized less costly, scaled-back server hardware, without tape drives, backup software and tape media. This solution came in at about $2,000 per server, plus the cost for NBO server licensing (which varies according to your Novell licensing agreement - commercial, government, education, etc.). The hardware savings alone justified the cost of NBO licenses.
The reason that the less-costly server hardware made sense, is that since the NBO server model provided easily-replaceable, "cookie-cutter" servers at each site, the less-redundant lower-end server hardware was warranted. If replacing a dead server involves a great deal of time, effort and expertise, then you want to make sure every subsystem in the server has built-in redundancy (as much as you can afford). If replacing a server suddenly become less of a hassle due to new server imaging and data replication strategies, then hardware redundancy is much less of a priority.
It should be said that we still retained hardware disk redundancy (RAID-1) in the low-end server models, but not the RAID-5 disks in the high-end server models. Also, the obvious savings for the tape drive, tape media, and backup software made a significant difference.
We did not attempt to calculate the labor costs for changing and storing data tapes, running cleaning tapes, and monitoring success/failure of tape backup jobs at each site, although that could be considered. My customer's main concern was that he currently had no reliable way to ensure that these tasks were being accomplished consistently at each site. So he was very happy to replace this traditional local tape backup solution with data replication, merely to be able to ensure that the job was getting done.
Also, for this customer, there were 10 additional small branch office sites that did not have servers at all. Most of his branch sites were T1 connected, but these 10 (newer) sites had DSL connections to the Internet, instead of the private WAN connections. The ability to have NBO servers that can use Internet connections for LDAP authentication and data replication allows him to place new servers in these sites, where traditional NetWare servers would not have worked well.
So the "how much does it cost?" question always has to be answered in the context of what are your own needs, does the solution fit your needs, and how much any other solution (which would also fit the needs) would cost instead.
UPDATE: Novell has recently opted to include Nterprise Branch Office 2 bundled with Open Enterprise Server (OES). So if you are (or want to be) an OES (NetWare 6.5 or OES Linux) customer, then there is no additional cost for licensing the Nterprise Branch Office product. See http://www.novell.com/products/openenterpriseserver/branchoffice.html.
Q10: "What are the disaster recovery benefits of Nterprise Branch Office?"
A10: First, let us clarify: the implementation of NBO is not a substitution for a complete Disaster Recovery plan, which is more of business solution, and not merely an implementation of technology. That said, let's examine the disaster recovery benefits.
The State of Montana has an excellent success story with the implementation of Nterprise Branch Office (see "State of Montana"). One of their main concerns in choosing a product to provide network services to nearly 200 offices spread across the state, was the ability to recover from server failures - since they are famous for lightning strikes all around the state.
They have reported that the ability to quickly image a new server, and restore its configuration settings, user accounts, and data from the Central Office server - and simply ship the new replacement server to the affected site - has dramatically reduced their costs for IT management, downtime, and travel.
I would like to interject a recommendation, which really applies whether you use NBO with data replication, or traditional tape backup solutions: a recovery solution is only as good as it is tested and verified.
More than once I have gone to a customer site to assist with data loss or even a total server failure, and found that their data backup solution did not serve them well. One customer did not bother to monitor the backup jobs, and for over a year was simply replacing blank tapes in his server's tape drive. One customer lost his email server, and although he was successfully backing up all the user data, he did not have the email agent (for open database files) for the backup software, and thus lost all of his company's email.
The same warnings apply, if you choose to replace tape backups with data replication. You should monitor the process, to ensure replications actually occur on schedule. And periodically, you should have a "fire drill" to ensure that you have the correct procedures in hand to recover a failed server.
I like to recommend to my customers to add a little bit of disaster recovery into the budget. With an NBO server solution, it's simple, and cost-effective. You need twenty-five servers? Buy twenty-seven. It is always wise to have a non-production server on which to test installations and configurations. And it can be handy when you need to troubleshoot problems (if you can replicate and solve a problem on a reference server, chances are that same solution is the fix for the production server). And keep at least one "cold spare" server on the shelf, ready to deploy in an instant, in case of a server failure at any site.
This is similar to the "N+1" redundancy that we employ all the time for server disks, power systems, and network equipment. Usually, with the hardware cost savings you can realize with NBO servers, buying a couple of extra boxes, to provide more rapid recovery in the event of server failures, is no problem at all.
Then you need to periodically stage a "fire drill". Just pretend that a particular site is down, and go through the process of re-imaging and restoring a server. The "fire drill" server doesn't actually need to be deployed to the site - this can all occur on an isolated test network. With NBO, you'll be surprised at just how easy it is to recover a server.
Q11: "Help! I made a change on my NBO server, and now it's no longer available. What can I do?"
A11: This problem recently occurred for one of my customers. If this is happening to you right now, you probably aren't leisurely reading this, but "sweating bullets" because you have a server down. Here is what happened:
The IT administrator needed to replace the Ethernet switch at a site with an NBO server, and sent a technician to the site to swap out the switch. But to accommodate the new switch, he wanted to set the network card settings on the NBO server from "auto-sense" to 100mb full duplex. That's reasonable. He accessed the server from the NBO Web administration portal, under IP Settings, and configured the NIC settings appropriately. Of course, the NBO server then re-cycled - but it never came back online on the network, even after a power down and reboot.
That's when he called for help. Fortunately, we had a reference server, with identical hardware (as above) at the Central Office, so we could repeat the configuration changes that he made on site. And what do you know, the exact same thing happened - the server didn't come back online.
What we could see on the reference server, since we were physically at the console, was why it didn't come back up. After about 5 minutes of trying to start the networking, it dropped out its normal locked NBO console to the traditional NetWare system console screen, and we could see that the NIC driver attempted to load. But switching to the logger screen, we found that it was waiting for a user response as to the speed at which to load, we simply had to select "100mb Full Duplex" (option 2), and then the server continued to boot up normally.
Fortunately, we still had the technician on-site, and we simply had him switch to the logger screen, type "2" for the 100mb full duplex option, and the server booted up normally, and we were then able to access it remotely.
A little investigation turned up the cause of the problem. We unlocked the NBO console (see Question 12) on the remote site server, so that we could access all server screens in NetWare Remote Manager (https://<serverIPaddress>:8009). Then we could open INETCFG on the NBO server to really see what was happening with the networking.
It turns out that the model of server he was using had a Broadcom embedded NIC. This was perfectly fine, except that the NIC driver uses slightly different load commands than most standard NICs. For example, the standard load command to load a NIC driver at 100mb full duplex would be (as seen in View Configuration / LAN Board Commands):
LOAD <DriverName> NAME=<BoardName> FRAME=Ethernet_II SLOT=<SlotNumber> SPEED=100 DUPLEX=2
However, this particular Broadcom NIC driver utilizes a slightly different command to do the same thing:
LOAD B57 NAME=B57_1_EII FRAME=Ethernet_II SLOT=<SlotNumber> SPEED=100FD
Since the driver does not understand the standard load commands (SPEED=100 DUPLEX=2), the server paused on bootup, waiting for the user to supply the proper parameters. This would apply to all models using the Broadcom (B57.LAN) NIC. The NBO Web administration portal merely inserts the standard commands into the INITSYS.NCF, whereas INETCFG uses the correct format pulled from the driver's information file.
Thus the solution here was to use INETCFG to correctly configure the server NIC, only because of the non-standard commands for the Broadcom driver.
Q12: "I need to access the system console on my remote NBO server. How can I unlock the console remotely?"
A12: Normally the NBO server console is locked, which is desirable for remote sites, where you don't want anyone to monkey around with an open server console. But there may be times when you actually want to see the (NetWare) server console, in order to load antivirus or other software on the server, or (as above) to change specific server settings. How to do that when you are remote?
By default, NBO servers provide administrative access via HTTPS, which allows both the NBO Web administration portal and NetWare Remote Manager to work, and also via SSH. To unlock the NBO server console, we need to first access the server via SSH.
This can be done using any SSH client - I prefer PuTTY (see PuTTY). Using the SSH client, specify the server IP or DNS name, default port 22, and SSH as the protocol. You can usually name and save the session, for easy access later. Open the session, and you are prompted for a username and password - use the server's local supervisor account. This gives you remote SSH access to the NBO console on the server.
To unlock the NBO server console, type "login supervisor" and provide the password again, and then type "unlock" and then "y" (yes). This opens up all of the server consoles, which now can be accessed either directly on the server (locally), or through NetWare Remote Manager (remotely) under Manage Server / Console Screens.
The SSH client session only has access to the NBO console screen. When you are finished with accessing the other server console screens, type "lock" at the NBO console screen, and the server returns to its normal (locked) mode. Then close the SSH session.
NOTE: If you SSH to an already unlocked NBO server, the session will take you to the server's system console screen.
Q13: "I am repeatedly getting IO errors on the initial data synchronization for my remote NBO servers, but it seems to go OK after that. What could be happening?"
A13: The RSYNC remote-update protocol often has issues when the network connections aren't perfect. With the initial synchronization of the NBO server, all of the server's data (DATA: volume), as well as configuration information (ADMIN: volume) will get transferred from the NBO server to the configured replication location on the Central Office server. After the initial synchronization, only new and updated files are transferred, so the synchronizations can happen very speedily.
In an ideal world, when a significant amount of data is on the NBO server (as in data migrated from an older server) you would perform the initial sync on a LAN to the Central Office server. However, in some implementation scenarios, that isn't too practical, since the migration to the NBO server may take place at the remote site.
Another recommended option is to perform a full tape backup of the data on the old server prior to migration, take that tape to the Central Office, and then restore the data from tape to the synchronization location for that site, prior to performing any RSYNC transfers. This allows you to have most of the data in place on the Central Office server, so that the initial RSYNC synchronization behaves more like an incremental sync.
If the WAN connection to the remote site is extremely slow or unreliable, one of the two options above may be your only choice. Some admins have successfully used a variation of the tape backup method: using a removable USB hard drive to transfer the data from the remote site server to the Central Office server.
Another option is to set the NBO server to replicate every 30 minutes or so, until all of the data finally gets synchronized. This way, if the replication stops due to some network error, it simply picks up where it left off on the next synchronization cycle, until all the data is synchronized. At that point, you can set the replication schedule back to the normal once-a-day timing to fit in with all of the other NBO servers.
Some admins use this "keep trying" method to achieve full synchronization over a weekend period, and then they can set the replication schedule to their desired time.
Using the NBO Web management portal, select "Replication" under Service Configuration in the left navigation panel, click "Set Replication Schedule", and then click the "Enable interval scheduling" radio button, and set the interval to 30. To return to once-a-day replication, click "Enable timer scheduling" in the same window, and set the begin and end times (in 24-hour format).
Q14: "I think my remote Branch Office servers are successfully replicating, but it's hard to tell. Isn't there an easier way to monitor the data replication?"
A14: The default methods of monitoring data replication on NBO servers leave a bit to be desired. One method is to open the NBO Web management portal every day, open the Replication page, and click on the "Download Replication Log" button. This can be referred to as "point management", since you would have to perform this same task for each of your NBO servers individually. Even then, the default information is less than - well - informative:
2006/03/08 23:00:41  ADMIN: volume
2006/03/08 23:00:57  DATA: volume
If you wish to see more information about the replication process, you will have to set the replication logging to verbose. Unfortunately, this option isn't in the GUI Web management portal, so this change needs to be made at the console. At the Novell Branch Office console screen on the NBO server, first "login supervisor" and provide the password, then type:
set replication verbose=yes [Enter]
To apply this change, type:
The next replication will use verbose logging, so you can see a list of the files that are transferred. If you need to make this change remotely to the NBO server, see Question 12 on how to remotely access the server console.
To make this verbose logging the default when imaging new NBO servers (see Question 5), then include this same "set replication verbose=yes" under the TAPSettings_Backup section of the autoload.nbo file.
So far, we've really only solved half of the problem - the client (NBO server) side of the replication. To monitor multiple replications, we really don't want to have to use "point management" to individually monitor each of the remote servers. The smart thing would be to monitor the data synchronization at the Central Office, where all the replicated data ends up.
Here again we want to modify the defaults to suit our own needs. By default, the Central Office RSYNC server will log all activity of the RSYNC daemon to a single file, named rsyncd.log, in the same folder with the rsync.nlm file. However, when we have multiple (NBO) clients replicating to one RSYNC server, that single rsyncd.log file can quickly become huge, and immensely confusing.
One of the first things I do is to move the RSYNC log files to their own directory, such as SYS:\rsync\logs. This requires a modification of the rsyncd.conf file to read:
log file = SYS:/rsync/logs/rsyncd.log
NOTE: If your Central Office RSYNC server is running on a cluster, replace SYS: with the appropriate cluster volume.
The next thing I do is to create a simple script to cycle the logs on a daily basis, so I can see a fresh new log for each day of the week. We can use CRON or the Scheduler in NetWare Remote Manager (https://<serverIPaddress>:8009) to schedule this script to run every day:
REM COPYLOGS.NCF REM *** Increment RSYNC logs 1-7 *** load toolbox.nlm delay 3 echo Incrementing RSYNC logs ... copy sys:\rsync\logs\rsyncd6.log sys:\rsync\logs\rsyncd7.log /q > sys:\null.txt copy sys:\rsync\logs\rsyncd5.log sys:\rsync\logs\rsyncd6.log /q > sys:\null.txt copy sys:\rsync\logs\rsyncd4.log sys:\rsync\logs\rsyncd5.log /q > sys:\null.txt copy sys:\rsync\logs\rsyncd3.log sys:\rsync\logs\rsyncd4.log /q > sys:\null.txt copy sys:\rsync\logs\rsyncd2.log sys:\rsync\logs\rsyncd3.log /q > sys:\null.txt copy sys:\rsync\logs\rsyncd1.log sys:\rsync\logs\rsyncd2.log /q > sys:\null.txt copy sys:\rsync\logs\rsyncd.log sys:\rsync\logs\rsyncd1.log /q > sys:\null.txt copy sys:\rsync\logs\empty.log sys:\rsync\logs\rsyncd.log /q > sys:\null.txt flag sys:\rsync\logs\rsyncd.log -R > sys:\null.txt unload toolbox.nlm
This script increments the rsyncd.log file each day, using TOOLBOX.NLM on the server to perform the file copies. Thus rsyncd.log records this day's activity, rsyncd1.log is from 1 day ago, and so forth. Since you wouldn't want to run this script during an active RSYNC session, be sure to schedule it to run during non-replication hours. You will also have to create the (empty) "empty.log" file to start out. (Also see: a better way to handle RSYNC logs).
The level of detail in the central rsyncd.log file is governed by the setting within each module of the rsyncd.conf file (each module corresponds to a remote location). If the setting for a location is:
Then we will see the same level of verbosity as we previously had set on the client side. You would also want to modify the schedule of replications from each of the remote sites so that they occur at separate times, say, 15 or 30 minutes apart. This way the central log file entries are grouped together by each site, making it much easier to read.
The final issue now is event notification. There currently is no type of automatic event (success/failure) notification built into Nterprise Branch Office for data replication. One "quick and dirty" method used by my customers, in conjunction with the log file incrementing above, is to use NetWare login script commands to display the current rsyncd.log file each time the administrator logs in.
The administrator merely adds this segment onto his existing Central Office container login script:
IF MEMBER OF ".NBOADMINS " MAP ROOT X:=\\RSYNC_Server\SYS\rsync\logs EXIT "notepad.exe X:\rsyncd.log" END
Of course, create the "NBOADMINS" group, and add to it the appropriate members. This group would need to have R-F rights, at very least, to the SYS:\rsync\logs folder on the Central Office server.
This script opens the current log file in Notepad, so the admin can quickly view the details of the recent synchronizations. Then the admin can just close the log file. Of course, previous log files can be viewed by simply opening them directly from the \logs folder. The log files themselves are typically backed up to tape with the regular backup solution employed at the Central Office, and then recycled every seven days.
Q15: "Can I salvage deleted user files on my Nterprise Branch Office server?"
A15: Yes, you can. Since the underlying platform for the NBO (2.0) server is NetWare, and the file system is NSS, the salvage utility is a built-in capability.
In order to exercise the salvage capability, you need to have supervisor (S) rights to a file or directory, as always. Users have the S right to their home directory by default, so if they are accessing the NBO server via the Novell client, they merely have to right-click the parent folder in which they have deleted a file, select Salvage Files, and highlight the file (or files) they wish to salvage, and click Salvage File.
If the user has deleted a directory or folder, with files and subfolders, only the selected folder will be salvaged - the salvage utility does not perform a recursive salvage operation. In order to restore the subfolders and files, the user will have to right-click the newly-salvaged parent folder, select Salvage Files, and highlight the files and folders, and click Salvage File (or click Salvage All to restore all files).
Of course, the network administrator can also salvage files for users by logging in as Supervisor with the Novell client, and mapping a drive to the NBO server's DATA:\users\remote folder, and performing the same steps for the user.
If you are accessing your files on an NBO server using NetStorage File Access in the Virtual Office Web portal, you can still salvage files, using a little different procedure. This time open the parent directory in which you deleted a file, and then select View | Show Deleted Files from the menu. The deleted files and folders appear in red and crossed-out. Click the file or folder you wish to salvage (so that a checkmark appears next to it), and select File | Undelete from the menu, and OK to confirm (or rename).
As above, the salvage (or undelete) operation is not recursive, so users will have to repeat this procedure for files or subfolders within a salvaged folder. To return to the normal file view within NetStorage, select View | Hide Deleted Files from the menu.
Q16: "I deleted and re-created a corrupt user account on my NBO server, and now it looks like the user's home directory is completely empty. What happened?"
A16: For this NBO administrator, there is good news. Sometimes, whether in eDirectory or any other directory, user accounts can become corrupt, and the easiest (and sometimes only) solution is to delete and re-create the user account. In this instance, the user account in the NBO server's eDirectory appeared to be corrupt, not allowing the user to log in, but the matching Central Office user account functioned normally.
The administrator simply deleted the NBO user account, using the Web-based NBO management portal, and had the user login to the NBO server again, which automatically triggered the user access provisioning mechanism. The NBO server made an LDAP query to the configured context in the Central Office tree, validated the user's name and password, and replicated the user (again) to the NBO server's tree. A new user home directory was automatically created for the user. This is the exact process that occurs when the user logs in to the NBO server for the very first time.
What is not immediately apparent here is what happens when the administrator deleted the original NBO user account. In a traditional NetWare environment, the user account is deleted, and nothing happens to the user's home directory, producing an "orphaned" home directory. Usually this is desirable, since we don't necessarily want to automatically delete a user's data when we delete a user account - we can do that manually, if we so choose, at a later time.
The NBO management portal does something similar when a user account is deleted. It automates the process of "documenting" a user deletion, while keeping the user data intact, by simply renaming the home directory of the deleted user. Thus the home folder of user "JohnDoe" becomes renamed to "JohnDoe (User Deleted Mar 23, 2006 3.51.41 PM)". All of the data is still intact in this renamed folder.
In a traditional NetWare environment, when we have to deleted and recreate a user account, we would have to manually have to re-associate the user to his home directory by changing the values in the Home Directory volume and path attributes (Environment tab) on the re-created user account. Then we would have to manually reset the trustee rights on the user's home folder itself to add the re-created user as the trustee.
With the NBO server, once we deleted the old user account, the home folder is automatically renamed, as above. All we really have to do is rename that folder to its original name (i.e., "JohnDoe"), and then have the user log in again to trigger the automatic user provisioning on the NBO server. When the user provisioning mechanism detects that the home folder already exists (in DATA:\users\remote\), then it merely assigns the new user account the trustee rights (SRWCEMFA) to the existing folder.
Since the admin deleted the original user account using the NBO management portal (https://<serverIPaddress>:2222), it would be simple to go the File Access dialogue (second button on the top menu bar), browse to DATA:\users\remote, highlight the automatically renamed home folder, and select "Rename" on the left menu bar, to remove the added "(User Deleted ?" portion of the directory name. Then merely have the user login to the NBO server, to be automatically provisioned, this time with the same home directory that he had before.
The easiest solution, therefore, once the problem of the empty home directory was discovered, is to again delete the user account in the NBO management portal, rename the original home folder, and have the user login again to be automatically provisioned, this time with his original home directory intact.
Q17: "A user deleted a folder on the NBO server, and I need to restore that folder from the Central Office server. Exactly how do I do that?"
A17: This scenario is the typical "OOPS" factor that network administrators must deal with on a day-to-day basis. We are all human, and with the ease of today's point-and-click computing, it is also all too easy to accidentally click on, and delete the wrong file or folder.
We'll assume here that the deleted folder, and its files, are not recoverable using the salvage utility (see Question 15) for some reason. Also, if there are only a few files to restore, it would be easy for the admin to merely map drives simultaneously to the NBO server and the RSYNC server, and directly copy the (replicated) files back to the NBO server. But if we are unsure as to the extent of the files deleted or corrupted, we may have to resort to performing a restore operation.
We have been (hopefully) replicating the Branch Office server's data to the Central Office every night to provide for data redundancy, and now it's time to collect on that investment.
The data replication that we schedule for NBO servers, through the NBO Web administration portal, uses RSYNC to transfer new and updated files on each replication cycle from the NBO server to the configured replication location on the Central Office server. This time we want to reverse that process, and use the same RSYNC mechanism to replicate missing files back to the NBO server from the Central Office. We do this by executing the RESTORE command on the NBO server.
Unfortunately, the RESTORE command is not a "targeted" utility, where we can specify exactly which files or folders we wish to restore. Also, it can only restore from the current replication set - it does not have the archiving capabilities to allow you to restore a previous version (say, from two weeks ago) of files or folders. Executing a RESTORE command will bring back any files deleted on the NBO server since the last replication, unless those files already exist, or are newer.
IMPORTANT: All user files must be closed on the Central Office volume (source) when restoring data back to the Branch Office appliance (target), or else the attempted restore operation will fail for that volume.
To launch the RESTORE operation on an NBO server, login to the NBO console (as Supervisor), and simply type RESTORE at the console. If your NBO server is remote, see Question 12 on how to remotely access the console. Users will have to wait until the RESTORE operation completes before accessing their files.
Any existing files, or any files newer than the last replication are not overwritten. Only files that exist at the Central Office replication location, but do not exist at the NBO server, will be replicated back to the NBO server.
Normally, the replicated files on the Central Office server or cluster are also regularly archived to tape, providing not only backups of the backup data, but also archived versions of files. This allows an admin to restore previous versions of files to the Central Office replication location, and subsequently replicate those back to the NBO server using the RESTORE command.
TIP: This works equally well for "restoring" files that were never originally on the NBO server. If the admin wishes to put new files, such as document templates, on his remote NBO server, then he only needs to copy them to the Central Office replication location, and then run the RESTORE command. This works best if done immediately after a replication session, to ensure that all other files are already in sync.
NOTE: The default behavior for the RESTORE command is "update" - meaning that if there are newer files on the NBO server (target) than on the Central Office (source), then they will be left alone. But if for some reason you wish to replicate the files exactly as they are on the Central Office replication location, you can change this behavior to "delete" - meaning to overwrite all files on the target, and delete any files that do not exist at the source. To do this, at the NBO server console type:
set replication restore delete=yes apply
To return to the default behavior (update), type:
set replication restore update=yes apply
Q18: "I need to test the disaster recovery feature of Nterprise Branch Office, so I want to stage a 'fire drill'. What steps do I have to take?"
A18: This question shows a good amount of wisdom. Every backup solution is only as good as it is tested and verified. It is also good to have the process well-documented, from start to finish, so that in the event of an emergency, you are not scratching your head, and trying to remember what steps to take.
This "fire drill", or dry run of a server recovery, does not have to happen during off-hours, or to have any effect whatsoever upon the production server. We can perform the entire process in a lab environment, and document the process as we go along.
In an ideal world, you would have a spare server - an identical model to your production servers, on the shelf, ready to be deployed at a moment's notice. In the real world, we sometimes have to use what we can get to do the job that we need to do. But for an NBO server recovery, it is not at all necessary to use identical hardware, but merely hardware that is compatible with Nterprise Branch Office (most server-class models are; in fact, most PC-class models will do for a test).
The easiest and surest way to find out if the server hardware is compatible, is to boot the computer to the NBO imaging CD, and attempt to image the server. If the drive controller and network card are recognized and matched to the correct (NetWare) drivers, then it will successfully image in about 20 minutes (on a newer computer).
The minimum requirements for the NBO server are: Pentium III (or equivalent) processor, at least 512 MB RAM, and a 10 GB hard drive (plus room enough for any amount of data you plan to restore to the server).
The safest route for performing a test server recovery is to first build a test NBO server in the lab environment, create a test container in the production tree with test users, and configure the test NBO server to replicate to that container with the LDAP connector. You probably did (or should do) this before you deployed your first production NBO server. Login to the test NBO server with the test user accounts, and put some test data in their home and shared directories. Set up data replication to the Central Office server, and perform (and verify) a replication. And then simulate a disaster - wipe the test server's hard disk.
The server recovery starts with the configuration file for the server to be "recovered". At every replication (you are replicating your servers, aren't you?), the configuration file is copied to the Central Office (RSYNC) server. Thus if the NBO server is named "NBO-TEST", and the replication is configured to go to VOL1:\RSYNC\NBO-TEST on the Central Office server, then the NBO configuration file will be found at:
Copy this file to a floppy, renaming it to "autoload.nbo". This will be used during the server imaging process to restore all of the configuration settings to the server. If the server will be restored in its original LAN environment, no changes will be needed to the configuration file. If you need to recover the server on a different LAN, you need to edit the file, and use a search-and-replace function to replace all instances of the server IP address to match the subnet of the target LAN environment. Also replace the gateway settings under TAPSettings_Gateway, to specify the correct router address on the recovery network.
You will also need the exported SSL certificate that was used to provide authentication for the LDAP connector. This is the same certificate file used to originally setup the NBO server. If you don't have that certificate file handy, it can also be retrieved from the replication area on the Central Office server, at:
NOTE: If you specified secondary or tertiary LDAP authentication sources on the NBO server, the corresponding certificate files will be lsmldap2.der and lsmldap3.der.
The recovery computer needs to be set to boot from the CD first, to avoid errors when attempting to boot to the floppy. The recovery process starts merely by booting the computer to the NBO imaging CD, and placing the floppy with the autoload.nbo configuration file in the floppy drive.
Upon booting, the imaging software prompts you that it is about to image the hard disk; reply (Y) yes. If you previously installed NBO on this computer, you might also see the options to (P) preserve existing data, or (D) delete any existing data (choose delete).
After about 10-20 minutes (on newer machines), the server beeps at you, letting you know that the imaging process is completed. On the server console is the usual license agreement verbage, so answer (Y) yes to both agreements. The server then parses through the autoload.nbo file on the floppy to apply the specified settings, and then automatically reboots itself.
Now the server, if the settings were specified correctly in the autoload.nbo file, is ready to be accessed through the NBO Web administration portal.
All of the server configuration settings were pre-supplied and set by the autoload.nbo configuration file. However, there are some things that cannot be contained in a text file, such as the SSL certificates, so now these need to be configured correctly.
Access the NBO server from the NBO Web portal (https://<serverIPaddress>:2222), using the IP address specified in the configuration settings file. Accept the server certificate, and login as supervisor, with a blank password.
Now we can verify the NBO server configuration, and re-establish its secure authentication. In the NBO Web administration portal:
- Verify IP address settings under "IP Address"
- Verify LDAP authentication settings under "Authentication Sources". Click LDAP, and ensure "Enable LDAP" is checked
- Verify the IP address of Primary Host. Click "Upload Certificate", click "Browse" and browse to the file location of the exported SSL certificate file, and click "Upload"
- Verify the LDAP context for the server (e.g.: ou=NBOTEST,o=ACME)
- Click OK, and click "Apply All Settings" on the Authentication Sources screen
- Verify NTP server settings under "Date & Time"
- Verify time zone settings under "Time Zone"
- Verify DHCP service settings under "DHCP"
- Verify replication (RSYNC) settings under "Replication". Ensure "Enable Replication" is checked, verify the Central Office server IP address, ensure "Enable SSL" is checked, and click "Authenticate to Central Office". Input the admin name and password, and click "Authenticate"
Now the NBO server is reconfigured to communicate securely to the Central Office, and has all of its settings restored. Now we want to restore all of the data that was replicated to the Central Office server back to the NBO server.
To restore the server data, merely go to the NBO server console, type "login supervisor" (still with blank password), and type the command "RESTORE".
The server now performs a reverse synchronization of the replication it normally performs. All data which was replicated from the ADMIN: and DATA: volumes that is stored at the Central Office (RSYNC) server (in the replication directory configured for this server) will be replicated back to the NBO server.
The NBO server console gives updates on the restore progress, and notifies you when it is complete. When it is finished, you can test login of the test user accounts, and verify that their data is indeed restored. If the LDAP settings and certificates were properly accomplished (as above), the users should be able to login, and their user account settings and passwords will be replicated again from the configured container in the Central Office eDirectory tree. Since the Supervisor account is local, not synchronized, we need to reset the supervisor password, using the NBO Web Administration portal:
- Click on the User Management icon on the top navigation bar in the NBO Administration Web portal, and highlight the Supervisor user under the appusers container.
- Under Manage in the left menu page, click on Modify User/Group/Context. Highlight the contents in the New Password field, and enter the new password. Repeat for the Confirm New Password field, and click OK.
This action will prompt you to login again to the NBO Administration Web portal with the new Supervisor password.
That's it! At this point the server restoration is complete. Now wasn't that easy?
What Works for You
I have developed these answers from my own experience with the Nterprise Branch Office product. In essence, these answer reflect what "works for me" in the network environments in which I have tested and deployed NBO.
Obviously, you have to do what works for you, in your own network environment. If you've read along this far in this article, it's apparent that you have some real interest in the Nterprise Branch Office product.
Since every network and every organization is going to be somehow different, with different requirements and considerations, I wouldn't dream of telling you exactly how you should do it in your own situation. You may have some good ideas of your own! So put your two cents in - responses are welcomed!
If you have other questions about Branch Office or would like to share your experiences, send them in here.
Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com