AppNote: Using DFS in a Clustered Environment
Novell Cool Solutions: AppNote
By Hans-Robert Vermeulen
Digg This -
Updated: 1 Jan 2005
- The DFS Management Context
- The VLDB Service
- Client behavior
- Client 32
- CIFS Client
- NetStorage Client
- NetWare Servers
- The Test Environment
- Configuring DFS on a Cluster
- Creating the DFS Management Context
- Preparing LDIF files for the Cluster Resource
- Create a StartVLDB.NCF File
- Modifying the Cluster Resource
- DFS Clustering Quick Configuration Guide
In NetWare 6.x, Novell Distributed File Services (DFS) allows you to develop a logical view of your storage.
DFS provides you with the option to create smaller Pools and volumes while presenting them as a single volume to end-users. This minimizes the time required for backup, restore and repair actions and will minimize overall disaster recovery time.
DFS uses a Volume Location Database (VLDB), which maps the physical location of NSS volumes on all servers in the tree.
By itself, DFS offers the possibility to copy the VLDB database to a second server for redundancy and availability, but there are situations where this isn't enough, or not desired.
For instance, your DFS Management context is high up and numerous clusters and servers are depending on the VLDB Database. In this case, you could cluster enable VLDB to have even more availability of the VLDB database. You even could run a VLDB database both inside and outside of the cluster.
Here I will describe how to get the VLDB Database running as a clustered resource, effectively allowing more flexibility in server assignment.
The DFS Management Context
Before you create a DFS junction, you must create at least one DFS Management Context at an O or OU level in the Novell eDirectory tree where you want to create the junction. We will create a DFS Management Context at the cluster level, being used exclusively for our clustered environment.
When you create a DFS Management Context, you specify which servers run the VLDB service and hold the actual database.
The VLDB Service
VLDB isn't cluster aware. Although plans have been uttered to make it cluster aware, at present it is not.
Currently, at startup, VLDB.NLM will check the actual NCP servername. If this servername is not in the DFS Management context list, VLDB.NLM will error out. This is one of the reasons why VLDB.NLM is not cluster aware. We will see the result of this in our configuration.
The Novell Client 32 will first read the junction, as this is just a file on the volume. The information in the junction gives the client an eDirectory path for the management context. In the management context, the list of replica servers is read. The client will start with the first server in the list and try to connect. Once a connection has been established, a VLDB query is sent to that server. If the VLDB responds, client 32 will exit and continue to query the VLDB for the junction information. The response from the VLDB provides Client 32 with the server and volume that the junction represents. The information (physical path) is passed back to Windows. If the VLDB service is unavailable on the first server, the next server will be tried sequentially.
To increase performance, a replica containing the DFS management context should be placed on all servers capable of hosting the VLDB cluster resource. In most environments this will be the cluster container itself.
If the replica is not local, the client will contact a different server hosting the DFS Management context. This server should be local to the user and not across a WAN. The information received from the DFS Management context is cached at the client.
Note: DFS support is included from the 4.91 client on, however Novell recommends using the latest support pack.
When hitting the VLDB Management context through CIFS, the CIFS client will randomly choose a server from the list.
The DFS behavior of the NetStorage client is identical to the the behavior in Client32. The NetStorage DFS code was actually used for Client32 as well.
NetWare servers also act as VLDB clients when NSS volumes are created or deleted. Information on the volume is stored in the VLDB database.
The Test Environment
In this report we will focus on DFS inside a single NetWare 6.5 cluster environment. We are using the following simple eDirectory structure for our test environment:
Note: All cluster nodes have been installed in the CL1 container. In most environments, all cluster nodes will be either in a separate Cluster container, which is a preferred design, or in a services container. If more servers exist in the services container, alongside the cluster nodes, the DFS Management Context created will apply to those servers as well.
On our cluster we have created three Cluster resources, ISCSI1, ISCSI2 and VLDB, all of which are cluster enabled volumes.
The VLDB volume will be used to store the VLDB database normally stored in SYS:/ETC. To store the database, a VLDB directory has been created on the VLDB volume. The volume can be as small as 50MB.
ISCSI1 and ISCSI2 will be used to test the DFS junctions. ISCSI1 has a DATA junction, pointing to the root of the ISCSI2 volume that contains a DATA1, DATA2 and a DATA3 directory.
Configuring DFS on a Cluster
First, we will create the DFS Management Context in our CL1.SERVICES.NOVELL container:
- Open ConsoleOne and browse to the CL1 container.
- Right-click the CL1 context in the eDirectory tree, then click New > DFS Management Context, or use the "Create DFS Management Context" button in the toolbar.
Select the server currently running the VLDB cluster resource volume and click the Right-arrow to move it into the Selected list.
Note: At this time it is not possible to select the Virtual server as the selected resource. We will change this configuration later.
In our case, SERVER2 is currently running the VLDB volume.
- Change the default Database Location from SYS:\ETC to VLDB:\VLDB and unmark the "Load NLMs automatically when the server restarts" checkbox.
- Click Finish.
DFS has now created the VLDB database in the VLDB volume and added an attribute for SERVER2 as the VLDB host on the CL1 container, the root of the DFS Management Context.
Next, copy the following files from SYS:ETC of the original server (SERVER2 in our case) to SYS:ETC of every cluster node involved.
Preparing LDIF Files for the Cluster Resource
On startup, VLDB checks the physical NCP server of the node it is started on. If this NCP object is not part of the DFS-VLDB-HOSTS attribute, the VLDB service will fail to start.
This means we only want the server it will currently run on to be part of the DFS-VDLB-HOSTS attribute.
We do not want the inactive cluster nodes to be part of the attribute, as this would result in unwanted additional network traffic, caused by clients trying to access the VLDB database on the wrong server.
The following steps are required to allow VLDB.NLM to load on other cluster nodes.
For each cluster node required to run the VLDB resource, create a <SERVERNAME>.LDIF file in the VLDB directory on the VLDB server that will remove other servers and add the current server. The name of the file must match the SERVERNAME.LDIF format exactly!
This file should have the following contents:#This LDIF file is used for DFS in a cluster environment. Version: 1
dn: <DFS Management context> changetype: modify delete: DFS-VLDB-Hosts DFS-VLDB-Hosts: <The second server> DFS-VLDB-Hosts: <The third server> - add: DFS-VLDB-Hosts DFS-VLDB-Hosts: <This server> -
Note: All entries are in LDAP format of CN=Leaf,OU=Container,OU=Container,O=Org
Note: The "Version: 1" line is part of the LDIF file, do not change this. If you need a version number of your own, start with a # to make it a comment line.
In our scenario, we end up with the following LDIF files:
#This LDIF file is used for DFS in a cluster environment. Version: 1 dn: ou=CL2,ou=Services,o=Novell changetype: modify delete: DFS-VLDB-Hosts DFS-VLDB-Hosts: cn=Server2,ou=CL1,ou=Services,o=Novell - add: DFS-VLDB-Hosts DFS-VLDB-Hosts: cn=Server1,ou=CL1,ou=Services,o=Novell -
#This LDIF file is used for DFS in a cluster environment. Version: 1 dn: ou=CL2,ou=Services,o=Novell changetype: modify delete: DFS-VLDB-Hosts DFS-VLDB-Hosts: cn=Server1,ou=CL1,ou=Services,o=Novell - add: DFS-VLDB-Hosts DFS-VLDB-Hosts: cn=Server2,ou=CL1,ou=Services,o=Novell
Note: For more information on LIDF formats, see TID 10052014.
Create a StartVLDB.NCF file
Now we need to be able to import those files depending on what server we start it from. This requires a special NCF file. Create a StartVLDB.NCF file in the VLDB directory with the following contents:
%if !env %servername=="" then cmd ice -b -S LDIF -c -f VLDB:VLDB/[env %servername].ldif -D LDAP -s <cluster resource IP> -p 389 -d <user name> -w <password> delay 20 VLDB
The user name and password need to be specified in clear text. It is therefore recommended that you create a user with Compare, Read, Write rights to the DFS-VLDB-HOSTS attribute only.
The <cluster resource IP> can only point to the cluster resource IP address if this specific object is in a DS replica on the server.
The delay is required and might need tuning if the VLDB service is run in a larger eDirectory environment. The DFS-VLDB-HOSTS attribute is a Synchronize Immediate type. This means a Sync will be triggered of that attribute within 10 seconds after the change is made, however it depends entirely on the environment when the change is finalized on all replicas. It is best to have the object local to the server, but still it is recommended to put in a delay for the object to be able to synchronize first.
You can further enhance the functionality by adding the following line:
%if !loaded VLDB.NLM then cmd SEND "VLDB DID NOT START CORRECTLY" TO Admin
This will send an alert to Admin (if logged in to that server!) notifying him something went wrong and VLDB is not loaded correctly.
Or if you experience problems with timely synchronization of the attribute you can try again with a larger delay:
%if !loaded VLDB.NLM then cmd Delay 30 %if !loaded VLDB.NLM then cmd VLDB
In our example the file looks like this:
%if !env %servername=="" then cmd ice -b -S LDIF -c -f
VLDB:VLDB/[env %servername].ldif -D LDAP -s 192.168.2.161 -p 389 -d cn=admin,o=novell -w novell delay 20 VLDB
Modifying the Cluster Resource
VLDB.NLM will only need to run on the node actively hosting the VLDB resource. Therefore we will need to add VLDB.NLM to the load and unload script of the VLDB resource.
Note: Make sure the VLDB.NLM is not loaded from any Autoexec.ncf file.
- Open the VLDB_SERVER cluster resource and add VLDB to the Cluster Resource Load script.
nss /poolactivate=VLDB mount VLDB VOLID=254 CLUSTER CVSBIND ADD CL1-VLDB-VS 192.168.2.161 NUDP ADD CL1-VLDB-VS 192.168.2.161 add secondary ipaddress 192.168.2.161 VLDB:\VLDB\StartVLDB.ncf
UNLOAD VLMSG UNLOAD VDQAD UNLOAD VLDB del secondary ipaddress 192.168.2.161 CLUSTER CVSBIND DEL CL1-VLDB-VS 192.168.2.161 NUDP DEL CL1-VLDB-VS 192.168.2.161 nss /pooldeactivate=VLDB /overridetype=question
Note: The "VLDB EXIT" command could also be used, however situations were found during test in which "VLDB EXIT" would not unload and cause the cluster resource to go comatose.
To test your configuration, create some junctions and test the fail-over of the volume containing the junction, the volume the junction is pointing to and the VLDB resource.
DFS Clustering quick configuration guide
- Create a small Pool/Volume for the VLDB database and name it VLDB.
- Create a cluster resource from the VLDB volume and assign it to the appropriate nodes.
- In C1 right click the container where all servers/volumes of DFS are located, typically the cluster container.
- Select New > DFS Management Context.
- Select the server currently running the VLDB cluster resource volume and click the Right-arrow to move it into the Selected list.
- Enter VLDB:\VLDB as location for database.
- Unmark "Load NLMs automatically when the server restarts" checkbox.
- Copy the following files from the "original" VLDB servers SYS:\ETC directory to the VLDB:\VLDB directory: VLDBCFG.DAT, VLDBCFG.SAV and VLDBPATH.DAT.
- Prepare the required LDIF files for the cluster resource.
- Create the StartVLDB.NCF file.
- Add "VLDB:\VLDB\StartVLDB.ncf " to the Cluster Resource Load script.
- Add "UNLOAD VLMSG" "UNLOAD VDQAD" and "UNLOAD VLDB" to the beginning of the Cluster Resource Unload script.
- Unload VLDB from your "original" VLDB server.
- Test your configuration.
Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com