Trustees Problems in Cluster Services

Trustees Problems

Each server tracks file system trustees based on the Object ID of the trustee. With NetWare^® 5, each server has its own view of Object IDs; any given NDS^® object will have a different Object ID depending on which server is viewing the object. Because Object IDs are server centric, when you move a volume from one server to another, Novell^® Cluster Services^TM must adjust the Object IDs of each trustee based on the new server’s perspective.

The TRUSTMIG.NLM module is used to adjust object IDs.

Note: NetWare 6 uses globally unique Object IDs, so TRUSTMIG.NLM is not required with NetWare 6.

TRUSTMIG.NLM normally functions well in accounting for and updating file system trustees. However, there might be times when something happens that causes trustee errors. Some of the more common problems include the following:

An administrator manually moves a volume to another server without using the TRUSTMIG utility.
A server fails in the middle of the trustee migration process.
A trustee assignment is changed when TRUSTMIG is not monitoring the volume.
An NDS object that has file system trustee assignments is deleted or moved when TRUSTMIG is not monitoring a volume but the volume is mounted. This could be caused by the volume resource being offline.
NSS corruption occurs on the volume.
An administrator has removed the TRUSTMIG command or the WATCH parameter from the load script.

Understanding the Function of TRUSTMIG
To help troubleshoot trustee issues, it is important to first understand the function of TRUSTMIG.NLM in a cluster. NetWare stores file system trustees in the metadata of the NSS volumes as a 4-byte Trustee ID (TID). These IDs are then resolved back to NDS objects based on the Entry ID table stored in the SYS:_NETWARE directory of the server. Each server has its own Entry ID table, which is different for each server.

When a volume moves from one server to another, the TIDs in the metadata are not valid for the new server. TRUSTMIG resolves this issue by keeping a TRUSTMIG.FIL file in a hidden _NETWARE directory on each shared volume that it is configured to monitor. The header record in TRUSTMIG.FIL has a marker that tells it which cluster node the file is currently valid on. It also contains a state marker, which tells the server the current migration status. This is used to recover from a failed migration. Finally, the TRUSTMIG.FIL file contains a list of all the Distinguished Names (DNs) of trustees for the volume, along with each trustee's assignments.

When a volume is mounted on a new server, TRUSTMIG goes through a routine to translate all of the DNs to the new server's TIDs. This process first produces temporary TIDs and then produces the actual TIDs, thus giving it the ability to recover even if the server were to crash in the middle of a migration.

Recovering from a Partially Migrated Volume
Any time a volume crashes in the middle of a trustee migration, Novell Cluster Services forces you to mount that volume back on the previous host server where the trustees were valid. This is necessary only when a node fails in the middle of the trustee migration process, which can happen only when a volume load script is being run on a node and that node fails. An example of this is when a volume fails over from one node to another and then the second node also fails. In this example, you have to repair the trustee assignments by mounting the volume on either the first or second node; do not mount the volume on any other node.

This problem happens only with a double-node failure. This means that one node fails, the second node runs TRUSTMIG, and then it also fails. The volume trustee assignments have to be restored on either the first or second node. If you attempt to mount the volume on a node other than the first or second, you will see an error similar to the following on the console of the node you are attempting to mount the volume on:

CLUSTER-<FATAL>-<18008>: You can't go to another node. Go back to either NODE1 or NODE2

It is extremely rare to get a double-node failure and rarer still that the second failure happens during the TRUSTMIG process. Novell Cluster Services is designed to deal with this kind of double-node failure. Simply move the volume back to the first node and then perform the following steps:

Bring the cluster volume resource offline.

Run the TRUSTOOL utility to finish the partial migration back to the previous host server.

The TRUSTOOL utility allows the migration to finish if you run it on the server where the volume was being moved to. If you run TRUSTOOL on the server that hosted the volume prior to the failure, it will back out of the migration. If it is run on a server other than the two described above, TRUSTOOL informs the administrator which server it must be run from.

To run the TRUSTOOL utility, execute the following command:

TRUSTOOL <VOLUME> FIX

Note: TRUSTOOL FIX deactivates the volume if it isn't already deactivated. Make sure the volume resource is offline before performing this action.

Activate and mount the volume.

Create a Trustool dump file by executing the following command:

TRUSTOOL <VOLUME> DUMP

This command writes all DNs and their migration status to the server console screen, as well as to the file SYS:ETC\TRUSTDMP.TXT.

Check the server console screen or the text file to verify that the trustees are correct.

You can also use NetWare Administrator, ConsoleOne^®, or Windows* Explorer to validate trustees. To do this you must manually access the volume through the server that has it mounted because you haven't yet brought the cluster-enabled volume online.

If necessary, add or edit any missing trustees or trustee assignments.

Purge any bad trustees.

The purging process deletes any bad DNs in the TRUSTMIG.FIL file. First it creates the TRUSTMIG.BAK file and then it re-creates the TRUSTMIG.FIL after purging the bad DNs.

Bad DNs occur whenever you delete an NDS object without first removing the object's file system trustee assignments. Bad DNs also occur when NDS doesn't synchronize quickly enough after new users with trustee assignments are created in the tree.

To purge bad trustees, execute the following command:

TRUSTOOL <VOLUME> PURGE

Create a new Trustool dump file by executing the following command:

TRUSTOOL <VOLUME> DUMP

Check the server console screen or the text file to verify that the trustees are correct.

Dismount and deactivate the volume.

Bring the volume online by using ConsoleOne to start the cluster volume resource.

Verify once again that the trustees are correct.

Migrate the volume to the server that you want it to run on.

Run TRUSTOOL DUMP on the new server and compare the dump file to the file you created in Step 8.

Preventing a Catastrophic Trustee Loss
Although the TRUSTMIG and TRUSTOOL utilities provide a great deal of protection and recovery in the event of trustee loss, you should still back up your trustees periodically just in case you have a serious volume error that prevents TRUSTOOL from recovering your trustees. Because the utilities described above rely on the TRUSTMIG.FIL file, any serious error that corrupts this file can cause an irreversible loss of trustees.

Novell has an unsupported utility called TBACKUP. This utility can be found at http://support.novell.com/misc/patlst.htm#tools, You can use TBACKUP to periodically back up your trustees. Many third-party companies also have tools that back up trustees. One of these is FSTRUST by DreamLAN Network Consulting Ltd. (http://www.dreamlan.com/main.htm).

Back up your trustees using your preferred method at intervals based on the frequency of file system trustees changes. If you have a lot of file system trustee activity, consider backing up your trustees weekly or even daily. For less active sites, consider doing it monthly.

A trademark symbol (^®, ^TM, etc.) denotes a Novell trademark. An asterisk (*) denotes a third-party trademark. For information on trademarks, see Legal Notices.