There has been a great deal of confusion surrounding obituaries stored in the directory and, as a result, some people have developed poor business practices to deal with then. Unlike some directory products, Novell eDirectory ensures referential integrity between objects. For example, if Group A has a member, User B, and User B is deleted, the directory automatically removes the reference to User B from Group A. Obituaries exist as operational attributes placed on objects by eDirectory as another way of ensuring referential integrity during delete, move, rename, restore, and other operations.
There are three general classifications for obituaries:
Obituaries, with the exception of Tracking obituaries, must move through a set of synchronizing states:
The states are recorded in the Flags field in the obituary attribute. Before an obituary can move to the next state, the current state must have been synchronized to all replicas of the real object. In order to determine whether all replicas in the ring have seen a given obituary state, a vector is computed from the transitive vector. In eDirectory 8.6 and later, a nonstored Obituary Vector is used. In previous versions of eDirectory, the Purge Vector is used. If the Modification Timestamp (MTS) on the obituary is older than the corrupted vector, the server responsible for that obituary can advance it to the next state.
For a Secondary obituary of type Back Link, the agent that holds the master replica of the object with the obituary is responsible for advancing the states. For a Secondary obituary of type Used By, the replica agent that created it is responsible for advancing the obituary states as long as that replica still exists. If it does not still exist, the agent holding the master of that partition takes over advancing the obituary states for the Used By obituary. For a Move Tree obituary, the master of the root partition is responsible for advancing the states.
Primary obituaries can be advanced in their states only after all Secondary obituaries have advanced through all of their states. After the Primary obituary reaches its last state, and that state synchronizes to all servers in the ring, all that remains is the object husk, which is an object without attributes---one which can subsequently be purged from the system by the Purge Process. Tracking obituaries are removed after the Primary obituary is ready to be removed or, in the case of Inhibit_move, the Tracking obituary is removed after the Primary obituary has moved to the OBF_NOTIFIED state on the master replica.
The replica responsible for processing obituaries does so on a background process (the Obituary Process), which is scheduled on a per-partition basis after a given partition finishes an inbound synchronization cycle. If there are no other replicas of the partition, the Outbound Replication Process is still scheduled on the heartbeat interval. The Outbound Replication Process then starts the Obituary Process. The Obituary Process cannot be manually scheduled, nor does it need to be. As synchronization occurs, the transitive vectors are updated, thus advancing the Purge Vector and Obit Vector. As these vectors move forward, the obituary states are allowed to move forward. This, together with the automatic scheduling done upon inbound synchronization, completes the obituary processing cycle. Therefore, the lifeblood of obituary processing is object synchronization.
For an object that is being removed, after all obituaries whose associated Primary obituary is of type Dead have been advanced to the last state (Purgeable), and that state has been synchronized to all replicas, a new process is responsible for removing the remaining entry husk from the database. The Purge Process runs automatically to remove these husks. You can manually schedule the Purge Process and modify its automatic schedule interval by using the Agent Configuration page in iMonitor.
This section contains the following examples:
Add the Primary obituary OBT_DEAD.
The Back Link attribute contains a list of servers that have an interest in this object and need to be notified of changes to this entry. For every DN listed in the Back Link attribute and all servers listed in the entry's partition replica attribute, eDirectory adds a Back Link obituary. The creation time of the Primary obituary, OBT_DEAD, is stored in the Secondary obituary.
The Used By attribute contains a list of partitions that have an interest in this object and need to be notified of changes to this entry. For every DN listed in the Used By attribute, eDirectory adds a Used By obituary. The creation time of the Primary obituary, OBT_DEAD, is stored in the Secondary obituary.
Remove all attributes but the obituaries.
The Outbound Replication Process then synchronizes this change to all other servers in the replica ring.
On the next inbound synchronization of this partition, the Obituary Process is started, which does the following:
If the obituary is a Primary obituary, there are no Secondary obituaries, and the attribute's modification time (MTS) on the obituary is older than the Purge Vector, then all servers have seen the change and this obituary will be removed.
If the obituary is a Back Link obituary and this server is the master, then this server is responsible for processing this obituary.
IMPORTANT: Perform the required operation for this state if it has not been done. Most often, this is done by notifying an external reference.
If the obituary is a Used By obituary and this server is the server where the delete occurred (determined by comparing the replica number in the obituary's MTS to our replica number), this server is responsible for processing this obituary.
Move acts much like Delete, but with the following changes:
Objects with obituaries are considered every time an agent outbound synchronizes, and by the obituary process, which is scheduled to run at the end of an inbound synchronization cycle.
On a regular basis, run the iMonitor Server Information report. This report walks the entire tree, communicates with every NCP server it can find, and reports any errors it finds. You can use this report to diagnose time synchronization and limber problems, or to find out if the current server is able to communicate with all other servers from this server's perspective. If selected in the configuration page, the server can also generate NDS Agent Health information for every server in the tree. See Configuring and Viewing Reports for more information on running the Server Information report.
If you are using iMonitor 2.0 or later, make sure that the Errors and Health Sub-report report options are enabled. The following items will be verified. You should browse the report and make sure that there are no errors.
If you are using iMonitor 1.5, select the Errors report option. The following items will be verified. You should browse the report and make sure that there are no errors.
Using the iMonitor Obituary Listing report or the iMonitor Object Statistics report, you can find any obituaries on your system. If you find any obituaries that you don't believe are being processed, see Troubleshooting Tips.
There are two general reasons that obituaries don't process: either the obituary has been orphaned (that is, the obituary exists on some servers but not all servers) or the obituary is stuck (that is, it exists on all servers but its states are not advancing for some reason).
Do the following to troubleshoot orphaned or stuck obituaries:
While browsing the entry with obituaries, click the Entry Synchronization link. The page displayed will show all attributes that have not been synchronized to all replicas.
Find the oldest time stamp on any of the obituary attribute values. The difference between that time and the current time should be greater than the interval shown in the Max Ring Delta field on the Partition Synchronization page.
Evaluate the transitive vector.
-625, -622, -634, and -635 communication problems. See Server Information Report for more details.
-601, and -603, indicating servers that have been improperly removed, or that the Server object might have a base class of Unknown.
Use the proper solution referred to in Troubleshooting Tips.
Before using any of these solutions, you must make sure that your data is safe. You might need to back up the directory database files, server configuration, and trustees. To increase the probability of success and to minimize future problems, upgrade to the latest eDirectory Support Packs.
Preferred method: If eDirectory 8.6 or later is on any of the servers in the replica ring, browse to the object in iMonitor, then select Send Single Entry. This will perform a nonauthoritative send to all other replicas.
Far less desirable method: If all servers in the replica ring that have a copy of the orphaned obituary are older than eDirectory 8.6, load DSBrowse with the -a option, browse to the object, then time-stamp the entry. This will make the object as it exists on this server the authoritative copy. We do not recommend making objects authoritative as a matter of practice.
Less desirable method: Run DSRepair with the time stamp option selected.
Less desirable method: Move a real replica to the server, wait for it to turn on, then wait for the obituary to be processed. If the obituary is not processed, use the information in Troubleshooting Tips to resolve the issue now that the object is on a real replica. After the obituary has processed, the replica can be removed if desired.
In the past, several different strategies have been employed to resolve stuck obituaries. Some of these strategies involve expensive partitioning operations, or the use of undocumented features that might cause problems in the future.
The first strategy was to switch which replica held the master. This would work in some cases because the master is the agent responsible for moving the Back Link obituaries through their various states. In the case where the replica was inconsistent and the master didn't hold the deleted object, switching masters to an agent that held the deleted entry with its obituaries would give the new agent the license to push the obituaries through their states and eventually purge it out. Send Single Entry is a much cleaner and less dangerous way to resolve obituaries that are stuck because the replica is inconsistent.
The second strategy used was to run DSRepair with certain switches to delete all obituaries. (There is a third-party application which resolves stuck obituaries by launching DSRepair.) We do not recommend this strategy. Using those switches will delete all obituaries on this agent, which means that obituaries that are not stuck might also be removed, creating further replica inconsistencies and more stuck obituaries. Because this is not a distributed operation, you must run DSRepair on all of the servers with stuck obituaries, which increases the odds that one of those servers has obituaries for another partition which will be prematurely deleted. The premature deletion of obituaries can cause additional orphaned obituaries and, in turn, cause problems which can be found years later when you change replicas types, add new replicas, or perform other partitioning operations.
The third strategy used was to make objects authoritative, either using DSBrowse with the advanced mode operation and time stamping the entry, or running DSRepair with the -0T switch. This forces the entry to become authoritative and synchronize out to all other replicas. This should be done with great care because you might lose data changed on other servers. We recommend that this be a rarely employed method of obituary cleanup.