Quick Start: Advanced tuning parameters for eDirectory 8.8 SP8 and eDirectory 9 explained

Every site is different in terms of tuning eDirectory.  One's own internal testing is the only way to find the tuning sweet spots for any given organization.  For more detail on the features discussed here please refer to the eDirectory Admin Guide found on the NetIQ Documentation site: https://www.netiq.com/documentation/edir88/.

---

Features enabled out of the box

Obituary Process Optimization

• Prior to 8.8 SP8 eDirectory had two redundant methods of processing obituaries: backlinks and DRLs (Distributed Reference Links).  The DRL method has been removed eliminating these unnecessary cycles.
• Additionally, if all servers in a ring are on 888, flag 2 is no longer used in processing delete, rename and move obituaries.  This results in the cycles required to process an obituary being reduced by 50-75%.  Some tests have shown a reduction of up to 230% in the total time to process obituaries. 
• Another side benefit is there are far fewer "UsedBy" and "Obit_UsedBy" obituary attributes that need to be processed.

Obiturary Process Scheduling Optimization

• Prior to 888, the obituary process would not run on a parition if the server was currently outbounding changes for that partition to other servers.  This would cause obituaries for that partition to be delayed.  In a busy environment with hundreds of changes per second this could lead to change cache buildup.  Now the obituary process can run in parallel with outbound synchronization thereby reducing obituary processing delays.

Priority Queue

• Synchronization, especially by server mode, could result in only a few servers in a given replica ring actively skulking to each other at any given time.  The previous way eDirectory stored the ordering of the servers it needs to schedule for outbound synchronization was not a priority queue.  Now the scheduling of synchronization is a priority queue that implements a FIFO ordering so that servers just contacted go to the end of the queue.  Servers with the oldest scheduled time go to the front of the list.  Additionally, servers who returned a "-698 - Server Busy", when contacted, are now rescheduled to the front of the queue.
             

---

Features not enabled by default

Asynchronous Outbound Synchronization

• In the past there was one thread responsible for interating through all entries in the change cache to see if there was a new value that needed to be sent to other servers.  This same thread was also responsible for putting the values in a packet, sending the packet over the wire and waiting on an achnowledgement from the receiving server before proceeding.  Performing these tasks sequentially were costly in terms of time required to process the changes out to other servers.  In eDirectory 8.8 SP8 the work has been split between two threads.  One examines the change cache, prepares the outgoing packets then fills a queue with the packets.  The second thread picks up the packets from the queue and sends them the remote server one at a time.  This has reduced the time to get changes out to other servers by up to 50% in some cases.

• The default setting for Asynchronous Outbound Synchronization is disabled.  In eDirectory 8.8 SP8 iMonitor provides an interface to enable this feature.  If this feature is enabled the server will be much more aggressive in sending changes out to the remote servers.  This can put pressure on the receiving servers resulting in higher utilization.  Therefore, another setting has been included with this feature, the Async Dispatcher Thread Delay.  This setting allows the administrator to control, in milliseconds, the frequency in which changes are sent to the remote servers.  The Async Dispatcher Thread Delay setting, by default, has no delay (0 milliseconds).  The allowed range for this setting is 0 - 999 milliseconds.  If this value is very small, meaning the thread fires more often, higher CPU and I/O utilization may be seen on the receiving servers due to the higher amount of inbound traffic.  Therefore, this setting should be monitored and fine tuned for the specific enviroment.

 NOTE: In eDirectory 9 Asynchronous Outbound Synchronization is turned on by default.  It is using the Background Process Delay with a Maximum CPU Utilization  of 80%.

 

• Background process scalability improvements were of primary importance during the development of eDirectory 8.8 SP8.  As mentioned earlier, the emphasis in the past was to make background processing a lower priority than that of applications and clients.  During 8.8 SP8's development an analysis of code and extensive testing revealed that many background processes were spending a significant percentage of time sleeping.  Dramatic gains in performance can be had in forcing these processes to fire more frequently.  These changes need to be carefully considered, especially on older servers, as an increased frequency can result in an increase of the utilization of the hardware.
• Currently the following three processes have a hard coded delay of 100ms delay time. 
  
  • Change Cache Processing Delay
  • ObitProc Delay
  • Purger Delay
• Better performance can sometimes be had by lowering the delay times so the processes spend less time sleeping.  This can be done via a Hard Limit or CPU Based Policy.  Lowering these values can improve the following:
  • obituary processing speed
  • purging of objects in change cache
  • faster analysis of objects in change cache resulting in increased synchronization speed
  NOTE: it is recommended to leave these at their defaults, especially the purger delay.
• Policies
  • Hard Limit Policy - default
    • This allows the administrator to define a fixed value, in milliseconds, for how long each process will be delayed.  However, this approach means the administrator must manually find the right settings. 
  • CPU Based Dynamic Policy
  • This allows the system to either step up or down the delay based on current overall CPU usage.  The default, when enabled, is an 80% limit of CPU with a 100ms sleep time.  The server will check this utilization every 100 milliseconds.  Should the load on the server become higher than the configured maximum the system will begin to the increase the sleep time by 5 millisecond increments until utilization falls below the maximum. 
  • A test was performed using the CPU policy with a max CPU of 40% and a max Delay of 0.  This resulted in an 80% improvement when processing a bulk import of one millio
  NOTE: it is recommended to leave these at their defaults: CPU based with an 80% maximum.  Setting this to a low hard limit (IE., 5 ms) will force the server to spend most of its time dredging the change cache instead of synching the changes it found in the last pass.
  

 

• eDirectory, by default, uses a dynamic (system computed) mode to determine both the mode of synchronization as well as the number of threads to be used in that mode.  Performance can be enhanced by both manually selecting the mode and setting the maximum number of threads to be used.  This can increase the number of other servers or partitions that a server can synchronize to simultaneously beyond what would have been auto calculated.
• Setting the Mode of Synchronization
  • Modes
    • By Partition.  This is the most conservative as it uses one thread per partition to update multiple servers.  By default, if this mode is selected the number of threads will equal the number of partitions on the server up to a maximum of 8 threads.  If the System Computed  Synchronization Threads is disabled and the Maximum Manual Synchronization Threads is set to 12 the maximum number of threads used can go up to 12.  Changes are then synchronized simultaneously with other replica holding servers. 
      • If there are more unique partitions than there are unique servers in that server's replica rings this mode is the most efficient if there are dynamic changes to objects in multiple partitions. 
      • For example, if there are 12 partitions but only three servers holding replicas and all 12 partitions contain object changes, then 12 threads can be scheduled to send these changes out for all 12 partitions at once.
    • By Server.  This mode is chosen if the number of unique servers in the server's replica rings is greater than the number of partitions on that server.
      • Maximum number of threads used = [(number of unique servers +1) /2]
      • This mode might be chosen in the following example:  There are two partitions in the tree and 9 servers holding replicas.  Therefore the calculation used would be (9 unique servers +1) /2 = 5 threads.    If partition mode was choosen the server would only schedule two threads to outbound to the other servers. 
    • Dynamic mode.  This is the default.  The system calculates the mode and threads available for outbound synchronization.
      • Dynamic is enabled by default.  This allows the system to choose which mode and the number of threads to use on startup.
• Setting the number of Synchronization Threads
  • Below are the new thread options:
    • Option A - System Computed Synchronization Threads  This is enabled by default.  Disabling this setting prevents the system from computing its own calculations of how many threads to use.  Once disabled, the administrator can set the amount of outbound threads using Option C below.
    • Option B - Maximum System Computed Synchronization Threads This number represent the maximum number of threads that can be used when the server is in dynamic mode.  By default the maximum threads dynamic mode can use is 8 threads.  This new option is only used with Option A enabled as well and allows the administrator to manually configure up to a maximum of 12 threads. 
    • Option C - Maximum Manual Setting Synchronization threads  This number is the maximum number of threads that can be used in either mode.  This option overrides Option A and allows the administrator to set the number of outbound threads up to the maximum value.  By manually selecting the mode and configuring the number of threads for outbound sync, administrators gain control over the number of outbound threads potentially used in outbound sync.  Therefore, using the example of 9 unique servers and 2 partitions in dynamic mode the server would choose server mode and calculate the number of threads to use in outbound synchronization to only 5.  However, by setting the following in iMonitor we can increase the number of threads so that all 9 servers can potentially receive synchronization changes at once.
    • Set the Synchronization Method to Server
    • Disable the System Computed Synchronization Threads
    • Set the Maximum Manual Setting Synchronization Threads to 9
      • NOTE: The Max. System Computed Synchronization Threads field and the Max. Manual Setting Synchronization Threads field are mutually exclusive.

• Method: Partition Mode has been selected.  We have only 6 servers but 12 busy partitions.   The maximum number of threads that can be used for outbound synchronization is 16.  If partition mode is selected and the number of threads  is set to 12 this server can simlutanously outbound changes in all 12 partitions to remote servers.
• System Computed Synchronization Threads: leave at enabled
• Maximum System Computed Synchronization Threads: 12

• Method: Server Mode has been selected. 
• System Computed Synchronization Threads: disabled.  In server mode, if this option was not disabled, the server would have calculated 4 threads to be used [(number of unique servers +1) /2]  Set the Maximum Manual
• Maximum Manual Synchronization Threads: 6

• The purger process is responsible for removing entries from the change cache once all servers have seen that object's changes.  The default is to run for 15 minutes followed by a 30 minute delay.  In earlier versions of eDirectory the purger would wait on the skulker to complete.  In sites that have continuous amounts of changes it could be hours before it fired.
• In eDirectory 8.8 SP8 the purger is allowed to run alongside the skulker.  Further, the delay setting is now configurable.  This can be set lower so that the purger process runs more frequently and objects are removed from change cache more quickly once they are seen by all replica bearing servers. 10 minutes is the recommended minimum so that it does not run excessively.

• By default several attributes are updated on a user object each time it is used to login.  Among them are:
• • Login Time
  • Last Login Time
  • Network Address
  • Revision
  • modifiersName
• These attribute changes must be synchronized to all other servers holding a replica for that user object.  In a busy environment this can add dramatically to the amount of data required to be synchronized between servers.  Previously these changes could only be disabled or enabled.  Some customers use these values to track logins so disabling them altogether was not an option for them. 
• In eDirectory 8.8 SP8 a new setting has been implemented that allows an administrator to set a time interval between logins wherein these changes are not recorded on the object.  The default is 0, meaning this option has not been enabled.  By entering a value other than 0 it is enabled.  When enabling this setting the typical interval set is 3600 seconds (1 hour).  For example, when a user logs in for the first time at 8:00 AM, eDirectory updates these login attributes and the interval starts. If the same user logs in again before 9:00 AM, eDirectory will not update these attributes' values.