Improving eDirectory Performance

The most significant setting that affects eDirectory performance is the cache. In earlier versions of NDS®, you could specify a block cache limit to regulate the amount of memory that the directory used for the cache. The default was 8 MB RAM for cache.

With eDirectory 8.5 or later, you can specify a block cache limit and an entry cache limit. The block cache, available in earlier versions of NDS, caches only physical blocks from the database. The entry cache, a feature introduced in eDirectory 8.5, caches logical entries from the database. The caching of entries reduces the processing time required to instantiate entries in memory from the block cache.

Although there is some redundancy between the two caches, each cache is designed to boost performance for different operations. Block cache is most useful for update operations. Entry cache is most useful for operations that browse the eDirectory tree by reading through entries, such as name resolution.

Both block and entry caches are useful in improving query performance. Block cache speeds up index searching. Entry cache speeds up the retrieval of entries referenced from an index.

The defaults for eDirectory 8.7.3 are listed below:


Distributing Memory between Entry and Block Caches

With an entry cache and a block cache, the total available memory for caching is shared between the two caches. The default is an equal division. To maintain the amount of block cache available in earlier versions of NDS 8, you need to double the total cache size for eDirectory. If you use the cache to boost LDIF-import performance, for example, you can either double the total cache size or change the default cache settings. To change the default cache settings, refer to Configuring Dynamically Adjusting and Hard Memory Limits.

The more blocks and entries that can be cached, the better the overall performance will be. The ideal is to cache the entire database in both the entry and block caches, although this is not possible for extremely large databases. Generally, you should try to get as close to a 1:1 ratio of block cache to DIB Set as possible. For entry cache, you should try to get close to a 1:2 or 1:4 ratio. For the best performance, exceed these ratios.


Using the Default Cache Settings

eDirectory provides two methods for controlling cache memory consumption: a dynamically adjusting limit and a hard memory limit. You can use either method, but you cannot use them at the same time because they are mutually exclusive. The last method used always replaces any prior settings.


Understanding the Dynamically Adjusting Limit

The dynamically adjusting limit causes eDirectory to periodically adjust its memory consumption in response to the ebb and flow of memory consumption by other processes. You specify the limit as a percentage of available physical memory. Using this percentage, eDirectory recalculates a new memory limit at fixed intervals. The new memory limit is the percentage of physical memory available at the time.

Along with the percentage, you can set a maximum and minimum threshold. The threshold is the number of bytes that eDirectory will adjust to. It can be set as either the number of bytes to use or the number of bytes to leave available. The minimum threshold default is 16 MB. The maximum threshold default is 4 GB.

If the minimum and maximum threshold limits are not compatible, the minimum threshold limit is followed. For example, you could specify the following settings:

Minimum threshold:

8 MB

Percentage of available physical
memory to use:


75

Maximum threshold:

Keep 10 MB available

When eDirectory adjusts its cache limit, there is 16 MB of available physical memory. eDirectory calculates a new limit of 12 MB. eDirectory checks to see whether the new limit falls within the range of minimum and maximum thresholds. In this example, the maximum threshold requires that 10 MB must remain available, so eDirectory sets the limit to 6 MB. However, the minimum threshold is 8 MB, so eDirectory sets the final limit to 8 MB.

With the dynamically adjusting limit, you also specify the interval length. The default interval is 15 seconds. The shorter the interval, the more the memory consumption is based on current conditions. However, shorter intervals are not necessarily better because the percentage recalculation will create more memory allocation and freeing.


Understanding the Hard Memory Limit

The hard memory limit is the method that earlier versions of eDirectory use to regulate memory consumption. You set a hard memory limit in one of the following ways:

  • Fixed number of bytes
  • Percentage of physical memory

    The percentage of physical memory at the interval becomes a fixed number of bytes.

  • Percentage of available physical memory

    The percentage of available physical memory at the interval becomes a fixed number of bytes.


Cleaning Up the Cache

NDS 8 creates multiple versions of blocks and entries in its cache for transaction integrity purposes. Earlier versions of NDS 8 did not remove these blocks and entries when they were no longer needed. In eDirectory 8.7.3, a background process periodically browses the cache and cleans out older versions. This helps minimize cache memory consumption. The default browsing interval is 15 seconds.


Configuring Dynamically Adjusting and Hard Memory Limits

You can configure dynamically adjusting and hard memory limits in either of the following methods:


Using Novell iMonitor

  1. Click Agent Configuration Agent Configuration button.

  2. Click Database Cache, then view the following information:

    Database Cache Information Description

    Maximum Size

    The maximum size (in KB) that the specified cache is allowed to grow to.

    Current Size

    The current size (in KB) of the specified cache.

    Items Cached

    The number of items in the specified cache.

    Old Versions Cached

    The number of old versions in the specified cache. Old versions of cache items are kept to maintain the consistency of read transactions in the database. In other words, if one thread is in a read transaction and another is in a write transaction, old versions of blocks modified by the writer are maintained on behalf of the reader. This is done so that the reader's results are guaranteed to produce a consistent view during the life of its transaction even though modifications are taking place during that time.

    Old Versions Size

    The size (in KB) of the old version items cached.

    Hits

    The number of times an item was successfully accessed from the specified cache.

    Hit Looks

    The number of items looked at in the cache before an item was successfully accessed from the specified cache. The hit-look-to-hit ratio is a measure of cache lookup efficiency. Normally, the ratio should be close to 1:1.

    Faults

    The number of times an item was not found in the specified cache and had to be obtained in a lower level cache or from the disk.

    Fault Looks

    The number of items looked at in the cache before it was determined that the desired item was not in the specified cache. The fault-look-to-fault ratio is a measure of cache lookup efficiency. Normally, the ratio should be close to 1:1.

  3. Choose from the following options:

    Option Description

    Dynamic Adjust

    Allows the eDirectory database to dynamically adjust the amount of system memory to be used for the cache based on the amount it thinks it needs and the parameters specified below.

    Cache Adjust Percentage

    The percentage of available memory allowed to be used for the record and block caches combined.

    Cache Size Constraints

    While dynamically adjusting, follow the specified constraints. Namely, use no less than the specified amount of memory for the cache and no more than the total amount of available memory minus the specified amount.

    Hard Limit

    The exact amount of system memory to be use for the cache.

    Cache Maximum Size

    The size (in KB) of the record and block caches combined.

    Block Cache Percentage

    The percentage of the system memory available for caching that should be allocated to the block cache. The remaining percentage will be allocated to the record cache.

    Cache Adjust Interval

    This interval applies only when Dynamic Adjust is set. It controls how often the cache size is adjusted, based on the specified percentage and constraints.

    Cache Cleanup Interval

    Controls how often unused old versions are removed from the cache.

    Cache Settings Permanent

    When this option is selected, any changes submitted through iMonitor will be made permanent, overwriting any previously saved settings or system defaults.

  4. Click Submit.


Using the _ndsdb.ini File

  1. Open _ndsdb.ini in a text editor.

    In NetWare®, this file is in sys:\netware. In Windows NT and Windows 2000, this file is generally in \Novell\NDS\DIBfiles.

  2. Add the applicable syntax to the file:

    Command Variable Explanation Definition

    cache=cache_bytes

    Fixed number of bytes you want used.

    Sets a hard memory limit.

    For example, to set a hard limit of 8 MB, enter

    cache=8000000

    cache=cache_options

    Multiple options can be specified in any order, separated by commas.

    • DYN

      Sets a dynamically adjusting limit.

    • HARD

      Sets a hard memory limit.

    • %:percentage

      Percentage of available or physical memory to use.

    • AVAIL or TOTAL

      Percentage of available or total physical memory for hard memory limit only.

    • MIN:number_of_bytes

      Minimum number of bytes.

    • MAX:number_of_bytes

      Maximum number of bytes.

    • LEAVE:number_of_bytes

      Minimum number of bytes to leave.

    Sets a hard memory or dynamically adjusting limit.

    For example, to set a dynamically adjusting limit of 75% of available memory and a minimum of 16 MB, enter

    cache=DYN, %:75,MIN:16000000

    Or to set a hard limit of 75% of total physical memory and a minimum of 16 MB, enter

    cache=HARD,%:75,MIN: 16000000

  3. (Optional) To specify the dynamic adjusting limit interval, add the following line:

    cacheadjustinterval=number_of_seconds

  4. (Optional) To specify the interval for cleaning up older versions of entries and blocks, add the following line:

    cachecleanupinterval=number_of_seconds

  5. (Optional) To change the percentage split between block cache and entry cache, add the following line:

    blockcachepercent=percent

    The variable percent should be between 0 and 100. The percentage you specify is the percentage of cache memory used for the block cache. The remaining percentage is used for the entry cache. We do not recommend setting the percentage to 0.

  6. Restart the eDirectory server for the changes to take effect.


Configuring Limits Using DSTrace

If you are using eDirectory for NetWare, you can configure the dynamically adjusting and hard memory limits in DSTrace. You do not need to restart the server for the changes to take effect.

  1. (Optional) To set a fixed hard limit, enter the following at the server console:

    SET DSTRACE=!MBamount_of_RAM_to_use_in_bytes

    For example, to set a hard limit of 8 MB, you would enter

    SET DSTRACE=!MB8388608

  2. (Optional) To set a calculated hard limit, enter the following at the server console. Include only the options you want to specify.

    SET DSTRACE=!MHARD,AVAIL OR TOTAL,%:percent,MIN:number_of_bytes,MAX:number_of_ bytes,LEAVE:number_of_bytes_to_leave,NOSAVE

    For example, to set a hard limit of 75% of total physical memory and minimum of 16 MB, and to specify not to save these options to the startup file, you would enter

    SET DSTRACE=!MHARD,%:75,MIN:16777216,NOSAVE

  3. (Optional) To set a dynamically adjusting limit, enter the following at the server console:

    SET DSTRACE=!MDYN,%:percent,MIN:number_of_bytes,MAX:
    number_of_bytes,LEAVE:number_of_bytes_to_leave,
    NOSAVE

    For example, to set a dynamic limit of 75% of available memory and a minimum of 8 MB, you would enter

    SET DSTRACE=!MDYN,%:75,MIN:8388608


Tuning LDAP for eDirectory

For information on basic LDAP server hardware and software configuration, tuning parameters, and tips on directory organization, see How to Configure and Optimize eDirectory LDAP Servers.


Managing the Memory

eDirectory uses memory for the database cache and for directory usage. These are separate allocated memory pools. The directory engine uses memory from available memory pools in the operating system as needed. The database uses a cache pool that is defined by parameters detailed below. Usually, the more database cache given to eDirectory, the better the performance. However, because eDirectory uses available system memory for its buffers, if clients are performing queries that require large data sets to be returned, the size of the database cache might need to be decreased to have enough system memory for the directory to handle building the query responses.

The database engine uses the database cache to hold the most recently accessed blocks. This cache is initially defined with a fixed size of 16 MB. The size of this cache can be changed from the command line in shipping versions of eDirectory. The following example command will set the eDirectory database cache to 80 million bytes:

set dstrace=!mb 80000000

You can also define a file named _ndsdb.ini in the sys:\_netware directory on a NetWare server, or in the directory containing the eDirectory database files on the Windows (normally install directory\nds\dbfiles) and UNIX environments (normally \var\nds\dib). This text file simply needs to contain a line such as the following:

cache=80000000

Don't add any white space around the equals (=) sign

The cache in eDirectory 8.7.3 can be initialized with a hard limit just as with earlier versions. In addition, the upper and lower limits can be set either as hard numbers or as a percentage of available memory. Dynamic allocation control parameters allow the cache size to grow or shrink depending on use. If the proper configuration parameters are set, the database cache dynamically grows or shrinks based on other system resource needs.

Editing the _ndsdb.ini file can manually control database memory usage. The format for INI file commands is given below:

cache=cacheBytes # Set a hard memory limit

Alternative formats are shown in the following table:

Command Description

cache=cache_options

Sets a hard limit or dynamically adjusting limit. Multiple cache options can be specified in any order, separated by commas. All are optional. They are as follows:

DYN or HARD

Dynamic or hard limit.

AVAIL or TOTAL

These only apply if a hard limit was chosen. Omit these options for a dynamic limit.

%:percentage

The percentage of available or total physical memory.

MIN:bytes

The minimum number of bytes.

MAX:bytes

The maximum number of bytes.

LEAVE:bytes

The minimum number of bytes to leave for the OS.

blockcachepercent=percentage

Splits the cache between the block and record cache.

If a hard limit is specified and the administrator wants to define the database cache to use a percentage of the memory, the administrator can select between a percentage of total memory or a percentage of available memory. Dynamic limits always refer to a percentage of available memory. The following command examples are all valid in the _ndsdb.ini file.

The following is an example dynamic limit of 75% available memory, a minimum of 16 million bytes, and 32 million bytes for the OS:

cache=DYN,%:75,MIN:16000000, LEAVE 32000000

The following is an example hard limit of 75% total physical memory, a minimum of 18 million bytes, and a maximum of 512 million bytes:

cache=HARD, TOTAL,%:75,MIN:18000000, MAX 512000000

The following is an example old style hard limit of 8 million bytes:

cache=8000000

The database cache is divided between block cache and record cache. Block cache holds data and index blocks that mirror the storage on the disk. Record cache holds in-memory representations of directory objects and attributes. If updating or adding to the directory, use the block cache setting. If performing mostly reads, use the record cache. It is possible to cause a thrashing condition in both caches if performing numerous sequential updates without allocating cache size properly. Unless specifically changed, the cache is allocated to be 50% block cache and 50% record cache. The blockcachepercent option can be included in the _ndsdb.ini file to specify the percentage of cache allocated to caching data and index blocks. (The default is 50%.) The remaining cache is used for entries.

For example, to designate 60% block cache and 40% record cache, enter the following:

blockcachepercent=60

Do not select 100% of the cache for either block or record cache and starve the other cache type. In general, do not allocate more than 75% of your cache memory to one or the other type.

Database cache settings can also be controlled using Novell iMonitor.

Although the cache size is dynamic depending on the amount of memory available, the DSTRACE command can still be used for custom environments.