Novell Home

My Favorites

Close

Please to see your favorites.

Handling ndsd (eDirectory) core files on Linux and Solaris

This document (3078409) is provided subject to the disclaimer at the end of this document.

Environment

Novell eDirectory 8.8 for Linux
Novell eDirectory 8.7.3 for Linux
Novell eDirectory 8.7.3 for Solaris
Novell eDirectory 8.8 for Solaris

Situation

When ndsd crashes a core file will be generated in the dib directory if the ulimit -c is configured to a value more than 0. By default, the dib directory is located:

eDirectory 8.7.3    /var/nds/dib
eDirectory 8.8.x    /var/opt/novell/eDirectory/data/dib

If ndsd crashes and the reason is not apparent, check for a core file in the dib directory. If there is no core file present, change the ulimit -c setting to unlimited.

Many Linux distributions set the ulimit value to '0' in /etc/profile or use 'ulimit -Sc 0' to prevent core files.

In order for ndsd to use this setting it is necessary to add it to the ndsd script and then restart ndsd. (it could also be added to the pre_ndsd_start script as this script is sourced when ndsd loads).

Modify the /etc/init.d/ndsd script and add the following on the 2nd line directly underneath “#!/bin/bash": 

ulimit -c unlimited

Resolution

Novell-getcore

Novell-getcore is a script used to gather and bundle the ndsd core file and all associated libraries necessary to analyze the core file.  Novell-getcore is installed as part of the NDSserv package, beginning with eDirectory 8.7.3.9 and eDirectory 8.8.2.  If you have an earlier  eDirectory version, the very first thing you should do is update eDirectory to the latest available version as the current version most likely has the fix! However, the novell-getcore script can be downloaded from http://download.novell.com

Just enter “novell-getcore” in the keyword field and click search.

Using novell-getcore to bundle core and necessary libraries:

1) Verify GDB is installed on the eDirectory server by typing "gdb -version".  GDB is required to be installed prior to using novell-getcore.

2) Create a bundle with novell-getcore to send to Novell Technical Support:

eDirectory 8.7.3 example:
novell-getcore -b /var/nds/dib/core.#### /usr/sbin/ndsd

eDirectory 8.8 example:
novell-getcore -b /var/opt/novell/eDirectory/data/dib/core.#### /opt/novell/eDirectory/sbin/ndsd

(where ####, is the PID number of ndsd when it cored)
Note: eDir 8.8.7 and 8.8.8 may not have a pid number at the end of the core name.

This will generate a gzip'd tar bundle in the same directory as the core file with a name like the following:

core_YYYYMMDD_162243_linux_ndsd_hostname.tar.gz


3)  Grab a supportconfig file from the server that cored.

On Linux use supportconfig/supportutils.  If you need the script it can be downloaded from the following page: http://www.novell.com/communities/node/2332/supportconfig-linux

On Solaris: Use unixinfo to create a unixinfo.log.  See TID 10075466 “How to create a UNIX configuration file”.

On Solaris:  Use pstack to get the stack of the core.  EX:  pstack core > ndsd.pstack

4) Upload the supportconfig or unixinfo.log and novell-getcore bundles to ftp://ftp.novell.com/incoming

NOTE:  Currently novell-getcore isn't functioning on Solaris.  Please gather the core file, the pstack output and a unixinfo.log, tar them together with the SR# and upload them to the ftp server (ftp.novell.com:/incoming)

Additional Information

Sometimes the reason ndsd crashes is due to memory corruption.  If this is the case, it is necessary to add variables setting to the ndsd environment to put the memory manager into a debug state. This will help to ensure that ndsd generates a core at the time the corruption occurs so the module that caused the corruption can more easily be identified in the core.

If ndsd cores due to stack corruption, Novell Technical Support will request that you add the appropriate memory manager setting and wait for another core to re-submit.

Linux

To set the necessary memory checking variable on Linux:

Modify the pre_ndsd_start script and the following at the very top, then restart the eDirectory instance.

MALLOC_CHECK_=2
export MALLOC_CHECK_
 
## Note in eDirectory 8.8.5 ftf2 (patch2) the location of the pre_ndsd_start has been moved from /etc/init.d to /opt/novell/eDirectory/sbin/.  The contents of the pre_ndsd_start script are sourced into ndsd at the time ndsd loads.  Be aware that any permanent settings will be overwritten if left in the ndsd script the next time an eDirectory patch is applied while the pre_ndsd_start script will not be modified.  For this reason changes to the 'ndsd' script itself should not be made.  This is the purpose of the pre/post_ndsd_start scripts.

MALLOC_CHECK_=2 should NOT be left pernamently.Once the cores have been gathered, remove this setting from the modified script and restart ndsd. This environment variable can have a performance impact on some systems due to the increased memory checking.  In eDirectory 8.8, it will cause ndsd to revert back to using malloc instead of tcmalloc_miminal which was added to enhance performance.

Another side effect of using MALLOC_CHECK_=2 is the possibility of increased coring.  Malloc will cause ndsd to core whenever a memory violation is detected whether or not it would have caused ndsd to crash under normal running conditions.

To verify this ndsd environment variable is set properly while ndsd is running, do the following as the user running the eDirectory instance ('root' most of the time):

strings /proc/`pgrep ndsd`/environ | grep -i MALLOC_CHECK_

The command above will not work on a server with multiple eDirectory instances (or ndsd processes).  To check a particular instance find that instance's process's PID and use that directly.  For PID 12345 the command would be the following:

strings /proc/12345/environ | grep -i MALLOC_CHECK_

After ndsd has cored, to verify the core file had the ndsd environment variable set, do the following:

strings core.#### | grep -i MALLOC_CHECK_

Bundle the core with MALLOC_CHECK_=2 set as in step 2.

For more information on Malloc check see: TID 3113982: Diagnosing Memory Heap Corruption in glibc with MALLOC_CHECK_

Solaris 

In current code, eDirectory uses libumem as the memory manager.

To configure libumem for debugging add the following to the pre_ndsd_start script at the top and restart ndsd:

UMEM_DEBUG=default

UMEM_LOGGING=transaction

export UMEM_DEBUG UMEM_LOGGING

Submit a new core with these settings in place.


Changing the location where cores files are generated

In certain situations it may be desirable to change the location where core files are generated.  By default ndsd core files are placed in the dib directory.  If space in this directory is limited or if another location is desired, the following can be done:

mkdir /tmp/cores
chmod 777 /tmp/cores
echo "/tmp/cores/core"> /proc/sys/kernel/core_pattern

This example would now generate the core.<pid> file in /tmp/cores

To revert back to placing cores in default location:

echo core > /proc/sys/kernel/core_pattern

Symbol build of ndsd libriaries

In some cases, a core file generated while running libraries with symbols included may be necessary to analyze the core.

This is particularly true when analyzing cores generated by the 64 bit version of ndsd since the parameters aren't located at a specific location.

The symbol versions of the libraries can be obtained from Novell eDirectory backline support.


 

Disclaimer

This Support Knowledgebase provides a valuable tool for NetIQ/Novell/SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:3078409
  • Creation Date:09-JAN-08
  • Modified Date:29-APR-14
    • NetIQeDirectory

Did this document solve your problem? Provide Feedback