Troubleshooting -641 -783 -299 Errors Starting an IDM driver - eDirectory fails to load (vrdim)

  • 7002449
  • 26-Jan-2009
  • 12-Oct-2020

Environment

Identity Manager 3.0.x
Identity Manager 3.5.x
Identity Manager 3.6.x
Identity Manager 4.0.x
Identity Manager 4.5.x
Identity Manager 4.6.x
Identity Manager 4.7.x

Situation

When trying to start an IDM driver through iManager, receive the error message: "Unable to start the driver, error: -641 "

When trying to start an IDM driver through iManager, receive the error message: "Unable to start the driver, error: -783 "

Same error codes (-641 and -783) seen when trying to start the driver through Designer

Search:  783 vrdim edirectory fails to start load-783 641 641 -299 299

Resolution

This issue can have several different causes. The errors just mean that the main IDM engine is not able to load properly when eDirectory loads.

The first step is to try rebooting the server that IDM/eDirectory is running on.  This has resolved the issue in a number of cases.

The second step is to make sure that no security or auditing software is involved.  LSM (SELinux, AppArmor or other similar programs) or auditing software such as Novell Audit can cause this problem.  Often the server with the auditing program will not be responding due to some problem such as a full disk.  Any problem with this type of program will cause the 783 error.
On NetWare check for a file sys:\etc\logevent.cfg and on linux check for a file /etc/logevent.conf  These files can point to auditing software that is tied into the server.

Troubleshooting steps per platform below:

IDM engine running on Windows Platform (pre IDM 4.7):
- IDM is installed in the same directory where eDirectory's dlms are (by default, C:\Novell\NDS)
01) Stop the eDirectory service
02) Move the file “dirxml.dlm” from the directory where eDirectory is installed
03) Start the eDirectory service
04) After eDirectory finishes loading, start DSTRACE.dlm
05) Go to the 'Edit' menu, then select 'Option'
06) Set ONLY the flags 'DirXML', 'DirXML Drivers', 'Misc Other'. Uncheck all the others and hit 'OK'
07) Go to the 'File' menu, then select 'New', then enter a filename for your trace file
08) Move the file “dirxml.dlm” back to its original location
09) Close/reopen the eDirectory services console
10) Select dirxml.dlm and hit the Start button
11) The trace file will show messages related to why the VRDIM module (IDM Engine) won't start


IDM engine running on Windows Platform (IDM 4.7 and later):
- IDM is installed in the same directory where eDirectory's dlms are (by default, C:\Novell\NDS)
01) Stop the eDirectory service
02) Move the file “dxevent.dll” from the directory where eDirectory is installed
03) Start the eDirectory service
04) After eDirectory finishes loading, start DSTRACE.dlm
05) Go to the 'Edit' menu, then select 'Option'
06) Set ONLY the flags 'DirXML', 'DirXML Drivers', 'Misc Other'. Uncheck all the others and hit 'OK'
07) Go to the 'File' menu, then select 'New', then enter a filename for your trace file
08) Move the file “dxevent.dll” back to its original location
09) Close/reopen the eDirectory services console
10) Select dxevent.dll and hit the Start button
11) The trace file will show messages related to why the VRDIM module (IDM Engine) won't start
 

IDM engine running on Linux/Solaris/AIX Platform (pre IDM 4.7):
01) Stop ndsd with the command "/etc/init.d/ndsd stop" (without the double-quotes)
02) Move the libvrdim.* files from their original directory to a different directory
     in eDirectory 8.7.3.x they reside at /usr/lib/nds-modules/
     in eDirectory 8.8.x 32 bit: they reside at /opt/novell/eDirectory/lib/nds-modules/
     in eDirectory 8.8.x 64 bit: they reside at /opt/novell/eDirectory/lib64/nds-modules/
03) Start ndsd with the command "/etc/init.d/ndsd start" (without the double-quotes)
04) Start ndstrace
05) Within ndstrace, type:
    set ndstrace=nodebug
    set ndstrace=+time
    set ndstrace=+tags
    set ndstrace=+misc
    set ndstrace=+dxml
    set ndstrace=+dvrs
    ndstrace file on
06) Leave ndstrace it running on screen
07) Move the libvrdim.* files back to their original location
08) Back on the ndstrace screen, type "load vrdim"
09) After you see the errors, stop ndstrace. The trace file will show messages related to why the VRDIM module (IDM Engine) won't start


With IDM 4.7, eDirectory 9.1.x will typically load despite vrdim failing to load.   Check to see if dxevent and vrdim are running.  You can do that with "ndstrace -c modules | grep dxevent" and "ndstrace -c modules | grep vrdim"
If dxevent is running, and vrdim is not, you can troubleshoot the loading of vrdim with the following.
01) Start ndstrace
02) Within ndstrace, type:
    set ndstrace=nodebug
    set ndstrace=+time
    set ndstrace=+tags
    set ndstrace=+misc
    set ndstrace=+dxml
    set ndstrace=+dvrs
    ndstrace file on
03) From a command prompt execute "ndstrace -c "load vrdim""
04) Look in the ndstrace.log file for the error encountered when vrdim attempts to load.

Cause

Those steps will show you exactly what the engine is doing as it loads, as well as all jvmloader related messages. Somewhere in the trace you probably will see the exact problem, and after that occurs, the -783 will happen.

  There are several possible causes for the error, some are listed bellow with suggestions to help fix and/or track them further:

- Cause: Misconfiguration of the JVM heap sizes
  Suggestions:  Clear any custom settings to the Initial heap size and Max heap size on the properties of the driver set, misc tab.   Then try to load the vrdim modules.    ****Too large of heap settings accounts for about 90% of these issues****



Additional detail on these errors can be found using the +misc flag in ndstrace. All jvmloader messages come under the MISC flag on Linux/Unix, and under the MISC OTHER flag on dstrace.dlm (Windows). Unfortunately there is no way of seeing them on Netware ( but we can always check the SYS:/ETC/JAVA.CFG file there)

- Cause LD_LIBRARY_PATH not set properly causing "error: libjvm.so: cannot open shared object file: No such file or directory" and "JVM interface initialization failed <failed, -299 (0xfffffed5)>, unloading DIRXML" errors.
Verify that echo $LD_LIBRARY_PATH returns a valid path.    . /opt/novell/eDirectory/bin/ndspath will get it running for the current session, then restarting ndsd should load vrdim.   The path can be set by exporting it after logging in as root, or adding it to an env_idm file in /etc/opt/novell/eDirectory/conf directory. 

LD_LIBRARY_PATH=//opt/novell/eDirectory/lib64/nds-modules/jre/lib/amd64://opt/novell/eDirectory/lib64/nds-modules/jre/lib/amd64/server://opt/novell/eDirectory/lib64/nds-modules/jre/lib/amd64/native_threads::$LD_LIBRARY_PATH:/opt/novell/eDirectory/lib64/apr:/opt/novell/eDirectory/lib64:/opt/novell/lib64:/opt/novell/eDirectory/lib64/nds-modules:$LD_LIBRARY_PATH

 
- Cause:  Installing Novell Identity Manager, versions 3.5.1 or 3.6 or 3.6.1, on a server that already has eDirectory 8.8.5 ftf1 will cause issues when IDM tries to load. Drivers will not start and give the errors -641 or -783.  Running ndstrace with +DXML and +DVRS flags will show the message:
DirXML JVM interface initialization failed <failed, -299 (0xfffffffffffffed5)>, unloading DIRXML
Suggestions: The easy fix in this case is to install eDirectory 8.8.5 ftf4.  See KB 7004299 for details on this problem.

- Cause: Corruption in the association between the driverset and the server.  a -602 will likely be seen loading vrdim with this issue.
  Suggestions: The easy fix in this case is to remove the association between the driverset and the server, cycle ndsd, add the association back again.  See Technical Information Document 7010712 - IDM does not auto load with error "Unable to find an active driver set", on this issue.


- Cause: Damage/corruption in the DirXML-ServerKeys attribute that resides in the local DIB's Pseudo-Server object
   Suggestions: DSDUMP (only done by Novell support) is needed to remove the attribute with  pre-IDM 3.6.  Use the new command within the dxcmd utility for IDM 3.6 and later.
- Cause: Using Dib Clone or dsbk will cause damage or corruption in the DirXML-ServerKeys attribute that resides in the local DIB's Pseudo-Server object
  Suggestions: DSDUMP (only done by Novell support) is needed to remove the attribute with  pre-IDM 3.6.  Use the new command within the dxcmd utility for IDM 3.6 and later.

- Cause: Too much memory allocated to eDirectory cache.
  Suggestions: Check the eDirectory cache configuration from iMonitor -> Agent Configuration -> Cache Maximum Size.  Try reducing this size if it seems high and reload dirxml.dlm (Windows) or restart ndsd (Linux).

- Cause: Corruption/damage/insufficient rights on the libraries IDM requires in the box (check the install log, also do an rpm -V on the IDM packages)
  Suggestions: This is by far the hardest to track down properly, but here are some options:
    > Use rpm -V liberally against all IDM-related libraries. The command line works well for that:
     rpm -qa | grep DXML | xargs rpm -V
    > ldd your IDM libraries to see the dependencies they have, and check those dependencies as well.  The checkbin.sh script can help greatly when checking dependencies, it is part of the ntsutils package that can be downloaded here: https://www.novell.com/communities/node/2332/supportconfig-linux
    > use strace / ltrace to track what is happening. strace works on scripts, so that will probably be the best option. ltrace can only be run against binary files. These tools provide a lot of information, and quite a bit of it requires basic C knowledge
    > For a quick solution, reinstall IDM on top of itself. That will simply overwrite the libraries/rights. If you run IDM 3.0.1 or 3.5.1 installer with the -DCLUSTER_INSTALL="true" option, it will only lay down the rpms again without extending the schema (even though it does show the messages that its doing that).

- Cause:   Issue with IDM schema may generate a -783 error.   You may see vrdim loaded in this case but when trying to load a driver it still generates a -783 error.
   Suggestion:   Re-extend schema using the idm-install-schema tool found in the /opt/novell/eDirectroy/bin directory on the server.


- Cause:   Server may have lost it's association with the driver set.  You may see vrdim loaded in this case but when trying to load a driver it still generates a -783 error.
   Suggestion:   You may have to remove the server from the driver set and add it back.   Caution:  Removing the server from the driver set and adding it back,  can cause the loss of server specific information on the drivers (GCV's, Driver Parameters) so the drivers may need to be redeployed from Designer, or a backup in order to restore that information.

If the above still does not resolve the issue, ensure the jclnt symbolic link is pointed to the correct directory.  View KB 7006778 for further information.