IDM Engine Event Caching
Novell Cool Solutions: AppNote
By Geoffrey Carman
Digg This -
Posted: 17 Oct 2007
The Identity Manager engine (Dirxml.nlm on Netware, dirxml.dlm on Windows, along with the dirxml.jar Java components) is the main power house in the IDM product.
One of the many great features of IDM is that it is event driven. That is, an event occurs in eDirectory, and IDM reacts to it. This is because eDirectory itself sees everything as events, and the IDM engine is tightly coupled to it. When an event occurs, the engine knows which drivers are associated with the current server it is running on, and what their filters look like. If your object class and attribute are in the filter, then the engine will cache that event.
The events are cached in the TAO files. (This is because the code name for the project was Tao for a while in the early days, and as we all know, it is so hard to go back and fix code names that are buried everywhere in the code that no one really ever bothers. Look at all the bhSomething schema attributes in your tree that come from the days when Novell Portal Services was called Black Hawk. Then NPS became the underlying engine for iManager. After it was all working, it was way too hard to go back and fix the schema names and all the code that references it!)
Each driver has a TAO file, named by the decimal value of the object ID (which is hex) of the driver object. Get that?
eDirectory identifies objects inside its internal database by object IDs. The Object ID is replica-specific. This means that if you have a replica ring of a partition with three servers holding replicas, then the Object ID of each object is assigned at the time it gets added to the server's local database. The main consequence is that each server may hold the same information about an object (say, an IDM DriverSet object), but each replica will almost certainly have a different Object ID for it in the local DIB set.
File system rights used to use Object ID's to reference objects in the directory, but with the advent of NSS on Netware and Linux, they moved to using a GUID (Globally Unique ID), a 128-bit number that is replicated and the same on all copies of the partition.
This was needed for several reasons, but one of the major ones is Cluster support. In the Netware 5.1 days, a cluster failover required that when a resource was shut down, the following things would happen:
- An NLM reads all trustees on the volume.
- The NLM store them as an XML file on the server shutting down the resource, as a map between Object DN's in the directory and Object IDs.
- The NLM reads the XML map and figures out what the current server's Object ID's are.
- These are replaced with the local server's Object ID values, which is how file system trustees were migrated.
As you can imagine, this took enough time to be annoying or crippling (depending on your perspective). With clustering in Netware 6 and higher, the GUID was used for trustees in the NSS file system. Because a GUID is synchronized and not replica-specific, all that work was no longer needed. Cluster failovers of NSS volumes were then super-speedy.
The easiest way I can think of to find the local object ID of an object is via iMonitor, which you are using already for Dstrace, right? This is at http://serverIpAddress:PORT/nds - where PORT is 8008 on Netware, 8010 on Windows, and 8028 on Linux/UNIX by default. iMonitor has an eDirectory browser built into it. Here's what you need to do:
1. Find the DirXML DriverSet object that holds your drivers, it will show you the driver objects as children.
2. Find the column named "Entry ID" - a hexadecimal 8 digit number.
3. Take that and paste it into a calculator and convert to decimal. That is the number you need.
(P.S.: In Windows > Start > Run, calc.exe will launch the Windows Calculator. Then there is a View menu with a Scientific option that will turn on Dec-> Hex conversion options for you).
Another way to do this is via DSBrowse on Netware, which will show the Entry ID for any object you look at as well. I used to think that you cannot get it via LDAP, but Aaron Burgemeister - who clearly knows more than I do - pointed out that it is available as localEntryID via LDAP. I was pretty sure it is just a level of detail that is not exposed at all. This is good to know!
There is a nice blog entry with a slightly different approach to this Hex-Dec Object ID issue at:
1. Go to where you eDirectory DIB is stored. This is SYS:_Netware on Netware, c:\novell\nds\dib on Windows, /var/opt/novell/eDirectory/data/dib on Unix/Linux, or wherever you installed it to. (Those paths may slightly vary depending on if you are using eDirectory 8.7.3.x or 8.8 and higher.)
2. On Netware, use TBX.NLM to look at the SYS:_Netware directory.
This is a very special directory that is hidden in many ways, since it holds the actual eDirectory database. (Once eDirectory went cross-platform it fell outside of Novell's control to limit access to the raw database files, and it is up to the administrator to hide them from view.)
3. Look for files named *.TAO, and more specifically, look for the decimal object ID you just figured out. That is the cache file for your IDM driver. On Linux/Unix, it will be uppercase "TAO", not "tao".
Now that we can look at the TAO file, we can do a few things with it.
First of all, an empty TAO file (no cached events) should be 12 bytes long. Events are stored in a binary format, so you cannot just look at the file directly - alas!
When the file size is greater than 12 bytes, there is an event in the queue. If the driver is not running, and it is disabled, then nothing queues up in the TAO file. If the driver is NOT disabled - whether it is running or not - events queue up. This has consequences for when you do not want to cache events on a driver. You need to disable it (which you can only do once it is not running) to stop the caching. Stopping the driver is not enough.
To view the contents of the driver cache you have basically these options:
- DXCMD, a Java tool that ships with IDM on all platforms. It is scriptable and takes a number of command line parameters. If you just run it, you login and get some menu choices. It is a very simple menu driven interface, where you select options from a list.
- The new iManager snap-ins, supposedly, which will have a cache browser. I have yet to see the iManager snap-in but I hear it is now available in the latest iManager 2.6 plug-ins for Identity Manager 3.5 (possibly for 3.5.1 the SP1 of 3.5). There was a blog posting recently about the new 3.5.1 snapin functionality and it looks GREAT! (Actually it looks better than great!) But it will require the 3.5.1 IDM engine to work, and the snapins will be released AFTER 3.5.1 is released.
- Driver Options, which lists your drivers.
To use Driver Options, you select the driver by its number in the list. You can start or stop the driver, and you can do a couple of other useful things at this point. Option 14 is the Driver Cache options. Within this you can view the cache and you can delete individual cached events, which can be useful.
NYCSUSE2:/tmp # dxcmd DirXML Command Line Utility version 2.0 Copyright (C) 2003-2006 Novell Inc., All Rights Reserved Enter user name: admin.services.acmeny Enter user's password: Logging in using: host: NYCSUSE2/10.1.1.2 user: admin.services.acmeny DirXML version is 18.104.22.16885. Driver set ACMEFLAT.services.acmeny.ACMEIDVAULT. is associated with the server. DirXML commands 1: Start driver 2: Stop driver 3: Driver operations... 4: Driver set operations... 5: Log events operations... 6: Get DirXML version 7: Job operations... 99: Quit Enter choice: Choose 3 for Driver Operations Select a driver driver name start option state ========================================================= 1: Active Directory Manual Running 2: CPS Manual Running 3: APPz Disabled Stopped 4: eDirectory Driver Manual Running 5: APPy Manual Running 6: IDVault - VPN Driver Manual Running 7: LinuxUnixSettings Manual Running 8: APPx-JDBC Manual Stopped 9: APPx-WO Manual Running 10: Move on Disable Disabled Stopped 11: Password Export Driver Manual Running 12: Solaris-Thin Client Manual Running 13: SRV_PWNotify Disabled Stopped 14: Triggered Actions Manual Running 15: UserApplication35 Manual Running 16: WorkOrder Manual Running 99: Exit Enter choice: 8 Select a driver operation for: APPx-JDBC.ACMEFLAT.services.acmeny.ACMEIDVAULT. 1: Start driver 2: Stop driver 3: Get driver state 4: Get driver start option 5: Set driver start option 6: Resync driver 7: Migrate from application into DirXML 8: Submit XDS command document to driver 9: Submit XDS event document to driver 10: Queue event for driver 11: Check object password 12: Initialize new driver object 13: Passwords operations 14: Cache operations 99: Exit Enter choice: 14 Choose 14 for Cache Operations Select a cache operation 1: Get driver cache limit 2: Set driver cache limit 3: View cached transactions 4: Delete cached transactions 99: Exit Enter choice: 3 Enter position token (default=0): Enter maximum transaction records to return (default=1): 10
The cached events will look just like the XDS document that show up in the Dstrace options. This should be very familiar to you. The TAO file does not store in a viewable format; it is in some binary format designed to speed up access. You need to use a tool like dxcmd to view the contents, alas.
This is good information to know. Every time you restart the driver, if you are having trouble processing the event, you will see the same event sent over and over again. This can be a good or bad thing.
If it is what you want, it is good. But if the event has hit on a bug or a bad IDM behavior, then it could be very bad! In that case, you will probably need to use DXCMD to clear the event out of the cache.
Select a driver operation for: APPz.ACMEFLAT.services.acmeny.ACMEIDVAULT. 1: Start driver 2: Stop driver 3: Get driver state 4: Get driver start option 5: Set driver start option 6: Resync driver 7: Migrate from application into DirXML 8: Submit XDS command document to driver 9: Submit XDS event document to driver 10: Queue event for driver 11: Check object password 12: Initialize new driver object 13: Passwords operations 14: Cache operations 99: Exit Enter choice: 14 Select a cache operation 1: Get driver cache limit 2: Set driver cache limit 3: View cached transactions 4: Delete cached transactions 99: Exit Enter choice: Enter choice: 3 Enter position token (default=0): Enter maximum transaction records to return (default=1): Enter name of file for response: <?xml version="1.0" encoding="UTF-8"?><nds dtdversion="3.5" ndsversion="8.x"> <source> <product version="22.214.171.12470719 ">DirXML</product> <contact>Novell, Inc.</contact> </source> <input> <modify class-name="User" event-id="PK_SEQUENCE=328288,table=CLIENTS,schema=IDM" qualified-src-dn="O=acmeny\OU=employees\OU=active\CN=joeuser" src-dn="\ACMEIDVAULT\acmeny\employees\active\joeuser" src-entry-id="34377" timestamp="0#0"> <modify-attr attr-name="APPID"> <remove-value> <value timestamp="1189810434#9" type="string">JKL</value> </remove-value> <add-value> <value timestamp="1190826654#2" type="string">JKL</value> </add-value> </modify-attr> <modify-attr attr-name="acmeABCProfile"> <remove-value> <value timestamp="1190154367#5" type="string">ABC</value> </remove-value> <remove-value> <value timestamp="1190382161#2" type="string">GHI</value> </remove-value> <remove-value> <value timestamp="1190382161#3" type="string">DEF</value> </remove-value> <add-value> <value timestamp="1190826654#4" type="string">ABC</value> </add-value> </modify-attr> <modify-attr attr-name="ABCUser"> <remove-value> <value timestamp="1190382161#6" type="state">true</value> </remove-value> <add-value> <value timestamp="1190826654#6" type="state">false</value> </add-value> </modify-attr> </modify> </input> </nds> Select a cache operation 1: Get driver cache limit 2: Set driver cache limit 3: View cached transactions 4: Delete cached transactions 99: Exit Enter choice: THEN A DELETE 1: Get driver cache limit 2: Set driver cache limit 3: View cached transactions 4: Delete cached transactions 99: Exit Enter choice: 4 Enter position token (default=0): Enter event-id value of first transaction record to delete (optional): Enter number of transaction records to delete (default=1): Select a cache operation 1: Get driver cache limit 2: Set driver cache limit 3: View cached transactions 4: Delete cached transactions 99: Exit Enter choice: THEN AN ERROR Select a cache operation 1: Get driver cache limit 2: Set driver cache limit 3: View cached transactions 4: Delete cached transactions 99: Exit Enter choice: 3 Enter position token (default=340): Enter maximum transaction records to return (default=1): novell.jclient.JCException: request -641 ERR_INVALID_REQUEST at novell.jclient.JClient.request(Native Method) at novell.jclient.JClient.ndsRequest(JClient.java:1197) at com.novell.nds.dirxml.util.DxCommand.sendWireRequest(DxCommand.java:892) at com.novell.nds.dirxml.util.DxCommand.readCache(DxCommand.java:1459) at com.novell.nds.dirxml.util.DxCommand.access$11200(DxCommand.java:44) at com.novell.nds.dirxml.util.DxCommand$ViewCacheHandler.run(DxCommand.java:4286) at com.novell.nds.dirxml.util.Menu.run(Menu.java:335) at com.novell.nds.dirxml.util.DxCommand$DriverCacheOpsHandler.run(DxCommand.java:4209) at com.novell.nds.dirxml.util.Menu.run(Menu.java:335) at com.novell.nds.dirxml.util.DxCommand$DriverOpsHandler.run(DxCommand.java:3071) at com.novell.nds.dirxml.util.Menu.run(Menu.java:335) at com.novell.nds.dirxml.util.DxCommand.mainMenu(DxCommand.java:1709) at com.novell.nds.dirxml.util.DxCommand.commandLine(DxCommand.java:380) at com.novell.nds.dirxml.util.DxCommand.main(DxCommand.java:337) Select a cache operation 1: Get driver cache limit 2: Set driver cache limit 3: View cached transactions 4: Delete cached transactions 99: Exit
You will note in the screen log above, I got a 641 error when trying to look at the cache file. This will always happen if the driver is currently running. To browse the cache file, you need to stop the driver. Unless your driver is stuck on an event, this almost makes sense, since the event is usually so fleeting you should not have time to see it.
But of course, when you need to see it most is when the driver is up and stuck, or up and hung for some reason. My hope is the new IDM snap-ins will be better able to get at an open cache file. Time will tell!
Now a couple of things about TAO files ...
They are empty, as I said above, and they are 12 bytes in size. As each event gets added, the files get bigger and bigger. However, they do not get smaller as each event is processed. Rather, when the last event in the queue is processed, the file drops back down to 12 bytes.
This makes sense when you think about it: appending data to a file is pretty easy, and it's computationally cheap. Pulling a value out from the middle of a file and shifting everything over to make it smaller is much more expensive for the system.
It used to be that there was a copy of the 12 byte TAO file on the Novell Support site, but that is no longer there. In principle you could just copy any TAO file, but Support does not recomend this. Rather, they suggest you stop the driver, disable it, then reenable it and start it. This deletes the TAO file and recreates an empty 12 byte file - at least until events start to fill it up!
Hopefully that gives you a good feel for how the caching in Identity Manager works, as well as providing some tools to understand it!
Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com