Cool Solutions

Using Global Configuration Values in XPATH

geoffc — Sun, 08 Nov 2009 09:41:35 -0700

Novell Identity Manager has several kinds of variables. Local variables are what the name suggests, and have two levels of scoping, driver and policy. Driver scoping means the variable hangs around after definition for the duration of the event, throughout the policies in the driver. This is a new feature in Identity Manager 3.5 and higher. Policy scoping, the default if you do not say otherwise means that the variable remains only through the duration of this policy object.

Global Configuration Variables (GCV from now on) are used to store things at the driver or the driver set level and can be very powerful. This is very useful if you are developing in a lab tree, before promoting the solution to production and your tree layout is different in the lab. You can store the references to base containers you use in a GCV and when it is time to move to production, just change those values to the correct ones for the production environment.

I detailed some examples of how to do that in a two part article about GCV'izing the Active Directory driver, see the following articles for some examples:

To reference these variables, there is a local variable token, and Global configuration variable token in Argument Builder, but a lot of fields do not use Argument Builder to specify. For example, when testing the condition, Source DN, you could want to test if it is in the Subtree held in a variable.

In the example articles, I showed how to use the notation ~GCVName~ to represent a GCV in policy. For local variables the notation is $VariableName.

An additional place where the use of variables like this comes in handy, is within an XPATH expression.

For example, set a local variable CurrentDN to the current objects DN, say via the Source DN() token. Lets say that the value is TREE-NAME\ACME\US\USERS\JSMITH

Have a GCV called eDirActiveUsers that holds the value, ACME\US\USERS and you want to test if the Current DN is the subtree of the eDirActiveUsers container. Lets say you also have a eDirDisabledUsers GCV that holds the value ACME\US\DISABLED

Now if you were using a condition, and you selected Source DN or Destination DN condition tests, then Policy Builder would give you a couple of options. You could test if it was in the subtree, in the container, equal, not equal and so on. So not much advice I can offer here, as it is already built in.

But what if CurrentDN is not the DN of the object in the current document? Say you are using a Work Order driver, and a Work Order is created, and it references an object elsewhere in the tree. Now you need to know before you proceed, is that user in the Active or Disabled container.

Well Source DN and Destination DN condition tests in an IF token or the main tests for a rule only refer to the current object. You could have some fun and store the value of Destination DN in another variable, lets call it CurrentDestDN and then do a Set Operation destination DN to the value of CurrentDN (the DN of the object the Work Order is referencing). Now the destination DN of the current operation is the one you want to test. Therefore you can use an IF token to test if Destination DN is in the subtree of ~eDirActiveUsers~ or ~eDirDisabledUsers~ as the case may be. Do remember when you done to set operation destination DN back to the value you stored in the CurrentDestDN variable if that makes sense in what you are trying to accomplish.

Now that previous example is a pretty stupid way to do it, but should work. It turns out there is probably a much simpler approach using XPATH.

The more I use and watch other people use XPATH the more I learn. There are a couple of sites with some examples of XPATH (and to me something as complex, Regular Expressions) at:

XPATH has a function called contains(), which takes two strings, and returns true if the second string is contained within the first string.

For more XPATH functions, you can look at:
http://www.w3.org/TR/1999/REC-xpath-19991116

This is a nice reference, as I often find myself thinking, gee it would be nice to get the length of this string, but no DirXML Script tokens support it, check out the XPATH link above, and lookee here, there is an XPATH string-length("string") function.

Back to our example,

contains("I am Bob", "Bob")

would return true. How does this help? Well XPATH in Identity Managers implementation also supports both classes of variables using $Local-Variable and ~GCVName~ notation.

So therefore

contains($CurrentDN, ~eDirActiveUsers~)

looks like it will work, right? Well it would except that the way Identity Manager handles a local variable versus a GCV is slightly different. Once you understand the difference it will be really easy to work with.

Global configuration values, when used in tokens are treated just like variables, and looked up each time you run across them in a rule. When I say look up, I mean retrieve from a list in memory, since a change to configuration of the driver, say changing a GCV but without restarting the driver will not cause it to take effect.

When you use the ~GCVName~ notation, something different happens. As the driver is started, it finds all instances of ~something~ and matches the 'something' part, then replaces it with the literal string stored in the GCV. Thus our line:

contains($CurrentDN, ~eDirActiveUsers~)

at run time will be interpreted as:

contains("TREE-NAME\ACME\US\USERS\JSMITH", ACME\US\USERS)

The $CurrentDN gets quotation marks because that is how variables are represented, but the GCV is a literal replacement of the string, so no quotes get added automatically.

Thus to make it actually work, we would need to use:

contains($CurrentDN, "~eDirActiveUsers~")

Then this would return true in our case, and you could react to that as appropriate.

Overall using a Global Configuration Value when you can for things that might change can be very powerful. I am a huge fan of using them.

Cool Solutions

Cool tricks using XPATH on nodesets

geoffc — Sun, 08 Nov 2009 09:41:17 -0700

Novell Identity Manager is a very powerful system that has many components. There is the core engine, that watches for events in eDirectory (dxevent) and sends them out to the drivers according to what the filters for each driver are listening for, processing the rules along the way (vrdim). There are driver shims for many different connected systems. There are well over a dozen different drivers for well known systems, some like the JDBC, LDAP, Delimited Text, and SOAP drivers are sufficiently generic to support dozens of different systems each. (For example the JDBC driver can talk to Oracle, MS SQL, Postgress, MySQL, or DB2 databases and that counts as a single driver). These drivers watch for events in their connected systems and send them to the engine for processing. Then there is the User Application which is a whole other world, almost a completely different product.

At the heart of all this is the engine, and in addition to seeing events in eDirectory it processes the rules, style sheets, and other objects that are part of the drivers.

There are many tools at your disposal for use by the engine. You can write your rules using XSLT style sheets, the original approach. This is probably the most complex approach, unless XSLT is something you are very comfortable with. NSure Identity Manager 2.0 introduced DirXML Script, which is an XML based language with nouns, verbs, tokens, actions, condition tests, and what not that allow you to process events in a much more user friendly manner. The tool used to build rules in DirXML Script, Policy Builder is very powerful, and is exposed in iManager's plugins for Identity Manager, and Designer for Identity Manager.

Within both XSLT style sheets and DirXML Script it is possible to use XPATH, the XML Path language to do a whole host of other things. XPATH includes a number of string processing functions like contains(), subtring-before(), and many more functions, see http://www.w3.org/TR/1999/REC-xpath-19991116 for the full definition. Note that only version 1.0 of XPATH is supported, not 2.0. Last I heard XPATH 2.0 was not fully backwards compatible and can break XPATH 1.0 rules. Thus upgrading could be very complex and the developers for Novell Identity Manager are still working out how to handle this issue.

One of the very cool things you can do with XPATH that greatly extends the power of Novell Identity Manager is the ability to call a Java function (I suppose any Java function you happen to have installed on the server). With Identity Manager 3.5 and higher they added the ability to call ECMAScript functions as well from XPATH. This way if you cannot do it in DirXML Script, you can write your own function in either Java and call it, or in ECMAScript and call it via XPATH.

In this article, http://www.novell.com/communities/node/4833/some-thoughts-xpath-novell-identity-manager I discussed a couple of general uses for XPATH, and the basic approaches you can take to using it.

Another function that came up recently for me, that turns out to be extremely powerful is using the DirXML Script action, do-strip-xpath action. Now I always thought of using this action to remove a value from the current XDS document. For example, you want to remove a <remove-all-values> node from the current operation, you could do-strip-xpath with an XPATH expression of something like <do-strip-xpath expression='modify-attr[@attr-name="SomeAttribute"]/remove-all-values'/> and this is a pretty useful thing to be able to do. Especially in the case where you know a remove all values is going to come from the connected system on a particular event, but your REALLY do not want it to do that. This way you can remove it from the XDS event document before it gets sent to eDirectory (or vica versa, depending on the channel you are currently in).

What I recently was pointed at, is the fact that you can use the do-strip-xpath action against any nodeset, not just the current operational document. After all, both are just DOM objects in memory.

The most common nodeset I deal with is the result of a Query token. For more on the Query token you could read the article:
http://www.novell.com/communities/node/4906/the-qu...

A particular example I ran into was querying the file and print eDirectory tree for ACL values. I was trying to find users with excessive amounts of rights. Turns out there were all sorts of optimization tricks I used, but this one is still my favorite. I queried back for all Organizational Units with ACL attributes on them, and that generated a relatively large document that I stored in a local variable as a nodeset. Now a nodeset in this context is really a DOM object stored in memory.

The thing is, I knew there were a stack of ACL's returned by my query that I could care less about, and really did not want to waste time processing.

Things like rights granted to WM:Workstation Registration. Turns out this tree had Zenworks 2.0 installed in it years ago, and in Zenworks 2.0 to register a workstation, the container object get granted W to itself on the attribute WM:Workstation Registration. This allowed the Zen workstation registration tool (wsreg32.exe was it?) to write a string with all the info about the workstation on the container. Then in NWAdmin (who remembers that anymore?) you could use a plugin to create Workstation objects from the registration info. Overall, this was an approach to the problem. Probably not a GOOD approach, in hindsight, but hey that was almost ten years ago now, so I will cut them some slack. (The problem was that you could easily get hundreds to thousands of registration values on a container object that had to be synchronized and really had no long term value. Also, even after you were done using it, old clients would continue populating the values onto the container object. At the same time, NDS (this was before they renamed it to eDirectory) had some issues with hugely multi valued attributes. Now a days it would not be a big deal. (Worth noting that Dsrepair still reports objects with over 1000 object references, even though it does not really matter that much today).

Anyway, every container in the tree had WM:Workstation Registration set, and normally that would be the type of ACL I would want to notice, except that it really does not matter to me. I considered looping through the nodeset in a for-each loop, and testing that it was not WM:Workstation Registration, but that turned out to take a reasonable amount of time. I.e. Looping through a few thousands nodes can take more time than you might like.

Then I had all the other default ACL's that I did care about like R to Login Script on every container. Again, that is one more test inside that loop through all the ACL values. More time.

Father Ramon from the Novell Support Forums (You do read and search the Forums when you need help, right? Of course you do! There is a web interface at http://forums.novell.com where you can login with your Novell ID that you use for opening incidents and downloading patches, and there is an NNTP interface at the same IP name. nntp://forums.novell.com and look for the novell.support.identity-manager.engine-drivers forum. Lots of excellent content there), suggested that I use do-strip-xpath against the local variable nodeset.

Thus for the local variable ACLS the line below accomplishes what I needed all by its lonesome.
<do-strip-xpath expression="$ACLS//value[(component[@name = 'protectedName'] = 'WM:Registered Workstation')]"/>

This is even more powerful when you realize that ACL is structured attribute with three components. If you want to set such a value, you need to provide all three components, like in this action:

<do-add-dest-attr-value class-name="User" name="ACL">
	<arg-value type="structured">
		<arg-component name="protectedName">
			<token-text xml:space="preserve">CN</token-text>
		</arg-component>
		<arg-component name="trustee">
			<token-text xml:space="preserve">\ACME\lab\users</token-text>
		</arg-component>
		<arg-component name="privileges">
			<token-text xml:space="preserve">33</token-text>
		</arg-component>
	</arg-value>
</do-add-dest-attr-value>

The ACL is written to an object that the right is granted upon. I.e. If the ACL is on the Tree Root, then the right specified by the value of the ACL is being granted at the Tree Root level. (This could be very bad, if you are not careful! This right will potentially inherit all the way down the tree! So please think twice, then a third time if doing something like this at the Tree Root level.)

Then the component 'protectedName' is the attribute that the trustee is being granted for. This would be the Attribute name in the eDirectory schema. The example above is for CN, but any attribute is possible. There are some special ones like [All Attributes], [Entry Rights], \[Inheritance Mask] (for Inherited Rights Filters), and possibly others.

The component 'trustee' is the user who is being granted the ACL to the object the ACL is written on. So in our case, it looks like a container named users.lab in the ACME tree is getting the right to the CN attribute, on wherever our destination object happens to be. (The example is a simple snippet, so that detail is not specified, but it could be!)

Finally the 'privileges' component is a bit mask of the values you set in the NDS Rights tab in Console 1, or via iManager, (or via NWAdmin32 even! Any tool than can manipulate eDirectory ACL's can do this. Technically that is what our set destination attribute example above is doing as well). The mapping of the values depends if it is an Entry or Attribute right, but for simplicity in the Attribute case, 33 is 32 for Supervisor rights plus 1 for Compare rights.

Alas, you cannot query for components of a structured attributes, which also would have solved my problem.

So instead, what we do is strip by XPATH the expression:
$ACLS//value[(component[@name = 'protectedName'] = 'WM:Registered Workstation')]

$ACLS means in the context of the variable $ACLS (which is conveniently a nodeset), //value means find any occurrence of a value node, with the predicate of component[@name = 'protectedName'] = 'WM:Registered Workstation'). Within that predicate, look for the component named 'protectedName', whose value is WM:Registered Workstation.

So this removes all value nodes where the trustee of the ACL is WM:Registered Workstation. Cool.

Your variable goes in with hundreds or thousands of values, and in one line of DirXML Script, hundreds of values I want removed, are removed! Yay!

Then do it again for Login Script:
<do-strip-xpath expression="$ACLS//value[(component[@name = 'protectedName'] = 'Login Script')]"/>

Then do it again for SAS:Login Configuration and SAS:Login Configuration Key, and now your thousands of nodes in your nodeset is a much smaller set that when you loop through it, takes hugely less time!

Looking at the trace on a SUSE Linux box, running eDir 8.8.1 FT2, on modern server hardware, it took about 15 milliseconds to remove all the ACL's with a privileges value below 4 (which would remove values of 1, 2, or 3 which would be Read and Compare or just Read or just compare, it is a bit mask) and there were TONS of those. The rest of do-strips took between 1-5 milliseconds.

You just cannot run through a for-each loop that fast.

So for my example I did 6 do-strip-xpath operations, and it took about 19 milliseconds total.

My initial brute force approach took over 40 minutes looping through all the values and testing and doing much more work. This approach cut the total time down to under 2 minutes. That is pretty darn fine! (10,000 object tree with all sorts of weird historical ACL values set all over the place).

I never would have thought that a local variable nodeset can be treated in a similar way to the current document nodeset and tweaked this way, but boy did it work well!

Now that I have this hammer of do-strip-xpath, everything is starting to look like a nodeset nail... Every time I have a nodeset variable I keep trying to think of a way to use this on it somehow.

Cool Solutions

Using XPATH to examine Association values

geoffc — Sun, 08 Nov 2009 09:40:05 -0700

Novell Identity Manager has an advantage over some of its competitors based on how it stores the association values for synchronized objects. Other products may use a directory to combine all the data about the users, but often they do not store the information about associations, that is connections between objects in the different systems, within that directory.

The way Novell Identity Manager does it is quite advantageous, since you can examine the directory on any object and see what the association values are, without needing to look into another location to retrieve the information.

The values are stored in eDirectory in an attribute called DirXML-Associations. The DirXML-Associations attribute uses an interesting syntax, called Path syntax. Path is a compound or structured attribute syntax that has three parts to it. An integer (I think it has a max range, but we only ever use 0-5 so not sure) named nameSpace, then a distinguished name syntax field called "volume", and finally a string field called "path". This was designed to store things like a home directory path, where you need to specify the name space the path is in, (DOS, NFS, LONG, OS2, MAC, etc... Shades of the old NetWare days eh?), then the distinguished name of the volume object, and finally a free form text field for the path on the volume.

The DN reference to the volume is the reason why if you deleted a server or volume object in eDirectory, all the Home Directory mappings would go away. DN syntax fields do not store the string of the DN of the object, rather it stores a 32 bit integer that describes the entry in the eDirectory database. Thus every tool that displays the DN to a human, basically does a look up for the current display name of the object. Thus renames, and moves, are automatically displayed correctly at all times. However, if you delete the object and then recreate it, even when the names are the same, the object IDs are different (eDirectory may eventually reuse an object ID, but there is no way to force it or request it, so you can basically be guaranteed a different object ID).

That does bring up an important point with DirXML-Associations values, and if you are updating the drivers. You might be tempted, when the time comes to do an upgrade to the system, to just delete all the DirXML Driver objects and re-import them from your lab model, but this illuminates a big pitfall. All the DirXML-Association values are based on the DN (the Volume part of the structured attribute) of the current DirXML Driver object. If that gets deleted, all the DirXML-Association values just magically go away.

If you have to take that approach, then delete everything UNDER the Driver object, but leave the Driver object behind so that the DirXML-Associations remain.

There are several good tools to back up the values. From the commercial world Blackbird Detroubler (http://www.blackbird-group.com/)is one of the best tools for backing up eDirectory in general, since it is smart enough to recognize cross links between objects. That is, if you deleted your DirXML Driver object accidentally (or on purpose) and regret it, it can restore the objects, but also it can restore the association values on all the objects that had them. It is quite impressive, and to the best of my knowledge a unique feature in the market. Other products will restore the objects (possibly with the same object ID, but not likely) and then it is up to you to restore the references on the other objects.

From the free world, Novell Professional services out of Hungary released the super cool tool, DAModifier. You can get it from: http://www.npsh.hu/associmod_en.html or http://www.novell.com/communities/node/958/association+modifier

This free tool is a Windows tool, that requires an NCP over Client 32 connection. You can pick the tree, the driver object, the context to search in, and the kind of association value you wanted. You can use it to show the current values, and save it to a file, and at a later date restore them from a file. This gets around the DN syntax issue, by storing and referencing the names of the DirXML Driver object, so that when it goes to restore the value, it has to lookup the actual current Object ID that belongs to the name.

For a listing of the various known association values (The "Path" part of the structured attribute) check out this article on the topic:
Open Call - IDM Association Values for eDirectory Objects

Within a driver, there are a couple of DirXML Script tokens, conditions, and actions that can look at Association values.

In the Condition block of rules, you can select an If association, and the tests are things like, Associated, not associated, available, and then the usual set of equal, not equal, less than, not less than, and friends.

The less than, greater than tests do not seem all that useful, until you consider that the value of the association, the "Path" part of the structured attribute is just a text string. So perhaps in the case of a driver to a Unix style system, it might be storing uid numbers, in which case testing if the number is below 1024 could be useful and tell you something about the user. (Now it happens that the bi-directional Linux Unix driver happens to use the users CN followed by the string User or Group, so a User named jsmith would have an association value of jsmithUser and group named Shovellers would have an association value of ShovellersGroup but it was a nice example even if it makes no sense in reality). Some databases might use the value of a primary key sequence number (say a column called pk_sequence) that is incremented, and you want to know if a users row in the database is below or above some value.

In the Action block of a rule, you can Add Association or Remove Association. Remove Association is a great action, if you want to dis-associate users after an event, say if one system deletes the user, we often convert the delete event into a Remove Association and disable user (Set Login Disabled equal true). Lets say in the case of a JDBC driver, where you are synchronizing a view, instead of a real table, and there is some criteria that defines your view, and users can change to be in or out of the view. Say some value is Active or Inactive and that defines your view. When they go inactive, it would come across as a delete, since it has disappeared from the view. In a case such as that, you would want to remove the association so that if the user becomes Active again, and appears in the view, you would want to match on the user again, but if the user still has an association value it will not match correctly.

In the Argument Builder, basically anywhere you can build a string, you can use the Association token to use the current association value. Source Attribute and Destination Attribute can reference a specific object other than the one in the current operation if you give a DN or Association value to look for.

The problem of all these approaches however if that they are focused on the Association of the current object, and for the current driver. There are many powerful use cases where in a different driver (often a Null or Loopback driver) it would be useful to know if the user is associated to a particular system.

The good news is that with XPATH you can accomplish much of what you need. I am trying to do a series of articles on the use of XPATH in Identity Manager, since the context for an Identity Manager event (which is key when selecting in XPATH), is different than you would see in documentation for XPATH elsewhere on the web. You can see the previous articles at:

You can query for the DirXML-Associations on an object with the Query token quite readily.

For example this action will set a local variable ASSOC-VAL with a node set filled with the data from the Query for a user with a workforceID of the local variable WORKFORCE-ID.

<do-set-local-variable name="ASSOC-QUERY" scope="policy">
	<arg-node-set>
		<token-query class-name="User" datastore="src">
			<arg-match-attr name="workforceID">
				<arg-value type="string">
					<token-local-variable name="WORKFORCE-ID"/>
				</arg-value>
			</arg-match-attr>
			<arg-string>
				<token-text xml:space="preserve">DirXML-Associations</token-text>
			</arg-string>
		</token-query>
	</arg-node-set>
</do-set-local-variable>

This will return a document something like the following into the node set.

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product version="3.5.11.20080307 ">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <output>
    <instance class-name="User" qualified-src-dn="O=LAB\OU=EMPLOYEES\OU=NEW\CN=JSmith" src-dn="\ACME-LAB\LAB\EMPLOYEES\NEW\LJohnson1" src-entry-id="56795">
      <attr attr-name="DirXML-Associations">
        <value timestamp="1217007039#71" type="structured">
          <component name="nameSpace">1</component>
          <component name="volume">\ACME-LAB\LAB\SERVICES\IDVAULT\APP-JDBC</component>
          <component name="path">PK_SEQUENCE=350110,table=CLIENTS,schema=IDM</component>
        </value>
        <value timestamp="1217007039#72" type="structured">
          <component name="nameSpace">1</component>
          <component name="volume">\ACME-LAB\LAB\SERVICES\IDVAULT\LinuxUnixSettings</component>
          <component name="path">YE98+yxf3AGAleAAAwAAAA==</component>
        </value>
        <value timestamp="1217007039#73" type="structured">
          <component name="nameSpace">1</component>
          <component name="volume">\ACME-LAB\LAB\SERVICES\IDVAULT\UserApplication35</component>
          <component name="path">"AnAssociation"</component>
        </value>
        <value timestamp="1217007039#74" type="structured">
          <component name="nameSpace">1</component>
          <component name="volume">\ACME-LAB\LAB\SERVICES\IDVAULT\Active Directory</component>
          <component name="path">f0648eab27d6da4283246583112d6319</component>
        </value>
        <value timestamp="1217007039#75" type="structured">
          <component name="nameSpace">1</component>
          <component name="volume">\ACME-LAB\LAB\SERVICES\IDVAULT\Corporate Password Sync</component>
          <component name="path">E101738</component>
        </value>
      </attr>
      </instance>
      </output>
      </nds>

You could also use the Source attribute token if you are interested in the current user object, or can specify its DN or Association value. That might look like:

<do-set-local-variable name="ASSOC-VALS" scope="policy">
	<arg-node-set>
		<token-src-attr name="DirXML-Associations"/>
	</arg-node-set>
</do-set-local-variable>

That will generate a query and result much like above in DSTrace, but the value held in the node set will be different, which means the XPATH to get the values will be different.

For the first case, where we used a Query, the nod set is based on the instance document which is where the current context is, so something like the following (assuming you know the DN of the driver you want to test for and it is stored in the local variable TARGET-DRIVER-DN):

$ASSOC-QUERY/attr[@attr-name="DirXML-Associations"]/value/component[(@name='volume') and (text()=$TARGET-DRIVER-DN)]/parent::component[@name='nameSpace']/text()='1'

<if-xpath op="true">$ASSOC-QUERY/attr[@attr-name="DirXML-Associations"]/value/component[@name='volume']/text()=$TARGET-DRIVER-DN</if-xpath>
<if-xpath op="true">$ASSOC-QUERY/attr[@attr-name="DirXML-Associations"]/value/component[@name='nameSpace']/text()='1'</if-xpath>

This ought to do it in one operation. Test that the component named volume, with a text of the DN of the target driver local variable, is true and then up a node to the parent, and look for the component named namespace, having a value of 1. That will evaluate true if one of the nodes meets those requirements.

For a the second case, where the node set contains the values returned by a Source attribute token, although the same basic instance document is returned, the current context is different and instead of being at the top of the instance document, the nodes are just the values (without the attr node above it).

Therefore you would have to use different XPATH, in this case it is easier to do it in two XPATH statements, that both need to be true.

<if-xpath op="true">$ASSOC-VALS/component[@name='volume']/text()=$TARGET-DRIVER-DN</if-xpath>
<if-xpath op="true">$ASSOC-VALS/component[@name='nameSpace']/text()='1'</if-xpath>

The node set is basically composed of several nodes, one per Value node returned for the user, so you want to check right at the component level, instead of the attr[@attr-name="DirXML-Associations"] level.

If you needed to do something with the value of the association, you could select it into a variable with an XPATH that looks something like setting a local variable equal to the XPATH expression, depending on the

$ASSOC-VALS/component[@name='path']/text()
or
$ASSOC-QUERY/attr[@attr-name="DirXML-Associations"]/component[@name='path']/text()

just after you had gotten to the correct node with a test as shown previously.

Cool Solutions

Using String Compares in XPATH Statements

geoffc — Sun, 08 Nov 2009 09:39:55 -0700

Novell Identity Manager supports several languages for manipulating events that come across as XDS documents. With DirXML 1.x came the use of XML Style sheets (XSLT) which is still supported. You can still see some rules in XSLT in some of the drivers. With NSure Identity Manager 2.0 came DirXML Script an XML based language that is very well suited for manipulating events.

Common across both languages has been the use of the XML path language, XPATH. One of the biggest issues with using XPATH in Identity Manager installations, is that most documentation and examples are written with a web or HTML focus. That is all fine and good in the abstract, but when trying to apply it to real world examples in the Identity Manager experience, much of the online documentation is not that helpful.

It is always useful to go straight to the source, and look at the RFC for XPATH 1.0. The RFC can be found at: http://www.w3.org/TR/xpath.

In order to make XPATH more available and understandable for others, I have been working on a series of articles about interesting things in XPATH, as they relate to Identity Manager. You can see some of the previous articles at:

Some thoughts on XPATH in Novell Identity Manager
Using Global Configuration Values in XPATH
Using XPATH to examine Association values
Cool tricks using XPATH on nodesets
The different attribute options in Identity Manager
Learning XPATH for IDM (This one is not mine, but still a good read!)

One of the neat things to do with XPATH is to do some string manipulations that the default DirXML Script token set does not include. For example, DirXML Script includes a substring token, but that is placement based. I.e. Substring from position 3 for 12 positions. This is very useful, but sometimes you want to do a slightly different substring function, say substring before the @ sign in an email address, to get the username part of the email address, or substring after the @ sign to get the domain name for the email address.

XPATH handily enough has a pair of string functions, substring-before() and substring-after() to help out with this. Thus, you could have the email address in a local variable EMAIL-ADDRESS and then call the XPATH functions substring-before($EMAIL-ADDRESS,"@") or substring-after($EMAIL-ADDRESS,"@" to get the information from the email address that we just discussed.

One big issue to watch out for in XPATH is that the functions are case sensitive. We get a little spoiled, since most string compare/tests in DirXML Script default to being case-insensitive. That is, a value of "Brian" is the same as "brian", is the same as "BrAiN" in most cases. In XPATH, this is not the case. Usually to get around this I just upper case the two strings I am about to compare while they are in local variables, just to be on the safe side.

There are a bunch of other string processing functions defined in XPATH 1.0, and some can be very useful, like contains(). A good example on how to use contains() is in this article:
Using Global Configuration Values in XPATH where the idea is to use contains() to recreate the "in subtree" test, that we get in DirXML Script for Source DN and Destination DN. The reason you might need this, is that the "in subtree" test is only applicable to the current objects source or destination DN. What if you had another object, say a group the user is a member of, and you want to test if the group is in a particular subtree. You could set the value of the attribute into a local variable, and then test if XPATH of contains($GROUP-DN,"\ACME\GROUPS\HR") is true or false.

There are many other functions that are useful like:

string() which converts the value you give to a string data type. Usually not necessary, but good to have available. Like number() that I discussed in: XPATH and math it is not always needed, but nice to have!

concat() which concatenates two strings together. Takes two strings, and outputs the concatenated string.

starts-with() is great for finding strings that start with some value. You could use a regular expression compare in DirXML Script and look for the regex token ^something where the carat symbol (^) indicates it needs to be at the beginning of the string.

string-length() is very useful, when you are trying to validate data. If you know the location code from the HR system is always four characters, a very quick test is if the XPATH string-length($LOC-CODE)=4 is true.

normalize-space() is a great function. This replaces multiple white spaces with a single one, and strips off leading and trailing white spaces. Very useful for cleaning up data coming from a database or the like, where a leading space is a valid case, but you would prefer to avoid that.

substring-before(), substring-after(), and contains() we already talked about above.

translate() is a function I do not often used, but it is meant to replace certain characters with others, notionally to allow simple case conversion, but is not sophisticated enough to do it for all languages, and so future support is planned for better case conversion.

One problem with using string functions in XPATH is that you cannot do wildcard compares. That is you cannot compare a $VARNAME=test* and expect it to work. The asterisk (*) is meant for node tests, where a single asterisk (*) means any node of the principal node type of the given axis. For example, child::* matches any element, attribute::* or @* matches any attribute.

In the Identity Manager implementations of XPath (which is 1.0, not XPATH 2.0), you can however use methods from java.lang.String that support regular expressions as extension functions, e.g. modify-attr[jstring:matches(@attr-name,'ACMEPROFILE.*')] (where jstring have been mapped to http://www.novell.com/nxsl/java/java.lang.String) or modify-attr[java.lang.String:matches(@attr-name,'ACMEPROFILE.*')] This is a DirXML Script provided functionality and is not available in XSLT.

Overall XPATH gets us a lot of functionality, the only real issue is that it is more complex than other approaches. However it is good to have multiple approaches in your toolkit for when you need it!

Cool Solutions

XPATH and math

geoffc — Sun, 08 Nov 2009 09:39:49 -0700

XPATH and math:

Novell Identity Manager has several languages supported to process events in the flow of events. There is the original XML Style sheets (XSLT) from the DirXML 1.x days, and with the release of NSure Identity Manager 2.0, DirXML Script got added. Along the way, the ability to call out to Java classes was maintained, and with the release of Novell Identity Manager 3.5 we got the ability to call out to ECMA Script functions (aka Java Script).

Throughout all of that we have the ability to use XPATH, the XML Path language in both DirXML Script, XSLT, and I think even ECMA Script.

I have been working on a series of articles about XPATH in Novell Identity Manager, and here are some links for the ones I have written so far:

and some other good articles by others:

In the article Some thoughts on XPATH in Novell Identity Manager I talked about some of the XPATH features we can use in Identity Manager.

One of the discussions in that article was the difference between selecting a node via XPATH (aka a node test) and doing something, like math, with XPATH.

There are a couple of caveats to using XPATH for math that are worth thinking about, and once you have thought about them, they are of course very obvious, and easy to remember, but until you spend that moment to think about it, they are somewhat non obvious.

First off, there is a number() function, that is useful to use, when you want to be sure the value you are about to use is really an integer. For example, you might use a counter variable in a for-each loop. In that case, you might initialize it before the loop begins by set local variable COUNTER to the string 0. Now is that a string or an integer, as far as XPATH is concerned? In my experience, most of the time, that is sufficient to be treated as an integer. However, it is easy to be certain, just use the number($COUNTER) function, whenever you use the variable for math.

Thus inside your for-each loop, you can increment it either as. set local variable equal to the XPATH $COUNTER + 1 or perhaps to be safe the XPATH number($COUNTER) + 1.

Subtraction is the next issue. From the XML Path language RFC at http://www.w3.org/TR/1999/REC-xpath-19991116 there is a note.

NOTE: Since XML allows - in names, the - operator typically needs to be preceded by white space. For example, foo-bar evaluates to a node-set containing the child elements named foo-bar; foo - bar evaluates to the difference of the result of converting the string-value of the first foo child element to a number and the result of converting the string-value of the first bar child to a number.

What that is basically saying is you need a space on either side of your minus sign. So while it might look like number($COUNTER)-1 should work, you really need to write it as number($COUNTER) - 1 which is a pretty easy thing to accommodate.

The reasoning being pretty obvious once you read the note. The dash symbol is valid in variable names, so unless it has spaces to either side, it is not clear to the parser that you mean do math, instead of using it as part of the name.

Trivial, but non obvious. In fact, I had stumbled upon this issue in a funny fashion, helping out a friend. I had suggested the XPATH $COUNTER+1 and that worked, but he was trying to then do subtraction and $COUNTER-1 was not working. I suggested he try number($COUNTER) - 1 thinking it was the number() function that would do the trick, but as I typed the suggestion, I had put in spaces around the dash (minus sign), without thinking about it, aiming only for clarity. Turns out I had gotten the correct idea without knowing why. The joys of learning complicated systems!

Next up is division. There is a mod operator, that acts much as you would expect, it returns the remainder from an integer division operation. I had used that before and had no issues.

I usually use the mod operator when I want a delay loop, because of replica synchronization. That is, an event happens to two objects on one replica in a driver there. The driver on this second box needs to see the event on both of them before proceeding, so since I cannot guarantee both changes will synchronize at the same time or even close enough, I usually build a test loop.

The loop uses a counter variable, that runs for a GCV called RetryCount (defined as say 120 or so times) to use the Query token for the needed attribute. The Query token as described in the article More thoughts on Source/Destination/Operation attribute tokens in Identity Manager always checks, and does not read from the cache, so it keeps trying to look for the second object. Then after that first number of loops, I pause for a GCV RetryPauseTime (defined as say 1000 milliseconds), and finally after a GCV RetryMax, (defined as say 601, or 10,000 tries) I exit the loop entirely and decide the second object never appeared. You can use the mod operator to divide the current counter value by the RetryCount GCV and show the left over. Whenever it is zero, it is time to take a break for RetryPauseTime.

Well that was mod, which is nice, now on to division. Without thinking about it, I tried an XPATH of number($COUNTER)/number($MEM-FREE) in a rule, and the driver refused to start, with the error:

DirXML Log Event -------------------
     Driver:   \ACMEIDVAULT\acmeny\services\FLAT\BlockingActions
     Channel:  Subscriber
     Status:   Error
     Message:  Code(-9130) Error in vnd.nds.stream://ACMEIDVAULT/acmeny/services/FLAT/BlockingActions/Subscriber/%5Bacme%5D+acmeEnt
itlementCounter+tools#XmlData:508 : An invalid XPATH expression 'number($COUNTER)/number($MEM-FREE)' is specified: java.lang.IllegalArgumentException: DOMEvaluator parser error: a node-test was expected.

Here we have the same basic issue. A node test in XPATH, usually includes the forward slash symbol, which I always think of as division. But in XPATH it is not division. Again, completely trivial once you spend a moment to think about it, but not obvious at first glance.

Turns out the real division operator is div. Easy enough to fix, change the XPATH number($COUNTER)/number($MEM-FREE) to number($COUNTER) div number($MEM-FREE) and we are good to go.

Has anyone else run into other similar obvious in hindsight things about XPATH, that are worth mentioning? Please feel free to comment or write an article about it! The more the merrier!

Cool Solutions

XPATH and the context node

geoffc — Sun, 08 Nov 2009 09:39:44 -0700

Note: I wrote another attempt at this that is helpful as well, in trying to explain this concept, you can find the new article at: Another Attempt at Explaining the Context Node".

Novell Identity Manager allows you to use a number of approaches to manipulate events that occur in the Identity Vault and in the connected systems.

With the original release of Identity Manager, when it was still called DirXML, the only real option was XSLT. XSLT is still available and some of the drivers still use it. XSLT is very powerful when you need to convert the syntax of the document from one XML based dialect to another, which is why the SOAP and Delimited Text drivers use it. The SOAP driver converts the XDS dialect that Identity Manager uses to the SOAP document dialect (SPML, DSML, or other) that you are trying to communicate with and then in the other direction as needed (Depending on if the event is in the Subscriber (XDS -> SOAP) or Publisher (SOAP -> XDS) channel).

The Delimited Text driver converts XDS event documents to comma separated values (CSV) and vica versa using XSLT.

The SAP HR driver requires the use of XSLT in order to format a funny looking Query document that is used to get the relationship between people (Technically Persons, but I find that plural of the word Person funny looking), Jobs, Positions, and Organizations within the HR system. The query uses nodes that the DTD supports, but are hard if not impossible to do any other way, so XSLT is probably your best approach to using them.

With NSure Identity Manager 2.0 we got DirXML Script, an XML based language that is used to manipulate events. With each subsequent release of Identity Manager more features have been added to DirXML Script. New verbs, tokens, actions, and conditions have been added, as well as interesting enhancements to existing tokens and functions. You can read more about those new features at:

One thing that has stayed the same, is that when it comes time to select parts of the document, do math, or call out to external Java classes you need to use XPATH.

XPATH is a tricky thing, where some parts of it are really easy (Like some of the string and math functions), other parts are pretty straight forward, and things like selecting can be simple in principle but surprisingly tricky in practice.

One source of information about XPATH is the RFC that defines XPATH:
http://www.w3.org/TR/1999/REC-xpath-19991116

The thing is that the RFC does not really provide examples, and most web resources on XPATH are confusing since they focus on using XPATH for HTML document manipulation, which is not obvious how they might apply to XDS event documents.

Other than that, there are a couple of good sites with some examples:

http://wiki.novell.com/index.php/XPATH_Examples
http://ldapwiki.willeke.com/Wiki.jsp?page=IDMCodeS...

I have been trying to write some articles about interesting XPATH tidbits, you can read more at:

We still need more I think, so here goes another one.

In the article about using XPATH for doing math versus selecting a node in a node set (Some thoughts on XPATH in Novell Identity Manager) I discussed a couple of ways you can use XPATH. What seems to be most confusing to people is where are we starting from, also known as: What is the current context node.

What illustrated this for me was using the XPATH Simulator in Designer. I kept trying to use it, and nothing ever seemed to work. I was sure I had a valid XPATH selection string, and it should have selected something in the sample document, but I just could not get it to do it. Finally, I walked through the XML document in the object view, and selected the <output> node, and suddenly results appeared!

The issue had been all along that what matters most for XPATH in selecting, is the context node. That is, where are we starting from. There are a number of selection methods that allow you to specify, anywhere this node occurs, but rarely is that sufficiently fine grained for us in an Identity Manager world.

For example, a node selection of @src-dn is one of the most common ones I find myself using. That is, select the value of the XML attribute of the current context node, called src-dn. The part is so important is the "of the current context node".

For example, if you query for some objects, store the returned node-set in a local variable and then want to loop through them and read out the src-dn then of course the XPATH select statement of "@src-dn" will not be useful, it will be selecting the same thing every time, the src-dn of the originating event.

Thus inside your loop you would be selecting for "$current-node/@src-dn" which leverages a built in local variable for the current node of any looping structure you might be in the middle of. Thus we specify the context to be the local variable, which comes from the loop, and how it looks as an XML node set depends entirely on what the node set you are looping through looks like.

The two most common node sets you will encounter in Novell Identity Manager for doing this sort of task are probably the results of a Query token or the result returned from the use of one of the tokens, Source Attribute, Attribute, or Destination Attribute. (For more on the difference between the three attribute tokens, please read:
The different attribute options in Identity Manager
More thoughts on Source/Destination/Operation attribute tokens in Identity Manager ).

Prior to Identity Manager 3.5, there was no Query token, (for more information see: The Query token in Identity Manager
Examples of using the ParseDN Token in Identity Manager) and we had to use the Java Command Processor. The Java Command Processor is still available for use, but not really needed any longer, since the three attribute tokens (Source Attribute, Attribute, and Destination Attribute), and the Query token very nicely wrap it in a trivial to use interface. That is probably my favorite part of a new Identity Manager version release! Seeing what thing that was a little tricky in the past, is now wrapped in an easy to use interface instead. (Unique name is another example of a wrapper of a reasonable amount of logic, based around the query functions, to generate, and then test for Unique names in a directory)

Regardless of which of the four options you choose to use, the engine in the background will generate a query document. It will ask the source or destination, depending on what you chose, to find all objects of the class you specify (or don't specify), that match some condition (or no match specified), and return a list of attributes (or all attributes if none specified). There are all sorts of interesting ways to tune this, by setting the number of returned values (query-ex functionality, see: The Query token in Identity Manager ), limiting the subtree, or specifying a full DN to return a subset of objects, or a single objects values respectively.

The Source Attribute, Attribute, Destination Attribute tokens basically do a query for a specified DN with no match criteria (since we know the DN, the current objects src-dn or dest-dn. Or in the case of Source and Destination Attribute tokens, we can specify the objects DN or association value) for a specified attribute.

The query document will look something like:

<ndsextra version="3.5" ndsversion="8.x">
  <source>
    <product version="3.5.10.20070918 ">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
    <query class-name="Organizational Unit" dest-dn="com\acme\People\GA\CN\" scope="entry">
      <read-attr attr-name="Object Class"/>
    </query>
  </input>
</nds>

Here we have a standard DirXML XDS document. It is a Query, for a specific object class (Organitional Unit) against the destination system (which happens to be eDirectory in this case, coming from an SAP HR example) for a specific DN (The value of dest-dn) and reading back the Object Class. (The reason for querying the Object class is because it is a mandatory attribute, even Unknown objects, have an object class of Unknown. See this article for more discussion on the issue: XXX comment on Father Ramons article about Object Class).

The response can look something like this:

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product version="3.5.10.20070918 ">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <output>
    <instance class-name="Organizational Unit" qualified-src-dn="dc=com\O=acme\OU=People\OU=GA\OU=CN" src-dn="\ACME-DEV-AUTH\com\acme\People\GA\CN" src-entry-id="74056">
      <attr attr-name="Object Class">
        <value timestamp="1207772827#7" type="string">Organizational Unit</value>
        <value timestamp="1207772827#8" type="string">ndsLoginProperties</value>
        <value timestamp="1207772827#9" type="string">ndsContainerLoginProperties</value>
        <value timestamp="1207772827#10" type="string">Top</value>
      </attr>
    </instance>
    <status level="success"></status>
  </output>
</nds>

In this case, if we were using either the Destination Attribute token, or the Query token we might have generated this query, and be planning on storing its result in a local variable that we set to be of type "nodeset" (as opposed to the default type of "string"). In that case, lets say we called the local variable CHINA-OU then we can start playing around with some XPATH.

Now is where it depends how you generated the query. Lets talk about the Query token first.

The first simple example is:
$CHINA-OU/@src-dn to get the object DN in the destination data store. Now this is a silly example, since we provided it to the query token in the first place, but hey, the point is still valid and would work in other circumstances.

Then we could test to see if we got any values back, and thus if the object we are looking for actually exists. In that case, you might use a condition that tests if the following XPATH statement is true:

$CHINA-OU/attr[@attr-name="Organizational Unit"]/value

That tests for the attr node, in the CHINA-OU node set, that has an XML attribute attr-name, that is equal to Organizational Unit, and then tests to see if there is a value node under it. If there is, it returns true, if there isn't (no value returned) then it would be false.

Thus you can test for the existence of an object.

The key point to take out of this is that we did not specify a XPATH path (was that redundant or what? Like RAM memory?) that was complex like:

$CHINA-OU/nds/output/instance/attr/[@attr-name="Organizational Unit"]/value

it was sufficient to know that the context node for this type of operation in Identity Managers view of XPATH is to start at the node under the output document.

If we had used the Destination Attribute token, and set a local variable to a node set of the results, what we would have in the nodeset would actually be the set of the value nodes. I do not think we would be able to get the @src-dn out of the document, since we would really only have the value nodes in memory.

Thus we could use a For Each loop on this nodeset, (That is, when you use the for-each action, the node set is the local variable CHINA-OU) and inside the loop, you could XPATH select things $current-node/text() to get the value of each node as you loop through it, or perhaps test in an IF-THEN test, if $current-node[type="string"] to get the string values only. This example seems kind of silly, but actually has utility when looking at Group memberships in systems that do not maintain referential integrity (like Lotus Notes). In those cases, the driver and engine will try and convert the values it is getting from the destination data store for members names, into DN's in eDirectory. Which would by @type="dn" but since the destination actually could care less what value is stored in that field, (it is basically a free form string field in Lotus Notes/Domino) the values that come back may not be valid DN's in eDirectory, or may not be associated yet. You might need to pull out those values to do something with them (like storing them in a multi valued attribute that is string syntax on the group object, or maybe you just want to clean them up, so you would strip them from the operation document with Strip by XPATH expression).

The case of a Query token can be more interesting, where you can return information about multiple objects, for multiple attributes, of which some might be multi-valued attributes. In order to take advantage of all the data you retrieved and get all the bits and pieces you want out of the results, you might need to have three nested for-each loops.

For a Query document that looks like:

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product version="3.5.10.20070918 ">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
    <query class-name="User" dest-dn="com\acme\People" scope="subtree">
      <search-class class-name="User"/>
      <read-attr>
      	<value>
      </read-attr>
    </query>
  </input>
</nds>

This would return all User objects in the People.acme.com container in the tree, so it could be one or it could be thousands, use with care! If you return too many values be aware you may run out of Java heap memory. (For thoughts on how much memory a node set takes up, look at these articles: More thoughts on the size of a node set in Identity Manager but suffice it to say, expect about 10K a node to be on the safe side. You can check how much Java heap you have allocated, free, and the maximum values by looking at the examples in this article: Reading and Displaying the Value of Java Heap in Identity Manager Rules.

Now what you will get back is a nodeset with an <instance> document for each object it found.

Inside the <instance> node, there will be an XML attribute src-dn="TREE\O\OU\OU\ObjectName", and then underneath the <instance> node there will be a series of <attr> nodes with an XML attribute attr-name="Attribute Name". Underneath the <attr> node will be either a single <value> node or many <value> nodes at the same level, for multi valued attributes.

You can see how you would need to use three nested for-each loops to get at everything in the query results.

As before, we set a node set local variable called BIGGER-QUERY with the value of that query shown above, and so our first for-each loops nodeset will be the local variable BIGGER-QUERY. (We could have said XPATH of $BIGGER-QUERY which would be the same thing).

Now in this loop, we can do things like set a local variable CURRENT-USER to the XPATH $current-node/@src-dn to store the user's DN for use inside all the nested loops.

This first for each loop would run through all the instance documents, one per User object found, until they are all done.

Then we would have another for-each loop inside the first one, to loop through the nodeset of either local variable current-node or XPATH $current-node.

We can test in this loop if the $current-node (now the local value of this loops current node), is $current-node/attr[@attr-name="Some Attribute"] and possibly do something with the value.

If the attribute is multi valued, then there will be several <value> nodes inside the <attr> node, so we would need a third for each loop to work through that set as well.

The nodeset for our third for each loop would be XPATH of $current-node/value and we could test again like above for @type="string" or the like. Perhaps we want to build a text string to send in an Audit event or an email or something else like that.

Hopefully this example shows pretty simply how the context node matters, and everything is based on where it currently is, when using XPATH.

I think I will need to continue on this thread, and possibly take another swing at it with more examples. Hmm.

Cool Solutions

Some thoughts on XPATH in Novell Identity Manager

geoffc — Sun, 08 Nov 2009 09:39:10 -0700

The original iteration of Novell's product started as DirXML 1 and (when it was still called Virtual Replicas, thus the engine's name as a binary is still "vrdim" internal to the system.

In that build, all work was done in XSLT, the XML Style sheet Transformation language.

With the release of NSure Identity Manager 2.0 (gotta love the name changes) DirXML Script was introduced, which is an XML dialect that specifies conditions, actions, nouns, verbs, and tokens in a reasonably easy to read syntax that has an astonishing amount of power.

With each release of new versions of Identity Manager the functionality of DirXML Script gets better and better. Examples are things like the Unique Name (http://www.novell.com/communities/node/2209/unique-name-token-functionality-idm-35), Time, and Convert Time tokens (http://www.novell.com/communities/node/2572/using-time-tokens-idm-35). In both cases, you could do all the work in a rule you wrote yourself (for the case of Unique Name) or via a call to a Java class (Time and Convert Time). But the thing was, pretty much everybody needed to use one of these tokens in a project, so why not build a nice interface around it (Time and Convert Time's interface versus the call to a Java class is such an improvement it is amazing to see! The Java time function is very very powerful, and the Time/Convert Time tokens expose most of it in a trivial to use fashion.).

XSLT is still here and supported in the latest versions of Novell Identity Manager, and in fact some drivers still use it by default, because what it does, is still not one hundred percent replicated by DirXML Script, or perhaps more correctly, XSLT is better suited to do the task. Examples are parts of the SAP HR/UAM drivers, where there is an external relationship between objects that is almost impossible (or so the original author told me in person, and that he had asked the worlds greatest expert on DirXML Script take on the task, before admitting it was currently impossible. If Father Ramon believed it could not be done in DirXML Script then it cannot be done) to do in DirXML Script.

The other example is the Delimited Text driver, which takes an incoming file CSV style file and performs a transform into an XDS formatted document. This is exactly the type of thing XSLT is designed for, and it truly shines in this space.

Having said that, most drivers use minimal XSLT these days as DirXML Script has become powerful enough to do almost everything needed.

If you cannot do it in DirXML Script, you have two more options, before falling back to XSLT. Call out to a custom Java class that you wrote to perform the specific task, or use XPATH. The scope of things you can do within your custom Java class is limited by your imagination and your access to API's.

XPATH is the XML Path Language, and a good definition of the RFC is at this link:
http://www.w3.org/TR/1999/REC-xpath-19991116

What is nice about this link is that you can find functions that are supported by XPATH in an XPATH statement. It is worth mentioning that Novell Identity Manager only supports XPATH 1.0 and not XPATH 2.0. The issue of 2.0 support is a complex one, since apparently XPATH 2.0 is not backwards compatible with XPATH 1.0, and thus an upgrade of support to 2.0 could cause you to have to re-evaluate all the use of XPATH in your current drivers. Clearly this is NOT a customer friendly event to consider. I will be interested in seeing how Novell intends to handle this issue in the future, as it will no doubt become more pressing.

XPATH, like XSLT is relatively confusing, and reading the RFC is not always enlightening. There are a bunch of Identity Manager focused resources you can look at, the Support forums (forums.novell.com over HTTP or NNTP) to start with. There are lots of examples floating in previous message posts in that group. There are some Wiki sites with samples like:
http://wiki.novell.com/index.php/XPATH_Examples
http://ldapwiki.willeke.com/Wiki.jsp?page=IDMCodeSnippets

One issue that I find stalls many first timers as they encounter XPATH is the difference between selecting and evaluating.

XPATH is used in two very different contexts within an Identity Manager solution.

The most common use is to select a node in an XML document. The second usage is to call some function or do something with XPATH. Both approaches use the same language but are very different and confusing.

Selecting a node with an XML document (in the XDS dialect for Identity Manager) you can use the XPATH simulator in Designer pretty effectively.

Here is a sample of an add User document, and some of the things you can use XPATH to easily select.

<nds dtdversion="3.5" ndsversion="8.x">
  <source>
    <product version="3.5.10.20070918 ">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <input>
    <modify class-name="User" dest-dn="\ACME-TREE\LAB\EMPLOYEES\ACTIVE\ljohnson" dest-entry-id="61535" event-id="ACMESMSLES10FS1#20080515175150#1#1" from-merge="true" qualified-src-dn="O=acme\OU=NYC\OU=FIN\CN=ljohnson" src-dn="acme\NYC\FIN\ljohnson" src-entry-id="34296">
      <association>{50BCE1E7-2D5F-dc01-80C5-330003000000}</association>
      <modify-attr attr-name="Password Expiration Time">
        <remove-all-values/>
      </modify-attr>
      <modify-attr attr-name="Login Time">
        <remove-all-values/>
      </modify-attr>
      <modify-attr attr-name="GUID">
        <remove-all-values/>
      </modify-attr>
      <modify-attr attr-name="NOVUser">
        <add-value>
          <value timestamp="1210875960#2" type="state">false</value>
        </add-value>
      </modify-attr>
      <modify-attr attr-name="Password Expiration Time">
        <add-value>
          <value timestamp="1210341755#17" type="time">1189381168</value>
        </add-value>
      </modify-attr>
      <modify-attr attr-name="GUID">
        <add-value>
          <value timestamp="1210341755#41" type="octet">gA8kk9Ad3RGIhwBQVrsTEw==</value>
        </add-value>
      </modify-attr>
    </modify>
  </input>
</nds>

It depends how you are using the XPATH to understand the context. For example, you might want to store a local variable with the DN of the source user. (Now this is not usually needed since there is a Source DN() token in Argument Builder, and an if-source-dn test in the Conditions section, but imagine you needed to do so.). You could set a local variable to the XPATH statement "@src-dn" which will select the value of src-dn in the modify node (acme\NYC\FIN\ljohnson). Lets say you called the variable SOURCE-DN.

You might have a condition with XPATH as the test. This can be XPATH is true or is not-true. Then you might want to say 'if XPATH expression is true "@src-dn"' to test if there is a value for src-dn in the document. (As you may already know, not every document comes across with a src-dn or dest-dn set, and sometimes you need to be sure it is present before you try to use the value, with blind trust that it exists). This will return true if there is a value, and false if there is no value.

Those were a couple of examples of using selection. Now for an example of actions or functionality, lets say you might want to do a in-subtree test against a variable. In that case, you could use the XPATH function contains(). You really should for our example use the if-sourceDN condition in Policy Builder, and the in-subtree test, but for the purpose of our contrived example lets assume you want to do it the hard way. One realistic scenario would be when the object path you want to check is not the current object, but your retrieved the path somehow, say via a Distinguished Name syntax attribute, or maybe via a Query for the object. The Source DN and Destination DN condition tests only really apply to the object that is the focus of the current operational document.

First off it helps to know that variables can be used in XPATH by referencing them with a dollar sign ($) before the name.

You could then test if XPATH expression is true, contains($SOURCE-DN,"ACME\NY\FIN") and expect either a true of false. If you wanted too you could use a Global Configuration Value as well, see this article http://www.novell.com/communities/node/4825/using-global-configuration-values-xpath for more details on that issue.

The easiest way I know of in Identity Manager to do math is in XPATH. I would set a local variable equal to an XPATH expression, and you can use variables and do basic math things. Like $TIME has the value of the current time in CTIME (Seconds since 1970) but I need 90 days from now. So I could set the local variable FUTURE-DATE to the XPATH of "$NOW + 90*24*60*60". If I am not sure it is really an integer in my variable, I could use the function number() to convert it an integer.

I could have used a GCV to hold the value of 90 * 24 * 60 *60 or maybe a GCV (lets call it DaysInactive) for the 90 and multiply it by 86400 (seconds in a day, 24 * 60 * 60). Which would look more like XPATH of "$NOW + ~DaysInactive~*86400".

String manipulation can be done in XPATH as well. While DirXML Script has a very powerful Replace All, Replace First, and Substring tokens, it does not do a good job of substring-before and substring-after, which XPATH supports.

Imagine you need to get the first part of a users email address, say the part before the @acme.com in ljohnson@acme.com. While you could use a Regular Expression to replace everything up to the @ symbol with nulls. Something like .+\@ will select everything up to @ symbol. But it is much easier to use the substring-after() XPATH function, so that my variable $EMAIL which holds the value ljohnson@acme.com, I could set the local variable $NAME equal to the XPATH statement of "substring-after($EMAIL,"@") and you would end up with what you wanted.

There are a number of XPATH functions you can use, like concat(), starts-with(), substring-before(), substring-after(), substring(), string-length(), normalize-space(), and translate(). Check the XPATH reference at http://www.w3.org/TR/1999/REC-xpath-19991116 that I mentioned above for more XPATH possibilities.

Normalize-space() is one of my favorites and it is quite powerful and useful for cleaning up strings. It will replace any multiple occurrences of white space with a single space, and removing any leading and trailing spaces. Great if you do not trust the exact syntax of the string you are getting from your connected system. If you are doing string compares, the safest way to be sure they actually compare correctly when you want them too is to normalize-space both of them before you do the compare.

Identity and Security

Using XPATH to Get the Position of a Node in a Node Set

geoffc — Sun, 08 Nov 2009 09:38:53 -0700

Using XPATH to get the position of a node in a node set:

Novell Identity Manager is a great tool. Lots of power, and several interesting languages are available for use.

There is XML Style sheets (XSLT) that were the first available language in DirXML 1.0. You could always call out to a Java class, so that was not all that limiting.

With the advent of NSure Identity Manager 2.0 we got access to DirXML Script, which is a great XML based language for processing XML documents. It is very readable and makes sense (in English only, sorry) to read, because the tokens, verbs, and nouns are all nicely described by their definition.

Probably the greatest strength about the DirXML Script is actually the interface that parses it, Policy Builder. There are currently two different implementations of it, one in iManager, and the other in Designer for Identity Manager.

I personally like the Designer interface better, but that is more of a preference for the widgets and the responsiveness of a thick client (Eclipse based) versus a web application, that is always, by definition, a little slower to respond.

Both are great, and what is often most powerful is that use the interface to design the rule in general. Then when you realize you need to make minor changes, you can easily flip over to the XML view and copy and paste, or edit something minor (or major) faster than you could in the interface. The best of both worlds.

One of the features both XSLT and DirXML Script offer is the ability to use the XML Path language (XPATH). The problem is Novell Identity Manager only supports XPATH 1.0, not the newer 2.0 standard, which means many searches for XPATH help will return 2.0 functions or ideas, that will not work in XPATH 1.0.

Additionally, the way Identity Manager uses XPATH is slightly different from how standard XPATH is implemented. (Usually related to the context node (see: XPATH and the context node)). This sometimes makes it hard to translate web XPATH examples into Identity Manager examples.

Additionally you do not see a lot of articles about really cool ways to use XPATH in Identity Manager. Thus I was looking to add more articles on the issue to the public mix.

One source of information about XPATH is the RFC that defines XPATH:
http://www.w3.org/TR/1999/REC-xpath-19991116

Other than that, there are a couple of good sites with some examples:

I have been trying to write some articles about interesting XPATH tidbits, you can read more at:

We still need more I think, so here goes another one.

I was reading a neat web site called Stack Overflow, (www.stackoverflow.com) and looked for this interesting XPATH example I thought I would liberate (Information wants to be free after all, right?)
http://stackoverflow.com/questions/226405/find-position-of-a-node-using-xpath

When referring to a node set, it is possible to specify a couple of interesting XPATH descriptors to specify it by the node placement.

That is, if you are using an Append XML element, you need to specify an XPATH location for it. So you could say add-attr[@attr-name="Group Membership"]/value[2] to refer to the second node in the set of values. Or you could append it to the end with add-attr[@attr-name="Group Membership"]/value/last() or insert at the beginning, with add-attr[@attr-name="Group Membership"]/value/first() and whatnot.

But what if you want to know what place the node is in the list of values, how would you do figure it out via XPATH? I think I know how I would do it by looping through the node set in a for-each loop, using a counter value, and testing each value for the one I want and returning the counter value when done.

If this is a really large node set, then it probably is not very efficient to loop through the node set, and would be really cool to just do it in a magical XPATH statement.

Now to be fair, I am having trouble deciding when I would want to actually know this info in an Identity Manager context, since most multi valued attributes that are returned, while they look ordered, usually are not. Often they do not return the values in a predictable order, though they ought to return in the same order each time (as long as you use the same replica). (I.e. Usually not alphabetic, usually by the order it was added as a value). But I think it is an interesting exercise regardless. Ok, I'll admit it, I thought it was a really cool XPATH example and just had to write about it.

To demonstrate the issue, the article I found this in, used a sample node set of:

<a>
    <b>zyx</b>
    <b>wvu</b>
    <b>tsr</b>
    <b>qpo</b>
</a>

And wanted to know the position of the <b> node that has the value of 'tsr'.

The suggestion was an XPATH of:

count(a/b[.='tsr']/preceding-sibling::*)+1

This does not really help in an Identity Manager example, after all, what a/b in our context, so lets imagine a group membership node set, where we queried for Group membership and got back an <instance> document, that has a <attr> node, with an attribute of attr-name="Group Membership" and a bunch of <value> nodes.

So our node set looks something like: (I grabbed this from an Active Directory driver example)

<nds dtdversion="2.2">
  <source>
    <product version="3.5.11.20080307 ">DirXML</product>
    <contact>Novell, Inc.</contact>
  </source>
  <output>
    <instance class-name="Group" src-dn="CN=Domain Admins,CN=Users,DC=Test,DC=domain">
      <attr attr-name="Group Membership">
          <value type="dn">CN=Lateefah Smith,OU=Fac,OU=19TH,DC=Test,DC=domain</value>
          <value type="dn">CN=Alonzo Duncan,OU=CustSvc,OU=NY,DC=Test,DC=domain</value>
          <value type="dn">CN=Jeanie Jones,OU=CustSvc,OU=NY,DC=Test,DC=domain</value>
          <value type="dn">CN=Armando Corona,OU=CustSvc,OU=NY,DC=Test,DC=domain</value>
          <value type="dn">CN=idmUser,CN=Users,DC=Test,DC=domain</value>
          <value type="dn">CN=Administrator,CN=Users,DC=Test,DC=domain</value>
        </add-value>
      </modify-attr>
    </modify>
  </input>
</nds>

So what would XPATH be in our example?

count(attr/value[.='$TARGET']/preceding-sibling::*)+1

(I used a variable $TARGET instead of a literal string, since I think it is a bit clearer, but you could use a literal string like this instead 'CN=idmUser,CN=Users,DC=Test,DC=domain' to specify the value).

The way this works is it selects the node where the value equals the string we are looking for, then goes back one node, counts the number of nodes till there, and adds one to it. (I think they used preceding-sibling plus one, because it is a built in test that lets you go back one, thus the need to add one.

Neat eh? The best use I can think of this, is really the only time I needed to really use the [x] (you know, like [0] or [2] for the zeroth or second position) notation to pick a node. I was writing direct SQL commands in the JDBC driver, like in this article: Using the JDBC Driver and Direct SQL

In that case I needed to send several SQL statements in one operation, (in the end I went back and did it all in one SQL statement once I got better at SQL, but on my first attempt I needed to do several SQL statements) and each SQL statement you want to send needs an XML node of <jdbc:sql> under a <jdbc:statement> node with the SQL statement. In which case, you want to be sure you insert it into a specific node. In that case, the ordering did matter, but I concede it is a bit of a stretch. But come on, it is a really neat XPATH example!

Cool Solutions

Using DNS Aliases with SPNEGO

matt — Fri, 06 Nov 2009 16:46:56 -0700

One of the great features of Novell Access Manager is the integrated single sign-on capability from Microsoft Active Directory (AD) domain member workstations. Through the use of Kerberos and the Simple and Protected GSS-API Negotiation Mechanism (SPNEGO), the Access Manager Identity Server (IdS) can seamlessly authenticate a Windows desktop.

Users logon to the desktop using their normal credentials and then when they attempt to access an Access Manager protected site, they are not required to login again. Instead, a token is passed to the Access Manager IdS from the workstation. The IdS then verifies that token and allows the user access per the policies as defined in Access Manager.

Complete details on configuring Access Manager can be found in the product documentation. This basically works by the client requesting a service ticket from the domain controller for the IdS. The actual name it passes to the domain controller is known as the Service Principal Name (SPN).

The SPN is made up of three components, the protocol, the fully qualified domain name of the IdS and the client’s own AD domain name (known as the realm). So, for example, lets say that the DNS name of our IdS (the Base URL) is ids1.appdomain.com and our AD domain (the realm) is ad.appdomain.com. This would make our SPN, as sent by the workstation, the following:

HTTP/ids1.appdomain.com@AD.APPDOMAIN.COM

This is what would be sent to the domain controller (the protocol is always listed as HTTP even if it is HTTPS). The client gets back a token that has information about the user in a service ticket encrypted within the token. This is passed in the header to the IdS where it is decrypted (using the shared secret in the nidpkey.keytab file). At this point the user is authenticated and Access Manager will grant or deny access as appropriate.

This all works fine as long as the fully qualified domain name used to build the SPN matches the actual DNS host record (A record) returned when the Windows desktop does a DNS query for ids1.appdomain.com. But what happens if a DNS alias record (CNAME record) is used? Lets say now that the actual hostname of the server acting as the IdS is linux1.appdomain.com and that the DNS record for ids1.appdomain.com is actually a CNAME pointing at linux1.appdomain.com:

linux1.appdomain.com. 		IN A		10.1.1.1
ids1.appdomain.com.		IN CNAME	linux1.appdomain.com.

What happens in this scenario? When the client builds the SPN, it will look up ids1.appdomain.com which results in the CNAME being returned. It will then take the actual host record and use that to build the SPN, resulting in:

HTTP/linux1.appdomain.com@AD.APPDOMAIN.COM

This will be sent to the domain controller and will obviously fail since the SPN is incorrect, resulting in the browser being presented with a basic authentication dialog box (the IdS falls back to NTLM authentication).

The preferred solution to this problem is to put in a second host entry for the IdS, not a CNAME:

linux1.appdomain.com. 		IN A		10.1.1.1
ids1.appdomain.com.		IN A		10.1.1.1

This would result in a host record being returned to the client when it looks up ids1.appdomain.com and that is the value that would be used to build the SPN. However, there are situations where it may not be possible to enter another host record in DNS. For example, some fault tolerant layer-4 switching solutions provide for management of DNS entries as well in order to support disaster recovery scenarios (such as F5 Networks’ Global Traffic Manager). In this case, the switch may be managing and changing the DNS entries for the virtual IP addresses. Some organizations might use a dedicated or unique zone name for this and therefore have all application names referencing the switch managed entries thorough DNS aliases. In this case, a CNAME must be used.

This will work with Access Manager as long as the true, resolvable, host entry is used for the SPN. So in this example, if a CNAME is used for ids1, the value of linux1.appdomain.com would need to be used for the user ID in AD, in the Kerberos class properties (see figure 1), and in the bcsLogin.conf on the IdS server as shown below:

com.sun.security.jgss.accept {
com.sun.security.auth.module.Krb5LoginModule required
debug="true"
useTicketCache="true"
ticketCache="/opt/novell/java/jre/lib/security/spnegoTicket.cache"
doNotPrompt="true"
principal="HTTP/linux1.appdomain.com@AD.APPDOMAIN.COM"
useKeyTab="true"
keyTab="/opt/novell/java/jre/lib/security/nidpkey.keytab"
storeKey="true";
};

Figure 1: Kerberos Class Properties

Click to view.

However, the URL listed in the local trusted site list in the browser must still be the actual IdS base URL (ids1.appdomain.com in this example), not the true hostname as referenced in the A record.

Using the Kerberos feature in Access Manager is a great way to provide seamless single sign-on to Windows desktops. But it is important to understand how the client is resolving the IdS and building the SPN in order to ensure it functions reliably.

Cool Solutions

DBCOPY Cleanup Tool

soundsolutionsinc — Fri, 06 Nov 2009 16:30:28 -0700

BACKGROUND:
We use DBCOPY to stage our GroupWise backups to a different location on the server. This function works just fine in the background. The problem is that the files from the target server are not deleted if they no longer exist in the source server. So they will just stay there over time. Until now.

Using a tool called AutoDelete from Cyber-D, you can now schedule the older files to be purged from your staging area. Simply download this free utility at http://cyber-d.blogspot.com/2005/10/cyber-ds-auto-... (yup, free). Configure the paths for the folder structure that you want pruned, set the period of time that you want to search for (since we perform daily DBCOPYs, there's no need to keep anything older than 2 days).

Our DBCOPY kicks off at 8pm and our AutoDelete will kick off daily at 2am, pruning anything older than two days. DBCOPY runs from our NetWare servers, but I am setting up AutoDelete on one of our GroupWise servers running on Windows to prune all of the other NetWare servers.

WARNING:
If we request a restore to the DBCOPY area from tape, the Cyber-D AutoDelete tool will wipe that area clean of old files, so our restores need to be finished before the 8pm DBCOPY and the 2AM AutoDelete.

I initially found this tool to remove files in a folder that were older than 30 days and it has done a great job in doing so. I then used it on a folder that normally gets files deleted after 30 days, but I was finding that it needed to be cleaned out more frequently. Since I could not alter the purging period of the existing tool, AutoDelete came in really handy.

You'll find other uses for this tool in no time!

This is Freeware, please consider donating (I have no affiliation with this product or company). I believing in supporting companies who produce tools that work.

Cool Solutions