Novell Home

Collector Development Topics

KEYMAPS AND DATAMAPS

When writing a Collector, it is often necessary to leverage external references to "map" data from one type to another. The SDK APIs provides a couple important mapping classes to assist with this.

SDK Maps

KeyMap
Defines a map that allows you to "look up" additional information based on a particular key. You can use this, for example, to map location information to host IP addresses (but see the Sentinel Asset framework). The standard taxonomy map and many severity mappings are implemented with this class.
DataMap
Defines a map that allows you to transform one type of JS object into an Event, specifying which fields of the input object are mapped to fields in the output Event. The standard Event.send() method takes a DataMap as an argument that converts a Record into an Event.

Using KeyMaps

KeyMaps are pretty intuitive — you use a key to look up other data — but the implementation has a couple interesting features that we'll highlight here. First, please note that the source of a KeyMap is always a simple CSV file (we usually use the extension ".map"); you can create it using LibreOffice Calc, Microsoft Excel, or just a text editor, as long as the file contains CSV. Second, note that whitespace is matched so you need to make sure you don't have any extra whitespace on your keys or they might not be matched.

To actually load a KeyMap into the Collector execution environment, you need to do two things: add the KeyMap CSV file to the Collector, and then load the KeyMap in the Collector code. The first is easy; if you're developing the Collector, just copy the file into the Collector directory (if you are customizing an existing Collector, you can use the Add Auxiliary File method.) The second is also pretty easy, just write the following in your initialize() method:

Collector.prototype.initialize = function() {
    this.MAPS.myMap = new KeyMap(this.CONFIG.collDir + "/mymap.map");
};
                

A couple points here:

  • The global configuration variable this.CONFIG.collDir points to the executing Collector itself; we then just append the name of a file that is stored in the plug-in to load.
  • This code loads the map file mymap.map, which is a CSV file, into a global storage area instance.MAPS used for storing maps. We've called this new map myMap, but it could have been any name you prefer.

Later when you want to use this map, a fairly standard way to do so might work as follows. Suppose you have a map file that contains:

~Key,FirstName,PhoneNumber
userkey1,John,202-123-1234                
                

And let's suppose that we've extracted some value from our input data so that rec.RXMap.userkey == "userkey1". Well, then we can:

Record.prototype.normalize = function() {
    this.RXMap.mappeduser = instance.MAPS.myMap.lookup(this.RXMap.userkey);
};                
                

This will result in rec.RXMap.mappeduser[0] == "John" and also rec.RXMap.mappeduser[1] == "202-123-1234". We could then refer to these two attributes in our Rec2Evt.map file to copy them into the output Event, something like:

CustomerVar24,RXMap.mappeduser[0]
CustomerVar25,RXMap.mappeduser[1]
                

The point here is that you start with a CSV file that defines a relationship between a key in column one, and a set of values in columns 2...N. You can then use the KeyMap to look up those additional values based on the key; columns 2...N will be returned as an array of Strings (zero-based, of course) which you can then do other stuff with.

There are a couple variations on this theme, however, that make this more interesting. First of all, if all you want is a single specific column value back from your map, you can specify that column number (again, zero-based) as part of the lookup() method. If you do so, you'll get back a single String (not an array). For example, we could have called instance.MAPS.myMap.lookup(this.RXMap.userkey,0) and we would have gotten just the String "John" back.

But that's not all. You can also tell the KeyMap constructor that the map has a header row that contains names for the columns. All you have to do is give the constructor a hint (either a line number or the first few characters of the header), and instead of using implicit column numbers for the map it will use the names for the columns taken from the header (lines before the header are ignored). Then when you do your lookup, you use that column name to get back the string:

Collector.prototype.initialize = function() {
    this.MAPS.myMap = new KeyMap( this.CONFIG.collDir + "/mymap.map", "~");
};
    
Record.prototype.normalize = function() {
    this.RXMap.userFirstName = instance.MAPS.myMap.lookup(this.RXMap.userkey, "PhoneNumber");
};

But that's not all! KeyMaps can also be refreshed, so that if some other part of your code (or an external process) is modifying your KeyMap CSV file, you can reload it at any time. Obviously this is an expensive operation, so be careful you aren't trying to load multi-megabyte maps every second or so. You can also extend a map using a second CSV file (the CSV column structure must be the same), and this allows you to take multiple similar input files and load them into a single KeyMap. We do this for the Novell Identity Manager Collector, for example, because events can be defined in LSC files from different sources.

Built-in Maps

There are a few built-in maps that are pre-defined for common types of mapping activity. Depending on the product you are integrating with, you may choose to use these maps or not. Here they are:

Taxonomy map
This is the standard taxonomy map, discussed extensively in the section on taxonomy.
User privilege level map
The keys in this map are matched against the InitiatorUserName field to assign privilege level information to users performing activity in the enterprise. This is intended to capture "administrative" users, in other words users that have some special meaning to the source product like root in Unix or Administrator in Windows. This is not intended to capture privilege levels based on dynamically-assigned roles or similar. Simply fill in the privlevel.map file according to the usage notes at the top of the file.
Data sensitivity map
The keys in this map are matched against the TargetDataName field to assign sensitivity level information for the data objects that are affected by the activity. This is intended to help prioritize analysis of events that modify extremely sensitive files that have special meaning to the product being integrated with, such as files within /etc in Unix. This is not intended to catch cases where files are sensitive because an enterprise has arbitrarily placed sensitive data in them. Simply fill in the datasensitivity.map file according to the usage notes at the top of the file.

If you do not intend to use the user privilege level map or the data sensitivity map, you can simply delete the map source files.

Using DataMaps

DataMaps are used much more rarely than KeyMaps, because nearly all the time you need only one: the standard Rec2Evt.map that defines how the input record is mapped to the output Event. And that DataMap is automatically handled by the template — loaded from file into the global instance.MAPS.Rec2Evt DataMap, and then automatically applied by the Event.send() method. It is conceivable, however, that in some odd circumstance one might wish to support multiple DataMaps and switch between them — perhaps you are dealing with a source that has three distinct event structures, for example, and the easiest way to handle them is to use different DataMaps.

There are a couple ways to achieve this under the current template:

  • You could load additional maps under other names, make a copy of the default Rec2Evt map to another map, and then overwrite the default instance.MAPS.Rec2Evt map with a copy of one of the existing DataMaps just before the template sends the event (if you copy an existing DataMap into another variable, you should just be copying the reference, not duplicating the entire map).
  • You could load additional maps, and then write your own code to call Event.send() with the selected map for that particular Event (the send() method takes an optional DataMap as an argument). In this case, you should circumvent the template's call to send the Event, either by returning false after your parsing is done, or setting the instance.SEND_EVENT flag to false.

Note that loading DataMaps is a little more complicated than KeyMaps — by default, they are tweaked to make them safer and faster — so looking at the map in memory might be slightly confusing.

Collector Development Guide

Development Topics

© 2014 Novell