Collector Development Topics
KEYMAPS AND DATAMAPS
When writing a Collector, it is often necessary to leverage external references to "map" data from one type to another. The SDK APIs provides a couple important mapping classes to assist with this.
SDK Maps
- KeyMap
- Defines a map that allows you to "look up" additional information based on a particular key. You can use this, for example, to map location information to host IP addresses (but see the Sentinel Asset framework). The standard taxonomy map and many severity mappings are implemented with this class.
- DataMap
- Defines a map that allows you to transform one type of JS object into an Event, specifying which fields of the input object are mapped to fields in the output Event. The standard Event.send() method takes a DataMap as an argument that converts a Record into an Event.
Using KeyMaps
KeyMaps are pretty intuitive — you use a key to look up other data — but the implementation has a couple interesting features that we'll highlight here. First, please note that the source of a KeyMap is always a simple CSV file (we usually use the extension ".map"); you can create it using LibreOffice Calc, Microsoft Excel, or just a text editor, as long as the file contains CSV. Second, note that whitespace is matched so you need to make sure you don't have any extra whitespace on your keys or they might not be matched.
To actually load a KeyMap into the Collector execution environment, you need to do two things: add the KeyMap CSV file to the Collector, and then load the KeyMap in
the Collector code. The first is easy; if you're developing the Collector, just copy the file into the Collector directory (if you are customizing an existing
Collector, you can use the Add Auxiliary File method.) The second is also pretty easy, just write the following in your initialize()
method:
Collector.prototype.initialize = function() { this.MAPS.myMap = new KeyMap(this.CONFIG.collDir + "/mymap.map"); };
A couple points here:
- The global configuration variable
this.CONFIG.collDir
points to the executing Collector itself; we then just append the name of a file that is stored in the plug-in to load. - This code loads the map file
mymap.map
, which is a CSV file, into a global storage areainstance.MAPS
used for storing maps. We've called this new mapmyMap
, but it could have been any name you prefer.
Later when you want to use this map, a fairly standard way to do so might work as follows. Suppose you have a map file that contains:
~Key,FirstName,PhoneNumber userkey1,John,202-123-1234
And let's suppose that we've extracted some value from our input data so that rec.RXMap.userkey == "userkey1"
. Well, then we can:
Record.prototype.normalize = function() { this.RXMap.mappeduser = instance.MAPS.myMap.lookup(this.RXMap.userkey); };
This will result in rec.RXMap.mappeduser[0] == "John"
and also rec.RXMap.mappeduser[1] == "202-123-1234"
. We could then refer
to these two attributes in our Rec2Evt.map
file to copy them into the output Event, something like:
CustomerVar24,RXMap.mappeduser[0] CustomerVar25,RXMap.mappeduser[1]
The point here is that you start with a CSV file that defines a relationship between a key in column one, and a set of values in columns 2...N. You can then use the KeyMap to look up those additional values based on the key; columns 2...N will be returned as an array of Strings (zero-based, of course) which you can then do other stuff with.
There are a couple variations on this theme, however, that make this more interesting. First of all, if all you want is a single specific column value back from your map,
you can specify that column number (again, zero-based) as part of the lookup()
method. If you do so, you'll get back a single String (not an array). For example, we could have
called instance.MAPS.myMap.lookup(this.RXMap.userkey,0)
and we would have gotten just the String "John" back.
But that's not all. You can also tell the KeyMap constructor that the map has a header row that contains names for the columns. All you have to do is give the constructor a hint (either a line number or the first few characters of the header), and instead of using implicit column numbers for the map it will use the names for the columns taken from the header (lines before the header are ignored). Then when you do your lookup, you use that column name to get back the string:
Collector.prototype.initialize = function() { this.MAPS.myMap = new KeyMap( this.CONFIG.collDir + "/mymap.map", "~"); }; Record.prototype.normalize = function() { this.RXMap.userFirstName = instance.MAPS.myMap.lookup(this.RXMap.userkey, "PhoneNumber"); };
But that's not all! KeyMaps can also be refreshed, so that if some other part of your code (or an external process) is modifying your KeyMap CSV file, you can reload it at any time. Obviously this is an expensive operation, so be careful you aren't trying to load multi-megabyte maps every second or so. You can also extend a map using a second CSV file (the CSV column structure must be the same), and this allows you to take multiple similar input files and load them into a single KeyMap. We do this for the Novell Identity Manager Collector, for example, because events can be defined in LSC files from different sources.
Built-in Maps
There are a few built-in maps that are pre-defined for common types of mapping activity. Depending on the product you are integrating with, you may choose to use these maps or not. Here they are:
- Taxonomy map
- This is the standard taxonomy map, discussed extensively in the section on taxonomy.
- User privilege level map
- The keys in this map are matched against the InitiatorUserName field to assign privilege level information
to users performing activity in the enterprise. This is intended to capture "administrative" users, in other words users that have some special meaning
to the source product like
root
in Unix orAdministrator
in Windows. This is not intended to capture privilege levels based on dynamically-assigned roles or similar. Simply fill in theprivlevel.map
file according to the usage notes at the top of the file. - Data sensitivity map
- The keys in this map are matched against the TargetDataName field to assign sensitivity level information
for the data objects that are affected by the activity. This is intended to help prioritize analysis of events that modify extremely sensitive files that have special
meaning to the product being integrated with, such as files within
/etc
in Unix. This is not intended to catch cases where files are sensitive because an enterprise has arbitrarily placed sensitive data in them. Simply fill in thedatasensitivity.map
file according to the usage notes at the top of the file.
If you do not intend to use the user privilege level map or the data sensitivity map, you can simply delete the map source files.
Using DataMaps
DataMaps are used much more rarely than KeyMaps, because nearly all the time you need only one: the standard Rec2Evt.map
that defines
how the input record is mapped to the output Event. And that DataMap is automatically handled by the template — loaded from file into the
global instance.MAPS.Rec2Evt
DataMap, and then automatically applied by the Event.send()
method. It is conceivable, however,
that in some odd circumstance one might wish to support multiple DataMaps and switch between them — perhaps you are dealing with a source that has
three distinct event structures, for example, and the easiest way to handle them is to use different DataMaps.
There are a couple ways to achieve this under the current template:
- You could load additional maps under other names, make a copy of the default
Rec2Evt
map to another map, and then overwrite the defaultinstance.MAPS.Rec2Evt
map with a copy of one of the existing DataMaps just before the template sends the event (if you copy an existing DataMap into another variable, you should just be copying the reference, not duplicating the entire map). - You could load additional maps, and then write your own code to call
Event.send()
with the selected map for that particular Event (thesend()
method takes an optional DataMap as an argument). In this case, you should circumvent the template's call to send the Event, either by returningfalse
after your parsing is done, or setting theinstance.SEND_EVENT
flag to false.
Note that loading DataMaps is a little more complicated than KeyMaps — by default, they are tweaked to make them safer and faster — so looking at the map in memory might be slightly confusing.
- Back to Develop to Sentinel
Collector Development Guide
- Overview
- Getting Started
- Initial Build
- Plug-in Contents
- Data Parsing
- Build Process
- Event Construction
- Taxonomy
- Connector Interaction
- Common Code
- Parameters
- Additional Information