Novell Home

Collector Development Guide


In previous sections we've covered parsing input data received from an event source and putting that data into an output Sentinel event. This parsing logic is one of the major benefits provided by the Collector; namely, that it normalizes the input data into a regular field structure.

The second major benefit that Collectors provide is that of classification. The basic idea here is that we attempt to map various properties of the inbound event into standard taxonomies or classification schemes. This feature is just as important as the parsing, since it allows event consumers to search for and analyze events across any source using standard terms. For example, a consumer might want to see all authentication events from their firewalls: without classification, this could be extremely difficult — different vendors might have different terms like "login," "session created," and so forth to refer to the same basic type of activity, and identifying all the enterprise's firewalls might itself be a challenge. Hence with taxonomy a query like:

obsip: OR obsip: ...(etc) AND evt: "Login" AND evt: "Authentication Success" AND ...(etc)

can be collapsed into:

rv32: "FW" AND e.xdastaxname: "XDAS_AE_AUTHENTICATION"

(the "rv32" and "xdastaxname" might seem a little magical, but you can use the UI query builder to generate this search query quite easily).

Taxonomies in Sentinel

There are actually several taxonomies in Sentinel, each designed to support common requests for security, compliance, and forensic purposes. Each will be described in detail below:

Action Taxonomy
The action taxonomy classifies the type of activity described by the event record, in a sense it classifies the "verb". This taxonomy is typically used to answer questions like "show me all service start events."
Outcome Taxonomy
The outcome taxonomy classifies the outcome or result of the attempted activity - did it succeed, did it fail? This taxonomy is typically used to clarify part of an action-type question, like "show me all failed attempts to create user accounts."
Observer Taxonomy
The observer taxonomy classifies the type of system from which the event record was received. This taxonomy is typically used to answer questions like "show me all events from my databases."

It should also be noted that the Sentinel Event Schema itself is a form of taxonomy — because fields have semantic meaning, having data in a particular field means that the event relates to that class of object. For example, if TargetAccountName is non-null, then you know that the event relates to activity against an account. Sentinel has prioritized several object classes to top-level domain objects based on security and compliance significance, and hence those fields have their own designated schema fields. These domain objects include Accounts, Hosts, Services, Trusts (groups, roles, profiles), and DataObjects (objects that just contain data). Accounts, Hosts, and Services can act as security principals, meaning that they can "take action" and show up as an initiator of an activity (but a file, for example, cannot in and of itself cause an activity to take place). In rare cases a Trust can serve as an initiator, for example when an activity is allowed to take place due to an anonymous trust relationship. All the security principal objects plus the other object types (basically, data objects) can all be targets of activity.

Action Taxonomy

The action taxonomy is used to classify the type of activity that the event record is reporting on. The intent is to group the universe of possible activities into a relatively concise set of descriptive classes, to make searching for and analyzing common activities across different types of event sources easier.

Sentinel's action taxonomy has undergone some evolution over time. Users of previous versions of Sentinel may be familiar with Sentinel's legacy taxonomy, which used a set of four fields to present a hierarchical set of event classes. With Sentinel 6.1 we mapped that legacy taxonomy to the only existing developing standard at the time, The Open Group's XDAS. The XDAS implementation uses five fields — four numeric fields that function somewhat like the legacy taxonomy, plus a fifth field that contains a text string that maps to the numeric tuple. In Sentinel 7 we've taken additional steps to make the XDAS taxonomy easier to use and understand.

The XDAS taxonomy works like this:

XDAS Registry
The registry field contains a numeric code that represents the registry that defines the namespace for all event classes with this registry code. The only defined registry at this time is The Open Group, with code '0'.
XDAS Provider
The provider field contains a numeric code that identifies a provider of event class information that has registered with the registry. The Open Group is itself a provider (code '0') that provides a base set of event classes; this field allows third-party vendors to define their own sub-namespace of event classes if they choose to do so.
XDAS Class
The class field contains a numeric code that represents the broad class of activity that is being described; "Account Management", for example, or "Service and Application Utilization Events".
XDAS Identifier
The identifier contains a numeric code that represents the specific class of event activity being described, such as "Create Account" or "Invoke Service or Application".

Each unique tuple (all four fields described) represents an individual event type within the taxonomy. There is a fifth field, XDAS Taxonomy Name, which maps a human-readable string to the unique tuple. In addition, each unique tuple is mapped to a legacy taxonomy tuple. Here's an example:

Legacy Taxon L1Legacy Taxon L2 Legacy Taxon L3Legacy Taxon L4
XDAS_AE_CREATE_ACCOUNT - The creation of an account representing a security principal within a domain.0000 SYSTEMUSERCREATE 

When developing a Collector, it is important to review the full taxonomy and think about how each received event record will map to that taxonomy.

The Sentinel Collector template implements the selection of the appropriate taxonomy using a simple map file. The way this works is that the Collector developer identifies some piece of each input record that can serve as a "key" — a unique identifier for that specific type of event. This could be a vendor-specific event code, some snippet of the event name or description, or some combination of other fields. This key is then passed to the Event.setTaxKey() method, which will then pull in the relevant data from the taxonomy map. The developer of course also has to fill in the map with the set of possible keys and the mapped taxonomy names.

The map itself,, is a simple CSV file. The first column holds the selected event key, the second column holds the XDASTaxonomyName that is mapped to that key, and the third column holds the XDASOutcomeName (see below) that is mapped to that key. Since the file contains both the action and outcome taxonomies, we'll wait on examples until after we cover the outcome taxonomy.

When writing a Collector, you should make every effort to understand what an input record represents in terms of activity, and to map that to a pre-existing defined action taxonomy. In some cases this is not possible or easy, however, because the vendor of the source generating the event records doesn't have the same conception of what an "event" is or is performing an activity not yet defined. In that case you have several options:

  • Leave the taxonomy blank. Although not the preferred solution, for truly vendor-specific events that won't correlate to activity in any other system this may be appropriate.
  • Combine multiple input records into a single output event. In some cases the event source generates several individual records to represent what we consider to be one unitary action. In these cases it can be beneficial to use the Collector code to combine two or more input records into a single output event (for example, by using the Session class). This increases the complexity of the Collector, but may enhance the clarity of the output.
  • Define your own action classes. To do this officially, you would need to register with The Open Group as a provider and then define official class and identifier numbers for each event you are dealing with. Attachmate/NetIQ/Novell has done this for some of its products. If the release of your Collector is likely to be limited, however, you may be able to get away with just defining your own codes and documenting them carefully.

Outcome Taxonomy

The outcome taxonomy is similar to the action taxonomy, except that it classifies the outcome or result of the attempted action. As with the action taxonomy, there is a legacy version and a more recent mapping to XDAS. The outcome taxonomy uses the following fields:

XDAS Outcome
The outcome field contains a numeric value that specifies the broad class of outcome — success, failure, denial, or unknown. This is often as much detail as one needs.
XDAS Detail
The detail field contains a numeric value that specifies more specifically what the outcome was, to some extent this corresponds to providing a reason for the outcome. For example this might be "Invalid User Credentials" or "Service Failure".

As with the action taxonomy, each unique tuple of outcome/detail values maps to a readable string, such as:


As for the legacy taxonomy, the way the outcome is reported is simply to append the word "FAILURE" or "DENIAL" to the third level of the taxonomy.

The outcome taxonomy for any particular event is specified in the same taxonomy map file as the action taxonomy, by simply specifying the outcome name in the third column.

Combined Examples

Most of the time, the patterns by which an input event can be mapped to Sentinel's defined taxonomies boil down to two cases:

  • The source produces a unique event for each action/outcome. For example, a source might produce separate and distinct "authentication accepted" and "authentication denied" events.
  • The source produces one event for each action type, and then provides a result code or descriptor. For example, the source might produce a single "authentication" event, with a field that holds a "result".

In the latter case, we usually end up combining the two pieces of information — the event type and result — into a single combined key.

For our first example, assume we have a source that produces a structured record which is then parsed into rec.RXMap, and ends up looking something like this:

rec.RXMap: { tstamp: "1234567", eventname: "authentication success", targetacct: "..." }
rec.RXMap: { tstamp: "1234568", eventname: "service start failed", service: "..." }

For this scenario, we'd construct a simple like this:


Our code would be something quite simple like this:


For our second example, let's say the source uses a couple different fields to record the type and result:

rec.RXMap: { tstamp: "1234567", eventname: "authentication", result: "success", targetacct: "..." }
rec.RXMap: { tstamp: "1234568", eventname: "service start", result: "out of memory", service: "..." }

For this scenario, we'll combine the eventname and result fields into a compound key, like this (the _ character is used just to make the file more readable):

e.setTaxKey(this.RXMap.eventname + "_" + this.RXMap.result);

and then our taxonomy map file looks like this:


In practice of course, often vendors will use codes or other opaque identifiers for their events, and you'll have to do more extensive parsing.

NOTE: In the 6.1 template, the file also contained the relevant legacy taxonomy entries. With the 2011.1 template these are now automatically calculated.

Observer Taxonomy

The observer taxonomy is a bit different a somewhat simpler than the action and outcome taxonomies. As with the other taxonomies, its purpose is to classify the output events into nice logical groups to help make it easier to search and analyze the data. In this case, however, we're not trying to classify each event, we're trying to classify what type of device/application the event came from.

Sentinel has a pre-defined set of observer types that you should use, noting that there's a general catchall category ('O') if the device you are writing a Collector for does not exist in the list. If this happens, feel free to post on our Forums to discuss the issue, and we'll consider extending our list. The general rule of thumb is that each category should be a generally-recognized type of product, for example something that the analyst community considers a distinct category with multiple competitors.

Specifying the observer category for a Collector is very simple: you'll be prompted to select a category when the Collector is created. If you missed that step or you change your mind, simply edit the file and change the line that starts pcat=. Make sure the code you place there matches one of the defined categories.

It should also be noted that some event sources perform multiple functions, for example a firewall might also do network routing. In general that does not matter — the goal here is to allow consumers to request data from "their firewalls" — but if for some reason you feel strongly about this it is possible to modify the observer category on the fly, on a per-event basis. Basically the category is set in the prototype event (see Event Construction), and hence you can override that value downstream in your parsing if you wish.

Collector Development Guide

Development Topics

© 2014 Novell