Collector Development Topics

SESSIONS

In some circumstances, event sources do not provide all information required to create a Sentinel Event in a single input record. This may happen if the event source produces beginning/ending records for the start and end of a transaction; if the event source generates an event with details that are left out of subsequent events; or if the records generated don't correspond well to what Sentinel considers a complete "Event".

The Collector SDK provides a Session class to assist with handling these types of scenarios. In essence, the Session class allows you to store one or more Records in the Session for later processing.

Complex Event Scenarios

Consider the following read-world scenario: let's say we have a directory server that stores information about various types of domain objects; hosts, user accounts, groups, and so forth. This directory is used by systems within the enterprise to grant access to various resources, and when a new employee comes on board one of the first things that happens is that an account is created for that user within the directory.

When an administrator creates an account for someone, however, the directory actually implements this as a two-stage process: first, it creates a generic object in the directory; then, it converts that object into an account type which sets the schema and object semantics. This process is reflected in events coming from the directory server — when an account is created, Sentinel sees an object create event closely followed by an object modify, assign class "account" event.

So how should the Collector report this information? Clearly the important bit is that an account was created, and indeed there is an event taxonomy specifically for this event — and that's exactly what people will be looking for on reports about account management. But we can't really label the first event account create because the same event will appear for other object types. We could report the object modify, assign class "account" event as an account create, but that misrepresents what the directory is actually doing, and could cause confusion if someone tries to look at activity related to that specific directory object — it will appear as if there are two create events for that object. What we really want is to report a single account create when both events are seen in the event stream, and not report each individual event.

The problem we face immediately in the Collector is that by default, once one record has been received and processed the next record completely replaces it, leaving no trace of the earlier record. And in this case, there might be important information in the first record that needs to be combined with the second record to allow us to construct our combined account create Event. We could create ad hoc global objects to hold this information, but this will get complicated quickly — we might have to deal with expiring old events that don't eventually become accounts, worry about overwriting overlapping account creation sessions, etc. Luckily, the SDK API comes with a pre-built class called a Session that will take care of these details for you.

Setting Up a Session

So how do we set up a Session? There are a few options here, but basically you need to do the following:

Select a Session key that can be used uniquely to identify the Session. Consider that you may have multiple event sources feeding you data in parallel, and also those event sources may have overlapping Sessions. In this case, a compound key consisting of the Event Source UUID (rec.s_RV24) and the object name might be appropriate.
Create a Session using the selected key.
Decide how the Session will "know" when it's ready to be processed. In this case it might simply be that we get that second event record; in more complex cases we might expect a specific number of events, or just wait a certain length of time before we process the Session. The Session class has a variety of methods — addEvtTimer(), addWallTimer(), and setting max records in the constructor — to help the Session know when it's ready to be processed.
Add the first input record to the Session (using Session.store()).
The way the Session is set up, we try to avoid making the developer step out of the normal Collector processing loop to go and deal with a Session. Instead, we give the Session the knowledge it needs to parse itself (this is a little like the SQLQuery class). The Session will automatically detect when it's ready to be parsed (if you've set max records or a timer) and then will parse itself. The developer needs to define an anonymous method that knows how to parse the Session and attach it to the Session using Session.addParser().

OK, so for our example scenario let's say we get one of these object create events, and based on that we create a new Session with a good key, set the max records that can be stored in the Session to two, and add this first record. Maybe we've also defined a parser for the Session, so we add that to the Session.

At some point later, we'll get a second input record that describes what class was assigned to the directory object. If we add that second record to the Session, then our Session will automatically expire and trigger parsing. The parser then just needs to combine the information from the two records in the Session to create a combined create account event (or some other type of create, if the class did not turn out to be an account).

What if we never get that second record? Well, one thing we can do is set a fallback timer for the Session, and make our parser intelligent enough so that if it doesn't have a second event it just issues a generic create object event.

Session Parsers

Session parsers are a little tricky because they happen automatically, and when writing them you need to consider the context of where they are actually executing. The context is actually the Session itself, so keep in mind that this refers to the Session that is being parsed. Also, since the Session parser is not running as part of the main Collector loop, you have to handle some of the things that are ordinarily handled for you: creating an Event to send, and actually sending it. In fact, some Sessions will generate multiple Events as output; you have that flexibility.

The other key piece of information you'll need is to understand what you get back from the Session when you go to parse the records in that Session. You use the Session.retrieve() method to fetch an array of all the Record objects stored in the Session, but please note that the 0th array element is a blank Record. The intent is that you can use that blank Record as a template and copy various fields from the other Record objects, but you can also build upon any of the other Records if you prefer.

Your Session parser should (usually):

Create a new Event based on the prototype Event.
Retrieve the Records stored in the Session.
Parse the data from the Records and place the partial results into the 0th Record.
Send the Session, passing the Event object as a parameter (the Session.send() method will automatically convert the 0th Record into the passed Event, and send it).

Session Example

Here's an example based on the scenario above:

Collector.prototype.initialize = function() {
    // Define a Session parser and store it in a global area for easy retrieval
    this.PARSER.parseCreateSessions = function() {
        var newEvt = new Event(instance.protoEvt);  // Construct a new Event from the prototype
        var sessRecs = this.retrieve();  // Get the stored Records from the Session
        sessRecs[0] = sessRecs[1];  // Overwrite the blank 0th Record with the first "create object" Record
        if (sessRecs[2].RXMap.NewClass == "Account") {
            sessRecs[0].RXMap.NewClass = "Account";
            newEvt.setTaxKey("Create_" + sessRecs[2].RXMap.NewClass);    
        }
        sessRecs[0].parseRecData(newEvt);  // Call a method that knows how to parse normal Records from this source
        this.send(newEvt);
    }
};

Record.prototype.parse = function(e) {
    // Detect object creation events
    if (this.RXMap.EventName == "create") {
        sess = new Session(this.s_RV24 + ":" + this.RXMap.ObjectName, 2);
        sess.store(this);
        sess.addWallTimer(30); // If we don't get an object class, this is a backup
        sess.addParser(instance.PARSER.parseCreateSessions);
        return false;   // Short circuit the normal loop
    } else if (this.RXMap.EventName == "modify" && this.RXMap.TargetField == "Class") {
        var sess = Session.get(this.s_RV24 + ":" + this.RXMap.ObjectName); // Check for existing Session
        if (!sess) {  // This isn't a create; handle like a normal object modify
            return this.parseRecData(e);
        } else {
            sess.store(this);
            return false;    // Short circuit the normal loop; Session parsing should trigger immediately since we now have two records in the Session 
        }
    } else {
        return this.parseRecData(e);
    }
};

One last point: how can we debug this? Since the Session expires and is parsed automatically, you won't see this happen in the normal part of the loop — during the preParse(), parse(), normalize() phase, for example. Instead, the Sessions are processed during the Collector reset() phase, which is called just at the beginning of each loop. To debug this, step through the main loop until you get back to the beginning, then step into the reset() method. There you should see a block that tests for expired Sessions, and if there are any will run the parsers against them. Step down into the parser and you'll see your Session parsing code executing.

Back to Develop to Sentinel