Novell Home

Collector Development Guide

INITIAL BUILD

One of the nice things about the default Collector template, the one that was cloned to create your new Collector Plug-in, is that it is already a fully-functioning plug-in that you can build and deploy into Sentinel to start your testing. The template is set up to support most of the standard Connector types, so you can even begin capturing log records immediately from your source so you know what you will be dealing with. There are some limitations, however:

  • If the data source has to be queried (like a database), the template of course won't know the query.
  • The template has no idea how to parse the inbound data, so it will just take the default input and pass it along unparsed.
  • If the data source has a very non-standard format (such as a file that doesn't delimit records using newlines) you may get partial records.

In many cases, however, you can fairly easily get your new Collector up and running at least to the point where you will be able to see live data coming from your source, and establish a known-good state as a starting point.

Before running a build, however, note that there are two types of builds:

Development build
This assembles the Collector into a running state, but does not do things like build the documentation. It is intended to produce something you can quickly use for debugging and functional testing.
Production build
This does a complete build including documentation and so forth.

Obviously you should choose the build best suited for your current purpose, which at this stage is likely the development build.

Running either type of build is really easy:

  1. Start Eclipse and make sure you are in the Sentinel perspective.
  2. Expand the collectors section in the Sentinel SDK pane at left, then expand the vendor and product directories for your Collector.
  3. Right-click on the 2011.1 node for your Collector and select Create Development Build.

The above process will run a build on the current files that make up your new Collector, which are stored under that 2011.1 node in the tree. The resulting build output will be stored, by default, in the same tree under which you unpacked the SDK, e.g. /plugin-sdk/content/build/Collector/<your name>.

Import Into ESM

After building the Collector, import it into ESM using the Scripts palette:

  • Start the Sentinel Control Center and then Event Source Management.
  • In the Scripts palette, click on the green '+' sign.
  • Browse to and select your new Collector plug-in.
  • Import into ESM.
  • Deploy your Collector to a Collector Manager using either the Connect to Event Source wizard or right-clicking on the CM and selecting Add Collector

Since most of the standard Connectors are supported by the default template, you should be able to hook your new Collector up to your data source without too much difficulty. Exceptions will include sources that need a specific query (like a SQL query) or custom settings to work correctly. Here are two examples:

Connect a Syslog Source

In its default mode, syslog sources will automatically be configured by the Syslog Connector and will appear in the interface without any manual work on the Sentinel side. They won't be automatically hooked up to your Collector, however:

  1. Start the Event Source Management (ESM) Live View.
  2. Ensure that a Syslog Event Source Server is running on the Collector Manager where you will be deploying your Collector and that it is configured to listen on the desired ports and auto-configure event sources (this is the default).
  3. Since your Collector is not yet set to auto-configure, manually deploy a Collector and Connector:
    1. Right-click on the Collector Manager or Sentinel node and select Add Collector. Follow the prompts selecting your vendor/product and your new Collector for deployment, and leaving everything else defaulted.
    2. Right-click on the new Collector node and select Add Connector. Follow the prompts to deploy a Syslog Connector with the default settings.
  4. Configure your data source to send syslog traffic to that same Collector Manager (defaults: udp/514 tcp/1468).
  5. As soon as data starts to flow, a new Event Source node should appear in the ESM UI representing the new data source. It will likely be hooked up to the Generic Event Collector.
  6. Right-click on the new Event Source node and select Move.
  7. In the dialog, select the Collector/Connector pair that you previously created above.

Once the Event Source node is moved, inbound syslog data should flow to your new Collector.

Connect a File Source

Files are a very common source of log data, or can be used temporarily to hold a snapshot of "live" source data for analysis. For this to work, the test file must be on the same Collector Manager where you will be configuring the Collector:

  1. Start the Event Source Management (ESM) Live View.
  2. Select the Connect to Event Source button.
  3. Follow the prompts selecting your vendor/product and your new Collector for deployment, and leaving everything else defaulted.
  4. Select FILE from the list of connection methods, then click Next.
  5. Click Next again to create a new set of components.
  6. Select the Collector Manager host you are deploying to.
  7. Click Next to accept the defaults on the Collector properties (parameters), Collector runtime configuration, and Configure Connector screens.
  8. Specify or browse to and select the file to be processed by the Collector: /tmp/sample.csv.
  9. Select the defaults for the next few screens and complete the wizard.
  10. Right-click to start the Event Source and data should flow through your Collector.

Prep for Development

Now that you've shown that you can get the plug-in working without actually editing anything, you need to do a little prep for the heavy lifting part of development. The first critical piece here is to capture a good sample data set; this serves a number of purposes:

  • By looking at the sample data, you can see what sort of manipulations the Connector applies to the data when presenting it to the Collector.
  • You can also get a look at the additional metadata the Connector injects into the event record, which can be very helpful for determining context.
  • By capturing a good set of sample events (logins, resource access, configuration, etc) you can begin to develop a parsing plan that accounts for the full variety of data that your source will produce.
  • By capturing a data set that covers the gamut of event types and formats, you can create an automated test suite that will automatically alert you if there are issues with the parsing, unanticipated changes from one version of the Collector to the next, and so forth.
Creating a Connector Dump

As you can see, getting a really good data set can be extremely helpful. What we recommend for this procedure is to use the Generic Event Collector hooked up to the new event source using any of the available Connectors, similar to what you did above (but you can even specify test queries for databases using this method). Here's how:

  1. Add your new Event Source to the Generic Event Collector by right-clicking on the Collector and selecting Add Connector, then in turn right-clicking on that and selecting Add Event Source OR if you already created your new Event Source node, above, right-click on it and select Move, then select the Generic Event Collector as the new parent node for the Event Source.
  2. Right-click on the Connector and, on the General tab, select the Save raw data to file option — specify an output file.
    • This will cause a Connector Dump file to be created that contains the raw output from the Connector.
  3. Restart your Event Source and then exercise as many functions of the source as possible to generate as wide a range of events as possible. Log in, open files, reconfigure the device/app, create accounts, do everything that you want to capture for user activity, and then some.
  4. Check your Connector Dump file and make sure that event records corresponding to your activity were written there (you may need to stop the Connector to flush the Dump to disk).

Introduction to Debugging

In the previous section we captured a Connector Dump that represents the exact output data that will be coming from the Connector for this data source. Now it's time to look at that data directly within the Collector, so you can see exactly how it will appear to your parsing code. The following instructions assume that you've already set up your Collector and captured a Connector Dump, as above.

  1. Right-click on your new Collector and select Add Connector.
  2. Add the File Connector and configure it to read from the Connector Dump file created as above.
  3. Right-click on the Event Source, select the Connection Mode (Advanced) tab, and select the Replay mode.
    • This tells the Collector to treat the file specially and “unwrap” the Dump records into a virtual “live” record.
  4. Right-click on your Collector and select Debug.
  5. Select Yes to run in live mode, then hit Play once to load the Collector code into the debugger. Don't hit Play twice or it will start to execute the actual script.
  6. Scroll all the way to the bottom of the file, then scroll up a few lines until you see the line:
    if (!(rec.customPreparse(curEvt) && rec.preParse(curEvt))) {
  7. Put a breakpoint on this line (click to the left of the line).
  8. Hit Play again and the Collector should advance to your breakpoint.
  9. Hit the Step Into button to drop into the customPreparse() method.

To explain what's happening here a bit:

  • The Collector script consists of a set of several Javascript files, the ones you see in the development directory or if you unpack the plug-in.
  • These files are concatenated together to make one large script at runtime.
  • Most of the files define classes and methods and so forth. All these definitions come first before they are used.
  • Hence, the actual runnable code isn't until the very bottom of the file, and consists of only a few lines of code that calls out to the other methods. In fact, the last 40 or so lines is actually where the script starts executing, and if you look at those lines you should be able to map most of the method calls to the states that we discussed in Getting Started.
  • As we discussed before, the Collector is implemented using object-oriented techniques, hence the customPreparse() method is called on the Record object rec.
  • When the customPreparse() method is called, the scope of execution drops into the rec object, meaning that the top-level scope, and the variable this, refer to the rec object. You can see the list of variables change in the bottom left-hand corner of the debugger when you drop into the method to reflect this.
  • The line rec = conn.read() just above is where the first record is fetched from the Connector input queue. Hence, the rec object contains the first record from the input.
  • Putting these two things together, when you drop into the customPreparse() method you are “inside” the rec object, and all the variables listed in the bottom left pane of the debugger are member attributes of the rec object.
  • Also note that the customPreparse() method is empty — it's not used in released Collectors.
At this point the variables in that bottom left pane represent exactly what the Connector will deliver for a sample input record. The exact set of attributes will vary depending on what type of data source this is, what dataset was received, how the query (if there is one) was defined, how the Connector is configured, and so forth. As a general rule, however:
  • Line-oriented data (files, syslog, etc) will show up in the s_RXBufferString variable.
  • Map-oriented data (structured records like DB query results, etc) will show up as an RXMap object, with multiple sub-attributes representing each input field.
  • A variety of metadata is also included; these are typically top-level fields like s_Version, s_SyslogRelayIp, and so forth (in general, prefixed by 's_' or 'i_').

Look through the data that you received from your device, and see exactly what you'll be getting from the Connector. Your Collector code will have to parse that data, so you need to know it well!

You can step through the default code in the debugger if you like at this point, but in the next section we'll talk more about how the Collector plug-in is put together, and come back to actual parsing and debugging later.

Collector Development Guide

Development Topics

© 2014 Novell