Reports

Report Plug-in Development

This documentation is broken into two parts: a Development Guide which walks through the entire process of developing a report, and a Reference Topics section which contains reference material for quick access and more detail. This page contains an overview and basic steps to get you started; further Guide topics walk you through the full process of report development. If this is your first attempt at report development, you should also visit the report previewing configuration page to get things set up correctly so that you can run quick previews of your report designs directly within iReport.

Report Development Guide

Overview

Sentinel™ includes a reporting engine that allows users to query data from the Sentinel platform — including events, configuration, and metadata — and present that information in highly-customizable textual or graphical forms. Before delving into the specific of developing report plug-ins, note that this feature has evolved over time:

Sentinel 6.1 shipped with an OEM version of Crystal Reports (now Business Objects / SAP) that could be integrated to report against the Oracle or SQL Server database that contained Sentinel's configuration and event data. Since Crystal reports were not packaged as true plug-ins, they are not covered here (refer to the Sentinel 6.1 documentation.
Sentinel 6.1 RD introduced an embedded Jasper reporting engine that was built directly into the Sentinel environment, and reports that could be executed and viewed directly in the Sentinel UI. The backend database was also extended to PostgreSQL.
Sentinel Log Manager (SLM) introduced a file-based event store, which can also be queried from reports - there are now two different data sources for reporting information. SLM also introduced the "Save as report" feature that allows customers to take existing reports and treat them as templates to create new reports.
Sentinel 7 brought the file-based event store into the main product line, but also introduced ways to replicate information from the event store to the database to support more complex reporting needs.

The Plug-in SDK supports building reports for all platforms mentioned above except legacy Sentinel 6.1. Due to the differing data storage architectures, a single report may or may not work on more than one platform depending on exactly what type of data it will be querying. The SDK will help guide you in choosing the appropriate starting template for a given type of report, and each report includes metadata that will prevent it from being imported into a platform on which it is not supported.

Sentinel Datastores

Sentinel stores important data in different places, depending on factors like efficiency, speed of access, and what platform you are using. Before developing a report, you'll need to know where the different datasets live and how to access them. First, here's a short list of the different types of data that Sentinel stores:

Event data: This is the raw data collected from the event source and parsed by the Collector. There may be additional data injected by the Collector or by the Mapping Service, and conversely in some cases data is normalized out of the event structure for storage. In general, the event data is described by the Event Schema, although some fields as mentioned are normalized into separate tables, and some fields are not persisted at all.
Event Metadata: This category represents data that is not delivered as part of the event, but is loaded from some other source like an asset management system. Usually such data is associated with an event by the Mapping Service, but note that even after this has happened some metadata will be stored with the event, and some is normalized into separate tables.
Configuration data: Configuration data includes all the internal Sentinel configuration such as users; Event Source Management node configuration; management information about Collector Managers, Correlation Engines, and Correlation rules; correlation output (associations between correlated events and normal events); incident data, and so forth. Note that anomaly rules (Security Intelligence) is not included in this category.
Security Intelligence data: The new Security Intelligence feature uses a custom storage facility based on MongoDB to store its configuration and runtime information. Note that this feature is not currently query-able via the normal Sentinel reporting facilities.

The following table describes how the datasets described above are stored for each platform.

	Event data	Event metadata	Configuration data	Security Intelligence Data
Sentinel 6.1	Oracle/SQL Server: EVENTS_RPT_V3 table¹	Oracle/SQL Server: various²	Oracle/SQL Server	N/A
Sentinel 6.1 RD	Postgres: EVENTS_RPT_V3	Postgres: various²	Postgres	N/A
Sentinel Log Manager	Files via Lucene	N/A	Postgres	N/A
Sentinel 7	Files via Lucene	Postgres: various³	Postgres	MongoDB

See the Event Schema for details
Metadata is normalized into separate tables, see the Database Views for details; typically a UUID of some kind is left in the event table for linking to the normalized tables
Note that fewer fields are normalized into DB tables in Sentinel 7 so that more queries can be driven by Lucene only; see the Event Schema for details

Joining Datasets

One of the trickier parts of reporting is figuring out how to join together separate datasets to provide a comprehensive view of the data of interest. Typical scenarios for this type of work include:

Displaying event data along with information about the Sentinel components (ESM nodes) that processed it.
Displaying event data with asset, identity, and/or vulnerability information mapped to the event.
Showing information about correlated events along with information about the raw events that triggered that correlated event.
Showing incident information along with event data that is associated with that incident.

In Sentinel 6.1 and Sentinel 6.1 RD, this is relatively easy - normal event data, event metadata, and Sentinel configuration data are all stored in a single database, so joining different datasets together is as simple as designing the correct SQL query with JOIN statements. In platforms that use file-based storage, however, this task can be more complex.

Sentinel Log Manager is a log management solution only, and hence it does not support advance features like the Mapping Service, Correlation, and Incidents - features that might require joined datasets. This leaves Sentinel 7 as the platform where you are most likely to need to join data from multiple datasets — and possibly across Lucene and SQL — to get the combined dataset you want.

For many types of reports — reports that query only event data, reports that only query Sentinel configuration parameters, etc — figuring out how to join datasets is not important. In fact, Sentinel 7 has de-normalized some fields that used to be normalized into separate tables back into the base event; this helps make it less common to need to join Lucene data against the database. There are four basic reporting scenarios for Sentinel 7:

Reports that only query event data.
Reports that only query configuration or event metadata (without reference to events)
Reports that present event data enriched with event metadata
Reports that show correlated event or incident details along with associated normal events

To determine which category your report will fall in, you can examine the Event Schema and the Database Views; if the data you wish to report on is all in one or the other category, then your report falls into one of the first two categories, above. If, however, the data you want to display appears both in the database and in the event schema, then you will fall into one of the latter two categories.

Sentinel 7 provides two methods for joining event data to the configuration database:

Report Data Definitions (RDD): An RDD sets up a persistent query that extracts data from the Sentinel database and stores it in a new database table. Reports can then be written that join that new table, containing extracted event data, against the configuration database.
Postgres triggers: The correlation and incident tables in the configuration database contain associations to event data, with the linking key being the event UUID. For this one case where you have the event UUID, Sentinel has installed background Postgres triggers that can take the UUID, query Lucene, and fetch back the relevant event data.

Selecting Your Report Template

Armed with the background knowledge above, you should now be able to look through the Event Schema and the Database Views to find those fields that are of interest for your reporting needs. Once you know which fields you want, there are several ways to create new reports within Sentinel:

For SLM and Sentinel 7, if there is an existing report definition that meets your needs, save for some additional filtering, you can use the "Save as report" feature to inject a custom filter into most existing report definitions and save that as a new report. For this to work, the existing query and report layout must be what you are looking for.
Also for SLM and Sentinel 7, there are a few special reports built into Sentinel that perform generalized event queries, and then allow you to modify how the returned data is displayed on the report by using parameters. These reports include the Event Details and Event Summary reports, and typically they allow you to specify one or more fields over which the report should calculate summaries, and then optionally a set of event fields to display.
If none of the existing report definitions can be used to meet your needs, you can create a new report using this SDK and customize it any way you like. You can still start from an existing report by duplicating that report's source into your report, then modifying as appropriate.

If you end up creating a custom report using the SDK, use the following procedure:

In Eclipse, make sure you are in the Sentinel perspective.
Find the reports entry in the Sentinel SDK pane at left, right-click on it and select Create New Report Plug-in.
Give your report a descriptive name in the Report Name field, and specify a Category (the category groups related reports together).
Specify appropriate keywords for your report to make it easier to find in the Sentinel UI.
Based on your reading above, select the appropriate platform, data to be queried, and type of report from the drop-downs. This will select an appropriate starting template.
If you choose a Sentinel 7 Both report, you may be asked to pick a Report Data Definition to use with this report. If you need one but it doesn't yet exist, you should probably go create one and then come back here.

After you click Finish, Eclipse will create a new report based on the most similar template per your selections. If necessary, right click on reports again and select Refresh, then expand reports to see your new plug-in (grouped by Category).

If you have found some other pre-existing report that is closer to what you want than the default template, you can at this point copy that report into your new report development area. To do so, find the source report in the tree in Eclipse by drilling down through Category, Report Name, platform, and then SDK version, then select all the *.jrxml, Template*.properties, and *Chart.properties files, right-click and select Copy, then browse down to the same level in your new report in the dialog and select OK. Do NOT copy the plugin.properties file it it's visible.