Sentinel Log Manager receives two separate but similar data streams from the Collector Managers: the event data and the raw data. The data is moved from the online, compressed, file-based storage to a user-configured, compressed networked storage location on a regular basis.
The raw data files are unprocessed events that are received by the Connector and sent directly to the Sentinel Log Manager message bus.This data is written to the Sentinel Log Manager server. When the event is sent to the message bus, the following additional information is also sent without altering the original event:
SHA-256 hash of the event
Chaining indicator (which is reset to 0 whenever the Sentinel Log Manager event source is restarted)
Raw Data ID (in s_RV25)
Event source, Connector, Collector, and Collector Manager node IDs
Event ID (stored in s_RV25)
Event source, Connector, Collector, and Collector Manager node IDs
All raw data is sent to the Sentinel Log Manager without filtering. Because the raw data is not searched or used to generate reports, the data is not indexed.
In Sentinel Log Manager, raw data is always stored. Raw data is stored in partitions that are based on the time and the event source. Raw data partitions are individual files. They are created every hour, and are closed within 10 minutes after the elapsed time.Older, inactive partitions are compressed.
The raw data files can be stored in one of the following locations:
Local storage location: <SLM data directory>/rawdata/online
Networked storage location: <SLM archive directory>/rawdata_archive
When a raw data file is closed, it is renamed to identify the closed files. Files in the open state have a .open extension. When they are closed, they are renamed with a .log extension. At the configured interval, after they are closed, they are compressed and given a .zip extension. The compressed raw data files are moved from the local storage to the networked storage location.
The following table describes the directory structure of the online raw data under the installation directory:
Table 3-1 Raw Data Directory Structure
If the raw data files are stored in the online raw data location, the full path name of the file is in the following format:
<SLM data directory>/rawdata/online/<event source UUID>/<Date>/<RawDataFile>
For example:
/var/opt/novell/sentinel_log_mgr/data/rawdata/online/A75CF6A0-4948-102D-A615-000C29A9C3DB/2010-05/24-0600.zip
In this example, /var/opt/novell/sentinel_log_mgr/data is the data directory for Sentinel Log Manager.
If the raw data files are stored in the networked storage location, the full path name would be as follows:
<SLM archive directory>/rawdata_archive/<event source UUID>/<Date>/<RawDataFile>
For example:
/slm_archive_data/rawdata_archive/A75CF6A0-4948-102D-A615-000C29A9C3DB/2010-05/24-0600.zip
In this example, /slm_archive_data is the archive directory configured by the user.
Each raw data event is represented as a single line in a raw data file. Each line is a JSON object with the following format:
{ "EventDate": "<date>", "EventRecordID:" "<event record uuid>", "RawData": "<raw data>", "RawDataHash": "<SHA256 hash of raw data, in hex format>", "EventSourceManagerID", "<uuid of event source manager>", "CollectorID", "<uuid of collector>", "EventSourceID:", "<uuid of event source>", "ChainID", "<chain ID>", "ChainSequence", "<Sequence number>" }
The following table describes each of the fields in the raw data event:
Table 3-2 Raw Data Representation
The following examples show three raw data records:
{ "EventDate":"05\/24\/2010 06:15:06.676", "EventRecordID":"A75CF6A0-4948-102D-A61C-000C29A9C3DB", "RawData":"Sep 22 10:22:00 testhost Message #100", "RawDataHash":"7003c0e0be4ddf43a3b49026a37483f59c7f839950f581ec9fde5dea43da90f5", "EventSourceManagerID":"C76D2820-C395-1029-BB86-001321B5C0B3", "CollectorID":"A75CF6A0-4948-102D-A613-000C29A9C3DB", "EventSourceGroupID":"A75CF6A0-4948-102D-A614-000C29A9C3DB", "EventSourceID":"A75CF6A0-4948-102D-A615-000C29A9C3DB", "ChainID":"1274696106664", "ChainSequence":"0" } { "EventDate":"05\/24\/2010 06:15:07.358", "EventRecordID":"A75CF6A0-4948-102D-A624-000C29A9C3DB", "RawData":"Sep 22 10:22:00 testhost Message #99", "RawDataHash":"f5681ba965144d2d22b13188767d94540b5fe57904afcee5821854bde2afca72", "EventSourceManagerID":"C76D2820-C395-1029-BB86-001321B5C0B3", "CollectorID":"A75CF6A0-4948-102D-A613-000C29A9C3DB", "EventSourceGroupID":"A75CF6A0-4948-102D-A614-000C29A9C3DB", "EventSourceID":"A75CF6A0-4948-102D-A615-000C29A9C3DB", "ChainID":"1274696106664", "ChainSequence":"1" } { "EventDate":"05\/24\/2010 06:15:07.988", "EventRecordID":"A75CF6A0-4948-102D-A62A-000C29A9C3DB", "RawData":"Sep 22 10:22:00 testhost Message #98", "RawDataHash":"98435b5dba95633699b88d07782109876e8ceb4169d567602f2c92657118645d", "EventSourceManagerID":"C76D2820-C395-1029-BB86-001321B5C0B3", "CollectorID":"A75CF6A0-4948-102D-A613-000C29A9C3DB", "EventSourceGroupID":"A75CF6A0-4948-102D-A614-000C29A9C3DB", "EventSourceID":"A75CF6A0-4948-102D-A615-000C29A9C3DB", "ChainID":"1274696106664", "ChainSequence":"2" }
Event data is processed by the Collector running on the Collector Manager. For more information about event processing and parsing, see Section 4.0, Configuring Data Collection. Event data is subject to filtering rules set up on the event source, Connector, and Collector, so event data can be dropped, if necessary.
The event data partitions are closed after two days, and no more events are written to them. Even though the duration of the partition is only for one day, partitions are closed after two days to accommodate events arriving at the last moment. After the partitions are closed, they are compressed and archived.
Online partitions are stored in the /var/opt/novell/sentinel_log_mgr/data/eventdata directory, which is on the local file system. Partitions are created based on the dates and retention policies.
A central partition index is maintained in the database that keeps track of all the existing partitions and their location.
The following table describes the directory structure under the installation directory where event data is stored:
Table 3-3 Event Data Directory Structure