| Object type: Data sync policy array | |
|---|---|
| Collection of data sync policy objects | |
| Field | Description |
| EventDataSyncPolicy | This is an array of <B><Data Sync Policy></B> objects. See below for a description of fields in Data Sync Policy objects. |
| Object type: Data sync policy object | |
|---|---|
| Information about the data sync policy | |
| Field | Description |
| alwaysSchedule | Boolean flag specifying whether data syncing is to run continuously ("true") or according to a schedule ("false"). If "false", the ScheduleItems field specifies the schedule to be used for syncing data. |
| backOffPeriod | The number of seconds to backoff between queries when the synced-to time is up to date. |
| countColumn | Column in destination table where event counts are to be stored. The type of this column should be capable of storing an integer. |
| dbConnectionConfig | This is a nested JSON <B><Database Connection></B> object that specifies the destination database to sync data to. It is used in conjunction with the <B>table</B> field to specify the destination table events are to be synced to. See below for description of fields in Database Connection objects. NOTE: If this field is missing, the policy is syncing event data to a table in the internal embedded database. |
| doSummaries | Boolean flag specifying whether this policy creates event summary records instead of syncing individual events. If this field is not present, it defaults to false - i.e., the policy is NOT a summary policy. If true, then the <B>summaryPeriod</B>, <B>countColumn</B>, <B>timeColumn</B>, and <B>summaryKeyColumn</B> fields will aos contain information. |
| enabled | Boolean flag specifying whether the policy is enabled or not. |
| fieldMappingStatus | This is a nested JSON <B><Field Mapping Status></B> object that indicates if there are problems with the destination table and/or column mappings. See below for description of fields in Field Mapping Status objects. NOTE: If this field is missing, there are no problems with the table and its column mappings. |
| filter | The Lucene filter specifying which events are to be synced. |
| forReporting | This is a flag that indicates whether the data sync policy is associated with a report. If true, the data sync policy was created for a specific report to sync event data for the report into the internal embedded database. The following fields should also be present to give more information about the report plugin the policy is associated with: <B>reportPluginName</B>, <B>reportPluginDisplayName</B>, <B>reportPluginDescription</B>, <B>reportPluginVersion</B>, and <B>reportPluginReleaseDat</B>. |
| id | This is the UUID of the data sync policy |
| maxBatchSize | The maximum number of events to write to the destination database in a single transaction. |
| maxEPSSize | The maximum number of events per second to sync. |
| partitionTable | A flag indicating whether the table is partitioned. NOTE: This only applies to data sync policies that sync data to the internal Postgres database. |
| reportPluginDescription | This is the description of the report plugin the data sync policy is associated with. It will only be present if the <B>forReporting</B> field is "true". |
| reportPluginDisplayName | This is the display name of the report plugin the data sync policy is associated with. It will only be present if the <B>forReporting</B> field is "true". |
| reportPluginName | This is the name of the report plugin the data sync policy is associated with. It will only be present if the <B>forReporting</B> field is "true". |
| reportPluginReleaseDate | This is the release date of the report plugin the data sync policy is associated with. It will only be present if the <B>forReporting</B> field is "true". |
| retentionPeriod | The number of days to retain data before deleting it. If missing, or if it contains a value <= 0, data is not deleted. |
| ScheduleItems | If the <B>alwaysSchedule</B> field is "false", this is a nested JSON object that specifies the schedule to use for doing data sync. Note that it contains a single internal field <B>scheduleItem</B> that is an array of <B><Schedule Item></B> objects. See below for description of fields in Schedule Item objects. |
| startSyncTime | This is the time that the data sync started from. It is specified in milliseconds since midnight, January 1, 1970 (UTC). |
| summaryKeyColumn | Column in destination table that holds a special summary key. The type of this column should be a VARCHAR capable of storing a 36 character UUID. This column is used internally when it is necessary to update a summary record. NOTE: This column should be indexed for performance reasons. |
| summaryPeriod | Number of minutes to summarize events over. All events having a common set of event fields (as specified in the <B>TableColumnMap</B> field) will be counted over time periods of this length. A single record with a count of events found during the time period will be stored in the destination table. NOTE: This must be a positive number. If omitted, or the value is <= 0, the policy will be treated as a normal data sync policy - i.e., it will NOT produce summaries. |
| syncInternalEvents | Boolean flag indicating whether or not to sync internal events. |
| table | This is a nested JSON <B><Table></B> object that specifies the destination table events are to be synced to. See below for description of fields in Table objects. NOTE: If this field is missing, the policy is syncing event data to the "events" table. |
| TableColumnMap | This is a nested JSON object that specifies the mappings between event fields and destination table columns. Note that it contains a single internal field <B>ColumnMap</B> that is an array of <B><Column Map></B> objects. See below for description of fields in Column Map objects. |
| timeColumn | Column in destination table where event time will be stored. Event time for a summary record is defined to be the time at the beginning of the summary period. For example, if the summary period is two minutes, then event times would potentially fall on every two minute boundary (such as 12:00, 12:02, 12:04, etc). The count would be the count of all events which occurred starting from that time for the duration of the summary. If the time column contained a time of 12:02, then the summary record contains a count is for all events that occurred between >= 12:02 and < 12:04 (note it is exclusive of 12:04). NOTE: This column should be indexed for performance reasons. |
| Object type: Data sync schedule item | |
|---|---|
| Information about the data sync schedule for this policy | |
| Field | Description |
| dayOfWeek | Day of the week data sync should occur in. 0=Sunday, 1=Monday, etc. -1=Every Day |
| duration | Number of minutes the data sync should last. 1 through 1440 (number of minutes in a day) |
| startHour | Hour of the day the data sync should start. 0 through 23. |
| startMinute | Minute of the hour the data sync should start. 0 through 59. |
| Object type: Database connection object | |
|---|---|
| Information about the connection to the database where the data sync data is to be stored | |
| Field | Description |
| database | Name of database. |
| dbPlatform | Type of database. Valid values are: "postgresql", "oracle11g", and "mssql2008". |
| hostName | Name or IP address of host where database resides. |
| password | Password of database user. |
| port | Port number for communication with database system. |
| userName | User name of database user to login to database. |
| Object type: Database table object | |
|---|---|
| Information about the database table where the data sync data is to be stored | |
| Field | Description |
| schemaName | Name of the schema for the destination table. NOTE: This is an optional field. It will default to the schema of the database user specified in the database connection information. |
| tableName | Name of the destination table. |
| Object type: Database field mapping status object | |
|---|---|
| Status of database mapping | |
| Field | Description |
| InvalidMappings | This is a nested JSON object that specifies which of the mappings between event fields and destination table columns are invalid. Note that it contains a single internal field <B>columnMap</B> that is an array of <B><Column Map></B> objects. See below for description of fields in Column Map objects. If this field is missing, there are no invalid field/column mappings. |
| tableStatus | A status code indicating any problems with the table. 0=No problems, 1=Error occurred checking table (see server log), 2=Table does not exist, 3=The "summary time" column is not indexed, 4=The "summary key" column is not indexed, 5=The "summary time" and "summary key" columns are not indexed. 6=The "event time" field is not synced as a timestamp (or date) and needs to be in order to support a retention period or table partitioning. 7=The column holding the "event time" field is not indexed and needs to be in order to support a retention period. NOTE: Status codes 3, 4, and 5 are applicable only to data sync policies that do summaries. |
| Object type: Data sync column mapping object | |
|---|---|
| Information about how the lucene fields map to the database fields | |
| Field | Description |
| columnName | Name of column in the database that the event field is to be stored in. |
| columnSize | Size of database column. NOTE: This only applies if the database column is a VARCHAR. |
| columnType | Data type of database column. Should be a java.sql.Types value (BIGINT, VARCHAR, etc.). |
| eventField | Name of event field that is to be synced. NOTE: These are the names of the event fields as specified <a target="_top" href="http://www.novell.com/developer/event_schema.html">here</a> in the <B>Tag Name</B> column. |
| nullable | Flag indicating whether database column can have null values. 0=Nulls not allowed, 1=Nulls allowed, 2=Unknown if nulls allowed. |
GET /datasync/policies
Data Sync Policy Fields { "id": "102B21D0-BE9B-102D-83DB-001A6B6D3CF6", "policyName": "My Data Sync Policy", "enabled": "true", "filter": "sev:[3 TO 5]", "syncInternalEvents": "false", "lagTime": "10", "retentionPeriod": "90", "partitionTable": "false", "backOffPeriod": "60", "maxEPSSize": "1000", "maxBatchSize": "100", "alwaysSchedule": "false", "ScheduleItems": { "scheduleItem": [{<Schedule Item>},{<Schedule Item>}...]}, "dbConnectionConfig":{<Database Connection>}, "table": {<Table>}, "fieldMappingStatus": {<Field Mapping Status>}, "TableColumnMap": { "ColumnMap": [{<Column Map>},{<Column Map>} ....]} "doSummaries": "false", "summaryPeriod": "0", "countColumn": "summary_count", "timeColumn": "summary_time", "summaryKeyColumn": "summary_key", "startSyncTime": "1288177541000", "forReporting": "false" } Schedule Item Fields { "dayOfWeek": "0", "startHour": "11", "startMinute": "23", "duration": "120" } Database Connection Fields { "hostName": "164.99.19.125", "port": "5432", "database": "SIEM", "userName": "appuser", "password": "star1111", "dbPlatform": "postgresql", } Table Fields { "schemaName": "my_schema", "tableName": "my_event_table" } Field Mapping Status Fields { "tableStatus": "2", "InvalidMappings": {"columnMap":[{<Column Map>},{<ColumnMap>}...]} } Column Map Fields { "eventField": "msg", "columnName": "msg", "columnType": "12", "nullable": "1", "columnSize": "4000" }