| Object type: Data sync policy object | ||
|---|---|---|
| Information about the data sync policy | ||
| Field | Required | Description |
| alwaysSchedule | false | Boolean flag specifying whether data syncing is to run continuously ("true") or according to a schedule ("false"). If "false", the <B>ScheduleItems<B> field specifies the schedule to be used for syncing data. |
| backOffPeriod | false | The number of seconds to backoff between queries when the synced-to time is up to date. |
| countColumn | false | Column in destination table where event counts are to be stored. The type of this column should be capable of storing an integer. |
| dbConnectionConfig | false | This is a nested JSON <B><Database Connection></B> object that specifies the destination database to sync data to. It is used in conjunction with the <B>table</B> field to specify the destination table events are to be synced to. See below for description of fields in Database Connection objects. |
| doSummaries | false | Boolean flag specifying whether this policy creates event summary records instead of syncing individual events. If this field is not present, it defaults to false - i.e., the policy is NOT a summary policy. If true, then the <B>summaryPeriod</B>, <B>countColumn</B>, <B>timeColumn</B>, and <B>summaryKeyColumn</B> fields will aos contain information. |
| enabled | false | Boolean flag specifying whether the policy is enabled or not. |
| filter | false | The Lucene filter specifying which events are to be synced. |
| maxBatchSize | false | The maximum number of events to write to the destination database in a single transaction. NOTE: A transaction may exceed this size if it is required to sync all of the events in a particular time boundary in a single transaction |
| maxEPSSize | false | The maximum number of events per second to sync. (NOTE: This is not currently used) |
| policyName | false | Name of data sync policy. NOTE: The policy name must be unique. If it is not unique, the create policy call will fail. |
| retentionPeriod | false | The number of days to retain synced data. If missing, or zero, data will not be deleted. Otherwise, it will be deleted after the specified number of days |
| ScheduleItems | false | If the <B>alwaysSchedule</B> field is "false", this is a nested JSON object that specifies the schedule to use for doing data sync. Note that it contains a single internal field <B>scheduleItem</B> that is an array of <B><Schedule Item></B> objects. See below for description of fields in Schedule Item objects. |
| startSyncTime | false | This is the time that the data sync started from. It is specified in milliseconds since midnight, January 1, 1970 (UTC). |
| summaryKeyColumn | false | Column in destination table that holds a special summary key. The type of this column should be a VARCHAR capable of storing a 36 character UUID. This column is used internally when it is necessary to update a summary record. NOTE: This column should be indexed for performance reasons. |
| summaryPeriod | false | Number of minutes to summarize events over. All events having a common set of event fields (as specified in the <B>TableColumnMap</B> field) will be counted over time periods of this length. A single record with a count of events found during the time period will be stored in the destination table. NOTE: This must be a positive number. If omitted, or the value is <= 0, the policy will be treated as a normal data sync policy - i.e., it will NOT produce summaries. |
| syncInternalEvents | false | Boolean flag indicating whether or not to sync internal events. |
| table | false | This is a nested JSON <B><Table></B> object that specifies the destination table events are to be synced to. See below for description of fields in Table objects. |
| TableColumnMap | false | This is a nested JSON object that specifies the mappings between event fields and destination table columns. Note that it contains a single internal field <B>ColumnMap</B> that is an array of <B><Column Map></B> objects. See below for description of fields in Column Map objects. |
| timeColumn | false | Column in destination table where event time will be stored. Event time for a summary record is defined to be the time at the beginning of the summary period. For example, if the summary period is two minutes, then event times would potentially fall on every two minute boundary (such as 12:00, 12:02, 12:04, etc). The count would be the count of all events which occurred starting from that time for the duration of the summary. If the time column contained a time of 12:02, then the summary record contains a count is for all events that occurred between >= 12:02 and < 12:04 (note it is exclusive of 12:04). NOTE: This column should be indexed for performance reasons. |
| Object type: Data sync schedule item | ||
|---|---|---|
| Information about the data sync schedule for this policy | ||
| Field | Required | Description |
| dayOfWeek | false | Day of the week data sync should occur in. 0=Sunday, 1=Monday, etc. -1=Every Day |
| duration | false | Number of minutes the data sync should last. 1 through 1440 (number of minutes in a day) |
| startHour | false | Hour of the day the data sync should start. 0 through 23. |
| startMinute | false | Minute of the hour the data sync should start. 0 through 59. |
| Object type: Database connection object | ||
|---|---|---|
| Information about the connection to the database where the data sync data is to be stored | ||
| Field | Required | Description |
| database | false | Name of database. |
| dbPlatform | false | Type of database. Valid values are: "postgresql", "oracle11g", and "mssql2008". |
| hostName | false | Name or IP address of host where database resides. |
| password | false | Password of database user. |
| port | false | Port number for communication with database system. |
| userName | false | User name of database user to login to database. |
| Object type: Database table object | ||
|---|---|---|
| Information about the database table where the data sync data is to be stored | ||
| Field | Required | Description |
| schemaName | false | Name of the schema for the destination table. NOTE: This is an optional field. It will default to the schema of the database user specified in the database connection information. |
| tableName | false | Name of the destination table. |
| Object type: Data sync column mapping object | ||
|---|---|---|
| Information about how the lucene fields map to the database fields | ||
| Field | Required | Description |
| columnName | false | Name of column in the database that the event field is to be stored in. |
| columnSize | false | Size of database column. NOTE: This only applies if the database column is a VARCHAR. |
| columnType | false | Data type of database column. Should be a java.sql.Types value (BIGINT, VARCHAR, etc.). |
| eventField | false | Name of event field that is to be synced. NOTE: These are the names of the event fields as specified <a target="_top" href="http://www.novell.com/developer/event_schema.html">here</a> in the <B>Tag Name</B> column. |
| nullable | false | Flag indicating whether database column can have null values. 0=Nulls not allowed, 1=Nulls allowed, 2=Unknown if nulls allowed. |
| Object type: Data sync policy object | |
|---|---|
| Information about the newly created data sync policy | |
| Field | Description |
| alwaysSchedule | Boolean flag specifying whether data syncing is to run continuously ("true") or according to a schedule ("false"). If "false", the ScheduleItems field specifies the schedule to be used for syncing data. |
| backOffPeriod | The number of seconds to backoff between queries when the synced-to time is up to date. This should be the value that was specified for the create request. |
| countColumn | Column in destination table where event counts are to be stored. The type of this column should be capable of storing an integer. |
| dbConnectionConfig | This is a nested JSON <B><Database Connection></B> object that specifies the destination database to sync data to. It is used in conjunction with the <B>table</B> field to specify the destination table events are to be synced to. See above for description of fields in Database Connection objects. This should be whatever was specified for the create request. |
| doSummaries | Boolean flag specifying whether this policy should do event summaries instead of syncing individual events. If this field is not present, it defaults to false - i.e., the policy is NOT a summary policy. If true, then the <B>summaryPeriod</B>, <B>countColumn</B>, <B>timeColumn</B>, and <B>summaryKeyColumn</B> fields must also be supplied. |
| enabled | Boolean flag specifying whether the policy is enabled or not. NOTE: It is possible for a policy to NOT be enabled even if the REST request specified that it should be. The policy may not be enabled if the destination database or destination table cannot be accessed. |
| filter | The Lucene filter specifying which events are to be synced. |
| forReporting | This is a flag that indicates whether the data sync policy is associated with a report. It should be false - this REST api is NOT allowed to create a data sync policy and associate it with a report. |
| id | This is the UUID of the newly created data sync policy. |
| maxBatchSize | The maximum number of events to write to the destination database in a single transaction. This should be the value that was specified for the create request. NOTE: A transaction may exceed this size if it is required to sync all of the events in a particular time boundary in a single transaction. |
| maxEPSSize | The maximum number of events per second to sync. This should be the value that was specified for the create request. |
| policyName | Name of data sync policy. |
| retentionPeriod | The number of days to retain data before deleting it. If missing, or if it contains a value <= 0, data is not deleted. This should be the value that was specified for the create request. |
| ScheduleItems | If the <B>alwaysSchedule</B> field is "false", this is a nested JSON object that specifies the schedule to use for doing data sync. Note that it contains a single internal field <B>scheduleItem</B> that is an array of <B><Schedule Item></B> objects. See above for description of fields in Schedule Item objects. This should be whatever was specified for the create request. |
| startSyncTime | This is the time that the data sync will be started from. It is specified in milliseconds since midnight, January 1, 1970 (UTC). This should be whatever was specified for the create request. |
| summaryKeyColumn | Column in destination table that holds a special summary key. The type of this column should be a VARCHAR capable of storing a 36 character UUID. This column is used internally when it is necessary to update a summary record. NOTE: This column should be indexed for performance reasons. |
| summaryPeriod | Number of minutes to summarize events over. All events having a common set of event fields (as specified in the <B>TableColumnMap</B> field) will be counted over time periods of this length. A single record with a count of events found during the time period will be stored in the destination table. NOTE: This must be a positive number. If omitted, or the value is <= 0, the policy will be treated as a normal data sync policy - i.e., it will NOT produce summaries. |
| syncInternalEvents | Boolean flag indicating whether or not to sync internal events. This should be the value that was specified for the create request. |
| table | This is a nested JSON <B><Table></B> object that specifies the destination table events are to be synced to. See above for description of fields in Table objects. This should be whatever was specified for the create request. |
| TableColumnMap | This is a nested JSON object that specifies the mappings between event fields and destination table columns. Note that it contains a single internal field <B>ColumnMap</B> that is an array of <B><Column Map></B> objects. See above for description of fields in Column Map objects. This should be whatever was specified for the create request. |
| timeColumn | Column in destination table where event time will be stored. Event time for a summary record is defined to be the time at the beginning of the summary period. For example, if the summary period is two minutes, then event times would potentially fall on every two minute boundary (such as 12:00, 12:02, 12:04, etc). The count would be the count of all events which occurred starting from that time for the duration of the summary. If the time column contained a time of 12:02, then the summary record contains a count is for all events that occurred between >= 12:02 and < 12:04 (note it is exclusive of 12:04). NOTE: This column should be indexed for performance reasons. |
POST /datasync/policy
Data Sync Policy Fields { "policyName": "My Data Sync Policy", "enabled": "true", "filter": "sev:[3 TO 5]", "syncInternalEvents": "false", "lagTime": "10", "retentionPeriod": "90", "partitionTable": "false", "backOffPeriod": "60", "maxEPSSize": "1000", "maxBatchSize": "100", "alwaysSchedule": "false", "ScheduleItems": { "scheduleItem": [{<Schedule Item>},{<Schedule Item>}...]}, "dbConnectionConfig":{<Database Connection>}, "table": {<Table>}, "fieldMappingStatus": {<Field Mapping Status>}, "TableColumnMap": { "ColumnMap": [{<Column Map>},{<Column Map>} ....]} "doSummaries": "false", "summaryPeriod": "0", "countColumn": "summary_count", "timeColumn": "summary_time", "summaryKeyColumn": "summary_key", "startSyncTime": "1288177541000", "forReporting": "false" } Schedule Item Fields { "dayOfWeek": "0", "startHour": "11", "startMinute": "23", "duration": "120" } Database Connection Fields { "hostName": "164.99.19.125", "port": "5432", "database": "SIEM", "userName": "appuser", "password": "star1111", "dbPlatform": "postgresql", } Table Fields { "schemaName": "my_schema", "tableName": "my_event_table" } Field Mapping Status Fields { "tableStatus": "2", "InvalidMappings": {"columnMap":[{<Column Map>},{<ColumnMap>}...]} } Column Map Fields { "eventField": "msg", "columnName": "msg", "columnType": "12", "nullable": "1", "columnSize": "4000" }
{ "id": "102B21D0-BE9B-102D-83DB-001A6B6D3CF6", "policyName": "My Data Sync Policy", "enabled": "true", "filter": "sev:[3 TO 5]", "syncInternalEvents": "false", "lagTime": "10", "retentionPeriod": "90", "partitionTable": "false", "backOffPeriod": "60", "maxEPSSize": "1000", "maxBatchSize": "100", "alwaysSchedule": "false", "ScheduleItems": { "scheduleItem": [{<Schedule Item>},{<Schedule Item>}...]}, "dbConnectionConfig":{<Database Connection>}, "table": {<Table>}, "fieldMappingStatus": {<Field Mapping Status>}, "TableColumnMap": { "ColumnMap": [{<Column Map>},{<Column Map>} ....]} "doSummaries": "false", "summaryPeriod": "0", "countColumn": "summary_count", "timeColumn": "summary_time", "summaryKeyColumn": "summary_key", "startSyncTime": "1288177541000", "forReporting": "false" }