InsightIDR

Rapid7 Universal Event Sources

InsightIDR can now universally support selected data types from any product’s logs, so long as you convert the log output from your product to JSON that matches the Universal Event Format (UEF) contract.

This allows you to send data from otherwise unsupported products into InsightIDR and still experience the same functionality as supported event sources, such as User Behavior Analytics. All logs that meet the UEF contract will also appear in log search.

Need help transforming your logs?

Read instructions on transforming your logs in this Rapid7 blog post or on the Transform Logs to UEF help page.

Required Format

Field
Required?
Validation
Description

version

Yes

Must be a documented version string for the Universal Event event_type.

Each event_type is versioned. This version reflects the version of the specific event_type for a Universal Event.

event_type

Yes

Any documented supported value.

The event type of this Universal Event.

time

Yes

Must be a valid ISO 8601 extended timestamp with millisecond precision, such as the following:
yyyy-MM-ddTHH:mm:SS.SSSZ

The ISO 8601 extended timestamp.

See Time Validation for more information.

custom_data

No

Must be a JSON object.

Use this field to send any additional information. This data will be available for log search and LEQL queries.

Please refer to the documentation for each event_type to see additional required and optional fields that apply.

Timestamps

Timestamps with fewer than 3 significant digits in the sub-second portion will be augmented to 3 significant digits. For example, 2018-06-07T12:34:56.1Z will be interpreted as 2018-06-07T12:34:56.100Z.

Additionally, 2018-06-07T12:34:56Z (seconds only precision) will be interpreted as 2018-06-07T12:34:56.000Z.

Timestamps with greater than 3 significant digits in the sub-second portion will be truncated to 3 significant digits. For example, 2018-06-07T12:34:56.123456Z (sub-millisecond precision) will be interpreted as 2018-06-07T12:34:56.123Z.

Expected Format

You must send events to the InsightIDR collector in UTF-8 format, with each log line representing a single event and a newline delimiting each event.

For example, a DHCP event would be represented as: {"event_type":"DHCP_LEASE","version": “v1”,"time": "2018-06-08T18:18:18.123Z","client_hostname":"pc.acme.com","client_ip":"10.6.102.53","operation": "OBTAIN"}

The same event in expanded JSON format would be:

{
  "version": “v1”,
  "event_type": "DHCP_LEASE",
  "time": "2018-06-07T18:18:31.000123+0100",

  // event type-specific fields here
  “client_hostname”: ”pc.acme.com”,
  “client_ip”: ”10.6.102.53”,
  “operation”: “OBTAIN”
}

What If Logs Do Not Match the Format?

InsightIDR will only process UEF events that meet the exact format stipulated for a given event_type and version. If the logs do not contain the required information or violate the contract of the UEF, InsightIDR will not parse the logs and they will not be processed by the InsightIDR analytics engine or appear in log search.

Possible Violations of UEF

Possible violations of the UEF contract include:

  • Malformed JSON objects
  • Log lines that contain more than one or only part of a UEF event
  • UEF events with unrecognized child fields on the root member object
  • UEF events without all of the required fields
  • UEF events containing an invalid value for an optional or required field
    • For example: If a field is child of the root member object and the contract mandates the value be an IPv4 address, but the value for that field is not a valid IPv4 address

Supported Event Types

InsightIDR supports the following universal event sources: