InsightIDR

Data Collection Methods

InsightIDR gathers information about your network and assets in two different ways: the Collector and the Insight Agent.

The Collector

The Collector is an on-premise component of InsightIDR that gathers data from event sources.

The Insight Agent

The Insight Agent, however, is a service on your individual assets (employee computers, guest laptops, workstations, etc) that collects system information and sends it back to InsightIDR for analysis and user attribution services.

Rapid7 recommends deploying the Insight Agent to critical infrastructure, which provides real-time detection of advanced attacks on critical infrastructure, and to assets that leave the corporate network, as the agent will monitor those systems continuously.

Endpoint Monitoring

The optional monitoring service is called the Endpoint Monitor and can be configured to look at all assets or only a certain range of assets when the agent is in Scan Mode. This service is not natively installed on your machine.

For systems that are not as critical, but are on your network, use the Endpoint Monitor.

Network Data Collection

The following sections describe the most common data collection methods for log data. In some cases, you'll need to provide the directory or file location where the Collector can access the server logs. You can specify a local folder path or a Windows UNC (Universal Naming Convention) path to a hosted network drive.

The Collection Method in an event source configuration specifies how the data will be either pushed to or pulled by your Collector.

WMI

WMI (Windows Management Instrumentation) allows your Collector to poll event sources and pull in the events used for user attribution. WMI is available for all Windows-based event sources, and it is recommended for data collection whenever possible.

WMI requires access via an admin account & communication over ports 135, 139 & 445.

Listen for Syslog

Almost every event source supports a Listen for Syslog collection method. This configuration allows you to forward log events from your event source to your Collector on a unique port, just as you would with a syslog server over a predefined port. Collects will accept syslog from UDP and TCP traffic - UDP is lighter on the network and generally recommended outside of strict compliance environments.

See Syslog Logging for more information.

Log Aggregator

If you have an existing SIEM or log aggregator in place, then you can forward logs from that source to your InsightIDR Collector so long as the logs are sent in their raw form. The same predefined, unique port rule applies -- the Collector is designed to only listen for one event log stream per port. To forward multiple event source logs from a single log aggregator, you'll need to configure each event source log to be sent to a unique port.

SIEM

In some deployments, a SIEM may already collect data. You can configure your SIEM to send logs to the collector by selecting the appropriate SIEM under Log Aggregator when configuring the event source in InsightIDR.

Watch Directory

The watch directory is the network location of a watch directory where log files are copied. This method monitors a specified directory on a local or remote host and uploads files added to the directory, at 30-second scan intervals. Use this method for log files that roll over to new files, for example, Microsoft DHCP and IIS (Internet Information Services) log files.

Note that if the directory contains other files, you will want to enter a file pattern to specify which files should be collected from the directory.

To enhance security, you can select the watch shared remote directory option, which requires the Collector to authenticate to the directory as it would for any file share.

Tail File

InsightIDR will watch a log file and ingest any new data that is added to it. This is the network location of a tail file where log data is stored. This method watches a specific file written to disk using the equivalent of the Unix tail command, at 20-second scan intervals. Use this method for log files that are written continuously to a single file, such as Windows DNS log files or Microsoft DNS.