Hello,
we plan to ingest the access and error logs of Apache2 / nginx in IDR to alert unusual behavior.
Unfortunately, there are no standard event sources for this.
Does anyone have experience with these logs, custom parsing of the logs and creating rules to monitor the most important aspects?
I would be very pleased if you could share your experiences with me.
Elisabeth
You could set up a Custom log event source and set it to tail the apache logs remotely (you’d need to set up an smb share and provide credentials)
Alternatively the Insight Agent offers the ability to tail any log file locally and send it to Log Search see here Configure the Insight Agent to Send Additional Logs | InsightIDR Documentation
David
Hello David,
Thank you for your feedback.
Feeding logs into IDR only for the search is not practical. In addition to searching the logs, we also want to create detection rules to alert on unusual behavior. Additional logs collected by the Insight Agent will bypass the custom parsing module and the detection engine.
For these reasons, we planned to use the custom log event source and set up a custom parsing rule. We also discuss which activities should be monitored for web servers.
Do you have experience with parsing these logs? Do you have any recommendations for web server log activities that should be monitored?
Regards,
Elisabeth
Theres an alternative configuration you can use with the logging.json method to send in the data through an agent via the collector listening on a network port. This would enable you to build Custom Detection Rules (not just basic detection rules) as well as Custom Parsing Rules.
The logging.json configuration would look like this
{
"config":
{
"datahub": "<COLLECTOR_FQDN>:20001",
"state-file": "/opt/rapid7/ir_agent/components/insight_agent/common/config/logs.state",
"logs":
[
{
"token": "",
"enabled": true,
"name": "Apache Logs",
"path": "/var/log/nginx*.log"
}
]
}
}
You would need to set up a Custom Log Event Source in IDR, set it to Listen on Network port and enable the TCP encrypted option. Navigate to data collection, add event source, Add Raw Data -> Custom Logs
Note some important keys in the above configuration:
After saving the logging.json in the
/opt/rapid7/ir_agent/components/insight_agent/common/config.''
Once the le_realtime job starts - you should see a log lines like this
[agent.jobs.le_realtime]: Following Apache Log - /var/log/nginx*.log
it should begin tailing the Apache logs and streaming them to the event source.
Within a minute of the above log event (provided the log file you are tailing is active) you should see the events when hitting the View Raw Log button.
One limitation of this method is that if you tail multiple logs, all of them will be sent to the same log within log search, since the collector using a custom log event source is limited in this manner.
However you can work around this by leveraging the name key within the configuration which will appear in each log as part of the syslog header which is prepended by default.
David
Hi David,
I'm trying to use the method you suggested in your post. However, after adding my collector hostname to the datahub attribute, I see the following error in agent.log:
LOG:
2026-10-02 03:24:21,225 [ERROR] [6552:140712810968640:agent.jobs.le_realtime]: Err or establishing SSL socket. Error message: [SSL: CERTIFICATE_VERIFY_FAILED] certif icate verify failed: Hostname mismatch, certificate is not valid for ‘FQDN of collector'. (_ssl.c:1081)
2026-10-02 03:24:31,247 [ERROR] [6552:140712810968640:agent.jobs.le_realtime]: Error establishing SSL socket. Error message: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'FQDN of collector'. (_ssl.c:1081)
2026-10-02 03:24:41,275 [ERROR] [6552:140712810968640:agent.jobs.le_realtime]: Error establishing SSL socket. Error message: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'FQDN of collector'. (_ssl.c:1081)
2026-10-02 03:24:51,297 [ERROR] [6552:140712810968640:agent.jobs.le_realtime]: Error establishing SSL socket. Error message: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'FQDN of collector'. (_ssl.c:1081)
followed the steps in the Rapid7 documentation:
Configured the custom log in InsightIDR.
Downloaded the TLS certificate.
Appended the certificate to cafile.pem and restarted the agent service.
Note: I am using your logging.json on my Linux server to forward to my collector but the error always shown in agent.log and unable to forward to my collector
This error is indicating the cert is not valid, in order to troubleshoot further I’d recommend raising a support case so we can get a closer look at the logs and the config.
David
Thanks for the response! Could you share the full guidelines for this process? Also, I have two quick questions:
No the proxy is not supported with the datahub approach
You would need to append the TLS cert if you intend to use SSL. Are these machines local to the collector? Is SSL strictly required or could it be suppressed?
David
Hi David,
Thanks for clarifying the proxy behavior and TLS requirements.
Network & Security: The servers are on an internal network, but our security team requires TLS encryption for log transport.
The Issue: When pointing datahub to our local collector FQDN (test-example.lab.com:2025) with TLS enabled, the agent fails with an SSL verification error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch because the port presents the platform's AWS Global Accelerator certificate (*.awsglobalaccelerator.com).
We could take a look at this over a call, it may however be necessary to leverage some third party tool such as Nxlog if this configuration/setup cannot be corrected.
David
