Log Sets Missing From Custom Parsing Tool

Hi folks! I’m trying to parse some data coming from AWS via a lambda function but when I load up the custom parsing tool the log sets don’t appear in the tool. Is this something to do with how they’ve been ingested? Is there a workaround?

You can only custom parse logs which flow through an event source, other logs such as Data Sent via the Insight Agent, cannot be custom parsed.

Which logs are you trying to custom parse? You can workaround it using regex in your search query, as you can build capture groups on the fly Components for Building a Query | InsightIDR Documentation

Hey @david_smith1 , data send from the agent like additional logs neither? It actually makes no sense for me, I mean, in a specific case I´m grabbing the logs from a .log file perfectly structured in order to be able to easily parsing.

Correct the additional logs option cannot be custom parsed unless you send the logs via a collector, you can do so, however the method is not publicly documented.

Instead of the default logging.json you would use this instead

{
"config": {
"datahub": "COLLECTOR_FQDN:20001",
"state-file": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\state.file",
"formatter": "plain",
"logs": [
{
"token": "",
"enabled": true,
"name": "Insight Agent log",
"path": "C:\\DNSLogs\\dns*.log"
}
]
}
}

Note you would need to set up a collector based custom log event source, listening on network port. Select an unused TCP port such as 20001 and select send encrypted.

Leave the token as “”, it needs to remain empty.

Ensure you update the datahub parameter to include the FQDN of the collector.

When using this method you will then be able to custom parse your logs.

David