Group By Fields In Syslog

Hello,
I have some raw syslogs from Cisco ISE feeding into IDR.
I would like to do a query that groups by a field within the syslog data. It is not a Key field.
See below for sanitised example:

    {
  "timestamp": "2021-05-20T11:05:51.000Z",
  "facility": "LOCAL_USE_6",
  "severity": "NOTICE",
  "hostname": "xxxx",
  "appname": "CISE_Failed_Attempts",
  "source_data": "<181>May 20 12:05:51 ukdcaisenode01 CISE_Failed_Attempts 0000012226 1 0 2021-05-20 12:05:51.602 +01:00 0007382428 5400 NOTICE Failed-Attempt: Authentication failed, ConfigVersionId=86, Device IP Address=x.x.x.x, Device Port=59606, DestinationIPAddress=x.x.x.x, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=INVALID, Protocol=Radius, RequestLatency=37, NetworkDeviceName=xxxxxx, User-Name=INVALID,

I want to be able to do a query such as:
where(appname = CISE_Failed_Attempts) groupby(UserName)

This does not return any results, and im assuming this is because UserName is not a Key, its a value within a Key.

Fairly new to IDR, so any tips would be welcomed here!

Hi Ross,

I’d recommend you try out our Custom Parsing tool to extract the values you are looking to query on. Here are the docs: Custom Parsing Tool | InsightIDR Documentation

With this feature you can extract KVPs from log lines by highlighting values in your logs and providing a key name for those values.

This does not apply retroactively however, so in order to query against the old data, you would need to use a Regex capture group, see more here: Use a Search Language | InsightIDR Documentation

What that would look like for your example,

where(/UserName=(?P<User_name>[^,]*/ User_name=INVALID)

this query anchors on “UserName=” and then the regex here states match everything up until the next comma. Once you extract your KVP and name it, you can use it for key comparison or groupby or a calculation.

Hope this helps

David

Thanks David, very helpful.
I looked at the Custom Parsing Tool, however, as the “UserName” field is not in the same position in every log, it seems that the Parsing Tool would not play nicely with this. Is this correct?

Thanks for the Regex, saved me many hours of re-learning it…

Thanks,
Ross

Hi Ross,

you can use a filter step when building out the rule and perhaps you might need multiple parsing rules if the key changes in some log lines. Also we have manual regex custom parsing coming very soon. So the limitations of our automatic parser not always working will be (somewhat) easily overcome by writing your own regex.

David