Any interest in Sharing Log Parsing Regex's?

InsightIDR
1

We have Silverfort for backend MFA protection and I recently built the regex parsers for ingesting those events into InsightIDR. Was curious if anyone wanted me to share the parsing regex?

I also have quite a few others I can share like Canon ImageRunner Syslog and far more verbose parsing for the Citrix Netscalers (parses all of the events we use not just the SSLVPN logs) and Cisco Umbrella (Will actually capture the user and dns category) than the default parser provides.

I write all of my parsers in regex as I find they are more flexible and accurate than the highlight method.

2 Likes
2

Sharing is caring, I for one would love to see some of your regex, feel free to share!!!

3

Would love to see your Netscaler regex’s, I have created my own but go the hybrid route where I highlight then go back and update the regex to make it more accurate.
I have the following regex’s if anyone is interested:

  • DHCP - added ‘source’ to the data as Rapid7 doesn’t parse this field. Not always there, but can be useful if it is
  • DNS - regex to parse unparsed logs
  • Checkpoint - Rapid7 does not parse the following fields: Policy name, rule name, rule number
    I have more, but they are for less common apps.
    There should be a channel or thread dedicated to best practices for log ingest configurations and parsing. People shouldn’t have to recreate the wheel if someone else already figured it out.
1 Like
4

The Netscaler regex would be great to see!

5

Would love to see some working examples as I will be creating my own from scratch as well. The highlight method only seems to work if the fields are ingested in the same order. For logs like Mimecast, they take a few forms, and highlighting them in different places does not seem to work.

6

So there’s actually several parsing rules applied to this log source, so you’ll have to create several parsing rules and apply them to your netscaler logs but here’s the parsers I have. I don’t have every log available parsed out since we don’t use all of the features.

Name:
netscaler - AAA LOGIN_FAILED

Filter:
AAA LOGIN_FAILED

Extracted Fields:
^<132>\s(?P<timestamp>[^\s]+)\s.*?default\s(?P<feature>\w+)\s(?P<vendor_message>[^\s]+)\s.*?(?:User\s(?P<user>[^\s]+)\s)?\-\sClient_ip\s(?P<source_ip>[^\s]+)\s\-\sFailure_reason\s\"(?P<failure_reason>[^\"]+)"(?:\s\-\sBrowser\s(?P<user_agent>.+))?

Name:
netscaler - GUI CMD_EXECUTED

Filter:
GUI

Extracted Fields:
^<134>\s(?P<timestamp>[^\s]+)\s.*?default\s(?P<feature>\w+)\s(?P<vendor_message>[^\s]+).*User\s(?P<user>[^\s]+)\s\-\sRemote_ip\s(?P<source_ip>[^\s]+).*?Command\s\"(?P<command>[^\"]+).*Status\s\"(?P<result>[^\"]+)\"

Name:
netscaler - ICAEND_CONNSTAT

Filter:
ICAEND_CONNSTAT

Extracted Fields:
^<134>\s(?P<timestamp>[^\s]+)\s.*?default\s(?P<feature>\w+)\s(?P<vendor_message>[^\s]+).*Source\s(?P<source_ip>[^\:]+)\:(?P<source_port>[^\s]+)\s\-\sDestination\s(?P<destination_ip>[^\:]+)\:(?P<destination_port>[^\s]+).*?domainname\s(?P<user>\w+).*?Duration\s(?P<duration>[^\s]+).*?Total_bytes_send\s(?P<total_bytes_send>\d+)\s-\sTotal_bytes_recv\s(?P<total_bytes_recv>\d+)

Name:
netscaler - SSLVPN ICASTART

Filter:
SSLVPN ICASTART

Extracted Fields:
^<134>\s(?P<timestamp>[^\s]+)\s.*?default\s(?P<feature>\w+)\s(?P<vendor_message>[^\s]+).*Source\s(?P<source_ip>[^\:]+)\:(?P<source_port>[^\s]+)\s\-\sDestination\s(?P<destination_ip>[^\:]+)\:(?P<destination_port>[^\s]+).*?domainname\s(?P<user>\w+).*?applicationName\s(?P<application_name>(?:\w\s\d+|[^\s])+)

Name:
netscaler - SSLVPN LOGIN

Filter:
SSLVPN LOGIN

Extracted Fields:
^<134>\s(?P<timestamp>[^\s]+)\s.*?default\s(?P<feature>\w+)\s(?P<vendor_message>[^\s]+)\s.*?Context\s(?P<user>[^\@]+)\@(?:\w+\.com\@)?(?P<source_ip>[^\s]+).*?Vserver\s(?P<destination_ip>[^\:]+)\:(?P<destination_port>[^\s]+).*?Browser_type\s\"(?P<user_agent>[^\"]+)\".*?SSLVPN_client_type\s(?P<sslvpn_client_type>[^\s]+)

Name:
netscaler - SSLVPN LOGOUT

Filter:
SSLVPN LOGOUT

Extracted Fields:
^<134>\s(?P<timestamp>[^\s]+)\s.*?default\s(?P<feature>\w+)\s(?P<vendor_message>[^\s]+)\s.*?User\s(?P<user>[^\s]+)\s\-\sClient_ip\s(?P<source_ip>[^\s]+).*?Vserver\s(?P<destination_ip>[^\:]+)\:(?P<destination_port>[^\s]+).*?Duration\s(?P<duration>[^\s]+).*?Total_bytes_send\s(?P<total_bytes_send>\d+)\s-\sTotal_bytes_recv\s(?P<total_bytes_recv>\d+).*?LogoutMethod\s\"(?P<logout_method>\w+)\"

Name:
NetScaler - TCP

Filter:
default TCP

Extracted Fields:
^<134>\s(?P<timestamp>[^\s]+).*?default\s(?P<feature>\w+)\s(?P<vendor_message>[^\s]+).*?Source\s(?P<source_ip>[^:]+)\:(?P<source_port>[^\s]+)\s.*?Destination\s(?P<destination_ip>[^\:]+)\:(?P<destination_port>[^\s]+).*?Total_bytes_send\s(?P<total_bytes_send>\d+)\s-\sTotal_bytes_recv\s(?P<total_bytes_recv>\d+)