Dear All,
We are trying to create an Detection Rule from the Unparsed LogSet. But we were to do so.
Basically, Unparsed LogSet contains the IP Address Hits for Particular URL’s.
Trying to right an Custom Query like “trigger an alert if for example, the same IP is observed 1000 times in 5 minutes performing the same action”.
Any guidance to help writing an query to accomplish same.
Hi @rnandakumar ,
if your logs in the unparsed data logset are in key:value pair format you will be able to leverage a Custom Detection Rule. If the keys you are looking at are not in KVP format a Custom Parsing rule will need to be applied first to extract those values and assign key names.
For example if you have a log that looks something like this
{“timestamp”:“2024-02-14T15:16:21.000Z”, “source_ip”: “100.110.10.1”, “action”: “request_login”}
You would want to build your Custom Detection rule to focus on the Unparsed Data logset, select the individual log in question, and then apply any relevant where clause, such as where(action!=“unknown”) or whatever is relevant, a where is optional.
Then in the Match Conditions it will want to be configured like this
Hello @david_smith … Thanks for the response. Yes, logset are unparsed and not in KVP Format.
Looks like I might need help on the Parsing Rule, I tried to create parsing rule and extracted the fields called SRC_IP and Application_Name for the below sample unparsed logs.
SRC_IP – I’m assigning for IP Address mentioned like 1.1.1.1
Application_Name – I’m assigning for URL: https://example.com
But, I’m not sure where is the issue, when searching whit those fields it showing any results. Any insights on this?.
I’d recommend raising a support case for this so we can access your account and take a closer look @rnandakumar
Great. Thanks for the Information. Yes. We do have the Support Case: [06206882]. Could you please and let us know.
Currently, I’m able to Parse the Data and guessing its working fine.
But Event type is getting appended with “custom.data”. Is that usual behavior?. Why because, there is error during creation of detection rule.
Since we are concentrating on the Source_IP, we have written the condition like below, but it doesn’t producing any investigations, even though we have more than 10 count of IP’s with in 5 Minutes.
Any help can be provided here?
The custom_data.keyname is expected.
However the unrecognized keys is a little troubling, I have a feeling this is a limitation of the guided custom rule creation experience. @nicholas_mifsud1 might have more to add here.
This rule should work as expected from what I can tell, if we see 10 occurrences from any source_ip+url_host combination in 5 minutes.
David
Thanks for the response. But, its not producing any investigations, eventhough we see more than 10 hits in 5 Minutes.
Any thing missed here?. Please review and help.
Hi @rnandakumar
Apologies for not getting back to you sooner. At first glance at this thread it would seem that you have set this up correctly, but I expect there is some nuance of the custom parsing that we’re overlooking. I will investigate this asap and get back to you later today with an update.
We’ve had a look into this and the problem is that the rule is both grouping by and counting unique values of the source_IP field. Because of that all groups will only ever have a single unique IP.
To resolve you either need to remove the field from the group by (if you want to be alerted on every 10 unique IPs), or remove the count by to get an alert when each unique IP happens 10 times.
We know this is complicated and are always looking to make this less prone to misconfiguration. By default we prevent keys being used in both fields, but because this data was from the custom parser and not in our ‘known’ list our validation did not apply - so we will be looking to rectify that v. soon.
Please let us know if that doesn’t help and/or if you have any further qs
Thanks for the response. You mean to say we have to configure like below?.
Please review and let us know.
That will alert only once when 10 unique IPs are seen in a 5 minute window. If that is what you want?
If you want to alert when a unique IP is seen 10 times in a 5 minute window then you need to put the source_ip field in the group by section (and remove it from the other part)