Noobie here looking for some guidance.
I’ve read through the custom parsing tool document a few times and I get it but I’m still having some trouble with what I’m trying to extract from my Sonicwall firewall logs.
Example of normal log you’d use with custom parse tool (as seen on Using the Parsing Tool in InsightIDR - YouTube):
2019-10-01 13:50:47 10.0.0.241 GET /SMS_MP/.sms_aut MPLIST 80 - 10.0.0.241 SMS_MP_CONTROL_MANAGER - 200 0 0 410 4
I get that from this, I’d just be highlighting the fields I want and then naming them, then the rule would be able to extract them.
Unparsed source_data from my firewall log:
<129> id=firewall sn=ABCDEF123456 time="2021-09-28 17:41:32 UTC" fw=70.12.123.123 pri=1 c=32 gcat=3 m=609 src=51.15.146.240:5078:X2 dst=172.31.255.251:5060:X7 msg="IPS Prevention Alert: SCAN SIPVicious Activity 1, SID: 5616, Priority: Medium" msg="IPS Prevention Alert: SCAN SIPVicious Activity 1" sid=5616 ipscat="SCAN SIPVicious Activity 1" ipspri=2 n=1302 pktdatId=7013041180384428203 fw_action="drop"
At first I thought I’d extract id, sn, time, fw, gcat, fw_action, etc but then I realized that’s not correct since I would want to be selecting the actual values of those fields, not the fields themselves (or both). So would I be selecting only the field values in this case? For example, for id=firewall I would just select firewall? Otherwise it’s like I’d need to pre-parse them before parsing them.
Hi David,
What you have stated is correct. When you are creating your key-value pairs of what you want parsed (i.e. in the Extracted Fields step), you highlight the values, not the keys. Highlight the value (s). Verify that the values for the key-value-pair are highlighted in yellow as expected, and then click Validate. Once you click on Validate, you can then give the field a name, which is where you specify the key name, like “id”, “time”, etc.
Hopefully this makes sense!
Teresa
2 Likes
Yeah that does make sense and I actually did that initially, then deleted my fields because I didn’t think that would work. My worry was that, for example, if I had (from the yet unparsed text) gcat=6 and pri=2 and I am selecting the 6 and 2 to be parsed into different fields, wouldn’t there be the chance that they could get mixed up at some point? Also will IDR and/or regex just ignore the other text (the unparsed fields)?
Yes, the parsing rule will just ignore the other text, i.e. the unparsed fields. That is, you can parse out all of the data if you want to, or just parse out those fields that you are interested in.
Good luck!
hmm… I can tell the custom parsing is going to have issues… I’m working on adding fields and I see that when I mouse-over some of the fields I’ve created, the logs highlight accordingly and some are fine, such as serial number; but others such as message (msg field) are showing all kinds of different things being selected on different log lines. And several are like that.
I guess this pretty much just comes down to regex which I don’t know at all yet.
You are definitely on the right track. You actually have a few things you can try.
-
You can try parsing every field. Sometimes this lets the machine-generated regular expression figure out what to do and the logs will then parse as desired.
-
I wrote a blog series (2 blogs long) to try to help figure out how to use the Manual Regex mode if you don’t have much experience with regex. The first blog is here: Introducing the Manual Regex Editor in IDR’s Parsing Tool: Part 1 | Rapid7 Blog
Regex parsing rules are always a little messy, so a “condensed” blog on this topic is still not that short! However, if you have an hour or so, you can probably learn what you need to create the parsing rule yourself.
- As a shortcut, you can use a tool like regex101.com to test out your regular expression. If you have a little familiarity with regex, you can copy your raw logs into the Test String box. Switch the Data Parsing Tool to Manual Regex mode and copy out the regex that is there and not quite working. Then use the Explanation in regex101 to fix up the regex so it works. Then you can just copy it back into InsightIDR and test it out.
You can also open up a support case with Rapid7 to get some help with the rule.
Hopefully this helps!
Teresa
3 Likes
Teresa,
Wow awesome thanks! I will read through your blog post now. And thanks for the link to regex101, I was looking for something like that!
1 Like
Hi @David_Williams ,
In addition to all of @tjc_tester 's great advice, you can also train the parsing tool in guided mode. If you select the “message” value on one line and notice that the wrong values are extracted on other lines, you can correct those lines by selecting the expected value on each of those lines. As you add more sample data this way, the algorithm will refine the parsing rule and hopefully provide better results.
Good luck!
Andrew
1 Like
Andrew,
Oh nice! I was wondering if there was something like that I could do but didn’t figure it out… I’ll give it a shot now though