Hi there community!
I’m running into some hurdles when I try to setup custom log parsing on one of our event sources.
The logs are RADIUS/NPS logs from our Windows NPS server set up to allow clients to authenticate on our internal network.
I have been using the log parser in a bunch of projects where I have managed to get the coverage which we needed however this time I can’t get things to work reliably.
The main reason seems to be that when the NPS service is writing the logs to the logfile which I have the IDR collector ingest the NPS service is not putting the fields in the same order.
The order seems to be inconsistent not only among different “called station”/“NPS Client” and events but also for the same called station and event it can come out in different order in the log file.
Since the order can be different I have issues to get it mapped correctly in IDR.
I have been going over making corrections with the help of the guided parser - however after I have done a couple of corrections it all of a sudden do not seem to know what to parse any longer and just don’t take any fields.
Is there anyone that have had similar kind of issue before?
@support_rapid7 are there any plans on expanding the functionality of the custom parser?
It would have been great if it could actually detect the log keys based automatically in the cases where the logs are in ex XML or JSON format and that you just have to specify which of the key-values you want to enable.
Thanks in advance
Hey Richard,
there are currently no plans to enhance the custom parser in its current form. However what you describe is a known limitation of the custom parser. Insofar as it will typically work well when events are well structured and uniform, and will fail when logs have missing or optional fields/values.
We recommend leveraging the filter in your parsing rule to include only the log events which contain certain strings, this can help in the case where a field is not always present, and then you would need to create another parsing rule to cover the instances where the number of fields differs.
For further assistance I would recommend you raise a support case so that we can evaluate your rules and perhaps suggest some workarounds or tips, since we will need to look at your specific logs and use cases.
David
Thanks for your response David.
The problem I have is not that the fields are missing in the logfiles. They are all there but they can come in different order from time to time.
This is also not possible to solve with using the filter, as I mention this also happen from time to time for the same kind of events and from the same source.
It seems like that when the NPS service is writing to the log file it just takes the fields in anyway and not the same order all the time. But all fields I have been looking for have still been in the logfiles just that the parser fails to keep track on the different orders I want to mark.
For what it’s worth I am running into this exact same issue with NPS logs and NAC. I have given up on trying to parse these through the log files and am planning to try sending the NPS events from Event Viewer to InsightIDR using nxlog’s json output to see if that will automatically parse out the different fields.
1 Like
I have had that thought of using NXlog as well but not had the time to really dig into that option.
Please update here if you get that parsing the different fields better.