Log Sources Dashboard

[ajain]

Hi Team,

Can we define a log search to fetch insight about data being ingested from our different log sources, as we would like to fetch this data via IDR API but not sure this can be accomplished or not?

[mmur_gt4e (Mike)]

Hey ajain, What kind of data you want to retrieve?

You may looking for this:

docs.rapid7.com/insightidr/log-search-api/#operation/getOrganizationsLogUsage

[ajain]

Thanks this helps, another thing which I was looking for was to measure data volume ingested per log source. There is a API endpoint Data Size Broken Down By Log in the above link and output contains “id”. How can we determine which log source this ID belongs to so we can measure metrics on our end?

[ajain]

For ref, If we have log source as Okta how to determine which is the respective ID to map it against the API result to understand the volume of logs ingested?

[ajain]

Also, can we get logs in EPM(Events per month)?

[david_smith1 (David Smith)]

We have a precomputed query that shows usage per log, you can find it in the Dashboard cards view here

Dashboard\Query Events per log

Hi @talford , there is a Pre-Computed query (PCQ) which gets you what you want. However it will not include a log if there are no events received. If you navigate to Dashboards, either create a new Dashboard or add a Card to an existing one, then when building the card, select the Pre-Computed Query tab Then Search for ‘Usage’ and you should see this [Screenshot 2024-01-05 at 4.57.58 PM] This query will return a table view of each log source and how many bytes were received. David

[ajain]

@david_smith1 I am looking for the following things?

• events per month data for each log source
• Total volume for logs ingested per month in events per month, how we see in IDR->Settings->monthly data usage

[david_smith1 (David Smith)]

@ajain in terms of events per month you would be able to build a custom query and save it as a Precomputed query -

groupby(#log)calculate(count)timeslice(1d)limit(10000)

select all log sources and Save it as a Precomputed Query

This would allow you to see the day over day trends of each log source,

As an aside you could build out PCQs for each event source, selecting the logs from that event source, such as O365 which sends logs to Ingress Auth and Cloud Service Activity and Admin Activity and then build a single dashboard card for each event source, then build a report which would include each event source on a unique card with the total count of events trending over time?

David