I have a workflow that collects the investigations and sends it our DWH (data warehouse). Now it happens that we have 3 pages of investigations.
How do I iterate through the other pages? I was look into a loop but that does not seems to be working for me as the loop item (array) is of course the investigation itself.
Or does this mean I need to sent each investigation by itself (hope not).
Can you share the plugins you are using?
What part of the investigation do you want to go to your data warehouse?
The plugin actions should say what data type they expect.
The action to send to the data warehouse would say “string”, or “Array”. The data type needs to match.
Without knowing what tools exactly you are using I would imagine you would need to:
List investigations which outputs the array:
Loop Through the investigations using each individual investigation ID
Inside of the loop send the relevant data for each investigation to your Data Warehouse.
From your description above, I would expect to loop through and send each investigation individually, unless your data warehouse has an “add investigations in bulk” type of API endpoint you could target.
Thanks for reaching out @darrick_hall1,
Can you share the plugins you are using?
We are using the Rapid7 InsightIDR plugin:
I am getting an output, similar to as described by the documentation:
Example input according to doc:
{
"email": "user@example.com",
"end_time": "2020-06-01T12:11:13+05:30",
"index": 0,
"priorities": [
"LOW",
"MEDIUM",
"HIGH",
"CRITICAL"
],
"size": 100,
"sort": "Created time Ascending",
"sources": [
"USER",
"ALERT"
],
"start_time": "2020-06-01T12:11:13+05:30",
"statuses": [
"CLOSED"
]
}
Input from my end:
{
"email": "",
"end_time": "2024-03-05T00:00:00+01:00",
"index": 0,
"priorities": [],
"size": 0,
"sort": "",
"sources": [],
"start_time": "2024-02-25T23:00:00+00:00",
"statuses": [
"CLOSED"
]
}
Output:
{
"$success": true,
"investigations": "<<referenced:bigdata>>",
"metadata": {
"index": 0,
"size": 100,
"total_data": 225,
"total_pages": 3
}
}
To modify the index each time so you are on a new page I created a workflow that you can use as a reference. Essentially instead of looping over an array, you would do a loop until. So you would choose x number of times to loop. You could set it at a 100 as an example.
Every time it loops there is a count done. It starts at 0.
I use Math to add 1 to that count. It breaks when the math result equals the total pages. Let me know if that helps.
List Investigations By Pages.icon (24.6 KB)
I use the API for stuff like that because you can export everything based on time, ticket status, etc. and you get all the same data with less fuss. For example, retrieving all open investigations with PowerShell looks like this:
$headers = New-Object “System.Collections.Generic.Dictionary[[String],[String]]”
$headers.Add(“Accept-version”, “investigations-preview”)
$headers.Add(“Accept”, “application/json”)
$headers.Add(“X-Api-Key”, “”)
$response = Invoke-RestMethod ‘https://us3.api.insight.rapid7.com/idr/v2/investigations?size=20&sort=created_time,DESC&statuses=OPEN’ -Method ‘GET’ -Headers $headers
$response | ConvertTo-Json
Maybe you don’t want to want to get all open tickets, because you could end up with duplicate data. In that case, you could create a task that runs once every 24 hours. you’d have to add something like this to the PowerShell script to generate the current time and current time -24h
$currentDateTime = (Get-Date).ToString(“yyyy-MM-ddTHH:mm:ssZ”)
$previousDateTime = (Get-Date).AddHours(-24).ToString(“yyyy-MM-ddTHH:mm:ssZ”)