Pagination problem when export IDR Detection Rules

InsightIDR API
1

I am trying to get an export of IDR detection rules per the documentation here [ SIEM | © Rapid7 ] because the web interface method does not export the workflow column.

However, I can’t get past a pagination problem. I only get the first 100 records returned, and in a never-ending loop.

Here is what I have:

import os

import requests

import pandas as pd

# Configuration

API_KEY = "My_Org_Key"

BASE_URL = "https://us.api.insight.rapid7.com/idr/v1/rules/summary"

OUTPUT_DIR = r"My_Local_Win_Path"

PAGE_SIZE = 100

# Create output directory if it doesn't exist

os.makedirs(OUTPUT_DIR, exist_ok=True)

headers = {

*"X-Api-Key": API_KEY,*

*"Accept": "application/json"*

}

page_index = 0

while True:

*params = {*

    *"item": \["workflows", "name", "PRIORITY_LEVEL"\],*

    *"size": PAGE_SIZE,*

    *"index": page_index*

*}*



*response = requests.get(*

    *BASE_URL,*

    *headers=headers,*

    *params=params*

*)*



*response.raise_for_status()*



*data = response.json()*



*# Adjust this based on the API response structure*

*records = data.get("data", \[\])*



*if not records:*

    *print(f"No more records found at index {page_index}.")*

    *break*



*# Save page to CSV*

*csv_file = os.path.join(*

    *OUTPUT_DIR,*

    *f"rules_summary_page\_{page_index}.csv"*

*)*



*pd.DataFrame(records).to_csv(csv_file, index=False)*



*print(f"Saved {len(records)} records to {csv_file}")*



*# Stop if fewer than PAGE_SIZE records returned*

*if len(records) < PAGE_SIZE:*

    *print("Last page reached.")*

    *break*



*page_index += 1*

Thanks,

Cw.

2

For anyone looking for a possible web interface solution to this here is what I ended up doing. It is a few minutes of effort but works for me since I am only interested in “HIGH” and “CRITICAL” detections and I’ve spent way too many hours trying to get the API approach to work:

  1. Navigate to the Rapid7 Detections Library web interface.
  2. Open the browser developer tools (F12)
  3. Open the "Network" tab
  4. Create a filter to search for "rules"
  5. Navigate through each page of rules and watch for the .json response to build
  6. Open each .json response as it is created and copy/paste the contents to a .json text file.
  7. Ask Copilot to create a .csv from the .json file
  8. Ask Copilot to only include the following: name,rule action,priority_level,description,event_types,workflows

Hopefully someone finds this useful.

Cw.