Paging in API

InsightVM API
1

Spent an afternoon attempting to return all our “reports”. We have over 1700 and the api limit 500.

I’m seeing a totalPages and totalResources value come back in the results mentioned in docs. I tried

#this url works and returns 500
insightserver.example.com/api/3/reports?page=0&size=500

When I try a simple look in python

for x in range(0, 3):
url = ‘nsightserver.example.com/api/3/reports?page=’ + str(x) + ‘&size=500’

the same 500 results come back each time

When I try putting a specific page, I get an error.

url = ‘https://insightserver.example.com/api/3/reports?page=1&size=500’

Documentation and examples aren’t clear how to do this…or what I can find.

An example would be helpful. Using vscode to step through code so I see the values.

2

Here is the python script.

import requests
import configparser
from requests.auth import HTTPBasicAuth

def getReportInfo(reportId):
try:
url = “https://in.example.com:3780/api/3/reports/” + reportId
except:
print(“nofound”)
r = requests.get(url, auth=(login, pwd), verify=False).json()
return r

def log(value, filename):
f = open("/Rapid7/files/" + filename, “a+”)
f.write(value + “\n”)
f.close()
return

load credentials

config = configparser.ConfigParser()
config.read("/sampleFile/file", encoding=None)

login = config.get(“ABC”, “user”)
pwd = config.get("CBD, “pass”)
logOutputFile = “reportInfo.csv”

Headers for Logfile

log(“name,owner,template,timezone,str(id),reportFrequency,reportFormat,reportOwner”, logOutputFile)

for x in range(0,3):
url = ‘https://in.example.com:3780/api/3/reports?index=’ + str(x) + ‘&size=500&sort=id,DESC’
response = requests.get(url, auth=(login, pwd), verify=False).json()
reports = response[“resources”]

for report in reports:
    id = str(report["id"])
    print("id:" + str(id))

reports = ""
response = ""
3

the docs here show the index increasing.

https://docs.rapid7.com/insight/api-overview/

4

@aiuex79_aiuex79 Not an exact example, but the script included in this post walks through paging results with the InsightVM console API.

Looking at your examples, I see one that is using index as the parameter for the page where it should be page specifically for the console api: InsightVM API (v3). For example, /resources?page=2&size=10 would be how you pull the next 10 results from page 2.

5

The documentation within the /api/3/reports sections has this… Looks like what you posted. The link I posted in the overview uses the “index” example. Neither code sample is helpful…most likely my newness to the API. I want to use this paging to cleanup old reports.

Paging

Pagination is supported on certain collection resources using a combination of two query parameters, page and size. As these are control parameters, they are prefixed with the underscore character. The page parameter dictates the zero-based index of the page to retrieve, and the size indicates the size of the page. For example, /resources?page=2&size=10 will return page 3, with 10 records per page, giving results 21-30. The maximum page size for a request is 500.

Index is used at this link - Insight Platform API Overview | Insight Cloud Documentation

Pagination

Requests that return multiple items can be paginated. When a response is paginated, it will contain a metadata section describing the current page and a data section containing the response data.

For example:

TEXT

1{2  "data": [3    {"id": 3, "name": "vm-101"},4    {"id": 4, "name": "vm-102"}5  ],6  "metadata": {7    "index": 1,8    "size": 2,9    "total_pages": 4,10    "total_data": 711  }12}13

The metadata section may contain the following attributes:

  • Index - The current page, starting from 0. This value will always be provided.
  • Size - The number of data items in the current page. This value will always be provided.
  • Total_pages - The total number of pages that make up the complete response. This will be provided if possible.
  • Total_data - The total number of data items that make up the complete response. This will be provided if possible.
  • Sort - The attributes used to sort the complete response. This will be provided if the response is sorted.

You can control which page is returned using query parameters.

For example:

TEXT

1https://us.api.insight.rapid7.com/idr/v1/investigations?index=3&size=50&sort=type,ASC,created,DESC

The query parameters used for pagination include:

  • Index - The page to return, starting from 0

  • Size - The number of items to return per page. NOTE: The returned size may be smaller than the requested size in order to maintain API performance. Always check the metadata.size parameter in the response.

  • Sort - The key,order pairs to sort by, separated by commas. The key is one of the supported sort keys for a given API. Order is one of ASC, for ascending, or DESC, for descending. Multiple sort keys can be provided by specifying multiple key,order pairs. Please note that this may not be supported by some APIs. You can verify this from your product’s API documentation.

6

Sorry Zac…thanks for responding. My post got accidentally posted before I could say “I’ll definitely check out your example”. :slight_smile:

7

One unfortunate thing at the moment between products is that InsightIDR uses index and InsightVM console api uses page. So just keep that in mind. That example should be a decent starting place - just needs modified for your specific resource/needs. The full script is at the bottom of the initial post and you can expand it to copy/paste it somewhere to get started.

8

Besides here is there any other suggested resources for samples? Just curious if I’m missing something.

9

As I loop through increasing the page count from 0, 1, 2, 3 but when I look in the metadata…Shows 0, 0, 2, 3. The first loop works, i get a 500 error when trying the second and other loops. I looked in the logs and can’t determine what the 500 error is. Looks like a null…

10

@aiuex79_aiuex79 I’m not aware of any API examples maintained anywhere other than the posts here. There are several around running reports, asset searches, etc with full scripts to base things off of.

If you are paging and getting a 500, it’s very possible there is an exception not properly being handled and it is something our VM engineering team would need to look into. It might be a very specific scan that is causing a problem, but they can likely help identify it and see if there is anything that needs to be done on our side to resolve it for good. I would recommend opening a support ticket with:

  • Simple curl request of that page
  • Response of the request
  • Logs from nsc.log when that request is made

That should get support the details they need to work with engineering.

11

Thanks Zak. The 500 is weird. My return using the reports API shows 4 pages (0,1.2,3)

When hitting with postman, pages 0, 3 work all the time. Pages 1 is always a 500 and Page 2 works rarely but 500 usually. I opened a case with support. They can’t offer support for scripting or programming. I get that, but postman with a static page, there i something. I don’t see anything obvious in nsc.log.

12

Thanks Steve for letting me know. I’m going to see if I can track down the ticket internally and will get back to you.