I ask a lot of questions here on the Discuss forums, so I thought I’d mix it up a bit by sharing something which blocked me for a while and what I did to get it working.
The goal seemed simple: retrieve a few thousand records from an API, and take action on each of the records. Of course I encountered some issues along the way, which is why I wrote this thread. I was able to solve the problem with a fair bit of testing, and by refactoring what I know about Python/Bash scripting into a more InsightConnect-ish way of thinking.
I’m using a Loop Step to request multiple pages of a paginated API using the HTTP/REST plugin, and since the API I’m using doesn’t return metadata about which page it has returned, or how many pages there are in total, I have to figure that out myself.
Let’s say there are up to 500 records per page, and an unknown number of total records/pages to retrieve. I need to supply the page number each time:
/path/to/api/endpoint?page=1
/path/to/api/endpoint?page=2
etc.
This calls for a Loop step, but how to specify that page number? The InsightConnect data model only exposes an Object with the name of the Loop step - I’ve imaginatively called it Loop API in this case - and that Object doesn’t have any exposed properties:
I used an Artifact step to see what’s inside that Loop API object:
{"$item":0,"$outputs":[{}],"__jobExecutionScopeKey00000000-0000-0000-0000-000000000000":["e4a26679-9f70-414c-9312-612cbb7352dd"]}
$item looks promising, and it turns out to be what I’m after.
Some indexing confusion to sort out here:
- the API pagination in my case is 1-indexed, so requesting &page=0 returns nothing; your mileage may vary by API
- the
$itemproperty of the Loop step Object is 0-indexed, even though the Iteration step # in the InsightConnect UI is 1-indexed (see red boxes in the next screenshot)
So I need to turn the $item property into $item + 1 for the HTTP/REST step to fire into the API, but how?
My best answer so far is to use jq, which feels a bit like using a hedge-trimmer to slice a cake, but it works. I supply the jq Action with $item from the Loop step, and add 1 to it:
I did try editing the Output schema of the jq Action to export the number I want with a specific name, but InsightConnect consistently resets it to have just one variable: a String called json_out, so I went with it.
Finally, I feed that calculated page index number into the HTTP/REST Action route field:
/path/to/api/endpoint?page={{["JQ Index Arithmetic"].[json_out]}}
Then I set the Loop step to be Repeat Until, set a very high value for the Repeat N Times value - a logical test on something inside the loop would be more reliable, but this will work unless you have more than half a million records - and then finally added a Break Loop step to test the number of items returned by the API for each page. If it’s less than 500, then we’ve reached the end of the data.
I’m pretty sure someone out there can show a more elegant want to do any/all of the above using InsightConnect - for starters, jq cannot be the best way to do the arithmetic - but I got this working and I wanted to share in case it helps someone with a similar issue.
