Loop step on JSON from HTTP REST API

InsightConnect
1

Goal: retrieve data from Insight Platform API Get Users (or other arbitrary JSON sources) and run a Loop step on each member of the JSON array.

Problem: A Loop Step requires an InsightConnect array - you can’t specify an arbitrary object as it’s a dropdown - but I’ve not been able to figure out a way to transform the JSON I have into an object which the Loop step will recognize as something to loop over.

For example, here’s what the body_object output of an HTTP REST API query to the InsightIDR users API endpoint looks like:

{
  "object": [
    {
      "email": "user0.lastname@organization.domain",
      "federated": true,
      "first_name": "user0",
      "id": "000bb000-ec0a-0000-b000-000c00d0000c",
      "last_login": null,
      "last_name": "lastname",
      "platform_admin": false,
      "status": "ACTIVE",
      "timezone": "Europe/Rome"
    },
    {
      "email": "User1.Lastname@organization.domain",
      "federated": true,
      "first_name": "User1",
      "id": "00a000a0-0bce-0000-bc0d-00000aaa000a",
      "last_login": "0000-00-00T00:00:00Z",
      "last_name": "Lastname",
      "platform_admin": false,
      "status": "ACTIVE",
      "timezone": "UTC"
    },
    {
      "email": "user2.lastname@organization.domain",
      "federated": true,
      "first_name": "user2",
      "id": "00e0f00d-c000-0a00-b000-0a0000ddb000",
      "last_login": "0000-00-00T00:00:00Z",
      "last_name": "lastname",
      "platform_admin": false,
      "status": "ACTIVE",
      "timezone": "UTC"
    }
  ]
}

How do I get InsightConnect to acknowledge that this is a repeating data structure which I’d like to iterate over? Is there some jq magic? A plugin which will convert this into an array? Something else?

The Loop Step documentation has a lot of useful info on how Loop Step outputs work, but almost nothing on inputs, so I’m stuck.

While we’re here: does anyone knows how to get this structure into a Python Script plugin step, or does the params.get call require that the Python Script plugin receives a dict with named keys as input?

1 Like
2

Hey Adam!

This is a great question - to wire up your REST → loop steps, you can edit the output of the REST step to more granularly define the fields returned in your API call’s response body. Once an array is explicitly defined, the Loop step form should then allow you to select that array to iterate over.

To do this, open your REST step and go to the Configure Details panel, and you should see an Edit option available next to the Output section at the bottom. You’ll see a few fields already defined here - most of these should be left as is - but the body_object output field is the one that contains the full API request body including your user data. Within that body_object field, you can do + Add Variable to define specific fields within that object based on the data from your API. For your case, it looks like you’ll want a variable named object, with Data Type = Array, and Array Type = Object (you could even define subfields in this object with email, first_name etc. but that is not required).

Once that’s done, you should be able to select that body_object.object array variable from the REST step in your loop step.

Just a warning, this output editting should only be done for special steps like REST where there are dynamic outputs - you need to be careful to enter the EXACT field names and data types or else it may not work as expected :slight_smile:

For the Python step, yes, you can pass in any input object and can reference any fields in that object by doing params.get("field name") - e.g. in this case, you could pass in the body_object from your REST step as Input, and should be able to do params.get("object") to access the array. You may have seen this already but there’s some more info on using Python steps here: Python 2 or 3 Script | InsightConnect Documentation

I’ve attached a screenshot of a REST step I have with the customized output schema for a similar use case - hope this helps!

custom output

4 Likes
3

Thank you very much @charlotte_mchugh3 - that’s such a detailed and thoughtful response. Exactly what I needed to know!

I’ll admit that I was so focused on the Loop step that I didn’t stop to wonder if there were any output config options for the HTTP/REST step. Now I know to look, I see the Edit button:

image

I did have another read of the HTTP/REST Requests plugin documentation just now but I don’t think it mentions output configuration.

[CM] For your case, it looks like you’ll want a variable named object , with Data Type = Array, and Array Type = Object (you could even define subfields in this object with email , first_name etc. but that is not required).

Yep. I’ve done this just now and it works just as expected: array appears in the Loop step dropdown. I like that explicitly defining the fields (or even some of them) is an option but not a requirement. It’s useful to be able to refer to variables using the InsightConnect UI when they’re defined, but if there are lots, it’d be a bit arduous to have to define them all by hand in a web dialog so it’s great to have the choice.

[CM] Just a warning, this output editting should only be done for special steps like REST where there are dynamic outputs - you need to be careful to enter the EXACT field names and data types or else it may not work as expected :slight_smile:

I believe it :upside_down_face:

[CM] For the Python step, yes, you can pass in any input object and can reference any fields in that object by doing params.get("field name") - e.g. in this case, you could pass in the body_object from your REST step as Input, and should be able to do params.get("object") to access the array.

Thanks! I thought I tried this already but I must have done something wrong. Now I know for sure that it’s a valid method, I’ll give this another go and report back.

1 Like
4

Glad you got it working! And that’s a good callout about the documentation, I’ll pass that on to the team and see if we can get some more details in there :slight_smile:

1 Like