Formatting Column Names returned from Surface Command Query

InsightConnect Workflows
1

hey there,

I have a surface command query returning not very "safe" column names. The API used returns labels for a column instead of the column name, i.e. CVSS (V3) Score.

I've added a jq step after the query return to reformat the names, but now all the steps that was using the query. yes i could go to each step and update,

i was wondering if there was a way to modify the variable(Output from the query step) itself instead of creating a new one?

Also open to another way of accomplish this task

As an addendum t this question, how do i output the result to reference it the same way as the output of the query - which is an json array of objects?

i've tried many different ways but but nothing iis working.

The reformat step is jq, flags: , filter:(.. | select(has("CVSS (V3) Severity")?)) |= with_entries(if .key == "CVSS (V3) Severity" then .key = "CVSS_V3_Severity" else . end) and the output as

image
image509×246 18.9 KB

how when i try to refernce it in the next step with the ip put as {{["ReformatKeys"].[CleanedKeys]}} or {{["ReformatKeys"].[json_out]}} i get an error saying it's null.

2

sorry to answer my own question . . .

the answer for the second part is Type Converter plugin, String to Object. Even though you set the output of the jq to array object, it seems to be a string. the hint was a ton of \n in the file. Although i am sure i did something wrong in the out phase of the jq step.

jq operates on json, seems like it would retun and object., So every time i have a jq step, i then have to do a type conversion from string to object??

3

If you use the JQ plugin the output is hardcoded as a string. Unless the output you are seeking is a string, you need to format your output as an object, then use the string to object with type converter.

4

appreciate the response Darrick and I think you hit directly on my greatest ignorance of the workflow steps. Depending on the context a json string could also be an object. ATM i don't know what i need.

I execute a rather large query of vulnerabilities and asset joined returned as json(output:{{["Get-Adjusted-Vulns"].[items]}} - Array of Items). So a row for each unique pair of vuln and asset. Is this a string or an obj?

The next step is jq transforming the keys json "safe", the output is {{["Format_GetVulns_To_Safe_Column_Names"].[json_out]}} - The output JSON

The output looks like this

image
image181×412 6.2 KB

The next step is jq which pulls out unique vulnIDs. When trying to select the input for this step i have the options but none of them work for jq to operate on

image
image443×245 15.2 KB

ReformatKeys and myobj - error our staying not 'action input JSON was invalid', <ValidationError: "'' is not of type 'object'"> which is labeled as an object and the .json_out is a string, but is polluted with \n, so it is an actual string.

I think I want the output of a jq step to a json string that i can use in another jq step for further formatting, etc. How do i know when i need a type conversion? and how/what should i be setting the output of a jq step to get back a clean json object/string? i still don't know what to call it.

5

Always Return an Object That Contains an Array

This example shows how to modify a jq statement so it always returns an object that contains an array, instead of returning the array alone.
In InsightConnect’s JQ editor, do not include jq at the beginning — just use the filter directly.

Example JSON Payload

{
"servers": [
{"hostname": "web01", "ip": "10.0.0.1"},
{"hostname": "web02", "ip": "10.0.0.2"},
{"hostname": "db01", "ip": "10.0.0.3"}
]
}

Filter That Returns Only an Array

.servers | map(.hostname)

Output:
["web01", "web02", "db01"]

Filter That Returns an Object Containing That Array

{data: (.servers | map(.hostname))}

Output:
{
"data": ["web01", "web02", "db01"]
}

One-Liner Pattern for Any Filter

{data: (<your_filter_here>)}

This ensures your output is always an object with a consistent structure:
{"data": [...]}

6

I always format every output as an object. It is just habit. I haven't had a use case where I need to send output of JQ to another JQ statement though. It seems like you should be able to accomplish everything you need with a single JQ statement.

7

great response! and i feel like always returning an object is the way to go. according to your response i think i am screwing up the filter more than the output, although i i am still not sure which object i should be setting int he return type.

i agree about completeing everything i need to do in one one jq step. the multi step approach is me being a novice.

My end target object(example below) from the vulns query is list of vulnerabilities with a key in each that includes all the assets that has that vulnerability. The first jq step was to make the keys safe making the next jq step easier to quote properly.

"items": [
{
"ARS_Severity_Adjusted": "High",
"ARSadjusted": "750.0",
"Active_Risk_Score": "1000",
"CVSS_V3_Score": "8.3",
"CVSS_V3_Severity": "High",
"IsCISAExploitable": "false",
"Operating System": "Microsoft Windows Server 2022 21H2",
"Title": "Google Chrome Vulnerability: CVE-2025-2783 Incorrect handle provided in unspecified circumstances in Mojo on Windows",
"Vulnerability_ID": "google-chrome-cve-2025-2783",

  "exposure": "Internal",
  "isAIFM": "false",
  "isDORA": "false",
  "assests": [
    {"hosaname": "0f3f1a42a847451", 
    "IP": "0f3f1a42a847451",
    "d_macs": "12:52:20:02:57:67, 12:ca:8a:06:af:89",
    },
  {"hosaname": "0f3f1a42a847451", 
    "IP": "0f3f1a42a847451",
    "d_macs": "12:52:20:02:57:67, 12:ca:8a:06:af:89",
    },]
},
8

To be transparent, I’m not deeply fluent in JQ and typically rely on AI tools to assist with JQ-related tasks. As long as you provide the AI with a clear example of your data structure, it should generate accurate and usable filters.

If your data includes sensitive information, be sure to anonymize or randomize it before sharing. You can easily create a small Python script to ingest your data, preserve the same key structure, and replace all values with randomized or placeholder data. This allows you to safely share sample payloads with AI tools without exposing any confidential information.

9

using chatgpt to great one big beautiful jq was the idea! I did that, and it looks like it would work, but i am back to the issue of output type from one step to the input of the next step.

the one instruction it givges is - Here’s a jq filter you can drop straight into the Rapid7 InsightConnect jq step, assuming the input is a JSON array of objects:

input is a JSON array of objects: - i thought i was doing that

the outputs from the query step are

image
image541×241 18.1 KB

the input put to the jq step according to the download file from testing, looks like this

image
image244×297 6.16 KB

the jq filter field starts like this v- which is where i think the problem is

image
image349×368 5.23 KB

the error i get

image
image856×233 12.5 KB

10

The JQ step only accepts an object as input. Not an array of objects.

Object Only
Object Only533×228 23.8 KB

11

ok, setting it to object. so in order to pass an array of maps to the filter
had a feeling you were going to say that. When i select myobj as the input i get the error

Failed validating 'type' in schema['properties']['json_in']:
{'description': 'Data in JSON format to be passed to jq',
'order': 1,
'title': 'JSON Input',
'type': 'object'}
On instance['json_in']:

('action input JSON was invalid', <ValidationError: "'' is not of type 'object'">)

so is this a place where i need to do a type conversion?

12

Whatever step has the data you want to pass to jq the very first variable from that step is the entire output. The data type is an object. Pass the entire output of the step into your jq step, even if you only want to work with a section of that object such as a specific array.

13

gotcha, pass the entire object, then in the jq filter refer to the key in that object. that worked!! thank you!

14

going to post this here as it is in the same vein, just different steps. I think i ned the same rule like you said for in and out jq - always an object. the step were talking about is working. there are 2 top keys, vulns which hasl athe vulns and vuln_list which is formatted to go into an IN statement in IVM query.

output of the jq step

image
image520×214 13.3 KB

i am struggling to reach into myobj.vuln_list. i had this working in a different workflow, but i created a artifact first, then did a type conversation. is there any way to reach directly into the return of the jq?

the object
{
"vulns": [
{
"ARS_Severity_Adjusted": "High",
"ARSadjusted": "750.0",
"Active_Risk_Score": "1000",
"CVSS_V3_Score": "7.8",
"CVSS_V3_Severity": "High",
"IsCISAExploitable": "false",
"Title": "Amazon Linux 2023: CVE-2025-32463: Important priority package update for sudo",
"Vulnerability_ID": "amazon_linux_2023-cve-2025-32463",
"exposure": "Internal",
"assets": [
{
"Hostname": "ip-10-18-136-91",
"IP": "10.18.136.91",
"Operating_System": "Amazon Linux 2023 2023.7.20250527",
"d_macs": "02:f6:87:00:1a:77, 02:42:07:b8:7f:13",
"isAIFM": "false",
"isDORA": "false"
},
...
}
],
"vuln_list": "'amazon_linux_2023-cve-2025-32463','apache-httpd-cve-2011-3192','apache-tomcat-cve-2025-24813','apple-osx-coremedia-cve-2025-24085','cisco-anyconnect-cve-2020-3153','cisco-anyconnect-cve-2020-3433''"
}

15

Apologies I am not good at understanding without some better examples.

When you show data it is good to know what step the data comes from, what the output from your JQ step looks like and what it is named.

As an example:

I have a step that query's system x for new vulns called GetNewVulns. The output from that step is:

Random Data Example


I then pass this variable from GetNewVulns to JQ (Include screenshots if possible). 

The JQ step is called ExtractWithJQ and the filter works. It gives me this random string of data... (More screenshots or examples)

I want to get specific data from ExtractWithJQ step, but am uncertain how.

With what you are providing currently it is difficult to interpret what the actual issue is. To better help us get you a great solution can you follow the format provided above?

16

totally appreciate the hand holding. as i'm sure you can tell, new to the community. Used your post to create a forum post template of sorts.

i got this one solved, although i am still a little confused on return types. but i'm sure that will come the more i build. Thank you!

17

and we're going to try this out Moose . . . lol. I don't think i understand the outputs.

I have a simple python step(named "limit number of vulns while building") that is just picking three items out of a json array. The output of this step which is correct, looks like

image
image415×404 15.7 KB

data
{
"$success": true,
"limited_vulnerabilities": [
{
"ARS_Severity_Adjusted": "High",
"ARSadjusted": "750.0",
"Active_Risk_Score": "1000",
"CVSS_V3_Score": "7.8",
"CVSS_V3_Severity": "High",
"IsCISAExploitable": "false",
"Title": "Amazon Linux 2023: CVE-2025-32463: Important priority package update for sudo",
"Vulnerability_ID": "amazon_linux_2023-cve-2025-32463",
"assets": [
{
"Hostname": "ip-10-18-136-91",
"IP": "172.17.0.1",
"Operating_System": "Amazon Linux 2023 2023.7.20250527",
"d_macs": "02:42:96:b9:25:b9, 02:f6:87:00:1a:77",
"isAIFM": "false",
"isDORA": "false"
},
{
"Hostname": "shane-ec2",
"IP": "172.29.0.1",
"Operating_System": "Amazon Linux 2023 2023.7.20250609",
"d_macs": "02:42:46:d9:05:3a, 02:23:98:51:1c:87, 02:42:9c:3d:c2:ff, 02:42:6d:b8:74:e8",
"isAIFM": "false",
"isDORA": "false"
}
],
"exposure": "Internal",
"solution": {
"Best Solution": "Update sudo to the latest version available from Amazon, using tools like yum or dnf. By default, AL2023 instances do not automatically receive additional critical and important security updates at launch. The instance includes updates available at the time the version of AL2023 and the chosen AMI were created. For more information on updating your instance, visit this documentation. (https://docs.aws.amazon.com/linux/al2023/ug/updating.html)",
"Solution Type": "WORKAROUND",
"Vulnerability-Name": "Amazon Linux 2023: CVE-2025-32463: Important priority package update for sudo",
"nexpose_id": "amazon_linux_2023-cve-2025-32463",
"vulnerability_id": "393601"
}
},
{
"ARS_Severity_Adjusted": "High",
"ARSadjusted": "750.0",
"Active_Risk_Score": "1000",
"CVSS_V3_Score": "7.5",
"CVSS_V3_Severity": "High",
"Credentials_Succeeded": "true",
"IsCISAExploitable": "false",
"Title": "Apache HTTPD: Range header remote DoS (CVE-2011-3192)",
"Vulnerability_ID": "apache-httpd-cve-2011-3192",
"assets": [
{
"Hostname": "HC1APTR5SV",
"IP": "10.14.76.30",
"Operating_System": "Microsoft Windows Server 2019 Standard Edition 1809",
"d_macs": "00:50:56:9e:4c:15",
"isAIFM": "false",
"isDORA": "false"
}
],
"exposure": "Internal",
"solution": {
"Best Solution": "Download and apply the upgrade from: https://httpd.apache.org/download.cgi (https://httpd.apache.org/download.cgi)",
"Solution Type": "ROLLUP",
"Vulnerability-Name": "Apache HTTPD: Range header remote DoS (CVE-2011-3192)",
"nexpose_id": "apache-httpd-cve-2011-3192",
"vulnerability_id": "27785"
}
},
{
"ARS_Severity_Adjusted": "High",
"ARSadjusted": "750.0",
"Active_Risk_Score": "1000",
"CVSS_V3_Score": "9.8",
"CVSS_V3_Severity": "Critical",
"Credentials_Succeeded": "true",
"IsCISAExploitable": "false",
"Title": "Apache Tomcat: Important: Remote Code Execution and/or Information disclosure and/or malicious content added to uploaded files via write enabled Default Servlet - (CVE-2025-24813)",
"Vulnerability_ID": "apache-tomcat-cve-2025-24813",
"assets": [
{
"Hostname": "HC1-PSFT07-VM",
"IP": "10.14.120.77",
"Operating_System": "Microsoft Windows 10 22H2",
"d_macs": "00:50:56:9e:79:9d",
"isAIFM": "false",
"isDORA": "false"
}
],
"exposure": "Internal",
"solution": {
"Best Solution": "Download and apply the upgrade from: http://archive.apache.org/dist/tomcat/tomcat-11/v11.0.12/ (http://archive.apache.org/dist/tomcat/tomcat-11/v11.0.12/)",
"Solution Type": "ROLLUP",
"Vulnerability-Name": "Apache Tomcat: Important: Remote Code Execution and/or Information disclosure and/or malicious content added to uploaded files via write enabled Default Servlet - (CVE-2025-24813)",
"nexpose_id": "apache-tomcat-cve-2025-24813",
"vulnerability_id": "340976"
}
}
],
"total_vulnerabilities": 40
}

The step configuration, the outputs are

image
image599×211 14.3 KB

The next step is a loop over the three items. However, when i try to configure the For Each loop i don't get any of the outputs from the previous step, neither the ones in the step or from the step configuration.

image
image382×557 24.5 KB

I have a feeling this is because i didn't add an output to the python step before the loop. I tried that and still didn't see it.

So i guess my question is how do i "create" a map/collection that contains a key holding the list of vulns output that can be referenced by the loop step?