Security-related tasks often come with long lists of data that you have to work through, whether it’s a list of indicators, such as URLs, or users, or something more complex like alerts. And if you’re looking to automate the processing of that data with InsightConnect, you need to be able to reference the entries within that list to do something with each of them. So how exactly do you do that? Well, the most common way would be to use the Loop Step.
How do you know when you need to use a loop step?
The most telltale sign of needing to use a Loop within a workflow is that you have a list, or an array as it’s called in InsightConnect, and that you want the workflow to be able to do something with the individual entries within that list. A common example might look like the following. You create a workflow that consumes some form of text, perhaps from a potential phishing email or from an alert description, and you want to extract any URLs from that text. An easy way to do the extraction would be to pass the text variable into the URL Extractor Action of the ExtractIt plugin like so:
Now when we extract those URLs, there could be any number of them actually contained within the text. So if we zoom in on the output of the extraction step:
We will see that the URLs are represented as an array. This means if we want to access the actual URLs within that array, we will need to loop over it.
Remember, an array is just a list of items. An array of three URLs may look like the following:
["google.com", "bing.com", "yahoo.com"]
Creating your loop
The Loop Step can be accessed in the workflow builder for any workflow that is in edit mode, by simply adding a new step and selecting Loop.
When you select the loop step, you will notice there are two different types of loops - a Repeat Over loop and a Repeat Until loop. Repeat Over loops are typically used when you have a list of items that you want to iterate through, while Repeat Until loops are used when you want the loop to execute a set number of times. Repeat Over loops tend to be more common, so we will focus on those for now, but most of the concepts discussed here carry over for both.
Now, the first thing we will need to configure with our loop is what array from our workflow we want to iterate through. I will continue with the previous example and loop over the URLs that I decided to extract earlier in my workflow.
When I save that configuration, I will be brought inside of my loop within the workflow builder. And from there, I can begin to configure any actions that I want to take against each individual entry of my array.
Adding steps inside of your loop
Steps can be added inside of a loop just as they would be anywhere else within a workflow, but there are some minor nuances that you will want to be aware of! To demonstrate these, we can continue with our URL extraction example. For the purposes of this walkthrough, we can keep it simple and say that we want to gather some intelligence for each URL within our array. So inside of my loop, I will add an action step to fetch a scanned URL report from VirusTotal.
Within this action step, I want to be able to pass an item from my array to the step as an input. To do this, you can use the variable picker and select the variable called {{[“loop_name”].[$item]}}.
In this case the array only contains strings of text (our URLs), but if our array were an array of objects and we wanted to reference a specific key within the object, it would follow the format {{[“loop_name”.[$item].[variable_name]}}.
Once we save this VirusTotal step, our workflow will be set up to receive some text, maybe from a phishing email, parse the text for URLs, and then loop over and look up each URL within VirusTotal individually.
Configuring loop outputs
Now that our loop is configured to run and is performing an action against each entry within the array of URLs, the next thing I want to think about is how to pass data from my loop to the rest of the workflow. For this example in particular, I may want to pass any URLs with malicious analysis results from VirusTotal to the rest of my workflow so I could perform some type of action against them. To get started, I will click on the Configure Loop button in the upper right hand portion of my screen and I will add a loop output.
Since I am trying to configure another list of known bad URLs as an output, I will select the array type of output. I will configure the step to add the URL to this new list if there are any indications of the URL being malicious in its latest analysis.
When I exit my loop, notice that this variable is now available for use in later workflow steps, and I could even loop over my list of known bad URLs if I wanted to!
View loop steps in job details
The final topic to cover as it relates to loops is how to review the results of loops when your workflow executes. Loop steps will appear in the All Outputs section of the Job Details page just as any other step would, but note that you can also select individual loop iterations to see their output. Loop iterations will show you exactly what happened inside the loop for each entry of your array.
Hopefully this overview was helpful, please feel free to add any questions!