How would you advise reading a csv for MD5 hash matches from a list of hashes?

InsightConnect
1

I would like to send an email with a csv attached and have icon check it for matches from a list of md5 hashes.

read email > csv to json > pattern match?

2

@hayden_redd You don’t have to convert the CSV to JSON, you can pass it directly to the ExtractIt step or the Pattern Match step to pull out the MD5 data (assuming there’s more than MD5’s in it) from the CSV, and then you could run the Grep plugin to match against the list you want to check them against. Depending on the format, you will likely want to compare array of the MD5s from the CSV against an array of your list of MD5 hashes. Utility plugins like String and Type Convertor may help get that data into the correct format.

We also have CSV plugin if you want to get more granular with the CSV data and pull out specific columns for extraction, say for example, there’s multiple columns and column 5 has the MD5s. In this case, it’s better to just grab the column than regex across them with Pattern Match or ExtractIt.

1 Like
3

why do I keep getting

UnicodeDecodeError: ‘utf-8’ codec can’t decode bytes in position 15-16: invalid continuation byte

when im using the Filter Bytes action from the csv plugin

what is the right input to use here?

{{[“new email”].[icon_email].[flattened_attached_files].[0].[content]}}

4

What’s the value for:
{{[“new email”].[icon_email].[flattened_attached_files].[0].[content_type]}}

I’ve seen various email servers do strange things with attachments.

5

“content_type”:“image/png”

6

it was an .xlsx ugh, trying now as an actual csv ha

1 Like
7

Heh, I’ve done that. What keyed me off was the line 15-16 comment. If something is going to fail to decode it usually does it in the 1st or 2nd character.

1 Like
8

We shipped a new action to the Type Converter plugin called Array Match that will help with use-cases like this where you have two lists (arrays) and want to find matches between them e.g. MD5 hash in one list is also present in another.

9

@jon_schipp what would you recommend for extracting emails from a csv? I have been trying to use the extractit plugin as suggested here. But the plugin keeps telling me NO :joy:

rapid7/ExtractIt:3.0.6. Step name: email_extractor
An error occurred during plugin execution!

The type of the provided file is not supported. Supported file types are text/binary, such as: PDF, DOCX, PPTX, XLSX, ODT, ODP, ODF, TXT, ZIP
Traceback (most recent call last):
File “/usr/local/lib/python3.8/site-packages/extractit_rapid7_plugin-3.0.6-py3.8.egg/icon_extractit/util/extractor.py”, line 107, in extract
base64.b64decode(provided_file.encode(DEFAULT_ENCODING)).decode(DEFAULT_ENCODING)
UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x89 in position 1: invalid start byte