How do you de-sanitize your URLs?

for instance if there is a new TAP alert that contains a URL, guess what? it is sanitized, in other words. hxxps://evil[.]co

so how then do you all deal with that?

Also, how do you decode URL encoding?

here is the URL (randomized for privacy)

https://www[.]google[.]com/url?source=web&url=https%3A%2F%2Fwww[.]evil[.]com

There is a URL Decode action in the TAP Plugin, it decodes these via regex parsing.
If you want it directly from TAP you can call their API

https://tap-api-v2.proofpoint.com/v2/url/decode

1 Like

Nice, I don’t see a URL Decode action in the latest TAP plugin though?

There are two separate Proofpoint plugins, and the URL Decode action is on the Proofpoint URL Defense one. We’ve got it in the docs here.

@brandon_mcclure curious if you’ve seen any differences between the TAP url_decode API and our decode plugin that uses regex? I wasn’t aware that they had an endpoint for this.

Thanks Holly. Getting errors, looks like an API call via Python script plugin will be where to go from here. This URL is one where google.com contains another URL, useful for evading controls.

Are you getting errors in configuring the step, or when the step runs? If you have an example of the error, we can take a look and see if it’s a bug we need to fix on our side.

errors out when the step runs. “Failed to parse the URL”. This is the threat url from the URL Defense Alert.

I am doing this pattern now:

  1. decode the URL
  2. regex to search and replace the sanitization characters, e.g. hxxps => https, [.] => .
1 Like

I do where they only put a single slash after the protocol, the regex left it like that when the API call put it back in (I have an Advanced Regex after to add this just in case).
But also, the API includes the TAP cluster, message GUID, and intended recipient which is nice information to have.

2 Likes

@hayden_redd Are you able to share the encoded URL with us? We can try to reproduce and fix if so

can I email it to you, do you have a Rapid7 address?

@hayden_redd Yep, and we appreciate it: jonathan_schipp@rapid7.com

What is your method to call PP API? Do you use a plugin or a python script, etc?

I found the HTTP (formerly REST) Plugin works fine.
Connection uses https://tap-api-v2.proofpoint.com and the header info.
Then Step has the Route of /v2/url/decode and body as {"urls":[]}

1 Like

for the v2/campaign/ids endpoint you have to both use basic auth creds AND service app auth?? The POST endpoints work just fine using only basic auth…

See my response to you asking this in its own thread

@hayden_redd We updated the plugin to fix the issues decoding the URL in the original post.

@brandon_mcclure We are also taking a look at the missing forward-slash in the protocol prefix. E.g. https:/ vs. https://.

2 Likes

We released a new version that fixes the missing forward-slash after decoding that @brandon_mcclure reported.

2 Likes