Proofpoint_url_defense Not Decoding correctly

InsightConnect Plugins
1

I have this Action in a workflow and it isn’t correctly decoding the URL.

My input is coming in looking like this
encoded_url:https://urldefense.com/v3/__https://www.myclientline.net/publicS/publicServ/ClientLineEnrollment/complete.jsp__;!!MiZrGOEI0vA!NohzERuuhFAIty3SEYDFKwxCVMolJEd8Nz9rLcG4K7F8vlC_LdVlbApnP4DYHhi0l4wPQz_sD0PvfecXmeU0yUsS$

and the output is looking like this
* $success:true
* decoded_url:https://urldefense.com/v3///https://www.myclientline.net/publicS/publicServ/ClientLineEnrollment/complete.jsp//

I was expecting this
decodedUrl:
https://www.myclientline.net/publicS/publicServ/ClientLineEnrollment/complete.jsp

The docker logs show
[2020-03-24 18:22:08 +0000] [1] [INFO] Starting gunicorn 19.7.1
[2020-03-24 18:22:08 +0000] [1] [INFO] Listening at: http://0.0.0.0:10001 (1)
[2020-03-24 18:22:08 +0000] [1] [INFO] Using worker: threads
[2020-03-24 18:22:08 +0000] [10] [INFO] Booting worker with pid: 10
rapid7/Proofpoint URL Defense:1.0.1. Step name: url_decode
URL has been decoded
rapid7/Proofpoint URL Defense:1.0.1. Step name: url_decode
URL has been decoded

2

That doesn’t look right. I think the input to the plugin is wrong somehow. My guess is this needs stripped off “;!!MiZrGOEI0vA!NohzERuuhFAIty3SEYDFKwxCVMolJEd8Nz9rLcG4K7F8vlC_LdVlbApnP4DYHhi0l4wPQz_sD0PvfecXmeU0yUsS$”

leaving

https://urldefense.com/v3/__https://www.myclientline.net/publicS/publicServ/ClientLineEnrollment/complete.jsp__

Can you split on ( ; ) and take the first element maybe?

3

I can try, but that was the URL from the array that ExtractIt provided.
I also passed this URL to the ProofPoint TAP API and it returned the string that I expected.

Upon some digging into the code I see that this is using ppdecode from git, this only looks to support ProofPoint v1&v2 when the URL I’m passing is v3. It makes since when I look at the code on github it is just doing string replacements that match the return that I’m seeing. This might need an upgrade to see if the URL is v3 and if it is do a POST call to ‘https://tap-api-v2.proofpoint.com/v2/url/decode’ with a payload of "{“urls”:[]}

I don’t know if this will work with v1&v2 URLs if you have one to test maybe we can make a new Plugin called proofpoint_url_defense_api or something that just does the requests call?

4

Example payload
{
“urls”: [
“https://urldefense.com/v3/https://www.myclientline.net/publicS/publicServ/ClientLineEnrollment/complete.jsp;!!MiZrGOEI0vA!NohzERuuhFAIty3SEYDFKwxCVMolJEd8Nz9rLcG4K7F8vlC_LdVlbApnP4DYHhi0l4wPQz_sD0PvfecXmeU0yUsS$”
]
}

5

I see I see…Let me talk to the team about this one. We can get a fix in for this one way or another.

6

Just an update: We’ve got a fix that should be in tonight or Monday. Doing some last bits of testing and QA.

7

Sure, just as I finished writing my plugin :wink:
I was just playing around anyways trying to learn the icon stuff, thought a simple RESTful call would be a great way to get my feet wet

8

Fixed it.

https://extensions.rapid7.com/extension/proofpoint_url_defense/v1.1.0

Try that out and let us know if there are any problems.

9

Also, let us know how writing a plugin goes. If there’s anything there we can improve, we’d love to hear about it.

10

Almost, it is working correctly with a URL Defense Encoded URL.
But when I pass an unencryptd one like http://www.google.com it fails with a python error.

rapid7/Proofpoint URL Defense:1.1.0. Step name: url_decode
Unexpected issue occurred decoding URL
Error parsing URL

I got it returning a parsed error, but I didn’t update the success variable correctly

  • $success:true
  • encodedUrl:http://www.google.com
  • error:Invalid URL - encoded URL is not a URL Defense URL
  • success:false

plugin writing is going smooth, I can tell that I am barely touching the tip of the iceberg though and there will be a lot more that I can do with it.

11

I’m running it through our unit tests, and I’d expect the plugin to fail like that. It’s a malformed Proof Point URL. What would you expect the behavior to be in the case of a malformed URL?

12

One more thing, I checked with one of our archs about this. We have no control over $success unfortunately. That really just tells us that the plugin did indeed return something and didn’t throw any errors.

13

So I’ve been back and forth about this over the weekend.
What I’ve come up with might just be what my company wants to see so I don’t know if it would be a globally agreed way.
in my example I’m thinking about changing ‘success’ to ‘decryptionSuccess’ and if this is false, then set decryptedUrl equal to encryptedUrl. This way $success is correct in that the plugin completed successfully, but the decryption failed. This will allow the Workflow to continue with the variable decryptedUrl both if it could decrypt or if it couldn’t. I could also just do a Workflow check to see if it is a valid encrypted URL first before sending it to the action. I’m back and forth on both options…

14

Yea, that’s reasonable, and how I was thinking about approaching it anyway. We’ve got other plugins that behave in the same way on case of failure. We still give output, but put an error message with it and/or a boolean of some sort that indicates failure.

Ok…give me a little bit and I’ll have a fix in for you.

15

Try this one (1.2.0):
https://extensions.rapid7.com/extension/proofpoint_url_defense

1 Like
16

So close…
Everything looks good, except the successfully decoded URL is missing a / in the https:// for some reason

  • encodedUrl:https://urldefense.com/v3/https:/www.myclientline.net/publicS/publicServ/ClientLineEnrollment/complete.jsp;!!MiZrGOEI0vA!IYcXCO-F4L4ITR_Xurj6OmRzulrKGbrPp8v5Jvk6lSPhRjPRH0eNq9UZgzrZARHUbI9S2ZlB8rSZNT0_sKMOKxeM$
  • $success:true
  • decoded:true
  • decoded_url:https:/www.myclientline.net/publicS/publicServ/ClientLineEnrollment/complete.jsp

It looks like they do this in the encoded URL

17

If you look at your input there (encodedURL) it only has one slash. If that’s how it comes in, the plug in (IMHO) is doing the right thing and spitting it back out with one slash. I’m not sure we can fix malformed URLs in this plugin.

Did Proof Point screw that up when it was encoded?

18

When I passed this URL to the API it came back correctly formatted so I’m wounding if that was intended in v3 to obfuscate the URL

19

I don’t think that’s it…if we go back to your original post, I get this…

encoded_url:https://urldefense.com/v3/https://www.myclientline.net/publicS/publicServ/ClientLineEnrollment/complete.jsp;!!MiZrGOEI0vA!NohzERuuhFAIty3SEYDFKwxCVMolJEd8Nz9rLcG4K7F8vlC_LdVlbApnP4DYHhi0l4wPQz_sD0PvfecXmeU0yUsS$

That one has two slashes in it. So I don’t think it’s URL Defense removing that slash.

Was it encoded in a workflow? Or did you get it from ExtractIT or something like that?

20

So I’m stumped…
I see that in the first note, but when I go to that email that triggered it there was only 1 slash. I looked at the Job though that ran from that trigger and the input to ExtractIT has 2 slashes.
I looked through all the emails in that inbox (it is new so there were mostly just the test one I sent to it) and they all have 1 slash in Outlook. I kept Replying to all in that inbox so I kept using that same email to re-trigger the Workflow. I’m not sure what changed.
what about doing something like this just before returning the object?

decoded_url_split = decoded_url.split(':/')
decoded_url_protocol = decoded_url_split[0]
decoded_url_protocol_len = len(decoded_url_protocol) + 2
decoded_url_destination = decoded_url[decoded_url_protocol_len:]
if decoded_url_destination[0] == '/':
    decoded_url_destination = decoded_url_destination[1:]
decoded_url = "{}://{}".format(decoded_url_protocol,decoded_url_destination)