CloudTrail SQS or API?

Hi, looking for some advice on the best option to send CloudTrail logs to InsightIDR. Per the documentation, there are 2 options. Configure an SQS queue or API.
https://docs.rapid7.com/insightidr/aws-cloudtrail-sqs/
https://docs.rapid7.com/insightidr/aws-cloudtrail-api

What are the pros and cons of each option? thanks in advance for any pointers.

Hi @jmathison the SQS option is much better for higher volume cloudtrail event sources. The way this functions is that it is listening to the SQS queue to prompt it for new events to download. So effectively a “push” method to tell the collector to download more events. It’s much more efficient.

The API method is a pull method, the collector polls for new events and its not as efficient for high volume cloud trails.

When you configure using the AWS CloudTrail API, InsightIDR queries the API periodically to see what has changed and then downloads the logs. When you configure with SQS, InsightIDR receives messages through an SQS notification when the S3 logs are created and ready to download. Both methods work well, however when you use the SQS method, InsightIDR is often able to gather logs faster because messages are created as soon as the S3 files are ready. For instructions on how to configure AWS CloudTrail with SQS, see the AWS CloudTrail SQS documentation.

David

2 Likes

Thanks for the info David!

1 Like