Agent autoupdate

A few days ago we installed the Insight agent v2.7.22 on Ubuntu 20.4 expecting it to auto-update to the latest version (v4.0.0.1), but it has not updated yet.

How quickly are agents supposed to auto-update, provided they have a working connection to the Insight Platform?

Is the agent unable to autoupdate if it is more than one major version behind?

@nnielsen

Good question. I can answer from my own experience. In the past I deployed agents that were on v3 and it updated the agents automatically within a day or so of v4 being available. I have not tried this with a v2 agent though.

The official documentation from Rapid7 suggests agents above v2 should support automatic updates. Below are some potentially helpful resources. Worst case, you can download v3 through the following link and deploy it to another Ubuntu server. Give it a day or so to see if an auto-update is performed: https://docs.rapid7.com/insight-agent/insight-agent-version-history/

This might help you pinpoint whether the issue is with v2 or the environment. Additional helpful resources are below.

Thanks for chiming in, @Cyb3r
Previously, we have deployed v2.7.22 to Windows Servers and clients, and seen the agent update to the latest supported version within hours to a day, but this is our first install of v2.7.22 after v4.0.0 was released (and possibly our first install of v2.7.22 on Linux).

I also noticed just now that v3.1.2.38 on Windows Server 2012R2 has not updated to the latest agent v3.2.5 supported for R2.

According to documentation, v2.7.22 should able to do auto-updates, but apparently there are caveats regarding agent versions and platform OSs.

I have also noticed the same behavior on Windows Server 2012 R2. However, I’d need to take a look at one of our agents to see what a successful autoupdate event looks like in the logs.

Have you tried looking at the agent.log on the Ubuntu system to see if it is throwing any errors?

/opt/rapid7/ir_agent/components/insight_agent/common/agent.log

I’ll have a look at the Ubuntu log next week, thanks for pointing it out :slight_smile:

I don’t know where to look for updates in the logs, but in the Insight Platform - Agents, each host has the agent version listed with a ‘Last updated’ timestamp. For all of our 2012 servers, it is ‘N/A’.

Insight Agent requirements - operating system support | Insight Agent Documentation says

How OS supportability affects agent functionality

The Insight Agent software receives regular updates (including new features, improvements, and defect fixes) designed to maintain agent performance for all supported OS versions. Running the agent on a supported version ensures that the agent software continues to receive these updates. Rapid7’s Customer Support team can also assist with any questions and troubleshoot any issues that arise with agents installed on supported OS versions.

Reading between the lines, UNsupported OS versions may not get ANY updates, and they may not be automatically updated to the latest available agent for that OS. However, Windows Server 2012 is listed as EOL for Insight Agent Support on 2023-10-31, so the agent should have auto-updated, at least up until that date.

The same holds for Ubuntu 20.04, having EOL for Insight Agent Support on 2030-04-02.

Thanks for pasting this here. You are correct, Windows Server 2012 is under support until Oct 31, 2023. However, Windows Server 2012 R2 went EOL on Apr 10, 2023. The max supported version—according to the documentation—is 3.2.5 which explains why your systems may not have upgraded from v3.1.2.38 to v3.2.5. Double check your assets page to see whether your servers are R2 or non-R2, they are both under different support schedules according to the article.

Unsupported Windows OSes

Correct, that’s what it says - well spotted! I’m very surprised that Rapid7 ended agent support for the ‘successor’ 2012 R2 before it’s predecessor 2012, and well before the OS went out of (free) support from Microsoft. It would be interesting to know the reasoning behind that decision.

Our 2012 R2 servers have Rapid7 agent v3.1.2.38 while our 2012 servers have v4.0.0.1.

Anyways, it does not explain why Ubuntu 20.04 (under OS and agent support) did not autoupdate.

1 Like

I am curious to hear their response as well. If I have time tomorrow, I will try to deploy a v3 agent to one of our Ubuntu 20.04 systems and see how it long it takes to update—and if there are any logs in agent.log that indicate the prerequisite checks.

@nnielsen

Okay, I deployed v2.7.22.6 to one of my test Ubuntu 20.04 machines. Let’s give it a few hours or until end of day for results.

@nnielsen

It has been 24 hours and my system on v2.7.22.6 has not auto updated. I uninstalled the v2 agent and deployed v3.3.6.32 a few hours ago. I looked at the config.json file for both versions and following values were the same across both,

“enable_updates”: true,
“p2p_update_enabled”: true,

Upon checking the logs, it appears my system successfully updated to v4.0.0.1. This further proves that despite the config.json stating updates are set true—I presume this means auto-updating—it did not update from v2 to v4. Perhaps, v2 is not capable of auto-updating due to it being an older release? My system updated without any issues once I redeployed v3 with no changes made to my network in-between deployments.

Additional data from agent.log—Timezone in UTC -500.

2023-11-22 14:36:28,081 [INFO] [agent.agent]: Agent Semantic Version: 3.3.6.32
2023-11-22 15:08:11,473 [INFO] [agent.agent_socket.SMS.139706545456688]: SocketTracker host(us3.endpoint.ingress.rapid7.com:443) jailed(False) status(NSocketStatus.ACTIVE) status updated to ACTIVE.
2023-11-22 15:08:15,467 [INFO] [root]: Removing event listener - <bound method AgentSocket._update_agent_socket_state of <AgentSocket @ 0x7f0aa6098090 Sent: 350391 Recv: 14087>>
2023-11-22 15:13:01,355 [INFO] [agent.agent]: Agent Semantic Version: 4.0.0.1

Based on the data in the logs it took approximately 45m to update and it utilized the following host for its communications us3.endpoint.ingress.rapid7.com:443.

I would recommend uninstalling the v2 agent. Redeploying a v3 agent and then waiting a few hours to see if it updates without making any modifications to UFW or other firewall technologies. If it updates successfully on v3 then that proves something is not functioning correctly on v2. Similarly, you can package the latest agent installer v4.0.0.1 and use that moving forward which seems to work on auto-update. If it still does not update on v3 then make sure your systems can communicate to us3.endpoint.ingress.rapid7.com:443 which appears to be a precursor to the successful agent update events.

Hopefully this helps.

I can shed some light on this!

The reason for this specific case, with Windows Server 2012 R2, is that a native library used by the Agent (an XML parser) stopped supporting the OS in April of this year and led to us stopping our support for the OS as well.
The release notes in April covers this, but not in the greatest detail - and I don’t have any further insight on the specific here, I’m afraid: Insight Agent Release Notes

Another thing to note is that the primary driver in our OS support list is the technologies used by the Agent. Since it’s built using Python and Golang, we follow the support of these languages on the OS level as well.
Golang specifically only supports Windows operating systems as long as they’re supported by Microsoft, and this doesn’t take things like ESU into account.

For updating from version 2.x on Unix, I’ve seen the same in a few environments lately and I’m currently running some lab tests on a few different distributions. I don’t have a clear answer yet, but will try to update this thread with what I find.

1 Like

@Cyb3r - Thank you for testing it thoroughly, and confirming that agent v2.7.22.6 will not autoupdate to the current version (v4.0.0.1), whereas a v3.x (v3.3.6.32) will.
We reinstalled using v4.0.0.1, and we expect autoupdate to work again going forward.

@kneser - Thank you for sharing your knowledge, it is much appreciated :slight_smile:

Windows Server 2012 R2 went EOS from Microsoft together with Windows Server 2012 on 2023-10-10, so discontinuing active agent support on on 2023-04-03 for an OS that was still supported by Microsoft at the time (not ESU) is a very disturbing move.

It is also surprising that the agent autoupdate function cannot update any autoupdate capable agent to the latest available agent supported for the platform. Apparently, autoupdate fails to update if the latest and greatest agent version is not supported on the current OS platform OR if updating will skip over a major version, i.e. v2.x to 4.y. These caveats are important and should be visibly documented.