Agent autoupdate

A few days ago we installed the Insight agent v2.7.22 on Ubuntu 20.4 expecting it to auto-update to the latest version (v4.0.0.1), but it has not updated yet.

How quickly are agents supposed to auto-update, provided they have a working connection to the Insight Platform?

Is the agent unable to autoupdate if it is more than one major version behind?

@nnielsen

Good question. I can answer from my own experience. In the past I deployed agents that were on v3 and it updated the agents automatically within a day or so of v4 being available. I have not tried this with a v2 agent though.

The official documentation from Rapid7 suggests agents above v2 should support automatic updates. Below are some potentially helpful resources. Worst case, you can download v3 through the following link and deploy it to another Ubuntu server. Give it a day or so to see if an auto-update is performed: https://docs.rapid7.com/insight-agent/insight-agent-version-history/

This might help you pinpoint whether the issue is with v2 or the environment. Additional helpful resources are below.

Thanks for chiming in, @Cyb3r
Previously, we have deployed v2.7.22 to Windows Servers and clients, and seen the agent update to the latest supported version within hours to a day, but this is our first install of v2.7.22 after v4.0.0 was released (and possibly our first install of v2.7.22 on Linux).

I also noticed just now that v3.1.2.38 on Windows Server 2012R2 has not updated to the latest agent v3.2.5 supported for R2.

According to documentation, v2.7.22 should able to do auto-updates, but apparently there are caveats regarding agent versions and platform OSs.

I have also noticed the same behavior on Windows Server 2012 R2. However, I’d need to take a look at one of our agents to see what a successful autoupdate event looks like in the logs.

Have you tried looking at the agent.log on the Ubuntu system to see if it is throwing any errors?

/opt/rapid7/ir_agent/components/insight_agent/common/agent.log

I’ll have a look at the Ubuntu log next week, thanks for pointing it out :slight_smile:

I don’t know where to look for updates in the logs, but in the Insight Platform - Agents, each host has the agent version listed with a ‘Last updated’ timestamp. For all of our 2012 servers, it is ‘N/A’.

Insight Agent requirements - operating system support | Insight Agent Documentation says

How OS supportability affects agent functionality

The Insight Agent software receives regular updates (including new features, improvements, and defect fixes) designed to maintain agent performance for all supported OS versions. Running the agent on a supported version ensures that the agent software continues to receive these updates. Rapid7’s Customer Support team can also assist with any questions and troubleshoot any issues that arise with agents installed on supported OS versions.

Reading between the lines, UNsupported OS versions may not get ANY updates, and they may not be automatically updated to the latest available agent for that OS. However, Windows Server 2012 is listed as EOL for Insight Agent Support on 2023-10-31, so the agent should have auto-updated, at least up until that date.

The same holds for Ubuntu 20.04, having EOL for Insight Agent Support on 2030-04-02.

Thanks for pasting this here. You are correct, Windows Server 2012 is under support until Oct 31, 2023. However, Windows Server 2012 R2 went EOL on Apr 10, 2023. The max supported version—according to the documentation—is 3.2.5 which explains why your systems may not have upgraded from v3.1.2.38 to v3.2.5. Double check your assets page to see whether your servers are R2 or non-R2, they are both under different support schedules according to the article.

Unsupported Windows OSes

Correct, that’s what it says - well spotted! I’m very surprised that Rapid7 ended agent support for the ‘successor’ 2012 R2 before it’s predecessor 2012, and well before the OS went out of (free) support from Microsoft. It would be interesting to know the reasoning behind that decision.

Our 2012 R2 servers have Rapid7 agent v3.1.2.38 while our 2012 servers have v4.0.0.1.

Anyways, it does not explain why Ubuntu 20.04 (under OS and agent support) did not autoupdate.

1 Like

I am curious to hear their response as well. If I have time tomorrow, I will try to deploy a v3 agent to one of our Ubuntu 20.04 systems and see how it long it takes to update—and if there are any logs in agent.log that indicate the prerequisite checks.

@nnielsen

Okay, I deployed v2.7.22.6 to one of my test Ubuntu 20.04 machines. Let’s give it a few hours or until end of day for results.

@nnielsen

It has been 24 hours and my system on v2.7.22.6 has not auto updated. I uninstalled the v2 agent and deployed v3.3.6.32 a few hours ago. I looked at the config.json file for both versions and following values were the same across both,

“enable_updates”: true,
“p2p_update_enabled”: true,

Upon checking the logs, it appears my system successfully updated to v4.0.0.1. This further proves that despite the config.json stating updates are set true—I presume this means auto-updating—it did not update from v2 to v4. Perhaps, v2 is not capable of auto-updating due to it being an older release? My system updated without any issues once I redeployed v3 with no changes made to my network in-between deployments.

Additional data from agent.log—Timezone in UTC -500.

2023-11-22 14:36:28,081 [INFO] [agent.agent]: Agent Semantic Version: 3.3.6.32
2023-11-22 15:08:11,473 [INFO] [agent.agent_socket.SMS.139706545456688]: SocketTracker host(us3.endpoint.ingress.rapid7.com:443) jailed(False) status(NSocketStatus.ACTIVE) status updated to ACTIVE.
2023-11-22 15:08:15,467 [INFO] [root]: Removing event listener - <bound method AgentSocket._update_agent_socket_state of <AgentSocket @ 0x7f0aa6098090 Sent: 350391 Recv: 14087>>
2023-11-22 15:13:01,355 [INFO] [agent.agent]: Agent Semantic Version: 4.0.0.1

Based on the data in the logs it took approximately 45m to update and it utilized the following host for its communications us3.endpoint.ingress.rapid7.com:443.

I would recommend uninstalling the v2 agent. Redeploying a v3 agent and then waiting a few hours to see if it updates without making any modifications to UFW or other firewall technologies. If it updates successfully on v3 then that proves something is not functioning correctly on v2. Similarly, you can package the latest agent installer v4.0.0.1 and use that moving forward which seems to work on auto-update. If it still does not update on v3 then make sure your systems can communicate to us3.endpoint.ingress.rapid7.com:443 which appears to be a precursor to the successful agent update events.

Hopefully this helps.

I can shed some light on this!

The reason for this specific case, with Windows Server 2012 R2, is that a native library used by the Agent (an XML parser) stopped supporting the OS in April of this year and led to us stopping our support for the OS as well.
The release notes in April covers this, but not in the greatest detail - and I don’t have any further insight on the specific here, I’m afraid: Insight Agent Release Notes

Another thing to note is that the primary driver in our OS support list is the technologies used by the Agent. Since it’s built using Python and Golang, we follow the support of these languages on the OS level as well.
Golang specifically only supports Windows operating systems as long as they’re supported by Microsoft, and this doesn’t take things like ESU into account.

For updating from version 2.x on Unix, I’ve seen the same in a few environments lately and I’m currently running some lab tests on a few different distributions. I don’t have a clear answer yet, but will try to update this thread with what I find.

1 Like

@Cyb3r - Thank you for testing it thoroughly, and confirming that agent v2.7.22.6 will not autoupdate to the current version (v4.0.0.1), whereas a v3.x (v3.3.6.32) will.
We reinstalled using v4.0.0.1, and we expect autoupdate to work again going forward.

@kneser - Thank you for sharing your knowledge, it is much appreciated :slight_smile:

Windows Server 2012 R2 went EOS from Microsoft together with Windows Server 2012 on 2023-10-10, so discontinuing active agent support on on 2023-04-03 for an OS that was still supported by Microsoft at the time (not ESU) is a very disturbing move.

It is also surprising that the agent autoupdate function cannot update any autoupdate capable agent to the latest available agent supported for the platform. Apparently, autoupdate fails to update if the latest and greatest agent version is not supported on the current OS platform OR if updating will skip over a major version, i.e. v2.x to 4.y. These caveats are important and should be visibly documented.

1 Like

Hey! A few quick updates here, sorry it’s a little late - busy couple of weeks :slight_smile:

Windows Server 2012 R2 EOL:
Unfortunately, I don’t have any further context around the EOL side of things. Ultimately it’s a decision made by the Product Managers for the Agent, and it mostly revolves around prioritising support for currently in-date operating systems.
At a guess, and only a guess (this is not authoritative information), the work required to keep supporting Server 2012 R2 might have taken too much resources or perhaps could not have been completed prior to the OS ending general support from Microsoft’s side.

Generally I’d suggest getting in touch with your Customer Success Manager if you want to dig deeper into this decision. I know a few customers have opted to go via this route, but I don’t know the outcomes there.

Agent updates on version 2.7.x:
As for the autoupdate issues with older Agent versions, I’ve got an answer on this. Operating System returns were introduces in bootstrap version 2.3.0.3, which shipped with Agent version 3.0.8. Prior to these versions, bootstrap doesn’t return the OS of the asset the Agent is running on.

Agent updates are handled via the bootstrap, in which our Platform pushes Agent updates in accordance with organisation settings and such. As part of this, we have a service that prevents updates to certain operating systems that are no longer supported on newer versions. This service requires there to be an OS return; if there is a null value, it won’t push any updates to that bootstrap.

So, given that you’re running Agents below version 3.0.8 in which the OS return was introduced, the service would not push any updates out to them on account of no OS return.

From chatting to dev about this, there’s no great way of adjusting the service to allow for updates without causing further problems as we have no way of knowing which operating systems they run on from a bootstrap perspective.
As an example of why removing the rule is not ideal, if we did so then any Agents running on Windows Server 2008 or Windows 7 SPO1, with a last-supported version of 2.7.22, would update to 3.0.8 which would break the installation and the Agent would not function anymore.

Our approach currently is to prevent these situations, and we currently recommend customers manually upgrade Agents below 3.0.8 that are supported up to or above this version. It’s an ongoing discussion to see if we can find a way around the limitations here without unintentionally breaking things, but I’d go with manual updates as I don’t expect us to make any changes here in the foreseeable future given it’s near the end of the year.

And for reference, the last supported versions for out-of-support operating systems are listed here:

2 Likes

Great information, thank you!