Hi everyone. I’m currently running Windows Server 2019 scan engines on 6.6.185 and since early this week I;m seeing my scan engines run out of RAM when running our normal network vulnerability scans. The scans we ran on Monday ran without issue but since Tuesday we have seen almost all the scans paunse and fail to complete. I’m seeing the scan engines using almost all of their RAM and failing to free up the RAM even after the scan jobs are cancelled. Is anyone else seeing this issue? I have a case open with support but haven’t gotten any replies on that. I have tried allocating more RAM to the scanners (increasing the from 16gb to 24) but the scanner process just continues using more RAM until it completely exhausts the server and the scan job pauses. This is pretty much a work stoppage so far as our ability to complete scans.
Peter, same issue. The XDR on the server is consuming a ton of memory. We are going to test pausing XDR this weekend. Will let you know.
I’ve been scheduling my scan engine servers to reboot every night at 2am as we don’t have any scans running at that time and that seems to be mitigating the issue for now. I can make the issue occur if I run a big scan job against a scan pool that has already run a large job and not been rebooted. After the reboot, memory usage returned to a normal baseline but will increase when the first scan job is run and does not return to the baseline level when the scan completes. Of 16gb mine hit 11-12gb used after the first scan job and stay at that level. So if I try to run another large scan using the scan scanner pool the severs only have 3gb of RAM free for the second scan job out of 16 total.
Yes, I have also encountered the same issue on some of my 31 scanners. They are all running on RHEL and have between 12 to 16GB of RAM, depending on the size of the scan sites. Yesterday, one of my scanners ran out of memory. I had to restart the server and the scan.
We will also set up a scheduling to reboot the scan engines on a daily basis.
Being a Java based app the scan engine will use up a lot of memory. On thing to check is in the console settings under Administration → Console → Platform Data Synchronization → Scan Engines
enable Retrieve incremental scan results from distributed engines
seems to have helped on my end
We get the same issue anytime Crowdstrike updates the agent on the scan servers. We ended up moving our R7 servers into a special Crowdstrike update policy and once every couple of months we manually update the CS agent. Then for a day or so I’ll get stuck with paused scans and reboots of the R7 servers till everything updates and settles down.