Hi all,
Got a strange issue that’s been plaguing our internal platform for a couple of weeks now.
I’ve newly joined the company and setting myself some goals to remediate any issues that have been present.
Recently, I have been informed the Rapid 7 agent’s seem to be getting killed / stopped across our internal devices.
I’m currently dealing with support, however figured it’d be best to bring this to the community as well in-case anyone has experienced the same issue before.
I’ve been running procmon throughout the days and analysing the agent.log file and the follow keeps cropping up:
2022-08-31 10:34:20,236 [DEBUG] [agent.agent_id_reporter]: ID Reporter socket timed out after 2 seconds
2022-08-31 10:34:20,807 [WARNING] [agent.agent]: Shutting down agent…
2022-08-31 10:34:20,807 [INFO] [agent.agent_beacon]: Shutting down beacon service
2022-08-31 10:34:20,807 [DEBUG] [agent.agent_proxy_config.AgentProxyConfigManager]: Shutting down <agent.agent_proxy_config.AgentProxyConfigManager object at 0x000002B981C206A0>
2022-08-31 10:34:20,807 [DEBUG] [agent.agent_beacon]: Interrupt ignored, not in sending state
2022-08-31 10:34:20,807 [INFO] [agent.message_bus]: Shutting down message service
2022-08-31 10:34:20,807 [WARNING] [agent.message_bus]: Hard shutting down msgsock!
2022-08-31 10:34:20,807 [DEBUG] [agent.message_bus.sndr.14440]: Got dead pill - shutting down job sender
2022-08-31 10:34:20,807 [DEBUG] [agent.message_bus]: Interrupt ignored, not in sending state
2022-08-31 10:34:20,807 [INFO] [agent.agent_socket.HAS.2995795965984]: Requesting shutdown: JobMessageSender
2022-08-31 10:34:20,807 [DEBUG] [agent.agent_socket.AGS.2995795913600.msgpack://127.0.0.1:0]: Resetting ttl to system default: 128
2022-08-31 10:34:20,807 [INFO] [agent.agent_socket.HAS.2995795965984]: Clearing out old singleton instance 2995795965984:2995795965984
2022-08-31 10:34:20,807 [INFO] [agent.event_tracker]: Shutting down network interface tracker service
2022-08-31 10:34:20,807 [DEBUG] [agent.message_bus.rcvr.14876]: job socket closed - removing from read_sockets list
2022-08-31 10:34:20,807 [INFO] [agent.agent_socket.SMS.2995795966992]: Shutting down
2022-08-31 10:34:20,808 [INFO] [agent.event_engine_manager]: Requesting shutdown of the event engine!
2022-08-31 10:34:20,808 [WARNING] [agent.message_bus.rcvr.14876]: Shutting down
2022-08-31 10:34:20,808 [DEBUG] [agent.agent_socket.SMS.2995795966992]: SMT.2995795994752.win-r3db8vc15o3:5508 - shutting down
2022-08-31 10:34:20,808 [INFO] [agent.event_engine_manager]: Checking if event engine is supported
2022-08-31 10:34:20,808 [DEBUG] [agent.agent_socket.SMS.2995795966992]: SMT.2995795995808.eu.endpoint.ingress.rapid7.com:443 - shutting down
2022-08-31 10:34:20,808 [INFO] [agent.agent_socket.SMT.2995795994752.win-r3db8vc15o3:5508]: *** jail_socket - received shutdown event
2022-08-31 10:34:20,808 [INFO] [agent.event_engine_manager]: Rebuild not set - shutting down
2022-08-31 10:34:20,808 [DEBUG] [agent.agent_socket.SMS.2995795966992]: Clearing our server list
2022-08-31 10:34:20,808 [DEBUG] [agent.agent_socket.SMT.2995795995808.eu.endpoint.ingress.rapid7.com:443]: *** set_wait_good - received shutdown event
2022-08-31 10:34:20,809 [DEBUG] [agent.agent_socket.SMT.2995795994752.win-r3db8vc15o3:5508]: Socket tracker completed
2022-08-31 10:34:20,809 [DEBUG] [agent.agent_socket.SMS.2995795966992]: SocketTracker-win-r3db8vc15o3:5508 - waiting for completion
2022-08-31 10:34:20,810 [DEBUG] [agent.agent_socket.AGS.2995795967856.cmsgpack://eu.endpoint.ingress.rapid7.com:443]: Resetting ttl to system default: 128
2022-08-31 10:34:20,811 [DEBUG] [agent.agent_socket.SMS.2995795966992]: SocketTracker-win-r3db8vc15o3:5508 - has completed
2022-08-31 10:34:20,812 [DEBUG] [agent.agent_socket.SMS.2995795966992]: SocketTracker-eu.endpoint.ingress.rapid7.com:443 - waiting for completion
2022-08-31 10:34:20,812 [DEBUG] [agent.agent_socket.SMT.2995795995808.eu.endpoint.ingress.rapid7.com:443]: Socket tracker completed
2022-08-31 10:34:20,813 [DEBUG] [agent.agent_socket.SMS.2995795966992]: SocketTracker-eu.endpoint.ingress.rapid7.com:443 - has completed
2022-08-31 10:34:20,813 [INFO] [agent.agent_socket.SMS.2995795966992]: SmartSocket has be shutdown and reset
2022-08-31 10:34:20,814 [DEBUG] [agent.agent_socket.HAS.2995795965984]: Unregistering and shutting down any sockets in the list, from the event manager.
2022-08-31 10:34:20,814 [DEBUG] [root]: Un-Registering new handlers from - cmsgpack://win-r3db8vc15o3:5508
2022-08-31 10:34:20,814 [INFO] [root]: Removing event listener - <bound method AgentSocket._update_agent_socket_state of <AgentSocket @ 0x2b9835abc40 Sent: 0 Recv: 0>>
2022-08-31 10:34:20,814 [DEBUG] [root]: Un-Registering new handlers from - cmsgpack://eu.endpoint.ingress.rapid7.com:443
2022-08-31 10:34:20,815 [INFO] [root]: Removing event listener - <bound method AgentSocket._update_agent_socket_state of <AgentSocket @ 0x2b9835abf70 Sent: 22328 Recv: 3469>>
2022-08-31 10:34:20,815 [DEBUG] [agent.agent_socket.HAS.2995795965984]: Unregistering and shutting down any jailed sockets from the event manager.
2022-08-31 10:34:20,815 [INFO] [agent.message_bus.sndr.14440]: Initial queue size 1
2022-08-31 10:34:20,831 [WARNING] [agent.jobs.windows.ui_realtime.1384]: event_engine_parser - connection error: [WinError 10054] An existing connection was forcibly closed by the remote host
2022-08-31 10:34:20,832 [ERROR] [agent.jobs.windows.ui_realtime.1384]: Engine sent dead pill - shutting down
2022-08-31 10:34:20,832 [INFO] [agent.jobs.windows.ui_realtime.1384]: Event engine receiver shutdown
2022-08-31 10:34:20,833 [DEBUG] [agent.jobs.windows.ui_realtime.1384]: ProcessWatcherThread setting event code to 1
2022-08-31 10:34:20,833 [INFO] [agent.jobs.windows.ui_realtime.1384]: Unregistering from event - 1
2022-08-31 10:34:20,834 [INFO] [agent.event_engine_manager]: Checking if event engine is supported
2022-08-31 10:34:20,833 [WARNING] [agent.jobs.windows.ui_realtime.1384.EventLogMonitor]: Shutdown event - EventLogMonitor periodic scan
2022-08-31 10:34:20,833 [WARNING] [agent.jobs.windows.ui_realtime.1384]: Shutdown event - monitor
2022-08-31 10:34:20,834 [DEBUG] [agent.jobs.windows.ui_realtime.1384.EventProcessingTracker]: check_external_flags_periodically thread terminated.
2022-08-31 10:34:20,834 [WARNING] [agent.platforms.windows.eventlog_api.EvtxLogFollower]: Shutting down
2022-08-31 10:34:20,834 [INFO] [agent.jobs.windows.ui_realtime.1384]: Waiting for ProcessMonitor to complete
2022-08-31 10:34:20,835 [INFO] [agent.jobs.windows.ui_realtime.1384.ProcessMonitor]: Waiting on ProcessWatcherThread
2022-08-31 10:34:20,908 [WARNING] [agent.jobs.windows.ui_realtime.1384]: Shutdown event is set - exit status non-zero
2022-08-31 10:34:21,016 [INFO] [agent.jobs.tem_realtime.19980]: Shutdown requested from signal handler. signum: 2
2022-08-31 10:34:21,016 [INFO] [agent.jobs.windows.ui_realtime.1384]: agent.jobs.windows.ui_realtime - exiting
2022-08-31 10:34:21,016 [INFO] [agent.jobs.network_monitor.2232]: Shutdown requested from signal handler. signum: 2
2022-08-31 10:34:21,017 [WARNING] [agent.jobs.tem_realtime.19980]: Received signum: 2 - setting event
2022-08-31 10:34:21,017 [DEBUG] [agent.jobs.tem_realtime.19980]: MainThread setting event code to 2
2022-08-31 10:34:21,017 [DEBUG] [agent.jobs.windows.ui_realtime.1384]: MainThread setting event code to 1 - interrupting comms (if running)
2022-08-31 10:34:21,017 [WARNING] [agent.jobs.network_monitor.2232]: Received signum: 2 - setting event
2022-08-31 10:34:21,018 [WARNING] [agent.jobs.windows.tem_realtime.TemRealtime]: ** Shutdown requested
2022-08-31 10:34:21,018 [DEBUG] [agent.agent_socket.AGS.2520280429280.msgpack://127.0.0.1:64923]: Resetting ttl to system default: 128
2022-08-31 10:34:21,019 [INFO] [agent.job_manager]: Received metric info from job: agent.jobs.le_realtime - uuid: 8d479879-817c-4dc0-a3f2-8e5189401e67 - {‘processInfo’: {‘pid’: 2456, ‘ppid’: 20492, ‘cpuUtil’: ‘0.00’, ‘coreCount’: 16, ‘memUsage’: {‘percent’: ‘0.17’, ‘rss’: ‘55.73 MB’, ‘private’: ‘40.76 MB’}, ‘ioInfo’: {‘readCount’: 1326, ‘readBytes’: 7261744, ‘writeCount’: 81, ‘writeBytes’: 29948}}}
2022-08-31 10:34:21,019 [WARNING] [agent.job_manager]: Job thread has exited. Ensuring all jobs have been stopped
2022-08-31 10:34:21,019 [INFO] [agent.jobs.le_realtime.2456]: Shutdown requested from signal handler. signum: 2
2022-08-31 10:34:21,019 [DEBUG] [agent.jobs.network_monitor.2232]: MainThread setting event code to 2
2022-08-31 10:34:21,020 [DEBUG] [agent.job_manager]: Requested stopJob on e07e2fba-1613-455d-9147-8a92c034226c, marking as 4
2022-08-31 10:34:21,020 [DEBUG] [agent.agent_beacon]: Appending system information to beacon
2022-08-31 10:34:21,020 [DEBUG] [agent.jobs.windows.ui_realtime.1384]: Running cleanup
2022-08-31 10:34:21,019 [INFO] [agent.jobs.tem_realtime.IntervalTriggerThread]: Shutdown requested. Exiting trigger
2022-08-31 10:34:21,020 [WARNING] [agent.jobs.le_realtime.2456]: Received signum: 2 - setting event
2022-08-31 10:34:21,021 [INFO] [agent.job_manager]: Sending shutdown request signal: 2 to job: agent.jobs.windows.ui_realtime - uuid: e07e2fba-1613-455d-9147-8a92c034226c
2022-08-31 10:34:21,021 [DEBUG] [agent.job_manager]: Told job: agent.jobs.windows.ui_realtime - uuid: e07e2fba-1613-455d-9147-8a92c034226c to shutdown
2022-08-31 10:34:21,021 [DEBUG] [agent.jobs.windows.ui_realtime.1384]: Waiting for agent.jobs.windows.ui_realtime to finish…
2022-08-31 10:34:21,021 [DEBUG] [agent.jobs.le_realtime.2456]: MainThread setting event code to 2
2022-08-31 10:34:21,022 [INFO] [agent.jobs.le_realtime]: Initializing configured log from C:\Program Files\Rapid7\Insight Agent\components\insight_agent\common\config\logging.json
2022-08-31 10:34:21,019 [WARNING] [agent.jobs.tem_realtime.TemEventTriggerThread]: ** Shutdown requested
2022-08-31 10:34:21,021 [INFO] [agent.jobs.windows.ui_realtime.1384]: ** Shutdown event - ExceptionWorker
2022-08-31 10:34:21,023 [INFO] [agent.jobs.le_realtime]: Initializing log transport
2022-08-31 10:34:21,019 [INFO] [agent.jobs.tem_realtime.IntervalTriggerThread]: Shutdown requested. Exiting trigger
2022-08-31 10:34:21,022 [DEBUG] [agent.jobs.windows.ui_realtime.1384]: Waiting for ExceptionWorker to finish…
2022-08-31 10:34:21,023 [INFO] [agent.jobs.le_realtime]: Initializing log monitor
2022-08-31 10:34:21,019 [INFO] [agent.jobs.tem_realtime.IntervalTriggerThread]: Shutdown requested. Exiting trigger
2022-08-31 10:34:21,023 [INFO] [agent.jobs.le_realtime]: Shutting down
2022-08-31 10:34:21,024 [INFO] [agent.jobs.le_realtime]: Shut down
2022-08-31 10:34:21,019 [INFO] [agent.jobs.tem_realtime.IntervalTriggerThread]: Shutdown requested. Exiting trigger
2022-08-31 10:34:21,024 [INFO] [agent.jobs.le_realtime]: terminated
2022-08-31 10:34:21,019 [INFO] [agent.jobs.tem_realtime.IntervalTriggerThread]: Shutdown requested. Exiting trigger
2022-08-31 10:34:21,024 [INFO] [agent.jobs.le_realtime.2456]: agent.jobs.le_realtime - exiting
2022-08-31 10:34:21,020 [INFO] [agent.jobs.tem_realtime.IntervalTriggerThread]: Shutdown requested. Exiting trigger
2022-08-31 10:34:21,025 [DEBUG] [agent.jobs.le_realtime.2456]: MainThread setting event code to 2 - interrupting comms (if running)
2022-08-31 10:34:21,020 [INFO] [agent.jobs.tem_realtime.IntervalTriggerThread]: Shutdown requested. Exiting trigger
2022-08-31 10:34:21,025 [DEBUG] [agent.agent_socket.AGS.2026693149264.msgpack://127.0.0.1:64923]: Resetting ttl to system default: 128
2022-08-31 10:34:21,020 [INFO] [agent.jobs.tem_realtime.IntervalTriggerThread]: Shutdown requested. Exiting trigger
2022-08-31 10:34:21,025 [INFO] [agent.jobs.windows.tem_realtime.TemRealtime]: *** TEM Realtime job completed successfully
2022-08-31 10:34:21,026 [DEBUG] [agent.jobs.le_realtime.2456]: Running cleanup
2022-08-31 10:34:21,026 [INFO] [agent.jobs.tem_realtime.19980]: agent.jobs.tem_realtime - exiting
2022-08-31 10:34:21,026 [DEBUG] [agent.jobs.le_realtime.2456]: Waiting for MessageWorker to finish…
2022-08-31 10:34:21,026 [DEBUG] [agent.jobs.tem_realtime.19980]: MainThread setting event code to 2 - interrupting comms (if running)
2022-08-31 10:34:21,026 [DEBUG] [agent.agent_socket.AGS.2870734787152.msgpack://127.0.0.1:64923]: Resetting ttl to system default: 128
2022-08-31 10:34:21,026 [INFO] [agent.jobs.le_realtime.2456]: ** Shutdown event - ExceptionWorker
2022-08-31 10:34:21,027 [DEBUG] [agent.jobs.tem_realtime.19980]: Running cleanup
2022-08-31 10:34:21,026 [DEBUG] [agent.jobs.le_realtime.2456]: Waiting for ExceptionWorker to finish…
2022-08-31 10:34:21,027 [DEBUG] [agent.jobs.tem_realtime.19980]: Closing OOB queues
2022-08-31 10:34:21,028 [DEBUG] [agent.jobs.tem_realtime.19980]: Deleted working directory: C:\Program Files\Rapid7\Insight Agent\components\insight_agent\common\ir_agent_tmp\agent.jobs.tem_realtime_1c222301-fd63-4b9e-a0fa-a5017870d924_wqyerw4i
2022-08-31 10:34:21,028 [INFO] [agent.jobs.tem_realtime.19980]: Cleanup completed, graceful exit: True
2022-08-31 10:34:21,028 [DEBUG] [root]: Shutting down logging for this process…
2022-08-31 10:34:21,031 [WARNING] [agent.jobs.network_monitor.2232]: Shutdown event is set - exit status non-zero
2022-08-31 10:34:21,125 [WARNING] [agent.jobs.le_realtime.2456]: Shutdown event is set - exit status non-zero
2022-08-31 10:34:21,140 [DEBUG] [agent.jobs.le_realtime.2456]: Closing OOB queues
2022-08-31 10:34:21,140 [INFO] [agent.jobs.le_realtime.2456]: Cleanup completed, graceful exit: True
2022-08-31 10:34:21,141 [DEBUG] [root]: Shutting down logging for this process…
2022-08-31 10:34:21,203 [WARNING] [agent.platforms.windows.eventlog_api]: Shutting down evtx event_stream
2022-08-31 10:34:21,203 [DEBUG] [agent.platforms.windows.eventlog_api.EvtxLogFollower]: Shutting down all event subscriptions
2022-08-31 10:34:21,204 [DEBUG] [agent.platforms.windows.eventlog_api.2520307456704.EvtxWatcher.security]: Shutting down
2022-08-31 10:34:21,204 [DEBUG] [agent.platforms.windows.eventlog_api.2520307455600.EvtxWatcher.securityWithSource]: Shutting down
2022-08-31 10:34:21,204 [DEBUG] [agent.platforms.windows.eventlog_api.2520307417872.EvtxWatcher.system]: Shutting down
2022-08-31 10:34:21,204 [DEBUG] [agent.platforms.windows.eventlog_api.2520309329200.EvtxWatcher.application]: Shutting down
2022-08-31 10:34:21,205 [DEBUG] [agent.platforms.windows.eventlog_api.2520309329296.EvtxWatcher.applicationWithSource]: Shutting down
2022-08-31 10:34:21,205 [DEBUG] [agent.platforms.windows.eventlog_api.2520307454208.EvtxWatcher.Microsoft-Windows-Windows Defender/OperationalWithSource]: Shutting down
2022-08-31 10:34:21,205 [DEBUG] [agent.platforms.windows.eventlog_api.2520307759952.EvtxWatcher.systemWithSource]: Shutting down
2022-08-31 10:34:21,219 [DEBUG] [agent.jobs.network_monitor.2232]: No poisoned ips captured
2022-08-31 10:34:21,312 [INFO] [agent.jobs.network_monitor.2232]: ProbeWatcher shutdown event - exiting
2022-08-31 10:34:21,312 [INFO] [agent.jobs.network_monitor.2232]: *** Received shutdown event
2022-08-31 10:34:21,313 [INFO] [agent.jobs.network_monitor.2232]: Waiting for NetBiosPoisonMonitor to complete
2022-08-31 10:34:21,313 [DEBUG] [agent.jobs.network_monitor.2232]: NetBiosPoisonMonitor - checking 1 threads
2022-08-31 10:34:21,313 [INFO] [agent.jobs.network_monitor.2232]: NetBiosPoisonMonitor - thread ProbeWatcher completed
2022-08-31 10:34:21,313 [INFO] [agent.jobs.network_monitor.2232]: Shutting down NetBiosPoisonMonitor
2022-08-31 10:34:21,314 [INFO] [agent.jobs.network_monitor.2232]: NetBiosPoisonMonitor completed
2022-08-31 10:34:21,314 [INFO] [agent.jobs.network_monitor.2232]: *** Network monitor(s) completed successfully
2022-08-31 10:34:21,314 [INFO] [agent.jobs.network_monitor.2232]: agent.jobs.network_monitor - exiting
2022-08-31 10:34:21,314 [DEBUG] [agent.jobs.network_monitor.2232]: MainThread setting event code to 2 - interrupting comms (if running)
2022-08-31 10:34:21,314 [DEBUG] [agent.jobs.network_monitor.2232]: Running cleanup
2022-08-31 10:34:21,315 [DEBUG] [agent.jobs.network_monitor.2232]: Waiting for ExceptionWorker to finish…
2022-08-31 10:34:21,315 [INFO] [agent.jobs.network_monitor.2232]: ** Shutdown event - ExceptionWorker
2022-08-31 10:34:21,436 [DEBUG] [agent.jobs.network_monitor.2232]: Closing OOB queues
2022-08-31 10:34:21,436 [INFO] [agent.jobs.network_monitor.2232]: Cleanup completed, graceful exit: True
2022-08-31 10:34:21,437 [DEBUG] [root]: Shutting down logging for this process…
2022-08-31 10:34:22,240 [DEBUG] [agent.agent_id_reporter]: ID Reporter socket timed out after 2 seconds
2022-08-31 10:34:23,048 [DEBUG] [agent.agent_beacon]: Sending 856 bytes for a beacon payload
2022-08-31 10:34:23,048 [DEBUG] [agent.agent_socket.HAS.2995795965984]: BeaconThread overriding default retry of 3 to 1 attempts
2022-08-31 10:34:23,048 [WARNING] [agent.agent_socket.HAS.2995795965984]: Interrupting BeaconThread before sending
2022-08-31 10:34:23,048 [WARNING] [agent.agent_beacon]: Failed to send beacon: No server available
2022-08-31 10:34:23,048 [WARNING] [agent.agent_beacon]: Beacon did not run successfully!
2022-08-31 10:34:23,048 [DEBUG] [agent.agent_beacon]: Refreshing hostname
2022-08-31 10:34:23,048 [DEBUG] [agent.agent_beacon]: Refreshing domain name
2022-08-31 10:34:23,048 [DEBUG] [agent.agent_beacon]: Refreshing fqdn
2022-08-31 10:34:23,048 [DEBUG] [agent.agent_beacon]: Checking persistent jobs state
2022-08-31 10:34:23,049 [DEBUG] [agent.agent_beacon]: Beacon sending shutdown
2022-08-31 10:34:23,049 [INFO] [agent.agent_socket.HAS.2995795965984]: Requesting shutdown: BeaconThread
2022-08-31 10:34:23,049 [INFO] [agent.agent_socket.HAS.2995795965984]: Already been shutdown
2022-08-31 10:34:23,049 [WARNING] [agent.agent_beacon]: Beacon has stopped running
2022-08-31 10:34:24,026 [ERROR] [agent.job_manager]: Did not recv ACK from job: agent.jobs.windows.ui_realtime - uuid: e07e2fba-1613-455d-9147-8a92c034226c for shutdown request
2022-08-31 10:34:24,026 [DEBUG] [agent.job_manager]: Closing OOB queues for job e07e2fba-1613-455d-9147-8a92c034226c.
2022-08-31 10:34:24,027 [DEBUG] [agent.job_manager]: Requested stopJob on 1c222301-fd63-4b9e-a0fa-a5017870d924, marking as 4
2022-08-31 10:34:24,027 [DEBUG] [agent.job_manager]: Closing OOB queues for job 1c222301-fd63-4b9e-a0fa-a5017870d924.
2022-08-31 10:34:24,027 [DEBUG] [agent.job_manager]: Requested stopJob on 1b43ad9f-ada0-4c70-8f8b-34e50e71d3f9, marking as 4
2022-08-31 10:34:24,028 [DEBUG] [agent.job_manager]: Closing OOB queues for job 1b43ad9f-ada0-4c70-8f8b-34e50e71d3f9.
2022-08-31 10:34:24,028 [DEBUG] [agent.job_manager]: Requested stopJob on 8d479879-817c-4dc0-a3f2-8e5189401e67, marking as 4
2022-08-31 10:34:24,028 [DEBUG] [agent.job_manager]: Closing OOB queues for job 8d479879-817c-4dc0-a3f2-8e5189401e67.
2022-08-31 10:34:24,136 [DEBUG] [agent.agent_socket.HAS.2995795965984]: JobMessageSender —> requesting server
2022-08-31 10:34:24,136 [DEBUG] [agent.agent_socket.SMS.2995795966992]: Request ignored - shutting down tracker
2022-08-31 10:34:24,136 [WARNING] [agent.agent_socket.HAS.2995795965984]: JobMessageSender no servers available yet
2022-08-31 10:34:24,136 [WARNING] [agent.agent_socket.HAS.2995795965984]: HASocket - No servers available
2022-08-31 10:34:24,137 [WARNING] [agent.message_bus.sndr.14440]: Unable to send job status
2022-08-31 10:34:24,137 [ERROR] [agent.message_bus.sndr.14440]: Queue size growing during shutdown - prev:1 now:3
2022-08-31 10:34:24,137 [INFO] [agent.message_bus.sndr.14440]: Buffer name: JOB_DATA count: 130
2022-08-31 10:34:24,137 [WARNING] [agent.message_bus.sndr.14440]: Caching 130 JOB_DATA msg(s) - Cache size: 0B - Cache items: 0 - Cache Max: 10485760B
2022-08-31 10:34:24,142 [DEBUG] [agent.message_cache]: Requesting to cache entry that’s 696028 bytes
2022-08-31 10:34:24,142 [INFO] [agent.message_cache]: Used: 696028 bytes - Free: 9789732 bytes
2022-08-31 10:34:24,143 [INFO] [agent.agent_socket.HAS.2995795965984]: Requesting shutdown: JobMessageSender
2022-08-31 10:34:24,143 [INFO] [agent.agent_socket.HAS.2995795965984]: Already been shutdown
2022-08-31 10:34:24,143 [INFO] [agent.message_bus.sndr.14440]: Buffer name: JOB_DATA count: 0
2022-08-31 10:34:24,143 [INFO] [agent.message_bus.sndr.14440]: Buffer name: FILE_UPLOAD_COMPLETE count: 0
2022-08-31 10:34:24,143 [INFO] [agent.message_bus.sndr.14440]: Buffer name: FILE_SET_UPDATE count: 0
2022-08-31 10:34:24,143 [INFO] [agent.message_bus.sndr.14440]: Pulled 1 message from buffer
2022-08-31 10:34:24,143 [INFO] [agent.message_bus.sndr.14440]: Cache info: Cache size: 696028B - Cache items: 1 - Cache Max: 10485760B
2022-08-31 10:34:24,145 [DEBUG] [agent.message_cache]: packed info - pad_len: 12 crc8: 3454104322
2022-08-31 10:34:25,821 [ERROR] [agent.event_engine_manager]: ** No shutdown response from controller
2022-08-31 10:34:25,821 [CRITICAL] [agent.event_engine_manager]: Process - Already terminated - exit code: 3221225786
2022-08-31 10:34:25,828 [INFO] [agent.event_engine_manager]: Rebuilding components
2022-08-31 10:34:25,828 [INFO] [agent.event_engine_manager]: Client: b’\xda\xe3g\x064\xc9\xa8Cf-\xc9\x8d\xccv\x9a\xef5|:2’ unregistering from event: 1 - type: 1
2022-08-31 10:34:25,829 [INFO] [agent.agent]: Waiting for JobMessageReceiver to complete…
2022-08-31 10:34:25,829 [INFO] [agent.agent]: JobMessageReceiver completed.
2022-08-31 10:34:25,829 [INFO] [agent.agent]: Waiting for JobMessageSender to complete…
2022-08-31 10:34:25,829 [INFO] [agent.agent]: JobMessageSender completed.
2022-08-31 10:34:25,829 [INFO] [agent.agent]: Waiting for JobManager to complete…
2022-08-31 10:34:25,829 [DEBUG] [agent.jobs.windows.ui_realtime.1384]: Unregister response - {‘result’: False, ‘error’: “UNREGISTER - Engine shutting down - CID: b’\xda\xe3g\x064\xc9\xa8Cf-\xc9\x8d\xccv\x9a\xef5|:2’ - Event Code: 1 - Filter: 1”}
2022-08-31 10:34:25,830 [WARNING] [agent.jobs.windows.ui_realtime.1384]: Unable to unregister from event {‘result’: False, ‘error’: “UNREGISTER - Engine shutting down - CID: b’\xda\xe3g\x064\xc9\xa8Cf-\xc9\x8d\xccv\x9a\xef5|:2’ - Event Code: 1 - Filter: 1”}
2022-08-31 10:34:25,830 [INFO] [agent.jobs.windows.ui_realtime.1384.ProcessMonitor]: ProcessWatcherThread completed
2022-08-31 10:34:25,831 [INFO] [agent.jobs.windows.ui_realtime.1384]: ProcessMonitor completed
2022-08-31 10:34:25,831 [INFO] [agent.jobs.windows.ui_realtime.1384]: Waiting for LocalAccountMonitor to complete
2022-08-31 10:34:25,831 [INFO] [agent.jobs.windows.ui_realtime.1384]: LocalAccountMonitor completed
2022-08-31 10:34:25,831 [INFO] [agent.jobs.windows.ui_realtime.1384]: Waiting for EventLogMonitor to complete
2022-08-31 10:34:25,832 [INFO] [agent.jobs.windows.ui_realtime.1384]: EventLogMonitor completed
2022-08-31 10:34:25,832 [INFO] [agent.jobs.windows.ui_realtime.1384]: Waiting for HostnameToIpMonitor to complete
2022-08-31 10:34:25,832 [INFO] [agent.jobs.windows.ui_realtime.1384]: HostnameToIpMonitor completed
2022-08-31 10:34:25,833 [INFO] [agent.jobs.windows.ui_realtime.1384]: Shutting down <agent.jobs.windows.ui_realtime.ProcessMonitor object at 0x0000024ACE049B20>
2022-08-31 10:34:25,833 [INFO] [agent.jobs.windows.ui_realtime.1384]: Shutting down <agent.jobs.windows.ui_realtime.LocalAccountMonitor object at 0x0000024ACE049B80>
2022-08-31 10:34:25,834 [INFO] [agent.jobs.windows.ui_realtime.1384]: Shutting down <agent.jobs.windows.ui_realtime.EventLogMonitor object at 0x0000024ACE049BB0>
2022-08-31 10:34:25,834 [INFO] [agent.jobs.windows.ui_realtime.1384.EventLogMonitor]: Shutting down EventLogMonitor
2022-08-31 10:34:25,834 [WARNING] [agent.platforms.windows.eventlog_api.EvtxLogFollower]: Shutting down
2022-08-31 10:34:25,834 [INFO] [agent.jobs.windows.ui_realtime.1384]: Shutting down <agent.jobs.windows.ui_realtime.HostnameToIpMonitor object at 0x0000024ACC122C10>
2022-08-31 10:34:25,834 [WARNING] [agent.jobs.windows.ui_realtime.1384]: *** Realtime job completed successfully
2022-08-31 10:34:25,868 [DEBUG] [agent.jobs.windows.ui_realtime.1384]: Closing OOB queues
2022-08-31 10:34:25,868 [INFO] [agent.jobs.windows.ui_realtime.1384]: Cleanup completed, graceful exit: True
2022-08-31 10:34:25,868 [DEBUG] [root]: Shutting down logging for this process…
2022-08-31 10:34:25,912 [INFO] [agent.agent]: JobManager completed.
2022-08-31 10:34:25,912 [INFO] [agent.agent]: Waiting for BeaconThread to complete…
2022-08-31 10:34:25,912 [INFO] [agent.agent]: BeaconThread completed.
2022-08-31 10:34:25,912 [INFO] [agent.agent]: Waiting for AgentIdReporter to complete…
2022-08-31 10:34:25,912 [INFO] [agent.agent]: AgentIdReporter completed.
2022-08-31 10:34:25,912 [INFO] [agent.agent]: Waiting for CcsBeaconThread to complete…
2022-08-31 10:34:25,912 [INFO] [agent.agent]: CcsBeaconThread completed.
2022-08-31 10:34:25,912 [DEBUG] [agent.agent]: Message Bus sent 22328 received 3469
2022-08-31 10:34:25,913 [DEBUG] [agent.agent]: Beacon sent 22328 received 3469
2022-08-31 10:34:25,913 [DEBUG] [agent.agent]: Thread still running: Name=MainThread,Daemon=False
2022-08-31 10:34:25,913 [DEBUG] [agent.agent]: Thread still running: Name=ExportManagerThread,Daemon=True
2022-08-31 10:34:25,913 [DEBUG] [agent.agent]: Thread still running: Name=Thread-4,Daemon=True
2022-08-31 10:34:25,913 [DEBUG] [agent.agent]: Thread still running: Name=Dummy-65,Daemon=True
2022-08-31 10:34:25,914 [DEBUG] [comtypes]: Calling CoUnititialize()
2022-08-31 10:34:25,914 [DEBUG] [comtypes]: CoUnititialize() done.
We currently have our internal labs which hosts a collector not in operation at the moment (I am working on this,) and the agent should then divert comms direct to platform. However, we’re seeing across the board the events of shutdown job being received but not entirely sure as to where this could be?
Kind regards,