Data warehousing exports logs

InsightVM
1

Hello,

Is there a way to monitor the progress of data warehousing exports through log files?
If yes, where can I find those logs files?

I followed this guide: Configuring data warehousing settings | InsightVM Documentation

Thanks!

2

there isnt so much as a progress status to check, but you can grep -i warehouse in the nsc.log

3

i wrote a python script that emails me if the export did not run..

here is a very basic script, you can modify to add your own email alert or however you want to alert and then setup a cron to run daily.

import re
import logging
from datetime import datetime

# Configuration
LOG_FILE_PATH = "/opt/rapid7/nexpose/nsc/logs/nsc.log"
LOG_OUTPUT_PATH = "/path/to/warehouse.log"

# Set up logging
logging.basicConfig(
    filename=LOG_OUTPUT_PATH,
    level=logging.INFO,
    format='%(asctime)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

# Define today's date in the format used in the logs
today = datetime.now().strftime('%Y-%m-%d')

# Regular expressions to search for patterns
success_message_pattern = re.compile(r"Warehouse export completed successfully")
started_pattern = re.compile(r"\[Started: (.*?)\]")
duration_pattern = re.compile(r"\[Duration: (.*?)\]")

# Status flag
log_found = False

# Read and parse log file
try:
    with open(LOG_FILE_PATH, 'r') as file:
        lines = file.readlines()

    for line in lines:
        if today in line and success_message_pattern.search(line):
            log_found = True
            started = started_pattern.search(line)
            duration = duration_pattern.search(line)

            logging.info(
                f"Warehouse export completed successfully. "
                f"Started: {started.group(1) if started else 'N/A'} "
                f"Duration: {duration.group(1) if duration else 'N/A'}"
            )
            break

    if not log_found:
        logging.warning(f"No warehouse export success log found for {today}.")

except FileNotFoundError:
    logging.error(f"Log file not found: {LOG_FILE_PATH}")
except Exception as e:
    logging.exception("Unexpected error occurred during log analysis.")