Out of work hours activity detection

testuser · January 14, 2025, 12:55pm

Hello folks,

I want to collect specific action values from the logs during non-working hours based on the country tags in the logs, determine working hours for each country, and if the activity occurs outside working hours, generate an alarm.

Using Python, I have generated an array with 3 objects:

[
{“actor”: “xxx”, “country_code”: “DE”, “timestamp”: “2025-01-14T11:07:24”},
{“actor”: “yyy”, “country_code”: “ES”, “timestamp”: “2025-01-14T11:07:24”},
{“actor”: “zzz”, “country_code”: “DE”, “timestamp”: “2025-01-14T11:34:38”}
]

I want to give this array into a loop, check the country code, compare the time, and proceed accordingly to generate alarms. Alternatively, I think this can be done using predefined steps without Python by performing log queries one by one. Does anyone have any ideas or previous experience with this?

darrick_hall1 · January 14, 2025, 4:37pm

Can you describe the entire workflow in more detail?

What is the triggering event?

What is feeding the python step? You are using Python to get the array, but what is your input, and what is the python step actually doing?

testuser · January 15, 2025, 8:26am

In GitHub logs, for the git.push action, it is necessary to identify users performing activities outside the local working hours of different subsidiaries and generate alerts for such cases. Since IDR rules do not support activation based on specific time ranges, I am attempting to implement this process using a workflow.

To achieve this, I am using a timer trigger that runs every hour. Subsequently, I collect logs from the event source in GitHub where action=git.push. Due to JSON inconsistencies in the logs, I am using string replacement to replace "false" with "False". I successfully extracted the actor, country_tag, and timestamp details from the logs in string format using a Python script, as shown in the example above.

The next step involves feeding the Python script output into a loop. If the country_code is ES, it should check Spain’s local working hours, and if there is a log outside the working hours, an alert should be generated. We have more than four local working regions, and the working hour ranges for these regions need to be defined by me and checked by ICON. This is the project plan.

If there is a better alternative, I am open to re-creating the workflow.

When I configure the Python script output as an object, the output in the artifact appears as shown in the example above. In my previous workflows, when I used a single-object output, I could create a variable as an array using an array match to provide input to the loop. However, this time I have a 3-object array, and it seems to be treated as an object because it is not a single object.

def run(params={}):
input_array = {{[“Action - 4”].[result_string]}}

output_json = []


for log in input_array:
    country_code = str(log["message"]["actor_location"]["country_code"])
    actor = str(log["message"]["actor"])
    timestamp = str(log["timestamps"]["custom"]).split(".")[0]

    if country_code:
        output_json.append({"country_code":country_code,"actor":actor,"timestamp":timestamp})


return {
    "output_json": output_json
}

output_json:
[
{“actor”: “xxx”, “country_code”: “DE”, “timestamp”: “2025-01-14T11:07:24”},
{“actor”: “yyy”, “country_code”: “ES”, “timestamp”: “2025-01-14T11:07:24”},
{“actor”: “zzz”, “country_code”: “DE”, “timestamp”: “2025-01-14T11:34:38”}
]

darrick_hall1 · January 16, 2025, 3:43am

I am very sorry about the late reply.

So the solution I would propose would be to use the JQ plugin. JQ is Java Query, and recently I have found it to be quite useful. You can remove large loops, you can transform data. I believe you could probably remove your python step as well and match all false and False as one like value if you were to perform some trial and error.

The JQ plugin is looking for an object as input. This is going on the assumption that your array you want to loop through is called output_json. If it is not, then you will need to format accordingly. Pass the entire object to the JQ step as input.

This is the JQ statement you will use in the filter section of the plugin input:

def tz_offset(country_code):
  if country_code == "US" then -5
  elif country_code == "DE" or country_code == "ES" then 1
  else 0
  end;

def in_working_hours(hour):
  hour >= 9 and hour <= 17;

.output_json
| map(
    . + {
      local_hour: (tz_offset(.country_code) + ((.timestamp + "Z") | fromdateiso8601 | . / 3600 | floor % 24)),
      out_of_hours: (
        tz_offset(.country_code) + ((.timestamp + "Z") | fromdateiso8601 | . / 3600 | floor % 24) | (. < 9 or . > 17)
      )
    }
  )

I will break each section down so you know what it is doing.

To start I am making the assumption that your time is UTC in your output. You will have to adjust accordingly if that is not accurate.

This function assigns a time zone offset (in hours) based on the country code:

"US": UTC-5 (Eastern Time)
"DE" or "ES": UTC+1 (Central European Time)
else 0: Default for other countries, assumes UTC. :

def tz_offset(country_code):
  if country_code == "US" then -5
  elif country_code == "DE" or country_code == "ES" then 1
  else 0
  end;

This function checks whether a given hour is within working hours, I chose 9 to 5 and it is on military time. If working hours are 8AM to 4PM it would be 8 and 16. Adjust to whatever is good for you:

hour >= 9: Starts at 9 AM.
hour <= 17: Ends at 5 PM.

Returns true if the hour is within this range, otherwise false.

def in_working_hours(hour):
  hour >= 9 and hour <= 17;

.output_json: Accesses the array of objects within the "output_json" field.
map(...): Iterates over each object in the array, applying a transformation to it.

.output_json
| map(...)

This line calculates the local time in the country for each record:

tz_offset(.country_code):

Gets the time zone offset for the country_code.

.timestamp + "Z":

Appends "Z" to indicate that the timestamp is in UTC format (required by fromdateiso8601).

fromdateiso8601:

Converts the timestamp into a Unix timestamp (seconds since 1970-01-01T00:00:00Z).

. / 3600:

Converts Unix timestamp to hours.

floor % 24:

Ensures the result is a whole number and wraps it to a 24-hour clock.

tz_offset + ...:

Adds the time zone offset to calculate the local hour.

local_hour: (tz_offset(.country_code) + ((.timestamp + "Z") | fromdateiso8601 | . / 3600 | floor % 24))

This calculates the same local_hour but directly checks if it’s outside working hours:

. < 9: Before 9 AM.
. > 17: After 5 PM.
out_of_hours: true if the time is outside working hours, false otherwise.

out_of_hours: (
  tz_offset(.country_code) + ((.timestamp + "Z") | fromdateiso8601 | . / 3600 | floor % 24) | (. < 9 or . > 17)
)

The .+ operator adds the new fields (local_hour and out_of_hours) to the original object, creating an updated version.

. + { local_hour: ..., out_of_hours: ... }

This was the input I provided:

{"output_json":[{"actor":"xxx","timestamp":"2025-01-14T11:07:24","country_code":"DE"},{"actor":"yyy","timestamp":"2025-01-14T11:07:24","country_code":"ES"},{"actor":"zzz","timestamp":"2025-01-14T11:34:38","country_code":"DE"},{"actor":"aaa","timestamp":"2025-01-14T18:00:00","country_code":"DE"},{"actor":"bbb","timestamp":"2025-01-14T08:30:00","country_code":"ES"},{"actor":"ccc","timestamp":"2025-01-14T16:59:59","country_code":"US"},{"actor":"ddd","timestamp":"2025-01-14T21:00:00","country_code":"US"},{"actor":"eee","timestamp":"2025-01-14T04:00:00","country_code":"DE"},{"actor":"fff","timestamp":"2025-01-14T14:00:00","country_code":"ES"},{"actor":"ggg","timestamp":"2025-01-14T06:00:00","country_code":"US"}]}

This was the output received:

[ { "actor": "xxx", "country_code": "DE", "timestamp": "2025-01-14T11:07:24", "local_hour": 12, "out_of_hours": false }, { "actor": "yyy", "country_code": "ES", "timestamp": "2025-01-14T11:07:24", "local_hour": 12, "out_of_hours": false }, { "actor": "zzz", "country_code": "DE", "timestamp": "2025-01-14T11:34:38", "local_hour": 12, "out_of_hours": false }, { "actor": "aaa", "country_code": "DE", "timestamp": "2025-01-14T18:00:00", "local_hour": 19, "out_of_hours": true }, { "actor": "bbb", "country_code": "ES", "timestamp": "2025-01-14T08:30:00", "local_hour": 9, "out_of_hours": false }, { "actor": "ccc", "country_code": "US", "timestamp": "2025-01-14T16:59:59", "local_hour": 11, "out_of_hours": false }, { "actor": "ddd", "country_code": "US", "timestamp": "2025-01-14T21:00:00", "local_hour": 16, "out_of_hours": false }, { "actor": "eee", "country_code": "DE", "timestamp": "2025-01-14T04:00:00", "local_hour": 5, "out_of_hours": true }, { "actor": "fff", "country_code": "ES", "timestamp": "2025-01-14T14:00:00", "local_hour": 15, "out_of_hours": false }, { "actor": "ggg", "country_code": "US", "timestamp": "2025-01-14T06:00:00", "local_hour": 1, "out_of_hours": true } ]

I hope this helps get you started.

testuser · January 17, 2025, 10:50am

Hi Darrick,
I will check as soon as possible. Thanks for your reply.

testuser · January 17, 2025, 2:37pm

This is good, but now I need to understand how to process this. I want to check all objects one by one, and when the out_of_hours field is "false" (i.e., activity occurs outside of working hours), the alarm should change based on the user’s name or country. For example: “OutofWorking_Hours_activity_from_DE_user:XXX”

[
{“actor”: “xxx”, “country_code”: “DE”, “timestamp”: “2025-01-14T11:07:24”},
{“actor”: “yyy”, “country_code”: “ES”, “timestamp”: “2025-01-14T11:07:24”},
{“actor”: “zzz”, “country_code”: “DE”, “timestamp”: “2025-01-14T11:34:38”}
]

I initially planned to pass this output to an array, where the array would consist of objects with three fields each. I thought I could loop through the array and provide conditions for the country tag to extract and use the values. However, that approach didn’t work out very well.

The output we get from JQ is string. At this point, I’m a bit stuck.

darrick_hall1 · January 17, 2025, 3:39pm

You can modify the JQ using this statement instead.

def tz_offset(country_code):
  if country_code == "US" then -5
  elif country_code == "DE" or country_code == "ES" then 1
  else 0
  end;

def in_working_hours(hour):
  hour >= 9 and hour <= 17;

{
  results: (
    .output_json
    | map(
        . + {
          local_hour: (tz_offset(.country_code) + ((.timestamp + "Z") | fromdateiso8601 | . / 3600 | floor % 24)),
          out_of_hours: (
            tz_offset(.country_code) + ((.timestamp + "Z") | fromdateiso8601 | . / 3600 | floor % 24) | (. < 9 or . > 17)
          )
        }
      )
    )
}

Next you will use the Type Converter Plugin and the action you would use is called String to Object.

Then you can test it and you will see your desired output.

If you open the Type Converter step, and you hit edit, you can modify the output. I’ve included a Snippet that does this for you, so you have something to go off of.

JQ Convert Time.snpt (8.2 KB)

testuser · January 17, 2025, 3:59pm

Thank you so much Darrick! You’ve saved me from struggling with Python. The output is exactly how I wanted, and I can now process it further as needed.

darrick_hall1 · January 17, 2025, 4:08pm

Not a problem. Glad it worked out. Please be sure to test the times that it provides, and make sure the output is working in accordance with the local times. I made some assumptions on what time format your logs are showing.