Health check

Maand allow jobs to define specialized command using Python scripts stored in the _modules folder.

Setup

This example demonstrates how to implement a health check for the node_exporter job, ensuring the service runs correctly across all workers.

Create the _modules Folder

1. Navigate to the node_exporter job directory:

   $ cd workspace/jobs/node_exporter
   

2. Create the _modules folder:

   $ mkdir _modules
   

Define the Health Check Command

1. Create the command_health_check.py file:

   $ cat > _modules/command_health_check.py
   

2. Add the following Python code:

   import requests
   import os
   
   def execute():
       workers = os.getenv("WORKER_WORKERS")
       workers = workers.split(",")
       for worker in workers:
           r = requests.get(f"http://{worker}:9100/metrics")
           r.raise_for_status()
   

   What It Does:

   • Retrieves the list of worker IPs from the WORKER_WORKERS environment variable.
   • Sends HTTP requests to each worker on port 9100 to verify node_exporter is running.
   • Raises an exception if a node's health check fails.

Run

$ maand health_check

Example output

health check successed : node_exporter

Maand execute health checks part of deploy command, this make sure deployments are always verified.