Script for checking inactive Services

[shahidirfan]

I have developed a bash script that monitors the running services in myVestaCP. Should any service cease to function, the script will restart it after two minutes. It operates effectively. However, for safety reasons, it is advisable to test it on a non-production service or to create a snapshot of your VPS prior to usage.

Code: Select all

#!/bin/bash

# Comprehensive Service Monitor and Recovery Script for Debian-based systems
# This script sets up a systemd service to monitor and automatically restart critical services if they become inactive.
# Services monitored: MariaDB, Apache2, Nginx, PHP7.4, Redis, Memcached

# Ensure the script is run as root
if [ "$EUID" -ne 0 ]; then
    echo "Please run as root"
    exit 1
fi

# Define the path for the monitor script
MONITOR_SCRIPT="/usr/local/bin/service_monitor.sh"

# Define the path for the systemd service file
SERVICE_FILE="/etc/systemd/system/service-monitor.service"

# Log file path
LOG_FILE="/var/log/service_monitor.log"

# Create the monitor script
cat << EOF > "$MONITOR_SCRIPT"
#!/bin/bash

# Log file path
LOG_FILE="/var/log/service_monitor.log"

# Array of services to monitor
SERVICES=("mariadb" "apache2" "nginx" "php7.4-fpm" "redis-server" "memcached")

# Function to log messages
log_message() {
    echo "\$(date '+%Y-%m-%d %H:%M:%S') - \$1" >> "\$LOG_FILE"
}

# Function to check and restart a service
check_and_restart_service() {
    local service=\$1
    if ! systemctl is-active --quiet "\$service"; then
        log_message "Service \$service is inactive. Waiting 2 minutes before attempting restart."
        sleep 120
        if ! systemctl is-active --quiet "\$service"; then
            log_message "Attempting to restart \$service"
            if systemctl restart "\$service"; then
                log_message "Service \$service restarted successfully"
            else
                log_message "Failed to restart \$service. Manual intervention may be required."
            fi
        else
            log_message "Service \$service is now active. No restart needed."
        fi
    fi
}

# Main loop
while true; do
    for service in "\${SERVICES[@]}"; do
        # Use a subshell to handle each service check concurrently
        (
            check_and_restart_service "\$service"
        ) &
    done

    # Wait for all background processes to finish
    wait

    # Sleep for 2 minutes before the next check
    sleep 120
done
EOF

# Make the monitor script executable
chmod +x "$MONITOR_SCRIPT"

# Create the systemd service file
cat << EOF > "$SERVICE_FILE"
[Unit]
Description=Service Monitor and Recovery
After=network.target

[Service]
ExecStart=$MONITOR_SCRIPT
Restart=always
User=root

[Install]
WantedBy=multi-user.target
EOF

# Reload systemd, enable and start the service
systemctl daemon-reload
systemctl enable service-monitor.service
systemctl start service-monitor.service

echo "Service monitor has been set up and started."
echo "You can check its status with: systemctl status service-monitor"
echo "Logs are available at: $LOG_FILE"

[myVesta]

Our suggestion for Main looop:

Code: Select all

apt_running=$(ps -Af | grep 'apt' | grep -v 'grep apt' | wc -l)
if [ "$apt_running" -gt 0 ]; then
	echo "apt is running, maybe apt upgrade, we should not restart services";
	sleep 120,
	continue;
fi

[shahidirfan]

Thank you for your suggestions; I have updated the script accordingly.

Code: Select all

#!/bin/bash

# Comprehensive Service Monitor and Recovery Script for Debian-based systems
# This script sets up a systemd service to monitor and automatically restart critical services if they become inactive.
# Services monitored: MariaDB, Apache2, Nginx, PHP7.4, Redis, Memcached

# Ensure the script is run as root
if [ "$EUID" -ne 0 ]; then
    echo "Please run as root"
    exit 1
fi

# Define the path for the monitor script
MONITOR_SCRIPT="/usr/local/bin/service_monitor.sh"

# Define the path for the systemd service file
SERVICE_FILE="/etc/systemd/system/service-monitor.service"

# Log file path
LOG_FILE="/var/log/service_monitor.log"

# Create the monitor script
cat << EOF > "$MONITOR_SCRIPT"
#!/bin/bash

# Log file path
LOG_FILE="/var/log/service_monitor.log"

# Array of services to monitor
SERVICES=("mariadb" "apache2" "nginx" "php7.4-fpm" "redis-server" "memcached")

# Function to log messages
log_message() {
    echo "\$(date '+%Y-%m-%d %H:%M:%S') - \$1" >> "\$LOG_FILE"
}

# Function to check and restart a service
check_and_restart_service() {
    local service=\$1
    if ! systemctl is-active --quiet "\$service"; then
        log_message "Service \$service is inactive. Waiting 2 minutes before attempting restart."
        sleep 120
        if ! systemctl is-active --quiet "\$service"; then
            log_message "Attempting to restart \$service"
            if systemctl restart "\$service"; then
                log_message "Service \$service restarted successfully"
            else
                log_message "Failed to restart \$service. Manual intervention may be required."
            fi
        else
            log_message "Service \$service is now active. No restart needed."
        fi
    fi
}

# Main loop
while true; do
    # Check if apt is running
    apt_running=\$(ps -Af | grep 'apt' | grep -v 'grep apt' | wc -l)
    if [ "\$apt_running" -gt 0 ]; then
        log_message "apt is running, possibly due to system updates. Skipping service checks."
        sleep 120
        continue
    fi

    for service in "\${SERVICES[@]}"; do
        # Use a subshell to handle each service check concurrently
        (
            check_and_restart_service "\$service"
        ) &
    done

    # Wait for all background processes to finish
    wait

    # Sleep for 2 minutes before the next check
    sleep 120
done
EOF

# Make the monitor script executable
chmod +x "$MONITOR_SCRIPT"

# Create the systemd service file
cat << EOF > "$SERVICE_FILE"
[Unit]
Description=Service Monitor and Recovery
After=network.target

[Service]
ExecStart=$MONITOR_SCRIPT
Restart=always
User=root

[Install]
WantedBy=multi-user.target
EOF

# Reload systemd, enable and start the service
systemctl daemon-reload
systemctl enable service-monitor.service
systemctl start service-monitor.service

echo "Service monitor has been set up and started."
echo "You can check its status with: systemctl status service-monitor"
echo "Logs are available at: $LOG_FILE"