Check container upgrade

Monitoring plugin to check if containers are upgradable. By default all running container are checked.

Checks are done by running Icinga/Nagios compatible check plugins inside containers. These plugins are listed inside the CHECK_PLUGINS associative array (on top of the file) and by default, the following plugin are declared:

• /usr/lib/nagios/plugins/check_apt: for Debian based image, provide by the monitoring-plugins-basic debian package
• /usr/lib/nagios/plugins/check_apk: for Alpine based image, see project for install instructions

Note: The first plugin detected as installed will be used.

This script also include a set of cron modes to automatically rebuild and deploy containers image of a docker compose project:

• check cron (use --check-mode): check if containers need to be updated and marked then to be rebuilt;
• rebuild cron (use --rebuild-mode): rebuild containers marked to be rebuilt;
• deploy cron (use --deploy-cron): deploy rebuilt containers.

Installation

git clone https://gitea.zionetrix.net/bn8/check_container_upgrade.git /usr/local/src/check_container_upgrade
mkdir -p /usr/local/lib/nagios/plugins
ln -s /usr/local/src/check_container_upgrade/check_container_upgrade /usr/local/lib/nagios/plugins/
echo "nagios ALL=NOPASSWD: /usr/local/lib/nagios/plugins/check_container_upgrade" > /etc/sudoers.d/nagios-containers
chmod 0400 /etc/sudoers.d/nagios-containers
echo "command[check_container_upgrade]=sudo /usr/local/lib/nagios/plugins/check_container_upgrade" > /etc/nagios/nrpe.d/containers.cfg
service nagios-nrpe-server reload

Configure crons

mkdir /var/log/check_container_upgrade
cat << EOF > /etc/cron.d/containers
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
00 2 * * * root /usr/local/lib/nagios/plugins/check_container_upgrade -f /srv/docker/docker-compose.yml --build -v -l /var/log/containers/check_container_upgrade.log --check-cron
30 2 * * * root /usr/local/lib/nagios/plugins/check_container_upgrade -f /srv/docker/docker-compose.yml --build -v -l /var/log/containers/check_container_upgrade.log --rebuild-cron
0 4 * * * root /usr/local/lib/nagios/plugins/check_container_upgrade -f /srv/docker/docker-compose.yml --build -v -l /var/log/containers/check_container_upgrade.log --deploy-cron
30 4 * * * root /usr/bin/docker image prune -a -f > /dev/null
EOF
cat << EOF > /etc/logrotate.d/containers
/var/log/check_container_upgrade/*.log {
        weekly
        missingok
        rotate 53
        compress
        copytruncate
        notifempty
}
EOF

Usage

Usage : check_container_upgrade [-d] [-E /path/to/engine] [container1,...]
    -E [path]               Force a specific engine (possible values: auto docker podman,
                            default: auto)
    -x [container]          Exclude specified container (could be repeat)
    -M [integer]            Max number of container checks to run in parallel
                            (default: 4, 0=no limit)
    -f [docker-compose.yml] To check upgrade on docker compose project, specified the path of the
                            docker-compose.yml file
    -b|--build|--rebuild    Trigger container build if upgrade detected (only possible if a docker
                            compose file if provided)
    --rebuild-path          Specify rebuild data directory path (default: /var/log/check_container_upgrade)
    --rebuild-cron          Start in rebuild cron mode: rebuild containers detected and mark to be
                            rebuilt on status file.
    --deploy-cron           Start in deploy cron mode: deploy containers known as rebuilt in status
                            file.
    --check-cron            Start in check cron node: check if containers need to be updated and
                            trigger their rebuild.
    -d                      Debug mode
    -l                      Log file
    -C                      Console logging (even if log file is specify)
    -X                      Enable bash tracing (=set -x)
    -h                      Show this message

Copyright

Copyright (c) 2024 Benjamin Renard brenard@zionetrix.net

License

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.