"Monitors" by "Jaysin Trevino" on Flickr

Using monit to monitor Docker Containers

As I only run a few machines with services that matter on them (notably, my home server and my web server), I don’t need a full-on monitoring service, so instead rely on a system called monit.

Monit is an open source piece of software, used to monitor (see, it’s easily named 😄) and, if possible remediate issues with things it sees wrong.

I use this for watching whether particular services are running (and if not, restart them), for whether the ink in my printer is empty, and to monitor the free space and SMART status on my disks.

Today I noticed that a Docker container had stopped, and I’d not noticed. It wasn’t a big thing, but it gnawed at me, so I had a bit of a look around to see what I can find about this.

I found this blog post, titled “Monitoring Docker Containers with Monit”, from 2014, which suggested monitoring the result from docker top… and would you believe it, that’s a valid trick 🙂

So, here’s what I’m doing! Each container has it’s own file called/etc/monit/scripts/check_container_<container-name>.sh which has just this command in it:

#! /bin/bash
docker top "<container-name>"
exit $?

Note that you replace <container-name> in both the filename and the script itself with the name of the container – for example, the container hello-world would be monitored with the file check_container_hello-world.sh, and the line in that file would say docker top "hello-world".

I then have a file in /etc/monit/conf.d/ called check_container_<container-name> which has this content

CHECK PROGRAM <container-name> WITH PATH /etc/monit/scripts/check_container_<container-name>.sh
  START PROGRAM = "/usr/bin/docker start <container-name>"
  STOP PROGRAM = "/usr/bin/docker stop <container-name>"
  IF status != 0 FOR 3 CYCLES THEN RESTART
  IF 2 RESTARTS WITHIN 5 CYCLES THEN UNMONITOR

I then ensure that in /etc/monit/monitrc the line “include /etc/monit/conf.d/*” is included and not commented out, and then restart monit with systemctl restart monit.

Featured image is “Monitors” by “Jaysin Trevino” on Flickr and is released under a CC-BY license.

JonTheNiceGuy

He/Him. Husband and father. Linux advocating geek. Co-Host on the AdminAdmin Podcast, occasional conference speaker. I work for Amazon Web Services and any opinions expressed are my own.

6 thoughts to “Using monit to monitor Docker Containers”

    1. Thanks Melroy for your comment.

      In this case, I’m not certain it’s better. I had a quick read of this stack overflow post, entitled “Why is it better to use “#!/usr/bin/env NAME” instead of “#!/path/to/NAME” as my shebang?”. One of the comments suggests that for an interactive terminal, where you may want to replace python for python3 or python3.7, for tasks executed where you can’t guarantee the path (for example, in a CRON job), being explicit about the source of the binary gives more of a guarantee that your script will run.

      While I didn’t explicitly state it, I’m using Ubuntu 20.04 to run this script, so I know where my Bash interpreter will be. If I were writing this to be more portable across multiple distributions of Linux, I might be tempted to write it differently.

      Is there a reason you prefer using #!/usr/bin/env bash versus #!/bin/bash?

  1. Jon,
    thanks for the article i was looking for something like this a long time ago. I’m newbie but I have done some changes to your setup. The changes are the next ones:
    – For the ”START PROGRAM = “/usr/bin/docker start “” I add the user and group to start the docker container so it look like:
    START PROGRAM = “/usr/bin/docker start ” as uid “” and gid “”
    to choose the correct user to restart the container, we don’t want root to run that container.
    – In your the test script I was getting and error (”unable to create context store: $HOME is not defined”), so i also tell which user should do the test so it looks like
    ”su – -c ‘docker top “‘””.

    I think what i have done is correct, what do you think?

    1. Thanks for the reply Alvaro! That looks really good. I’ve rebuilt my server machine twice since I wrote this, and haven’t put monitoring on each time (which is really rather foolish of me!) but your comments both sound very sensible!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.