As I only run a few machines with services that matter on them (notably, my home server and my web server), I don’t need a full-on monitoring service, so instead rely on a system called monit.
Monit is an open source piece of software, used to monitor (see, it’s easily named 😄) and, if possible remediate issues with things it sees wrong.
I use this for watching whether particular services are running (and if not, restart them), for whether the ink in my printer is empty, and to monitor the free space and SMART status on my disks.
Today I noticed that a Docker container had stopped, and I’d not noticed. It wasn’t a big thing, but it gnawed at me, so I had a bit of a look around to see what I can find about this.
I found this blog post, titled “Monitoring Docker Containers with Monit”, from 2014, which suggested monitoring the result from docker top… and would you believe it, that’s a valid trick 🙂
So, here’s what I’m doing! Each container has it’s own file called/etc/monit/scripts/check_container_<container-name>.sh which has just this command in it:
#! /bin/bash
docker top "<container-name>"
exit $?Note that you replace <container-name> in both the filename and the script itself with the name of the container – for example, the container hello-world would be monitored with the file check_container_hello-world.sh, and the line in that file would say docker top "hello-world".
I then have a file in /etc/monit/conf.d/ called check_container_<container-name> which has this content
CHECK PROGRAM <container-name> WITH PATH /etc/monit/scripts/check_container_<container-name>.sh
START PROGRAM = "/usr/bin/docker start <container-name>"
STOP PROGRAM = "/usr/bin/docker stop <container-name>"
IF status != 0 FOR 3 CYCLES THEN RESTART
IF 2 RESTARTS WITHIN 5 CYCLES THEN UNMONITORI then ensure that in /etc/monit/monitrc the line “include /etc/monit/conf.d/*” is included and not commented out, and then restart monit with systemctl restart monit.
Featured image is “Monitors” by “Jaysin Trevino” on Flickr and is released under a CC-BY license.
Hi there! For bash scripts try to use shebang: #!/usr/bin/env bash
Thanks for the post!
Thanks Melroy for your comment.
In this case, I’m not certain it’s better. I had a quick read of this stack overflow post, entitled “Why is it better to use “#!/usr/bin/env NAME” instead of “#!/path/to/NAME” as my shebang?”. One of the comments suggests that for an interactive terminal, where you may want to replace python for python3 or python3.7, for tasks executed where you can’t guarantee the path (for example, in a CRON job), being explicit about the source of the binary gives more of a guarantee that your script will run.
While I didn’t explicitly state it, I’m using Ubuntu 20.04 to run this script, so I know where my Bash interpreter will be. If I were writing this to be more portable across multiple distributions of Linux, I might be tempted to write it differently.
Is there a reason you prefer using
#!/usr/bin/env bashversus#!/bin/bash?Jon,
thanks for the article i was looking for something like this a long time ago. I’m newbie but I have done some changes to your setup. The changes are the next ones:
– For the ”START PROGRAM = “/usr/bin/docker start “” I add the user and group to start the docker container so it look like:
START PROGRAM = “/usr/bin/docker start ” as uid “” and gid “”
to choose the correct user to restart the container, we don’t want root to run that container.
– In your the test script I was getting and error (”unable to create context store: $HOME is not defined”), so i also tell which user should do the test so it looks like
”su – -c ‘docker top “‘””.
I think what i have done is correct, what do you think?
Thanks for the reply Alvaro! That looks really good. I’ve rebuilt my server machine twice since I wrote this, and haven’t put monitoring on each time (which is really rather foolish of me!) but your comments both sound very sensible!
Thanks for your great article! It helped me a lot
Thanks Heiser!
Awesome, thanks!
Sorry for the late reply. But it should be more portable, because we don’t have to write the absolute path to an interpreter.
I’m still not convinced. I respect your opinion that your code should be more portable, and yes, I’ve written code that uses that method (grumble grumble OSX grumble)… But in this case (which for context was several years ago now), I wrote it with a specific target in mind, which meant I knew exactly what path to use.
I personally think the only time you need to write with this portability in mind is when you’re writing for multiple OS targets – Ubuntu to OSX is a common one, but to something where the paths are not defined as per LSB is another… Perhaps Nix (not used, so not sure).
Is there another case I should be aware of danger89?