As I only run a few machines with services that matter on them (notably, my home server and my web server), I don’t need a full-on monitoring service, so instead rely on a system called monit.
Monit is an open source piece of software, used to monitor (see, it’s easily named 😄) and, if possible remediate issues with things it sees wrong.
I use this for watching whether particular services are running (and if not, restart them), for whether the ink in my printer is empty, and to monitor the free space and SMART status on my disks.
Today I noticed that a Docker container had stopped, and I’d not noticed. It wasn’t a big thing, but it gnawed at me, so I had a bit of a look around to see what I can find about this.
I found this blog post, titled “Monitoring Docker Containers with Monit”, from 2014, which suggested monitoring the result from docker top… and would you believe it, that’s a valid trick 🙂
So, here’s what I’m doing! Each container has it’s own file called/etc/monit/scripts/check_container_<container-name>.sh which has just this command in it:
#! /bin/bash
docker top "<container-name>"
exit $?Note that you replace <container-name> in both the filename and the script itself with the name of the container – for example, the container hello-world would be monitored with the file check_container_hello-world.sh, and the line in that file would say docker top "hello-world".
I then have a file in /etc/monit/conf.d/ called check_container_<container-name> which has this content
CHECK PROGRAM <container-name> WITH PATH /etc/monit/scripts/check_container_<container-name>.sh
START PROGRAM = "/usr/bin/docker start <container-name>"
STOP PROGRAM = "/usr/bin/docker stop <container-name>"
IF status != 0 FOR 3 CYCLES THEN RESTART
IF 2 RESTARTS WITHIN 5 CYCLES THEN UNMONITORI then ensure that in /etc/monit/monitrc the line “include /etc/monit/conf.d/*” is included and not commented out, and then restart monit with systemctl restart monit.
Featured image is “Monitors” by “Jaysin Trevino” on Flickr and is released under a CC-BY license.
Hi there! For bash scripts try to use shebang: #!/usr/bin/env bash
Thanks for the post!
Thanks Melroy for your comment.
In this case, I’m not certain it’s better. I had a quick read of this stack overflow post, entitled “Why is it better to use “#!/usr/bin/env NAME” instead of “#!/path/to/NAME” as my shebang?”. One of the comments suggests that for an interactive terminal, where you may want to replace python for python3 or python3.7, for tasks executed where you can’t guarantee the path (for example, in a CRON job), being explicit about the source of the binary gives more of a guarantee that your script will run.
While I didn’t explicitly state it, I’m using Ubuntu 20.04 to run this script, so I know where my Bash interpreter will be. If I were writing this to be more portable across multiple distributions of Linux, I might be tempted to write it differently.
Is there a reason you prefer using
#!/usr/bin/env bashversus#!/bin/bash?Jon,
thanks for the article i was looking for something like this a long time ago. I’m newbie but I have done some changes to your setup. The changes are the next ones:
– For the ”START PROGRAM = “/usr/bin/docker start “” I add the user and group to start the docker container so it look like:
START PROGRAM = “/usr/bin/docker start ” as uid “” and gid “”
to choose the correct user to restart the container, we don’t want root to run that container.
– In your the test script I was getting and error (”unable to create context store: $HOME is not defined”), so i also tell which user should do the test so it looks like
”su – -c ‘docker top “‘””.
I think what i have done is correct, what do you think?
Thanks for the reply Alvaro! That looks really good. I’ve rebuilt my server machine twice since I wrote this, and haven’t put monitoring on each time (which is really rather foolish of me!) but your comments both sound very sensible!
Thanks for your great article! It helped me a lot
Thanks Heiser!
Awesome, thanks!
Sorry for the late reply. But it should be more portable, because we don’t have to write the absolute path to an interpreter.
I’m still not convinced. I respect your opinion that your code should be more portable, and yes, I’ve written code that uses that method (grumble grumble OSX grumble)… But in this case (which for context was several years ago now), I wrote it with a specific target in mind, which meant I knew exactly what path to use.
I personally think the only time you need to write with this portability in mind is when you’re writing for multiple OS targets – Ubuntu to OSX is a common one, but to something where the paths are not defined as per LSB is another… Perhaps Nix (not used, so not sure).
Is there another case I should be aware of danger89?
Hi, great find.
However on my Debian/Raspberry, it also needs semicolons after the docker & exit commands.
For example:
#! /bin/bash
docker top “homeassistant”;
exit $?;
Cheers,
Jord
Hey Jord! Thanks for the comment, and it took me on a bit of a goose chase.
Your “need” here is a bit odd, and I wonder whether you edited that script on a mac and copied it over. Your command isn’t understanding that a new line means “the last command is done” and instead is requiring an “end-of-command” character, the semi-colon.
I did a bit of checking, because it’s been a while since I did line-ending-wrangling (more-or-less since I stopped having to use an operating system from Microsoft as my daily-driver).
This is what a Linux system expects to see from a native file.
user@vps:~$ echo "Hello World" > endings
user@vps:~$ echo "Goodbye" >> endings
user@vps:~$ file endings
endings: ASCII text
Cool, so let’s look at what makes that valid for Linux.
\rand\nare two different line-endings, Carriage Return and Line Feed, respectively. So let’s run a set of commands and see how that ends up.printf, by the way, is a great way to send different control characters into a file, which is fun, and often doesn’t work withecho.user@vps:~$ printf "Hello World\nGoodbye\n" > n-endings
user@vps:~$ file n-endings
n-endings: ASCII text
user@vps:~$ printf "Hello World\rGoodbye\r" > r-endings
user@vps:~$ file r-endings
r-endings: ASCII text, with CR line terminators
user@vps:~$ printf "Hello World\r\nGoodbye\r\n" > rn-endings
user@vps:~$ file rn-endings
rn-endings: ASCII text, with CRLF line terminators
user@vps:~$ printf "Hello World\n\rGoodbye\n\r" > nr-endings
user@vps:~$ file nr-endings
nr-endings: ASCII text, with CR, LF line terminators
So
text\nis what your Linux based file system expects to see. Just out of interest, let me open up a couple of those files in the text editor nano.In r-endings, I got this line in the footer:
[ Read 2 lines (Converted from Mac format) ]In rn-endings, I got this line in the footer:
[ Read 2 lines (Converted from DOS format) ]So, the quickest way I know to fix this, on a one-by-one basis, is to edit the file with nano
nano myfile, then press Ctrl+o (the shortcut at the bottom shows ^O) to write out the file and this then shows you can press Alt-m (or, in Nano terms, “Meta-m” written as “M-M”) to change it from “Mac Format” to “Linux Format”, and then hit enter. Exit with Ctrl+x. And if it is using “DOS Format”, same principle, but use Alt-d (or “M-D”) to toggle the DOS format flag off when you go into write.Of course, if it’s not that, then I’m confused on why that script wants end-of-command terminators between each instruction, when a New-Line character would do!
Hi,
I do indeed work on a Mac, but used nano on a ssh-ed terminal to actually type in the script. I saved it with the default settings (not selecting Mac or Dos format).
file indicates : “Bourne-Again shell script, ASCII text executable”.
Sure is weird.
Oooooo that is odd. Might need to break out my Pi and take a look then :) The important thing is that this has been helpful and has worked. The *interesting* thing is that it didn’t work the way I expected!