Running monit inside Kubernetes

Sometimes you may want to run monit inside a Kubernetes cluster just to validate what you’re getting from your standard monitoring solution with a second monitor that does not require that much configuration or tinkering. In such cases the Dockerfile bellow might come handy:

FROM ubuntu:bionic
RUN apt-get update
RUN apt-get install monit bind9-host netcat fping -y
RUN ln -f -s /dev/fd/1 /var/log/monit.log
COPY monitrc /etc/monit
RUN chmod 0600 /etc/monit/monitrc
EXPOSE 2812
ENTRYPOINT [ "/usr/bin/monit" ]
CMD [ "-I", "-c", "/etc/monit/monitrc" ]

I connected to it via kubectl -n monit-test port-forward --address=0.0.0.0 pod/monit-XXXX-YYYY 2812:2812. Most people do not need --address=0.0.0.0, but I run kubectl inside a VM for some degree of compartmentalization. Stringent, I know…

Why would you need something like this you ask? Well imagine the case where you have multiple pods running, no restarts, everything fine, but randomly you get connection timeouts to the clusterIP address:port pair. If you have no way of reproducing this, don’t you want an alert the exact moment it happens? That was the case for me.

And also the fun of using a tool in an unforeseen way.

Having monit complement ansible

Here is a weird thing:

When running /etc/init.d/milter-greylist restart via ansible (either direct or via a playbook) it hangs. I had no time to debug this, so I reverted to the next best workaround since the machine was already running monit:

Have ansible distribute greylist.conf and then have monit restart the process. So here is a simple playbook:

- hosts: greylist
  user: root
  tasks:
  - name: copy local milter-greylist configuration to hosts
    action: copy src=/usr/local/etc/greylist.conf dest=/etc/milter-greylist/greylist.conf

and here is how monit finishes the task:

check file greylist.conf with path /etc/milter-greylist/greylist.conf
 if changed checksum then exec "/etc/init.d/milter-greylist restart"

Of course this is just a simple case of having the two cooperate. But once you get the hang of it, more elaborate schemes where ansible and monit can cooperate will pop out.

sudo within monit

When you need to execute sudo from monit, you have to check whether the requiretty flag is off in your sudoers file.

requiretty: If set, sudo will only run when the user is logged in to a real tty. When this flag is set, sudo can only be run from a login session and not via other means such as cron(8) or cgi-bin scripts. This flag is off by default.

I was working on a machine that had:

Defaults requiretty

and got bit by that.