----------------------------------------------------------------------- MoSheL - MOnitoring with SHEll Locally 2020- by Volker Tanger ----------------------------------------------------------------------- MoSheL (MOnitoring with SHEll Locally) is a simple, lightweight (both in size and system requirements) server monitoring package designed for secure and in-depth monitoring of single or multiple typical/critical internet systems. As most of the servers/services I want to monitor are remote systems, traditional NMS (relying on close-looped and/or unencrypted sessions) are either big, complicated to install for safe remote monitoring, ressource intense (when doing remote checks), lack a status history or a combination thereof. Thus I (re)wrote this small, easily configured system. It originally was intended for monitoring of single a handful of typical internet systems. After 17 years after the first installment and 11 years after the big rewrite this is the next iteration: even smaller, more decentralized, easier to adapt to local peculiarities (or changes in output formats). MoSheL supports email alerts (with flapping-prevention) out of the box - and whatever you can script. The system is programmed in plain (Bourne) SH, and to be compatible with BASH, ZSH an various others, so it can easily be deployed on most unixish systems. Monitoring is designed to be distributed over multiple systems, usually running locally. As no parameters are accepted from outside, checks cannot be tampered or misused from outside. For centralized monitoring usually only aggregate data (i.e. the current, non-okay status) is transferred, easing network load a lot. Agent data is transferred via HTTP(S) in pull-mode - which comes in handily if you are moinitoring web servers anyway. So no additional network daemons or weaknesses. As agent queries are only downloading locally generated static files, the possible attack surface is minimal. Requirements for MoSheL: * Unix Shell (tested with Bourne-Shell and BASH) * awk + standard Unix text tools (fgrep, mail, time, date, ...) for single checks only if performed: depending on what you want to check - see the "moshel.example" script for web interface: * webserver - which can server static files (= nearly any) * the "dygraphs" JavaScript library. Included in the archive within the /data/ directory Hardware requirements: Should be minimal. 39 checks on my 1vCore mailserver (including 40% checks of remote systems) take 3.5s (.75 CPU-seconds). RaspberryPi with SD card needs 34s (1.3 CPU-seconds) for the same checks. A central monitoring system loads a single static file snippet from each monitored server - usually a few hundred bytes every five minutes. The system is a shell script, so no big size components here, either. For a webserver (nearly) any HTTPD is fine. No database needed - everything is plain text. Updates will be available at http://www.wyae.de/software/moshel/ Please check there for updates prior to submitting patches! For bug reports and suggestions or if you just want to talk to me please contact me at volker.tanger@wyae.de ----------------------------------------------------------------------- Installation & Setup ----------------------------------------------------------------------- Get and untar the archive - usually into /usr/local/lib/moshel copy the whole data/ directory to WWWDIR (see below) Edit the MOSHEL file and set the environment MYNAME HOSTname of this server ALERTTO mail address where aerts should be sent to WEBURL the URL the snippet, overview and data files are available under WWWDIR where the HTML reports and status file are saved to CMPDIR location the tamper-detection files are copied to Copy the moshel.example shell script to moshel file and configure the checks to be run - usually you can set alert trigger levels Adapt the "moshel" script. Run it locally after each configuration change to make sure your command and match pattern does not contain any errors. Place the cron.d_moshel file to /etc/cron.d/moshel, maybe adapt it accordingly so moshel is called periodically. Via the web interface you can view the overall status - full and abbreviated status. But you cannot modify anything - which makes it quite safe for even non-admin multiuser use... ;-) ------------ Local Checks ------------ You can group checks for output with the command Category "HEADLINE" All other checks follow the pattern CheckSOMETHING Name without spaces 'command -opt' command with all parameters - without pipes! 'AWK pattern' AWK pattern, see "man awk", usually '/regex/{ print $POS }' where POS is the nth value in the result return line matching the regex value value the result is compated to 'alert message' The message printed whenever the check fails Checks are named / to be read as "expected" or "should-be". If the check fails / exceeds the value, its status is changed. Whenever a status changes to "alert" for 2 consecutive checks, an email alert is sent. CheckValueOver Number value should be over the one given CheckValueUnder Number value should be under the one given CheckCountLessThan count the number of lines the check yields, they should be below the number given CheckCountMoreThan count the number of lines the check yields, they should be above the number given CheckFileChanges The file should match the comparison file. Generate the comparison files with ./gen_compares with the moshel directory --------------- Action On Alert --------------- If something breaks (e.g. FCGI hickups between httpd and PHP process), then MoSheL can run actions if the problem occurs. Use this command AFTER "CleanUp" at the end of the script - the CleanUp is needed to ensure the check is not just flapping but "really" failing two consecutive runs. The command follows the pattern: ActionOnAlert CHECKNAME 'COMMAND TO BE RUN' ------------------ Central Monitoring ------------------ Well, a biiig name for what it does. Add a line to your /usr/local/lib/moshel/moshel file on the "central monitoring server" Centralize URL1/moshel URL2/moshel ... where you list all URLs to the MoSheL directories files of all "centrally monitored" servers. You can find the summaries at URL/moshel/summary.html For the "graphs" summary you will need to set the CORS headers accordingly (if you want to risk it). ----------------------------------------------------------------------- Customizing Checks & Writing your own ----------------------------------------------------------------------- Nearly all monitoring systems use shell scripts to extract system data, add wrappers around them, and then more and more layers. So changes in output format or differences between systems or versions break checks, so you have to constantly adapt and create multi-case-aware wrappers. MoSheL was designed to do away with the overhead and the wrappers. You can (and should) be able to work with your own commands and outputs - and correct pattern matching whenever stuff changes. If you have a nice (free) check that could be of use to other people, please send it to me so I can include it into the distribution. ----------------------------------------------------------------------- Shortcut: Distributable under GPL ----------------------------------------------------------------------- Copyright (C) 2020- Volker Tanger This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. or on their website http://www.gnu.org/copyleft/gpl.html -----------------------------------------------------------------------