CMON - a Monitoring Tool for MySQL Cluster

Download Installation Troubleshooting Roadmap Bugs Documentation Professional edition Screenshots Launchpad!

CMON is a daemon that aggregates information from MySQL Cluster that earlier was only accessible from the cluster log or the management client, such as:

cluster state
node state
backup statistics
statistics
cluster events (cluster log basically)

.. and let's you access the information using SQL, because CMON logs the information into ordinary MYISAM tables! So, it is really easy to use! In the package you also get get php scripts that you can put on your webserver to generate graphs and get a www interface to CMON. CMON can also start ndbd nodes and make decisions on how they should be started (with or without --initial).

CMON starts as a daemon and will automatically create cmon database and install the necessary tables automatically.

Community version

Stay on the edge - released often and with new features. There is also a Professional Edition.

Download - source dist only (latest first)

Version 0.15beta -Tested on Linux (RH, Ubuntu, Fedora, Suse) - (release notes, rev. 85 and onwards) - released 4th of May 2009 - download source dist. To upgrade - drop database cmon, then restart cmon v0.15b

Version 0.14 -Tested on Linux (RH, Ubuntu, Fedora, Suse) - (release notes to come) - released 27th of March 2009 - download source dist. To upgrade - drop database cmon, then restart cmon v0.14

Version 0.13 -Tested on Linux (RH, Ubuntu, Fedora, Suse) - (release notes) - released 28th of Jan 2009 - download source dist. You will have to execute this sql script if you are upgrading from an earlier version. You also need a free [MYSQLD] slot (check with ndb_mgm -e "show") for cmon to connect to.

Version 0.12 (versioning restared..)- Tested on Linux (RH, Ubuntu, Fedora, Suse) - (release notes) - released 12th of Dec 2008 - download source dist

Version 1.2.1 - Tested on Linux (RH, Ubuntu) - (change log) - released 7th of Oct 2008 - Don't user any longer

Download - Solaris 10 (sparc 64 bit) only

Because building on Solaris is more painful than i could ever expect here are some precompiled binaried. Please be careful which version you download!

Version 1.2.2 - for MySQL Cluster 6.2.15 - released 7th of Oct 2008 - download binary (don't forget to set LD_LIBRARY_PATH!)

Version 1.2.2 - for MySQL Cluster 6.3.17 - released 7th of Oct 2008 - download binary (don't forget to set LD_LIBRARY_PATH!)

Installation

Pre-requisites

MySQL Cluster 6.2 or MySQL Cluster 6.3 installed and started

In order to make it simple it is also recommended to have:

ndb_mgmd running on localhost (where cmon is going to be installed)

mysqld running on localhost (where cmon is going to be installed)

data nodes can be somewhere else or also on localhost

one free [mysqld] slot. If you do ndb_mgm -e show or sh ./show.sh you must have (example)


  ...

  [mysqld(API)]  X node(s)

  id=4@127.0.0.1  (mysql-5.1.30 ndb-6.3.20)

  id=5@127.0.0.1  (mysql-5.1.30 ndb-6.3.20)

  id=6 (not connected, accepting connect from localhost)

cmon

mysql_config on the PATH

autoconf, automake, libtool (e.g apt-get install autoconf apt-get install automake apt-get install libtool

compile and start cmon

Make sure you have the correct mysql_config on the path. If you have used the config tool (and the default install path):

Make sure you have: g++ and gcc installed (run those two commands from the command line). If they don't exists you have to install them!!

export PATH=/usr/local/mysql/mysql/bin:$PATH

or ..another popular PATH is

export PATH=/usr/local/mysql/bin:$PATH

Proceed with the following when you can execute the command mysql_config (then your PATH is good):

tar xvfz cmon-0.1.tgz

cd cmon-0.1

sh autogen.sh

./configure

On Solaris (you need Sun studio):

CC=cc-5.0 CXX=CC ./configure

You can also do (from old versioning is 1.2)

./configure --with-wwwroot=<path to www>

standard wwwroots for various linux distributions are:

Suse : /srv/www/htdocs

Red Hat : /var/www/

Fedora core 9 : /var/www/html

Debian/Ubuntu : /var/www/

BSD : /usr/local/www/

then you have to make it and install cmon

make

sudo make install

You can also do (from old versioning is 1.2)

sudo make install-extra

this will install /etc/init.d/cmon (but you have to to chkconfig..see below) and opy the www/cmon files to /var/www/cmon/ by default. You can specify the location of your www-root with ./configure -with-wwwroot=/my/www/root

Make sure you set the LD_LIBRARY_PATH to point to libmysqlclient_r and libndbclient. If you have used the config tool (and the default install path):

export LD_LIBRARY_PATH=/usr/local/mysql/mysql/lib:/usr/local/mysql/mysql/lib/mysql

If you don't set the LD_LIBRARY_PATH to where libmysqlclient_r.so.16 is, then you get the following error message when starting cmon:

cmon: error while loading shared libraries: libmysqlclient_r.so.16: cannot open shared object file: No such file or directory

cmon --help

cmon

This is equivalent of starting 'cmon' with:

cmon --mysqluser='root' --mysqlpasswd='' --mysqlhost='localhost' --mysqlport=3306  --mysqlsocket='/tmp/mysql.sock' --ndb-connectstring='localhost' &

if you don't have the mysqld on localhost you might have to GRANT and change the above to point to the mysqld you wish to use.

To check that everything works you can do:

tail -100f /var/log/messages

If all is ok then the following is printed:

23 21:02:13 stingray cmon: Starting cmon version 0.0.0 with the following parameters:  --mysqldatabase=cmon  --mysqluser=root  --mysqlpasswd=  --mysqlhost=localhost  --mysqlport=3306  --mysqlsocket=/tmp/mysql.sock  --ndb-connectstring=localhost  --savetime-clusterlog=24 (hours)  --savetime-statistics=24 (hours)

 23 21:02:13 stingray cmon: If that doesn't look correct, kill cmon and restart with -? for help on the parameters, or change the params in /etc/init.d/cmon

 23 21:02:13 stingray cmon: Table 'backup' did not exist. Schema incomplete. Going to recreate missing tables.

 23 21:02:13 stingray cmon: Recreated missing tables

 23 21:02:13 stingray cmon: Created table 'backup'

 23 21:02:13 stingray cmon: Created table 'backup_log'

...

 23 21:02:13 stingray cmon: Created table 'node_statistics'

 23 21:02:13 stingray cmon: Done

...

 23 21:02:13 stingray cmon: MonitorThread: MGM node id = 1

 23 21:02:13 stingray cmon: MonitorThread: NDBD node 2 is STARTED

 23 21:02:13 stingray cmon: MonitorThread: NDBD node 3 is STARTED

...

 23 21:02:13 stingray cmon: MonitorThread: Cluster STARTED   - all nodes are started


If you get problems then you should check:

 GRANTs  -  can the user you want connect and is the user allowed to create/drop tables?

 --mysqlsocket - perhaps you have the socket somewhere else than in /tmp/mysql.sock?

 --mysqlport - perhaps you use another port than 3306?

 --mysqlhost - perhaps you have the mysql server that cmon should log to on another host?

Don't worry if you see the following in the log:

Sep 26 16:43:12 myhost cmon: LogEventThread: line: 481, code: 1 msg: Read error.

Sep 26 16:43:12 myhost cmon: LogEventThread: line: 481, code: 3 msg:
Unknown event type.

Those messages are due to bugs in the mgmapi that is generating the events cmon reads.

If the above does not help then please send me an email: johan@severalnines.com

install www (php scripts)

There are php scripts in the www/cmon directory:

cd www/cmon

Download jpgraph, a graph drawing toolkit for php into www/cmon/

wget http://hem.bredband.net/jpgraph/jpgraph-1.26.tar.gz

Unpack jpgraph

tar xvfz jpgraph-1.26.tar.gz

if you have jpgraph installed somewhere else already, change in cmon_graph.inc.php. I know this isn't great but will be fixed better. If someone can let me know how to deal with PHP libraries then I would be happy.

copy www/cmon to your web servers www-root. E.g, my www-root is in /var/www/:

cp -r cmon-0.1/www/cmon /var/www/

Hopefully you have apache2, php and php-mysql installed and ready to go.

If not you have to install them. On my Ubuntu I do, but similar packages should exist for Redhat, Centos etc.:

apt-get install apache2

apt-get install libapache2-mod-php5

apt-get install php5

apt-get install php5-mysql

apt-get install php5-gd

on fedora (yum based stuff)

yum install php

yum install php-gd

yum install php-mysql

Restart the webserver!!

/etc/init.d/httpd restart

If you have the apache server on another host than where cmon and the mysqld that cmon uses for logging then you must change a few things (mysql host, username, password etc) in www/cmon/cmon_db_conf.inc.php

cat cmon_db_conf.inc.php:

$mysql_username = "root";

$mysql_password = "";

$mysql_hostname= "localhost";

Here is a recipe how to change $mysql_hostname


/*If you have mysql installed on another port etc then you may have to change to:

// Connect to the server at hostname using the default port

$mysql_hostname="hostname"

// Connect to the server at hostname using the specified port

$mysql_hostname="hostname:port"

// Connect to the server on the local machine using the provided local socket

$mysql_hostname=":/path/to/socket"

*/

install initd script

In cmon-x.y/initd/ is a initd script called cmon

To install it:

sudo cp cmon /etc/init.d/

On Redhat (and I presume Fedora, OpenSuse, Centos etc):

sudo chkconfig --add cmon

On Ubuntu/Debian:

sudo update-rc.d cmon start 99 2 3 4 5 . stop 20 2 3 4 5 .

On Solaris I don't know..

Troubleshooting

Web interface- "Unable to connect to MySQL"

A common problem is that you may see the message "Unable to connect to MySQL". This is very likely to be because that the mysqld and the apache is on localhost. Hence the apache will (when executing php scripts) try to connect to the mysqld using sockets. The mysql_connect will by default contact the mysqld on e.g /var/.../mysql.sock. If you have your socket somewhere else you need to change the following:

cd $wwwroot

edit cmon_db_conf.inc.php

change $mysql_hostname="localhost" to $mysql_hostname = ":/tmp/mysql.sock"

Similarily, you might have to change the hostname and/or port if the mysql server is somewhere else.

Graphs

If you have problems with graphs test this script (save as test.php and put in wwwroot, e.g /usr/local/apache/htdocs/cmon or /var/www/cmon or /var/www/html/cmon)

<?php phpinfo();?>

and then load the page http://127.0.0.1/test.php then you should see something about gd (GD Support --> enabled). You should also see in the 'gd' section something about PNG Support --> enabled.

If you don't see it you have to install GD support in php:

i haven't tested these ones, but there seem to be some RPMs for this at rpmfind.net and for Ubuntu I have found php5-gd package by issuing the command sudo apt-cache search gd |grep php. But I have built mine from source so I don't know how well the above works.

Memory

If you don't see any memory events (e.g, your Index/DataMemory graphs are empty then you have probably not set MemReportFrequency.

[ndbd default]

....

MemReportFrequency=5

....

Restart the management server and then the data nodes (either one by one or by a complete restart.

Error while loading shared libraries libmysqlclient_r.so.16

If you get the error above when starting cmon, mysql client or whatever then you should:

export LD_LIBRARY_PATH=/path/to/mysql/lib/:$LD_LIBRARY_PATH

e.g:

export LD_LIBRARY_PATH=/usr/local/mysql/mysql//lib/mysql:/usr/local/mysql/mysql/lib/:$LD_LIBRARY_PATH

Roadmap

Alert using emails when e.g a node crashes or DataMemory/IndexMemory reaches a threshold
Monitoring of replication links - note, no link failover, just monitoring
Monitoring of buffers (RedoBuffer etc) when that statistics is accessible from cluster
Configuration of cmon from web interface
Embed with ConfigTool (so cmon is a part there) and complete with process management - this will happen very very soon!

Bugs

CMON has a place on launchpad and there is a bug tracking tool there.
Please use it.

Documentation

Parameters

name	description
--logfile=destination/logfilename	cmon will write a logfile to destination/logfilename. The log is rotating and cmon keep five log files of 1MB each. When cmon is restarted it will overwrite the old log files. cmon will not log to syslog if this parameter is enabled. You must have write permission in the directory where you want cmon to write the log.

Other

statistics is delayed with about 10sec. This is because how the events flow from the data nodes to cmon, and from cmon to the cmon database.

Professional edition

cmon is GPL, but if you write proprietary code or want to use cmon commercially, then we wish you acquire a commercial license for cmon. Please contact sales@severalnines.com for more information regarding payment details. In the commercial version you will get binaries, upgrades and updates sent to you. You can also influence the roadmap.

License cost:

1 year per CMON instance - 495USD, 450EUR, 4995SEK

3 years - please contact sales@severalnines.com

Perpetual license - please contact sales@severalnines.com

OEM customers - please contact sales@severalnines.com