cmon eval blog dimensioning toolkit

Cluster Config

Configuration Tool for MySQL Cluster - version 1.1

The configuration tool is below all this initial text, so please read and scroll down!

Take me there now!

Read this first

The configuration tool lets you define a production quality MySQL Cluster and:

  • propose configuration files (config.ini and my.cnf
  • generate scripts for administration of MySQL Cluster from a single point
  • email the scripts in a tarball to you
  • It is important that you follow the instructions and don't make typos.

    The configuration files are proposal and you might have to make additions to them depending on the type of load and queries you are running. In particular, the scripts are intended to be very simple so that you can modify them as you wish.

    A set of scripts are also generated to let you:

  • download and compile the latest version of MySQL Cluster
  • distribute the Cluster software to the hosts that you have specified
  • bootstrap the cluster in order to create necessary directories and install the mysql server(s)
  • start and stop the cluster, as well as starting and stopping individual nodes
  • Please note that this has only been tested on Linux-based systems and you are strongly adviced to setup shared SSH keys in order to avoid typing the password a lot of times. You also need to have root credentials to take advantage of some features (lock memory to avoid swapping etc).

    The scripts are intended to be executed on one of the nodes in the cluster, or another node connected to the cluster. This node is denoted as the front-end node.

    Read the documention for more info about the scripts.

    MySQL Cluster 6.2 - specific

    You will have to download the MySQL Cluster 6.2 for your target platform. A link where to get the MySQL CLuster 6.2 is provided in the email containing the configuration files and the scripts.

    MySQL Cluster 6.3 - specific

    The generated scripts will download and build the latest MySQL Cluster 6.3 release, since it is only delivered in source format at the time of writing.

    The download and compile scripts requires that the following is installed on front-end node (thus, this will likely not work on Solaris):

  • gcc
  • bash
  • wget
  • same OS (kernel + CPU arch) since the scripts requires a homogenous cluster
  • Current Limiations and Important notes

  • Max one mysql server per computer is allowed
  • DiskPageBufferMemory and SharedGlobalMemory should be increased if using disk data. Please read my blog about disk data and this one (more to come) and use the dimensioning toolkit to help you scope out the config.
  • I have not tried this on localhost. If you want to run on localhost only, please use the eval/development guide for this.
  • You must have root priviliges to run this. The scripts will create a "mysql" user account. The mysql servers will run as the mysql user. The data nodes as user "root".
  • News in version 1.1

    You can now get a complete system with process management and monitoring setup in minutes!

  • initd scripts - you can install initd scripts on the hosts with the script install_initd_linux.sh (yes, only tested on Linux)
  • cmon scripts - you can install cmon (as well as initd scripts for it). If CMON is used with monit below, CMON will also chose the best way to recover a data node in case of failures.
  • monit - one script to build and install monit. Another to add it to inittab.
  • Check out the new directory structure here and learn about the scripts
  • News in version 1.0

  • Replication - you can define whether you cluster should be Master or Slave in geo replication. However, scripts for auto starting replication is not yet ready (2.0)
  • Multi-core - you can deploy several data nodes on one computer. Recommendation is to have 2 cores for each data node.
  • MySQL client - scripts for accessing the mysqlds are now generated (mysqlclient-.sh.
  • Please let me know about usability issues - johan at severalnines dot com
  • Disclaimer: Please note that the scripts are not Supported by MySQL nor Sun Microsystems!!

    Configure tool

    Cluster version: We recommend MySQL Cluster 6.3 for general purpose.
    If you have an old kernel (like linux 2.4) then you must currently use 6.2, as 6.3 use epoll and does not work in the 2.4 kernel. The main differences between 6.2 and 6.3 is that 6.3 has better support for Realtime and the geo-redundancy supports Conflict Detection and Resolution. Read here for more information about the differences.

    There are prebuilt binaries for both versions, but the source releases are made more frequently.

    Number of management servers Two management servers are recommended in a HA setup
    Number of Data Nodes: Each data node requires two cores. You should have atleast two computers (for two data nodes). Two or four computers (depending on the number of cores in each computer) for four data nodes. One data node is only for testing purpose only.
    Number of APIs (MySQLD): You should have atleast two MySQLD for a HA setup. The sum of nodes (management servers, data nodes and apis) must be less than 255 (max number of supported nodes in Cluster 6.2 and Cluster 6.3).
    DataMemory: MB The OS will require about 1GB of RAM. Approximately 900MB will be used by internal buffers in the Data node. The IndexMemoryis derived from the DataMemory and will be 12.5% of the DataMemory. This is enough for most applications since only the hash primary key is stored there (20B per record in the database). However, if you have a lot of UNIQUE indexes then you might have to increase this later (just edit the config.ini file). The minimum value for DataMemory is 80MB.

    Thus, if you have 8GB of RAM, then you could set
    DataMemory= 0.875*(8172MB-1024MB-900MB) = 5400MB.
    Number of tables (optional - not implemented yet): The default setup will support 4096 tables objects. Each INDEX (and UNIQUE) and BLOB/TEXT attribute also counts towards a table object. If you plan to run more, then you should think about this and set this to an approximate. The maximum number of table objects supported in MySQL Cluster is 20320. Each table object takes about 20KB of RAM.
    Replication role:
  • None - the cluster will not be used in replication
  • Master - the cluster will act as the replication master (one or more mysqlds will produce binary logs)
  • Slave - the cluster will act as a standby cluster