We just wanted to make it official: Severalnines ClusterControl now supports MariaDB 10!
As most of you know will know by now, MariaDB 10 is the newest and most advanced version of the popular MariaDB relational database system. Whilst remaining application-compatible with the MySQL database, it adds many new capabilities to address the most challenging web and enterprise application use cases. Cluster deployments would be based on MariaDB Galera Cluster 10, which is a complete merge of MariaDB 10.0.12 and Galera Cluster.
ClusterControl gives you the power to deploy, manage, monitor and scale entire MariaDB Galera clusters efficiently and reliably. ClusterControl acts as a virtual DBA assistant and frees system administrators and developers from the complexity and learning curves associated with database clusters.
We had the opportunity to showcase ClusterControl for MariaDB 10 recently during the European MariaDB Roadshow, which took us to Helsinki, Amsterdam, Paris, Hamburg, Berlin and Frankfurt. We were delighted of course to be invited by SkySQL as guest speakers and to have the opportunity to talk to existing and future MariaDB users.
There is one more date scheduled at the moment for this roadshow - London in September.
If you’ve missed the European Roadshow and/or would like a more in-depth presentation and demo of ClusterControl, please do register for our joint webinar with SkySQL on 30th September 2014.
ClusterControl uses the Apache HTTP Server to serve its web interface, but it is also possible to use nginx. nginx + PHP fastcgi is well-known for its capabilities to run on a small memory footprint compared to standard Apache + PHP DSO.
In this post, we will show you how to run ClusterControl on nginx web server by swapping out the default Apache web server installed during the initial deployment. This blog post does not mean that we officially support nginx, it just an alternative way that a portion of our users have been interested in. For instance, Phil Bayfield wrote a blog on the same topic a while back.
Before we jump into nginx configurations, let’s look at how the ClusterControl web application is configured with Apache web server. ClusterControl consists of two web components, a web UI and CMONAPI. These components are located in the Apache’s document root which might vary depending on the operating system.
1. Make sure ClusterControl UI and CMONAPI exist in the document root. Document root for RedHat/CentOS and Ubuntu 14.04 LTS (Apache 2.4) is located at /var/www/html while Debian and Ubuntu 12.04 and lower is located at /var/www. ClusterControl will be installed under this root directory and you should see something like this:
$ ls -l /var/www/html drwxr-xr-x. 6 apache apache 4096 Jul 9 16:31 cc-cmonapi-1.2.6 drwxr-xr-x. 5 apache apache 4096 Jul 9 16:31 cc-ui-1.2.6 lrwxrwxrwx. 1 root root 26 Jul 9 16:31 clustercontrol -> /var/www/html//cc-ui-1.2.6 drwxr-xr-x. 4 apache apache 4096 Jul 8 18:34 cmon lrwxrwxrwx. 1 root root 31 Jul 9 16:31 cmonapi -> /var/www/html//cc-cmonapi-1.2.6
2. Apache must be able to read custom configuration file (.htaccess) under the document root directory, by setting AllowOverride options to All. Example in /etc/httpd/conf/httpd.conf:
<Directory "/var/www/html"> Options Indexes FollowSymLinks AllowOverride All Order allow,deny Allow from all </Directory>
3. Require the following PHP modules to be installed and enabled:
The standard Apache installation via package manager will install PHP to run as dynamic shared object (DSO). Running on this mode will require you to restart Apache in case of php configuration changes.
We continuously see great interest in MySQL load balancing and HAProxy, so we thought it was about time we organised a live webinar on the topic!
As most of your will know, database clusters and load balancing go hand in hand.
Once your data is distributed and replicated across multiple database nodes, a load balancing mechanism helps distribute database requests, and gives applications a single database endpoint to connect to.
Instance failures or maintenance operations like node additions/removals, reconfigurations or version upgrades can be masked behind a load balancer. This provides an efficient way of isolating changes in the database layer from the rest of the infrastructure.
Wednesday, July 9th 2014
In this webinar, we will cover the concepts around the popular open-source HAProxy load balancer, and show you how to use it with your SQL-based database clusters. We will also discuss HA strategies for HAProxy with Keepalived and Virtual IP.
MongoDB is great at storing clickstream data, but using it to analyze millions of documents can be challenging. Hadoop provides a way of processing and analyzing data at large scale. Since it is a parallel system, workloads can be split on multiple nodes and computations on large datasets can be done in relatively short timeframes. MongoDB data can be moved into Hadoop using ETL tools like Talend or Pentaho Data Integration (Kettle).
In this blog, we’ll show you how to integrate your MongoDB and Hadoop datastores using Talend. We have a MongoDB database collecting clickstream data from several websites. We’ll create a job in Talend to extract the documents from MongoDB, transform and then load them into HDFS. We will also show you how to schedule this job to be executed every 5 minutes.
We have an application collecting clickstream data from several websites. Incoming data is mostly inserts generated from user actions against HTML Document Object Model (DOM) and stored in a MongoDB collection called domstream. We are going to bulk load our data in batch from the MongoDB collection into Hadoop (as an HDFS output file). Hadoop can then be used as a data warehouse archive on which we can perform our analytics.
For step by step instructions on how to set up your Hadoop cluster, please read this blog post. Our architecture can be illustrated as below:
Our goal is to bulk load the MongoDB data to an HDFS output file every 5 minutes. The steps are:
We’ll be using Talend Open Studio for Big Data as our ETL tool. Download and install the application on your local workstation. We'll use it to design and deploy the process workflow for our data integration project.
Extract the downloaded package and open the application. Accept the license and create a new project called Mongo2Hadoop. Choose the corresponding project and click Open. You can skip the TalendForge sign-in page and directly access the Talend Open Studio dashboard. Click on Job under Create a new section and give the job a name. We are going to use the same name with project name.
We’re delighted to be joining Team MariaDB at SkySQL with a talk on ‘Automation & Management of Database Clusters’ as they tour across several European regions in the coming weeks.
Whether you’re a MariaDB enthusiast or newbie, a MySQL veteran or newbie, or even a MongoDB user who also happens to run MySQL or MariaDB databases: this roadshow is a good place to find out about the latest developments on the MariaDB database as well as related tools and technologies.
From our own perspective, we’ll be updating participants on how to best automate & manage their database clusters, and demo some of the capabilities of ClusterControl.
The theme of this year’s MariaDB roadshow is ‘Reaching New Heights - Proven Features and Cool New Technologies’.
You will hear about the latest MariaDB innovations including how the MariaDB 10 GA release significantly increases the performance of MariaDB. You will also learn how the new MaxScale - an open source proxy platform for MariaDB and MySQL - provides you with a new tool for even better scalability.
There are six dates/locations to chose from:
If you need to deploy a database cluster behind your corporate firewall, on servers with no internet access, then this blog is for you. Most of the installation tutorials out there require servers to have an internet connection, and that would not work very well with companies under strict policies and compliance guidelines.
Our online Galera Configurator supports deployment without Internet connection for Percona XtraDB Cluster. You would need to prepare a minimum of four hosts (three DB nodes and one management node), generate a Deployment Package from the Configurator and copy it to the ClusterControl host. In this post, we will show you how to deploy Percona XtraDB Cluster 5.5 in an environment without internet connection on CentOS 6.5 (installed with minimal installation). Our setup is illustrated below:
Prior to performing the offline installation:
Mount the CentOS installation DVD using the external DVD drive on the ClusterControl host. On this host, we will configure Apache web server to host the local repository via HTTP to be accessed by the database hosts.
Thanks to everyone who attended and participated last week’s joint webinar on ClusterControl 1.2.6! We had great questions from participants (thank you), most of which are transcribed below with our answers to them.
If you missed the sessions or would like to watch the webinar again & browse through the slides, they are now available online.
Webinar topics discussed:
We won’t bore you with buzzwords like volume, velocity and variety. This post is for MySQL users who want to get their hands dirty with Hadoop, so roll up your sleeves and prepare for work. Why would you ever want to move MySQL data into Hadoop? One good reason is archival and analytics. You might not want to delete old data, but rather move it into Hadoop and make it available for further analysis at a later stage.
In this post, we are going to deploy a Hadoop Cluster and export data in bulk from a Galera Cluster using Apache Sqoop. Sqoop is a well-proven approach for bulk data loading from a relational database into Hadoop File System. There is also Hadoop Applier available from MySQL labs, which works by retrieving INSERT queries from MySQL master binlog and writing them into a file in HDFS in real-time (yes, it applies INSERTs only).
We will use Apache Ambari to deploy Hadoop (HDP 2.1) on three servers. We have a clustered Wordpress site running on Galera, and for the purpose of this blog, we will export some user data to Hadoop for archiving purposes. The database name is wordpress, we will use Sqoop to import the data to a Hive table running on HDFS. The following diagram illustrates our setup:
The ClusterControl node has been installed with an HAproxy instance to load balance Galera connections and listen on port 33306.
All hosts are running CentOS 6.5 with firewall and SElinux turned off. All servers’ time are using NTP server and synced with each other. Hostname must be FQDN or define your hosts across all nodes in /etc/hosts file. Each host has been configured with the following host definitions:
192.168.0.100 clustercontrol haproxy mysql 192.168.0.101 mysql1 galera1 192.168.0.102 mysql2 galera2 192.168.0.103 mysql3 galera3 192.168.0.111 hadoop1 hadoop1.cluster.com 192.168.0.112 hadoop2 hadoop2.cluster.com 192.168.0.113 hadoop3 hadoop3.cluster.com
Create an SSH key and configure passwordless SSH on hadoop1 to other Hadoop nodes to automate the deployment by Ambari Server. In hadoop1, run following commands as root:
$ ssh-keygen -t rsa # press Enter for all prompts $ ssh-copy-id -i ~/.ssh/id_rsa hadoop1.cluster.com $ ssh-copy-id -i ~/.ssh/id_rsa hadoop2.cluster.com $ ssh-copy-id -i ~/.ssh/id_rsa hadoop3.cluster.com
Following the release of ClusterControl 1.2.6 a couple of weeks ago, we are now looking forward to demonstrating this latest version of the product on Tuesday next week, May 13th.
This release contains key new features (along with performance improvements and bug fixes), which we will be demonstrating live during the webinar.
Tuesday, May 13th at 09:00 BST (UK) / 10:00 CEST (Germany, France, Sweden)
Tuesday, May 13th at 9:00 Pacific Time (US) / 12:00 Eastern Time (US)
Vinay Joosery, CEO, Severalnines
ClusterControl 1.2.6 introduces integration with Active Directory and LDAP authentication. This allows users to log into ClusterControl by using their corporate credentials instead of a separate password. LDAP groups can be mapped onto ClusterControl user groups to apply roles to the entire group, so it is very convenient for larger organizations who have a centralized LDAP-compliant authentication system. This blog shows you how to configure LDAP authentication in ClusterControl, and allow users to use their Active Directory or LDAP username and password to log in to ClusterControl.
LDAP authentication can be configured from ClusterControl, in the Admin dashboard (ClusterControl > Admin > LDAP Settings). If you are running ClusterControl v1.2.5 or older, please upgrade to v1.2.6. It requires PHP LDAP package to be installed and enabled. To install, you can just simply use following command:
$ yum install php-ldap
$ apt-get install php5-ldap
You may need to restart Apache web server to ensure that PHP LDAP is loaded properly. If you are upgrading your ClusterControl installation using the s9s_upgrade_cmon script, PHP LDAP will be installed automatically.
The following is a simple organization structure running on OpenLDAP: