February 3, 2014
By Severalnines

Liferay is an open-source content management system written in Java. It is used by a number of high traffic sites, as this survey suggests. 

Clustering Liferay and other components such as the database and the file system is a good way to handle the performance requirements of a high traffic site. The latest Liferay version has introduced features that simplify clustering, such as built-in support for Ehcache clustering, Lucene replication, read/write splitting capabilities for database (in case if you run on master-slave architecture) and support for various file systems for the portal repository. 

 

In this post, we are going to show you how to cluster Liferay in a multi-node load-balanced setup. The database backend will be based on Galera Cluster for MySQL, and the file system clustered using Ceph FS.

 

Architecture

 

We will have a three-node database cluster, with two of the MySQL instances co-located with Liferay portal. The third MySQL instance is co-located with ceph-admin. Another two nodes, ceph-osd0 and ceph-osd1 will be used as a storage pool for Liferay repository using CephFS. ClusterControl will be hosted on ceph-osd1. We will be using Liferay Portal 6.2 Community Edition GA1 and all hosts are running on CentOS 6.4 64bit. All commands shown below are executed in root environment. SElinux and iptables are turned off.

 

Our hosts definition in all nodes:

192.168.197.111		liferay1 galera1
192.168.197.112		lifeary2 galera2
192.168.197.113		ceph-mds ceph-admin ceph-mon1 galera3
192.168.197.114		ceph-osd0 ceph-mon2 haproxy1
192.168.197.115		ceph-osd1 ceph-mon3 haproxy2 clustercontrol

 

Deploying MySQL Galera Cluster

 

1. Use the Galera Configurator to deploy a three-node Galera Cluster. Use galera1, galera2 and galera3 for the MySQL nodes, and ceph-osd1 (192.168.197.115) for the ClusterControl node. Once deployed, create a schema for Liferay by using Manage >> Schema and Users >> Create a database called “liferay”. Then assign all privileges to the schema with a wildcard host (‘%’).

We will use the MySQL JDBC connector for load balancing and failover of database requests. 

 

Installing Ceph Storage Cluster

 

Ceph is a free clustered storage platform that provides unified object, block and file storage. It requires at least three nodes, one for the admin/monitor node and the rest for the storage cluster nodes (OSD). We’ll also add a monitor role to the storage nodes, for resiliency in case the admin node goes down. 

Please note that Ceph FS is relatively new and not yet production ready. If you’re considering this for production, you may instead want to look at another clustered file system. We covered the following in previous blog posts; GlusterFS, OCFS2, or GFS2.

Our distributed storage layer is illustrated below:

 

We’ll use a nice helper script to simplify the Ceph deployment called ceph-deploy. This script requires passwordless SSH setup between the admin node and OSD nodes. Then, we will mount it using the FUSE method on clients. 

 

The following steps should be performed on ceph-admin node.

 

1. SSH into ceph-admin and generate SSH key:

$ ssh-keygen -t rsa # press Enter for all questions

 

2. Copy the SSH key to OSDs to allow passwordless SSH:

$ ssh-copy-id -i ~/.ssh/id_rsa ceph-osd0
$ ssh-copy-id -i ~/.ssh/id_rsa ceph-osd1

 

3. Install ceph-deploy dependencies, install ceph-deploy package and create a directory for Ceph deployment:

$ yum install python-argparse python-distribute -y
$ rpm -Uhv http://ceph.com/rpm/el6/noarch/ceph-deploy-1.3.4-0.noarch.rpm
$ mkdir ~/ceph-deploy
$ cd ~/ceph-deploy

 

4. Start the Ceph deployment:

$ ceph-deploy new ceph-admin ceph-osd0 ceph-osd1
$ ceph-deploy install ceph-admin ceph-osd0 ceph-osd1
$ ceph-deploy mon create-initial

 

5. Define the public network for Ceph communication on ~/ceph-deploy/ceph.conf:

$ cat “public_network = 192.168.197.0/24>> ceph.conf

 

Then, push the configuration file to all Ceph nodes:

$ ceph-deploy --overwrite-conf config push ceph-admin
$ ceph-deploy --overwrite-conf config push ceph-osd0
$ ceph-deploy --overwrite-conf config push ceph-osd1

 

6. Create the Ceph monitor daemons:

$ ceph-deploy mon create ceph-admin ceph-osd0 ceph-osd1
$ ceph-deploy gatherkeys ceph-admin

 

7. On each OSD node, create a directory for OSD daemons:

$ ssh ceph-osd0 "mkdir /var/local/osd0"
$ ssh ceph-osd1 "mkdir /var/local/osd1"

 

8. Prepare the directory for OSDs and activate it:

$ ceph-deploy osd prepare ceph-osd0:/var/local/osd0 ceph-osd1:/var/local/osd1
$ ceph-deploy osd activate ceph-osd0:/var/local/osd0 ceph-osd1:/var/local/osd1

 

9. Copy the configuration file and admin key to all Ceph nodes so we can manage it without specifying keyring: 

$ ceph-deploy admin ceph-osd0 ceph-osd1 ceph-admin

 

10. Check the ceph deployment status. Make sure you achieve active+clean state which indicates that it has finished peering:

$ ceph -s
    cluster 8d8f1f38-ddb4-4d7b-a219-c4a886101788
     health HEALTH_WARN clock skew detected on mon.ceph-osd0, mon.ceph-osd1
     monmap e3: 3 mons at {ceph-admin=192.168.197.113:6789/0,ceph-osd0=192.168.197.114:6789/0,ceph-osd1=192.168.197.115:6789/0}, election epoch 8, quorum 0,1,2 ceph-admin,ceph-osd0,ceph-osd1
     osdmap e9: 2 osds: 2 up, 2 in
      pgmap v26: 192 pgs, 3 pools, 0 bytes data, 0 objects
            14792 MB used, 9260 MB / 25339 MB avail
                 192 active+clean

 

11. Create the metadata server on ceph-admin:

$ ceph-deploy mds create ceph-admin

 

Creating CephFS

 

Following steps should be performed on ceph-admin.

 

1. Create a separate pool for ceph client to mount for CephFS:

$ rados mkpool cephfs
$ ceph osd pool set cephfs size 2

 

2. Create and associate keyring and assign client privilege for the CephFS:

$ ceph auth get-or-create client.cephfs mon 'allow r' osd 'allow rwx pool=cephfs' -o ~/ceph-deploy/client.cephfs.keyring
$ ceph-authtool -p -n client.cephfs ~/ceph-deploy/client.cephfs.keyring > ~/ceph-deploy/client.cephfs

 

3. Copy the CephFS configuration file and keyring to liferay1 and liferay2:

$ scp ceph.conf client.cephfs client.cephfs.keyring liferay1:/etc/ceph/
$ scp ceph.conf client.cephfs client.cephfs.keyring liferay2:/etc/ceph/

 

Mounting Ceph

 

Following steps should be performed on liferay1 and liferay2.

 

1. Install ceph-fuse package:

$ yum install ceph-fuse -y

 

2. Create the mount point for CephFS and mount using FUSE:

$ mkdir /storage
$ ceph-fuse --keyring /etc/ceph/client.cephfs.keyring --name client.cephfs /storage

 

3. Verify if CephFS is mounted correctly:

$ mount | grep ceph
ceph-fuse on /storage type fuse.ceph-fuse (rw,nosuid,nodev,allow_other,default_permissions)

 

4. Add the following line into /etc/rc.local to allow automount on system startup:

cat/usr/bin/ceph-fuse --keyring /etc/ceph/client.cephfs.keyring --name client.cephfs /storage” >> /etc/rc.local

 

Installing Liferay

 

Following steps should be performed on liferay1 and liferay2 unless specified.

 

1. Liferay requires the latest Java from Oracle JDK (not OpenJDK available in yum repository). Download it by using following command:

$ wget --no-cookies --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com" "http://download.oracle.com/otn-pub/java/jdk/7u51-b13/jdk-7u51-linux-x64.rpm" --no-check-certificate -O jdk-7u51-linux-x64.rpm

 

2. Install Oracle JDK:

$ rpm -Uhv jdk-7u51-linux-x64.rpm

 

3. Check the Java version. Make sure it installs correctly:

$ java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

 

4. Download Liferay Portal Community Edition (bundled with Tomcat), install Liferay dependencies and extract the downloaded file into /home directory:

$ yum install -y unzip ImageMagick
$ wget http://downloads.sourceforge.net/project/lportal/Liferay%20Portal/6.2.0%20GA1/liferay-portal-tomcat-6.2.0-ce-ga1-20131101192857659.zip?r=http%3A%2F%2Fwww.liferay.com%2Fdownloads%2Fliferay-portal%2Favailable-releases&ts=1390908800&use_mirror=jaist
$ unzip liferay-portal-tomcat-6.2.0-ce-ga1-20131101192857659.zip -d /home

 

5. Create a symbolic link to simplify the Liferay path:

$ ln -sf /home/liferay-portal-6.2.0-ce-ga1/ /home/liferay

 

6. On liferay1, create a directory for Liferay clustered storage:

$ mkdir /storage/liferay

 

7. Create a file called portal-ext.properties to replace some default parameters in Liferay:

$ vim /home/liferay/tomcat-7.0.42/webapps/ROOT/WEB-INF/classes/portal-ext.properties

 

And add the following lines:

dl.store.file.system.root.dir=/storage/liferay/document_library
cluster.link.enabled=true
ehcache.cluster.link.replication=true
lucene.replicate.write=true
jdbc.default.driverClassName=com.mysql.jdbc.Driver
jdbc.default.url=jdbc:mysql:loadbalance://liferay1:3306,liferay2:3306,ceph-admin:3306/liferay?useUnicode=true&characterEncoding=UTF-8&useFastDateParsing=false&loadBalanceBlacklistTimeout=5000
jdbc.default.username=liferay
jdbc.default.password=liferaypassword

 

8. On liferay1, start Liferay:

$ /home/liferay/tomcat-7.0.42/bin/startup.sh

 

You can monitor the Liferay bootstrap process at /home/liferay/tomcat-7.0.42/logs/catalina.out. You will see “INFO: Server startup in 130336 ms” indicates the Liferay Portal has started.

 

9. Connect to liferay1 Tomcat server at http://192.168.197.111:8080/ and perform the initial setup. Once done, you will be redirected to a welcome page, similar to screenshot below:

 

10. Once completed, repeat step 8 for the other Liferay node (liferay2). If the bootstrapping completed successfully, both Liferay portals should run in active-active clustered mode now. You should see some activities on your database cluster, as seen from the ClusterControl dashboard:

 

Finally, you may add load balancers in front of the Liferay web apps such as HAproxy, Apache (with mod_jk, mod_rewrite and mod_proxy) or load balancing appliances.

 

References