Upgrading Corosync and Pacemaker From Squeeze To Wheezy

Configuration

Go into the cluster resource manager

Set the cluster in maintanance mode

Commit the change

Exit the cluster resource manager

Upgrade

Now we can upgrade the distribution from squeeze to whezzy.

Post Process

Reinstall pacemaker, this is neccesary because the pacemaker package was removed on the upgrade process.

If both nodes are upgraded, restart corosync on both nodes

Remove the maintanance mode, go into the cluster resource manager





Remove maintanance mode

Commit the change

Exit the cluster resource manager

Using csync2 with inotifywait

Installation

Configuration




Csync2 Cluster Synchronization Tool

Installation

Configuration

Create shared-key on master server

Add csync2 with default port 30865 to /etc/services on all servers

Create key and certificate on master server

Setup the csync2 configuration on master server (for more information see csync2 manual)

Copy csync2 keys and configuration to the slave servers

Usage





Create xinted csync configuration on master server

Create xinted csync configuration on slave servers

Retart xinetd on all servers

Start synchronization on master server

Now you can add the command to crontab

Ldirectord troubles with HTTPS negotiate check

If you defined an https negotiate check in your ldirectord configuration and you are using self-made certificates on the real servers, the internal check_http function of ldirectord fails because the perl LWP HTTPS module has changed the ssl certificat verification.

We can use the external check type and create a custom https check script, like this

This script checks only if a hello.htm file exists on the real webserver, it don’t matter what string is into the file.




Change ower and permission

Change check type to external in the ldirectord.cfg

Configure Pacemaker fencing and stonith with Hyper-V Driver

Notes

  • exlb1 => Cluster Node A on Host hyperv10
  • exlb2 => Cluster Node B on Host hyperv11
  • hyperv10 => Hyper-V Host
  • hyperv11 => Hyper-V Host

Requirements

Packages

Install requirements on the Debian nodes (exlb1 and exlb2)

WinRM

The Windows Remote Managment should be running on the Hyper-V host, recomanded is also a configured HTTPS transport, see KB2019527. Currently only Basic authentication is supported by libvirt. This method is disabled by default on the Hyper-V server and can be enabled via the WinRM commandline tool.

You need also to enable unencrypted authentication (only if HTTPS for WinRM is not enabled)

Installation




openwsman

Download openwsman from http://sourceforge.net/projects/openwsman/files/.
Extract tarball

Change into source directory

Prepare for building the libraries

Change into the build directory

Configure

Compile and install

Add openwsman to ldconfig

Reload ldconfig

libvirt

Download libvirt http://libvirt.org/sources/libvirt-1.1.1.tar.gz




Extract tarball

Change into source directory

Prepare configure

Configure

Compile and install

Configuration

Configure the credentials, on Debian node exlb2 for Hyper-V host hyperv10

Configure the credentials, on Debian node exlb1 for Hyper-V host hyperv11

Make auth.conf readonly by root

Check the libvirt auth page for more information.

Test

Show version of virsh

Test secure connection

Test insecure connection

Integration in Pacemaker

We need to change the “version” check in the stonith plugin of libvirt, the “version” parameter is not supported yet by libvirt:libvirt API support matrix. So we need to change “version” with another command, I am using “list”:

For some reason, the libvirt stonith plugin doesn’t find our libvirt binary in /usr/local/bin with the which command (I don’t know why). So quick and dirty, make a symbol link in /usr/bin

Now we can configure the stonith device in our cluster configuration

Go in the configure section

Use the command edit to add options with vi

Commit changes

Exit cluster resource manager

Create High-Available Loadbalancer with Pacemaker

Overview Loadbalancer

Notes

Read first: Installing Pacemaker and Corosync

  • lb1 => Cluster Node A
  • lb2 => Cluster Node B
  • web1 => Webserver 1, 192.168.1.30
  • web2 => Webserver 2, 192.168.1.31
  • web3 => Webserver 3, 192.168.1.32
  • web4 => Webserver 4, 192.168.1.33

Installation

Install ldirectord

Configuration

Enable ip forwarding

Reload parameters





Remove ldirectord from init.d because Pacemaker will mange it

Configure virtual and real servers

Create a dummy network card with the virtual IP on each real webserver

Configure arp settings on each real webserver

Reload

Create the negotiate page on on each real webserver

Integration in Pacemaker

Create virtual ip and ldirectord resource

Usage

Now you can check the ldirector status with

 

Create High-Available FTP Server with Pacemaker

Notes

First read Installing Pacemaker and Corosync, Create High-Available DRBD Device with Pacemaker and Create High-Available NFS Server with Pacemaker

Installation

Create a ssl certificate for tls connections

Configuration





Configure some parameters

Remove UnixAuthentication method

Activate PureDB authentification

Create a user, needed for our pure-ftpd virtual users

Edit init script to make it lsb compatible and add this at line 106

Start pure-ftpd for testing

Stop pure-ftpd and remove from init.d

Integration in Pacemaker

Use the Cluster Resource Manager tool to manage cluster resources on lb1 or lb2

If You have following the tutorials Installing Pacemaker and Corosync, Create High-Available DRBD Device with Pacemaker and Create High-Available NFS Server with Pacemaker You should have this status

Go into the configuration section

Configure resource

Update the created group services, You need to type edit on the configuration section and add resFTP manualy, I don’t find a way to alter the group from crm command line.

Commit changes

Exit configuration section

Show status

Create High-Available NFS Server with Pacemaker

Notes

Read first Installing Pacemaker and Corosync and Create High-Available DRBD Device with Pacemaker.

Installation

Remove nfs-kernel-server and nfs-common from init.d on lb1 and lb2



Move the pipefs working directory, this directory should not be on the shared drbd device on lb1 and lb2

Mount drbd device and move nfs directory on it on lb1

Remove nfs directory on lb2

Create symbolic link for the nfs directory on lb1 and lb2

Make sure that the permission are right on lb1 and lb2

Edit the nfs-common init script to configure our new rpc_pipefs directory on lb1 and lb2

Also in the id mapping daemon config on lb1 and lb2

Edit nfs-common to bind rpc ports to fixed value on node1 and node2 (192.168.33.190 is our virtual ip address that we will setup later)

Also in nfs-kernel-server on lb1 and lb2

We need also to configure fixed ports for the rpc lock daemon module on lb1 and lb2 this change needs a reboot

Configuration

Create some exports on lb1

Copy export file from lb1 to lb2

Ingeration in Pacemaker

Use the Cluster Resource Manager tool again to add the nfs resources on lb1 or lb2

Go into the configuration section

Add nfs-kernel-server and nfs-common resources

Update our services group, You need to type edit on the configuration section and add resNFScommon and resNFSserver manualy, I don’t find a way to alter the group from crm command line.

Commit changes

Exit configuration section

Check status on lb1 or lb2

Create High-Available DRBD Device with Pacemaker

Notes

Please read first Installing Pacemaker and Corosync

Installation

Install DRBD on lb1 and lb2

 Configuration

Edit DRBD global configuration on lb1

Create DRBD resource on lb1



Copy DRBD configuration and resource from lb1 to lb2

Create drbd metadata for resource data on lb1 and lb2

Start DRBD on lb1 and lb2

Promote lb1 to master, this will overwrite the drbd data on lb2

Check the process of synchronisation on lb1

Create filesystem on resource data on lb1

Mount device to test and umount it on node1 Finaly create on lb1 and lb2 the mount directory

Create same mountpoint on secondary node on lb2

Stop DRBD and remove it from init.d on lb1 and lb2

Integration in Pacemaker

Use the Cluster Resource Manager tool to create and manage cluster resources on lb1 or lb2

Go to the configuration section with

Configure DRBD resource

Configure DRBD Master/Slave

Configure filesystem mountpoint

Configure group (for future resources)

Configure colocations

Configure order

Configure location for lb1

Commit changes

Configuration overview

Exit the configuration section

Show status

Exit Cluster Resource Manager

Installing Pacemaker and Corosync

Notes

Pacemaker is the cluster resource manager (CRM), it recives informations from the communication layer (CCM) and it manages resources and executes events. Corosync is responsible for the cluster communication, called cluster communication manager (CCM), the alternative product of corosync is heartbeat.

Pacemaker and Corosync

In this documentation the primary pacemaker server is called lb1 (loadbalancer1) and the slave pacemaker server lb2 (loadbalancer2). The operation system on both server are debian wheezy, it should be working also on the newest debian release jessie.

Installation

Install pacemaker and corosync, don’t forget to run apt-get update befor you install new packages.

Configuration





Generate private key on lb1 to ensure authenticity and privacy of the messages in the cluster communication. The command below writes the private key in /etc/corosync/authkey. This private key must be copied to every node in the cluster, in our case lb1 (already generated) and lb2. If the key isn’t the same for every node, those nodes with nonmatching key will not able to join the cluster.

Copy the privat key to every node, in our case from lb1 to lb2.

The corosync configuration is stored in /etc/corosync/corosync.conf, the configuration must be the same on every node. We need only to modify the setting bindnetaddr on lb1 with the network interface that we should use, I use always a dedicated network interface. We can also add other network interfaces to guarantee the redundancy. Here an example for the dedicated network interface configuration:

lb1

lb2

Edit the corosync configuration and make the changes.

Here an example configuration for two network interface

Copy the corosync configuration from lb1 to lb2

Edit default corosync init.d configuration on lb1 and lb2 to allow the init.d start.

Start corosync on lb1 and lb2

Now we can use the cluster resource manager tool to manage cluster resources on lb1 or lb2.

Check the status and verify if both nodes are online

Type configure in the crm console to enter in the configuration section

Pacemaker default behavior is to stop all resources if the cluster doesn’t have quorum. A cluster is said to have quorum when more than half nodes are online, this will working fine in a three-nodes or more cluster, but not in a two-nodes cluster like in our case. Therefore a two-node cluster only has quorum when both nodes are running, this makes no sense, so we need to disable quorum.

Stonith is the pacemaker fencing implementation, we skip the stonith configuration for the moment, you can read this documentation for a fancing implementation with a Micro$oft Hyper-V host Configure Pacemaker fencing and stonith with Hyper-V Driver, for more information read Fencing and Stonith. Set stonith to false

Commit the changes

Show configuration

Leave the configuration section and quit the cluster resource manager

Check also the ring status