You are here

Part6: Fencing with fence_virsh. My Study Notes for Red Hat Certificate of Expertise in Clustering and Storage Management Exam (EX436)

FENCING

In this example we are going to use fence_virsh not recommended for production IMPORTANT if you use fence_virsh you need to disable selinux in ENFORCING MODE:

https://access.redhat.com/site/solutions/39028

We are going to confgure fence_virsh agents to get fencing working because our host is a ubuntu server and it doesn't have the fence_libvirt multicas daemon available right now.

We are going to configure fencing on our cluster nodes using ccs from centos4:

[root@centos4 ~]# ccs -h centos --addfencedev fence_centos agent=fence_virsh ipaddr=192.168.122.1 login=liquid passwd=isa28bel option=off port=centos
[root@centos4 ~]# ccs -h centos --addfencedev fence_centos2 agent=fence_virsh ipaddr=192.168.122.1 login=liquid passwd=isa28bel option=off port=centos2
[root@centos4 ~]# ccs -h centos --addfencedev fence_centos3 agent=fence_virsh ipaddr=192.168.122.1 login=liquid passwd=isa28bel option=off port=centos3

agent=we put the agent we are going to use in out case fence_virsh
ipaddr= the ip address of the physical host
login=ssh user with priviledges to use virh on the host
passwd=pass for the user
option=on,off or reboot
port=the server we ean't to get fenced

We can check how the config file is modified:

[root@centos ~]# cat /etc/cluster/cluster.conf | grep -i fence

Now we are going to add a method(of fencing) to each of our cluster nodes, we are going to call it primary, we could add more methods(seconday for example):
[root@centos4 ~]# ccs -h centos --addmethod primary centosclu
Method primary added to centosclu.
[root@centos4 ~]# ccs -h centos --addmethod primary centos2clu
Method primary added to centos2clu.
[root@centos4 ~]# ccs -h centos --addmethod primary centos3clu
Method primary added to centos3clu.

cluster.conf:

And finally we are going to add our devices to our methods and to the nodes, so it all comes together:

[root@centos4 ~]# ccs -h centos --addfenceinst fence_centos centosclu primary
[root@centos4 ~]# ccs -h centos --addfenceinst fence_centos2 centos2clu primary
[root@centos4 ~]# ccs -h centos --addfenceinst fence_centos3 centos3clu primary

cluster.conf:

As we said before we can have several fence methods for each node, and each method with a different fence agent.

No we are going to sync and update all nodes.

[root@centos4 ~]# ccs -h centos --sync --activate
[root@centos4 ~]#

Now we can see, we have a fencing:

[root@centos3 ~]# ccs_tool lsnode

Cluster name: newcluster, config_version: 23

Nodename Votes Nodeid Fencetype
centosclu 1 1 fence_centos
centos2clu 1 2 fence_centos2
centos3clu 1 3 fence_centos3
[root@centos3 ~]#

So now we have to test it out, first with the fence_node binary, after that in a real situation:

[root@centos3 ~]# fence_node -vv centosclu
fence centosclu dev 0.0 agent fence_virsh result: success
agent args: nodename=centosclu agent=fence_virsh ipaddr=192.168.122.1 login=liquid option=off passwd=isa28bel port=centos
fence centosclu success
[root@centos3 ~]#
[root@centos3 ~]# clustat
Cluster Status for newcluster @ Mon Aug 12 17:03:46 2013
Member Status: Quorate

Member Name ID Status
------ ---- ---- ------
centosclu 1 Offline
centos2clu 2 Online
centos3clu 3 Online, Local

From the host:
[liquid@liquid-ibm:~]$ virsh list --all
Id Name State
----------------------------------
9 centos2 running
10 centos3 running
14 centos4 running
- centos shut off

Aug 12 17:03:27 centos3 fence_node[5172]: fence centosclu success
Aug 12 17:03:33 centos3 corosync[2227]: [TOTEM ] A processor failed, forming new configuration.
Aug 12 17:03:45 centos3 corosync[2227]: [QUORUM] Members[2]: 2 3
Aug 12 17:03:45 centos3 corosync[2227]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 12 17:03:45 centos3 kernel: dlm: closing connection to node 1
Aug 12 17:03:45 centos3 corosync[2227]: [CPG ] chosen downlist: sender r(0) ip(1.1.1.3) ; members(old:3 left:1)
Aug 12 17:03:45 centos3 corosync[2227]: [MAIN ] Completed service synchronization, ready to provide service.

Aug 12 17:09:00 centos3 fence_node[5217]: fence centos2clu success
Aug 12 17:09:07 centos3 corosync[2227]: [TOTEM ] A processor failed, forming new configuration.
Aug 12 17:09:19 centos3 corosync[2227]: [CMAN ] quorum lost, blocking activity
Aug 12 17:09:19 centos3 corosync[2227]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Aug 12 17:09:19 centos3 corosync[2227]: [QUORUM] Members[1]: 3
Aug 12 17:09:19 centos3 corosync[2227]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 12 17:09:19 centos3 corosync[2227]: [CPG ] chosen downlist: sender r(0) ip(1.1.1.4) ; members(old:2 left:1)
Aug 12 17:09:19 centos3 corosync[2227]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 12 17:09:19 centos3 kernel: dlm: closing connection to node 2

[root@centos3 ~]# fence_node -vv centos2clu
fence centos2clu dev 0.0 agent fence_virsh result: success
agent args: nodename=centos2clu agent=fence_virsh ipaddr=192.168.122.1 login=liquid option=off passwd=isa28bel port=centos2
fence centos2clu success
[root@centos3 ~]# clustat
Cluster Status for newcluster @ Mon Aug 12 17:09:44 2013
Member Status: Inquorate

Member Name ID Status
------ ---- ---- ------
centosclu 1 Offline
centos2clu 2 Offline
centos3clu 3 Online, Local

Now that we checked that fence_node is working, let's hang node centos3 and check how the other nodes fence it correctly:

Different ways to triger a fence situation:
cman_tool kill -n nodename.fqdn.org

1. service network stop

2. echo c > /proc/sysrq-trigger # hangs the system

3. pull the network cable on one node

We hang node3:

[root@centos3 ~]# echo c > /proc/sysrq-trigger

and check the cluster logs:

[root@centos cluster]# tail -5 corosync.log
Aug 13 19:19:19 corosync [TOTEM ] A processor failed, forming new configuration.
Aug 13 19:19:31 corosync [QUORUM] Members[2]: 1 2
Aug 13 19:19:31 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Aug 13 19:19:31 corosync [CPG ] chosen downlist: sender r(0) ip(1.1.1.2) ; members(old:3 left:1)
Aug 13 19:19:31 corosync [MAIN ] Completed service synchronization, ready to provide service.
[root@centos cluster]# tail -5 fenced.log
Aug 13 19:14:36 fenced fence centos3clu failed
Aug 13 19:19:31 fenced fencing node centos3clu
Aug 13 19:19:36 fenced fence centos3clu success

Unix Systems: 

Add new comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.

Fatal error: Class CToolsCssCache contains 1 abstract method and must therefore be declared abstract or implement the remaining methods (DrupalCacheInterface::__construct) in /homepages/37/d228974590/htdocs/sites/all/modules/ctools/includes/css-cache.inc on line 52