DRBD是一个用软件实现的、无共享的、服务器之间镜像块设备内容的存储复制解决方案。
DRBD的核心功能通过Linux的内核实现，最接近系统的IO栈，但它不能神奇地添加上层的功能比如检测到EXT3文件系统的崩溃。
DRBD的位置处于文件系统以下，比文件系统更加靠近操作系统内核及IO栈。
DRBD相当于给磁盘做了一个网络的raid1。
heartbeat（Linux-HA）的工作原理：heartbeat最核心的包括两个部分，心跳监测部分和资源接管部分，心跳监测可以通过网络链路和串口进行，而且支持冗 余链路，它们之间相互发送报文来告诉对方自己当前的状态，如果在指定的时间内未收到对方发送的报文，那么就认为对方失效，这时需启动资源接管模块来接管运 行在对方主机上的资源或者服务。
DRBD Protocol参数：

ProtocolA@数据一旦写入磁盘并发送到网络中就认为完成了写入操作

ProtocolB@收到接收确认就认为完成了写入操作。

ProtocolC@收到写入确认就认为完成了写入操作。

################################################

暗紫色文字都是log或命令输出，红色部分为文件内容。

################################################

vm1 10.0.0.11 主
vm2 10.0.0.12 从
VIP: 10.0.0.20

以下操作vm1 vm2都要做：
1 软件安装：
wget http://elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
ls
rpm -ivh elrepo-release-6-6.el6.elrepo.noarch.rpm
yum search drbd
yum install drbd84 kmod-drbd84
yum install heartbeat

2 创建分区
[root@vm1 ~]# fdisk -l

Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00076c33

Device BootStart EndBlocks IdSystem
/dev/sda1 * 126204800 83Linux
Partition 1 does not end on cylinder boundary.
/dev/sda226 157 1048576 82Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3 157261119717120 83Linux

Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9aa21f28

Device BootStart EndBlocks IdSystem

[root@vm1 ~]# fdisk /dev/sdb

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-130, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-130, default 130):
Using default value 130

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@vm1 ~]# partprobe
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).As a result, it may not reflect all of your changes until after reboot.
[root@vm1 ~]#

3 创建DRBD配置文件
[root@vm1 ~]# cat /etc/drbd.conf

global { usage-count yes; }
common { syncer { rate 10M; } }
resource db {
protocol C;
net {
}
on vm1 {
device/dev/drbd1;
disk/dev/sdb1;
address 10.0.0.11:7789;
meta-diskinternal;
}
on vm2 {
device/dev/drbd1;
disk/dev/sdb1;
address 10.0.0.12:7789;
meta-diskinternal;
}
}


以上操作vm1 vm2都要做。

4 创建DBRD资源
[root@vm1 ~]# drbdadm create-md db##db是DRBD配置文件中的resource名字
You want me to create a v08 style flexible-size internal meta data block.
There appears to be a v08 flexible-size internal meta data block
already in place on /dev/sdb1 at byte offset 1069248512

Do you really want to overwrite the existing meta-data?
[need to type 'yes' to confirm] yes

md_offset 1069248512
al_offset 1069215744
bm_offset 1069182976

Found ext3 filesystem
1044124 kB data area apparently used
1044124 kB left usable by current configuration

Even though it looks like this would place the new meta data into
unused space, you still need to confirm, as this is only a guess.

Do you want to proceed?
[need to type 'yes' to confirm] yes

initializing activity log
NOT initializing bitmap
Writing meta data...
New drbd meta data block successfully created.
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11
[root@vm1 ~]#

[root@vm2 ~]# drbdadm create-md db
You want me to create a v08 style flexible-size internal meta data block.
There appears to be a v08 flexible-size internal meta data block
already in place on /dev/sdb1 at byte offset 1069248512

Do you really want to overwrite the existing meta-data?
[need to type 'yes' to confirm] yes

md_offset 1069248512
al_offset 1069215744
bm_offset 1069182976

Found ext3 filesystem
1044124 kB data area apparently used
1044124 kB left usable by current configuration

Even though it looks like this would place the new meta data into
unused space, you still need to confirm, as this is only a guess.

Do you want to proceed?
[need to type 'yes' to confirm] yes

initializing activity log
NOT initializing bitmap
Writing meta data...
New drbd meta data block successfully created.
[root@vm2 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11
[root@vm2 ~]#

5 启动DRBD
[root@vm1 ~]# /etc/init.d/drbd start
Starting DRBD resources: [
create res: db
prepare disk: db
adjust disk: db
adjust net: db
]
..........
***************************************************************
DRBD's startup script waits for the peer node(s) to appear.
- If this node was already a degraded cluster before the
reboot, the timeout is 0 seconds. [degr-wfc-timeout]
- If the peer was available before the reboot, the timeout
is 0 seconds. [wfc-timeout]
(These values are for resource 'db'; 0 sec -> wait forever)
To abort waiting enter 'yes' [12]: yes

.
[root@vm1 ~]# ls /dev/drbd*
/dev/drbd1

/dev/drbd:
by-diskby-res
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----s
ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:1044124

[root@vm2 ~]#/etc/init.d/drbd start
Starting DRBD resources: [
create res: db
prepare disk: db
adjust disk: db
adjust net: db
]
.
[root@vm2 ~]# ls /dev/drbd*
/dev/drbd1

/dev/drbd:
by-diskby-res

[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:105528 nr:0 dw:0 dr:106200 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:938596
[=>..................] sync'ed: 10.2% (938596/1044124)K
finish: 0:00:44 speed: 21,104 (21,104) K/sec

6 设置vm1为primary
[root@vm1 ~]# drbdsetup /dev/drbd1 primary --force=yes
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:105528 nr:0 dw:0 dr:106200 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:938596
[=>..................] sync'ed: 10.2% (938596/1044124)K
finish: 0:00:44 speed: 21,104 (21,104) K/sec
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:1044124 nr:0 dw:0 dr:1044796 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

7 建立文件系统，并挂载。
[root@vm1 ~]# mkfs.ext4 /dev/drbd1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65280 inodes, 261031 blocks
13051 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8160 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 31 mounts or
180 days, whichever comes first.Use tune2fs -c or -i to override.
[root@vm1 ~]#
[root@vm1 ~]# mount /dev/drbd1 /drbd/
[root@vm1 ~]# df -h
FilesystemSizeUsed Avail Use% Mounted on
/dev/sda319G9.7G7.9G56% /
tmpfs 246M 0246M 0% /dev/shm
/dev/sda1 190M 53M128M30% /boot
/dev/drbd1988M1.3M936M 1% /drbd

8 挂载测试；
vm1上：
[root@vm1 ~]# cd /drbd/
[root@vm1 drbd]# touch aa
[root@vm1 drbd]# umount /drbd/
umount: /drbd: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
[root@vm1 drbd]# cd
[root@vm1 ~]# cd
[root@vm1 ~]# umount /drbd/
[root@vm1 ~]# drbdadm secondary db
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:1077352 nr:0 dw:33228 dr:1045509 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@vm1 ~]#

切换到vm2上验证：
[root@vm2 dev]# drbdadm primary db
[root@vm2 dev]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:1077352 dw:1077352 dr:672 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@vm2 dev]# mkdir /drbd
[root@vm2 dev]# mount /dev/drbd1 /drbd/
[root@vm2 dev]# cd /drbd/
[root@vm2 drbd]# ls
aalost+found
[root@vm2 drbd]#

9 切换回vm1:
切换，将vm1设为主：

[root@vm2 ~]# umount /drbd/
[root@vm2 ~]# drbdadm secondary db
[root@vm2 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:36 nr:1077352 dw:1077388 dr:1025 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

[root@vm1 ~]# drbdadm primary db
[root@vm1 ~]# mount /dev/drbd1 /drbd/
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:1077356 nr:36 dw:33268 dr:1046530 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

10 配置heartbeat.
[root@vm1 ~]# cp /usr/share/doc/heartbeat-2.1.3/haresources /etc/ha.d/
[root@vm1 ~]# cp /usr/share/doc/heartbeat-2.1.3/ha.cf /etc/ha.d/
[root@vm1 ~]# cp /usr/share/doc/heartbeat-2.1.3/authkeys /etc/ha.d/
[root@vm1 ~]# chmod 600 /etc/ha.d/authkeys

VIP 10.0.0.20

设置通信秘钥
[root@vm1 ~]# cat /etc/ha.d/authkeys| egrep -v "^#"
auth 3
3 md5 Hello!

创建heartbeat配置文件
[root@vm1 ~]# grep -v "^#" /etc/ha.d/ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 5
warntime 5
initdead 120
ucast eth0 10.0.0.12
auto_failback off
nodevm1
nodevm2
ping10.0.0.1

创建mysql管理脚本，仅为简单说明问题
[root@vm1 ~]# cat /etc/ha.d/resource.d/manage_mysql
#!/bin/bash

case$1 in

'start')

/usr/local/mysql/bin/mysqld_safe --defaults-file=/drbd/mysql3307/my.cnf &
;;

'stop')
if [[ `ps -ef | grep mysqld | wc -l` <2 ]] ;then
exit
else
mysqladmin -S /tmp/mysql3307.sockshutdown
fi
;;

*)
;;
esac

[root@vm1 ~]# chmod +x /etc/ha.d/resource.d/manage_mysql
定义heartbeat管理资源
[root@vm1 ~]# grep -v "^#" /etc/ha.d/haresources
vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql

[root@vm1 ~]# scp -r /etc/ha.d/ vm2:/etc

修改vm2的/etc/ha.d/ha.cf内容如下
[root@vm2 ha.d]# grep -v "^#" /etc/ha.d/ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 5
warntime 5
initdead 120
ucast eth0 10.0.0.11
auto_failback off
nodevm1
nodevm2
ping10.0.0.1

11 启动验证
先启动DRBD
[root@vm1 ~]# /etc/init.d/drbd start
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:1452 nr:1148 dw:2600 dr:26342 al:9 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

[root@vm2 resource.d]# /etc/init.d/drbd start
[root@vm2 resource.d]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:1412 nr:6112 dw:7524 dr:24838 al:9 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

启动heartbeat：
[root@vm1 ~]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO:Resource is stopped
Done.
[root@vm2 ~]#/etc/init.d/heartbeat start
Starting High-Availability services: INFO:Resource is stopped
Done.

检查log
[root@vm1 ~]# tail -20f /var/log/ha-log
Apr 11 17:19:58 vm1 heartbeat: [7734]: info: Pacemaker support: false
Apr 11 17:19:58 vm1 heartbeat: [7734]: WARN: Logging daemon is disabled --enabling logging daemon is recommended
Apr 11 17:19:58 vm1 heartbeat: [7734]: info: **************************
Apr 11 17:19:58 vm1 heartbeat: [7734]: info: Configuration validated. Starting heartbeat 3.0.4
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: heartbeat: version 3.0.4
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: Heartbeat generation: 1460298765
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: bound send socket to device: eth0
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: set SO_REUSEPORT(w)
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: bound receive socket to device: eth0
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: set SO_REUSEPORT(w)
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: started on port 694 interface eth0 to 10.0.0.12
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ping heartbeat started.
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: G_main_add_TriggerHandler: Added signal manual handler
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: G_main_add_TriggerHandler: Added signal manual handler
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: Local status now set to: 'up'
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: Link 10.0.0.1:10.0.0.1 up.
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: Status update for node 10.0.0.1: status ping
Apr 11 17:20:36 vm1 heartbeat: [7735]: info: Link vm2:eth0 up.
Apr 11 17:20:36 vm1 heartbeat: [7735]: info: Status update for node vm2: status up
harc(default)[7747]:2016/04/11_17:20:36 info: Running /etc/ha.d//rc.d/status status
Apr 11 17:20:37 vm1 heartbeat: [7735]: info: Comm_now_up(): updating status to active
Apr 11 17:20:37 vm1 heartbeat: [7735]: info: Local status now set to: 'active'
Apr 11 17:20:37 vm1 heartbeat: [7735]: info: Status update for node vm2: status active
harc(default)[7768]:2016/04/11_17:20:37 info: Running /etc/ha.d//rc.d/status status
Apr 11 17:20:48 vm1 heartbeat: [7735]: info: remote resource transition completed.
Apr 11 17:20:48 vm1 heartbeat: [7735]: info: remote resource transition completed.
Apr 11 17:20:48 vm1 heartbeat: [7735]: info: Initial resource acquisition complete (T_RESOURCES(us))
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[7821]:2016/04/11_17:20:48 INFO:Resource is stopped
Apr 11 17:20:48 vm1 heartbeat: [7785]: info: Local Resource acquisition completed.
harc(default)[7887]:2016/04/11_17:20:49 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp(default)[7887]: 2016/04/11_17:20:49 received ip-request-resp IPaddr::10.0.0.20 OK yes
ResourceManager(default)[7910]: 2016/04/11_17:20:49 info: Acquiring resource group: vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[7938]:2016/04/11_17:20:49 INFO:Resource is stopped
ResourceManager(default)[7910]: 2016/04/11_17:20:49 info: Running /etc/ha.d/resource.d/IPaddr 10.0.0.20 start
IPaddr(IPaddr_10.0.0.20)[8034]: 2016/04/11_17:20:49 INFO: Adding inet address 10.0.0.20/24 with broadcast address 10.0.0.255 to device eth0
IPaddr(IPaddr_10.0.0.20)[8034]: 2016/04/11_17:20:49 INFO: Bringing device eth0 up
IPaddr(IPaddr_10.0.0.20)[8034]: 2016/04/11_17:20:49 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.0.0.20 eth0 10.0.0.20 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[8020]:2016/04/11_17:20:49 INFO:Success
ResourceManager(default)[7910]: 2016/04/11_17:20:49 info: Running /etc/ha.d/resource.d/drbddisk db start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[8162]: 2016/04/11_17:20:49 INFO:Resource is stopped
ResourceManager(default)[7910]: 2016/04/11_17:20:49 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /drbd ext4 start
Filesystem(Filesystem_/dev/drbd1)[8249]:2016/04/11_17:20:49 INFO: Running start for /dev/drbd1 on /drbd
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[8241]: 2016/04/11_17:20:49 INFO:Success
ResourceManager(default)[7910]: 2016/04/11_17:20:49 info: Running /etc/ha.d/resource.d/manage_mysqlstart


[root@vm2 ~]# tail -20f /var/log/ha-log
Apr 11 17:20:36 vm2 heartbeat: [16250]: info: Pacemaker support: false
Apr 11 17:20:36 vm2 heartbeat: [16250]: WARN: Logging daemon is disabled --enabling logging daemon is recommended
Apr 11 17:20:36 vm2 heartbeat: [16250]: info: **************************
Apr 11 17:20:36 vm2 heartbeat: [16250]: info: Configuration validated. Starting heartbeat 3.0.4
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: heartbeat: version 3.0.4
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: Heartbeat generation: 1460351868
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: bound send socket to device: eth0
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: set SO_REUSEPORT(w)
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: bound receive socket to device: eth0
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: set SO_REUSEPORT(w)
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: started on port 694 interface eth0 to 10.0.0.11
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ping heartbeat started.
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: G_main_add_TriggerHandler: Added signal manual handler
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: G_main_add_TriggerHandler: Added signal manual handler
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: Local status now set to: 'up'
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: Link 10.0.0.1:10.0.0.1 up.
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: Status update for node 10.0.0.1: status ping
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: Link vm1:eth0 up.
Apr 11 17:20:37 vm2 heartbeat: [16251]: info: Status update for node vm1: status active
Apr 11 17:20:37 vm2 heartbeat: [16251]: info: Comm_now_up(): updating status to active
Apr 11 17:20:37 vm2 heartbeat: [16251]: info: Local status now set to: 'active'
harc(default)[16261]: 2016/04/11_17:20:37 info: Running /etc/ha.d//rc.d/status status
Apr 11 17:20:48 vm2 heartbeat: [16251]: info: local resource transition completed.
Apr 11 17:20:48 vm2 heartbeat: [16251]: info: Initial resource acquisition complete (T_RESOURCES(us))
Apr 11 17:20:48 vm2 heartbeat: [16280]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys vm2] to acquire.
Apr 11 17:20:48 vm2 heartbeat: [16251]: info: remote resource transition completed.

检查drbd、磁盘挂载、mysql进程信息：
vm1上：
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:2004 nr:1148 dw:3152 dr:38227 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@vm1 ~]# df -h
FilesystemSizeUsed Avail Use% Mounted on
/dev/sda319G 11G7.5G58% /
tmpfs 246M 0246M 0% /dev/shm
/dev/sda1 190M 53M128M30% /boot
/dev/drbd1988M178M760M19% /drbd

[root@vm1 ~]#ps -ef | grep mysql
root8337 10 17:20 ?00:00:00 /bin/sh /usr/local/mysql/bin/mysqld_safe --defaults-file=/drbd/mysql3307/my.cnf
mysql 859983370 17:20 ?00:00:00 /usr/local/mysql/bin/mysqld --defaults-file=/drbd/mysql3307/my.cnf --basedir=/usr/local/mysql --datadir=/drbd/mysql3307 --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=/drbd/mysql3307/vm1.err --pid-file=/drbd/mysql3307/vm1.pid --socket=/tmp/mysql3307.sock --port=3307
root863318740 17:24 pts/300:00:00 grep mysql

[root@vm1 ~]# mysql -uroot -pmysql -h20.0.0.20 -P3307
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.27-log Source distribution

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> Ctrl-C -- exit!
Aborted

vm2上：
[root@vm2 resource.d]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:1412 nr:6664 dw:8076 dr:24838 al:9 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@vm2~]#df -h
FilesystemSizeUsed Avail Use% Mounted on
/dev/sda319G 11G7.5G58% /
tmpfs 246M 0246M 0% /dev/shm
/dev/sda1 190M 53M128M30% /boot
[root@vm2 ~]#ps -ef | grep mysql
root 16304 118100 17:24 pts/200:00:00 grep mysql
[root@vm2 resource.d]#

[root@vm2 resource.d]# mysql -uroot -pmysql -h20.0.0.20 -P3307
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.27-log Source distribution

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> Ctrl-C -- exit!
Aborted
[root@vm2 resource.d]#

关掉vm1,模拟故障：
[root@vm1 ~]# init 0

[root@vm2 ~]# tail -20f /var/log/ha-log
Apr 11 17:27:00 vm2 heartbeat: [16251]: info: Received shutdown notice from 'vm1'.
Apr 11 17:27:00 vm2 heartbeat: [16251]: info: Resources being acquired from vm1.
Apr 11 17:27:00 vm2 heartbeat: [16310]: info: acquire local HA resources (standby).
Apr 11 17:27:00 vm2 heartbeat: [16310]: info: local HA resource acquisition completed (standby).
Apr 11 17:27:00 vm2 heartbeat: [16251]: info: Standby resource acquisition done [all].
Apr 11 17:27:00 vm2 heartbeat: [16311]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys vm2] to acquire.
harc(default)[16336]: 2016/04/11_17:27:00 info: Running /etc/ha.d//rc.d/status status
mach_down(default)[16353]:2016/04/11_17:27:00 info: Taking over resource group IPaddr::10.0.0.20
ResourceManager(default)[16380]:2016/04/11_17:27:00 info: Acquiring resource group: vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[16408]: 2016/04/11_17:27:00 INFO:Resource is stopped
ResourceManager(default)[16380]:2016/04/11_17:27:00 info: Running /etc/ha.d/resource.d/IPaddr 10.0.0.20 start
IPaddr(IPaddr_10.0.0.20)[16504]:2016/04/11_17:27:00 INFO: Adding inet address 10.0.0.20/24 with broadcast address 10.0.0.255 to device eth0
IPaddr(IPaddr_10.0.0.20)[16504]:2016/04/11_17:27:00 INFO: Bringing device eth0 up
IPaddr(IPaddr_10.0.0.20)[16504]:2016/04/11_17:27:00 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.0.0.20 eth0 10.0.0.20 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[16490]: 2016/04/11_17:27:00 INFO:Success
ResourceManager(default)[16380]:2016/04/11_17:27:00 info: Running /etc/ha.d/resource.d/drbddisk db start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[16632]:2016/04/11_17:27:01 INFO:Resource is stopped
ResourceManager(default)[16380]:2016/04/11_17:27:01 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /drbd ext4 start
Filesystem(Filesystem_/dev/drbd1)[16719]: 2016/04/11_17:27:01 INFO: Running start for /dev/drbd1 on /drbd
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[16711]:2016/04/11_17:27:01 INFO:Success
ResourceManager(default)[16380]:2016/04/11_17:27:01 info: Running /etc/ha.d/resource.d/manage_mysqlstart
mach_down(default)[16353]:2016/04/11_17:27:01 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default)[16353]:2016/04/11_17:27:01 info: mach_down takeover complete for node vm1.
Apr 11 17:27:01 vm2 heartbeat: [16251]: info: mach_down takeover complete.
Apr 11 17:27:07 vm2 heartbeat: [16251]: WARN: node vm1: is dead
Apr 11 17:27:07 vm2 heartbeat: [16251]: info: Dead node vm1 gave up resources.
Apr 11 17:27:07 vm2 heartbeat: [16251]: info: Link vm1:eth0 dead.

[root@vm2 ~]# ps -ef | grep mysql
root 16808 10 17:27 ?00:00:00 /bin/sh /usr/local/mysql/bin/mysqld_safe --defaults-file=/drbd/mysql3307/my.cnf
mysql17085 168080 17:27 ?00:00:00 /usr/local/mysql/bin/mysqld --defaults-file=/drbd/mysql3307/my.cnf --basedir=/usr/local/mysql --datadir=/drbd/mysql3307 --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=/drbd/mysql3307/vm2.err --pid-file=/drbd/mysql3307/vm2.pid --socket=/tmp/mysql3307.sock --port=3307
root 17112 118100 17:28 pts/200:00:00 grep mysql

[root@vm2 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
ns:1820 nr:6856 dw:8760 dr:36747 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:84
[root@vm2 ~]# df -h
FilesystemSizeUsed Avail Use% Mounted on
/dev/sda319G 11G7.5G58% /
tmpfs 246M 0246M 0% /dev/shm
/dev/sda1 190M 53M128M30% /boot
/dev/drbd1988M178M760M19% /drbd

[root@vm2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:9b:2c:f6 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.12/24 brd 10.0.0.255 scope global eth0
inet 10.0.0.20/24 brd 10.0.0.255 scope global secondary eth0
inet6 fe80::a00:27ff:fe9b:2cf6/64 scope link
valid_lft forever preferred_lft forever

[root@vm2 ~]#mysql -uroot -pmysql -h20.0.0.20 -P3307
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.27-log Source distribution

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

VIP、磁盘资源、mysql进程都已在vm2上。

完！

########################################

后记

资源文件配置解释：

[root@vm1 ~]# grep -v "^#" /etc/ha.d/haresources
vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql

vm1 主服务器的hostname,后面都是由heartbeat负责管理的资源。

IPaddr::10.0.0.20VIP

drbddisk::dbDRBD资源，db为drbd.conf中resource的名字

Filesystem::/dev/drbd1::/drbd::ext4需要挂载的文件系统、挂载点、文件类型

manage_mysql系统或自己定义的资源管理文件，需要能配合start\stop参数。一般位于/etc/init.d或/etc/ha.d/resource.d目录下，

这个manage_mysql是我自己写的。

执行/etc/init.d/heartbeat start命令后，主节点的/var/log/ha-log内容如下：

ResourceManager(default)[2006]: 2016/04/11_18:05:35 info: Acquiring resource group: vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[2052]:2016/04/11_18:05:35 INFO:Resource is stopped
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[2061]:2016/04/11_18:05:35 INFO:Resource is stopped
Apr 11 18:05:35 vm1 heartbeat: [1980]: info: Local Resource acquisition completed.
ResourceManager(default)[2006]: 2016/04/11_18:05:35 info: Running /etc/ha.d/resource.d/IPaddr 10.0.0.20 start
IPaddr(IPaddr_10.0.0.20)[2218]: 2016/04/11_18:05:35 INFO: Adding inet address 10.0.0.20/24 with broadcast address 10.0.0.255 to device eth0
IPaddr(IPaddr_10.0.0.20)[2218]: 2016/04/11_18:05:35 INFO: Bringing device eth0 up
IPaddr(IPaddr_10.0.0.20)[2218]: 2016/04/11_18:05:35 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.0.0.20 eth0 10.0.0.20 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[2204]:2016/04/11_18:05:35 INFO:Success
ResourceManager(default)[2006]: 2016/04/11_18:05:35 info: Running /etc/ha.d/resource.d/drbddisk db start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[2348]: 2016/04/11_18:05:35 INFO:Resource is stopped
ResourceManager(default)[2006]: 2016/04/11_18:05:35 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /drbd ext4 start
Filesystem(Filesystem_/dev/drbd1)[2434]:2016/04/11_18:05:35 INFO: Running start for /dev/drbd1 on /drbd
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[2426]: 2016/04/11_18:05:35 INFO:Success
ResourceManager(default)[2006]: 2016/04/11_18:05:35 info: Running /etc/ha.d/resource.d/manage_mysqlstart
Apr 11 18:05:35 vm1 heartbeat: [1979]: info: all HA resource acquisition completed (standby).
Apr 11 18:05:35 vm1 heartbeat: [1590]: info: Standby resource acquisition done [all].
harc(default)[2523]:2016/04/11_18:05:35 info: Running /etc/ha.d//rc.d/status status
mach_down(default)[2560]: 2016/04/11_18:05:35 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default)[2560]: 2016/04/11_18:05:36 info: mach_down takeover complete for node vm2.
Apr 11 18:05:36 vm1 heartbeat: [1590]: info: mach_down takeover complete.
harc(default)[2626]:2016/04/11_18:05:36 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp(default)[2626]: 2016/04/11_18:05:36 received ip-request-resp IPaddr::10.0.0.20 OK yes
ResourceManager(default)[2680]: 2016/04/11_18:05:36 info: Acquiring resource group: vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[2741]:2016/04/11_18:05:36 INFO:Running OK
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[2929]: 2016/04/11_18:05:36 INFO:Running OK
ResourceManager(default)[2680]: 2016/04/11_18:05:36 info: Running /etc/ha.d/resource.d/manage_mysqlstart
Apr 11 18:05:41 vm1 heartbeat: [1590]: WARN: node vm2: is dead
Apr 11 18:05:41 vm1 heartbeat: [1590]: info: Dead node vm2 gave up resources.
Apr 11 18:05:41 vm1 heartbeat: [1590]: info: Link vm2:eth0 dead.

#######################

在主节点执行/etc/init.d/heartbeat stop命令后，/var/log/ha-log内容如下：

ResourceManager(default)[17275]:2016/04/11_18:05:31 info: Running /etc/ha.d/resource.d/manage_mysqlstop
ResourceManager(default)[17275]:2016/04/11_18:05:33 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /drbd ext4 stop
Filesystem(Filesystem_/dev/drbd1)[17343]: 2016/04/11_18:05:33 INFO: Running stop for /dev/drbd1 on /drbd
Filesystem(Filesystem_/dev/drbd1)[17343]: 2016/04/11_18:05:33 INFO: Trying to unmount /drbd
Filesystem(Filesystem_/dev/drbd1)[17343]: 2016/04/11_18:05:33 INFO: unmounted /drbd successfully
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[17335]:2016/04/11_18:05:33 INFO:Success
ResourceManager(default)[17275]:2016/04/11_18:05:33 info: Running /etc/ha.d/resource.d/drbddisk db stop
ResourceManager(default)[17275]:2016/04/11_18:05:34 info: Running /etc/ha.d/resource.d/IPaddr 10.0.0.20 stop
IPaddr(IPaddr_10.0.0.20)[17492]:2016/04/11_18:05:34 INFO: IP status = ok, IP_CIP=
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[17473]: 2016/04/11_18:05:34 INFO:Success
Apr 11 18:05:34 vm2 heartbeat: [17262]: info: All HA resources relinquished.
Apr 11 18:05:35 vm2 heartbeat: [16251]: WARN: 1 lost packet(s) for [vm1] [64:66]
Apr 11 18:05:35 vm2 heartbeat: [16251]: info: No pkts missing from vm1!
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: killing HBFIFO process 16254 with signal 15
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: killing HBWRITE process 16255 with signal 15
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: killing HBREAD process 16256 with signal 15
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: killing HBWRITE process 16257 with signal 15
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: killing HBREAD process 16258 with signal 15
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: Core process 16254 exited. 5 remaining
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: Core process 16256 exited. 4 remaining
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: Core process 16255 exited. 3 remaining
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: Core process 16258 exited. 2 remaining
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: Core process 16257 exited. 1 remaining
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: vm2 Heartbeat shutdown complete.

可以看到启动时，资源是按照haresources文件中定义的顺序从前到后依次执行。
关闭时，资源是按照haresources文件中定义的顺序从后到前依次释放。