Mysql MHA部署-手动切换

架构说明：

参考：http://www.zhaibibei.cn/mysql/mha/

1 检查从库现有状态(188,223)

---show slave status\G

2 查看管理节点日志(222)

通过如下命令事实查看MHA当前状态

tail -f /etc/mha/manager/mha.log

3 关闭MHA的管理进程(222)

[root@rac4 ~]# masterha_stop -conf=/etc/mha/mha.conf

MHA Manager is not running on mha(2:NOT_RUNNING).

4.手动切换

相关命令如下：

masterha_master_switch -master_state=alive –orig_master_is_new_slave –conf=/etc/mha/mha.conf

-master_state=alive 代表告诉MHA原master还是存活的，不需要将其从配置文件删除

–orig_master_is_new_slave 参数代表原master会自动同步新的master

--还有一些其他的参数如下

-running_updates_limit 如果主库的写操作时间超过了该参数，则退出切换

–interactive=0 代表直接确认，不需要输入YES

切换过程如下：

[root@rac4 ~]# masterha_master_switch -master_state=alive -orig_master_is_new_slave -conf=/etc/mha/mha.conf

Sat Mar 14 22:17:37 2020 - [info] MHA::MasterRotate version 0.56.

Sat Mar 14 22:17:37 2020 - [info] Starting online master switch..

Sat Mar 14 22:17:37 2020 - [info]

Sat Mar 14 22:17:37 2020 - [info] * Phase 1: Configuration Check Phase..

Sat Mar 14 22:17:37 2020 - [info]

Sat Mar 14 22:17:37 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Sat Mar 14 22:17:37 2020 - [info] Reading application default configuration from /etc/mha/mha.conf..

Sat Mar 14 22:17:37 2020 - [info] Reading server configuration from /etc/mha/mha.conf..

Sat Mar 14 22:17:38 2020 - [info] GTID failover mode = 1

Sat Mar 14 22:17:38 2020 - [info] Current Alive Master: rac1(192.168.2.187:3306)

Sat Mar 14 22:17:38 2020 - [info] Alive Slaves:

Sat Mar 14 22:17:38 2020 - [info] rac2(192.168.2.188:3306) Version=5.7.28-log (oldest major version between slaves) log-bin:enabled

Sat Mar 14 22:17:38 2020 - [info]GTID ON

Sat Mar 14 22:17:38 2020 - [info]Replicating from 192.168.2.187(192.168.2.187:3306)

Sat Mar 14 22:17:38 2020 - [info]Primary candidate for the new Master (candidate_master is set)

Sat Mar 14 22:17:38 2020 - [info] rac3(192.168.2.223:3306) Version=5.7.28-log (oldest major version between slaves) log-bin:enabled

Sat Mar 14 22:17:38 2020 - [info]GTID ON

Sat Mar 14 22:17:38 2020 - [info]Replicating from 192.168.2.187(192.168.2.187:3306)

Sat Mar 14 22:17:38 2020 - [info]Not candidate for the new Master (no_master is set)

It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on rac1(192.168.2.187:3306)? (YES/no): YES

Sat Mar 14 22:19:01 2020 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..

Sat Mar 14 22:19:01 2020 - [info] ok.

Sat Mar 14 22:19:01 2020 - [info] Checking MHA is not monitoring or doing failover..

Sat Mar 14 22:19:01 2020 - [info] Checking replication health on rac2..

Sat Mar 14 22:19:01 2020 - [info] ok.

Sat Mar 14 22:19:01 2020 - [info] Checking replication health on rac3..

Sat Mar 14 22:19:01 2020 - [info] ok.

Sat Mar 14 22:19:01 2020 - [info] Searching new master from slaves..

Sat Mar 14 22:19:01 2020 - [info] Candidate masters from the configuration file:

Sat Mar 14 22:19:01 2020 - [info] rac1(192.168.2.187:3306) Version=5.7.28-log log-bin:enabled

Sat Mar 14 22:19:01 2020 - [info]GTID ON

Sat Mar 14 22:19:01 2020 - [info] rac2(192.168.2.188:3306) Version=5.7.28-log (oldest major version between slaves) log-bin:enabled

Sat Mar 14 22:19:01 2020 - [info]GTID ON

Sat Mar 14 22:19:01 2020 - [info]Replicating from 192.168.2.187(192.168.2.187:3306)

Sat Mar 14 22:19:01 2020 - [info]Primary candidate for the new Master (candidate_master is set)

Sat Mar 14 22:19:01 2020 - [info] Non-candidate masters:

Sat Mar 14 22:19:01 2020 - [info] rac3(192.168.2.223:3306) Version=5.7.28-log (oldest major version between slaves) log-bin:enabled

Sat Mar 14 22:19:01 2020 - [info]GTID ON

Sat Mar 14 22:19:01 2020 - [info]Replicating from 192.168.2.187(192.168.2.187:3306)

Sat Mar 14 22:19:01 2020 - [info]Not candidate for the new Master (no_master is set)

Sat Mar 14 22:19:01 2020 - [info] Searching from candidate_master slaves which have received the latest relay log events..

Sat Mar 14 22:19:01 2020 - [info]

From:

rac1(192.168.2.187:3306) (current master)

+--rac2(192.168.2.188:3306)

+--rac3(192.168.2.223:3306)

To:

rac2(192.168.2.188:3306) (new master)

+--rac3(192.168.2.223:3306)

+--rac1(192.168.2.187:3306)

Starting master switch from rac1(192.168.2.187:3306) to rac2(192.168.2.188:3306)? (yes/NO): yes

Sat Mar 14 22:19:32 2020 - [info] Checking whether rac2(192.168.2.188:3306) is ok for the new master..

Sat Mar 14 22:19:32 2020 - [info] ok.

Sat Mar 14 22:19:32 2020 - [info] rac1(192.168.2.187:3306): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host.

Sat Mar 14 22:19:32 2020 - [info] rac1(192.168.2.187:3306): Resetting slave pointing to the dummy host.

Sat Mar 14 22:19:32 2020 - [info] ** Phase 1: Configuration Check Phase completed.

Sat Mar 14 22:19:32 2020 - [info]

Sat Mar 14 22:19:32 2020 - [info] * Phase 2: Rejecting updates Phase..

Sat Mar 14 22:19:32 2020 - [info]

Sat Mar 14 22:19:32 2020 - [info] Executing master ip online change script to disable write on the current master:

Sat Mar 14 22:19:32 2020 - [info] /etc/mha/script/master_ip_online_change --command=stop --orig_master_host=rac1 --orig_master_ip=192.168.2.187 --orig_master_port=3306 --orig_master_user='monitor' --orig_master_password='123456' --new_master_host=rac2 --new_master_ip=192.168.2.188 --new_master_port=3306 --new_master_user='monitor' --new_master_password='123456' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave

Sat Mar 14 22:19:32 2020 756531 Set read_only on the new master.. ok.

Sat Mar 14 22:19:32 2020 761639 Set read_only=1 on the orig master.. ok.

Sat Mar 14 22:19:32 2020 763740 Killing all application threads..

Sat Mar 14 22:19:32 2020 763785 done.

Disabling the VIP an old master: rac1

eth0:2: ERROR while getting interface flags: No such device

Sat Mar 14 22:19:33 2020 - [info] ok.

Sat Mar 14 22:19:33 2020 - [info] Locking all tables on the orig master to reject updates from everybody (including root):

Sat Mar 14 22:19:33 2020 - [info] Executing FLUSH TABLES WITH READ LOCK..

Sat Mar 14 22:19:33 2020 - [info] ok.

Sat Mar 14 22:19:33 2020 - [info] Orig master binlog:pos is mysql-bin.000009:194.

Sat Mar 14 22:19:33 2020 - [info] Waiting to execute all relay logs on rac2(192.168.2.188:3306)..

Sat Mar 14 22:19:33 2020 - [info] master_pos_wait(mysql-bin.000009:194) completed on rac2(192.168.2.188:3306). Executed 0 events.

Sat Mar 14 22:19:33 2020 - [info] done.

Sat Mar 14 22:19:33 2020 - [info] Getting new master's binlog name and position..

Sat Mar 14 22:19:33 2020 - [info] mysql-bin.000006:3938

Sat Mar 14 22:19:33 2020 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='rac2 or 192.168.2.188', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';

Sat Mar 14 22:19:33 2020 - [info] Executing master ip online change script to allow write on the new master:

Sat Mar 14 22:19:33 2020 - [info] /etc/mha/script/master_ip_online_change --command=start --orig_master_host=rac1 --orig_master_ip=192.168.2.187 --orig_master_port=3306 --orig_master_user='monitor' --orig_master_password='123456' --new_master_host=rac2 --new_master_ip=192.168.2.188 --new_master_port=3306 --new_master_user='monitor' --new_master_password='123456' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave

Sat Mar 14 22:19:33 2020 227880 Set read_only=0 on the new master.

Enabling the VIP 192.168.2.189 on the new master: rac2

SIOCSIFADDR: No such device

eth0:2: ERROR while getting interface flags: No such device

SIOCSIFNETMASK: No such device

arping: Device eth0 not available.

Sat Mar 14 22:19:33 2020 - [info] ok.

Sat Mar 14 22:19:33 2020 - [info]

Sat Mar 14 22:19:33 2020 - [info] * Switching slaves in parallel..

Sat Mar 14 22:19:33 2020 - [info]

Sat Mar 14 22:19:33 2020 - [info] -- Slave switch on host rac3(192.168.2.223:3306) started, pid: 12424

Sat Mar 14 22:19:33 2020 - [info]

Sat Mar 14 22:19:34 2020 - [info] Log messages from rac3 ...

Sat Mar 14 22:19:34 2020 - [info]

Sat Mar 14 22:19:33 2020 - [info] Waiting to execute all relay logs on rac3(192.168.2.223:3306)..

Sat Mar 14 22:19:33 2020 - [info] master_pos_wait(mysql-bin.000009:194) completed on rac3(192.168.2.223:3306). Executed 0 events.

Sat Mar 14 22:19:33 2020 - [info] done.

Sat Mar 14 22:19:33 2020 - [info] Resetting slave rac3(192.168.2.223:3306) and starting replication from the new master rac2(192.168.2.188:3306)..

Sat Mar 14 22:19:33 2020 - [info] Executed CHANGE MASTER.

Sat Mar 14 22:19:33 2020 - [info] Slave started.

Sat Mar 14 22:19:34 2020 - [info] End of log messages from rac3 ...

Sat Mar 14 22:19:34 2020 - [info]

Sat Mar 14 22:19:34 2020 - [info] -- Slave switch on host rac3(192.168.2.223:3306) succeeded.

Sat Mar 14 22:19:34 2020 - [info] Unlocking all tables on the orig master:

Sat Mar 14 22:19:34 2020 - [info] Executing UNLOCK TABLES..

Sat Mar 14 22:19:34 2020 - [info] ok.

Sat Mar 14 22:19:34 2020 - [info] Starting orig master as a new slave..

Sat Mar 14 22:19:34 2020 - [info] Resetting slave rac1(192.168.2.187:3306) and starting replication from the new master rac2(192.168.2.188:3306)..

Sat Mar 14 22:19:34 2020 - [info] Executed CHANGE MASTER.

Sat Mar 14 22:19:34 2020 - [info] Slave started.

Sat Mar 14 22:19:34 2020 - [info] All new slave servers switched successfully.

Sat Mar 14 22:19:34 2020 - [info]

Sat Mar 14 22:19:34 2020 - [info] * Phase 5: New master cleanup phase..

Sat Mar 14 22:19:34 2020 - [info]

Sat Mar 14 22:19:34 2020 - [info] rac2: Resetting slave info succeeded.

Sat Mar 14 22:19:34 2020 - [info] Switching master to rac2(192.168.2.188:3306) completed successfully.

5.切换阶段

执行master_ip_online_change时当前master不可写

新的主库设置只读

旧的主库设置只读

禁用原主库VIP

旧主库设全局锁

获取新主库master信息

新主库设置VIP

新主库取消只读

异步从库重新同步至新主库

原主库释放全局锁

原主库同步至新主库

6.查看切换后状态

---223

mysql> show slave status \G;

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.2.188

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: mysql-bin.000006

Read_Master_Log_Pos: 3938

Relay_Log_File: mysql-relay.000005

Relay_Log_Pos: 454

Relay_Master_Log_File: mysql-bin.000006

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

----187

mysql> show slave status \G;

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: rac2

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: mysql-bin.000006

Read_Master_Log_Pos: 3938

Relay_Log_File: mysql-relay.000007

Relay_Log_Pos: 454

Relay_Master_Log_File: mysql-bin.000006

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Replicate_Do_DB:

Replicate_Ignore_DB:

......

7.数据测试

---188：

mysql> use jumptest

mysql> insert into cjc01 values(3000);

Query OK, 1 row affected (0.02 sec)

mysql> select * from cjc01;

+------+

| id |

+------+

| 100 |

|3 |

| 3000 |

+------+

3 rows in set (0.00 sec)

---187：

mysql> select * from cjc01;

+------+

| id |

+------+

| 100 |

|3 |

| 3000 |

+------+

3 rows in set (0.01 sec)

---223：

mysql> select * from cjc01;

+------+

| id |

+------+

| 100 |

|3 |

| 3000 |

+------+

3 rows in set (0.01 sec)