• ADADADADAD

    MYSQL主从复制部署MHA自动故障转移[ mysql数据库 ]

    mysql数据库 时间:2024-12-24 19:11:18

    作者:文/会员上传

    简介:

    一. 环境描述操作系统:RHEL 6.4 mysql :5.6.27角色IP 地址主机名Server ID类型主库192.168.222.156oradb10写入备库、MHA管理服务器192.168.222.157mysqldb211读、监控集群组

    以下为本文的正文内容,内容仅供参考!本站为公益性网站,复制本文以及下载DOC文档全部免费。

    一. 环境描述

    操作系统:RHEL 6.4

    mysql :5.6.27

    角色

    IP 地址

    主机名

    Server ID

    类型

    主库

    192.168.222.156

    oradb

    10

    写入

    备库、MHA管理服务器

    192.168.222.157

    mysqldb2

    11

    读、监控集群组

    二、部署MHA

    1.主库安装 MHA NODE

    1.1 安装需要的PERL模块

    yum -y install *zlib-dev*

    install.sh

    #!/bin/bash

    wget http://xrl.us/cpanm--no-check-certificate

    mv cpanm /usr/bin

    chmod 755 /usr/bin/cpanm

    cat >/root/list<<EOF

    install DBD::mysql

    EOF

    for package in `cat /root/list`

    do

    cpanm $package

    done




    1.2 安装 MHA NODE

    从 https://github.com/yoshinorim/mha4mysql-node 下载最新版本的 mha4mysql-node 源代码(目前是 0.57版)

    unzip mha4mysql-node-master.zip

    cd mha4mysql-node-master

    perl Makefile.PL

    make

    make install

    2. 备库安装 MHA Manager

    2.1 安装需要的perl模块

    yum -y install *zlib-dev*

    install.sh

    #!/bin/bash

    wget http://xrl.us/cpanm--no-check-certificate

    mv cpanm /usr/bin

    chmod 755 /usr/bin/cpanm

    cat >/root/list<<EOF

    install DBD::mysql

    install Config::Tiny

    install Log::Dispatch

    install Parallel::ForkManager

    install Time::HiRes

    EOF

    for package in `cat /root/list`

    do

    cpanm $package

    done

    2.2 安装MHA NODE

    从 https://github.com/yoshinorim/mha4mysql-node 下载最新版本的 mha4mysql-node 源代码(目前是 0.57版)

    unzip mha4mysql-node-master.zip

    cd mha4mysql-node-master

    perl Makefile.PL

    make

    make install

    2.3 安装 MHA Manager

    从 https://github.com/yoshinorim/mha4mysql-manager下载最新版本的 mha4mysql-manager 源代码(目前是 0.57版)

    unzip mha4mysql-manager-master.zip

    cd mha4mysql-manager-master

    perl Makefile.PL

    make

    make install

    3. 配置 SSH 登录无密码验证

    3.1 主库配置到 MHA manager 节点(备库节点)SSH登录无密码验证

    vi /etc/hosts

    192.168.222.157 mysqldb2

    192.168.222.156 oradb

    ssh-keygen -t rsa

    ssh-coyp-id -i /root/.ssh/id_rsa.pub root@192.168.222.157

    ssh-coyp-id -i /root/.ssh/id_rsa.pub root@192.168.222.156

    ssh root@192.168.222.156

    ssh root@oradb

    ssh root@192.168.222.157

    ssh root@mysqldb2

    3.2 MHA manager 节点(备库节点)配置到主库配置 SSH登录无密码验证

    vi /etc/hosts

    192.168.222.157 mysqldb2

    192.168.222.156 oradb

    ssh-keygen -t rsa

    ssh-coyp-id -i /root/.ssh/id_rsa.pub root@192.168.222.157

    ssh-coyp-id -i /root/.ssh/id_rsa.pub root@192.168.222.156

    ssh root@192.168.222.156

    ssh root@oradb

    ssh root@192.168.222.157

    ssh root@mysqldb2

    4. 搭建主从复制

    4.1 修改主库my.cnf 配置服务器ID为10

    vi /app/oracle/oradata2/mysql-5.6.27/my.cnf

    [mysqld]

    server_id = 10

    4.1 修改备库my.cnf 配置

    vi /usr/local/mysql/my.cnf

    [mysqld]

    server_id = 10

    # slave config

    log_bin=/usr/local/mysql/data/mysql-bin

    relay_log=/usr/local/mysql/data/relay-bin

    binlog_format=row

    relay_log_purge=0

    read_only=1

    4.2 在主库上备份

    mysqldump --master-data=2 --single-transaction -R -A> all.sql

    其中 --master-data=2代表备份时刻记录 master的 Binlog 位置和 Position,--single-transaction 意思是获取一致性快照, -R 意思是备份相关的存储过程

    ,-A代表备份所有的 schema。

    4.3 在主库上创建复制用户和 MHA监控用户

    mysql> grant replication slave on *.* to 'repl'@'192.168.222.%' identified by '123456';

    mysql> grant all on *.* to 'root'@'192.168.222.%' identified by '123456'; /*MHA 监控用户*/

    4.4 查看主库上备份时刻 Binlog 的名称和位置,MASTER_LOG和 MASTER_LOG_POS

    head -n 30 all.sql|grep -i"CHANGE MASTER TO"

    -- CHANGE MASTER TOMASTER_LOG_FILE='mysql-bin.000068', MASTER_LOG_POS=325;

    4.5 将备份复制到备库

    scp all.sql root@192.168.222.157:/root/olm

    4.6 在备库导入数据

    mysql> source /root/olm/all.sql

    mysql> flush privileges;

    mysql>

    change master tomaster_host='192.168.222.156',master_user='repl',master_password='123456',

    master_port=3306,master_log_file='mysql-bin.000068',master_log_pos=325;

    start slave;

    --查看复制状态

    show slave status \G

    5. 配置MHA

    5.1 创建 MHA 工作目录,并且创建相关配置文件

    mkdir -p /etc/masterha

    vi /etc/masterha/app1.cnf

    [server default]

    manager_log=/masterha/app1/app1.log

    manager_workdir=/masterha/app1

    master_binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data

    master_ip_failover_script=/usr/local/bin/master_ip_failover

    master_ip_online_change_script=/usr/local/bin/master_ip_online_change

    password=123456

    ping_interval=1

    remote_workdir=/tmp

    repl_password=123456

    repl_user=repl

    secondary_check_script=/usr/local/bin/masterha_secondary_check-s mysqldb2 -s oradb --user=root --master_host=oradb--master_ip='192.168.222.156' --master_port=3306

    shutdown_script=""

    ssh_user=root

    user=root

    [server1]

    hostname=192.168.222.156

    port=3306

    [server2]

    candidate_master=1

    check_repl_delay=0

    hostname=192.168.222.157

    port=3306

    5.2 设置定期清理中继日志的脚本

    crontab -e

    0 4 * * */usr/local/bin/purge_relay_logs --user=root --password=123456 --disable_replay_log_purge --port=3306 --workdir=/usr/local/mysql/data --disable_relay_log_purge >> /usr/loca/masterha/log/purge_relay_logs.log 2>&1

    5.2 编辑 /etc/bashrc 文件,在文件末尾添加以下内容,避免MHA执行故障切换时无法找到 mysqlbinlog

    PATH="$PATH:/usr/local/mysql/bin"

    export PATH

    5.3 检查 SSH 的配置

    masterha_check_ssh --conf=/etc/masterha/app1.cnf

    5.4 检查MYSQL复制

    masterha_check_repl --conf=/etc/masterha/app1.cnf




    5.5 启动 MHA Manger 监控

    tail -f /masterha/app1/app1.log

    --在另一个SSH 窗口启动 MHA Manger 监控。--remove_dead_master_conf 该参数代表当发生主从切换后,老的主库的IP将会从配置文件中移除。

    --ignore_last_failover 在缺省情况下,如果MHA检测到连续发生宕机,而且两次宕机时间不足8小时的话,则不会进行 Failover,之所以这样限制

    --是为了避免 ping-pong效应。该参数代表忽悠上次 MHA 触发切换产生的文件,默认情况下,MHA 发生切换后将会在 /masterha/paa1下产生 app1.

    --failover.complete 文件,下次再次切换的时候如果发现目录存在该文件将不允许触发切换,除非在第一次切换后收到 rm -rf /masterha/paa1/app1.--failover.complete,出于方便考虑,我们每次启动MHA时会添加 --ignore_last_failover 参数。

    nohup masterha_manager--conf=/etc/masterha/app1.cnf --remove_dead_master_conf--ignore_last_failover< /dev/null >/masterha/app1/manager.log 2>&1&

    --查看 MHA Manager 监控是否正常

    masterha_check_status --conf=/etc/masterha/app1.cnf

    app1 (pid:4758) isrunning(0:PING_OK), master:192.168.222.156

    --在默认情况下,10秒内状态为10:INITIALIZING_MONITOR,当状态转变为0:PING_OK后表明已经开启了到 MASTER 端的监控。

    关闭MHA的命令如下:

    masterha_stop --conf=/etc/masterha/app1.cnf


    6. 配置 VIP

    6.1 在主库下载集群心跳软件 keepalived,并进行安装

    访问 http://www.keepalived.org/software/ 下载 keepalived-1.2.18.tar.gz

    tar vzxf keepalived-1.2.18.tar.gz

    cd keepalived-1.2.18

    ./configure--prefix=/usr/local/keepalived

    make

    make install

    cp/usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

    cp/usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

    mkdir /etc/keepalived

    cp/usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

    cp/usr/local/keepalived/sbin/keepalived /usr/sbin/

    cd /etc/keepalived/

    vi keepalived.conf

    --下面是添加的配置

    [root@oradb ~]# cat/etc/keepalived/keepalived.conf

    ! Configuration File forkeepalived

    global_defs {

    notification_email {

    acassen@firewall.loc

    failover@firewall.loc

    sysadmin@firewall.loc

    }

    notification_email_from Alexandre.Cassen@firewall.loc

    smtp_server 127.0.0.1

    smtp_connect_timeout 30

    router_id MySQL-ha

    }

    vrrp_instance VI_1 {

    state BACKUP

    interface eth0

    virtual_router_id 51

    priority 150

    advert_int 1

    authentication {

    auth_type PASS

    auth_pass 1111

    }

    virtual_ipaddress {

    192.168.222.158/24

    }

    }

    service keepalived start

    --是用 ip add 命令可以看到 eth0上已经绑定了虚拟IP 192.168.222.158 ,注意使用 ifconfig 是看不到虚拟IP的

    [root@oradb ~]# ip add

    1: lo:<LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

    valid_lft forever preferred_lft forever

    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP>mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:53:7e:f0 brd ff:ff:ff:ff:ff:ff

    inet 192.168.222.156/24 brd 192.168.222.255 scope global eth0

    inet 192.168.222.158/24scope global secondary eth0

    inet6 fe80::20c:29ff:fe53:7ef0/64 scope link

    valid_lft forever preferred_lft forever

    3: eth2:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000

    link/ether 00:0c:29:53:7e:fa brd ff:ff:ff:ff:ff:ff

    inet 192.168.1.11/24 brd 192.168.1.255 scope global eth2

    inet6 fe80::20c:29ff:fe53:7efa/64 scope link

    valid_lft forever preferred_lft forever

    6.2 在备库下载集群心跳软件 keepalived,并进行安装

    访问 http://www.keepalived.org/software/ 下载 keepalived-1.2.18.tar.gz

    tar vzxf keepalived-1.2.18.tar.gz

    cd keepalived-1.2.18

    ./configure--prefix=/usr/local/keepalived

    make

    make install

    cp/usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

    cp/usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

    mkdir /etc/keepalived

    cp/usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

    cp/usr/local/keepalived/sbin/keepalived /usr/sbin/

    cd /etc/keepalived/

    vi keepalived.conf

    --下面是添加的配置

    ! Configuration File forkeepalived

    global_defs {

    notification_email {

    acassen@firewall.loc

    failover@firewall.loc

    sysadmin@firewall.loc

    }

    notification_email_from Alexandre.Cassen@firewall.loc

    smtp_server 127.0.0.1

    smtp_connect_timeout 30

    router_id MySQL-ha

    }

    vrrp_instance VI_1 {

    state BACKUP

    interface eth0

    virtual_router_id 51

    priority 120

    advert_int 1

    authentication {

    auth_type PASS

    auth_pass 1111

    }

    virtual_ipaddress {

    192.168.222.158/24

    }

    }

    service keepalived start

    6.2 MHA引入 keepalived

    在MHA监控节点(备库)编辑 master_ip_failover 脚本

    vi /usr/local/bin/master_ip_failover

    #!/usr/bin/env perl

    # Copyright (C) 2011 DeNA Co.,Ltd.

    #

    # This program is free software; you can redistribute it and/or modify

    # it under the terms of the GNU General Public License as published by

    # the Free Software Foundation; either version 2 of the License, or

    # (at your option) any later version.

    #

    # This program is distributed in the hope that it will be useful,

    # but WITHOUT ANY WARRANTY; without even the implied warranty of

    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

    # GNU General Public License for more details.

    #

    # You should have received a copy of the GNU General Public License

    # along with this program; if not, write to the Free Software

    # Foundation, Inc.,

    # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

    ## Note: This is a sample scriptand is not complete. Modify the script based on your environment.

    use strict;

    use warnings FATAL => 'all';

    use Getopt::Long;

    use MHA::DBHelper;

    my (

    $command, $ssh_user, $orig_master_host,

    $orig_master_ip, $orig_master_port, $new_master_host,

    $new_master_ip, $new_master_port, $new_master_user,

    $new_master_password, $cmd

    );

    GetOptions(

    'command=s' =>\$command,

    'ssh_user=s' =>\$ssh_user,

    'orig_master_host=s' =>\$orig_master_host,

    'orig_master_ip=s' =>\$orig_master_ip,

    'orig_master_port=i' =>\$orig_master_port,

    'new_master_host=s' => \$new_master_host,

    'new_master_ip=s' =>\$new_master_ip,

    'new_master_port=i' =>\$new_master_port,

    'new_master_user=s' =>\$new_master_user,

    'new_master_password=s' => \$new_master_password,

    );

    exit &main();

    sub main {

    if ( $command eq "stop" || $command eq "stopssh" ) {

    # $orig_master_host, $orig_master_ip, $orig_master_port are passed.

    # If you manage master ip address at global catalog database,

    # invalidate orig_master_ip here.

    my $exit_code = 1;

    eval {

    # updating global catalog, etc

    $exit_code = 0;

    };

    if ($@) {

    warn "Got Error: $@\n";

    exit $exit_code;

    }

    exit $exit_code;

    }

    elsif ( $command eq "start" ) {

    # all arguments are passed.

    # If you manage master ip address at global catalog database,

    # activate new_master_ip here.

    # You can also grant write access (create user, set read_only=0, etc)here.

    my $exit_code = 10;

    eval {

    my $new_master_handler = newMHA::DBHelper();

    # args: hostname, port, user, password,raise_error_or_not

    $new_master_handler->connect($new_master_ip, $new_master_port,

    $new_master_user, $new_master_password,1 );

    ## Set read_only=0 on the new master

    $new_master_handler->disable_log_bin_local();

    print "Set read_only=0 on the newmaster.\n";

    $new_master_handler->disable_read_only();

    ## Creating an app user on the new master

    print "Creating app user on the newmaster..\n";

    #FIXME_xxx_create_user($new_master_handler->{dbh} );

    #$new_master_handler->enable_log_bin_local();

    #$new_master_handler->disconnect();

    ## Update master ip on the catalogdatabase, etc

    #FIXME_xxx;

    #$cmd = 'ssh root@192.168.222.156 \./lvs-admin stop';

    #$cmd ='ssh'.$ssh_user.'@'.$orig_master_ip.'\'./lvs-admin stop\'';

    $cmd = 'ssh '.$ssh_user.'@'.$orig_master_ip.' \./lvs-admin stop';

    system($cmd);

    $exit_code = 0;

    };

    if ($@) {

    warn $@;

    # If you want to continue failover, exit10.

    exit $exit_code;

    }

    exit $exit_code;

    }

    elsif ( $command eq "status" ) {

    # do nothing

    exit 0;

    }

    else {

    &usage();

    exit 1;

    }

    }

    sub usage {

    print

    "Usage: master_ip_failover --command=start|stop|stopssh|status--orig_master_host=host --orig_master_ip=ip --orig_master_port=port--new_master_host=host --new_master_ip=ip --new_master_port=port\n";

    }

    在脚本中绿色背景的代码是新增的,红色背景的代码是注释的。


    在主库添加关闭 keepalived 的脚本

    vi /root/lvs-admin

    case "$1" in

    "stop")

    echo;

    echo "stop keepalived ......";

    service keepalived stop

    ;;

    "start")

    echo;

    echo "start keepalive ......";

    service keepalived start

    ;;

    *)

    echo "please input you select!"

    ;;

    esac

    chmod 755 /root/lvs-admin

    7. 测试主库故障MHA自动把备库提升为主库并把虚拟IP漂移到新的主库(原来的备库)

    7.1 在备库停止复制默认有二进制日志没传到备库

    mysql> stop slave;

    7.2 sysbench 生成测试数据,在主库上的 sbtest 库下创建 sbtest 表,共100万条数据

    mysql> create database sbtest;

    ./sysbench --num-threads=4--test=oltp --oltp-table-size=1000000 --oltp-read-only=off --init-rng=on --max-requests=0 \

    --oltp-dist-type=uniform--max-time=1800 --mysql-user=root --db-driver=test --mysql-table-engine=innodb\

    --oltp-test-mode=complex prepare

    7.3 关闭主库 MHA会自动把备份库提升为新的主库,并把虚拟IP漂移到新的主库(原来的备库)

    --关闭主库

    mysqladmin -u root shutdown

    --在MHA监控节点(备库)实时查看MHA日志

    tail -f /masterha/app1/app1.log

    Thu Feb 2 23:53:06 2017 - [info] MHA::MasterMonitorversion 0.57.

    Thu Feb 2 23:53:07 2017 -[error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

    6) replicates is not defined inthe configuration file!

    Thu Feb 2 23:53:07 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln427] Error happened on checking configurations. at /usr/local/share/perl5/MHA/Master

    Monitor.pm line 329.

    Thu Feb 2 23:53:07 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoringservers.

    Thu Feb 2 23:53:07 2017 - [info] Got exit code 1 (Notmaster dead).

    Thu Feb 2 23:54:15 2017 - [info] MHA::MasterMonitorversion 0.57.

    Thu Feb 2 23:54:16 2017 -[error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306from which slave 192.168.222.157(192.168.222.157:330

    6) replicates is not defined inthe configuration file!

    Thu Feb 2 23:54:16 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/share/perl5/MHA/Master

    Monitor.pm line 329.

    Thu Feb 2 23:54:16 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Thu Feb 2 23:54:16 2017 - [info] Got exit code 1 (Notmaster dead).

    Thu Feb 2 23:58:13 2017 - [info] MHA::MasterMonitorversion 0.57.

    Thu Feb 2 23:58:14 2017 -[error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

    6) replicates is not defined inthe configuration file!

    Thu Feb 2 23:58:14 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/share/perl5/MHA/Master

    Monitor.pm line 329.

    Thu Feb 2 23:58:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln525] Error happened on monitoring servers.

    Thu Feb 2 23:58:14 2017 - [info] Got exit code 1 (Notmaster dead).

    Thu Feb 2 23:59:48 2017 - [info] MHA::MasterMonitorversion 0.57.

    Thu Feb 2 23:59:49 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm,ln671] Master 192.168.222.156:3306 from which slave192.168.222.157(192.168.222.157:330

    6) replicates is not defined inthe configuration file!

    Thu Feb 2 23:59:49 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/share/perl5/MHA/Master

    Monitor.pm line 329.

    Thu Feb 2 23:59:49 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Thu Feb 2 23:59:49 2017 - [info] Got exit code 1 (Notmaster dead).

    Fri Feb 3 00:04:10 2017 - [info] MHA::MasterMonitorversion 0.57.

    Fri Feb 3 00:04:11 2017 -[error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

    6) replicates is not defined inthe configuration file!

    Fri Feb 3 00:04:11 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/share/perl5/MHA/Master

    Monitor.pm line 329.

    Fri Feb 3 00:04:11 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Fri Feb 3 00:04:11 2017 - [info] Got exit code 1 (Notmaster dead).

    Fri Feb 3 00:09:04 2017 - [info] MHA::MasterMonitorversion 0.57.

    Fri Feb 3 00:09:05 2017 -[error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

    6) replicates is not defined inthe configuration file!

    Fri Feb 3 00:09:05 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/share/perl5/MHA/Master

    Monitor.pm line 329.

    Fri Feb 3 00:09:05 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Fri Feb 3 00:09:05 2017 - [info] Got exit code 1 (Notmaster dead).

    Fri Feb 3 00:13:13 2017 - [info] MHA::MasterMonitorversion 0.57.

    Fri Feb 3 00:13:14 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm,ln671] Master 192.168.222.156:3306 from which slave192.168.222.157(192.168.222.157:330

    6) replicates is not defined inthe configuration file!

    Fri Feb 3 00:13:14 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/share/perl5/MHA/Master

    Monitor.pm line 329.

    Fri Feb 3 00:13:14 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Fri Feb 3 00:13:14 2017 - [info] Got exit code 1 (Notmaster dead).

    Fri Feb 3 00:20:21 2017 - [info] MHA::MasterMonitorversion 0.57.

    Fri Feb 3 00:20:23 2017 -[error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

    6) replicates is not defined inthe configuration file!

    Fri Feb 3 00:20:23 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/share/perl5/MHA/Master

    Monitor.pm line 329.

    Fri Feb 3 00:20:23 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Fri Feb 3 00:20:23 2017 - [info] Got exit code 1 (Notmaster dead).

    Fri Feb 3 00:21:54 2017 - [info] MHA::MasterMonitorversion 0.57.

    Fri Feb 3 00:21:55 2017 - [info] GTID failover mode =0

    Fri Feb 3 00:21:55 2017 - [info] Dead Servers:

    Fri Feb 3 00:21:55 2017 - [info] Alive Servers:

    Fri Feb 3 00:21:55 2017 - [info] 192.168.222.156(192.168.222.156:3306)

    Fri Feb 3 00:21:55 2017 - [info] 192.168.222.157(192.168.222.157:3306)

    Fri Feb 3 00:21:55 2017 - [info] Alive Slaves:

    Fri Feb 3 00:21:55 2017 - [info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major versionbetween slaves) log-bin:enabled

    Fri Feb 3 00:21:55 2017 - [info] Replicating from192.168.222.156(192.168.222.156:3306)

    Fri Feb 3 00:21:55 2017 - [info] Primary candidate for the new Master(candidate_master is set)

    Fri Feb 3 00:21:55 2017 - [info] Current AliveMaster: 192.168.222.156(192.168.222.156:3306)

    Fri Feb 3 00:21:55 2017 - [info] Checking slaveconfigurations..

    Fri Feb 3 00:21:55 2017 - [info] Checking replicationfiltering settings..

    Fri Feb 3 00:21:55 2017 - [info] binlog_do_db= , binlog_ignore_db=

    Fri Feb 3 00:21:55 2017 - [info] Replication filtering check ok.

    Fri Feb 3 00:21:55 2017 - [info] GTID (with auto-pos)is not supported

    Fri Feb 3 00:21:55 2017 - [info] Starting SSHconnection tests..

    Fri Feb 3 00:21:58 2017 - [info] All SSH connectiontests passed successfully.

    Fri Feb 3 00:21:58 2017 - [info] Checking MHA Nodeversion..

    Fri Feb 3 00:21:58 2017 - [info] Version check ok.

    Fri Feb 3 00:21:58 2017 - [info] Checking SSHpublickey authentication settings on the current master..

    Fri Feb 3 00:21:59 2017 - [info] HealthCheck: SSH to192.168.222.156 is reachable.

    Fri Feb 3 00:21:59 2017 - [info] Master MHA Nodeversion is 0.57.

    Fri Feb 3 00:21:59 2017 - [info] Checking recoveryscript configurations on 192.168.222.156(192.168.222.156:3306)..

    Fri Feb 3 00:21:59 2017 - [info] Executing command: save_binary_logs--command=test --start_pos=4--binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=

    /tmp/save_binary_logs_test--manager_version=0.57 --start_file=mysql-bin.000068

    Fri Feb 3 00:21:59 2017 - [info] Connecting toroot@192.168.222.156(192.168.222.156:22)..

    Creating /tmp if not exists.. ok.

    Checking output directory is accessible or not..

    ok.

    Binlog found at /app/oracle/oradata2/mysql-5.5.37/data, up tomysql-bin.000068

    Fri Feb 3 00:21:59 2017 - [info] Binlog setting checkdone.

    Fri Feb 3 00:21:59 2017 - [info] Checking SSHpublickey authentication and checking recovery script configurations on allalive slave servers..

    Fri Feb 3 00:21:59 2017 - [info] Executing command : apply_diff_relay_logs--command=test --slave_user='root' --slave_host=192.168.222.157--slave_ip=192.168.222.15

    7 --slave_port=3306 --workdir=/tmp--target_version=5.6.27-log --manager_version=0.57 --relay_log_info=/usr/local/mysql/data/relay-log.info --relay_dir=/usr/local/mys

    ql/data/ --slave_pass=xxx

    Fri Feb 3 00:21:59 2017 - [info] Connecting toroot@192.168.222.157(192.168.222.157:22)..

    Checking slave recovery environment settings..

    Opening /usr/local/mysql/data/relay-log.info ... ok.

    Relay log found at /usr/local/mysql/data, up to relay-bin.000002

    Temporary relay log file is /usr/local/mysql/data/relay-bin.000002

    Testing mysql connection and privileges..Warning: Using a password onthe command line interface can be insecure.

    done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

    Fri Feb 3 00:22:00 2017 - [info] Slaves settingscheck done.

    Fri Feb 3 00:22:00 2017 - [info]

    192.168.222.156(192.168.222.156:3306)(current master)

    +--192.168.222.157(192.168.222.157:3306)

    Fri Feb 3 00:22:00 2017 - [info] Checkingmaster_ip_failover_script status:

    Fri Feb 3 00:22:00 2017 - [info] /usr/local/bin/master_ip_failover--command=status --ssh_user=root --orig_master_host=192.168.222.156--orig_master_ip=192.168.222.

    156 --orig_master_port=3306

    Fri Feb 3 00:22:00 2017 - [info] OK.

    Fri Feb 3 00:22:00 2017 - [warning] shutdown_scriptis not defined.

    Fri Feb 3 00:22:00 2017 - [info] Set master pinginterval 1 seconds.

    Fri Feb 3 00:22:00 2017 - [info] Set secondary checkscript: /usr/local/bin/masterha_secondary_check -s mysqldb2 -s oradb--user=root --master_host=oradb --master_ip=

    '192.168.222.156'--master_port=3306

    Fri Feb 3 00:22:00 2017 - [info] Starting ping healthcheck on 192.168.222.156(192.168.222.156:3306)..

    Fri Feb 3 00:22:00 2017 - [info] Ping(SELECT)succeeded, waiting until MySQL doesn't respond..

    Fri Feb 3 00:25:06 2017 - [warning] Got error onMySQL select ping: 2013 (Lost connection to MySQL server during query)

    Fri Feb 3 00:25:06 2017 - [info] Executing secondarynetwork check script: /usr/local/bin/masterha_secondary_check -s mysqldb2 -soradb --user=root --master_host=orad

    b --master_ip='192.168.222.156'--master_port=3306 --user=root --master_host=192.168.222.156 --master_ip=192.168.222.156 --master_port=3306 --master_user=root --ma

    ster_password=123456--ping_type=SELECT

    Fri Feb 3 00:25:06 2017 - [info] Executing SSH checkscript: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data--outpu

    t_file=/tmp/save_binary_logs_test--manager_version=0.57 --binlog_prefix=mysql-bin

    Fri Feb 3 00:25:07 2017 - [warning] Got error onMySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

    Fri Feb 3 00:25:07 2017 - [warning] Connection failed2 time(s)..

    Monitoring server mysqldb2 isreachable, Master is not reachable from mysqldb2. OK.

    Fri Feb 3 00:25:08 2017 - [warning] Got error onMySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

    Fri Feb 3 00:25:08 2017 - [warning] Connection failed3 time(s)..

    Monitoring server oradb isreachable, Master is not reachable from oradb. OK.

    Fri Feb 3 00:25:08 2017 - [info] Master is notreachable from all other monitoring servers. Failover should start.

    Fri Feb 3 00:25:08 2017 - [info] HealthCheck: SSH to192.168.222.156 is reachable.

    Fri Feb 3 00:25:09 2017 - [warning] Got error onMySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

    Fri Feb 3 00:25:09 2017 - [warning] Connection failed4 time(s)..

    Fri Feb 3 00:25:09 2017 - [warning] Master is notreachable from health checker!

    Fri Feb 3 00:25:09 2017 - [warning] Master192.168.222.156(192.168.222.156:3306) is not reachable!

    Fri Feb 3 00:25:09 2017 - [warning] SSH is reachable.

    Fri Feb 3 00:25:09 2017 - [info] Connecting to amaster server failed. Reading configuration file /etc/masterha_default.cnf and/etc/masterha/app1.cnf again, and tryi

    ng to connect to all servers tocheck server status..

    Fri Feb 3 00:25:09 2017 - [warning] Globalconfiguration file /etc/masterha_default.cnf not found. Skipping.

    Fri Feb 3 00:25:09 2017 - [info] Reading applicationdefault configuration from /etc/masterha/app1.cnf..

    Fri Feb 3 00:25:09 2017 - [info] Reading serverconfiguration from /etc/masterha/app1.cnf..

    Fri Feb 3 00:25:10 2017 - [warning] SQL Thread isstopped(no error) on 192.168.222.157(192.168.222.157:3306)

    Fri Feb 3 00:25:10 2017 - [info] GTID failover mode =0

    Fri Feb 3 00:25:10 2017 - [info] Dead Servers:

    Fri Feb 3 00:25:10 2017 - [info] 192.168.222.156(192.168.222.156:3306)

    Fri Feb 3 00:25:10 2017 - [info] Alive Servers:

    Fri Feb 3 00:25:10 2017 - [info] 192.168.222.157(192.168.222.157:3306)

    Fri Feb 3 00:25:10 2017 - [info] Alive Slaves:

    Fri Feb 3 00:25:10 2017 - [info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major versionbetween slaves) log-bin:enabled

    Fri Feb 3 00:25:10 2017 - [info] Replicating from192.168.222.156(192.168.222.156:3306)

    Fri Feb 3 00:25:10 2017 - [info] Primary candidate for the new Master(candidate_master is set)

    Fri Feb 3 00:25:10 2017 - [info] Checking slaveconfigurations..

    Fri Feb 3 00:25:10 2017 - [info] Checking replicationfiltering settings..

    Fri Feb 3 00:25:10 2017 - [info] Replication filtering check ok.

    Fri Feb 3 00:25:10 2017 - [info] Master is down!

    Fri Feb 3 00:25:10 2017 - [info] Terminatingmonitoring script.

    Fri Feb 3 00:25:10 2017 - [info] Got exit code 20(Master dead).

    Fri Feb 3 00:25:10 2017 - [info] MHA::MasterFailoverversion 0.57.

    Fri Feb 3 00:25:10 2017 - [info] Starting masterfailover.

    Fri Feb 3 00:25:10 2017 - [info]

    Fri Feb 3 00:25:10 2017 - [info] * Phase 1:Configuration Check Phase..

    Fri Feb 3 00:25:10 2017 - [info]

    Fri Feb 3 00:25:11 2017 - [warning] SQL Thread isstopped(no error) on 192.168.222.157(192.168.222.157:3306)

    Fri Feb 3 00:25:11 2017 - [info] GTID failover mode =0

    Fri Feb 3 00:25:11 2017 - [info] Dead Servers:

    Fri Feb 3 00:25:11 2017 - [info] 192.168.222.156(192.168.222.156:3306)

    Fri Feb 3 00:25:11 2017 - [info] Checking masterreachability via MySQL(double check)...

    Fri Feb 3 00:25:11 2017 - [info] ok.

    Fri Feb 3 00:25:11 2017 - [info] Alive Servers:

    Fri Feb 3 00:25:11 2017 - [info] 192.168.222.157(192.168.222.157:3306)

    Fri Feb 3 00:25:11 2017 - [info] Alive Slaves:

    Fri Feb 3 00:25:11 2017 - [info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major versionbetween slaves) log-bin:enabled

    Fri Feb 3 00:25:11 2017 - [info] Replicating from192.168.222.156(192.168.222.156:3306)

    Fri Feb 3 00:25:11 2017 - [info] Primary candidate for the new Master(candidate_master is set)

    Fri Feb 3 00:25:11 2017 - [info] Starting SQL thread on192.168.222.157(192.168.222.157:3306) ..

    Fri Feb 3 00:25:11 2017 - [info] done.

    Fri Feb 3 00:25:11 2017 - [info] Starting Non-GTIDbased failover.

    Fri Feb 3 00:25:11 2017 - [info]

    Fri Feb 3 00:25:11 2017 - [info] ** Phase 1:Configuration Check Phase completed.

    Fri Feb 3 00:25:11 2017 - [info]

    Fri Feb 3 00:25:11 2017 - [info] * Phase 2: DeadMaster Shutdown Phase..

    Fri Feb 3 00:25:11 2017 - [info]

    Fri Feb 3 00:25:11 2017 - [info] Forcing shutdown sothat applications never connect to the current master..

    Fri Feb 3 00:25:11 2017 - [info] Executing master IPdeactivation script:

    Fri Feb 3 00:25:11 2017 - [info] /usr/local/bin/master_ip_failover--orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.156--orig_master_port=3306 --com

    mand=stopssh --ssh_user=root

    Fri Feb 3 00:25:11 2017 - [info] done.

    Fri Feb 3 00:25:11 2017 - [warning] shutdown_scriptis not set. Skipping explicit shutting down of the dead master.

    Fri Feb 3 00:25:11 2017 - [info] * Phase 2: DeadMaster Shutdown Phase completed.

    Fri Feb 3 00:25:11 2017 - [info]

    Fri Feb 3 00:25:11 2017 - [info] * Phase 3: MasterRecovery Phase..

    Fri Feb 3 00:25:11 2017 - [info]

    Fri Feb 3 00:25:11 2017 - [info] * Phase 3.1: GettingLatest Slaves Phase..

    Fri Feb 3 00:25:11 2017 - [info]

    Fri Feb 3 00:25:11 2017 - [info] The latest binarylog file/position on all slaves is mysql-bin.000068:530

    Fri Feb 3 00:25:11 2017 - [info] Latest slaves(Slaves that received relay log files to the latest):

    Fri Feb 3 00:25:11 2017 - [info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major versionbetween slaves) log-bin:enabled

    Fri Feb 3 00:25:11 2017 - [info] Replicating from192.168.222.156(192.168.222.156:3306)

    Fri Feb 3 00:25:11 2017 - [info] Primary candidate for the new Master(candidate_master is set)

    Fri Feb 3 00:25:11 2017 - [info] The oldest binarylog file/position on all slaves is mysql-bin.000068:530

    Fri Feb 3 00:25:11 2017 - [info] Oldest slaves:

    Fri Feb 3 00:25:11 2017 - [info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major versionbetween slaves) log-bin:enabled

    Fri Feb 3 00:25:11 2017 - [info] Replicating from192.168.222.156(192.168.222.156:3306)

    Fri Feb 3 00:25:11 2017 - [info] Primary candidate for the new Master(candidate_master is set)

    Fri Feb 3 00:25:11 2017 - [info]

    Fri Feb 3 00:25:11 2017 - [info] * Phase 3.2: SavingDead Master's Binlog Phase..

    Fri Feb 3 00:25:11 2017 - [info]

    Fri Feb 3 00:25:12 2017 - [info] Fetching deadmaster's binary logs..

    Fri Feb 3 00:25:12 2017 - [info] Executing command onthe dead master 192.168.222.156(192.168.222.156:3306): save_binary_logs--command=save --start_file=mysql-bin.00

    0068 --start_pos=530--binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data--output_file=/tmp/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlog--hand

    le_raw_binlog=1--disable_log_bin=0 --manager_version=0.57

    Creating /tmp if not exists.. ok.

    Concat binary/relay logs from mysql-bin.000068pos 530 to mysql-bin.000068 EOF into/tmp/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlog ..

    Binlog Checksum enabled

    Dumping binlog format description event, from position 0 to 120.. ok.

    Dumping effective binlog data from/app/oracle/oradata2/mysql-5.5.37/data/mysql-bin.000068 position 530 totail(61261921).. ok.

    Binlog Checksum enabled

    Concat succeeded.

    Fri Feb 3 00:25:16 2017 - [info] scp fromroot@192.168.222.156:/tmp/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlogto local:/masterha/app1/saved_

    master_binlog_from_192.168.222.156_3306_20170203002510.binlogsucceeded.

    Fri Feb 3 00:25:17 2017 - [info] HealthCheck: SSH to192.168.222.157 is reachable.

    Fri Feb 3 00:25:17 2017 - [info]

    Fri Feb 3 00:25:17 2017 - [info] * Phase 3.3:Determining New Master Phase..

    Fri Feb 3 00:25:17 2017 - [info]

    Fri Feb 3 00:25:17 2017 - [info] Finding the latestslave that has all relay logs for recovering other slaves..

    Fri Feb 3 00:25:17 2017 - [info] All slaves receivedrelay logs to the same position. No need to resync each other.

    Fri Feb 3 00:25:17 2017 - [info] Searching new masterfrom slaves..

    Fri Feb 3 00:25:17 2017 - [info] Candidate masters from the configuration file:

    Fri Feb 3 00:25:17 2017 - [info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major versionbetween slaves) log-bin:enabled

    Fri Feb 3 00:25:17 2017 - [info] Replicating from192.168.222.156(192.168.222.156:3306)

    Fri Feb 3 00:25:17 2017 - [info] Primary candidate for the new Master(candidate_master is set)

    Fri Feb 3 00:25:17 2017 - [info] Non-candidate masters:

    Fri Feb 3 00:25:17 2017 - [info] Searching from candidate_master slaves whichhave received the latest relay log events..

    Fri Feb 3 00:25:17 2017 - [info] New master is192.168.222.157(192.168.222.157:3306)

    Fri Feb 3 00:25:17 2017 - [info] Starting masterfailover..

    Fri Feb 3 00:25:17 2017 - [info]

    From:

    192.168.222.156(192.168.222.156:3306)(current master)

    +--192.168.222.157(192.168.222.157:3306)

    To:

    192.168.222.157(192.168.222.157:3306)(new master)

    Fri Feb 3 00:25:17 2017 - [info]

    Fri Feb 3 00:25:17 2017 - [info] * Phase 3.3: NewMaster Diff Log Generation Phase..

    Fri Feb 3 00:25:17 2017 - [info]

    Fri Feb 3 00:25:17 2017 - [info] This server has all relay logs. No need togenerate diff files from the latest slave.

    Fri Feb 3 00:25:17 2017 - [info] Sending binlog..

    Fri Feb 3 00:25:20 2017 - [info] scp from local:/masterha/app1/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlogto root@192.168.222.157:/tmp/saved_

    master_binlog_from_192.168.222.156_3306_20170203002510.binlogsucceeded.

    Fri Feb 3 00:25:20 2017 - [info]

    Fri Feb 3 00:25:20 2017 - [info] * Phase 3.4: MasterLog Apply Phase..

    Fri Feb 3 00:25:20 2017 - [info]

    Fri Feb 3 00:25:20 2017 - [info] *NOTICE: If anyerror happens from this phase, manual recovery is needed.

    Fri Feb 3 00:25:20 2017 - [info] Starting recovery on192.168.222.157(192.168.222.157:3306)..

    Fri Feb 3 00:25:20 2017 - [info] Generating diffs succeeded.

    Fri Feb 3 00:25:20 2017 - [info] Waiting until allrelay logs are applied.

    Fri Feb 3 00:25:20 2017 - [info] done.

    Fri Feb 3 00:25:20 2017 - [info] Getting slave status..

    Fri Feb 3 00:25:20 2017 - [info] Thisslave(192.168.222.157)'s Exec_Master_Log_Pos equals toRead_Master_Log_Pos(mysql-bin.000068:530). No need to recover from Exec_M

    aster_Log_Pos.

    Fri Feb 3 00:25:20 2017 - [info] Connecting to thetarget slave host 192.168.222.157, running recover script..

    Fri Feb 3 00:25:20 2017 - [info] Executing command:apply_diff_relay_logs --command=apply --slave_user='root'--slave_host=192.168.222.157 --slave_ip=192.168.222.157

    --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlog--workdir=/tmp --target_version=5.6.27-log --timestamp=201702

    03002510 --handle_raw_binlog=1--disable_log_bin=0 --manager_version=0.57 --slave_pass=xxx

    Fri Feb 3 00:26:48 2017 - [info]

    MySQL client version is 5.6.27.Using --binary-mode.

    Applying differential binary/relaylog files/tmp/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlog on192.168.222.157:3306. This may take long time.

    ..

    Applying log files succeeded.

    Fri Feb 3 00:26:48 2017 - [info] All relay logs were successfully applied.

    Fri Feb 3 00:26:48 2017 - [info] Getting new master'sbinlog name and position..

    Fri Feb 3 00:26:48 2017 - [info] mysql-bin.000002:196188901

    Fri Feb 3 00:26:48 2017 - [info] All other slaves should start replicationfrom here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.222.157',MASTER_POR

    T=3306,MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=196188901,MASTER_USER='repl', MASTER_PASSWORD='xxx';

    Fri Feb 3 00:26:48 2017 - [info] Executing master IPactivate script:

    Fri Feb 3 00:26:48 2017 - [info] /usr/local/bin/master_ip_failover--command=start --ssh_user=root --orig_master_host=192.168.222.156--orig_master_ip=192.168.222.1

    56 --orig_master_port=3306 --new_master_host=192.168.222.157--new_master_ip=192.168.222.157 --new_master_port=3306--new_master_user='root' --new_master_password=xx

    x

    Set read_only=0 on the new master.

    Creating app user on the newmaster..

    sh:sshroot@192.168.222.156./lvs-admin stop: No such file or directory

    Fri Feb 3 00:26:49 2017 - [info] OK.

    Fri Feb 3 00:26:49 2017 - [info] ** Finished masterrecovery successfully.

    Fri Feb 3 00:26:49 2017 - [info] * Phase 3: MasterRecovery Phase completed.

    Fri Feb 3 00:26:49 2017 - [info]

    Fri Feb 3 00:26:49 2017 - [info] * Phase 4: SlavesRecovery Phase..

    Fri Feb 3 00:26:49 2017 - [info]

    Fri Feb 3 00:26:49 2017 - [info] * Phase 4.1:Starting Parallel Slave Diff Log Generation Phase..

    Fri Feb 3 00:26:49 2017 - [info]

    Fri Feb 3 00:26:49 2017 - [info] Generating relaydiff files from the latest slave succeeded.

    Fri Feb 3 00:26:49 2017 - [info]

    Fri Feb 3 00:26:49 2017 - [info] * Phase 4.2:Starting Parallel Slave Log Apply Phase..

    Fri Feb 3 00:26:49 2017 - [info]

    Fri Feb 3 00:26:49 2017 - [info] All new slaveservers recovered successfully.

    Fri Feb 3 00:26:49 2017 - [info]

    Fri Feb 3 00:26:49 2017 - [info] * Phase 5: Newmaster cleanup phase..

    Fri Feb 3 00:26:49 2017 - [info]

    Fri Feb 3 00:26:49 2017 - [info] Resetting slave infoon the new master..

    Fri Feb 3 00:26:49 2017 - [info] 192.168.222.157: Resetting slave infosucceeded.

    Fri Feb 3 00:26:49 2017 - [info] Master failover to192.168.222.157(192.168.222.157:3306) completed successfully.

    Fri Feb 3 00:26:49 2017 - [info] Deleted server1entry from /etc/masterha/app1.cnf .

    Fri Feb 3 00:26:49 2017 - [info]

    ----- Failover Report -----

    app1: MySQL Master failover192.168.222.156(192.168.222.156:3306) to 192.168.222.157(192.168.222.157:3306)succeeded

    Master 192.168.222.156(192.168.222.156:3306)is down!

    Check MHA Manager logs atmysqldb2:/masterha/app1/app1.log for details.

    Started automated(non-interactive)failover.

    Invalidated master IP address on192.168.222.156(192.168.222.156:3306)

    The latest slave192.168.222.157(192.168.222.157:3306) has all relay logs for recovery.

    Selected192.168.222.157(192.168.222.157:3306) as a new master.

    192.168.222.157(192.168.222.157:3306):OK: Applying all logs succeeded.

    192.168.222.157(192.168.222.157:3306):OK: Activated master IP address.

    Generating relay diff files fromthe latest slave succeeded.

    192.168.222.157(192.168.222.157:3306):Resetting slave info succeeded.

    Master failover to192.168.222.157(192.168.222.157:3306) completed successfully.

    三、部署 MHA遇到的问题

    问题:

    [root@localhostmha]# ./install.sh

    --> Working on install

    Fetchinghttp://www.cpan.org/authors/id/D/DA/DAGOLDEN/install-0.01.tar.gz ... OK

    Configuring install-0.01 ... OK

    Building and testing install-0.01... FAIL

    ! Installing install failed. See/root/.cpanm/work/1485934105.3288/build.log for details. Retry with --force toforce install it.

    --> Working on DBD::mysql

    Fetchinghttp://www.cpan.org/authors/id/M/MI/MICHIELB/DBD-mysql-4.041.tar.gz ... OK

    ==> Found dependencies: DBI

    --> Working on DBI

    Fetching http://www.cpan.org/authors/id/T/TI/TIMB/DBI-1.636.tar.gz... OK

    Configuring DBI-1.636 ... N/A

    ! Configure failed for DBI-1.636.See /root/.cpanm/work/1485934125.3353/build.log for details.

    ! Installing the dependenciesfailed: Module 'DBI' is not installed

    ! Bailing out the installation forDBD-mysql-4.041.

    问题:

    /usr/bin/ld: cannot find -lz

    usr/bin/ld: cannot find -lz

    collect2: ld returned 1 exitstatus

    make: ***[blib/arch/auto/DBD/mysql/mysql.so] Error 1

    -> FAIL Installing DBD::mysqlfailed. See /root/.cpanm/work/1485938925.8246/build.log for details. Retry with--force to force install it.

    2 distributions installed

    处理:

    yum -y install *zlib-dev*

    [root@localhost mha]# ./install.sh

    install is up to date. (0.01)

    --> Working on DBD::mysql

    Fetching http://www.cpan.org/authors/id/M/MI/MICHIELB/DBD-mysql-4.041.tar.gz... OK

    Configuring DBD-mysql-4.041 ... OK

    Building and testingDBD-mysql-4.041 ... OK

    Successfully installedDBD-mysql-4.041 (upgraded from 4.013)

    1 distribution installed

    [root@localhostmha4mysql-node-0.53]# perl Makefile.PL

    *** Module::AutoInstall version1.03

    *** Checking for Perldependencies...

    [Core Features]

    - DBI ...loaded. (1.609)

    - DBD::mysql ...loaded. (4.041)

    *** Module::AutoInstallconfiguration finished.

    Checking if your kit iscomplete...

    Looks good

    Generating a Unix-style Makefile

    Writing Makefile formha4mysql::node

    Writing MYMETA.yml and MYMETA.json

    [root@localhostmha4mysql-node-0.53]# make

    cp lib/MHA/BinlogManager.pmblib/lib/MHA/BinlogManager.pm

    cp lib/MHA/BinlogPosFindManager.pmblib/lib/MHA/BinlogPosFindManager.pm

    cp lib/MHA/BinlogPosFinderXid.pmblib/lib/MHA/BinlogPosFinderXid.pm

    cp lib/MHA/BinlogHeaderParser.pmblib/lib/MHA/BinlogHeaderParser.pm

    cp lib/MHA/BinlogPosFinder.pmblib/lib/MHA/BinlogPosFinder.pm

    cp lib/MHA/BinlogPosFinderElp.pmblib/lib/MHA/BinlogPosFinderElp.pm

    cp lib/MHA/NodeUtil.pmblib/lib/MHA/NodeUtil.pm

    cp lib/MHA/SlaveUtil.pmblib/lib/MHA/SlaveUtil.pm

    cp lib/MHA/NodeConst.pmblib/lib/MHA/NodeConst.pm

    cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs

    "/usr/bin/perl""-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' --blib/script/apply_diff_relay_logs

    cp bin/filter_mysqlbinlogblib/script/filter_mysqlbinlog

    "/usr/bin/perl""-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog

    cp bin/purge_relay_logsblib/script/purge_relay_logs

    "/usr/bin/perl""-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' --blib/script/purge_relay_logs

    cp bin/save_binary_logsblib/script/save_binary_logs

    "/usr/bin/perl""-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' --blib/script/save_binary_logs

    Manifying 4 pod documents

    [root@localhostmha4mysql-node-0.53]# make install

    Manifying 4 pod documents

    Installing/usr/local/share/perl5/MHA/NodeUtil.pm

    Installing/usr/local/share/perl5/MHA/BinlogPosFindManager.pm

    Installing/usr/local/share/perl5/MHA/NodeConst.pm

    Installing/usr/local/share/perl5/MHA/BinlogPosFinder.pm

    Installing/usr/local/share/perl5/MHA/BinlogHeaderParser.pm

    Installing/usr/local/share/perl5/MHA/BinlogPosFinderXid.pm

    Installing/usr/local/share/perl5/MHA/BinlogPosFinderElp.pm

    Installing/usr/local/share/perl5/MHA/BinlogManager.pm

    Installing/usr/local/share/perl5/MHA/SlaveUtil.pm

    Installing/usr/local/share/man/man1/purge_relay_logs.1

    Installing /usr/local/share/man/man1/filter_mysqlbinlog.1

    Installing/usr/local/share/man/man1/save_binary_logs.1

    Installing/usr/local/share/man/man1/apply_diff_relay_logs.1

    Installing/usr/local/bin/apply_diff_relay_logs

    Installing/usr/local/bin/save_binary_logs

    Installing /usr/local/bin/filter_mysqlbinlog

    Installing/usr/local/bin/purge_relay_logs

    Appending installation info to/usr/lib64/perl5/perllocal.pod

    问题10:从 https://github.com/yoshinorim/mha4mysql-node 下载 mha4mysql-node 安装

    [root@mysqldb2mha4mysql-node-master]# perl Makefile.PL

    Can't locate inc/Module/Install.pmin @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5/usr/share/perl5 .) at Makefile.PL line 1.

    BEGIN failed--compilation abortedat Makefile.PL line 1.

    处理:

    --查看 Makefile.PL 第一行的内容

    [root@mysqldb2 mha4mysql-node-master]# vi Makefile.PL

    use inc::Module::Install;

    --安装模块

    cpanm inc::Module::Install

    [root@mysqldb2 mha4mysql-node-master]# perl Makefile.PL

    include /root/olm/mha/mha4mysql-node-master/inc/Module/Install.pm

    include inc/Module/Install/Metadata.pm

    include inc/Module/Install/Base.pm

    include inc/Module/Install/Makefile.pm

    include inc/Module/Install/Scripts.pm

    include inc/Module/Install/AutoInstall.pm

    include inc/Module/Install/Include.pm

    include inc/Module/AutoInstall.pm

    *** Module::AutoInstall version 1.17

    *** Checking for Perl dependencies...

    [Core Features]

    - DBI ...loaded.(1.609)

    - DBD::mysql ...loaded. (4.041)

    *** Module::AutoInstall configuration finished.

    include inc/Module/Install/WriteAll.pm

    include inc/Module/Install/Win32.pm

    include inc/Module/Install/Can.pm

    include inc/Module/Install/Fetch.pm

    Checking if your kit is complete...

    Warning: the following files are missing in your kit:

    META.yml

    Please inform the author.

    Generating a Unix-style Makefile

    Writing Makefile for mha4mysql::node

    Writing MYMETA.yml and MYMETA.json

    Writing META.yml

    [root@mysqldb2 mha4mysql-node-master]# make

    cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm

    cp lib/MHA/BinlogPosFindManager.pmblib/lib/MHA/BinlogPosFindManager.pm

    cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm

    cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm

    cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm

    cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm

    cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm

    cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm

    cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm

    cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs

    "/usr/bin/perl" "-Iinc" -MExtUtils::MY -e'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs

    cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog

    "/usr/bin/perl" "-Iinc" -MExtUtils::MY -e'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog

    cp bin/purge_relay_logs blib/script/purge_relay_logs

    "/usr/bin/perl" "-Iinc" -MExtUtils::MY -e'MY->fixin(shift)' -- blib/script/purge_relay_logs

    cp bin/save_binary_logs blib/script/save_binary_logs

    "/usr/bin/perl" "-Iinc" -MExtUtils::MY -e'MY->fixin(shift)' -- blib/script/save_binary_logs

    Manifying 4 pod documents

    [root@mysqldb2 mha4mysql-node-master]# make install

    Manifying 4 pod documents

    Installing /usr/local/share/perl5/MHA/NodeUtil.pm

    Installing /usr/local/share/perl5/MHA/NodeConst.pm

    Installing /usr/local/share/perl5/MHA/BinlogHeaderParser.pm

    Installing /usr/local/share/perl5/MHA/BinlogManager.pm

    Installing /usr/local/share/perl5/MHA/SlaveUtil.pm

    Installing /usr/local/share/man/man1/purge_relay_logs.1

    Installing /usr/local/share/man/man1/filter_mysqlbinlog.1

    Installing /usr/local/share/man/man1/save_binary_logs.1

    Installing /usr/local/share/man/man1/apply_diff_relay_logs.1

    Installing /usr/local/bin/apply_diff_relay_logs

    Installing /usr/local/bin/save_binary_logs

    Installing /usr/local/bin/purge_relay_logs

    Appending installation info to /usr/lib64/perl5/perllocal.pod

    问题11:处理完问题10后

    [root@mysqldb2 debian]# /usr/local/bin/masterha_check_ssh--conf=/etc/masterha/app1.cnf

    "NI_NUMERICHOST" is not exported by the Socket module

    "getaddrinfo" isnot exported by the Socket module

    "getnameinfo" isnot exported by the Socket module

    Can't continue after import errors at/usr/local/share/perl5/MHA/NodeUtil.pm line 29.

    BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/NodeUtil.pmline 29.

    Compilation failed in require at/usr/local/share/perl5/MHA/SlaveUtil.pm line 27.

    BEGIN failed--compilation aborted at/usr/local/share/perl5/MHA/SlaveUtil.pm line 27.

    Compilation failed in require at /usr/local/share/perl5/MHA/DBHelper.pmline 26.

    BEGIN failed--compilation aborted at/usr/local/share/perl5/MHA/DBHelper.pm line 26.

    Compilation failed in require at/usr/local/share/perl5/MHA/HealthCheck.pm line 30.

    BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/HealthCheck.pmline 30.

    Compilation failed in require at/usr/local/share/perl5/MHA/Server.pm line 28.

    BEGIN failed--compilation aborted at/usr/local/share/perl5/MHA/Server.pm line 28.

    Compilation failed in require at /usr/local/share/perl5/MHA/Config.pmline 29.

    BEGIN failed--compilation aborted at/usr/local/share/perl5/MHA/Config.pm line 29.

    Compilation failed in require at/usr/local/share/perl5/MHA/SSHCheck.pm line 32.

    BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/SSHCheck.pmline 32.

    Compilation failed in require at /usr/local/bin/masterha_check_sshline 25.

    BEGIN failed--compilation aborted at/usr/local/bin/masterha_check_ssh line 25.

    处理:

    [root@mysqldb2 mha4mysql-node-master]# cpanm Socket

    --> Working on Socket

    Fetchinghttp://www.cpan.org/authors/id/P/PE/PEVANS/Socket-2.024.tar.gz ... OK

    ==> Found dependencies: ExtUtils::Constant

    --> Working on ExtUtils::Constant

    Fetchinghttp://www.cpan.org/authors/id/N/NW/NWCLARK/ExtUtils-Constant-0.23.tar.gz ...OK

    Configuring ExtUtils-Constant-0.16 ... OK

    Building and testing ExtUtils-Constant-0.23 ... OK

    Successfully installed ExtUtils-Constant-0.23 (upgraded from 0.22)

    Configuring Socket-2.024 ... OK

    Building and testing Socket-2.024 ... OK

    Successfully installed Socket-2.024 (upgraded from 1.82)

    2 distributions installed

    [root@mysqldb2 mha4mysql-manager-master]#/usr/local/bin/masterha_check_ssh --conf=/etc/masterha/app1.cnf

    Wed Feb 1 19:41:34 2017 -[warning] Global configuration file /etc/masterha_default.cnf not found.Skipping.

    Wed Feb 1 19:41:34 2017 -[info] Reading application default configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 19:41:34 2017 -[info] Reading server configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 19:41:34 2017 -[info] Starting SSH connection tests..

    Wed Feb 1 19:41:51 2017 -[debug]

    Wed Feb 1 19:41:35 2017 -[debug] Connecting via SSH fromroot@192.168.222.157(192.168.222.157:22) toroot@192.168.222.156(192.168.222.156:22)..

    Wed Feb 1 19:41:50 2017 -[debug] ok.

    Wed Feb 1 19:41:55 2017 -[debug]

    Wed Feb 1 19:41:34 2017 -[debug] Connecting via SSH fromroot@192.168.222.156(192.168.222.156:22) toroot@192.168.222.157(192.168.222.157:22)..

    Wed Feb 1 19:41:55 2017 -[debug] ok.

    Wed Feb 1 19:41:55 2017 -[info] All SSH connection tests passed successfully.

    问题12:

    [root@mysqldb2 mha]# masterha_check_repl--conf=/etc/masterha/app1.cnf

    Wed Feb 1 20:04:52 2017 -[warning] Global configuration file /etc/masterha_default.cnf not found.Skipping.

    Wed Feb 1 20:04:52 2017 -[info] Reading application default configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:04:52 2017 -[info] Reading server configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:04:52 2017 -[info] MHA::MasterMonitor version 0.57.

    Creating directory /masterha/app1.. done.

    Wed Feb 1 20:04:57 2017 -[info] GTID failover mode = 0

    Wed Feb 1 20:04:57 2017 -[info] Dead Servers:

    Wed Feb 1 20:04:57 2017 -[info] Alive Servers:

    Wed Feb 1 20:04:57 2017 -[info] 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:04:57 2017 -[info] 192.168.222.157(192.168.222.157:3306)

    Wed Feb 1 20:04:57 2017 -[info] Alive Slaves:

    Wed Feb 1 20:04:57 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27 (oldest major version between slaves) log-bin:disabled

    Wed Feb 1 20:04:57 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:04:57 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Wed Feb 1 20:04:57 2017 -[info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:04:57 2017 -[info] Checking slave configurations..

    Wed Feb 1 20:04:57 2017 -[warning] log-bin is not set on slave192.168.222.157(192.168.222.157:3306). This host cannot be a master.

    Wed Feb 1 20:04:57 2017 -[info] Checking replication filtering settings..

    Wed Feb 1 20:04:57 2017 -[info] binlog_do_db= , binlog_ignore_db=

    Wed Feb 1 20:04:57 2017 -[info] Replication filtering check ok.

    Wed Feb 1 20:04:57 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln364] None of slaves canbe master. Check failover configuration file or log-bin settings in my.cnf

    Wed Feb 1 20:04:57 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/bin/masterha_check_repl line 48.

    Wed Feb 1 20:04:57 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Wed Feb 1 20:04:57 2017 -[info] Got exit code 1 (Not master dead).

    MySQL Replication Health is NOT OK!

    处理:

    vi /usr/local/mysql/my.cnf

    log_bin=/usr/local/mysql/data/mysql-bin

    #relay_log=/usr/local/mysql/data/relay-bin

    skip_slave_start

    /usr/local/mysql/bin/mysqld_safe --defaults-file=/usr/local/mysql/my.cnf--user=mysql &

    mysql> start slave;

    [root@mysqldb2 mysql]# masterha_check_repl--conf=/etc/masterha/app1.cnf

    Wed Feb 1 20:16:43 2017 -[warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

    Wed Feb 1 20:16:43 2017 -[info] Reading application default configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:16:43 2017 -[info] Reading server configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:16:43 2017 -[info] MHA::MasterMonitor version 0.57.

    Wed Feb 1 20:16:44 2017 -[info] GTID failover mode = 0

    Wed Feb 1 20:16:44 2017 -[info] Dead Servers:

    Wed Feb 1 20:16:44 2017 -[info] Alive Servers:

    Wed Feb 1 20:16:44 2017 -[info] 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:16:44 2017 -[info] 192.168.222.157(192.168.222.157:3306)

    Wed Feb 1 20:16:44 2017 -[info] Alive Slaves:

    Wed Feb 1 20:16:44 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

    Wed Feb 1 20:16:44 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:16:44 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Wed Feb 1 20:16:44 2017 -[info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:16:44 2017 -[info] Checking slave configurations..

    Wed Feb 1 20:16:44 2017 -[info] read_only=1 is not set on slave192.168.222.157(192.168.222.157:3306).

    Wed Feb 1 20:16:44 2017 -[warning] relay_log_purge=0 is not seton slave 192.168.222.157(192.168.222.157:3306).

    Wed Feb 1 20:16:44 2017 -[info] Checking replication filtering settings..

    Wed Feb 1 20:16:44 2017 -[info] binlog_do_db= , binlog_ignore_db=

    Wed Feb 1 20:16:44 2017 -[info] Replication filtering check ok.

    Wed Feb 1 20:16:44 2017 -[info] GTID (with auto-pos) is not supported

    Wed Feb 1 20:16:44 2017 -[info] Starting SSH connection tests..

    Wed Feb 1 20:16:50 2017 -[info] All SSH connection tests passed successfully.

    Wed Feb 1 20:16:50 2017 -[info] Checking MHA Node version..

    Wed Feb 1 20:16:51 2017 -[info] Version check ok.

    Wed Feb 1 20:16:51 2017 -[info] Checking SSH publickey authentication settings on the current master..

    Wed Feb 1 20:16:51 2017 -[info] HealthCheck: SSH to 192.168.222.156 is reachable.

    Wed Feb 1 20:16:51 2017 -[info] Master MHA Node version is 0.57.

    Wed Feb 1 20:16:51 2017 -[info] Checking recovery script configurations on192.168.222.156(192.168.222.156:3306)..

    Wed Feb 1 20:16:51 2017 -[info] Executing command:save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/binlog--output_file=/tmp/save_binary_logs_test --manager_version=0.57--start_file=mysql-bin.000058

    Wed Feb 1 20:16:51 2017 -[info] Connecting toroot@192.168.222.156(192.168.222.156:22)..

    Failed to save binary log: Binlog not found from /home/binlog! Ifyou got this error at MHA Manager, please set"master_binlog_dir=/path/to/binlog_directory_of_the_master" correctlyin the MHA Manager's configuration file and try again.

    at/usr/local/bin/save_binary_logs line 123.

    eval {...} called at/usr/local/bin/save_binary_logs line 70

    main::main() called at/usr/local/bin/save_binary_logs line 66

    Wed Feb 1 20:16:51 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln161] Binlog settingcheck failed!

    Wed Feb 1 20:16:51 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln408] Masterconfiguration failed.

    Wed Feb 1 20:16:51 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/bin/masterha_check_repl line 48.

    Wed Feb 1 20:16:51 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Wed Feb 1 20:16:51 2017 -[info] Got exit code 1 (Not master dead).

    MySQL Replication Health is NOT OK!

    问题12已经处理好,但出现了新的问题,命名为问题13

    问题13:

    [root@mysqldb2 mysql]# masterha_check_repl--conf=/etc/masterha/app1.cnf

    Wed Feb 1 20:16:43 2017 -[warning] Global configuration file /etc/masterha_default.cnf not found.Skipping.

    Wed Feb 1 20:16:43 2017 -[info] Reading application default configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:16:43 2017 -[info] Reading server configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:16:43 2017 -[info] MHA::MasterMonitor version 0.57.

    Wed Feb 1 20:16:44 2017 -[info] GTID failover mode = 0

    Wed Feb 1 20:16:44 2017 -[info] Dead Servers:

    Wed Feb 1 20:16:44 2017 -[info] Alive Servers:

    Wed Feb 1 20:16:44 2017 -[info] 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:16:44 2017 -[info] 192.168.222.157(192.168.222.157:3306)

    Wed Feb 1 20:16:44 2017 -[info] Alive Slaves:

    Wed Feb 1 20:16:44 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

    Wed Feb 1 20:16:44 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:16:44 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Wed Feb 1 20:16:44 2017 -[info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:16:44 2017 -[info] Checking slave configurations..

    Wed Feb 1 20:16:44 2017 -[info] read_only=1 is not set on slave192.168.222.157(192.168.222.157:3306).

    Wed Feb 1 20:16:44 2017 -[warning] relay_log_purge=0 is not seton slave 192.168.222.157(192.168.222.157:3306).

    Wed Feb 1 20:16:44 2017 -[info] Checking replication filtering settings..

    Wed Feb 1 20:16:44 2017 -[info] binlog_do_db= , binlog_ignore_db=

    Wed Feb 1 20:16:44 2017 -[info] Replication filtering check ok.

    Wed Feb 1 20:16:44 2017 -[info] GTID (with auto-pos) is not supported

    Wed Feb 1 20:16:44 2017 -[info] Starting SSH connection tests..

    Wed Feb 1 20:16:50 2017 -[info] All SSH connection tests passed successfully.

    Wed Feb 1 20:16:50 2017 -[info] Checking MHA Node version..

    Wed Feb 1 20:16:51 2017 -[info] Version check ok.

    Wed Feb 1 20:16:51 2017 -[info] Checking SSH publickey authentication settings on the current master..

    Wed Feb 1 20:16:51 2017 -[info] HealthCheck: SSH to 192.168.222.156 is reachable.

    Wed Feb 1 20:16:51 2017 -[info] Master MHA Node version is 0.57.

    Wed Feb 1 20:16:51 2017 -[info] Checking recovery script configurations on192.168.222.156(192.168.222.156:3306)..

    Wed Feb 1 20:16:51 2017 -[info] Executing command:save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/binlog--output_file=/tmp/save_binary_logs_test --manager_version=0.57--start_file=mysql-bin.000058

    Wed Feb 1 20:16:51 2017 -[info] Connecting toroot@192.168.222.156(192.168.222.156:22)..

    Failed to save binary log: Binlog not found from /home/binlog! Ifyou got this error at MHA Manager, please set"master_binlog_dir=/path/to/binlog_directory_of_the_master" correctlyin the MHA Manager's configuration file and try again.

    at/usr/local/bin/save_binary_logs line 123.

    eval {...} called at/usr/local/bin/save_binary_logs line 70

    main::main() called at/usr/local/bin/save_binary_logs line 66

    Wed Feb 1 20:16:51 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln161] Binlog settingcheck failed!

    Wed Feb 1 20:16:51 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln408] Masterconfiguration failed.

    Wed Feb 1 20:16:51 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/bin/masterha_check_repl line 48.

    Wed Feb 1 20:16:51 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Wed Feb 1 20:16:51 2017 -[info] Got exit code 1 (Not master dead).

    处理:

    修改 MHA配置文件中主库二进制日志的路径

    [server default]

    manager_log=/masterha/app1/app1.log

    manager_workdir=/masterha/app1

    #master_binlog_dir=/home/binlog 修改前的配置

    master_binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data

    master_ip_failover_script=/usr/local/bin/master_ip_failover

    master_ip_online_change_script=/usr/local/bin/master_ip_online_change

    password=123456

    ping_interval=1

    remote_workdir=/tmp

    repl_password=123456

    repl_user=repl

    report_script=/usr/local/bin/send_report

    secondary_check_script=/usr/local/bin/masterha_secondary_check -s'192.168.222.157' -s '192.168.222.156' --user=root--master_ip='192.168.222.156' --master_port=3306

    shutdown_script=""

    ssh_user=root

    user=root

    [server1]

    hostname=192.168.222.156

    port=3306

    [server2]

    hostname=192.168.222.157

    port=3306

    candidate_master=1

    check_repl_delay=0

    问题14:Can't exec "mysqlbinlog": No such file or directory at/usr/local/share/perl5/MHA/BinlogManager.pm line 106.

    [root@mysqldb2 mysql]# masterha_check_repl--conf=/etc/masterha/app1.cnf

    Wed Feb 1 20:25:05 2017 -[warning] Global configuration file /etc/masterha_default.cnf not found.Skipping.

    Wed Feb 1 20:25:05 2017 -[info] Reading application default configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:25:05 2017 -[info] Reading server configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:25:05 2017 -[info] MHA::MasterMonitor version 0.57.

    Wed Feb 1 20:25:06 2017 -[info] GTID failover mode = 0

    Wed Feb 1 20:25:06 2017 -[info] Dead Servers:

    Wed Feb 1 20:25:06 2017 -[info] Alive Servers:

    Wed Feb 1 20:25:06 2017 -[info] 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:25:06 2017 -[info] 192.168.222.157(192.168.222.157:3306)

    Wed Feb 1 20:25:06 2017 -[info] Alive Slaves:

    Wed Feb 1 20:25:06 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

    Wed Feb 1 20:25:06 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:25:06 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Wed Feb 1 20:25:06 2017 -[info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:25:06 2017 -[info] Checking slave configurations..

    Wed Feb 1 20:25:06 2017 -[info] read_only=1 is not set on slave192.168.222.157(192.168.222.157:3306).

    Wed Feb 1 20:25:06 2017 -[warning] relay_log_purge=0 is not seton slave 192.168.222.157(192.168.222.157:3306).

    Wed Feb 1 20:25:06 2017 -[info] Checking replication filtering settings..

    Wed Feb 1 20:25:06 2017 -[info] binlog_do_db= , binlog_ignore_db=

    Wed Feb 1 20:25:06 2017 -[info] Replication filtering check ok.

    Wed Feb 1 20:25:06 2017 -[info] GTID (with auto-pos) is not supported

    Wed Feb 1 20:25:06 2017 -[info] Starting SSH connection tests..

    Wed Feb 1 20:25:13 2017 -[info] All SSH connection tests passed successfully.

    Wed Feb 1 20:25:13 2017 -[info] Checking MHA Node version..

    Wed Feb 1 20:25:13 2017 -[info] Version check ok.

    Wed Feb 1 20:25:13 2017 -[info] Checking SSH publickey authentication settings on the current master..

    Wed Feb 1 20:25:13 2017 -[info] HealthCheck: SSH to 192.168.222.156 is reachable.

    Wed Feb 1 20:25:14 2017 -[info] Master MHA Node version is 0.57.

    Wed Feb 1 20:25:14 2017 -[info] Checking recovery script configurations on192.168.222.156(192.168.222.156:3306)..

    Wed Feb 1 20:25:14 2017 -[info] Executing command:save_binary_logs --command=test --start_pos=4--binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=/tmp/save_binary_logs_test--manager_version=0.57 --start_file=mysql-bin.000058

    Wed Feb 1 20:25:14 2017 -[info] Connecting toroot@192.168.222.156(192.168.222.156:22)..

    Creating /tmp if notexists.. ok.

    Checking output directory isaccessible or not..

    ok.

    Binlog found at/app/oracle/oradata2/mysql-5.5.37/data, up to mysql-bin.000058

    Wed Feb 1 20:25:14 2017 -[info] Binlog setting check done.

    Wed Feb 1 20:25:14 2017 -[info] Checking SSH publickey authentication and checking recovery scriptconfigurations on all alive slave servers..

    Wed Feb 1 20:25:14 2017 -[info] Executing command :apply_diff_relay_logs --command=test --slave_user='root'--slave_host=192.168.222.157 --slave_ip=192.168.222.157 --slave_port=3306--workdir=/tmp --target_version=5.6.27-log --manager_version=0.57--relay_log_info=/usr/local/mysql/data/relay-log.info --relay_dir=/usr/local/mysql/data/ --slave_pass=xxx

    Wed Feb 1 20:25:14 2017 -[info] Connecting toroot@192.168.222.157(192.168.222.157:22)..

    Can't exec "mysqlbinlog": No such file or directory at/usr/local/share/perl5/MHA/BinlogManager.pm line 106.

    mysqlbinlog version command failed with rc 1:0, please verify PATH,LD_LIBRARY_PATH, and client options

    at/usr/local/bin/apply_diff_relay_logs line 493.

    Wed Feb 1 20:25:14 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln208] Slaves settingscheck failed!

    Wed Feb 1 20:25:14 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln416] Slave configurationfailed.

    Wed Feb 1 20:25:14 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/bin/masterha_check_repl line 48.

    Wed Feb 1 20:25:14 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Wed Feb 1 20:25:14 2017 -[info] Got exit code 1 (Not master dead).

    MySQL Replication Health is NOT OK!

    处理:

    [root@mysqldb2 mysql]# vi /etc/bashrc

    #在文件的末尾添加

    PATH="$PATH:/usr/local/mysql/bin"

    export PATH

    问题15:Wed Feb 1 20:31:04 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checkingconfigurations. Can't exec "/usr/local/bin/master_ip_failover": Nosuch file or directory at /usr/local/share/perl5/MHA/ManagerUtil.pm line 68.

    [root@mysqldb2 mysql]# masterha_check_repl--conf=/etc/masterha/app1.cnf

    Wed Feb 1 20:30:56 2017 - [warning]Global configuration file /etc/masterha_default.cnf not found. Skipping.

    Wed Feb 1 20:30:56 2017 -[info] Reading application default configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:30:56 2017 -[info] Reading server configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:30:56 2017 -[info] MHA::MasterMonitor version 0.57.

    Wed Feb 1 20:30:57 2017 -[info] GTID failover mode = 0

    Wed Feb 1 20:30:57 2017 -[info] Dead Servers:

    Wed Feb 1 20:30:57 2017 -[info] Alive Servers:

    Wed Feb 1 20:30:57 2017 -[info] 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:30:57 2017 -[info] 192.168.222.157(192.168.222.157:3306)

    Wed Feb 1 20:30:57 2017 -[info] Alive Slaves:

    Wed Feb 1 20:30:57 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

    Wed Feb 1 20:30:57 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:30:57 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Wed Feb 1 20:30:57 2017 -[info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:30:57 2017 -[info] Checking slave configurations..

    Wed Feb 1 20:30:57 2017 -[info] read_only=1 is not set on slave192.168.222.157(192.168.222.157:3306).

    Wed Feb 1 20:30:57 2017 -[warning] relay_log_purge=0 is not seton slave 192.168.222.157(192.168.222.157:3306).

    Wed Feb 1 20:30:57 2017 -[info] Checking replication filtering settings..

    Wed Feb 1 20:30:57 2017 -[info] binlog_do_db= , binlog_ignore_db=

    Wed Feb 1 20:30:57 2017 -[info] Replication filtering check ok.

    Wed Feb 1 20:30:57 2017 -[info] GTID (with auto-pos) is not supported

    Wed Feb 1 20:30:57 2017 -[info] Starting SSH connection tests..

    Wed Feb 1 20:31:03 2017 -[info] All SSH connection tests passed successfully.

    Wed Feb 1 20:31:03 2017 -[info] Checking MHA Node version..

    Wed Feb 1 20:31:03 2017 -[info] Version check ok.

    Wed Feb 1 20:31:03 2017 -[info] Checking SSH publickey authentication settings on the current master..

    Wed Feb 1 20:31:04 2017 -[info] HealthCheck: SSH to 192.168.222.156 is reachable.

    Wed Feb 1 20:31:04 2017 -[info] Master MHA Node version is 0.57.

    Wed Feb 1 20:31:04 2017 -[info] Checking recovery script configurations on 192.168.222.156(192.168.222.156:3306)..

    Wed Feb 1 20:31:04 2017 -[info] Executing command:save_binary_logs --command=test --start_pos=4--binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=/tmp/save_binary_logs_test--manager_version=0.57 --start_file=mysql-bin.000058

    Wed Feb 1 20:31:04 2017 -[info] Connecting toroot@192.168.222.156(192.168.222.156:22)..

    Creating /tmp if notexists.. ok.

    Checking output directory isaccessible or not..

    ok.

    Binlog found at/app/oracle/oradata2/mysql-5.5.37/data, up to mysql-bin.000058

    Wed Feb 1 20:31:04 2017 -[info] Binlog setting check done.

    Wed Feb 1 20:31:04 2017 -[info] Checking SSH publickey authentication and checking recovery scriptconfigurations on all alive slave servers..

    Wed Feb 1 20:31:04 2017 -[info] Executing command :apply_diff_relay_logs --command=test --slave_user='root'--slave_host=192.168.222.157 --slave_ip=192.168.222.157 --slave_port=3306--workdir=/tmp --target_version=5.6.27-log --manager_version=0.57 --relay_log_info=/usr/local/mysql/data/relay-log.info --relay_dir=/usr/local/mysql/data/ --slave_pass=xxx

    Wed Feb 1 20:31:04 2017 -[info] Connecting toroot@192.168.222.157(192.168.222.157:22)..

    Checking slave recoveryenvironment settings..

    Opening /usr/local/mysql/data/relay-log.info... ok.

    Relay log found at/usr/local/mysql/data, up to mysqldb2-relay-bin.000005

    Temporary relay log fileis /usr/local/mysql/data/mysqldb2-relay-bin.000005

    Testing mysql connectionand privileges..Warning: Using a password on the command line interface can beinsecure.

    done.

    Testing mysqlbinlogoutput.. done.

    Cleaning up test file(s)..done.

    Wed Feb 1 20:31:04 2017 -[info] Slaves settings check done.

    Wed Feb 1 20:31:04 2017 -[info]

    192.168.222.156(192.168.222.156:3306) (current master)

    +--192.168.222.157(192.168.222.157:3306)

    Wed Feb 1 20:31:04 2017 -[info] Checking replication health on 192.168.222.157..

    Wed Feb 1 20:31:04 2017 -[info] ok.

    Wed Feb 1 20:31:04 2017 -[info] Checking master_ip_failover_script status:

    Wed Feb 1 20:31:04 2017 -[info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root--orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.156 --orig_master_port=3306

    Wed Feb 1 20:31:04 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. Can't exec"/usr/local/bin/master_ip_failover": No such file or directory at/usr/local/share/perl5/MHA/ManagerUtil.pm line 68.

    Wed Feb 1 20:31:04 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Wed Feb 1 20:31:04 2017 -[info] Got exit code 1 (Not master dead).

    MySQL Replication Health is NOT OK!

    Wed Feb 1 20:31:04 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln229] Failed to get master_ip_failover_scriptstatus with return code 1:0.

    Wed Feb 1 20:31:04 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at /usr/local/bin/masterha_check_replline 48.

    Wed Feb 1 20:31:04 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Wed Feb 1 20:31:04 2017 -[info] Got exit code 1 (Not master dead).

    MySQL Replication Health is NOT OK!

    处理:

    [root@mysqldb2 scripts]# cp master_ip_failover /usr/local/bin

    [root@mysqldb2 scripts]# pwd

    /root/olm/mha/mha4mysql-manager-master/samples/scripts

    问题16:

    [root@mysqldb2 scripts]# masterha_check_repl--conf=/etc/masterha/app1.cnf

    Wed Feb 1 20:40:41 2017 -[warning] Global configuration file /etc/masterha_default.cnf not found.Skipping.

    Wed Feb 1 20:40:41 2017 -[info] Reading application default configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:40:41 2017 -[info] Reading server configuration from /etc/masterha/app1.cnf..

    Wed Feb 1 20:40:41 2017 -[info] MHA::MasterMonitor version 0.57.

    Wed Feb 1 20:40:43 2017 -[info] GTID failover mode = 0

    Wed Feb 1 20:40:43 2017 -[info] Dead Servers:

    Wed Feb 1 20:40:43 2017 -[info] Alive Servers:

    Wed Feb 1 20:40:43 2017 -[info] 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:40:43 2017 -[info] 192.168.222.157(192.168.222.157:3306)

    Wed Feb 1 20:40:43 2017 -[info] Alive Slaves:

    Wed Feb 1 20:40:43 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major versionbetween slaves) log-bin:enabled

    Wed Feb 1 20:40:43 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:40:43 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Wed Feb 1 20:40:43 2017 -[info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

    Wed Feb 1 20:40:43 2017 -[info] Checking slave configurations..

    Wed Feb 1 20:40:43 2017 -[info] read_only=1 is not set on slave192.168.222.157(192.168.222.157:3306).

    Wed Feb 1 20:40:43 2017 -[warning] relay_log_purge=0 is not seton slave 192.168.222.157(192.168.222.157:3306).

    Wed Feb 1 20:40:43 2017 -[info] Checking replication filtering settings..

    Wed Feb 1 20:40:43 2017 -[info] binlog_do_db= , binlog_ignore_db=

    Wed Feb 1 20:40:43 2017 -[info] Replication filtering check ok.

    Wed Feb 1 20:40:43 2017 -[info] GTID (with auto-pos) is not supported

    Wed Feb 1 20:40:43 2017 -[info] Starting SSH connection tests..

    Wed Feb 1 20:40:59 2017 -[info] All SSH connection tests passed successfully.

    Wed Feb 1 20:40:59 2017 -[info] Checking MHA Node version..

    Wed Feb 1 20:40:59 2017 -[info] Version check ok.

    Wed Feb 1 20:40:59 2017 -[info] Checking SSH publickey authentication settings on the current master..

    Wed Feb 1 20:40:59 2017 -[info] HealthCheck: SSH to 192.168.222.156 is reachable.

    Wed Feb 1 20:40:59 2017 -[info] Master MHA Node version is 0.57.

    Wed Feb 1 20:40:59 2017 -[info] Checking recovery script configurations on192.168.222.156(192.168.222.156:3306)..

    Wed Feb 1 20:40:59 2017 -[info] Executing command:save_binary_logs --command=test --start_pos=4--binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=/tmp/save_binary_logs_test--manager_version=0.57 --start_file=mysql-bin.000058

    Wed Feb 1 20:40:59 2017 -[info] Connecting toroot@192.168.222.156(192.168.222.156:22)..

    Creating /tmp if notexists.. ok.

    Checking output directory isaccessible or not..

    ok.

    Binlog found at/app/oracle/oradata2/mysql-5.5.37/data, up to mysql-bin.000058

    Wed Feb 1 20:41:00 2017 -[info] Binlog setting check done.

    Wed Feb 1 20:41:00 2017 -[info] Checking SSH publickey authentication and checking recovery scriptconfigurations on all alive slave servers..

    Wed Feb 1 20:41:00 2017 -[info] Executing command :apply_diff_relay_logs --command=test --slave_user='root'--slave_host=192.168.222.157 --slave_ip=192.168.222.157 --slave_port=3306--workdir=/tmp --target_version=5.6.27-log --manager_version=0.57--relay_log_info=/usr/local/mysql/data/relay-log.info --relay_dir=/usr/local/mysql/data/ --slave_pass=xxx

    Wed Feb 1 20:41:00 2017 -[info] Connecting toroot@192.168.222.157(192.168.222.157:22)..

    Checking slave recoveryenvironment settings..

    Opening/usr/local/mysql/data/relay-log.info ... ok.

    Relay log found at/usr/local/mysql/data, up to mysqldb2-relay-bin.000005

    Temporary relay log fileis /usr/local/mysql/data/mysqldb2-relay-bin.000005

    Testing mysql connectionand privileges..Warning: Using a password on the command line interface can beinsecure.

    done.

    Testing mysqlbinlogoutput.. done.

    Cleaning up test file(s)..done.

    Wed Feb 1 20:41:00 2017 -[info] Slaves settings check done.

    Wed Feb 1 20:41:00 2017 -[info]

    192.168.222.156(192.168.222.156:3306) (current master)

    +--192.168.222.157(192.168.222.157:3306)

    Wed Feb 1 20:41:00 2017 -[info] Checking replication health on 192.168.222.157..

    Wed Feb 1 20:41:00 2017 -[info] ok.

    Wed Feb 1 20:41:00 2017 -[info] Checking master_ip_failover_script status:

    Wed Feb 1 20:41:00 2017 -[info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root--orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.156 --orig_master_port=3306

    Bareword "FIXME_xxx" not allowed while "strictsubs" in use at /usr/local/bin/master_ip_failover line 93.

    Execution of /usr/local/bin/master_ip_failover aborted due tocompilation errors.

    Wed Feb 1 20:41:00 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln229] Failed to get master_ip_failover_scriptstatus with return code 255:0.

    Wed Feb 1 20:41:00 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/bin/masterha_check_repl line 48.

    Wed Feb 1 20:41:00 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Wed Feb 1 20:41:00 2017 -[info] Got exit code 1 (Not master dead).

    MySQL Replication Health is NOT OK!

    处理:

    暂时注释 master_ip_failover 脚本

    ############################################################################################################################

    Keepalived

    ############################################################################################################################

    问题17:

    ./configure --prefix=/usr/local/keepalived

    configure: error:

    !!! OpenSSL is not properlyinstalled on your system. !!!

    !!! Can not include OpenSSLheaders files. !!!

    处理:

    yum -y install openssl-devel

    问题18:

    [root@mysqldb2 keepalived-1.3.2]# make

    Making all in lib

    make[1]: Entering directory `/root/olm/mha/keepalived-1.3.2/lib'

    make all-am

    make[2]: Entering directory `/root/olm/mha/keepalived-1.3.2/lib'

    CC memory.o

    CC utils.o

    CC notify.o

    notify.c: 在函数‘find_path’中:

    notify.c:226: 警告:此函数中的‘sgid_num’在使用前可能未初始化

    CC timer.o

    CC scheduler.o

    CC vector.o

    CC list.o

    CC html.o

    CC parser.o

    CC signals.o

    CC logger.o

    CC rttables.o

    AR liblib.a

    make[2]: Leaving directory `/root/olm/mha/keepalived-1.3.2/lib'

    make[1]: Leaving directory `/root/olm/mha/keepalived-1.3.2/lib'

    Making all in keepalived

    make[1]: Entering directory `/root/olm/mha/keepalived-1.3.2/keepalived'

    Making all in core

    make[2]: Entering directory`/root/olm/mha/keepalived-1.3.2/keepalived/core'

    CC main.o

    CC daemon.o

    CC pidfile.o

    CC layer4.o

    CC smtp.o

    CC global_data.o

    CC global_parser.o

    CC process.o

    CC namespaces.o

    namespaces.c: 在函数‘setns’中:

    namespaces.c:187: 错误:‘SYS_setns’未声明(在此函数内第一次使用)

    namespaces.c:187: 错误:(即使在一个函数内多次出现,每个未声明的标识符在其

    namespaces.c:187: 错误:所在的函数内也只报告一次。)

    make[2]: *** [namespaces.o] 错误 1

    make[2]: Leaving directory`/root/olm/mha/keepalived-1.3.2/keepalived/core'

    make[1]: *** [all-recursive] 错误 1

    make[1]: Leaving directory`/root/olm/mha/keepalived-1.3.2/keepalived'

    make: *** [all-recursive] 错误 1

    处理:

    --安装 glibc 库仍无法解决问题

    yum -y install glibc*

    --使用低版本keepalived (keepalived-1.2.18)

    问题19:

    [root@oradb sysbench]# ./sysbench --test=olm--oltp-table-size=1000000 --oltp-read-only=off --init-rng=on --num-threads=16--max-requests=0 --oltp-dist-type=uniform --max-time=1800 --mysql-user=root--db-driver=mysql --mysql-table-engine=innodb --oltp-test-mode=complexprepare

    ./sysbench: /usr/lib64/libmysqlclient.so.18: no version informationavailable (required by ./sysbench)

    处理:

    ln -s /app/oracle/oradata2/mysql-5.6.27/lib/libmysqlclient.so/usr/lib64/libmysqlclient.so.18

    问题:20

    [root@mysqldb2 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf

    Thu Feb 2 19:13:17 2017 -[warning] Global configuration file /etc/masterha_default.cnf not found.Skipping.

    Thu Feb 2 19:13:17 2017 -[info] Reading application default configuration from /etc/masterha/app1.cnf..

    Thu Feb 2 19:13:17 2017 -[info] Reading server configuration from /etc/masterha/app1.cnf..

    Thu Feb 2 19:13:17 2017 -[info] MHA::MasterMonitor version 0.57.

    Thu Feb 2 19:13:17 2017 -[error][/usr/local/share/perl5/MHA/Server.pm, ln180] Got MySQL error whenconnecting 192.168.222.157(192.168.222.157:3306) :1045:Access denied for user'root'@'mysqldb2' (using password: YES), but this is not a MySQL crash. CheckMySQL server settings.

    Thu Feb 2 19:13:17 2017 -[error][/usr/local/share/perl5/MHA/ServerManager.pm, ln301] at/usr/local/share/perl5/MHA/ServerManager.pm line 297.

    Thu Feb 2 19:13:18 2017 -[error][/usr/local/share/perl5/MHA/ServerManager.pm, ln309] Got fatal error,stopping operations

    Thu Feb 2 19:13:18 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened onchecking configurations. at/usr/local/share/perl5/MHA/MasterMonitor.pm line 329.

    Thu Feb 2 19:13:18 2017 -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened onmonitoring servers.

    Thu Feb 2 19:13:18 2017 -[info] Got exit code 1 (Not master dead).

    MySQL Replication Health is NOT OK!

    原因:从库没有MHA监控用户

    处理:

    1.再主库创建监控用户

    2.然后导出DUMP文件

    3.导入备库创建 主从复制

    4.执行权限刷新 flush privileges;

    问题21:

    Thu Feb 2 19:56:56 2017 -[info] Connecting to the target slave host 192.168.222.157, running recoverscript..

    Thu Feb 2 19:56:56 2017 -[info] Executing command: apply_diff_relay_logs --command=apply--slave_user='root' --slave_host=192.168.222.157 --slave_ip=192.168.222.157 --slave_port=3306--apply_files=/tmp/saved_master_binlog_from_192.168.222.156_3306_20170202195650.binlog--workdir=/tmp --target_version=5.6.27-log --timestamp=20170202195650--handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57--slave_pass=xxx

    Thu Feb 2 19:56:57 2017 -[info]

    MySQL client version is 5.6.27. Using --binary-mode.

    Applying differential binary/relay log files/tmp/saved_master_binlog_from_192.168.222.156_3306_20170202195650.binlog on192.168.222.157:3306. This may take long time.

    .....

    Thu Feb 2 19:56:57 2017 -[info]

    MySQL client version is 5.6.27. Using --binary-mode.

    Applying differential binary/relay log files/tmp/saved_master_binlog_from_192.168.222.156_3306_20170202195650.binlog on192.168.222.157:3306. This may take long time...

    FATAL: applying log files failed with rc 1:0!

    Error logs frommysqldb2:/tmp/relay_log_apply_for_192.168.222.157_3306_20170202195650_err.log(the last 200 lines)..

    .....

    at line 197: Cannot execute statement: impossible to write to binarylog since statement is in row format and BINLOG_FORMAT = STATEMENT

    ....

    --------------

    Bye

    at/usr/local/bin/apply_diff_relay_logs line 515.

    eval {...} called at/usr/local/bin/apply_diff_relay_logs line 475

    main::main() called at/usr/local/bin/apply_diff_relay_logs line 120

    Thu Feb 2 19:56:57 2017 -[error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln1398] Applying diffs failed with return code 22:0.

    Thu Feb 2 19:56:57 2017 -[error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln1561] Recovering masterserver failed.

    Thu Feb 2 19:56:57 2017 -[error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln177] Got ERROR: at /usr/local/bin/masterha_manager line 65.

    Thu Feb 2 19:56:57 2017 -[info]

    ----- Failover Report -----

    app1: MySQL Master failover 192.168.222.156(192.168.222.156:3306)

    Master 192.168.222.156(192.168.222.156:3306) is down!

    Check MHA Manager logs at mysqldb2:/masterha/app1/app1.log fordetails.

    Started automated(non-interactive) failover.

    Invalidated master IP address on192.168.222.156(192.168.222.156:3306)

    The latest slave 192.168.222.157(192.168.222.157:3306) has all relaylogs for recovery.

    Selected 192.168.222.157(192.168.222.157:3306) as a new master.

    Recovering master server failed.

    Got Error so couldn't continue failover from here.

    原因:二进制日志格式错误,需要修改为行格式

    处理:

    1.修改主库和备库的二进制日志格式

    vi my.cnf

    #添加

    binlog_format=row

    重启主备库MYSQL

    2. 启动mha监控

    nohup masterha_manager --conf=/etc/masterha/app1.cnf--remove_dead_master_conf --ignore_last_failover< /dev/null>/masterha/app1/manager.log 2>&1 &

    Thu Feb 2 20:24:51 2017 -[info] MHA::MasterMonitor version 0.57.

    Thu Feb 2 20:24:52 2017 -[info] GTID failover mode = 0

    Thu Feb 2 20:24:52 2017 -[info] Dead Servers:

    Thu Feb 2 20:24:52 2017 -[info] Alive Servers:

    Thu Feb 2 20:24:52 2017 -[info] 192.168.222.156(192.168.222.156:3306)

    Thu Feb 2 20:24:52 2017 -[info] 192.168.222.157(192.168.222.157:3306)

    Thu Feb 2 20:24:52 2017 -[info] Alive Slaves:

    Thu Feb 2 20:24:52 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major versionbetween slaves) log-bin:enabled

    Thu Feb 2 20:24:52 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Thu Feb 2 20:24:52 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Thu Feb 2 20:24:52 2017 -[info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

    Thu Feb 2 20:24:52 2017 -[info] Checking slave configurations..

    Thu Feb 2 20:24:52 2017 -[info] read_only=1 is not set on slave192.168.222.157(192.168.222.157:3306).

    Thu Feb 2 20:24:52 2017 -[warning] relay_log_purge=0 is not seton slave 192.168.222.157(192.168.222.157:3306).

    Thu Feb 2 20:24:52 2017 -[info] Checking replication filtering settings..

    Thu Feb 2 20:24:52 2017 -[info] binlog_do_db= , binlog_ignore_db=

    Thu Feb 2 20:24:52 2017 -[info] Replication filtering check ok.

    Thu Feb 2 20:24:52 2017 -[info] GTID (with auto-pos) is not supported

    Thu Feb 2 20:24:52 2017 -[info] Starting SSH connection tests..

    Thu Feb 2 20:24:54 2017 -[info] All SSH connection tests passed successfully.

    Thu Feb 2 20:24:54 2017 -[info] Checking MHA Node version..

    Thu Feb 2 20:24:54 2017 -[info] Version check ok.

    Thu Feb 2 20:24:54 2017 -[info] Checking SSH publickey authentication settings on the current master..

    Thu Feb 2 20:24:54 2017 -[info] HealthCheck: SSH to 192.168.222.156 is reachable.

    Thu Feb 2 20:24:55 2017 -[info] Master MHA Node version is 0.57.

    Thu Feb 2 20:24:55 2017 -[info] Checking recovery script configurations on192.168.222.156(192.168.222.156:3306)..

    Thu Feb 2 20:24:55 2017 -[info] Executing command:save_binary_logs --command=test --start_pos=4--binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=/tmp/save_binary_logs_test--manager_version=0.57 --start_file=mysql-bin.000064

    Thu Feb 2 20:24:55 2017 -[info] Connecting toroot@192.168.222.156(192.168.222.156:22)..

    Creating /tmp if notexists.. ok.

    Checking output directory isaccessible or not..

    ok.

    Binlog found at/app/oracle/oradata2/mysql-5.5.37/data, up to mysql-bin.000064

    Thu Feb 2 20:24:55 2017 -[info] Binlog setting check done.

    Thu Feb 2 20:24:55 2017 -[info] Checking SSH publickey authentication and checking recovery scriptconfigurations on all alive slave servers..

    Thu Feb 2 20:24:55 2017 -[info] Executing command :apply_diff_relay_logs --command=test --slave_user='root'--slave_host=192.168.222.157 --slave_ip=192.168.222.157 --slave_port=3306--workdir=/tmp --target_version=5.6.27-log --manager_version=0.57--relay_log_info=/usr/local/mysql/data/relay-log.info --relay_dir=/usr/local/mysql/data/ --slave_pass=xxx

    Thu Feb 2 20:24:55 2017 -[info] Connecting toroot@192.168.222.157(192.168.222.157:22)..

    Checking slave recovery environmentsettings..

    Opening/usr/local/mysql/data/relay-log.info ... ok.

    Relay log found at/usr/local/mysql/data, up to relay-bin.000005

    Temporary relay log fileis /usr/local/mysql/data/relay-bin.000005

    Testing mysql connectionand privileges..Warning: Using a password on the command line interface can beinsecure.

    done.

    Testing mysqlbinlogoutput.. done.

    Cleaning up test file(s)..done.

    Thu Feb 2 20:24:56 2017 -[info] Slaves settings check done.

    Thu Feb 2 20:24:56 2017 - [info]

    192.168.222.156(192.168.222.156:3306) (current master)

    +--192.168.222.157(192.168.222.157:3306)

    Thu Feb 2 20:24:56 2017 -[info] Checking master_ip_failover_script status:

    Thu Feb 2 20:24:56 2017 -[info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root--orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.156--orig_master_port=3306

    Thu Feb 2 20:24:56 2017 -[info] OK.

    Thu Feb 2 20:24:56 2017 -[warning] shutdown_script is not defined.

    Thu Feb 2 20:24:56 2017 - [info]Set master ping interval 1 seconds.

    Thu Feb 2 20:24:56 2017 -[info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s oradb --user=root --master_host=oradb--master_ip='192.168.222.156' --master_port=3306

    Thu Feb 2 20:24:56 2017 -[info] Starting ping health check on 192.168.222.156(192.168.222.156:3306)..

    Thu Feb 2 20:24:56 2017 -[info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

    3. 关闭主库

    mysqladmin -u root shutdown

    4. 启动MHA监控

    nohup masterha_manager --conf=/etc/masterha/app1.cnf--remove_dead_master_conf --ignore_last_failover< /dev/null>/masterha/app1/manager.log 2>&1 &

    Thu Feb 2 20:37:56 2017 -[warning] Got error on MySQL select ping: 2013 (Lost connection to MySQL serverduring query)

    Thu Feb 2 20:37:56 2017 -[info] Executing secondary network check script:/usr/local/bin/masterha_secondary_check -s oradb --user=root --master_host=oradb --master_ip='192.168.222.156'--master_port=3306 --user=root --master_host=192.168.222.156 --master_ip=192.168.222.156 --master_port=3306 --master_user=root--master_password=123456 --ping_type=SELECT

    Thu Feb 2 20:37:56 2017 -[info] Executing SSH check script: save_binary_logs --command=test--start_pos=4 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data--output_file=/tmp/save_binary_logs_test --manager_version=0.57--binlog_prefix=mysql-bin

    Thu Feb 2 20:37:57 2017 -[warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on'192.168.222.156' (111))

    Thu Feb 2 20:37:57 2017 -[warning] Connection failed 2 time(s)..

    Thu Feb 2 20:37:58 2017 -[warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on'192.168.222.156' (111))

    Thu Feb 2 20:37:58 2017 -[warning] Connection failed 3 time(s)..

    Thu Feb 2 20:37:59 2017 - [warning]Got error on MySQL connect: 2003 (Can't connect to MySQL server on'192.168.222.156' (111))

    Thu Feb 2 20:37:59 2017 -[warning] Connection failed 4 time(s)..

    Thu Feb 2 20:38:01 2017 -[warning] HealthCheck: Got timeout on checking SSH connection to192.168.222.156! at /usr/local/share/perl5/MHA/HealthCheck.pm line 342.

    Monitoring server oradb is reachable, Master is not reachable fromoradb. OK.

    Thu Feb 2 20:38:08 2017 -[info] Master is not reachable from all other monitoring servers. Failover shouldstart.

    Thu Feb 2 20:38:08 2017 -[warning] Master is not reachable from health checker!

    Thu Feb 2 20:38:08 2017 -[warning] Master 192.168.222.156(192.168.222.156:3306) is not reachable!

    Thu Feb 2 20:38:08 2017 -[warning] SSH is NOT reachable.

    Thu Feb 2 20:38:08 2017 -[info] Connecting to a master server failed. Reading configuration file/etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying toconnect to all servers to check server status..

    Thu Feb 2 20:38:08 2017 -[warning] Global configuration file /etc/masterha_default.cnf not found.Skipping.

    Thu Feb 2 20:38:08 2017 -[info] Reading application default configuration from /etc/masterha/app1.cnf..

    Thu Feb 2 20:38:08 2017 -[info] Reading server configuration from /etc/masterha/app1.cnf..

    Thu Feb 2 20:38:09 2017 -[warning] SQL Thread is stopped(no error) on192.168.222.157(192.168.222.157:3306)

    Thu Feb 2 20:38:09 2017 -[info] GTID failover mode = 0

    Thu Feb 2 20:38:09 2017 -[info] Dead Servers:

    Thu Feb 2 20:38:09 2017 -[info] 192.168.222.156(192.168.222.156:3306)

    Thu Feb 2 20:38:09 2017 -[info] Alive Servers:

    Thu Feb 2 20:38:09 2017 -[info] 192.168.222.157(192.168.222.157:3306)

    Thu Feb 2 20:38:09 2017 -[info] Alive Slaves:

    Thu Feb 2 20:38:09 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major versionbetween slaves) log-bin:enabled

    Thu Feb 2 20:38:09 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Thu Feb 2 20:38:09 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Thu Feb 2 20:38:09 2017 -[info] Checking slave configurations..

    Thu Feb 2 20:38:09 2017 -[info] read_only=1 is not set on slave192.168.222.157(192.168.222.157:3306).

    Thu Feb 2 20:38:09 2017 -[warning] relay_log_purge=0 is not seton slave 192.168.222.157(192.168.222.157:3306).

    Thu Feb 2 20:38:09 2017 -[info] Checking replication filtering settings..

    Thu Feb 2 20:38:09 2017 -[info] Replication filtering check ok.

    Thu Feb 2 20:38:09 2017 -[info] Master is down!

    Thu Feb 2 20:38:09 2017 -[info] Terminating monitoring script.

    Thu Feb 2 20:38:09 2017 -[info] Got exit code 20 (Master dead).

    Thu Feb 2 20:38:09 2017 -[info] MHA::MasterFailover version 0.57.

    Thu Feb 2 20:38:09 2017 -[info] Starting master failover.

    Thu Feb 2 20:38:09 2017 -[info]

    Thu Feb 2 20:38:09 2017 -[info] * Phase 1: Configuration Check Phase..

    Thu Feb 2 20:38:09 2017 -[info]

    Thu Feb 2 20:38:10 2017 -[warning] SQL Thread is stopped(no error) on 192.168.222.157(192.168.222.157:3306)

    Thu Feb 2 20:38:10 2017 -[info] GTID failover mode = 0

    Thu Feb 2 20:38:10 2017 -[info] Dead Servers:

    Thu Feb 2 20:38:10 2017 -[info] 192.168.222.156(192.168.222.156:3306)

    Thu Feb 2 20:38:10 2017 -[info] Checking master reachability via MySQL(double check)...

    Thu Feb 2 20:38:10 2017 -[info] ok.

    Thu Feb 2 20:38:10 2017 -[info] Alive Servers:

    Thu Feb 2 20:38:10 2017 -[info] 192.168.222.157(192.168.222.157:3306)

    Thu Feb 2 20:38:10 2017 -[info] Alive Slaves:

    Thu Feb 2 20:38:10 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

    Thu Feb 2 20:38:10 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Thu Feb 2 20:38:10 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Thu Feb 2 20:38:10 2017 -[info] Starting SQL thread on192.168.222.157(192.168.222.157:3306) ..

    Thu Feb 2 20:38:10 2017 -[info] done.

    Thu Feb 2 20:38:10 2017 -[info] Starting Non-GTID based failover.

    Thu Feb 2 20:38:10 2017 -[info]

    Thu Feb 2 20:38:10 2017 -[info] ** Phase 1: Configuration Check Phase completed.

    Thu Feb 2 20:38:10 2017 -[info]

    Thu Feb 2 20:38:10 2017 -[info] * Phase 2: Dead Master Shutdown Phase..

    Thu Feb 2 20:38:10 2017 -[info]

    Thu Feb 2 20:38:10 2017 -[info] Forcing shutdown so that applications never connect to the currentmaster..

    Thu Feb 2 20:38:10 2017 -[info] Executing master IP deactivation script:

    Thu Feb 2 20:38:10 2017 -[info] /usr/local/bin/master_ip_failover --orig_master_host=192.168.222.156--orig_master_ip=192.168.222.156 --orig_master_port=3306 --command=stop

    Thu Feb 2 20:38:10 2017 -[info] done.

    Thu Feb 2 20:38:10 2017 -[warning] shutdown_script is not set. Skipping explicit shutting down of thedead master.

    Thu Feb 2 20:38:10 2017 -[info] * Phase 2: Dead Master Shutdown Phase completed.

    Thu Feb 2 20:38:10 2017 -[info]

    Thu Feb 2 20:38:10 2017 -[info] * Phase 3: Master Recovery Phase..

    Thu Feb 2 20:38:10 2017 -[info]

    Thu Feb 2 20:38:10 2017 -[info] * Phase 3.1: Getting Latest Slaves Phase..

    Thu Feb 2 20:38:10 2017 -[info]

    Thu Feb 2 20:38:10 2017 -[info] The latest binary log file/position on all slaves ismysql-bin.000064:120

    Thu Feb 2 20:38:10 2017 -[info] Latest slaves (Slaves that received relay log files to the latest):

    Thu Feb 2 20:38:11 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

    Thu Feb 2 20:38:11 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Thu Feb 2 20:38:11 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Thu Feb 2 20:38:11 2017 -[info] The oldest binary log file/position on all slaves is mysql-bin.000064:120

    Thu Feb 2 20:38:11 2017 -[info] Oldest slaves:

    Thu Feb 2 20:38:11 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

    Thu Feb 2 20:38:11 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Thu Feb 2 20:38:11 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] * Phase 3.2: Saving Dead Master's Binlog Phase..

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[warning] Dead Master is not SSH reachable. Could not save it's binlogs.Transactions that were not sent to the latest slave (Read_Master_Log_Pos to thetail of the dead master's binlog) were lost.

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] * Phase 3.3: Determining New Master Phase..

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] Finding the latest slave that has all relay logs for recovering otherslaves..

    Thu Feb 2 20:38:11 2017 -[info] All slaves received relay logs to the same position. No need to resynceach other.

    Thu Feb 2 20:38:11 2017 -[info] Searching new master from slaves..

    Thu Feb 2 20:38:11 2017 -[info] Candidate masters from theconfiguration file:

    Thu Feb 2 20:38:11 2017 -[info] 192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

    Thu Feb 2 20:38:11 2017 -[info] Replicating from192.168.222.156(192.168.222.156:3306)

    Thu Feb 2 20:38:11 2017 -[info] Primary candidate for the newMaster (candidate_master is set)

    Thu Feb 2 20:38:11 2017 -[info] Non-candidate masters:

    Thu Feb 2 20:38:11 2017 -[info] Searching from candidate_masterslaves which have received the latest relay log events..

    Thu Feb 2 20:38:11 2017 -[info] New master is 192.168.222.157(192.168.222.157:3306)

    Thu Feb 2 20:38:11 2017 -[info] Starting master failover..

    Thu Feb 2 20:38:11 2017 -[info]

    From:

    192.168.222.156(192.168.222.156:3306) (current master)

    +--192.168.222.157(192.168.222.157:3306)

    To:

    192.168.222.157(192.168.222.157:3306) (new master)

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] * Phase 3.3: New Master Diff Log Generation Phase..

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] This server has all relay logs.No need to generate diff files from the latest slave.

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] * Phase 3.4: Master Log Apply Phase..

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] *NOTICE: If any error happens from this phase, manual recovery isneeded.

    Thu Feb 2 20:38:11 2017 -[info] Starting recovery on 192.168.222.157(192.168.222.157:3306)..

    Thu Feb 2 20:38:11 2017 -[info] This server has all relay logs.Waiting all logs to be applied..

    Thu Feb 2 20:38:11 2017 -[info] done.

    Thu Feb 2 20:38:11 2017 -[info] All relay logs were successfullyapplied.

    Thu Feb 2 20:38:11 2017 - [info]Getting new master's binlog name and position..

    Thu Feb 2 20:38:11 2017 -[info] mysql-bin.000003:120

    Thu Feb 2 20:38:11 2017 -[info] All other slaves should startreplication from here. Statement should be: CHANGE MASTER TOMASTER_HOST='192.168.222.157', MASTER_PORT=3306,MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=120, MASTER_USER='repl',MASTER_PASSWORD='xxx';

    Thu Feb 2 20:38:11 2017 -[info] Executing master IP activate script:

    Thu Feb 2 20:38:11 2017 -[info] /usr/local/bin/master_ip_failover--command=start --ssh_user=root --orig_master_host=192.168.222.156--orig_master_ip=192.168.222.156 --orig_master_port=3306--new_master_host=192.168.222.157 --new_master_ip=192.168.222.157--new_master_port=3306 --new_master_user='root' --new_master_password=xxx

    Undefined subroutine &main::FIXME_xxx_create_user called at/usr/local/bin/master_ip_failover line 89.

    Set read_only=0 on the new master.

    Creating app user on the new master..

    Thu Feb 2 20:38:11 2017 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm,ln1589] Failed to activate master IPaddress for 192.168.222.157(192.168.222.157:3306) with return code 10:0

    Thu Feb 2 20:38:11 2017 -[warning] Proceeding.

    Thu Feb 2 20:38:11 2017 -[info] ** Finished master recovery successfully.

    Thu Feb 2 20:38:11 2017 -[info] * Phase 3: Master Recovery Phase completed.

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] * Phase 4: Slaves Recovery Phase..

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] Generating relay diff files from the latest slave succeeded.

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] All new slave servers recovered successfully.

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] * Phase 5: New master cleanup phase..

    Thu Feb 2 20:38:11 2017 -[info]

    Thu Feb 2 20:38:11 2017 -[info] Resetting slave info on the new master..

    Thu Feb 2 20:38:11 2017 -[info] 192.168.222.157: Resetting slaveinfo succeeded.

    Thu Feb 2 20:38:11 2017 -[info] Master failover to 192.168.222.157(192.168.222.157:3306) completedsuccessfully.

    Thu Feb 2 20:38:11 2017 -[info] Deleted server1 entry from /etc/masterha/app1.cnf .

    Thu Feb 2 20:38:11 2017 -[info]

    ----- Failover Report -----

    app1: MySQL Master failover 192.168.222.156(192.168.222.156:3306) to192.168.222.157(192.168.222.157:3306) succeeded

    Master 192.168.222.156(192.168.222.156:3306) is down!

    Check MHA Manager logs at mysqldb2:/masterha/app1/app1.log fordetails.

    Started automated(non-interactive) failover.

    Invalidated master IP address on192.168.222.156(192.168.222.156:3306)

    The latest slave 192.168.222.157(192.168.222.157:3306) has all relaylogs for recovery.

    Selected 192.168.222.157(192.168.222.157:3306) as a new master.

    192.168.222.157(192.168.222.157:3306): OK: Applying all logssucceeded.

    Failed to activate master IP address for192.168.222.157(192.168.222.157:3306) with return code 10:0

    Generating relay diff files from the latest slave succeeded.

    192.168.222.157(192.168.222.157:3306): Resetting slave infosucceeded.

    Master failover to 192.168.222.157(192.168.222.157:3306) completedsuccessfully.


    MYSQL主从复制部署MHA自动故障转移.docx

    将本文的Word文档下载到电脑

    推荐度:

    下载
    热门标签: mhamysql故障