高可用 keepalived

Keepalived 编译及配置文件详解

Posted by Sunday on 2018-08-09

Keepalived简介

Keepalived是一个基于vrrp来实现的服务器高可用解决方案，可以利用其实现避免IP单点故障，类似的工具还有heartbeat、corosync。不过其不会单独出现，而是搭配着 LVS、Nginx、HAproxy，一起协同工作达到高可用的目的。

具体来说，其作用是检测服务器的状态，如果有一台web服务器宕机，或工作出现故障，KeepAlived将会检测到，并将有故障的服务器从系统中剔除，同时使用其他服务器代替该服务器的工作，当服务器工作正常后KeepAlived自动将服务器加入到服务器群中，这些工作全部自动完成，不需要人工干涉，需要人工做的只是修复故障的服务器。

VRRP协议

VRRP全称Vritual Router Redundancy Protocol，虚拟路由冗余协议。通过把几台提供路由功能的设备组成一个虚拟路由设备，使用一定的机制保证虚拟路由的高可用，从而达到保持业务的连续性与可靠性。

在配置组成的一个虚拟路由器中，有MASTER和BACKUP之分。MASTER是主节点，在一个虚拟路由器中，只能有一个MASTER，但可以有多个BACKUP，BACKUP是备用节点，也就是当master挂掉之后，BACKUP接管MASTER节点的所有资源，当有多个BACKUP节点时，根据其priority(优先级)的值的大小，来选举谁作为MASTER的替代者。当BACKUP节点的优先级值相同时，根据其IP地址的大小，来决定。

条件

  • 节点之间的时间必须同步。
  • 确保Firewalld及SELinux不会成为阻碍。
  • 各节点用于集群服务的网络接口必须支持MULTICAST（多播）通信。采用D类地址(224-239)。多播地址建议手动定义，因为若有多个集群服务都使用默认的，虽有认证机制，但仍会互发信息，可能会影响性能，更会产生无用日志信息。

时间同步

Firewalld配置

1
2
3
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface eth0 --destination 224.0.0.111 --protocol vrrp -j ACCEPT
firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface eth0 --destination 224.0.0.111 --protocol vrrp -j ACCEPT
firewall-cmd --reload

安装keepalived

1
2
3
4
5
6
yum install gcc openssl-devel libnl-devel libnfnetlink-devel ipvsadm -y
# apt-get install build-essential libssl-dev libnl-3-dev libnl-genl-3-dev libnl-route-3-dev
./configure --prefix=/usr/local/keepalived
make -j4 && make install
ln -s /sbin/keepalived /usr/sbin/
ln -s /etc/keepalived /etc/keepalived

修改keepalived日志路径

1
2
3
4
5
ln -sv /etc/sysconfig/keepalived /etc/sysconfig/
sed -i 's!KEEPALIVED_OPTIONS.*!KEEPALIVED_OPTIONS="-D -d -S 0"!' /etc/sysconfig/keepalived
echo "local0.*                                                /var/log/keepalived.log" >> /etc/rsyslog.conf
systemctl restart rsyslog keepalived
netstat -anp|grep syslog

启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
cat << EOF > /etc/systemd/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=syslog.target network-online.target

[Service]
Type=forking
PIDFile=/var/run/keepalived.pid
KillMode=process
EnvironmentFile=-/etc/sysconfig/keepalived
ExecStart=/usr/sbin/keepalived \$KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP \$MAINPID

[Install]
WantedBy=multi-user.targe
EOF

keepalived配置文件讲解

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
# 全局配置
global_defs {               # 全局定义
   notification_email {     # 通知邮件相关设置
    acassen@firewall.loc    # 通知邮件收件人
   }
   notification_email_from keepalived@sundayle.com  # 发件人
   smtp_server 127.0.0.1    # 使用本机邮件服务
   smtp_connect_timeout 30  # 设置连接smtp server的超时时间
   router_id LVS_DEVEL      # 标识当前节点唯一,节点间不能相同,默认hostname
   vrrp_mcast_group4 224.0.0.111 # vrrp的多播地址(ipv4)
   #vrrp_skip_check_adv_addr # 检查vrrp报文中的所有地址比较耗时 #设置此标志的意思是如果接收的到报文和上一个报文来至同一个路由器，则不执行检查。默认是跳过检查
   #vrrp_strict              # 严格执行VRRP协议规范，此模式不支持节点单播
   #vrrp_garp_interval 0     # 在一个网卡上每组gratuitous arp消息之间的延迟时间,默认为0
   #vrrp_gna_interval 0      # 在一个网卡上每组na消息之间的延迟时间,默认为0,单位秒
}
#检测脚本,定义keepalived在什么情况切换主备
vrrp_script chk_nginx {
    script "killall -0 nginx" # nginx进程存在返回0,否则返回非0 #也可以调用脚本 如script "/etc/keepalived/ngx_status.sh"
    interval 2                # 每2秒检测一次
    weight -10            # 检测失败(脚本返回值为非0数字)时，优先级减10
    fall 2                # 连续检测2次失败才算真正的失败。才会执行上面的weight -10,减去优先级
    rise 2                # 检测2次成功才算成功，不修改优先级
}

vrrp_sync_group VG_1 { # 同步切换
    group {
        VI_1
        VI_2
    }
}
# vrrp_sync_group/vrrp_instance 主要对外提供服务VIP区域及其相关属性
vrrp_instance VI_1 {     # VRRP实例
    state MASTER         # 只能有一个是MASTER,余下的都应该为BACKUP；
    interface eth1       # 对外提供网络的接口
    virtual_router_id 65 # 虚拟路由id标识,数字,必须和backup里相同(范围0-255)
    priority 100         # 优先级,数字必须比backup大
    advert_int 1         # 组播信息发送间隔,两个节点设置必须一样,秒
    authentication {     # 设置验证信息,两个节点必须一致(明文)
        auth_type PASS
        auth_pass 1111
    }
    # 虚拟地址，即VIP
    virtual_ipaddress {  # 可简写为单个地址，系统会默认计算掩码和设备
        192.168.10.10
      # 192.168.10.10/24
      # 192.168.10.10/24 dev eth1
    }
    track_script {
    chk_nginx
    }
    # 定义通知脚本（另添加）
        notify_master "/etc/keepalived/notify.sh master"
        notify_backup "/etc/keepalived/notify.sh backup"
        notify_fault "/etc/keepalived/notify.sh fault"
}

# 此区域是LVS配置。若用Keepalived+LVS，需要这段配置，若用其他，例如：Keepalived+Nginx，则无需配置。
# LVS包含两个子配置块：virtual_server_group/virtual_server
# virtual_server：虚拟服务器。每个虚拟服务器里面包含多个真实服务器real_server。

virtual_server 192.168.10.10 80 { # 虚拟IP 监听80端口
    delay_loop 6                  # 健康检查时间间隔,秒
    lb_algo rr                    # 负载调度算法,常见使用wlc或rr
    lb_kind NAT                   # LVS负载转发规则,DR,NAT,TUN等
    persistence_timeout 50        # 会话保持时间,秒
    protocol TCP                  # 转发协议一般有tcp和udp两种

    real_server 192.168.10.101 80 { # 配置真实服务器的地址与端口
        weight 1                  # 权重
        SSL_GET {
            url {
              path /health              # 健康检查的页面
              digest ff20ad2481f97b1754ef3e12ecd3a9cc # 计算出的MD5值
            }
            url {
              path /mrtg/
              digest 9b3a0c85a887a256d6939da88aabd8cd
            }
            connect_timeout 3     # 连接超时时间,秒
            nb_get_retry 3        # 失败重试次数,超过后移除
            delay_before_retry 3  # 失败重试间隔,秒
        }
    }
    real_server 192.168.10.102 80 {
        ...
    }
}
vrrp_instance VI_2 { 
...
}

检测脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
cat /etc/keepalived/ngx_status.sh 
#只有当script的返回状态结果为任意非0数字时，才会执行降权操作。
#当script正常执行时，也就是返回值为0时，不做任何操作。
#脚本示例：可以在script中调用，直接引用脚本路径即可
#该脚本检测ngnix的运行状态，并在nginx进程不存在时尝试重新启动ngnix，如果启动失败则停止keepalived，准备让其它机器接管
#!/bin/bash
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
    /usr/local/bin/nginx
    sleep 2
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
        /etc/init.d/keepalived stop
    fi
fi

notify邮件通知脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
cat /etc/keepalived/notify.sh

#!/bin/bash
contact='root@localhost'
                
notify() {
    local mailsubject="$(hostname) to be $1, vip floating"
    local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
    echo "$mailbody" | mail -s "$mailsubject" $contact
}
                
case $1 in
master)
    notify master   
    ;;
backup)
    notify backup   
    ;;
fault)
    notify fault    
    ;;
*)
    echo "Usage: $(basename $0) {master|backup|fault}"
    exit 1
    ;;
esac

启动服务

1
2
systemctl enable keepalived 
systemctl start keepalived

单主备实例

架构图

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
    +------+              +--------+
    |Client|     >>>      |Internet|
    +------+              +--------+
                              /\
                  +-----------------------+
                  | 公网VIP1:192.168.10.91|
                  +-----------------------+
                      /                \
+--------------------------+      +--------------------------+
| KA+Lvs/Nginx/HAProxy     |      | KA+Lvs/Nginx/HAProxy     |
| VIP1:Master (eth1)       |多播IP | VIP1:BACKUP (eth1)       |
| IP1:192.168.10.101(eth1) |      | IP1:192.168.10.102(eth1) |
+--------------------------+      +--------------------------+
                     \                /
                +-----------------------+
                | 公网VIP1:192.168.10.91 |
                +-----------------------+
                           \/
                        +------+
                        | Web  |
                        +------+

环境

MASTER: 192.168.10.101
BACKUP: 192.168.10.102
VIP: 192.168.10.91
OS: CentOS 7.4

Master配置文件

1
vim /etc/keepalived/keepalived.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka101
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance VG_1 {
    state MASTER
    interface eth1
    virtual_router_id 100
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday
    }
    virtual_ipaddress {
        192.168.10.91
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"          
}

Backup配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka102
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance VG_1 {
    state BACKUP
    interface eth1
    virtual_router_id 100
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday
    }
    virtual_ipaddress {
        192.168.10.91
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"          
}

notify.sh 邮件通知脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/bin/bash
contact='root@localhost'
                
notify() {
    local mailsubject="$(hostname) to be $1, vip floating"
    local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
    echo "$mailbody" | mail -s "$mailsubject" $contact
}
                
case $1 in
master)
    notify master   
    ;;
backup)
    notify backup   
    ;;
fault)
    notify fault    
    ;;
*)
    echo "Usage: $(basename $0) {master|backup|fault}"
    exit 1
    ;;
esac

主备测试

未启动keepalived 网卡信息

1
2
3
[root@ka101 ~]# ip a | grep eth1
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.168.10.101/24 brd 192.168.10.255 scope global eth1

启动keepalived

1
2
[root@ka101 ~]# systemctl start keepalived
[root@ka102 ~]# systemctl start keepalived

启动keepalived后 网卡信息

1
2
3
4
[root@ka101 ~]# ip a | grep eth1
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.168.10.101/24 brd 192.168.10.255 scope global eth1
    inet 192.168.10.91/32 scope global eth1

已成功添加VIP 192.168.10.91

停止Master keepalived

1
[root@ka101 ~]# systemctl stop keepalived

此时VIP已漂移到Backup主机

1
2
3
4
[root@ka102 ~]# ip a | grep eth1
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.168.10.102/24 brd 192.168.10.255 scope global eth1
    inet 192.168.10.91/32 scope global eth1

查看日志

1
2
3
4
5
6
[root@ka102 ~]# cat /var/log/messages
ug  9 18:18:27 ka102 Keepalived_vrrp[3535]: VRRP_Instance(VG_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.10.91
Aug  9 18:18:27 ka102 Keepalived_vrrp[3535]: Sending gratuitous ARP on eth1 for 192.168.10.91
Aug  9 18:18:27 ka102 Keepalived_vrrp[3535]: Sending gratuitous ARP on eth1 for 192.168.10.91
Aug  9 18:18:27 ka102 Keepalived_vrrp[3535]: Sending gratuitous ARP on eth1 for 192.168.10.91
Aug  9 18:18:27 ka102 Keepalived_vrrp[3535]: Sending gratuitous ARP on eth1 for 192.168.10.91

恢复Master

1
2
3
4
5
[root@ka102 ~]# systemctl start keepalived
[root@ka101 ~]# ip a | grep eth1
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.168.10.101/24 brd 192.168.10.255 scope global eth1
    inet 192.168.10.91/32 scope global eth1

以上状态表明，当MASTER恢复服务后，BACKUP的Keepalived会自动漂移到MASTER上。因为MASTER的权重值比BACKUP高。以上是BACKUP的漂移到MASTER的状态。

https://renwole.com/archives/1107

CATALOG
    FEATURED TAGS
    高可用 keepalived
    FRIENDS