keepalived nginx

Keepalived Nginx双网络(内外网)故障非同步漂移双主模式

Posted by Sunday on 2018-08-13

简介

有了keepalived+Lvs这样的高性能组合，为什么还需keepalived+Nginx呢。keepalived是为了Lvs而设计。Lvs是一个四层的负载均衡设备，虽然有着高性能的优势，但它无后端服务器的健康检查机制。keepalived为lvs提供一系列的健康检查机制，例如：TCP_CHECK,UDP_CHECK,HTTP_GET等。同时lvs也可以自己写健康检查脚脚本。或者结合ldirectory来实现后端健康检测。但LVS始终无法摆脱它是一个四层设备，无法对上层协议进行解析。而Nginx就不一样了，Nginx是一个七层的设备可以对七层协议进行解析，可以对一些请求进行过滤，还可以对请求结果进行缓存。这些都是Nginx独有的优势。但是keepalived并没有为Nginx提供健康检测。需要自己去写一些脚步来进行健康检测。

下面主要讲解Keepalived+Nginx的模式，不包含lvs。如果不是大型负载，一般用不到LVS，当然你也可以参阅：Keepalived + LVS-DR Nginx 双主

环境

准备四台服务器或虚拟机：

Web Nginx 内网：192.168.10.111/192.168.10.112
Keepalived 内网：192.168.10.101(ka101)/192.168.10.102(ka102)
Keepalived 公网：172.16.20.101/172.16.20.102
注：上面的ip请在 /etc/sysconfig/network-scripts/ifcfg-eth0 eth1 中固定配置。

Keepalived 内网VIP：192.168.10.91/192.168.10.92
Keepalived 公网VIP：172.16.20.91/172.16.20.92
OS：CentOS 7.4

先决条件

安装keepalived。
时间同步。
设置SELinux和防火墙。
互相之间/etc/hosts文件添加对方主机名（可选）。
确认网络接口支持多播（组播）新网卡默认支持。

以上部署请参阅：Keepavlied 编译 配置文件讲解及主备演示

Keepalived配置

ka101

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance Intranet_1 {
    state MASTER
    interface eth0
    virtual_router_id 171
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday0
    }
    virtual_ipaddress {
        192.168.10.91
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"  
}
vrrp_instance Intranet_2 {
    state BACKUP
    interface eth0
    virtual_router_id 172
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday1
    }
    virtual_ipaddress {
        192.168.10.92
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"  
}

vrrp_instance Extranet_1 {
    state MASTER
    interface eth1
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday2
    }
    virtual_ipaddress {
        172.16.20.91
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"          
}
vrrp_instance Extranet_2 {
    state BACKUP
    interface eth1
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday3
    }
    virtual_ipaddress {
        172.16.20.92
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"          
}

ka102配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance Intranet_1 {
    state BACKUP
    interface eth0
    virtual_router_id 171
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday0
    }
    virtual_ipaddress {
        192.168.10.91
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"  
}
vrrp_instance Intranet_2 {
    state MASTER
    interface eth0
    virtual_router_id 172
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday1
    }
    virtual_ipaddress {
        192.168.10.92
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"  
}

vrrp_instance Extranet_1 {
    state BACKUP
    interface eth1
    virtual_router_id 191
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday2
    }
    virtual_ipaddress {
        172.16.20.91
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"          
}
vrrp_instance Extranet_2 {
    state MASTER
    interface eth1
    virtual_router_id 192
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday3
    }
    virtual_ipaddress {
        172.16.20.92
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"          
}

启动Keepalived

ka101启动keepalived 及网卡状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[root@ka1 ~]# systemctl start keepalived
[root@ka1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:70:a1:05 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.101/24 brd 192.168.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.10.91/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.10.92/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe70:a105/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:70:a1:0f brd ff:ff:ff:ff:ff:ff
    inet 172.16.20.101/16 brd 172.16.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 172.16.20.91/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet 172.16.20.92/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe70:a10f/64 scope link 
       valid_lft forever preferred_lft forever

在ka102没有启动时，ka101添加了4个VIP。分别是

公网eth1: 172.16.20.91 172.16.20.92
内网eth0: 192.168.10.91 192.168.10.92

启动ka102后查看其网卡状态：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[root@ka1 ~]# systemctl start keepalived
[root@ka1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:4d:16:b3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.102/24 brd 192.168.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.10.92/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe4d:16b3/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:4d:16:bd brd ff:ff:ff:ff:ff:ff
    inet 172.16.20.102/16 brd 172.16.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 172.16.20.92/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe4d:16bd/64 scope l

ka102添加了2个VIP，分别是

公网eth1: 172.16.20.92
内网eth0: 192.168.10.92

再查看ka101网卡状态
172.16.20.92和192.168.10.92 已被移除。此时停止任意一台服务器（ka101、ka102）,4个VIP都通正常通信。切换过程有1秒延迟。

查看组播

1
2
3
4
5
6
7
8
9
[root@ka1 ~]# yum install tcpdump -y
[root@ka1 ~]# tcpdump -nn -i eth0 host 224.0.0.111
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:15:47.653995 IP 172.16.20.101 > 224.0.0.111: VRRPv2, Advertisement, vrid 191, prio 100, authtype simple, intvl 1s, length 20
15:15:47.654009 IP 192.168.10.102 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20
15:15:47.878423 IP 192.168.10.102 > 224.0.0.111: VRRPv2, Advertisement, vrid 172, prio 100, authtype simple, intvl 1s, length 20
15:15:47.878811 IP 172.16.20.102 > 224.0.0.111: VRRPv2, Advertisement, vrid 192, prio 100, authtype simple, intvl 1s, length 20
15:15:48.657193 IP 172.16.20.101 > 224.0.0.111: VRRPv2, Advertisement, vrid 191, prio 100, authtype simple, intvl 1s, length 20

配置后端Nginx

分别在后端服务器 192.168.10.111/192.168.10.112 安装Nginx

1
2
yum install epel-release -y
yum install nginx -y

测试环境为区分机器的不同，故将web页面设置服务器IP地址，但在生产环境中获取的内容是一致的。
分别在192.168.10.111/192.168.10.112执行如下命令：

1
2
echo "Server 192.168.10.111" > /usr/share/nginx/html/index.html
echo "Server 192.168.10.112" > /usr/share/nginx/html/index.html

测试是否访问正常

1
2
curl http://192.168.10.111
Server 192.168.10.111

配置Keepalived Ngin反向代理

分别在ka101/ka102上安装Nginx，我这里用yum安装
CentOS 7精简安装后，使用中发现没有killall命令。安装psmisc即可

1
2
yum install nginx psmisc -y
mv /etc/nginx/conf.d/default.conf{,.bak}

配置nginx反向代理

1
vim  /etc/nginx/conf.d/default.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
upstream webserverapps {
    server 192.168.10.111:80;
    server 192.168.10.112:80;
    #server 127.0.0.1:8080 backup;
   }

server {
    listen 80;
    server_name _;
    location / {
        proxy_pass http://webserverapps;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        client_max_body_size 10m;
        client_body_buffer_size 128k;
        proxy_connect_timeout 90;
        proxy_send_timeout 90;
        proxy_read_timeout 90;
        proxy_buffer_size 4k;
        proxy_buffers 4 32k;
        proxy_busy_buffers_size 64k;
        proxy_temp_file_write_size 64k;
        add_header Access-Control-Allow-Origin *;
    }
}

注意：生产环境请根据自己需求调整配置。

在ka101/ka102重启Nginx服务：

1
systemctl restart nginx

分别在ka101/ka102上测试：

1
2
3
4
5
6
7
[root@ka101 ~]# for i in `seq 6`; do curl 192.168.10.91; done
Server 192.168.10.111 
Server 192.168.10.112
Server 192.168.10.111 
Server 192.168.10.112 
Server 192.168.10.111 
Server 192.168.10.112

到目前为止，Nginx反代功能也已实现，下面我们将把Nginx与Keepalived结合起来，使Nginx支持高可用。

配置Keepalived Nginx高可用

分别在ka101/ka102配置文件/etc/keepalived/keepalived.conf的全局配置块global_defs下方添加vrrp_script配置块

1
2
3
4
5
6
7
vrrp_script chk_nginx {
    script "killall -0 nginx"
    interval 2
    weight -10
    fall 2
    rise 2
}

在所有vrrp_instance实例块里，添加track_script块：

1
2
3
track_script {
    chk_nginx
}

Keepalived最终配置

ka101

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   vrrp_mcast_group4 224.0.0.111
}
vrrp_script chk_nginx {
    script "killall -0 nginx"
    interval 2
    weight -10
    fall 2
    rise 2
}
vrrp_instance Intranet_1 {
    state MASTER
    interface eth0
    virtual_router_id 171
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday0
    }
    virtual_ipaddress {
        192.168.10.91
    }
    track_script {
    chk_nginx
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"  
}
vrrp_instance Intranet_2 {
    state BACKUP
    interface eth0
    virtual_router_id 172
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday1
    }
    virtual_ipaddress {
        192.168.10.92
    }
    track_script {
    chk_nginx
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"  
}
vrrp_instance Extranet_1 {
    state MASTER
    interface eth1
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday2
    }
    virtual_ipaddress {
        172.16.20.91
    }
    track_script {
    chk_nginx
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"          
}
vrrp_instance Extranet_2 {
    state BACKUP
    interface eth1
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sunday3
    }
    virtual_ipaddress {
        172.16.20.92
    }
    track_script {
    chk_nginx
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"          
}

客户端测试

1
2
3
4
5
6
7
[root@sunday ~]# for i in `seq 2`; do curl 172.16.20.92; done
Server 192.168.10.111
Server 192.168.10.112

[root@sunday ~]# for i in `seq 2`; do curl 192.168.10.91;done
Server 192.168.10.111
Server 192.168.10.112

在配置过程中出现了无法漂移的情况，跨网段问题。解决通道，还是要多看日志，多分析判断，最终还是能解决问题的。无论在何种情况下，既然选择了keepalived，就要坚信自己的初心。
如你在配置过程中出现任何问题，欢迎留言，共同解决问题。
转载来源: https://renwole.com/archives/1118

CATALOG
    FEATURED TAGS
    keepalived nginx
    FRIENDS