nginx

Nginx使用proxy_bind负载tcp socket，解决代理端口耗尽

Posted by Sunday on 2019-01-06

流量升高导致TIME_WAIT增加，连接大量失败的问题
nginx将php解析通过TCP转发给php-fpm,需要占用一个TCP

PHP-FPM TIME_WAIT

1
2
3
4
5
6
netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
SYN_RECV 186
TIME_WAIT 49245
ESTABLISHED 211
FIN_WAIT1 333
TIME_WAIT

TIME_WAIT 有4W多个 ,随着时间推移还会极速增加中，端口也就65536个, 用完了自然就没了!!

1
2
3
4
5
6
7
8
vim /etc/sysctl.conf

tcp_max_tw_buckets = 100000
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_tw_reuse = 1
#net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_syncookies = 1

保存后执行 sysctl -p 生效

net.ipv4.tcp_syncookies = 1开启SYN Cookies。当出现SYN等待队列溢出时，启用cookies来处理，可防范少量SYN攻击，默认为0
net.ipv4.tcp_tw_reuse = 1 开启重用。允许将TIME-WAIT套接字重新用于新的TCP连接，默认为0
net.ipv4.tcp_tw_recycle = 1 开启TCP连接中TIME-WAIT套接字的快速回收，默认为0
net.ipv4.tcp_fin_timeout = 30 修改默认的超时时间

修改后time_wait会大量减少

Nginx Proxy 突破64k

1
2
3
4
ifconfig em1:0 192.168.1.11/24 up
ifconfig em1:1 192.168.1.12/24 up
ifconfig em1:2 192.168.1.13/24 up
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
    split_clients "$remote_addr$remote_port" $split_ip {
        10%  192.168.1.11
        10%  192.168.1.12
        10%  192.168.1.13
        10%  192.168.1.14
        10%  192.168.1.15
        10%  192.168.1.16
        10%  192.168.1.17
        10%  192.168.1.18
        10%  192.168.1.19
        *    192.168.1.10
    }

    upstream backend {
    server 10.0.0.100:1234;
    server 10.0.0.101:1234;
    keepalive 128;
    }

    location / {
        proxy_bind $split_ip;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_http_version 1.1; 
        proxy_set_header Connection "";
    }

测试机

1
wrk -t4 -c10000  -d30s http://xxx.com/index.html

未添加proxy_bind

1
2
3
4
5
6
7
8
9
10
11
 ss -s
Total: 1340 (kernel 45585)
TCP:   51547 (estab 871, closed 50655, orphaned 0, synrecv 0, timewait 50655/0), ports 0

Transport Total     IP        IPv6
*     45585     -         -        
RAW   0         0         0        
UDP   1         1         0        
TCP   892       889       3        
INET      893       890       3        
FRAG      0         0         0

添加proxy_bind 简单测试突破65535端口

1
2
3
4
5
6
7
8
9
10
11
12
ss -s

Total: 21274 (kernel 51268)
TCP:   112189 (estab 20799, closed 91367, orphaned 1, synrecv 0, timewait 91365/0), ports 0

Transport Total     IP        IPv6
*     51268     -         -        
RAW   0         0         0        
UDP   1         1         0        
TCP   20822       20819       3        
INET      20823       20820       3        
FRAG      0         0         0

克服NGINX端口耗尽
使用HAProxy对300k并发tcp套接字连接进行负载均衡 端口耗尽 Keep-alive
nginx proxy_bind

CATALOG
    FEATURED TAGS
    nginx
    FRIENDS