容器云负载均衡之二：从IPVS DR模式下director不能访问VIP问题的探究

本文链接：https://blog.csdn.net/cloudvtech/article/details/80145568

一、前言

在《IPVS概览》一文中提到IPVS的Direct Routing模式是一个扩展性比较好的负载均衡方式。但是在默认的情况下，IPVS的Direct Routing模式需要指定一个Director，这个director负责绑定并对外暴露VIP，这个director所在的node上的进程是无法访问VIP获取后端real service的服务的。

在一般系统中，这可能不是问题，因为IPVS director作为单独的节点进行部署；但是在kubernetes环境中，IPVS被做成POD和service之后，再给IPVS单独指定worker节点，就会造成节点资源的浪费。本文就是基于这个目的，探究director上无法访问VIP的原因及其解决方案，以便于基于kubernetes部署IPVS服务。

转载自https://blog.csdn.net/cloudvtech

二、无法从IPVS director访问VIP服务

2.1 机器配置

IPVS director： 192.168.166.102/
IPVS real server：192.168.166.103/
VIP：192.168.166.111

real server上的服务是80端口的HTTP服务

2.2 IPVS director的设置

   ipvsadm -C
   ipvsadm -A -t 192.168.166.111 -s rr
   ipvsadm -a -t  192.168.166.111:80 -r 192.168.166.103:80 -w 1 -g

   ifconfig  ens33:0 192.168.166.111 broadcast 192.168.166.255 netmask 255.255.255.0 up
   route add -host 192.168.166.111  dev ens33:0

/etc/sysctl.conf

net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

2.3 IPVS real server的设置

/etc/sysctl.conf

net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.ens33.arp_ignore = 1
net.ipv4.conf.ens33.arp_announce = 2

ifconfig lo:0 192.168.166.111 broadcast 192.168.166.255 netmask 255.255.255.255 up
route add -host 192.168.166.111  dev lo:0

并且启动http服务

2.4 从外部可以访问VIP并且获得后端real server HTTP服务的返回

2.5 从director(192.168.166.102)访问VIP失败

curl 192.168.166.111

超时

IPVS的状态信息

[root@k8s-node1 ~]#  ipvsadm -L -n --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  192.168.166.111:80                  1        4        0      240        0
  -> 192.168.166.103:80                  1        4        0      240        0

director上的tcpdump

09:57:42.919671 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 783029 ecr 0,nop,wscale 7], length 0
09:57:42.920165 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 779155 ecr 783029,nop,wscale 7], length 0
09:57:43.921996 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 784032 ecr 0,nop,wscale 7], length 0
09:57:43.922565 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 780157 ecr 783029,nop,wscale 7], length 0
09:57:45.124180 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 781359 ecr 783029,nop,wscale 7], length 0
09:57:45.925794 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 786036 ecr 0,nop,wscale 7], length 0
09:57:45.926194 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 782161 ecr 783029,nop,wscale 7], length 0
09:57:48.127567 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 784363 ecr 783029,nop,wscale 7], length 0
09:57:49.937754 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 790048 ecr 0,nop,wscale 7], length 0
09:57:49.938057 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 786174 ecr 783029,nop,wscale 7], length 0
09:57:54.137227 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 790373 ecr 783029,nop,wscale 7], length 0
09:58:02.151601 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 798389 ecr 783029,nop,wscale 7], length 0
09:58:18.173820 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 814413 ecr 783029,nop,wscale 7], length 0

real server上的tcpdump

09:57:42.943819 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 783029 ecr 0,nop,wscale 7], length 0
09:57:42.943892 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 779155 ecr 783029,nop,wscale 7], length 0
09:57:43.946328 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 784032 ecr 0,nop,wscale 7], length 0
09:57:43.946385 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 780157 ecr 783029,nop,wscale 7], length 0
09:57:45.148027 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 781359 ecr 783029,nop,wscale 7], length 0
09:57:45.950287 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 786036 ecr 0,nop,wscale 7], length 0
09:57:45.950333 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 782161 ecr 783029,nop,wscale 7], length 0
09:57:48.151818 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 784363 ecr 783029,nop,wscale 7], length 0
09:57:49.962678 IP 192.168.166.102.33846 > 192.168.166.111.http: Flags [S], seq 831475674, win 43690, options [mss 65495,sackOK,TS val 790048 ecr 0,nop,wscale 7], length 0
09:57:49.962719 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 786174 ecr 783029,nop,wscale 7], length 0
09:57:54.162245 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 790373 ecr 783029,nop,wscale 7], length 0
09:58:02.177762 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 798389 ecr 783029,nop,wscale 7], length 0
09:58:18.201944 IP 192.168.166.111.http > 192.168.166.102.33846: Flags [S.], seq 2689589272, ack 831475675, win 28960, options [mss 1460,sackOK,TS val 814413 ecr 783029,nop,wscale 7], length 0

2.6 分析

看起来IPVS director以及顺利将请求转发到后端real server，并且real server也向源地址(director所在的192.168.166.102)返回了响应，这个响应也顺利到达了director，但是没有被上层应用接收到。

分析最后的数据包也未见异常：

Frame 2: 76 bytes on wire (608 bits), 76 bytes captured (608 bits)
    Encapsulation type: Linux cooked-mode capture (25)
    Arrival Time: Apr 30, 2018 23:13:02.824544000 CST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1525101182.824544000 seconds
    [Time delta from previous captured frame: 0.000468000 seconds]
    [Time delta from previous displayed frame: 0.000468000 seconds]
    [Time since reference or first frame: 0.000468000 seconds]
    Frame Number: 2
    Frame Length: 76 bytes (608 bits)
    Capture Length: 76 bytes (608 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: sll:ethertype:ip:tcp]
    [Coloring Rule Name: HTTP]
    [Coloring Rule String: http || tcp.port == 80 || http2]
Linux cooked capture
    Packet type: Unicast to us (0)
    Link-layer address type: 1
    Link-layer address length: 6
    Source: Vmware_75:24:37 (00:0c:29:75:24:37)
    Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.168.166.111, Dst: 192.168.166.102
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 60
    Identification: 0x0000 (0)
    Flags: 0x02 (Don't Fragment)
        0... .... = Reserved bit: Not set
        .1.. .... = Don't fragment: Set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (6)
    Header checksum: 0x6c95 [validation disabled]
        [Good: False]
        [Bad: False]
    Source: 192.168.166.111
    Destination: 192.168.166.102
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
Transmission Control Protocol, Src Port: 80 (80), Dst Port: 39852 (39852), Seq: 0, Ack: 1, Len: 0
    Source Port: 80
    Destination Port: 39852
    [Stream index: 0]
    [TCP Segment Len: 0]
    Sequence number: 0    (relative sequence number)
    Acknowledgment number: 1    (relative ack number)
    Header Length: 40 bytes
    Flags: 0x012 (SYN, ACK)
    Window size value: 28960
    [Calculated window size: 28960]
    Checksum: 0x2589 [validation disabled]
    Urgent pointer: 0
    Options: (20 bytes), Maximum segment size, SACK permitted, Timestamps, No-Operation (NOP), Window scale
        Maximum segment size: 1460 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 5297287, TSecr 5302934
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1]
        [The RTT to ACK the segment was: 0.000468000 seconds]

转载自https://blog.csdn.net/cloudvtech

三、使用iptables进行debug

根据文章《iptables概览》：

mangle表是在所有五个chain里面都有check point的表，所以可以在mangle表加入LOG target进行包的跟踪。

3.1 在director和real server的iptables插入debug LOG target如下

iptables -t mangle -A PREROUTING -s 192.168.166.102 -d 192.168.166.111 -j LOG --log-prefix "[PREROUTING|mangle] "
iptables -t mangle -A PREROUTING -d 192.168.166.102 -s 192.168.166.111 -j LOG --log-prefix "[PREROUTING|mangle] "
iptables -t mangle -A INPUT -s 192.168.166.102 -d 192.168.166.111 -j LOG --log-prefix "[INPUT|mangle] "
iptables -t mangle -A INPUT -d 192.168.166.102 -s 192.168.166.111 -j LOG --log-prefix "[INPUT|mangle] "
iptables -t mangle -A FORWARD -s 192.168.166.102 -d 192.168.166.111 -j LOG --log-prefix "[FORWARD|mangle] "
iptables -t mangle -A FORWARD -d 192.168.166.102 -s 192.168.166.111 -j LOG --log-prefix "[FORWARD|mangle] "
iptables -t mangle -A OUTPUT -s 192.168.166.102 -d 192.168.166.111 -j LOG --log-prefix "[OUTPUT|mangle] "
iptables -t mangle -A OUTPUT -d 192.168.166.102 -s 192.168.166.111 -j LOG --log-prefix "[OUTPUT|mangle] "
iptables -t mangle -A POSTROUTING -s 192.168.166.102 -d 192.168.166.111 -j LOG --log-prefix "[POSTROUTING|mangle] "
iptables -t mangle -A POSTROUTING -d 192.168.166.102 -s 192.168.166.111 -j LOG --log-prefix "[POSTROUTING|mangle] "

3.2 curl VIP:80

超时

3.3 查看director的iptables的LOG输出如下

Apr 30 18:32:19 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=lo SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0 
Apr 30 18:32:19 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0 
Apr 30 18:32:19 k8s-node1 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0 
Apr 30 18:32:19 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0 
Apr 30 18:32:19 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:19 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:20 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:20 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:22 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:22 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:26 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:26 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:35 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:35 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:51 k8s-node1 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:51 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0

3.4 查看real server的iptables的LOG输出如下

Apr 30 18:32:17 k8s-node2 kernel: [PREROUTING|raw] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0 
Apr 30 18:32:17 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0 
Apr 30 18:32:17 k8s-node2 kernel: [PREROUTING|nat] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0 
Apr 30 18:32:17 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0 
Apr 30 18:32:17 k8s-node2 kernel: INPUT|filter1IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50165 DF PROTO=TCP SPT=44598 DPT=80 WINDOW=43690 RES=0x00 SYN URGP=0 
Apr 30 18:32:17 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:17 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:17 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:17 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:18 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:18 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:18 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:18 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:21 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:21 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:21 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:21 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:25 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:25 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:25 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:25 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:33 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:33 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:33 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:33 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:49 k8s-node2 kernel: [OUTPUT|raw] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:49 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:49 k8s-node2 kernel: OUTPUT|filter1IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 18:32:49 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44598 WINDOW=28960 RES=0x00 ACK SYN URGP=0

3.5 分析

看起来数据包在进入director的mangel表PREROUTING链之后就丢失了。

查阅网络上相关问题，最后发现这个问题( https://www.linuxquestions.org/questions/linux-networking-3/packets-lost-after-mangle-prerouting-chain-4175437227/)与本文的问题类似，是关于地址验证的宽容度rp_filter设置的。

linux内核参数列表( https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)里面这样描述这个参数的：

rp_filter - INTEGER
	0 - No source validation.
	1 - Strict mode as defined in RFC3704 Strict Reverse Path
	    Each incoming packet is tested against the FIB and if the interface
	    is not the best reverse path the packet check will fail.
	    By default failed packets are discarded.
	2 - Loose mode as defined in RFC3704 Loose Reverse Path
	    Each incoming packet's source address is also tested against the FIB
	    and if the source address is not reachable via any interface
	    the packet check will fail.

	Current recommended practice in RFC3704 is to enable strict mode
	to prevent IP spoofing from DDos attacks. If using asymmetric routing
	or other complicated routing, then loose mode is recommended.

	The max value from conf/{all,interface}/rp_filter is used
	when doing source validation on the {interface}.

	Default value is 0. Note that some distributions enable it
	in startup scripts.

rp_filter参数有三个值，0、1、2，具体含义：
0：不开启源地址校验。
1：开启严格的反向路径校验。对每个进来的数据包，校验其反向路径是否是最佳路径。如果反向路径不是最佳路径，则直接丢弃该数据包。
2：开启松散的反向路径校验。对每个进来的数据包，校验其源地址是否可达，即反向路径是否能通（通过任意网口），如果反向路径不同，则直接丢弃该数据包。

所以看看将rp_filter设置成0，不进行任何源地址校验，是否可以让数据包往上层走。

转载自https://blog.csdn.net/cloudvtech

四、设置rp_filter为0继续进行测试

4.1 在sysctl.conf中加入如下配置

net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.ens33.rp_filter = 0

运行sysctl -p

4.2 curl VIP:80可以访问都后端HTTP服务

4.3 查看director的iptables的LOG输出如下

Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58279 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58279 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58279 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58280 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58280 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58280 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=58281 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK PSH URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=58281 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK PSH URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=58281 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK PSH URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2183 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2183 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=5201 TOS=0x00 PREC=0x00 TTL=64 ID=2184 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK PSH URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=5201 TOS=0x00 PREC=0x00 TTL=64 ID=2184 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK PSH URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58282 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58282 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58282 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58283 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK FIN URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58283 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK FIN URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58283 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK FIN URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2188 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK FIN URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:0c:29:75:24:37:08:00 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2188 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK FIN URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58284 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58284 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0 
Apr 30 23:55:45 k8s-node1 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58284 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0

4.4 查看real server的iptables的LOG输出如下

Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58279 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58279 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58280 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58280 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=58281 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK PSH URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=58281 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=229 RES=0x00 ACK PSH URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2183 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2183 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=4396 TOS=0x00 PREC=0x00 TTL=64 ID=2184 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=4396 TOS=0x00 PREC=0x00 TTL=64 ID=2184 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=857 TOS=0x00 PREC=0x00 TTL=64 ID=2187 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK PSH URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=857 TOS=0x00 PREC=0x00 TTL=64 ID=2187 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK PSH URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58282 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58282 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58283 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK FIN URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58283 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK FIN URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2188 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK FIN URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [POSTROUTING|mangle] IN= OUT=ens33 SRC=192.168.166.111 DST=192.168.166.102 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2188 DF PROTO=TCP SPT=80 DPT=44612 WINDOW=227 RES=0x00 ACK FIN URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [PREROUTING|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58284 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0 
Apr 30 23:55:43 k8s-node2 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.102 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58284 DF PROTO=TCP SPT=44612 DPT=80 WINDOW=309 RES=0x00 ACK URGP=0

转载自https://blog.csdn.net/cloudvtech

五、references

https://bugzilla.redhat.com/show_bug.cgi?id=1261410

http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.lvs_clients_on_realservers.html