在kubernetes1.28上发现nodeport不在node节点上建立监听端口及外部浏览器无法访问serivce为nodeport类型服务….

env

  • kubernetes-v1.28.8
  • kube-proxy ipvs
  • openEuler 20.03 (LTS-SP3) 2c4g (control-plan + worker)

K8S的1.24.0以上kube-proxy不监听nodeport端口问题

问题

service类型NodePort,pod正常运行,在node上没发现监听端口及浏览器无法访问

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

# Source: redis-stack/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: "redis-stack"
  namespace: default
  labels:
    app: "redis-stack"
spec:
  ports:
  - port: 6379
    name: "redis-stack"
    targetPort: 6379
  - port: 8001
    name: "redis-insight"
    targetPort: 8001
  type: NodePort
  selector:
    app: "redis-stack"
---
# Source: redis-stack/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: "redis-stack"
  namespace: default
  labels:
    app: "redis-stack"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "redis-stack"
  template:
    metadata:
      labels:
        app: "redis-stack"
    spec:
      containers:
      - name: "redis-stack"
        image: redis/redis-stack:7.2.0-v10
        imagePullPolicy: Always
        ports:
        - containerPort: 6379
          name: db
        - containerPort: 8001
          name: "redis-insight"

log

//查看集群信息
[root@euler-200 tmp]# kubectl get node
NAME        STATUS   ROLES                  AGE   VERSION
euler-200   Ready    control-plane,worker   11d   v1.28.8

//查看serivce信息，看到nodeport类型监听端口
    6379:30619/TCP,8001:31559/TCP

[root@euler-200 tmp]# kubectl get svc
NAME               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
external-service   ExternalName   <none>          gaga.local    80/TCP                          11d
kubernetes         ClusterIP      10.233.0.1      <none>        443/TCP                         11d
redis-stack        NodePort       10.233.43.183   <none>        6379:30619/TCP,8001:31559/TCP   52m    //8001:31559

//查看ipsvs监听信息
[root@euler-200 ~]# ipvsadm -Ln |grep 31559
TCP  169.254.25.10:31559 rr
TCP  172.20.20.200:31559 rr
TCP  172.24.20.200:31559 rr
TCP  10.233.73.0:31559 rr

//在node上curl发现是通的
[root@euler-200 tmp]# curl  172.24.20.200:31559
<!doctype html><html lang="en" dir="ltr"><head><meta charset="UTF-8"/><meta name="viewport" content="width=device-width,initial-scale=1,shrink-to-fit=no"><title>RedisInsight</title><base href="/"/><script defer="defer" src="/js/bundle.runtime.min.js"></script><script defer="defer" src="/js/bundle.elasticVendor.min.js"></script><script defer="defer" src="/js/bundle.monacoVendor.min.js"></script><script defer="defer" src="/js/bundle.utilityVendor.min.js"></script><script defer="defer" src="/js/bundle.reactVendor.min.js"></script><script defer="defer" src="/js/bundle.199.min.js"></script><script defer="defer" src="/js/bundle.main.min.js"></script><link href="/main.4c986a61d4c17ea16a7c.css" rel="stylesheet"></head><body><div id="root"></div></body></html>[root@euler-200 tmp]# 

//实际node上没有监听端口
[root@euler-200 tmp]# ss -lntp |grep 31559

//外部浏览器无法访问
http://172.24.20.200:31559/

解决

aliyun-kubernes log

nodeport Port is not displayed

kube-proxy: remove port opener

//aliyun
Kubernetes 1.24及以后的版本，去除了kube-proxy监听NodePort的逻辑，在NodePort与内核net.ipv4.ip_local_port_range范围有冲突的情况下，可能会导致偶发的TCP无法连接的情况，导致健康检查失败、业务异常等问题。升级前，请确保集群没有NodePort端口与任意节点net.ipv4.ip_local_port_range范围存在冲突。

//github的回答显示
在新版本中已经删除了kube-proxy打开端口套接字部分，但是会在iptables中进行转发，故不影响访问.（注意：需要ip_local_port_range包含service开放的端口）

//当前系统 net.ipv4.ip_local_port_range
[root@euler-200 ~]# sysctl get net.ipv4.ip_local_port_range
sysctl: cannot stat /proc/sys/get: No such file or directory
net.ipv4.ip_local_port_range = 32768    60999

//serivce 
redis-stack        NodePort       10.233.43.183   <none>        6379:30619/TCP,8001:31559/TCP   52m    //8001:31559

没有覆盖，默认和实际的存在非包含关系,serive_nodeport < ip_local_port_range

//重置ip_local_port_range
[root@euler-200 ~]# sysctl  net.ipv4.ip_local_port_range 30000 60000

vi /etc/sysctl.conf 
net.ipv4.ip_local_port_range=30000 60000

sysctl -p && sysctl -a |grep ip_local_port_range

[root@euler-200 ~]# sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 30000    60000

//testing
[root@euler-200 ~]# ss -lntp |grep 31559
[root@euler-200 ~]# 

浏览器已经可以打开了

Alt text

reference