银河麒麟Kylinos安装kubernetes集群

kubernetes , ,

概览:

前期准备

节点

我准备部署3个节点:一主二从

操作系统: Kylin-Server-V10-SP3-General-Release-2303-X86_64

分配： master: 1, worker: 2

在管理节点hosts文件增加主机域名和ip的对应关系。

1
2
3
192.168.6.21 k8s-m1
192.168.6.31 k8s-w1
192.168.6.32 k8s-w2

初始化

1
2
yum update -y
yum install nano vim tar socat net-tools ethtool wget mlocate bash-completion tree -y

运维工具安装

（可选）这个只需要再管理节点安装即可

1
yum install ansible

编辑ansible配置文件,/etc/ansible/hosts,追加以下新内容

1
2
3
4
5
6
[master]
192.168.6.21 ansible_python_interpreter=/usr/bin/python3.7

[worker]
192.168.6.31 ansible_python_interpreter=/usr/bin/python3.7
192.168.6.32 ansible_python_interpreter=/usr/bin/python3.7

ssh免密码

1
2
ssh-keygen
cat /etc/ansible/hosts | grep "192.168" | awk '{print $1}' | xargs -I {} ssh-copy-id {} # 就是自动化执行ssh-copy-id 192.168.6.[17,21-23]

检查一下

1
2
3
4
5
6
7
8
9
[root@k8s-m1 ~]# ansible all -m shell -a "uname -a"
192.168.6.32 | CHANGED | rc=0 >>
Linux k8s-w2 4.19.90-52.29.v2207.ky10.x86_64 #1 SMP Mon Sep 25 17:11:21 CST 2023 x86_64 x86_64 x86_64 GNU/Linux

192.168.6.31 | CHANGED | rc=0 >>
Linux k8s-w1 4.19.90-52.29.v2207.ky10.x86_64 #1 SMP Mon Sep 25 17:11:21 CST 2023 x86_64 x86_64 x86_64 GNU/Linux

192.168.6.21 | CHANGED | rc=0 >>
Linux k8s-m1 4.19.90-52.29.v2207.ky10.x86_64 #1 SMP Mon Sep 25 17:11:21 CST 2023 x86_64 x86_64 x86_64 GNU/Linux

推送hosts

请检查出现类似本地回环的配置

1
ansible all -m copy -a "src=/etc/hosts dest=/etc/hosts"

sealos

我是用的是部署工具sealos来部署。所以先下载并安装，请安装官网最新版本。

1
2
wget https://github.com/labring/sealos/releases/download/v4.3.5/sealos_4.3.5_linux_amd64.rpm \
   && yum localinstall sealos_4.3.5_linux_amd64.rpm

防火墙配置

以下配置根据自己的需求来更改和参考

安装

1
ansible all -m shell -a "yum install iptables-services -y"

编辑规则

1
2
3
4
5
6
7
8
9
[root@k8s-w1 ~]# cat /etc/sysconfig/iptables
# sample configuration for iptables service
# you can edit this manually or use system-config-firewall
# please do not ask us to add additional ports/services to this default configuration
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

分发规则到剩余服务器

1
2
ansible all -m copy -a "src=/etc/sysconfig/iptables dest=/etc/sysconfig/iptables"
ansible all -m shell -a "systemctl enable iptables.service --now"

优化配置

1
ansible all -m shell -a "ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime"

内核更新

（可选）有些组件对内核版本有要求，可以根据实际情况选择是否需要升级内核版本

开始安装

建议使用官方内核，更新其他镜像源可能存在系统不稳定的情况。

  • 我遇到的是进不去系统，账户输入完后点击Enter，快速闪过一些文字后会就回到登录页面，循环往复，暂时没探究其原因。
1
sealos run labring/kubernetes:v1.25.7 --masters 192.168.6.21 --nodes 192.168.6.31,192.168.6.32 # 若没有配置ssh免密码 就追加 `-p <服务器集群密码>`

集群节点状态

1
2
3
4
5
[root@k8s-m1 ~]# kubectl get node -A
NAME     STATUS     ROLES           AGE   VERSION
k8s-m1   NotReady   control-plane   42s   v1.25.7
k8s-w1   NotReady   <none>          5s    v1.25.7
k8s-w2   NotReady   <none>          5s    v1.25.7

k8s状态

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@k8s-m1 ~]# kubectl get pods -A
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
kube-system   coredns-565d847f94-46tmm         0/1     Pending   0          46s
kube-system   coredns-565d847f94-clkq9         0/1     Pending   0          46s
kube-system   etcd-k8s-m1                      1/1     Running   0          61s
kube-system   kube-apiserver-k8s-m1            1/1     Running   0          61s
kube-system   kube-controller-manager-k8s-m1   1/1     Running   0          61s
kube-system   kube-proxy-8vtwd                 1/1     Running   0          27s
kube-system   kube-proxy-djbnc                 1/1     Running   0          27s
kube-system   kube-proxy-v9znt                 1/1     Running   0          46s
kube-system   kube-scheduler-k8s-m1            1/1     Running   0          60s
kube-system   kube-sealos-lvscare-k8s-w1       1/1     Running   0          17s
kube-system   kube-sealos-lvscare-k8s-w2       1/1     Running   0          20s

cilium安装

（可选）这里可以开始直接替换kube-proxy(参考下方章节)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@k8s-m1 ~]# cilium install
[root@k8s-m1 ~]# cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:          OK
 \__/¯¯\__/    Operator:        OK
 /¯¯\__/¯¯\    Hubble Relay:    disabled
 \__/¯¯\__/    ClusterMesh:     disabled
    \__/

Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet         cilium             Desired: 4, Ready: 4/4, Available: 4/4
Containers:       cilium             Running: 4
                  cilium-operator    Running: 1
Cluster Pods:     2/2 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.13.0: 4
                  cilium-operator    quay.io/cilium/operator:v1.13.0: 1

重命名节点role

1
2
kubectl label node k8s-w1 kubernetes.io/role=worker
kubectl label node k8s-w2 kubernetes.io/role=worker

修改成功

1
2
3
4
5
[root@k8s-m1 ~]# kubectl get node -A
NAME     STATUS     ROLES           AGE     VERSION
k8s-m1   NotReady   control-plane   4m32s   v1.25.7
k8s-w1   NotReady   worker          3m55s   v1.25.7
k8s-w2   NotReady   worker          3m55s   v1.25.7

helm安装

1
sealos run labring/helm:v3.12.3

检查

1
2
[root@k8s-m1 ~]# helm version
version.BuildInfo{Version:"v3.12.3", GitCommit:"3a31588ad33fe3b89af5a2a54ee1d25bfe6eaa5e", GitTreeState:"clean", GoVersion:"go1.20.7"}

kube-proxy平替(可选)

注意⚠️，请不要在没有充分调研的情况在生产环境实施该方案。

kube-proxy卸载

1
2
kubectl -n kube-system delete ds kube-proxy
kubectl -n kube-system delete cm kube-proxy

iptables规则处理

1
ansible all -m shell -a "iptables-save | grep -v KUBE | iptables-restore"

clium卸载重装

⚠️ 请注意此步骤为可选。

安装参数介绍Kubernetes Without kube-proxy

在标准安装的基础上添加了一些配置，可以自主进行取舍。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
cilium install --version 1.14.3  \
  --helm-set bpf.masquerade=true  \
  --helm-set bpf.clockProbe=true  \
  --helm-set bpf.waitForMount=true  \
  --helm-set bpf.preallocateMaps=true  \
  --helm-set bpf.hostRouting=false  \
  --helm-set k8sServiceHost=apiserver.cluster.local  \
  --helm-set k8sServicePort=6443  \
  --helm-set ipv4NativeRoutingCIDR=172.26.131.117/32  \
  --helm-set prometheus.enabled=true  \
  --helm-set operator.prometheus.enabled=true  \
  --helm-set kubeProxyReplacement=strict  \
  --helm-set hubble.relay.enabled=true  \
  --helm-set hubble.ui.enabled=true  \
  --helm-set hubble.enabled=true  \
  --helm-set hubble.metrics.enableOpenMetrics=true  \
  --helm-set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_app\,destination_app\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}"

检查

Run kubectl -n kube-system exec ds/cilium -- cilium status --verbose

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.25 (v1.25.7) [linux/amd64]
Kubernetes APIs:        ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Strict   [enp6s18 192.168.6.32 (Direct Routing), kube-ipvs0 10.96.0.1]
Host firewall:          Disabled
CNI Chaining:           none
Cilium:                 Ok   1.14.3 (v1.14.3-252a99ef)
NodeMonitor:            Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon:   Ok   
IPAM:                   IPv4: 4/254 allocated from 10.0.1.0/24, 
Allocated addresses:
  10.0.1.123 (kube-system/hubble-relay-767765f86f-n4bhp)
  10.0.1.176 (router)
  10.0.1.226 (health)
  10.0.1.234 (kube-system/hubble-ui-5f5599766f-6hnww)
IPv4 BIG TCP:           Disabled
IPv6 BIG TCP:           Disabled
BandwidthManager:       Disabled
Host Routing:           Legacy
Masquerading:           BPF   [enp6s18, kube-ipvs0]   172.26.131.117/32 [IPv4: Enabled, IPv6: Disabled]
Clock Source for BPF:   ktime
Controller Status:      29/29 healthy
  Name                                  Last success   Last error   Count   Message
  cilium-health-ep                      5s ago         never        0       no error   
  dns-garbage-collector-job             11s ago        never        0       no error   
  endpoint-1658-regeneration-recovery   never          never        0       no error   
  endpoint-221-regeneration-recovery    never          never        0       no error   
  endpoint-2703-regeneration-recovery   never          never        0       no error   
  endpoint-3248-regeneration-recovery   never          never        0       no error   
  endpoint-gc                           11s ago        never        0       no error   
  ipcache-inject-labels                 6s ago         5m9s ago     0       no error   
  k8s-heartbeat                         11s ago        never        0       no error   
  link-cache                            6s ago         never        0       no error   
  metricsmap-bpf-prom-sync              6s ago         never        0       no error   
  neighbor-table-refresh                6s ago         never        0       no error   
  resolve-identity-1658                 5s ago         never        0       no error   
  resolve-identity-221                  6s ago         never        0       no error   
  resolve-identity-2703                 4s ago         never        0       no error   
  resolve-identity-3248                 5s ago         never        0       no error   
  sync-host-ips                         6s ago         never        0       no error   
  sync-lb-maps-with-k8s-services        5m6s ago       never        0       no error   
  sync-policymap-1658                   5m2s ago       never        0       no error   
  sync-policymap-221                    5m2s ago       never        0       no error   
  sync-policymap-2703                   5m2s ago       never        0       no error   
  sync-policymap-3248                   5m2s ago       never        0       no error   
  sync-to-k8s-ciliumendpoint (1658)     4s ago         never        0       no error   
  sync-to-k8s-ciliumendpoint (221)      6s ago         never        0       no error   
  sync-to-k8s-ciliumendpoint (2703)     4s ago         never        0       no error   
  sync-to-k8s-ciliumendpoint (3248)     5s ago         never        0       no error   
  sync-utime                            6s ago         never        0       no error   
  template-dir-watcher                  never          never        0       no error   
  write-cni-file                        5m11s ago      never        0       no error   
Proxy Status:            OK, ip 10.0.1.176, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range:   min 256, max 65535
Hubble:                  Ok   Current/Max Flows: 976/4095 (23.83%), Flows/s: 3.07   Metrics: Ok
KubeProxyReplacement Details:
  Status:                 Strict
  Socket LB:              Enabled
  Socket LB Tracing:      Enabled
  Socket LB Coverage:     Full
  Devices:                enp6s18 192.168.6.32 (Direct Routing), kube-ipvs0 10.96.0.1
  Mode:                   SNAT
  Backend Selection:      Random
  Session Affinity:       Enabled
  Graceful Termination:   Enabled
  NAT46/64 Support:       Disabled
  XDP Acceleration:       Disabled
  Services:
  - ClusterIP:      Enabled
  - NodePort:       Enabled (Range: 30000-32767) 
  - LoadBalancer:   Enabled 
  - externalIPs:    Enabled 
  - HostPort:       Enabled
BPF Maps:   dynamic sizing: on (ratio: 0.002500)
  Name                          Size
  Auth                          524288
  Non-TCP connection tracking   67837
  TCP connection tracking       135674
  Endpoint policy               65535
  IP cache                      512000
  IPv4 masquerading agent       16384
  IPv6 masquerading agent       16384
  IPv4 fragmentation            8192
  IPv4 service                  65536
  IPv6 service                  65536
  IPv4 service backend          65536
  IPv6 service backend          65536
  IPv4 service reverse NAT      65536
  IPv6 service reverse NAT      65536
  Metrics                       1024
  NAT                           135674
  Neighbor table                135674
  Global policy                 16384
  Session affinity              65536
  Sock reverse NAT              67837
  Tunnel                        65536
Encryption:                       Disabled        
Cluster health:                   3/3 reachable   (2023-10-23T15:05:37Z)
  Name                            IP              Node        Endpoints
  kubernetes/k8s-w2 (localhost)   192.168.6.32    reachable   reachable
  kubernetes/k8s-m1               192.168.6.21    reachable   reachable
  kubernetes/k8s-w1               192.168.6.31    reachable   reachable

初始配置

默认存储

k8s存储支持多种模式：本地存储:hostPath/emptyDir，传递网络存储:iscsi/nfs，分布式网络存 储：glusterfs/rbd/cephfs，以及云存储等；
k8s默认容器如果重建，则容器中文件将丢失，为了解决这些问题，通常我们会将容器中需要持久化的文件存储到其他可持久化存储目录中。

安装主机依赖

我选择以nfs作为默认存储

1
sealos exec "yum install nfs-utils -y" # 集群节点都需要安装

配置共享(可选)

nfs服务主机 我主机有限：所以将nfs服务是由k8s-m1提供的，共享文件:在nfs服务主机文件/etc/exportes添加需要共享的目录以及允许的网段。

1
/data/share 192.168.6.0/24(insecure,rw,async,no_root_squash)

在服务器k8s-m1(跟随nfs磁盘)启动nfs服务

1
2
systemctl enable nfs-server
systemctl start nfs-server

添加默认存储

注：nfs镜像是拉取的上游(k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2)的镜像

1
2
3
4
5
6
7
8
9
helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  --set image.repository=ka1i137/sig-storage-nfs-subdir-external-provisioner \
  --set image.tag=v4.0.2 \
  --set storageClass.name=nfs-client \
  --set storageClass.defaultClass=true \
  --set nfs.server=192.168.6.11 \
  --set nfs.path=/mnt/private/k8s/share

验证

1
2
3
[root@k8s-m1 ~]# kubectl get sc -A
NAME                   PROVISIONER                                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default)   cluster.local/nfs-subdir-external-provisioner   Delete          Immediate           true                   20s

部署服务

创建命名空间

1
kubectl create namespace app

编写yaml

whoami.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  namespace: app
  labels:
    app: whoami

spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami
          ports:
            - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: app
spec:
  type: NodePort
  ports:
    - name: http-80
      port: 80
      nodePort: 30888
      targetPort: 80
  selector:
    app: whoami

开始部署

1
2
3
4
5
6
7
8
9
[root@k8s-m1 app]# kubectl apply -f whoami.yaml 
deployment.apps/whoami created
service/whoami created
[root@k8s-m1 app]# kubectl get pod -A | grep "^app"
app  whoami-6bbfdbb69c-tss9n  0/1  ContainerCreating  0  22s
app  whoami-6bbfdbb69c-v26d4  0/1  ContainerCreating  0  22s
[root@k8s-m1 app]# kubectl get svc -n app
NAME     TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
whoami   NodePort   10.96.1.5    <none>        80:30888/TCP   36s

验证部署

1
2
3
4
5
6
7
8
9
10
11
[root@k8s-m1 app]# curl 192.168.6.17:30888 # ip为k8s集群ip其中一个都可以
Hostname: whoami-6bbfdbb69c-v26d4
IP: 127.0.0.1
IP: ::1
IP: 10.0.2.67
IP: fe80::5019:c7ff:fe03:42e9
RemoteAddr: 10.0.0.220:45186
GET / HTTP/1.1
Host: 192.168.137.11:30888
User-Agent: curl/7.76.1
Accept: */*

基本监控

metrics-server

yaml配置下载

1
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

若默认镜像无法下载,可以使用我转存的ka1i137/k8s-metrics-server:v0.6.2。

若显示tls错误，可以在启动参数添加- --kubelet-insecure-tls

1
2
3
4
5
6
7
8
9
10
spec:
  containers:
  - args:
    - --cert-dir=/tmp
    - --secure-port=4443
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --kubelet-use-node-status-port
    - --metric-resolution=15s
    - --kubelet-insecure-tls # 新增
    image: ka1i137/k8s-metrics-server:v0.6.2

dubernetes-dashboard

1
sealos run labring/kubernetes-dashboard:v2.7.0

编辑

开放访问

dashboard-adminuser.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

然后应用

1
kubectl apply -f dashboard-adminuser.yaml

最后获取登录token

1
kubectl -n kubernetes-dashboard create token admin-user

Tips 开放NodePort

1
kubectl edit svc -n kubernetes-dashboard   kubernetes-dashboard

可以看到svc的配置修改为如下。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-03-13T09:42:15Z"
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
  resourceVersion: "20876"
  uid: f160d21f-2244-46d1-894c-d0cbd96383c2
spec:
  clusterIP: 10.96.3.50
  clusterIPs:
  - 10.96.3.50
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - nodePort: 30880 # 新增
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: NodePort  # 修改
status:
  loadBalancer: {}

k9s

官网：k9scli.io

Ingress

我使用的ingress是traefik，可以根据自己的具体需求选择其他的ingress。配置在另一篇文章。

Very Nice!

Mardan MDevOps Engineer

Hi there, I’m Mardan(ka1i).