目录:

概述(官网)

Kubernetes 是一个可移植、可扩展的开源平台，用于管理容器化的工作负载和服务，可促进声明式配置和自动化。 Kubernetes 拥有一个庞大且快速增长的生态，其服务、支持和工具的使用范围相当广泛。

Kubernetes 这个名字源于希腊语，意为“舵手”或“飞行员”。k8s 这个缩写是因为 k 和 s 之间有八个字符的关系。 Google 在 2014 年开源了 Kubernetes 项目。 Kubernetes 建立在 Google 大规模运行生产工作负载十几年经验的基础上，结合了社区中最优秀的想法和实践。

时光回溯

让我们回顾一下为何 Kubernetes 能够裨益四方。

部署演进

传统部署时代：

早期，各个组织是在物理服务器上运行应用程序。由于无法限制在物理服务器中运行的应用程序资源使用，因此会导致资源分配问题。例如，如果在同一台物理服务器上运行多个应用程序，则可能会出现一个应用程序占用大部分资源的情况，而导致其他应用程序的性能下降。一种解决方案是将每个应用程序都运行在不同的物理服务器上，但是当某个应用程式资源利用率不高时，剩余资源无法被分配给其他应用程式，而且维护许多物理服务器的成本很高。

虚拟化部署时代：

因此，虚拟化技术被引入了。虚拟化技术允许你在单个物理服务器的 CPU 上运行多台虚拟机（VM）。虚拟化能使应用程序在不同 VM 之间被彼此隔离，且能提供一定程度的安全性，因为一个应用程序的信息不能被另一应用程序随意访问。

虚拟化技术能够更好地利用物理服务器的资源，并且因为可轻松地添加或更新应用程序，而因此可以具有更高的可扩缩性，以及降低硬件成本等等的好处。通过虚拟化，你可以将一组物理资源呈现为可丢弃的虚拟机集群。

每个 VM 是一台完整的计算机，在虚拟化硬件之上运行所有组件，包括其自己的操作系统。

容器部署时代：

容器类似于 VM，但是更宽松的隔离特性，使容器之间可以共享操作系统（OS）。因此，容器比起 VM 被认为是更轻量级的。且与 VM 类似，每个容器都具有自己的文件系统、CPU、内存、进程空间等。由于它们与基础架构分离，因此可以跨云和 OS 发行版本进行移植。

容器因具有许多优势而变得流行起来，例如：

敏捷应用程序的创建和部署：与使用 VM 镜像相比，提高了容器镜像创建的简便性和效率。
持续开发、集成和部署：通过快速简单的回滚（由于镜像不可变性），提供可靠且频繁的容器镜像构建和部署。
关注开发与运维的分离：在构建、发布时创建应用程序容器镜像，而不是在部署时，从而将应用程序与基础架构分离。
可观察性：不仅可以显示 OS 级别的信息和指标，还可以显示应用程序的运行状况和其他指标信号。
跨开发、测试和生产的环境一致性：在笔记本计算机上也可以和在云中运行一样的应用程序。
跨云和操作系统发行版本的可移植性：可在 Ubuntu、RHEL、CoreOS、本地、 Google Kubernetes Engine 和其他任何地方运行。
以应用程序为中心的管理：提高抽象级别，从在虚拟硬件上运行 OS 到使用逻辑资源在 OS 上运行应用程序。
松散耦合、分布式、弹性、解放的微服务：应用程序被分解成较小的独立部分，并且可以动态部署和管理 - 而不是在一台大型单机上整体运行。
资源隔离：可预测的应用程序性能。
资源利用：高效率和高密度。

为什么需要 Kubernetes，它能做什么？

容器是打包和运行应用程序的好方式。在生产环境中，你需要管理运行着应用程序的容器，并确保服务不会下线。例如，如果一个容器发生故障，则你需要启动另一个容器。如果此行为交由给系统处理，是不是会更容易一些？

这就是 Kubernetes 要来做的事情！ Kubernetes 为你提供了一个可弹性运行分布式系统的框架。 Kubernetes 会满足你的扩展要求、故障转移你的应用、提供部署模式等。例如，Kubernetes 可以轻松管理系统的 Canary (金丝雀) 部署。

Kubernetes 为你提供：

服务发现和负载均衡

Kubernetes 可以使用 DNS 名称或自己的 IP 地址来暴露容器。如果进入容器的流量很大， Kubernetes 可以负载均衡并分配网络流量，从而使部署稳定。
存储编排

Kubernetes 允许你自动挂载你选择的存储系统，例如本地存储、公共云提供商等。
自动部署和回滚

你可以使用 Kubernetes 描述已部署容器的所需状态，它可以以受控的速率将实际状态更改为期望状态。例如，你可以自动化 Kubernetes 来为你的部署创建新容器，删除现有容器并将它们的所有资源用于新容器。
自动完成装箱计算

你为 Kubernetes 提供许多节点组成的集群，在这个集群上运行容器化的任务。你告诉 Kubernetes 每个容器需要多少 CPU 和内存 (RAM)。 Kubernetes 可以将这些容器按实际情况调度到你的节点上，以最佳方式利用你的资源。
自我修复

Kubernetes 将重新启动失败的容器、替换容器、杀死不响应用户定义的运行状况检查的容器，并且在准备好服务之前不将其通告给客户端。
密钥与配置管理

Kubernetes 允许你存储和管理敏感信息，例如密码、OAuth 令牌和 ssh 密钥。你可以在不重建容器镜像的情况下部署和更新密钥和应用程序配置，也无需在堆栈配置中暴露密钥。

Kubernetes 不是什么

Kubernetes 不是传统的、包罗万象的 PaaS（平台即服务）系统。由于 Kubernetes 是在容器级别运行，而非在硬件级别，它提供了 PaaS 产品共有的一些普遍适用的功能，例如部署、扩展、负载均衡，允许用户集成他们的日志记录、监控和警报方案。但是，Kubernetes 不是单体式（monolithic）系统，那些默认解决方案都是可选、可插拔的。 Kubernetes 为构建开发人员平台提供了基础，但是在重要的地方保留了用户选择权，能有更高的灵活性。

Kubernetes：

不限制支持的应用程序类型。 Kubernetes 旨在支持极其多种多样的工作负载，包括无状态、有状态和数据处理工作负载。如果应用程序可以在容器中运行，那么它应该可以在 Kubernetes 上很好地运行。
不部署源代码，也不构建你的应用程序。持续集成（CI）、交付和部署（CI/CD）工作流取决于组织的文化和偏好以及技术要求。
不提供应用程序级别的服务作为内置服务，例如中间件（例如消息中间件）、数据处理框架（例如 Spark）、数据库（例如 MySQL）、缓存、集群存储系统（例如 Ceph）。这样的组件可以在 Kubernetes 上运行，并且/或者可以由运行在 Kubernetes 上的应用程序通过可移植机制（例如开放服务代理）来访问。
不是日志记录、监视或警报的解决方案。它集成了一些功能作为概念证明，并提供了收集和导出指标的机制。
不提供也不要求配置用的语言、系统（例如 jsonnet），它提供了声明性 API，该声明性 API 可以由任意形式的声明性规范所构成。
不提供也不采用任何全面的机器配置、维护、管理或自我修复系统。
此外，Kubernetes 不仅仅是一个编排系统，实际上它消除了编排的需要。编排的技术定义是执行已定义的工作流程：首先执行 A，然后执行 B，再执行 C。而 Kubernetes 包含了一组独立可组合的控制过程，可以连续地将当前状态驱动到所提供的预期状态。你不需要在乎如何从 A 移动到 C，也不需要集中控制，这使得系统更易于使用且功能更强大、系统更健壮，更为弹性和可扩展。

安装K8s

Kubeadm安装

k8s高可用

1.基本环境配置(所有节点)

配置: 2U2G128G CentOS7.9
节点:
- 192.168.8.11 k8s-master01
- 192.168.8.12 k8s-master02
- 192.168.8.13 k8s-master03
- 192.168.8.14 k8s-node01
- 192.168.8.15 k8s-node02
- 192.168.8.88 k8s-master-lb (不占用机器)

# A.所有节点关闭防火墙\selinux\dnsmasq\swap
systemctl disable --now firewalld
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager 

## 关闭selinux
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
## 关闭swap
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

# B.同步时间
yum -y install ntp 
ntpdate ntp1.aliyun.com

crontab -e 

*/5 * * * *  ntpdate  ntp1.aliyun.com

systemctl enable crond
systemctl restart crond
## 开机自动同步
vim /etc/rc.local 
ntpdate  ntp1.aliyun.com
# C.配置limit
ulimit -SHn 65535
vim /etc/security/limits.conf
# 末尾添加如下内容
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited

# D.Master01节点免密钥登陆其他节点
ssh-keygen -t rsa

# E.配置yum源
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

cat <<EOF > /etc/yum.repos.d/kubernetes.repo 
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enable=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF


# F.升级内核

yum -y install  psmisc telnet yum-utils device-mapper-persistent-data lvm2  ipvsadm ipset sysstat conntrack libseccomp

yum update -y --exclude=kernel*

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org　　#导入该源的秘钥

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-6.el7.elrepo.noarch.rpm　　#启用该源仓库

yum --disablerepo="*" --enablerepo="elrepo-kernel" list available　　#查看有哪些内核版本可供安装

yum --enablerepo=elrepo-kernel install kernel-lt kernel-lt-devel -y #安装稳定版本内核

# 打开并编辑 /etc/default/grub 并设置 GRUB_DEFAULT=0。意思是 GRUB 初始化页面的第一个内核将作为默认内核

grub2-mkconfig -o /boot/grub2/grub.cfg

# G.配置ipvs模块
cat << EOF > /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF
# 然后执行systemctl enable --now systemd-modules-load.service

# H.开启k8s集群中必须的内核参数
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
sysctl --system

# 重启
reboot

2.基本组建安装(所有节点)

## 安装启动docker\kubelet\kubeadm
yum install -y docker-ce-19.03.*  kubectl-1.20.0 kubelet-1.20.0 kubeadm-1.20.0  --disableexcludes=kubernetes

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://7sf2t0xw.mirror.aliyuncs.com"],
  "exec-opts":["native.cgroupdriver=systemd"]
}
EOF

mkdir /etc/systemd/system/kubelet.service.d
cd /etc/systemd/system/kubelet.service.d
vim 10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

systemctl daemon-reload
systemctl start docker && systemctl enable docker
systemctl enable --now kubelet

# 默认配置的pause镜像使用gcr.io仓库，国内可能无法访问，所以这里配置Kubelet使用阿里云的pause镜像
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
EOF

3.高可用组建安装

# 所有master节点安装HAProxy和KeepAlived
yum install keepalived haproxy -y
# 所有master节点配置HAProxy
vim /etc/haproxy/haproxy.cfg

global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01   192.168.8.11:6443  check
  server k8s-master02   192.168.8.12:6443  check
  server k8s-master03   192.168.8.13:6443  check

# 所有Master节点配置KeepAlived，配置不一样，注意区分
# 注意每个节点的IP和网卡（interface参数)
# Master01节点的配置
vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
    script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2
    rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33            # 网卡地址
    mcast_src_ip 192.168.8.11  # 当前节点的ip
    virtual_router_id 51        # 同一集群id相同，不同集群不能重复
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.8.88
    }
#    track_script {
#       chk_apiserver
#    }
}


# 健康检查脚本
vim  /etc/keepalived/check-apiserver.sh
#!/bin/bash

err=0
for k in $(seq 1 2)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi

chmod +x /etc/keepalived/check-apiserver.sh

# 启动haproxy、keepalived
systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived

4.集群初始化

# 使用kubeadm初始化集群
# master01节点创建kubeadm-config.yaml配置文件如下：
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 7t2weq.bjbawausm0jaxury
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.8.11
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master01
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  certSANs:
  - 192.168.8.88
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.8.88:16443
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.20.0
networking:
  dnsDomain: cluster.local
  podSubnet: 172.168.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}


# 更新kubeadm文件
kubeadm config migrate --old-config kubeadm-config.yaml  --new-config new.yaml
# 将new.yaml文件复制到其他master节点
scp new.yaml root@k8s-master02:/root/
scp new.yaml root@k8s-master03:/root/
# 之后所有Master节点提前下载镜像，可以节省初始化时间：
kubeadm config images pull --config /root/new.yaml
# Master01节点初始化，初始化以后会在/etc/kubernetes目录下生成对应的证书和配置文件，之后其他Master节点加入Master01即可
kubeadm init --config /root/new.yaml  --upload-certs

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.8.88:16443 --token 7t2weq.bjbawausm0jaxury \
    --discovery-token-ca-cert-hash sha256:04e52ac8e79522e780cadb8a7ae65471e52fafe19e71aab3a00cf64ef95ac037 \
    --control-plane --certificate-key 687c4a8c2ac5f242c21cf3e43d6496478ea8f07504a1d62423f0fe5d9e2815d0

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.8.88:16443 --token 7t2weq.bjbawausm0jaxury \
    --discovery-token-ca-cert-hash sha256:04e52ac8e79522e780cadb8a7ae65471e52fafe19e71aab3a00cf64ef95ac037

# 查看组件启动情况
kubectl get po -n kube-system

# Master01节点配置环境变量，用于访问Kubernetes集群：
cat <<EOF >> /root/.bashrc
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF
source /root/.bashrc

# 查看节点
[root@master01 ~]# kubectl get nodes
NAME       STATUS     ROLES                  AGE   VERSION
master01   NotReady   control-plane,master   13m   v1.20.0
# 查看启动服务
[root@master01 ~]# kubectl get po -n kube-system
NAME                               READY   STATUS    RESTARTS   AGE
coredns-54d67798b7-8vt5k           0/1     Pending   0          13m
coredns-54d67798b7-pf4q8           0/1     Pending   0          13m
etcd-master01                      1/1     Running   0          14m
kube-apiserver-master01            1/1     Running   0          14m
kube-controller-manager-master01   1/1     Running   0          14m
kube-proxy-86dls                   1/1     Running   0          13m
kube-scheduler-master01            1/1     Running   0          14m
# 所有节启动后
[root@master01 ~]# kubectl get nodes
NAME       STATUS     ROLES                  AGE     VERSION
master01   NotReady   control-plane,master   19m     v1.20.0
master02   NotReady   control-plane,master   4m48s   v1.20.0
node01     NotReady   <none>                 3m30s   v1.20.0
node02     NotReady   <none>                 3m28s   v1.20.0

5.Calico安装

# 以下步骤只在master01执行
## 将calico保存到本地
curl https://docs.projectcalico.org/archive/v3.21/manifests/calico.yaml -O
# 修改CIDR
vim calico.yaml
- name: CALICO_IPV4POOL_CIDR
  value: "172.168.0.0/16"

kubectl apply -f calico.yaml

6.Metrics部署

Metrics 用来查看集群节点中内存、磁盘、CPU和网络使用率

# 安装Metrice
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability.yaml

7.Dashboard部署

Web显示面板

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

kubectl proxy
# 更改dashboard的svc为NodePort
kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard
# 将ClusterIP更改为NodePort

# 查看端口
kubectl get svc -n kubernetes-dashboard
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.100.159.143   <none>        8000/TCP        18m
kubernetes-dashboard        NodePort    10.101.190.108   <none>        443:31084/TCP   19m
浏览器打卡 https://192.168.8.88:31084 选择Token模式登录
# 生成token
admin_account="k8s-admin"
kubectl create serviceaccount ${admin_account} -n kube-system

kubectl create clusterrolebinding ${admin_account} --clusterrole=cluster-admin --serviceaccount=kube-system:${admin_account}

# 获取Token
kubectl -n kube-system describe secrets $(kubectl -n kube-system get secret | grep ${admin_account} | awk '{print $1}')

# 其他Token
kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes- 
dashboard get secret | grep dashboard-admin | awk '{print $1}')

二进制安装

基本环境配置

主机信息，服务器IP地址不能设置为dhcp，要配置静态IP。

VIP（虚拟IP）不要和公司内网IP重复，首先去ping一下，不通才可用。VIP需要和主机在同一个局域网内！

192.168.8.200 k8s-master-lb # VIP虚IP不占用机器资源 # 如果不是高可用集群，该IP为Master01的IP
192.168.3.201 k8s-master01 # 2C2G 20G
192.168.3.202 k8s-master02 # 2C2G 20G
192.168.3.203 k8s-master03 # 2C2G 20G
192.168.3.204 k8s-node01 # 2C2G 20G
192.168.3.205 k8s-node02 # 2C2G 20G

K8s Service网段：10.96.0.0/12

K8s Pod网段：172.16.0.0/12

系统环境：

[root@master01 ~]#  cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
[root@master01 ~]#

虚拟机环境：

配置所有节点hosts文件

[root@master01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.3.201 k8s-master01
192.168.3.202 k8s-master02
192.168.3.203 k8s-master03
192.168.8.12 k8s-master-lb
192.168.3.204 k8s-node01
192.168.3.205 k8s-node02
[root@master01 ~]#

CentOS7安装yum源如下：

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

必备工具安装

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git -y

所有节点关闭firewalld、dnsmasq、selinux

systemctl disable --now firewalld
systemctl disable --now dnsmasq

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

所有节点关闭swap分区，fstab注释swap

swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

所有节点同步时间

安装ntpdate

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install ntpdate -y

所有节点同步时间。时间同步配置如下：

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' > /etc/timezone
ntpdate time2.aliyun.com
# 加入到crontab
crontab -e
*/5 * * * * ntpdate time2.aliyun.com

所有节点配置limit：

ulimit -SHn 65535

vim /etc/security/limits.conf
#末尾添加如下内容
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited

Master01节点免密钥登录其他节点，安装过程中生成配置文件和证书均在Master01上操作，集群管理也在Master01上操作，阿里云或者AWS上需要单独一台kubectl服务器。密钥配置如下：

[root@k8s-master01 ~]# ssh-keygen -t rsa

Master01配置免密码登录其他节点

for i in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02;do ssh-copy-id -i .ssh/id_rsa.pub $i;done

Master01下载安装文件

git config --global --unset http.proxy
git config --global --unset https.proxy
git clone https://github.com/dotbalo/k8s-ha-install.git

所有节点升级系统并重启，此处升级没有升级内核，升级内核可参考之前的内容一套教程搞定k8s安装到实战。

yum update -y --exclude=kernel* --skip-broken && reboot #CentOS需要升级，CentOS8可以按需升级系统

基本组件安装

本节主要安装的是集群中用到各种组件，比如Docker-ce、Kubernetes各组件等。

Docker安装

所有节点安装Docker-ce 19.03

yum install docker-ce-19.03.* -y

温馨提示：

由于新版kubelet建议使用systemd，所以可以把docker的CgroupDriver改成systemd

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://7sf2t0xw.mirror.aliyuncs.com"],
  "exec-opts":["native.cgroupdriver=systemd"]
}
EOF

所有节点设置开机自启动Docker：

sudo systemctl daemon-reload
sudo systemctl restart docker
systemctl enable --now docker

K8s及etcd安装

Master01下载kubernetes安装包

wget https://dl.k8s.io/v1.20.0/kubernetes-server-linux-amd64.tar.gz

下载etcd安装包

wget https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz

解压kubernetes安装文件

tar -xf kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

解压etcd安装文件

tar -zxvf etcd-v3.4.13-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.4.13-linux-amd64/etcd{,ctl}

版本查看

[root@k8s-master01 ~]# kubelet --version
Kubernetes v1.20.0
[root@k8s-master01 ~]# 
[root@k8s-master01 ~]# etcdctl version
etcdctl version: 3.4.13
API version: 3.4
[root@k8s-master01 ~]#

将组件发送到其他节点

MasterNodes='k8s-master02 k8s-master03'
WorkNodes='k8s-node01 k8s-node02'
for NODE in $MasterNodes;do echo $NODE;scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/;scp /usr/local/bin/etcd* $NODE:/usr/local/bin;done
for NODE in $WorkNodes;do scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin;done

所有节点创建/opt/cni/bin目录

mkdir -p /opt/cni/bin

切换分支

git branch -a
git checkout manual-installation-v1.20.x

生成证书

二进制安装最关键步骤，一步错误全盘皆输，一定要注意每个步骤都要是正确的

Master01下载生成证书工具

wget "https://pkg.cfssl.org/R1.2/cfssl_linux-amd64" -O /usr/local/bin/cfssl
wget "https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

所有Master节点创建etcd证书目录

mkdir /etc/etcd/ssl -p

所有节点创建kubernetes相关目录

mkdir -p /etc/kubernetes/pki

Master01节点生成etcd证书

生成证书的CSR文件：证书签名请求文件，配置了一些域名、公司、单位

cd /root/k8s-ha-install/pki

# 生成CA证书和CA证书的key
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca

# 曾在这里因为hostname问题导致etcd服务无法启动，后删除无用的ip地址，并修改节点域名为IP地址解决
cfssl gencert -ca=/etc/etcd/ssl/etcd-ca.pem -ca-key=/etc/etcd/ssl/etcd-ca-key.pem -config=ca-config.json -hostname=127.0.0.1,192.168.3.201,192.168.3.202,192.168.3.203 -profile=kubernetes etcd-csr.json|cfssljson -bare /etc/etcd/ssl/etcd

将证书复制到其他节点

MasterNodes='k8s-master02 k8s-master03'
WorkNodes='k8s-node01 k8s-node02'
for NODE in $MasterNodes;do
ssh $NODE "mkdir -p /etc/etcd/ssl"
for FILE in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem;do
scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}
done
done

Master01生成kubernetes证书

cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca

# 10.96.0是k8s service的网段，如果说需要更改k8s service网段，那就需要更改10.96.0.1
cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -hostname=10.96.0.1,192.168.8.12,127.0.0.1,kubernetes,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,192.168.3.201,192.168.3.202,192.168.3.203 -profile=kubernetes apiserver-csr.json|cfssljson -bare /etc/kubernetes/pki/apiserver

cfssl gencert -initca front-proxy-ca-csr.json|cfssljson -bare /etc/kubernetes/pki/front-proxy-ca

# 生成apiserver的聚合证书。Requestheader-client-xxx request-allowed-xxx:aggregator
cfssl gencert -ca=/etc/kubernetes/pki/front-proxy-ca.pem -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json|cfssljson -bare /etc/kubernetes/pki/front-proxy-client

# 生成controller-manage的证书
cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -profile=kubernetes manager-csr.json|cfssljson -bare /etc/kubernetes/pki/controller-manager

# set-cluster：设置一个集群项
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.8.12:8443 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 设置一个环境项，一个上下文
kubectl config set-context system:kube-controller-manager@kubernetes --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# set-credentials 设置一个用户项
kubectl config set-credentials system:kube-controller-manager --client-certificate=/etc/kubernetes/pki/controller-manager.pem --client-key=/etc/kubernetes/pki/controller-manager-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

#使用某个环境当做默认环境
kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -profile=kubernetes scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.8.12:8443 --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler --client-certificate=/etc/kubernetes/pki/scheduler.pem --client-key=/etc/kubernetes/pki/scheduler-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config set-context system:kube-scheduler@kubernetes --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config use-context system:kube-scheduler@kubernetes --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.8.12:8443 --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config set-credentials kubernetes-admin --client-certificate=/etc/kubernetes/pki/admin.pem --client-key=/etc/kubernetes/pki/admin-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config set-context kubernetes-admin@kubernetes --cluster=kubernetes --user=kubernetes-admin --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=/etc/kubernetes/admin.kubeconfig

创建ServiceAccount Key

openssl genrsa -out /etc/kubernetes/pki/sa.key 2048

openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

for NODE in k8s-master02 k8s-master03;do
for FILE in $(ls /etc/kubernetes/pki | grep -v etcd);do
scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE};
done;
for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig;do
scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE};
done;
done
# 确认生成的文件有23个
[root@k8s-master01 pki]# ls /etc/kubernetes/pki | wc -l
23
[root@k8s-master01 pki]#

Kubernetes系统组件配置

etcd配置

etcd配置大致相同，注意修改每个Master节点的etcd配置的主机名和IP地址

Master01

vim /etc/etcd/etcd.config.yml
name: 'k8s-master01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.3.201:2380'
listen-client-urls: 'https://192.168.3.201:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.3.201:2380'
advertise-client-urls: 'https://192.168.3.201:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.3.201:2380,k8s-master02=https://192.168.3.202:2380,k8s-master03=https://192.168.3.203:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false

Master02

vim /etc/etcd/etcd.config.yml
name: 'k8s-master02'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.3.202:2380'
listen-client-urls: 'https://192.168.3.202:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.3.202:2380'
advertise-client-urls: 'https://192.168.3.202:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.3.201:2380,k8s-master02=https://192.168.3.202:2380,k8s-master03=https://192.168.3.203:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false

Master03

vim /etc/etcd/etcd.config.yml
name: 'k8s-master03'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.3.203:2380'
listen-client-urls: 'https://192.168.3.203:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.3.203:2380'
advertise-client-urls: 'https://192.168.3.203:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.3.201:2380,k8s-master02=https://192.168.3.202:2380,k8s-master03=https://192.168.3.203:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false

创建Service

所有Master节点创建etcd service并启动

vi /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service

所有Master节点创建etcd的证书目录

mkdir /etc/kubernetes/pki/etcd
ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd
systemctl daemon-reload
systemctl enable --now etcd

查看etcd状态

[root@k8s-master01 etcd]# export ETCDCTL_API=3
[root@k8s-master01 pki]# etcdctl --endpoints="192.168.3.203:2379,192.168.3.202:2379,192.168.3.201:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem endpoint status --write-out=table
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.3.203:2379 | 40ebe70dffb00505 |  3.4.13 |   20 kB |     false |      false |        14 |          9 |                  9 |        |
| 192.168.3.202:2379 | 4087302bf9046cf1 |  3.4.13 |   20 kB |     false |      false |        14 |          9 |                  9 |        |
| 192.168.3.201:2379 | 8af8195104313994 |  3.4.13 |   20 kB |      true |      false |        14 |          9 |                  9 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@k8s-master01 pki]#

高可用配置

高可用配置（注意：如果不是高可用集群，haproxy和keepalived无需安装）

如果在云上安装也无需执行此章节的步骤，可以直接使用云上的lb，比如阿里云slb，腾讯云elb等

所有Master节点安装keepalived和haproxy

yum install keepalived haproxy -y

所有Master配置HAProxy，配置一样

vi /etc/haproxy/haproxy.cfg
global
  maxconn 2000
  ulimit-n 16384
  log 127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode http
  option httplog
  timeout connect 5000
  timeout client 50000
  timeout server 50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend k8s-master
  bind 0.0.0.0:8443
  bind 127.0.0.1:8443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01 192.168.3.201:6443 check
  server k8s-master02 192.168.3.202:6443 check
  server k8s-master03 192.168.3.203:6443 check

Master01 keepalived

所有Master节点配置KeepAlived，配置不一样，注意区分[root@k8s-master01 pki]# vim /etc/keepalived/keepalived.conf，注意每个节点的IP和网卡（interface参数）

!Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2
    rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 192.168.3.201
    virtual_router_id 51
    priority 101
    nopreempt
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.8.12
    }
    track_script {
        chk_apiserver
    }
}

Master02 keepalived

!Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2
    rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    mcast_src_ip 192.168.3.202
    virtual_router_id 51
    priority 100
    nopreempt
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.8.12
    }
    track_script {
        chk_apiserver
    }
}

Master03 keepalived

!Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2
    rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    mcast_src_ip 192.168.3.203
    virtual_router_id 51
    priority 100
    nopreempt
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.8.12
    }
    track_script {
        chk_apiserver
    }
}

健康检查配置

所有master节点

vi /etc/keepalived/check_apiserver.sh
#!/bin/bash

err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err +1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi
chmod +x /etc/keepalived/check_apiserver.sh

所有master节点启动haproxy和keepalived

systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived

VIP测试

[root@k8s-master01 pki]# ping 192.168.8.12
PING 192.168.8.12 (192.168.8.12) 56(84) bytes of data.
64 bytes from 192.168.8.12: icmp_seq=1 ttl=64 time=0.033 ms
64 bytes from 192.168.8.12: icmp_seq=2 ttl=64 time=0.081 ms
64 bytes from 192.168.8.12: icmp_seq=3 ttl=64 time=0.077 ms
^C
--- 192.168.8.12 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2042ms
rtt min/avg/max/mdev = 0.033/0.063/0.081/0.023 ms
[root@k8s-master01 pki]#

重要：如果安装了keepalived和haproxy，需要测试keepalived是否是正常的

[root@k8s-master01 pki]# telnet 192.168.8.12 8443
Trying 192.168.8.12...
Connected to 192.168.8.12.
Escape character is '^]'.
Connection closed by foreign host.
[root@k8s-master01 pki]#

如果ping不通且telnet没有出现']'，则认为VIP不可以，不可再继续往下执行，需要排查keepalived的问题，比如防火墙和selinux，haproxy和keepalived的状态，监听端口灯

所有节点查看防火墙状态必须为disable和inactive：systemctl status firewalld

所有节点查看selinux状态，必须为disable：getenforce

master节点查看haproxy和keepalived状态：systemctl status keepalived haproxy

Kubernetes系统组件配置（续）

Apiserver

所有节点创建相关目录

mkdir -p /etc/kubernetes/manifest/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes

所有Master节点创建kube-apiserver service，# 注意，如果不是高可用集群，192.168.8.200改为master01的地址

Master01配置

注意本文档使用的k8s service网段为10.96.0.0/12，该网段不能和宿主机的网段、Pod网段重复，请按需修改

vi /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2 \
      --logtostderr=true \
      --allow-privileged=true \
      --bind-address=0.0.0.0 \
      --secure-port=6443 \
      --insecure-port=0 \
      --advertise-address=192.168.3.201 \
      --service-cluster-ip-range=10.96.0.0/12 \
      --service-node-port-range=30000-32767 \
      --etcd-servers=https://192.168.3.201:2379,https://192.168.3.202:2379,https://192.168.3.203:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
      --client-ca-file=/etc/kubernetes/pki/ca.pem \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
      --authorization-mode=Node,RBAC \
      --enable-bootstrap-token-auth=true \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
      --requestheader-allowed-names=aggregator \
      --requestheader-extra-headers-prefix=X-Remote-Group \
      --requestheader-username-headers=X-Remote-User
      # --token-auth-file=/etc/kubernetes/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Master02配置

vi /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2 \
      --logtostderr=true \
      --allow-privileged=true \
      --bind-address=0.0.0.0 \
      --secure-port=6443 \
      --insecure-port=0 \
      --advertise-address=192.168.3.202 \
      --service-cluster-ip-range=10.96.0.0/12 \
      --service-node-port-range=30000-32767 \
      --etcd-servers=https://192.168.3.201:2379,https://192.168.3.202:2379,https://192.168.3.203:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
      --client-ca-file=/etc/kubernetes/pki/ca.pem \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
      --authorization-mode=Node,RBAC \
      --enable-bootstrap-token-auth=true \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
      --requestheader-allowed-names=aggregator \
      --requestheader-group-headers=X-Remote-Group \
      --requestheader-extra-headers-prefix=X-Remote-Extra- \
      --requestheader-username-headers=X-Remote-User
      # --token-auth-file=/etc/kubernetes/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Master03配置

vi /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2 \
      --logtostderr=true \
      --allow-privileged=true \
      --bind-address=0.0.0.0 \
      --secure-port=6443 \
      --insecure-port=0 \
      --advertise-address=192.168.3.203 \
      --service-cluster-ip-range=10.96.0.0/12 \
      --service-node-port-range=30000-32767 \
      --etcd-servers=https://192.168.3.201:2379,https://192.168.3.202:2379,https://192.168.3.203:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
      --client-ca-file=/etc/kubernetes/pki/ca.pem \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
      --authorization-mode=Node,RBAC \
      --enable-bootstrap-token-auth=true \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
      --requestheader-allowed-names=aggregator \
      --requestheader-group-headers=X-Remote-Group \
      --requestheader-extra-headers-prefix=X-Remote-Extra- \
      --requestheader-username-headers=X-Remote-User
      # --token-auth-file=/etc/kubernetes/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

启动apiserver

所有Master节点开启kube-apiserver

systemctl daemon-reload && systemctl enable --now kube-apiserver

检测kube-server状态

[root@k8s-master01 pki]# systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-06-21 21:15:19 CST; 11s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 4604 (kube-apiserver)
    Tasks: 8
   Memory: 276.4M
   CGroup: /system.slice/kube-apiserver.service
           └─4604 /usr/local/bin/kube-apiserver --v=2 --logtostderr=true --allow-privileged=true --bind-address=0.0.0.0 --secure-port=6443 --insecure-port=0 --advertise-address=192.168.3.201 --serv...

Jun 21 21:15:25 k8s-master-lb kube-apiserver[4604]: [-]poststarthook/rbac/bootstrap-roles failed: not finished
Jun 21 21:15:25 k8s-master-lb kube-apiserver[4604]: I0621 21:15:25.359623    4604 healthz.go:244] poststarthook/rbac/bootstrap-roles check failed: readyz
Jun 21 21:15:25 k8s-master-lb kube-apiserver[4604]: [-]poststarthook/rbac/bootstrap-roles failed: not finished
Jun 21 21:15:25 k8s-master-lb kube-apiserver[4604]: I0621 21:15:25.457019    4604 healthz.go:244] poststarthook/rbac/bootstrap-roles check failed: readyz
Jun 21 21:15:25 k8s-master-lb kube-apiserver[4604]: [-]poststarthook/rbac/bootstrap-roles failed: not finished
Jun 21 21:15:25 k8s-master-lb kube-apiserver[4604]: I0621 21:15:25.557168    4604 healthz.go:244] poststarthook/rbac/bootstrap-roles check failed: readyz
Jun 21 21:15:25 k8s-master-lb kube-apiserver[4604]: [-]poststarthook/rbac/bootstrap-roles failed: not finished
Jun 21 21:15:25 k8s-master-lb kube-apiserver[4604]: W0621 21:15:25.714977    4604 lease.go:233] Resetting endpoints for master service "kubernetes" to [192.168.3.201 192.168.3.202 192.168.3.203]
Jun 21 21:15:25 k8s-master-lb kube-apiserver[4604]: I0621 21:15:25.718182    4604 controller.go:606] quota admission added evaluator for: endpoints
Jun 21 21:15:25 k8s-master-lb kube-apiserver[4604]: I0621 21:15:25.726940    4604 controller.go:606] quota admission added evaluator for: endpointslices.discovery.k8s.io
[root@k8s-master01 pki]#

ControllerManager

所有Master节点配置kube-controller-manager service

vi /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
      --v=2 \
      --logtostderr=true \
      --address=127.0.0.1 \
      --root-ca-file=/etc/kubernetes/pki/ca.pem \
      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
      --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
      --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
      --leader-elect=true \
      --use-service-account-credentials=true \
      --node-monitor-grace-period=40s \
      --node-monitor-period=5s \
      --pod-eviction-timeout=2m0s \
      --controllers=*,bootstrapsigner,tokencleaner \
      --allocate-node-cidrs=true \
      --cluster-cidr=172.16.0.0/12 \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
      --node-cidr-mask-size=24

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

所有Master节点启动kube-controller-manager

systemctl daemon-reload
systemctl enable --now kube-controller-manager

查看启动状态

[root@k8s-master01 pki]# systemctl status kube-controller-manager
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-06-21 21:16:16 CST; 11s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 4721 (kube-controller)
    Tasks: 5
   Memory: 27.4M
   CGroup: /system.slice/kube-controller-manager.service
           └─4721 /usr/local/bin/kube-controller-manager --v=2 --logtostderr=true --address=127.0.0.1 --root-ca-file=/etc/kubernetes/pki/ca.pem --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pe...

Jun 21 21:16:20 k8s-master-lb kube-controller-manager[4721]: I0621 21:16:20.705608    4721 shared_informer.go:240] Waiting for caches to sync for ClusterRoleAggregator
Jun 21 21:16:20 k8s-master-lb kube-controller-manager[4721]: I0621 21:16:20.855590    4721 controllermanager.go:554] Started "pv-protection"
Jun 21 21:16:20 k8s-master-lb kube-controller-manager[4721]: I0621 21:16:20.855687    4721 controllermanager.go:539] Starting "replicaset"
Jun 21 21:16:20 k8s-master-lb kube-controller-manager[4721]: I0621 21:16:20.855787    4721 pv_protection_controller.go:83] Starting PV protection controller
Jun 21 21:16:20 k8s-master-lb kube-controller-manager[4721]: I0621 21:16:20.855807    4721 shared_informer.go:240] Waiting for caches to sync for PV protection
Jun 21 21:16:21 k8s-master-lb kube-controller-manager[4721]: I0621 21:16:21.015750    4721 controllermanager.go:554] Started "replicaset"
Jun 21 21:16:21 k8s-master-lb kube-controller-manager[4721]: I0621 21:16:21.015803    4721 controllermanager.go:539] Starting "nodeipam"
Jun 21 21:16:21 k8s-master-lb kube-controller-manager[4721]: I0621 21:16:21.015894    4721 replica_set.go:182] Starting replicaset controller
Jun 21 21:16:21 k8s-master-lb kube-controller-manager[4721]: I0621 21:16:21.015908    4721 shared_informer.go:240] Waiting for caches to sync for ReplicaSet
Jun 21 21:16:21 k8s-master-lb kube-controller-manager[4721]: I0621 21:16:21.055523    4721 node_ipam_controller.go:91] Sending events to api server.
[root@k8s-master01 pki]#

Scheduler

所有Master节点配置kube-scheduler service

vi /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
      --v=2 \
      --logtostderr=true \
      --address=127.0.0.1 \
      --leader-elect=true \
      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now kube-scheduler

TLS Bootstrapping配置

在Master01创建bootstrap

#注意，如果不是高可用集群，192.168.8.200:8443改为master01的地址，8443改为apiserver的端口，默认是6443

cd /root/k8s-ha-install/bootstrap
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.8.12:8443 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config set-credentials tls-bootstrap-token-user --token=c8ad9c.2e4d610cf3e7426e --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config set-context tls-bootstrap-token-user@kubernetes --cluster=kubernetes --user=tls-bootstrap-token-user --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config use-context tls-bootstrap-token-user@kubernetes --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

注意：如果要修改boostrap.secret.yaml的token-id和token-secret，需要保证下图红圈内内的字符串一致的，并且位数是一样的。还要保证上个命令的黄色字体：c8ad9c.2e4d610cf3e7426e与你修改的字符串要一致

mkdir -p /root/.kube; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config
kubectl create -f bootstrap.secret.yaml

Node节点配置

复制证书

Master01节点复制证书至Node节点

cd /etc/kubernetes/

for NODE in k8s-master02 k8s-master03 k8s-node01 k8s-node02; do
ssh $NODE mkdir -p /etc/kubernetes/pki /etc/etcd/ssl /etc/etcd/ssl
for FILE in etcd-ca.pem etcd.pem etcd-key.pem;do
scp /etc/etcd/ssl/$FILE $NODE:/etc/etcd/ssl/
done
for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do
scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}
done
done

Kubelet配置

所有节点创建相关目录

mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/

所有节点配置kubelet service

vi /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/local/bin/kubelet

Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

所有节点配置kubelet service的配置文件

vi /etc/systemd/system/kubelet.service.d/10-kubelet.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node=''"
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

创建kubelet的配置文件

注意：如果更改了k8s的service网段，需要更改kubelet-conf.yml的clusterDNS:配置，改成k8s Service网段的第十个地址，比如10.96.0.10

vi /etc/kubernetes/kubelet-conf.yml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

启动所有节点kubelet

systemctl daemon-reload
systemctl enable --now kubelet

查看集群状态

kube-proxy配置

以下操作在Master01执行

cd /root/k8s-ha-install
kubectl -n kube-system create serviceaccount kube-proxy
kubectl create clusterrolebinding system:kube-proxy --clusterrole system:node-proxier --serviceaccount kube-system:kube-proxy
SECRET=$(kubectl -n kube-system get sa/kube-proxy --output=jsonpath='{.secrets[0].name}')
JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET --output=jsonpath='{.data.token}'|base64 -d)
PKI_DIR=/etc/kubernetes/pki
K8S_DIR=/etc/kubernetes
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.8.12:8443 --kubeconfig=${K8S_DIR}/kube-proxy.kubeconfig
kubectl config set-credentials kubernetes --token=${JWT_TOKEN} --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
kubectl config set-context kubernetes --cluster=kubernetes --user=kubernetes --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
kubectl config use-context kubernetes --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

在master01将kube-proxy的systemd Service文件发送到其他节点

如果更改了集群Pod的网段，需要更改 kube-proxy/kube-proxy.conf的clusterCIDR:172.16.0.0/12参数为Pod的网段。vi kube-proxy/kube-proxy.conf（在k8s-ha-install目录下执行）

for NODE in k8s-master02 k8s-master03;do
scp ${K8S_DIR}/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig
scp kube-proxy/kube-proxy.conf $NODE:/etc/kubernetes/kube-proxy.conf
scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service
done

for NODE in k8s-node01 k8s-node02;do
scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig
scp kube-proxy/kube-proxy.conf $NODE:/etc/kubernetes/kube-proxy.conf
scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service
done

所有节点启动kube-proxy

systemctl daemon-reload
systemctl enable --now kube-proxy

安装Calico

以下步骤只在master01执行

cd /root/k8s-ha-install/calico/

修改calico-etcd.yaml的以下位置

sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.3.201:2379,https://192.168.3.202:2379,https://192.168.3.203:2379"#g' calico-etcd.yaml

ETCD_CA=`cat /etc/kubernetes/pki/etcd/etcd-ca.pem | base64 | tr -d '\n'`
ETCD_CERT=`cat /etc/kubernetes/pki/etcd/etcd.pem | base64 | tr -d '\n'`
ETCD_KEY=`cat /etc/kubernetes/pki/etcd/etcd-key.pem | base64 | tr -d '\n'`
sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml
sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml


set -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml
sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml


# 更改此处为自己的pod网段
POD_SUBNET="172.16.0.0/12"
sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: '"${POD_SUBNET}"'@g' calico-etcd.yaml
kubectl apply -f calico-etcd.yaml

查看容器状态

查看节点状态

安装CoreDNS

安装对应版本（推荐）

 cd /root/k8s-ha-install/

如果更改了k8s service的网段需要将coredns的serviceIP改成k8s service网段的第十个IP

sed -i "s#10.96.0.10#10.96.0.10#g" CoreDNS/coredns.yaml

安装coredns

kubectl create -f CoreDNS/coredns.yaml

安装最新版CoreDNS

git clone https://github.com/coredns/deployment.git
cd deployment/kubernetes
./deploy.sh -s -i 10.96.0.10 |  kubectl apply -f -
查看状态
kubectl get pods -n kube-system -I k8s-app=kube-dns

安装Metrics Server

在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率。

安装metrics server

cd /root/k8s-ha-install/metrics-server-0.4.x/

kubectl create -f .

等待metrics server启动然后查看状态

安装dashboard

Dashboard部署

Dashboard用于展示集群中的各类资源，同时也可以通过Dashboard实时查看Pod的日志和在容器中执行一些命令等。

安装指定版本dashboard（推荐）

cd /root/k8s-ha-install/dashboard/

kubectl create -f .

安装最新版

官方Github地址： https://github.com/kubernetes/dashboard

可以在官方dashboard查看到最新版dashboard

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.3/aio/deploy/recommended.yaml

创建管理员用户vim admin.yaml

apiVersion:v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system
kubectl apply -f admin.yaml -n kube-system

登录dashboard

在Chrome浏览器启动文件中加入启动参数，用于解决无法访问Dashboard的问题

--test-type --ignore-certificate-errors

更改dashboard的svc为NodePort：

kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard

将ClusterIP更改为NodePort（如果已经为NodePort忽略此步骤）：

查看端口号：

根据自己的实例端口号，通过任意安装了kube-proxy的宿主机或者VIP的IP+端口即可访问到dashboard：

访问dashboard： https://192.168.8.200:30825/#/login，选择登录方式为令牌（即token方式）

查看token值

[root@k8s-master-lb dashboard]#  kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
Name:         admin-user-token-hzblx
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: efd2583f-2cb9-4829-96e5-428901a2f931

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1411 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IllRQVIxdm1zLWowdUp5aXdBdU9KNnBhbTQtMkhBRkJ1U092c0FvN0VPU3cifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWh6Ymx4Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJlZmQyNTgzZi0yY2I5LTQ4MjktOTZlNS00Mjg5MDFhMmY5MzEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.gKfex-XB0Kj1B07d4duvKS3UHmXh2k17tda1HENefHGO92wz4mrYVdG6iwZ1L-_lbLgSYh-nD7XLr0DezvVgu-lb8JXYSlKNqoKlaG2w9FUtVgOxfrTJ_9o4PfiZUMbykGBUl4TwnPO4z6QQNEuCpuf1MEuxlCiXrMenYTTg4mtIMZMxXiZevTt69CNK5HboorCQ-_jArbzdRwevVuWzOP_3Sjv939X6pKyhUB0NE6tESywmvnZblpk8BNnY6wfWEPcJxwPYMPrlbZ3XHYH0Y63bo14MB-sC7lMzEOXTQioGA2IXNVDALcwwpvJ-1Ln7zqEtDfHJKg1cQsVr2tiKWA
You have new mail in /var/spool/mail/root
[root@k8s-master-lb dashboard]#

将token值输入到令牌后，单击登录即可访问Dashboard