一、概述
主要介绍搭建流程及使用注意事项，如果线上使用的话，请务必做好相关测试及压测。
1、基础环境准备
系统：ubuntu TLS 16.04 5台
docker-ce：17.06.2
kubeadm、kubelet、kubectl：1.12.0
keepalived、haproxy
etcd-3.2.22
2、安装前准备
1）k8s各节点SSH免密登录。
2）各Node必须关闭swap：swapoff -a，否则kubelet启动失败。vim /etc/sysctl.confvm.swappiness = 0sysctl -p
etcd采用单独部署的集群。
二、部署准备
1、安装docker：
apt-get updateapt-get install \ apt-transport-https \ ca-certificates \ curl \ software-properties-commoncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -apt-key fingerprint 0EBFCD88add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable"apt-get updateapt policy docker-ce找到自己需要的版本，然后安装apt install -y docker-ce=17.06.2~ce-0~ubuntu 修改docker的一些设置 vim /lib/systemd/system/docker.service ExecStart=/usr/bin/dockerd -H fd:// --graph=/data/docker --storage-driver=overlay2 配置docker的proxy，否则无法直接下载镜像。 mkdir /etc/systemd/system/docker.service.d
cat http-proxy.conf
[Service]Environment="HTTP_PROXY=http://10.42.3.110:8118" "NO_PROXY=localhost,127.0.0.1,0.0.0.0,10.42.73.110,10.10.25.49,10.2.140.154,10.42.104.113,10.2.177.142,10.2.68.77,172.11.0.0,172.10.0.0,172.11.0.0/16,172.10.0.0/16,10.,172.,.evo.get.com,.kube.hp.com,charts.gitlab.io,.mirror.ucloud.cn"cat https-proxy.conf
[Service]Environment="HTTPS_PROXY=http://10.42.3.110:8118" "NO_PROXY=localhost,127.0.0.1,0.0.0.0,10.42.73.110,10.10.25.49,10.2.140.154,10.42.104.113,10.2.177.142,10.2.68.77,172.11.0.0,172.10.0.0,172.11.0.0/16,172.10.0.0/16,10.,172.,.evo.get.com,.kube.hp.com,charts.gitlab.io,.mirror.ucloud.cn"
systemctl daemon-reload
systemctl restart docker
iptables设置，17.06.2默认的FORWARD是DROP，这样pod之间的通信就会收到影响，所以需要设置成ACCEPT
iptables -P FORWARD ACCEPTiptables-save
2、安装CA证书
1）安装cfssl证书管理工具
mkdir /opt/binwget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64chmod +x cfssl_linux-amd64mv cfssl_linux-amd64 /opt/bin/cfsslwget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64chmod +x cfssljson_linux-amd64mv cfssljson_linux-amd64 /opt/bin/cfssljsonwget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64chmod +x cfssl-certinfo_linux-amd64mv cfssl-certinfo_linux-amd64 /opt/bin/cfssl-certinfoecho "export PATH=/opt/bin:$PATH" > /etc/profile.d/k8s.sh source /etc/profile.d/k8s.sh
2)创建CA配置文件
mkdir ssl;cd ssl cfssl print-defaults config >config.json cfssl print-defaults csr >csr.json
根据生成的文件创建ca-config.json文件，过期时间设置为87600h
# cat ca-config.json
{ "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } }} 创建CA证书签名请求
# cat ca-csr.json
{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "GD", "L": "SZ", "O": "k8s", "OU": "System" } ]} 生成证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
copy证书到相应节点
mkdir /etc/kubernetes/ssl -p cp ~/ssl/ca* /etc/kubernetes/ssl scp -r /etc/kubernetes/ km12-02:/etc scp -r /etc/kubernetes/ km12-03:/etc
3、下载etcd，新版本的k8s要求etcd版本3.2.18以上，这里使用3.2.22
wget https://github.com/coreos/etcd/releases/download/v3.2.22/etcd-v3.2.22-linux-amd64.tar.gz tar xf etcd-v3.2.22-linux-amd64.tar.gz cd etcd-v3.2.22-linux-amd64 cp etcd etcdctl /opt/bin source /etc/profile.d/k8s.sh
制作etcd证书：
创建签名文件
# cat etcd-csr.json
{ "CN": "etcd", "hosts": [ "127.0.0.1", "10.42.13.230", "10.42.43.147", "10.42.150.212" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "GD", "L": "SZ", "O": "k8s", "OU": "System" } ]} 生成证书和私钥
cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubernetes/ssl/ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd mkdir /etc/etcd/ssl -p cp etcd*.pem /etc/etcd/ssl scp -r /etc/etcd/ km12-02:/etc scp -r /etc/etcd/ km12-03:/etc
部署etcd集群：
创建etcd的systemd unit文件
# mkdir -p /var/lib/etcd
# cat /etc/systemd/system/etcd.service
[Unit]Description=Etcd ServerAfter=network.targetAfter=network-online.targetWants=network-online.targetDocumentation=https://github.com/coreos[Service]Type=notifyWorkingDirectory=/var/lib/etcd/EnvironmentFile=-/etc/etcd/etcd.confExecStart=/opt/bin/etcd \--name=etcd-host0 \--cert-file=/etc/etcd/ssl/etcd.pem \--key-file=/etc/etcd/ssl/etcd-key.pem \--peer-cert-file=/etc/etcd/ssl/etcd.pem \--peer-key-file=/etc/etcd/ssl/etcd-key.pem \--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \--initial-advertise-peer-urls=https://10.42.13.230:2380 \--listen-peer-urls=https://10.42.13.230:2380 \--listen-client-urls=https://10.42.13.230:2379,http://127.0.0.1:2379 \--advertise-client-urls=https://10.42.13.230:2379 \--initial-cluster-token=etcd-cluster-1 \--initial-cluster=etcd-host0=https://10.42.13.230:2380,etcd-host1=https://10.42.43.147:2380,etcd-host2=https://10.42.150.212:2380 \--initial-cluster-state=new \--data-dir=/var/lib/etcdRestart=on-failureRestartSec=5LimitNOFILE=65536[Install]WantedBy=multi-user.target
将创建好的启动文件和之前解压的etcd文件都copy到其他master节点上
tar xf etcd-v3.2.22-linux-amd64.tar.gz cd etcd-v3.2.22-linux-amd64 cp etcd etcdctl /opt/bin scp -r /opt/bin km12-02:/opt scp -r /opt/bin km12-03:/opt source /etc/profile.d/k8s.sh copy到其他节点 ，注意修改几个地方，修改为相应的名字和地址：--name=etcd-host0--initial-advertise-peer-urls=https://10.42.13.230:2380--listen-peer-urls=https://10.42.13.230:2380--listen-client-urls=https://10.42.13.230:2379,http://127.0.0.1:2379--advertise-client-urls=https://10.42.13.230:2379
启动etcd
systemctl enable etcd systemctl start etcd systemctl status etcd
查看集群状态
etcdctl --key-file /etc/etcd/ssl/etcd-key.pem --cert-file /etc/etcd/ssl/etcd.pem --ca-file /etc/kubernetes/ssl/ca.pem member list etcdctl --key-file /etc/etcd/ssl/etcd-key.pem --cert-file /etc/etcd/ssl/etcd.pem --ca-file /etc/kubernetes/ssl/ca.pem cluster-health
为了方便使用etcdctl我们可以做一个alias,因为kubernetes默认使用etcdv3，而etcd部署默认使用etcdv2，所以在etcd里面查看不到kubernetes的数据，需要添加一个全局变量export ETCDCTL_API=3,
在所有master节点上添加如下内容：
# vim ~/.bashrc
#etcd
host1='10.42.13.230:2379'host2='10.42.43.147:2379'
host3='10.42.150.212:2379'
endpoints=$host1,$host2,$host3alias etcdctl='etcdctl --endpoints=$endpoints --key /etc/etcd/ssl/etcd-key.pem --cert /etc/etcd/ssl/etcd.pem --cacert /etc/kubernetes/ssl/ca.pem'export ETCDCTL_API=3
是内容生效：
#source ~/.bashrc
4、在另外的两台服务器上面部署keepalived、haproxy
apt install -y keepalived haproxy
1)创建keepalived配置文件
master:
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs { notification_email { root@localhost #发送邮箱 } notification_email_from keepalived@localhost #邮箱地址 smtp_server 127.0.0.1 #邮件服务器地址 smtp_connect_timeout 30 router_id km12-01 #主机名，每个节点不同即可 }vrrp_instance VI_1 { state MASTER #在另一个节点上为BACKUP interface eth0 #IP地址漂移到的网卡 virtual_router_id 6 #多个节点必须相同 priority 100 #优先级，备用节点的值必须低于主节点的值 advert_int 1 #通告间隔1秒 authentication { auth_type PASS #预共享密钥认证 auth_pass 571f97b2 #密钥 } unicast_src_ip 10.42.13.23 unicast_peer { 10.42.43.14 } virtual_ipaddress { 10.42.79.210/16 #VIP地址 }} slave1: cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost #发送邮箱 } notification_email_from keepalived@localhost #邮箱地址 smtp_server 127.0.0.1 #邮件服务器地址 smtp_connect_timeout 30 router_id km12-02 #主机名，每个节点不同即可 }vrrp_instance VI_1 { state BACKUP #在另一个节点上为MASTER interface eth0 #IP地址漂移到的网卡 virtual_router_id 6 #多个节点必须相同 priority 80 #优先级，备用节点的值必须低于主节点的值 advert_int 1 #通告间隔1秒 authentication { auth_type PASS #预共享密钥认证 auth_pass 571f97b2 #密钥 } unicast_src_ip 10.42.43.14 unicast_peer { 10.42.13.23 } virtual_ipaddress { 10.42.79.210/16 #漂移过来的IP地址 }} 启动keepalived
systemctl enable keepalivedsystemctl start keepalived
keepalived采用的是单播。
2)haproxy配置文件
cat /etc/haproxy/haproxy.cfg global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemondefaults mode tcp log global retries 3 timeout connect 10s timeout client 1m timeout server 1mfrontend kubernetes bind *:6443 mode tcp default_backend kubernetes-masterbackend kubernetes-master balance roundrobin server master 10.42.13.230:6443 check maxconn 2000 server master2 10.42.43.147:6443 check maxconn 2000
两台配置文件是一样的。
启动haproxy
systemctl enable haproxysystemctl start haproxy
5、安装指定版本的kubeadm、kubectl、kubelet
由于服务器不FQ，下载会有问题，所以我们需要换一下源：
cat </etc/apt/sources.list.d/kubernetes.listdeb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial mainEOF
安装
apt-get updateapt policy kubeadmapt-get install kubeadm=1.12.0-00 kubectl=1.12.0-00 kubelet=1.12.0-00 升级软件 apt-get upgrade kubeadm 回退版本 apt-get install aptitude 比如升到了kubeadm1.13.0，想退回到1.12.0 aptitude install kubeadm=1.12.0-00
卸载软件
apt-get --purge remove 包名称如果不记得包的全部名称，则可以使用如下命令查找dpkg --get-selections | grep kube
创建初始化的kubeadm-config.yaml
# cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1alpha3 controlPlaneEndpoint: 10.42.9.210:6443kind: ClusterConfigurationkubernetesVersion: v1.12.0networking: podSubnet: 192.244.0.0/16 serviceSubnet: 192.96.0.0/16apiServerCertSANs:- 10.42.13.230- 10.42.43.147- 10.42.150.212- km12-01- km12-02- km12-03- 127.0.0.1- localhost- 10.42.9.210etcd: external: endpoints: - https://10.42.13.230:2379 - https://10.42.43.147:2379 - https://10.42.150.212:2379 caFile: /etc/kubernetes/ssl/ca.pem certFile: /etc/etcd/ssl/etcd.pem keyFile: /etc/etcd/ssl/etcd-key.pem dataDir: /var/lib/etcd
clusterDNS:
- 10.96.0.2---
apiVersion: kubeproxy.config.k8s.io/v1alpha1kind: KubeProxyConfigurationmode: "ipvs"controlPlaneEndpoint是haproxy的地址，或vip
检查是否加载了ipvs：
lsmod | grep ip_vs
如果没有输出，则证明没有加载ipvs，需要手动加载，这里为了方便写了一个脚本：
# cat ipvs.sh
#!/bin/bashlist="ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4"for i in $listdomodprobe $idone
初始化
kubeadm init --config kubeadm-config.yaml
如果是已经初始化完成的集群想要修改为ipvs，做如下修改：
kubectl edit configmap kube-proxy -n kube-system ipvs: excludeCIDRs: null minSyncPeriod: 0s scheduler: "" syncPeriod: 30s kind: KubeProxyConfiguration metricsBindAddress: 127.0.0.1:10249 mode: "ipvs"
重新部署kube-proxy:
kubectl delete pod kube-proxy-ppsk90sd -n kube-system
将生成的pki拷贝到其他节点
scp -r /etc/kuberbetes/pki km12-02:/etc/kubernetes scp -r /etc/kubernetes/pki km12-03:/etc/kubernetes注意，到其他服务器上面删掉里面的两个文件： rm -rf apiserver.crt apiserver.key
将初始化的文件以及/etc/profile.d/k8s.conf拷贝到其他节点
scp -r kubeadm-config.yaml km12-02:scp -r kubeadm-config.yaml km12-03:scp -r /etc/profile.d/k8s.sh km12-02:/etc/profile.dscp -r /etc/profile.d/k8s.sh km12-03:/etc/profile.d
有与所有的节点都设置了taints，所以都不能部署，此时会看到coredns一直处于pending的状态，我们暂时把taints去掉，使其能正常部署
kubectl taint nodes --all node-role.kubernetes.io/master-
6、部署网络
网络有几种选择flannel+vxlan、flannel+host-gw、calico+BGP,经过测试calico+BGP这种方式网络性能是比较好的，这里给出两种方式部署网络flannel+host-gw和calico+BGP。推荐使用calico+BGP。
第一种calico+BGP：
calico的datastore有两种方式：kubernetes和etcd，这里建议使用etcd进行数据存储。calico默认使用ipip模式，如果想关闭ipip模式只需要将CALICO_IPV4POOL_IPIP设置为Off，然后就会使用BGP。
下载文件：
wget https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/hosted/calico.yaml
etcd使用前面部署好的etcd集群，所以需要修改calico.yaml文件：
主要修改几个地方：一个是etcd的相关配置，一个是CALICO_IPV4POOL_CIDR
# vim calico.yaml
kind: ConfigMapapiVersion: v1metadata: name: calico-config namespace: kube-systemdata: # Configure this with the location of your etcd cluster. etcd_endpoints: "https://10.42.13.230:2379,https://10.42.43.147:2379,https://10.42.150.212:2379" # If you're using TLS enabled etcd uncomment the following. # You must also populate the Secret below with these files. etcd_ca: "/calico-secrets/etcd-ca" etcd_cert: "/calico-secrets/etcd-cert" etcd_key: "/calico-secrets/etcd-key"
apiVersion: v1kind: Secrettype: Opaquemetadata: name: calico-etcd-secrets namespace: kube-systemdata: # Populate the following files with etcd TLS configuration if desired, but leave blank if # not using TLS for etcd. # This self-hosted install expects three files with the following names. The values # should be base64 encoded strings of the entire contents of each file. etcd-key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFB0NBUUVBd2F3ZHNKcHB5WTR1UjFtZXJocHl5dWlPU1B3N0lWS1Zxc3dXaFV6ZG1YVWRzN2swCmlMaVNGbDhIUjRaVDcvcEIzOGRuenJINE1pQit4TTRDVHRKVkZjTFJBK2tkNUwzZWdodC9KOHlBS3UyVnE2M2wKTWpBazlIb3NqUU96a2dLSnN1RlhyRFdaZG5FeGlvNkZrQXBncDFySmZJdFh1UkpFdDFRMGplblVTM0xLaFZVWQpkdTdTV0thVkc5T3RmS3BzZUsvc2tYcDJlVmJIdGpiRE9lQURodnhyNk1FcDV4SVJ2bmZlWXppWmpUaUJXMkUrCkdFOCtjTGVCTmVLMVYxWFYvR1pmR0FTN2orWFVLOEIxVHFEQ2ZzU3QrRTEzTGMvQWhnTTF2QTZjaDIzMm16RlUKWWw0L21VaU5ZeUFHZmNkTW9XMkR4ck96M28yQ0JoMm50R1FIbndJREFRQUJBb0lCQVFDdUF2dGFKaU5PbVpVegpQbTY0ZDhNVVZRY2l4SjVNUG11RFhvUU5tUmF5UHV4RkF1OFV1T0ZwZ0ZDR0QwZHpveU4wOFRGd0lhK2pPbGNzCmRQelYzVjNkZzRGUlJpODV5dThWYjZjOEZ4NUJjOTY2dGkvc2ZCMTRIQjNUUmJRZVNIRjRhOGZlVHZwbjFTUnkKSGNRYnUzZEVadW9LSVJqY2pIRjVWd3N4UnhzYi9hSGE3MzUzR2pwTDhsZThQVGtielVyVktGYkZDN1U2cE1IYwo3MGxTeW5uUUVmNC9QdjJFcGNWa1hjQVFvWk44MkJUNWNEbTFheWhaWWhaUk5wdEhmV0w5VWtHaUZma3h5TXRnCjllSFJPNFpoMnZ6RW12QWRGbTZKcjB3aDlOQVVMUDhkNlh5OStnUFJ0Q1I4K2p0WHd3SzgwazZZMlpTWXhSTS8KYlAxYytyc0JBb0dCQU1xdk12cHVjTTVrc04vdkl0NE9Uc09lNE5QMzhSQVdaWktycmt4TjEwTXFrSEM5V0cxNQpwYmcyWGVyN2ZYTUFycmtpZUhiUU5ySzB6OTVVN3VnMzMrZ3VvVzRMNUJrYlNYVHY0eDJqMFBuQ3lhOFlMNkwrCjhuTi95SFBmUW1scnZWazJOd1FPSzgyaHQwL0RuOHFQeENNZVY1RmJUOWNjVVN2ZEI1czZqMHcvQW9HQkFQU2UKQzFTRnhJYThFSzVobGxZdXhQSUFHd29BNEIxV2R3RXRLMUF2WjRaamdCdmJqRjEySi9yMGNoQ08zbnllbThGdQovTU51WnczM1hsTThLTmJJWGpja3Y5RG9zQ2lOUXVtRE1ZUVY0SGFCL3U2SHo0MkNiaW1IMUlOSmtacDg5eS9nClRsQmRzb3puMUZiMG9aSFBsamcxR3BlM1hlUW9IZjJnRFFtSFJleWhBb0dBQjhXYURkSklUeEc3V29yZjZtWGcKRFU0OVRzTjJJZWpKOGtyVS95WUkrQkZjd29Yd0t4cDhDVWxFWnF0SUR5M2FoVXpXRTdBK09MSUlPbjFGUC9CTQpqS21sRlFRdHRleSs3MVgzZm41MmwrMHBNS2FieFFORHByd2lvcGJRQkJ6V2dPSThUOUovU1g1RytpOEZKSTVJCnRoUUd4Wk1ieDZMVlZmbyt2V3dNYko4Q2dZQkVEUk9wMSt6c2JyVlZUZUM4NlJYeEhRWm9xZ1d2STdiSHBRRS8dFhCZmVwN1JJU3JUZFdONTlUY21WQmloSXA3Q0dWWklLQmFUVkJYeG9mTGFqYk5vTTlrSkRUSzBsVmZnRHBkZgozNVlxWWMvQ0hCWXBqL2VGcGp3QXFoN1BrNlJRdFY0VURYejJwaWYwYU9ucDNvNHo1TklaRXZJVDN2VTQ4YVd1CjJPc2pBUUtCZ0c0OXZiNjNyc1p0eHZ3ZkhSaUVPUWdZZE9uWkhET2I0K2MxU2hpQWluN0hpOGRjdFR3cnl4a2UKdzBwZzV0R09xa1d1QUJ5VTFSQWR1c0pnUnZQZDBFc2l2YVBVTXhmd0cyWjBEVFpIZGVjS0tjWkNrdXkxQ3NKdgpkVEJLbnRjNmhwUUUrRnl3eHpBc1VTSCtpYUJ4YzczRlp3SWEwWEw2T1BuOFpnTnl5Qi9lCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg etcd-cert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ0akNDQXNxZ0UJBZ0lVWFdsazdGY3JJR005T3Nxc08vOVBqTitjWGlrd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1d6RUxNQWtHQTFVRUJoTUNRMDR4Q3pBSkJnTlZCQWdUQWtkRU1Rc3dDUVlEVlFRSEV3SlRXakVNTUFvRwpBMVVFQ2hNRGF6aHpNUTh3RFFZRFZRUUxFd1pUZVhOMFpXMHhFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13CkhoY05NVGd4TVRBeE1EVTBNVEF3V2hjTk1qZ3hNREk1TURVME1UQXdXakJWTVFzd0NRWURWUVFHRXdKRFRqRUwKTUFrR0ExVUVDQk1DUjBReEN6QUpCZ05WQkFjVEFsTmFNUXd3Q2dZRFZRUUtFd05yT0hNeER6QU5CZ05WQkFzVApCbE41YzNSbGJURU5NQXNHQTFVRUF4TUVaWFJqWkRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDCkFRb0NnZ0VCQU1Hc0hiQ2FhY21PTGtkWm5xNGFjc3JvamtqOE95RlNsYXJNRm9WTTNabDFIYk81TklpNGtoWmYKQjBlR1UrLzZRZC9IWjg2eCtESWdmc1RPQWs3U1ZSWEMwUVBwSGVTOTNvSWJmeWZNZ0NydGxhdXQ1VEl3SlBSNgpMSTBEczVJQ2liTGhWNncxbVhaeE1ZcU9oWkFLWUtkYXlYeUxWN2tTUkxkVU5JM3AxRXR5eW9WVkdIYnUwbGltCmxSdlRyWHlxYkhpdjdKRjZkbmxXeDdZMnd6bmdBNGI4YStqQktlY1NFYjUzM21NNG1ZMDRnVnRoUGhoUFBuQzMKZ1RYaXRWZFYxZnhtWHhnRXU0L2wxQ3ZBZFU2Z3duN0VyZmhOZHkzUHdJWUROYndPbklkdDlwc3hWR0plUDVsSQpqV01nQm4zSFRLRnRnOGF6czk2TmdnWWRwN1JrQjU4Q0F3RUFBYU9Cb3pDQm9EQU9CZ05WSFE4QkFmOEVCQU1DCkJhQXdIUVlEVlIwbEJCWXdGQVlJS3dZQkJRVUhBd0VHQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQ01BQXcKSFFZRFZSME9CQllFRkdDa2lWS0RwcEtWUHp3TVkrU1hWY2NoeFB4OE1COEdBMVVkSXdRWU1CYUFGSlFEaTdGNQp2WlNaSFhFVTlqK0FXVE9Xemx6TU1DRUdBMVVkRVFRYU1CaUhCSDhBQUFHSEJBb3FqSnFIQkFvcWFIR0hCQW9xCnNZNHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQmlkNUh2VmNGUTNlUjhBZXNHekhaRHRsWUZsNGVSUVlsZ0kKd0orbC9XV09QamwxL0lwSDJtanBsWHlnNlJBUmhIeENnZXY1NG1BSGhyK0U4TktBOXMyZFdMNWpnd2twL2p3aApzaDhsUEljeVE2ZHdXdjJNYWZaTU1Xdng5OWIrd2wyRHRGQUpiTzM5bWU4QVFJc2FrRGgrV2ZlZk02dW5QVkdHClhqMjVFbjVwQlB4NHFyRTRUcTVmeURaYUF1ay9tN0NianQ2WklaUUthRjVwYmczMVpCNjZRTVdQbjYvU05lWmkKbEhkWHY3c2xweExvWkZFczdXdEpxM0VUcDE4aXRjTER3S3FDWGZaVzhVZkRWaWNHdlRNYmpBb3NXSG1uaFdnZwpEc0w5MVJhUW9qTTzZnd1d0VaZ3lMaEhrQi96NFR1UUFXT1NaRWU3clJ0ckx0UFNNdVE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K etcd-ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURxakNDQXBLZ0F3SUJBZ0lVR0tCU2dDUTdVU2t6b2E4WnRlNTJ6UERGUk1zd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1d6RUxNQWtHQTFVRUJoTUNRMDR4Q3pBSkJnTlZCQWdUQWtkRU1Rc3dDUVlEVlFRSEV3SlRXakVNTUFvRwpBMVVFQ2hNRGF6aHpNUTh3RFFZRFZRUUxFd1pUZVhOMFpXMHhFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13CkhoY05NVGd4TVRBeE1EVXpPREF3V2hjTk1qTXhNRE14TURVek9EQXdXakJiTVFzd0NRWURWUVFHRXdKRFRqRUwKTUFrR0ExVUVDQk1DUjBReEN6QUpCZ05WQkFjVEFsTmFNUXd3Q2dZRFZRUUtFd05yT0hNeER6QU5CZ05WQkFzVApCbE41YzNSbGJURVRNQkVHQTFVRUF4TUthM1ZpWlhKdVpYUmxjekNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBTjdLd0ZJUlFNL2hTZTdZdTRsSTEyVDRLdzJHYVc4bnA4YU9XZXZzNzhGQU9qYnAKQnBsSEZVQmFrVVcrN3pnU0FHQVMzRnU2Um14ZGk1T2c3a2dZUG9TbitNRDFkWjVIYm1oQjhCY2RtOGVnQnZRcApONUtlS0ovOTJTN09SSm9VeHpmWEhUN3V5bTlZZUhSa0owaTFzSlNJWVB5dXJUbGhOUGs2K3B6aEUzN3Rhd1NkClNHM3l5UlJOY25BOHFyRE5nQlVyUUJBVVRwa0N5SklyVG1oOEpqckl1YTdKMml4YTRidXo3ZWxGRURDYTJpcTcKVmNQZkZ3MVdnM09pMVNmSWVxWVlTUW0zQkpiRnZ2cjBCaUQzYVM5cDNJenNoNEtRYlkyUlhrKzZxTWxYb2ZNaAo4Vmp0MVU2Tmh4RHlkZGwwVDJrRWNwRkx3dWk4ZFRRSEtndU4yQTBDQXdFQUFhTm1NR1F3RGdZRFZSMFBBUUgvCkJBUURBZ0VHTUJJR0ExVWRFd0VCL3dRSU1BWUJBZjhDQVFJd0hRWURWUjBPQkJZRUZKUURpN0Y1dlpTWkhYRVUKOWorQVdUT1d6bHpNTUI4R0ExVWRJd1FZTUJhQUZKUURpN0Y1dlpTWkhYRVU5aitBV1RPV3psek1NQTBHQ1NxRwpTSWIzRFFFQkN3VUFBNElCQVFBamRMakRoMGN2bnNSai9STVhwbWFZRnEwaFowcmZnVmtWcnowK1VDdWRkZnZ3ClVISVEva0p2RFVCdDl5WEtud3ZqQ0JKR3grMWsySTlENVVUcDI4bUpXcUpFbXdWcllqaXo4eUhYamNVbUlNVWkKeHFrZUlwU2lYR0tVZ3NDV1d0U0cwR2Vnbi9telVPMERwVHBVMkdyM0ZjbXJPcGlVNFd0RURZVUZtUm4yMXc0Wgo1S3RnSkpPNWFXaGFMR3c2VU52MFVhRUhTTjZoUk9qWG9pMy8zK09KN1pRajc1YXFnU1k0TURheDZBSFZ4NENSCnk2Q0srNUx0NWhNc0lKVkhETjZkQXYyTlFBU0VxSFVZazJmVFJ1VjJjeExIcE1MQVVTc1JRaThZWmJwYys2TUMKZjkxOHp3MG03NjZ3eTdqYVFJYmJTTXJzbGUvTnhoZnFTQSs0anNFdwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
etcd-key：填写cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n' 的输出
etcd-cert：填写 cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n' 的输出
etcd-ca：填写cat /etc/kubernetes/ssl/ca.pem | base64 | tr -d '\n' 的输出
- name: CALICO_IPV4POOL_CIDR value: "192.244.0.0/16"
CALICO_IPV4POOL_CIDR的value需要与初始化设置的 podSubnet: 192.244.0.0/16相同，不能和serviceSubnet重复。
# Auto-detect the BGP IP address. - name: IP value: "autodetect" - name: IP_AUTODETECTION_METHOD value: "can-reach=8.8.8.8"
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP value: "Off"
部署
kubectl create-f calico.yaml
删除calico
kubectl delete -f calico.yamlmodprobe -r ipip ifconfig cali开头的网卡 down ip link delete cali开头的网卡 etcdctl del --prefix /calico
QA：部署过程中可能会碰到如下错误：
2018-11-19 08:51:49.719 [INFO][8] startup.go 264: Early log level set to info2018-11-19 08:51:49.719 [INFO][8] startup.go 284: Using HOSTNAME environment (lowercase) for node name2018-11-19 08:51:49.719 [INFO][8] startup.go 292: Determined node name: ui2018-11-19 08:51:49.720 [INFO][8] startup.go 105: Skipping datastore connection test2018-11-19 08:51:49.721 [INFO][8] startup.go 365: Building new node resource Name="ui"2018-11-19 08:51:49.721 [INFO][8] startup.go 380: Initialize BGP data2018-11-19 08:51:49.722 [INFO][8] startup.go 582: Using autodetected IPv4 address on interface br-99ef8e85b719: 172.18.0.1/162018-11-19 08:51:49.722 [INFO][8] startup.go 450: Node IPv4 changed, will check for conflicts2018-11-19 08:51:49.722 [WARNING][8] startup.go 879: Calico node 'ku' is already using the IPv4 address 172.18.0.1.2018-11-19 08:51:49.722 [WARNING][8] startup.go 1104: TerminatingCalico node failed to start
导致node节点的calico-node不能启动，这个主要是ip地址冲突导致，因为calico里面有个参数IP_AUTODETECTION_METHOD默认值是first-found.官方解释：
The method to use to autodetect the IPv4 address for this host. This is only used when the IPv4 address is being autodetected. See IP Autodetection methods for details of the valid methods. [Default: first-found]
https://docs.projectcalico.org/v3.4/reference/node/configuration
由于我们使用的ipv4，如果是ipv6的花还需要设置IP6_AUTODETECTION_METHOD。
这里的解决方式是在calico.yaml文件里面添加如下修改：
# Auto-detect the BGP IP address. - name: IP value: "autodetect" - name: IP_AUTODETECTION_METHOD value: "can-reach=8.8.8.8"
重新部署calico即可解决。
QA: 之前使用calico+ipip的模式，升级到1.13.0的时候出现一个请求长度的限制，超过1400就会被拒绝掉，切换到BGP模式就解决了这个问题，所以直接使用BGP模式。
安装calicoctl：
https://docs.projectcalico.org/v3.4/usage/calicoctl/install
我这里采用binary的方式安装
curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.3.2/calicoctlchmod +x calicoctl
我们之前是将数据存储到etcd里面，所以为calicoctl创建如下配置文件：
# cat /etc/calico/calicoctl.cfg
apiVersion: projectcalico.org/v3kind: CalicoAPIConfigmetadata:spec: etcdEndpoints: https://10.42.140.154:2379,https://10.42.104.113:2379,https://10.42.177.142:2379 etcdKeyFile: /etc/etcd/ssl/etcd-key.pem etcdCertFile: /etc/etcd/ssl/etcd.pem etcdCACertFile: /etc/kubernetes/ssl/ca.pem
第二种flannel+host-gw：
下载文件
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
需要作如下修改
vim kube-flannel.yml args: - --ip-masq - --kube-subnet-mgr - --iface=eth0
如果是自己定义的 podSubnet: 10.244.0.0/16 和serviceSubnet: 10.96.0.0/16，则还需要修改一个地方，修改为 podSubnet：
net-conf.json: | { "Network": "192.244.0.0/16", "Backend": { "Type": "host-gw" } } 部署
kubectl apply -f kube-flannel.yaml
删除flannel
kubectl delete -f flannel.yamlifconfig cni0 downip link delete cni0ifconfig flannel.1 downip link delete flannel.1rm -rf /var/lib/cni/
三、添加node节点
默认每个节点使用kubeadm初始化的时候都会创建一个token，可以选择任意一个master节点的token来添加节点，但是这样管理起来不是很方便。所一先把节点的所有token都删掉，然后创建一个token
用来添加节点，token默认ttl是24小时，可以讲ttl设置为0，这样token就永久有效，不建议将ttl设置成0，设置成2天轮换一次。具体步骤如下：
1、删掉原油默认的token
kubeadm tokne list找到默认的token，然后都删掉kubeadm token delete l36ybs.04f5zkvewgtsd703kubeadm token delete zvbdfg.9kx9fzozxfz8sekokubeadm token delete zvfoiw.r7p0rxtvgxc95owu使用kubeadm token create生成tokenkubeadm token create --ttl 72h
为了方便使用，写了一个脚本：
# cat uptk.sh#!/bin/bashtk=`kubeadm token list | awk '{print $1}' |grep -v TOKEN`kubeadm token delete $tkkubeadm token create --ttl 72h crontab -e * * */2 * * /data/scripts/uptk.sh 2>&1 & >>/dev/null
关于kubeadm管理token更多细节参考官方文档：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-token/
2、添加节点
准备一台新的服务器，做好免密交互，安装kubeadm、kubelet、docker等，按照上面准备master那样，只是不执行初始化，而是kubeadm join，注意版本要一致：
在master上操作：
查看集群的tokenkubeadm token list | grep authentication,signing | awk '{print $1}'查看discovery-token-ca-cert-hashopenssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' 为了方便使用将上述命令做成了一个生成脚本，需要在master节点上执行，当然可以根据服务器情况进行修改：
cat generate_kubeadm_join.sh#!/bin/bash#delete the invalid token and create new tokentk_old=`kubeadm token list | awk '{print $1}' | grep -v TOKEN`kubeadm token delete $tk_old >>/dev/null 2>& 1kubeadm token create --ttl 72h >>/dev/null 2>& 1#print option which kubeadm jointk_new=`kubeadm token list | awk '{print $1}' | grep -v TOKEN`hs=`openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'`port=6443ip=`ip a | grep eth0 | grep inet| awk -F / '{print $1}'| awk '{print $2}'`op="kubeadm join --token $tk_new $ip:$port --discovery-token-ca-cert-hash sha256:$hs"echo $op
在node节点上操作：
根据上面获取的token和hashkubeadm join --token c04f89.b781cdb55d83c1ef 10.42.9.210:6443 --discovery-token-ca-cert-hash sha256:986e83a9cb948368ad0552b95232e31d3b76e2476b595bd1d905d5242ace29af
在master上操作：
为node节点打标签kubectl label node knode1 node-role.kubernetes.io/node=
删除节点：
在master节点上：kubectl drain node1 --delete-local-data --force --ignore-daemonsetskubectl delete node node1在node节点上：kubeadm reset
ps：node1 是节点名称
如果部署的有问题，可以使用如下的方法进行重置：
#kubeadm resetifconfig cni0 downip link delete cni0ifconfig flannel.1 downip link delete flannel.1rm -rf /var/lib/cni/ rm -rf ~/.kube ipvsadm --clear
如果使用的etcd是external的方式，也就是外部集群，使用kubeadm reset重置集群的时候并不能删除etcd里面的数据，需要使用etcdctl工具手动删除。
由于前面已经设置了alias，所以可以直接使用etcdctl：
etcdctl del "" --prefix
四、部署监控
1、部署metrics-server
git clone https://github.com/cuishuaigit/k8s-monitor.gitcd k8s-monitor/metrics-server
我都是把这种服务部署在master节点上面，此时需要修改metrics-server-deployment.yaml
---apiVersion: v1kind: ServiceAccountmetadata: name: metrics-server namespace: kube-system---apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: metrics-server namespace: kube-system labels: k8s-app: metrics-serverspec: selector: matchLabels: k8s-app: metrics-server template: metadata: name: metrics-server labels: k8s-app: metrics-server spec: serviceAccountName: metrics-server tolerations: - effect: NoSchedule key: node.kubernetes.io/unschedulable operator: Exists - key: NoSchedule operator: Exists effect: NoSchedule volumes: # mount in tmp so we can safely use from-scratch images and/or read-only containers - name: tmp-dir emptyDir: {} containers: - name: metrics-server image: k8s.gcr.io/metrics-server-amd64:v0.3.1 imagePullPolicy: Always command: - /metrics-server - --kubelet-insecure-tls - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP volumeMounts: - name: tmp-dir mountPath: /tmp nodeSelector: metrics: "yes" 为master添加label
kubectl label nodes ku metrics=yes
部署
kubectl create -f metrics-server/deploy/1.8+/
如果有报错参考我的另外一片博客：https://www.cnblogs.com/cuishuai/p/9857120.html
接下来稍等几十秒就可以查看状态了
2、部署prometheus
下载相关文件使用上面clone的k8s-monitor部署cd k8s-monitor
1.搭建nfs服务动态提供持久化存储（如果已有可以跳过）
1.安装nfs sudo apt-get install -y nfs-kernel-serversudo apt-get install -y nfs-common sudo vi /etc/exports /data/opv *(rw,sync,no_root_squash,no_subtree_check)注意将*换成自己的ip段，纯内网的话也可以用*，代替任意sudo /etc/init.d/rpcbind restart sudo /etc/init.d/nfs-kernel-server restart sudo systemctl enable rpcbind nfs-kernel-server客户端挂在使用
sudo apt-get install -y nfs-commonmount -t nfs ku13-1:/data/opv /data/opv -o proto=tcp -o nolock为了方便使用将上面的mount命令直接放到.bashrc里面 2.创建namesapcekubectl creaet -f nfs/monitoring-namepsace.yaml 3.为nfs创建rbac kubectl create -f nfs/rbac.yaml 4.创建deploymentkubectl create -f nfs/nfs-deployment.yaml 5.创建storageclasskubectl create -f nfs/storageClass.yaml2.安装Prometheus
cd k8s-monitor/Promutheus/prometheus1.创建权限kubectl create -f rbac.yaml2.创建 node-exporterkubectl create -f prometheus-node-exporter-daemonset.yamlkubectl create -f prometheus-node-exporter-service.yaml3.创建 kube-state-metricskubectl create -f kube-state-metrics-deployment.yamlkubectl create -f kube-state-metrics-service.yaml4.创建 node-directory-size-metricskubectl create -f node-directory-size-metrics-daemonset.yaml5.创建 prometheuskubectl create -f prometheus-pvc.yamlkubectl create -f prometheus-core-configmap.yamlkubectl create -f prometheus-core-deployment.yamlkubectl create -f prometheus-core-service.yamlkubectl create -f prometheus-rules-configmap.yaml6.修改core-configmap里的etcd地址3.安装Grafana
cd k8s-monitor/Promutheus/grafana1.安装grafana servicekubectl create -f grafana-svc.yaml2.创建configmapkubectl create -f grafana-configmap.yaml3.创建pvckubectl create -f grafana-pvc.yaml4.创建gragana deploymentkubectl create -f grafana-deployment.yaml5.创建dashboard configmapkubectl create configmap "grafana-import-dashboards" --from-file=dashboards/ --namespace=monitoring6.创建job，导入dashboard等数据 kubectl create -f grafana-job.yamlprometheus和grafana都是采用的nodePort方式暴漏的服务，所以可以直接访问。
grafana默认的用户名密码：admin/admin