kubernetes+keepalived高可用部署

一、环境

1
2
3
4
5
6
7
8
9
10
11
12
13
14
下面是主机信息：
kubernetes1.yp14.cn  192.168.30.1 (etcd ，kube-apiserver，kube-controller-manager， kube-scheduler，flannel，keepalived) 系统: centos7.2
kubernetes2.yp14.cn  192.168.30.2 (etcd ，kube-apiserver，kube-controller-manager， kube-scheduler，flannel，docker，kubelet，kube-proxy，keepalived) 系统: centos7.2
docker.yp14.cn       192.168.30.3  (etcd，docker） 系统: ubuntu14.04
VIP                         192.168.30.10                   
PS： kubernetes1是keepalived主，kubernetes2是keepalived备

软件版本：
Kubernetes：v1.3.7
Keepalive：v1.2.24
etcd：v3.0.15
docker：1.11.2
flannel：0.5.3
pause：3.0

二、各组件用途

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
kube master
    kube-apiserver
    k8s的管理接口

    kube-scheduer
    k8s调度器，容器的启动、迁移、扩容缩减时候，选择哪个node，就看它了。

    kube-controller-manager
    k8s对node的控制行为，比如怎么去调用node启动一个容器。

kube node
    kubelet
    负责node的管理，基本所有操作都靠它。

    kube-proxy
    每个node里的container都在一个私有网络中，kube-proxy的作用就是做一个反向代理，让访问者访问这个node的时候，可以转发到内部对应的container。

etcd
    作为kubernetes的数据库，存储了k8s自身的信息、以及各种业务容器信息等。
    存储flannel网络配置信息，供各节点协调。

PS: 这是我狭隘的理解，仅供参考。

三、ETCD安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
PS： ETCD集群启动方式：配置文件都配置完成，首先启动kubernetes1主机,在启动kubernetes2主机，最后启动docker主机

启动成功可以查询集群命令
$ etcdctl member list
4eb1b87aa8b3b311: name=etcd0 peerURLs=http://192.168.30.1:2380 clientURLs=http://192.168.30.1:2379 isLeader=false
738d1cb28bd14833: name=etcd1 peerURLs=http://192.168.30.2:2380 clientURLs=http://192.168.30.2:2379 isLeader=true
bfd47a30ff185072: name=etcd2 peerURLs=http://192.168.30.3:2380 clientURLs=http://192.168.30.3:2379 isLeader=false

<--------------------------- kubernetes1节点安装 etcd start--------------------------->
第一步：
$ mkdir /usr/local/src/kubernetes
$ cd /usr/local/src/kubernetes
$ wget  https://github.com/coreos/etcd/releases/download/v3.0.15/etcd-v3.0.15-linux-amd64.tar.gz
$ tar xf etcd-v3.0.15-linux-amd64.tar.gz && cd etcd-v3.0.15-linux-amd64
$ cp etcd etcdctl /usr/local/sbin/ && chmod +x /usr/local/sbin/etcd  /usr/local/sbin/etcdctl

用systemctl控制etcd启动
$ cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/data/etcd/data-dir/
EnvironmentFile=-/etc/kubernetes/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/sbin/etcd"

[Install]
WantedBy=multi-user.target

$ mkdir /data/etcd/data-dir/ /etc/kubernetes/

第二步：
ETCD集群配置文件
$ cat /etc/kubernetes/etcd.conf
# [member]
ETCD_NAME=etcd0
ETCD_DATA_DIR="/data/etcd/data-dir/"
ETCD_LISTEN_PEER_URLS="http://192.168.30.1:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.30.1:2379,http://127.0.0.1:2379"

#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.30.1:2380"
ETCD_INITIAL_CLUSTER="etcd0=http://192.168.30.1:2380,etcd1=http://192.168.30.2:2380,etcd2=http://192.168.30.3:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="my-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.30.1:2379"

第三步：
启动etcd命令
$ systemctl daemon-reload
$ systemctl enable etcd
$ systemctl start etcd
<--------------------------- kubernetes1节点安装 etcd end--------------------------->


<--------------------------- kubernetes2节点安装 etcd start--------------------------->
第一步和第三步安装和kubernetes1节点安装一样，只有第二步配置文件不一样，配置文件如下：
$ cat /etc/kubernetes/etcd.conf
# [member]
ETCD_NAME=etcd1
ETCD_DATA_DIR="/data/etcd/data-dir/"
ETCD_LISTEN_PEER_URLS="http://192.168.30.2:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.30.2:2379,http://127.0.0.1:2379"

#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.30.2:2380"
ETCD_INITIAL_CLUSTER="etcd0=http://192.168.30.1:2380,etcd1=http://192.168.30.2:2380,etcd2=http://192.168.30.3:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="my-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.30.2:2379"
<--------------------------- kubernetes2节点安装 etcd end--------------------------->


<--------------------------- docker节点安装 etcd start--------------------------->
第一步：
$ mkdir /usr/local/src/kubernetes
$ cd /usr/local/src/kubernetes
$ wget  https://github.com/coreos/etcd/releases/download/v3.0.15/etcd-v3.0.15-linux-amd64.tar.gz
$ tar xf etcd-v3.0.15-linux-amd64.tar.gz && cd etcd-v3.0.15-linux-amd64
$ cp etcd etcdctl /usr/local/sbin/ && chmod +x /usr/local/sbin/etcd  /usr/local/sbin/etcdctl

第二步：
$ mkdir /usr/local/etcd && touch start.sh && chmod +x start.sh
$ mkdir /var/log/etcd/ /data/etcd/data-dir
$ cat /usr/local/etcd/start.sh
#!/bin/bash

/usr/local/sbin/etcd -name etcd2 --data-dir /data/etcd/data-dir -initial-advertise-peer-urls http://192.168.30.3:2380 \
-listen-peer-urls http://192.168.30.3:2380 \
-listen-client-urls http://192.168.30.3:2379,http://127.0.0.1:2379 \
-advertise-client-urls http://192.168.30.3:2379 \
-initial-cluster-token my-etcd-cluster \
-initial-cluster etcd0=http://192.168.30.1:2380,etcd1=http://192.168.30.2:2380,etcd2=http://192.168.30.3:2380 \
-initial-cluster-state new >> /var/log/etcd/etcd.log 2>&1 &

$ vim /etc/rc.local
在rc.local文件中加入下面两行，实现开机自动启动
#start etcd
/bin/bash /usr/local/etcd/start.sh
<--------------------------- docker节点安装 etcd end--------------------------->

四、Flannel安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
<--------------------------- kubernetes1节点安装 Flannel start--------------------------->
初始化flannel的etcd配置
$ $ etcdctl set /coreos.com/network/config '{ "Network": "172.19.0.0/16" }'
$ yum install flannel -y
$ grep -v ^$  /etc/sysconfig/flanneld | grep -v ^#
FLANNEL_ETCD="http://192.168.30.1:2379,http://192.168.30.2:2379,http://192.168.30.3:2379"
FLANNEL_ETCD_KEY="/coreos.com/network"

设置开机启动和启动服务
$ systemctl enable flanenld
$ systemctl restart flanneld

flannel启动后可以查看下面文件，可以看到所分配的ip网段
$ cat /run/flannel/subnet.env
<--------------------------- kubernetes1节点安装 Flannel end--------------------------->

<--------------------------- kubernetes2节点安装 Flannel start--------------------------->
$ yum install flannel docker -y
$ grep -v ^$  /etc/sysconfig/flanneld | grep -v ^#
FLANNEL_ETCD="http://192.168.30.1:2379,http://192.168.30.2:2379,http://192.168.30.3:2379"
FLANNEL_ETCD_KEY="/coreos.com/network"

设置开机启动和启动服务
$ systemctl enable flanenld
$ systemctl restart flanneld

flannel启动后可以查看下面文件，可以看到所分配的ip网段
$ cat /run/flannel/subnet.env
<--------------------------- kubernetes2节点安装 Flannel end--------------------------->

<--------------------------- docker节点安装 Flannel start--------------------------->
$ cd /usr/local/src/kubernetes/
$ wget https://github.com/coreos/flannel/releases/download/v0.5.3/flannel-0.5.3-linux-amd64.tar.gz
$ tar xf flannel-0.5.3-linux-amd64.tar.gz && cd flannel-0.5.3/
$ cp flanneld /usr/local/sbin && chmod +x /usr/local/sbin/flanneld
$ mkdir /usr/local/flanneld && touch start.sh && chmod +x start.sh
$ cat start.sh
#!/bin/bash

/usr/local/sbin/flanneld -etcd-endpoints="http://192.168.30.1:2379,http://192.168.30.2:2379,http://192.168.30.3:2379" -etcd-prefix=/coreos.com/network >> /var/log/flanneld/flanneld.log 2>&1 &
$ mkdir /var/log/flanneld/
$ vim /etc/rc.local
在rc.local文件中加入下面两行，实现开机自动启动
#start flanneld
/bin/bash /usr/local/flanneld/start.sh

flannel启动后可以查看下面文件，可以看到所分配的ip网段
$ cat /run/flannel/subnet.env
FLANNEL_NETWORK=172.19.0.0/16
FLANNEL_SUBNET=172.19.13.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=false
<--------------------------- docker节点安装 Flannel end--------------------------->

五、Docker安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
<--------------------------- kubernetes2节点安装 docker start--------------------------->
$ yum install docker -y
设置开机启动和启动服务
$ systemctl enable docker
$ systemctl restart flanneld docker

在这个文件下可以看到dokcer分到的ip网段，和Flannel是一个网络。Flannel作用就是提供给多个docker主机网络通信
$ cat /run/flannel/docker
DOCKER_OPT_BIP="--bip=172.19.70.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=1426"
DOCKER_NETWORK_OPTIONS=" --bip=172.19.70.1/24 --ip-masq=true --mtu=1426 "
<--------------------------- kubernetes2节点安装 docker end--------------------------->

<--------------------------- docker节点安装 docker start--------------------------->
从官方安装docker
$ curl -sSL https://get.docker.com/ | sh
$ cat /run/flannel/subnet.env
FLANNEL_NETWORK=172.19.0.0/16
FLANNEL_SUBNET=172.19.13.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=false

把flannel配置文件中的FLANNEL_SUBNET参数值，配置到docker配置文件中，如下所是
$ grep -v ^$ /etc/default/docker | grep -v ^#
DOCKER_OPTS="--bip=172.19.13.1/24"

设置开机启动和启动服务
$ update-rc.d docker default
$ service docker start
<--------------------------- docker节点安装 docker end--------------------------->

PS：最后测试kubernetes2主机和docker主机网络是否通信，正常是能够通信，如果不能够通信，请检查防火墙或者selinux等

注意：CentOS7这个版本的防火墙默认使用的是firewall，与之前的版本使用iptables不一样，所以我们要删掉firewall，使用iptables
1、关闭防火墙：sudo systemctl stop firewalld.service
2、关闭开机启动：sudo systemctl disable firewalld.service
3、安装iptables防火墙 执行以下命令安装iptables防火墙：sudo yum install iptables-services
4、配置iptables防火墙，打开指定端口（具体跟以前版本一样，网上介绍很多，这里不多介绍了）
5、设置iptables防火墙开机启动：sudo systemctl enable iptables
$ grep -v ^$ /etc/sysconfig/iptables | grep -v ^#
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p vrrp -j ACCEPT
COMMIT

六、kubernetes master安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
<--------------------------- kubernetes1节点安装 kubernetes master start--------------------------->
第一步：
下载二进制包
$ cd /usr/local/src/kubernetes
$ wget https://storage.googleapis.com/kubernetes-release/release/v1.3.7/kubernetes.tar.gz
$ tar xf kubernetes.tar.gz && mv kubernetes kubernetes-1.3.7
$ cd kubernetes-1.3.7/server/ && tar xf kubernetes-server-linux-amd64.tar.gz
$ cd kubernetes/server/bin/
$ mkidr /opt/bin
$ cp kube-apiserve kube-controller-manager kubectl kube-dns kubelet kube-proxy kube-scheduler /opt/bin
$ useradd kube -s /sbin/nologin && chmod +x /opt/bin/kube* && chown root.kube /opt/bin/kube*

在/etc/profile文件中添加 export PATH=$PATH:/opt/bin 一行
$ source /etc/profile

第二步：
用systemctl 来管理启动和开机启动
$ cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=/etc/kubernetes/apiserver
User=kube
ExecStart=/opt/bin/kube-apiserver $KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

$ cat /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=/etc/kubernetes/controller-manager
User=kube
ExecStart=/opt/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

$ cat /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=/etc/kubernetes/scheduler
User=kube
ExecStart=/opt/bin/kube-scheduler $KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

第三步：
创建配置文件并配置
$ cat /etc/kubernetes/apiserver
KUBE_API_ARGS="--etcd_servers=http://192.168.30.1:2379,http://192.168.30.2:2379,http://192.168.30.3:2379 --insecure-bind-address=0.0.0.0 --insecure-port=8080 --service-cluster-ip-range=172.30.0.0/16 --service-node-port-range=1-65535 --admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota --secure-port=0 --logtostderr=false --log-dir=/var/log/kubernetes --alsologtostderr=false --v=2"

$ cat /etc/kubernetes/controller-manager
KUBE_CONTROLLER_MANAGER_ARGS="--master=http://192.168.30.1:8080 --leader-elect=true --logtostderr=false --log-dir=/var/log/kubernetes --alsologtostderr=false --v=2"

$ cat /etc/kubernetes/scheduler
KUBE_SCHEDULER_ARGS="--master=http://192.168.30.1:8080 --leader-elect=true --logtostderr=false --log-dir=/var/log/kubernetes --alsologtostderr=false --v=2"

第四步:
下面开机启动和启动服务命令，还有就是加载新增目录 /usr/lib/systemd/system/ 下的文件，这样systemctl才能管理
$ systemctl daemon-reload
$ systemctl enable kube-apiserver kube-scheduler kube-controller-manager
$ systemctl start kube-apiserver kube-scheduler kube-controller-manager
<--------------------------- kubernetes1节点安装 kubernetes master end--------------------------->


<--------------------------- kubernetes2节点安装 kubernetes master start--------------------------->
第一步、第二步、第四步安装和kubernetes1节点安装一样

第三步：
创建配置文件并配置
$ cat /etc/kubernetes/apiserver
KUBE_API_ARGS="--etcd_servers=http://192.168.30.1:2379,http://192.168.30.2:2379,http://192.168.30.3:2379 --insecure-bind-address=0.0.0.0 --insecure-port=8080 --service-cluster-ip-range=172.30.0.0/16 --service-node-port-range=1-65535 --admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota --secure-port=0 --logtostderr=false --log-dir=/var/log/kubernetes --alsologtostderr=false --v=2"

$ cat /etc/kubernetes/controller-manager
KUBE_CONTROLLER_MANAGER_ARGS="--master=http://192.168.30.2:8080 --leader-elect=true --logtostderr=false --log-dir=/var/log/kubernetes --alsologtostderr=false --v=2"

$ cat /etc/kubernetes/scheduler
KUBE_SCHEDULER_ARGS="--master=http://192.168.30.2:8080 --leader-elect=true --logtostderr=false --log-dir=/var/log/kubernetes --alsologtostderr=false --v=2"
<--------------------------- kubernetes2节点安装 kubernetes master end--------------------------->

七、kubernetes node安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
<--------------------------- kubernetes2节点安装 kubernetes node start--------------------------->
第一步：
用systemctl 来管理启动和开机启动
$ cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=/etc/kubernetes/kubelet
ExecStart=/opt/bin/kubelet $KUBELET_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target

$ cat /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
Requires=network.service

[Service]
EnvironmentFile=/etc/kubernetes/proxy
ExecStart=/opt/bin/kube-proxy $KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

第二步：
创建配置文件并配置
$ cat /etc/kubernetes/kubelet
KUBELET_ARGS="--api-servers=http://192.168.30.10:8080 --hostname-override=192.168.30.2 --pod-infra-container-image=gcr.io/google_containers/pause:3.0 --logtostderr=false --log-dir=/var/log/kubernetes --alsologtostderr=false --v=2"

$ cat /etc/kubernetes/proxy
KUBE_PROXY_ARGS="--master=http://192.168.30.10:8080 --proxy-mode=iptables --logtostderr=false --log-dir=/var/log/kubernetes --alsologtostderr=false --v=2"

第三步:
下面开机启动和启动服务命令，还有就是加载新增目录 /usr/lib/systemd/system/ 下的文件，这样systemctl才能管理
$ systemctl daemon-reload
$ systemctl enable kubelet kube-proxy
$ systemctl start kubelet kube-proxy
<--------------------------- kubernetes2节点安装 kubernetes node end--------------------------->


<--------------------------- docker节点安装 kubernetes node start--------------------------->
第一步：
拷贝启动命令和启动文件
$ mkdir /opt/bin
在/etc/profile文件中添加 export PATH=$PATH:/opt/bin 一行
$ source /etc/profile
利用scp命令拷贝kubernetes2机器/opt/bin目录下的kubelet和kube-proxy文件到本机/opt/bin目录下
利用scp命令拷贝kubernetes1机器/usr/local/src/kubernetes/kubernetes-1.3.7/cluster/ubuntu/minion/init_scripts目录下kubelet kube-proxy文件
到/etc/init.d目录下
$ chmod +x /etc/init.d/kube*
修改/etc/init.d/kubele 和 /etc/init.d/kube-proxy 文件37行，修改如下：
if false &&  [ -x /sbin/initctl ] && /sbin/initctl version 2>/dev/null | grep -q upstart; then

第二步：
创建配置文件并配置
$ mkdir /var/log/kubernetes
$ cat /etc/default/kubelet
KUBELET_OPTS="\
--api_servers=http://192.168.30.10:8080 \
--hostname_override=192.168.30.3 \       
--pod-infra-container-image=docker.io/kubernetes/pause:latest \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--alsologtostderr=false \
--v=2"

$ cat /etc/default/kube-proxy
KUBE_PROXY_OPTS="\
--master=http://192.168.30.10:8080 \
--proxy-mode=iptables \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--alsologtostderr=false \
--v=2"

第三步:
启动和设置开机启动
$ service kubelet start
$ service kube-proxy start
$ update-rc kubelet default
$ update-rc kube-proxy default
<--------------------------- docker节点安装 kubernetes node end--------------------------->

八、keepalived安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
<--------------------------- kubernetes1节点安装 keepalived start--------------------------->
第一步：
安装依赖环境
yum -y install openssl-devel openssl postfix
yum install gcc gcc-c++ -y

配置postfix
$ cd /etc/postfix
$ vim main.cf
把main.cf文件中inet_protocols = all 修改成为 inet_protocols = ipv4
$ systemctl enable postfix
$ systemctl restart postfix

第二步：
下载keepalived1.2.24源码包并安装
$ wget -S http://www.keepalived.org/software/keepalived-1.2.24.tar.gz
$ tar -zxvf keepalived-1.2.24.tar.gz
$ cd keepalived-1.2.24
$ ./configure --prefix=/usr/local/keepalived
$ make && make install

第三步：
拷贝需要的文件
$ cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived
$ cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
$ chmod +x /etc/init.d/keepalived /usr/sbin/keepalived
$ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
$ mkdir -p /etc/keepalived/
$ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf

第四步：修改配置文件
$ cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     389736180@qq.com
   }

   notification_email_from root@kubernetes1.yp14.cn
   smtp_server 192.168.30.1
   smtp_connect_timeout 30
   router_id Master1
}

vrrp_script check_k8s {             
    script "/etc/keepalived/chk_k8s_master.sh"
    interval 3                      
    weight 5                        
    }

vrrp_instance VI_1 {
    # 使用单播通信,默认是组播通信
    unicast_src_ip  192.168.30.1
    unicast_peer {
                    192.168.30.2
    }
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100          
    advert_int 1
    mcast_src_ip 192.168.30.1

    authentication {
        auth_type PASS
        auth_pass 1111
    }

    virtual_ipaddress {        
        192.168.30.10 dev eth0
    }

    track_script {
        check_k8s
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault  "/etc/keepalived/notify.sh fault"
}

第五步：写监控和发邮件shell脚本
$ cat /etc/keepalived/chk_k8s_master.sh
#!/bin/bash
api=`ps aux | grep -v grep | grep '/opt/bin/kube-apiserver' | wc -l`
if [ $api = 1 ]; then
    con=`ps -ef | grep -v grep | grep '/opt/bin/kube-controller-manager' | wc -l`
    if [ $con = 1 ]; then
        sch=`ps -ef | grep -v grep | grep '/opt/bin/kube-scheduler' | wc -l`
        if [ $sch = 1 ]; then
            exit 0
        else
            exit 1
        fi
        exit 0
    else
        exit 1
    fi
    exit 0
else
    exit 1
fi

$ cat /etc/keepalived/notify.sh
#!/bin/bash
# description: An example of notify script

vip=192.168.30.10
contact='389736180@qq.com'


notify() {
    mailsubject="`hostname` to be $1: $vip floating"
    mailbody="`date '+%F %H:%M:%S'`: vrrp transition, `hostname` changed to be $1"
    echo $mailbody | mail -s "$mailsubject" $contact
}

case "$1" in
    master)
        notify master
        exit 0
    ;;
    backup)
        notify backup
        exit 0
    ;;
    fault)
        notify fault
        exit 0
    ;;
    *)
        echo 'Usage: `basename $0` {master|backup|fault}'
        exit 1
    ;;
esac

第六步：
把keepalived日志分离出来，默认日志在 /var/log/messages 文件中
$ mkdir /var/log/keepalived/
编辑/etc/sysconfig/keepalived：
KEEPALIVED_OPTIONS="-D -S 0"
编辑/etc/rsyslog.conf：
# 添加此行
local0.*                                        /var/log/keepalived/keepalived.log
$ mkidr -p /var/log/keepalived/
重启rsyslog
重启keepalived
按上面配置后，keepalived会把日志记录到/var/log/keepalived/keepalived.log，默认是记录在/var/log/message

第七步：
设置启动和开机启动
$ /etc/init.d/keepalived start
$ chkconfig keepalived on

第八步：
查看keepalived vip地址(vip地址使用ifconfig -a 命令是查看不到，而要使用ip addr命令才能看到)
$ ip addr
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1454 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:73:ec:6e brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.1/24 brd 10.10.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.30.10/32 scope global eth0
<--------------------------- kubernetes1节点安装 keepalived end--------------------------->


<--------------------------- kubernetes2节点安装 keepalived start--------------------------->
安装除 第四步：修改配置文件 不一样，其它步骤和kubernetes1步骤一样

第四步：修改配置文件
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     389736180@qq.com
   }

   notification_email_from root@kubernetes2.yp14.cn
   smtp_server 192.168.30.2
   smtp_connect_timeout 30
   router_id Master2
}

vrrp_script check_k8s {             
    script "/etc/keepalived/chk_k8s_master.sh"
    interval 3                      
    weight 5                        
    }

vrrp_instance VI_1 {
    # 使用单播通信,默认是组播通信
    unicast_src_ip  192.168.30.2
    unicast_peer {
                    192.168.30.1
    }
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 99          
    advert_int 1
    mcast_src_ip 192.168.30.2

    authentication {
        auth_type PASS
        auth_pass 1111
    }

    virtual_ipaddress {        
        192.168.30.10 dev eth0
    }

    track_script {
        check_k8s
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault  "/etc/keepalived/notify.sh fault"
}
<--------------------------- kubernetes2节点安装 keepalived end--------------------------->

九、服务启动顺序

1
2
3
4
5
6
7
8
9
10
1、启动 etcd集群服务，首先启动kubernetes1 etcd服务，在启动kubernetes2 etcde服务，最后启动docker etcd服务
2、启动 flannel和docker服务，并且测试各个主机的flannel网络是否通信
3、启动 Kubernetes master需要注意
{
  各个master节点的apiserver同时启动，指向相同的etcd cluster
  各个master 节点的controller-manager 和 scheduler  同时启动
  所有的node节点通过keepalive的虚拟VIP连接master
}
4、启动keepalived服务，首先启动kubernetes1 keepalived服务，然后启动kubernetes2 keepalived服务
5、启动各个主机上的Kubernetes node服务,这个服务不分先后，随便那台机器先启动都行


---本文结束感谢您的阅读。微信扫描二维码，关注我的公众号---






本文作者： Peng Yang
本文链接： https://www.yp14.cn/2016/11/28/kubernetes-keepalived高可用部署/
版权声明： 本作品采用 知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议 进行许可。转载请注明出处！
知识共享许可协议
赏

谢谢您的打赏

微信

扫一扫，分享到微信

微信分享二维码

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true