KubeEdge 初测

本文在已经成功部署了 KubeEdge 的集群中进行实测。目的是了解 KubeEdge 与 k8s 的异同。本文针对1.2版本。

一些说明

因为 KubeEdge 在 edgecore 上实现了 kubelet 部分功能,所以理论上是无缝接合的。
本文使用统一的镜像registry.cn-hangzhou.aliyuncs.com/latelee/webgin,该镜像的功能是提供 web 服务,返回运行时的 CPU、OS 和主机名称。笔者利用 docker manifest,可根据不同 CPU 拉取不同镜像,所以在 yaml 文件中统一使用同一名称,可自动匹配不同平台。

webgin镜像有几个版本:v1.0 v1.1 v1.2。

在主节点查看集群:

1
2
3
4
5
6
# kubectl get node
NAME STATUS ROLES AGE VERSION
edge-node Ready edge 9d v1.17.1-kubeedge-v1.2.1-dirty
edge-node2 Ready k8snode 105m v1.17.0
latelee.org.ttucon-2142ec Ready edge 9d v1.17.1-kubeedge-v1.2.1-dirty
ubuntu Ready master 9d v1.17.4

其中 edge-node2 为 k8s,版本为 v1.17.0,edge-node 和 latelee.org.ttucon-2142ec 为 KubeEdge 边缘端,后者是 arm 板子系统主机。

在部署 KubeEdge 时已经创建了 crds 了,查看之(本文中作用不大):

1
2
3
4
5
6
# kubectl get crds
NAME CREATED AT
clusterobjectsyncs.reliablesyncs.kubeedge.io 2020-03-17T06:45:08Z
devicemodels.devices.kubeedge.io 2020-03-17T06:44:50Z
devices.devices.kubeedge.io 2020-03-17T06:44:55Z
objectsyncs.reliablesyncs.kubeedge.io 2020-03-17T06:45:08Z

测试

测试 yaml 文件 webgin-service.yaml 如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: webgin-deployment
labels:
app: webgin
spec:
replicas: 3 # tells deployment to run 3 pods matching the template
selector:
matchLabels:
app: webgin
template:
metadata:
labels:
app: webgin
spec:
containers:
- name: webgin
image: registry.cn-hangzhou.aliyuncs.com/latelee/webgin:v1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
volumeMounts:
- mountPath: /etc/localtime
name: time-zone
volumes:
- name: time-zone
hostPath:
path: /etc/localtime
hostNetwork: true
---

apiVersion: v1
kind: Service # 指定为service
metadata:
labels:
run: webgin
name: webgin
namespace: default
spec:
ports:
- port: 10080 # 对外为88端口
targetPort: 80
selector:
app: webgin
type: LoadBalancer

释义:deployment + service 组合,副本数为3(因为有3台节点机器),hostNetwork 模式,挂载日期文件是为了输出真实时间。

在主节点创建 deployment:

1
kubectl apply -f webgin-service.yaml

查看 pod:

1
2
3
4
5
# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
webgin-deployment-57cbd68f7f-f8hq4 1/1 Running 0 5m2s 192.168.0.153 edge-node2 <none> <none>
webgin-deployment-57cbd68f7f-ktmfq 1/1 Running 0 5m2s 192.168.0.220 latelee.org.ttucon-2142ec <none> <none>
webgin-deployment-57cbd68f7f-xt7hf 1/1 Running 0 5m2s <none> edge-node <none> <none>

三个节点均已正常运行。但是最后一台的IP没有看到,下面测试获取信息时,没有看到该节点。

查看服务:

1
2
3
4
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
webgin LoadBalancer 10.97.98.218 <pending> 88:31059/TCP 12m

访问web服务:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
root@ubuntu:mytest# curl 10.97.98.218:88
Hello World v1.1
arch: arm os: linux hostname: latelee.org.ttucon-2142ec
uname: Linux latelee.org.ttucon-2142ec 4.14.67 #7 SMP PREEMPT Fri Feb 28 11:35:26 CST 2020 armv7l
Now: 2020-03-27 13:13:10

root@ubuntu:mytest#
root@ubuntu:mytest# curl 10.97.98.218:88
Hello World v1.1
arch: amd64 os: linux hostname: edge-node2
uname: Linux edge-node2 4.4.0-174-generic #204-Ubuntu SMP Wed Jan 29 06:41:01 UTC 2020 x86_64
Now: 2020-03-27 13:13:21

root@ubuntu:mytest#
root@ubuntu:mytest# curl 10.97.98.218:88
Hello World v1.1
arch: arm os: linux hostname: latelee.org.ttucon-2142ec
uname: Linux latelee.org.ttucon-2142ec 4.14.67 #7 SMP PREEMPT Fri Feb 28 11:35:26 CST 2020 armv7l
Now: 2020-03-27 13:13:29

结果:三台节点,在查看时,可以看到其中2台(平台不同),不同时候访问,其结果不同,理论上三台。

其它测试

对比kubectl describe命令。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
k8s:
# kubectl describe pod webgin-deployment-57cbd68f7f-f8hq4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m3s default-scheduler Successfully assigned default/webgin-deployment-57cbd68f7f-f8hq4 to edge-node2
Normal Pulling 9m3s kubelet, edge-node2 Pulling image "registry.cn-hangzhou.aliyuncs.com/latelee/webgin:v1.1"
Normal Pulled 8m57s kubelet, edge-node2 Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/latelee/webgin:v1.1"
Normal Created 8m57s kubelet, edge-node2 Created container webgin
Normal Started 8m57s kubelet, edge-node2 Started container webgin

KubeEdge:
# kubectl describe pod webgin-deployment-57cbd68f7f-ktmfq

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m25s default-scheduler Successfully assigned default/webgin-deployment-57cbd68f7f-ktmfq to latelee.org.ttucon-2142ec

结果:k8s 的输出信息相对全面一些。

对比kubectl logs命令。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
k8s:
# kubectl logs webgin-deployment-57cbd68f7f-f8hq4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m25s default-scheduler Successfully assigned default/webgin-deployment-57cbd68f7f-ktmfq to latelee.org.ttucon-2142ec
root@ubuntu:mytest# kubectl describe pod webgin-deployment-57cbd68f7f-ktmfqkubectl logs webgin-deployment-57cbd68f7f-f8hq4^C
root@ubuntu:mytest# kubectl logs webgin-deployment-57cbd68f7f-f8hq4
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET / --> main.myIndex (3 handlers)
gin server start...
[GIN-debug] Listening and serving HTTP on :80
[GIN] 2020/03/27 - 13:13:21 | 200 | 171.926µs | 192.168.0.102 | GET "/"

KubeEdge:
# kubectl logs webgin-deployment-57cbd68f7f-ktmfq
Error from server: Get https://192.168.0.220:10250/containerLogs/default/webgin-deployment-57cbd68f7f-ktmfq/webgin: dial tcp 192.168.0.220:10250: connect: connection refuse

结果:KubeEdge 不支持该命令。

对比kubectl exec命令。

1
2
3
4
5
6
7
k8s:
# kubectl exec -it webgin-deployment-57cbd68f7f-f8hq4 -- uname -a
Linux edge-node2 4.4.0-174-generic #204-Ubuntu SMP Wed Jan 29 06:41:01 UTC 2020 x86_64 GNU/Linux

KubeEdge:
# kubectl exec -it webgin-deployment-57cbd68f7f-ktmfq -- uname -a
Error from server: error dialing backend: dial tcp 192.168.0.220:10250: connect: connection refused

结果:KubeEdge 不支持该命令。

针对 hostNetwork 模式,在arm节点上查看容器IP(提供主要内容):

1
2
3
4
5
6
# docker exec -it 71605a5e17a3 ifconfig
docker0 Link encap:Ethernet HWaddr 02:42:00:00:00:94
inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0

eth0 Link encap:Ethernet HWaddr 4C:00:00:00:00:EC
inet addr:192.168.0.220 Bcast:192.168.0.255 Mask:255.255.255.0

在arm节点上,没有产生如veth216ffbc7之类名称的网络设备。

升级

此处简述即可。
1、
修改 yaml 配置文件,修改镜像版本号,即将registry.cn-hangzhou.aliyuncs.com/latelee/webgin:v1.1改为registry.cn-hangzhou.aliyuncs.com/latelee/webgin:v1.2
再更新:kubectl apply -f webgin-service.yaml。测试几次,均失败。一个是Pending,一个是Running。

2、 命令行修改镜像名

1
kubectl set image deployment webgin-deployment webgin=registry.cn-hangzhou.aliyuncs.com/latelee/webgin:v1.2

测试几次,发现有时成功,有时失败。失败时一个是Pending,一个是Terminating。(注:在k8s节点可以成功)

kubectl rollout status deployment webgin-deployment
在spec添加:

1
2
3
4
5
6
7
minReadySeconds: 5
strategy:
# indicate which strategy we want for rolling update
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1

修改,再应用,可行。

查看版本:

1
kubectl rollout history deployment webgin-deployment

回滚到指定的第2次升级的版本(2是指k8s记录的第2次升级)
kubectl rollout undo deployment webgin-deployment –to-revision=2

指定的镜像不存在,再升级(即故意让其失败)

应用时添加kubectl apply -f webgin-service.yaml --record=true
会记录版本信息,但似乎无用。

1
2
3
4
5
6
# kubectl rollout history deployment webgin-deployment
deployment.apps/webgin-deployment
REVISION CHANGE-CAUSE
1 kubectl apply --filename=webgin-service.yaml --record=true
2 kubectl apply --filename=webgin-service.yaml --record=true
3 kubectl apply --filename=webgin-service.yaml --record=true

疑问:如果更方便维护版本,即知道1做了什么,2做了什么

结论

两者还是存在差异。如部分命令不支持,如通信不稳定。

本文所用镜像,真实存在,但可能会不定时更新。本文所述,仅为本人实际测试之现象,不具通用性。