KubeEdge led部署

本文对 KubeEdge 的 led 灯示例进行测试。

KubeEdge 官方示例文件仓库为 https://github.com/kubeedge/examples ,将其下载到$GOPATH/src/github.com/kubeedge/ 目录,本文所用目录为 led-raspberrypi 。
本文所有修改见仓库 https://github.com/latelee/kube-examples/tree/master/led-raspberrypi

下面按测试步骤描述。

技术小结

一般步骤:
1、编写程序,编译,根据需求适配不同平台。
2、制作镜像。可与编译合并到 Makefile 或脚本中。
3、提交镜像,因为镜像会调度到不同节点,所以要提交到dockerhub,但是实际中,可以通过docker命令拷贝镜像到节点机器。在调试阶段比较方便。(注:笔者经常用此法调试)
4、修改并创建设备模型。
5、使用 kubectl 部署,一般选用 deployment,有时需要配合其它组件,如ConfigMap。
6、修改状态,观察实际效果。查看状态,观察反馈效果。

源码说明

sample-crds:crds 配置,指定了调度的节点、GPIO号,LED默认状态,等等。下称设备模型。
configuration:配置相关,需要读取 deviceProfile.json (运行时生成)和 config.yaml 文件。
light_mapper.go:主程序文件,主要匹配 crds 设备模型,并与真实硬件交互。与硬件交互主要使用github.com/stianeikeland/go-rpio/包,由于笔者没硬件环境,故其操作 GPIO 的代码注释掉,仅作示例。

Dockerfile:生成 docker 镜像文件,笔者扩展了 arm 平台(注:笔者去掉了硬件操作,故代码可适用不同平台)。
deployment.yaml:deployment 配置文件,指定调度节点、镜像和 configMap (如果不指定,生成不了 json 文件)。

理论上,不同的硬件,其操作不同,因此需要不同的 crds 为匹配。这也是为什么 crds 中要指定节点的原因。不过,笔者认为,实践中,可能存在批量操作,即同一批硬件,其硬件相同,功能相同,因此使用的程序也相同,如温度采集等。此情况下,可以通过节点的 label 来匹配调度的节点。当然,这不是本文关注的重点。

编译

笔者修改了 Makefile,如下:

1
2
3
4
5
6
7
8
9
# make led_light_mapper
.PHONY: default led_light_mapper
led_light_mapper:
export GOARCH=amd64; export GOOS="linux"; export GOARM=""; export CGO_ENABLED=1; export CC=cc; \
go build light_mapper.go
docker build -t latelee/led-light-mapper-x86:v1.1 . -f Dockerfile
export GOARCH=arm; export GOOS="linux"; export GOARM=7; export CGO_ENABLED=1; export CC=arm-linux-gnueabihf-gcc; \
go build light_mapper.go
docker build -t latelee/led-light-mapper-arm:v1.1 . -f Dockerfile-arm

分别使用不同编译器编译,并修改 docker 镜像地址。接着合并镜像:

1
2
3
4
5
6
7
8
9
10
11
12

docker push latelee/led-light-mapper-x86:v1.1
docker push latelee/led-light-mapper-arm:v1.1

export DOCKER_CLI_EXPERIMENTAL=enabled

docker manifest create latelee/led-light-mapper:v1.1 latelee/led-light-mapper-x86:v1.1 latelee/led-light-mapper-arm:v1.1

docker manifest annotate latelee/led-light-mapper:v1.1 latelee/led-light-mapper-x86:v1.1 --os linux --arch x86_64
docker manifest annotate latelee/led-light-mapper:v1.1 latelee/led-light-mapper-arm:v1.1 --os linux --arch armv7l

docker manifest push latelee/led-light-mapper:v1.1

创建设备模型

kubeedge在部署时已经创建了crds了,此处查看:

1
2
3
4
5
6
# kubectl get crds
NAME CREATED AT
clusterobjectsyncs.reliablesyncs.kubeedge.io 2020-02-20T08:28:32Z
devicemodels.devices.kubeedge.io 2019-12-31T08:41:34Z
devices.devices.kubeedge.io 2019-12-31T08:41:34Z
objectsyncs.reliablesyncs.kubeedge.io 2020-02-20T08:28:32Z

再创建led的crds。

1
2
3
cd $GOPATH/src/github.com/kubeedge/examples/led-raspberrypi
cd sample-crds
vim led-light-device-instance.yaml

修改 led-light-device-instance.yaml 文件,将节点改为 latelee.org.ttucon-2142ec
创建:

1
2
3
4
# kubectl apply -f .
```

查看

kubectl get deviceModel

NAME AGE
led-light 28h

kubectl get device

NAME AGE
led-light-instance-01 28h

1
查看详情:

kubectl describe devices.devices.kubeedge.io led-light-instance-01

1
2
3

### 部署
修改 deployment.yaml,如下:

apiVersion: apps/v1
kind: Deployment
metadata:
name: led-light-mapper-deployment
spec:
replicas: 1
selector:
matchLabels:
app: led-light-mapper
template:
metadata:
labels:
app: led-light-mapper
spec:
nodeName: latelee.org.ttucon-2142ec #edge-node2
hostNetwork: true
containers:

- name: led-light-mapper-container
  image: latelee/led-light-mapper:v1.1
  imagePullPolicy: IfNotPresent
  securityContext:
    privileged: true
  volumeMounts:
  - name: config-volume
    mountPath: /opt/kubeedge/
volumes:
- name: config-volume
  configMap:
    name: device-profile-config-edge-node2
restartPolicy: Always
1
2

部署:

kubectl apply -f deployment.yaml

1
2
3
4
等待调度完成。  

### 测试
修改 LED 状态值,把OFF改为ON,:

vim led-light-device-instance.yaml

1
2
3
再更新配置。  

或者直接实时修改:

kubectl edit device led-light-instance-01

1
2
3
4
5
6

注:
如果此时修改`led-light-device-model.yaml`的引脚号,再更新,是不成功的。

### 检查
在节点机器上,执行`docker ps`查看容器。再进入容器查看json文件:

docker exec -it 4bc0f93a0174 cat /opt/kubeedge/deviceProfile.json

{“deviceInstances”:[{“id”:”led-light-instance-01”,”name”:”led-light-instance-01”,”model”:”led-light”}],”deviceModels”:[{“name”:”led-light”,”properties”:[{“name”:”power-status”,”dataType”:”string”,”description”:”Indicates whether the led light is ON/OFF”,”accessMode”:”ReadWrite”,”defaultValue”:”OFF”},{“name”:”gpio-pin-number”,”dataType”:”int”,”description”:”Indicates whether the GPIO pin to which LED is connected”,”accessMode”:”ReadOnly”,”defaultValue”:18}]}],”protocols”:[{“protocol_config”:null}]}

1
2

查看日志(加-f):

docker logs -f 4bc0f93a0174
I0318 03:54:21.558189 1 light_mapper.go:242] Watching on the device twin values for device: led-light-instance-01
I0318 03:54:22.559353 1 light_mapper.go:272] Actual values are in sync with Expected value
I0318 03:54:22.559374 1 light_mapper.go:242] Watching on the device twin values for device: led-light-instance-01
I0318 03:54:23.560669 1 light_mapper.go:272] Actual values are in sync with Expected value
I0318 03:54:23.560695 1 light_mapper.go:242] Watching on the device twin values for device: led-light-instance-01
I0318 03:54:24.561883 1 light_mapper.go:248] Expected Value : ON
I0318 03:54:24.561909 1 light_mapper.go:252] Actual Value: OFF
I0318 03:54:24.561913 1 light_mapper.go:254] Equating the actual value to expected value
I0318 03:54:24.561918 1 light_mapper.go:257] Turning ON the light
I0318 03:54:24.561922 1 light_driver.go:11] TurnON pin: 18
I0318 03:54:24.562033 1 light_mapper.go:242] Watching on the device twin values for device: led-light-instance-01
I0318 03:54:25.563141 1 light_mapper.go:248] Expected Value : ON
I0318 03:54:25.563164 1 light_mapper.go:252] Actual Value: OFF
I0318 03:54:25.563168 1 light_mapper.go:254] Equating the actual value to expected value
I0318 03:54:25.563172 1 light_mapper.go:257] Turning ON the light
I0318 03:54:25.563195 1 light_driver.go:11] TurnON pin: 18
I0318 03:54:25.563281 1 light_mapper.go:242] Watching on the device twin values for device: led-light-instance-01

1
2
3
从日志中看到 GPIO 引脚的电平变化了。  

在云端查看状态:

kubectl get device led-light-instance-01 -oyaml -w

1
查看configmap详情:

kubectl get cm device-profile-config-edge-node2 -oyaml

1
2
### 排错
如果不使用 KubeEdge 部署的话,容器报错:

Error while reading from config map Error while reading from config map open /opt/kubeedge/deviceProfile.json: no such file or directory

1
2
3
4
5
有时候即使用 KubeEdge 部署,也会报相同的错误,原因未知。  
记:查看configmap,没有创建。疑惑:只部署了deployment,里面指定configmap名称而已,何时由谁创建configmap的?
与其中一位项目开发者进行邮件和github交流,得知在创建device时,kubeedge会自动创建configmap的。可能哪里出错,创建不成功。网上也有一些人遇到相同的问题。

非正当途径解决:根据上述json内容,创建名为deviceProfile.json的文件(一定是此名称,因为led程序代码使用这个文件名称),拷贝之。注意,json需用字符串形式,不能格式化。

kubectl create configmap led-config –from-file=deviceProfile.json

1
2
3
4
5
6
7
之后用` kubectl get cm -oyaml`查看。  

测试发现,导出的docker镜像,过一段时间会消失,此时,会卡在 ContainerCreating 阶段。


似乎自动创建的cm,同一节点,名称是一样的。
测试,先创建led设备模型,再创建temp设备模式,其结果如下:

kubectl describe cm device-profile-config-latelee1

Name: device-profile-config-latelee1
Namespace: default
Labels:
Annotations:

Data

deviceProfile.json:

{“deviceInstances”:[{“id”:”led-light-instance-01”,”name”:”led-light-instance-01”,”model”:”led-light”},{“id”:”temperature1”,”name”:”temperature1”,”model”:”temperature-model”}],”deviceModels”:[{“name”:”led-light”,”properties”:[{“name”:”power-status”,”dataType”:”string”,”description”:”Indicates whether the led light is ON/OFF”,”accessMode”:”ReadWrite”,”defaultValue”:”OFF”},{“name”:”gpio-pin-number”,”dataType”:”int”,”description”:”Indicates whether the GPIO pin to which LED is connected”,”accessMode”:”ReadOnly”,”defaultValue”:168}]},{“name”:”temperature-model”,”properties”:[{“name”:”temperature-status”,”dataType”:”string”,”description”:”Temperature collected from the edge device”,”accessMode”:”ReadOnly”,”defaultValue”:””}]}],”protocols”:[{“protocol_config”:null},{“protocol_config”:null}]}
Events:

1
2


docker save -o led.docker latelee/led-light-mapper-x86:v1.1

docker load -i led.docker

kubectl delete -f deployment.yaml

kubectl apply -f deployment.yaml

docker save -o temp.gdocker latelee/temp-mapper-x86

docker load -i temp.docker

kubectl delete pod –all –force –grace-period=0

kubectl get device temperature -oyaml -w

kubectl create configmap led-config –from-file=device.json

1
2

容器打印:

I0419 03:05:04.308734 1 led_demo.go:146] read configmap ok
I0419 03:05:04.308818 1 led_demo.go:148] device: configuration.DeviceInstance{ID:”led-demo-instance-01”, Name:”led-demo-instance-01”, Protocol:””, Model:”led-demo”}
I0419 03:05:04.308858 1 led_demo.go:151] device id: led-demo-instance-01
I0419 03:05:04.308871 1 led_demo.go:170] Changing the state of the device to online 111
I0419 03:05:35.354202 1 led_demo.go:250] Expected Value : ON
I0419 03:05:35.354246 1 led_demo.go:254] Actual Value: OFF
I0419 03:05:35.354278 1 led_demo.go:256] Equating the actual value to expected value
I0419 03:05:35.354297 1 led_demo.go:259] Turning ON the light
I0419 03:05:36.356098 1 led_demo.go:250] Expected Value : ON
I0419 03:05:36.356130 1 led_demo.go:254] Actual Value: OFF
I0419 03:05:36.356141 1 led_demo.go:256] Equating the actual value to expected value
I0419 03:05:36.356154 1 led_demo.go:259] Turning ON the light
`