kubeedge-counter-demo（二） 2020-12-04 实验 暂无评论 2466 次阅读 先说这是什么事。[上次的例程](https://www.proup.club/index.php/archives/657/)部署完了一部分（网页），这回部署另一部分（计数器）。 刚开始还好，计数器和网页都部署完了。 但发现网页控制不了计数器，就开始排错，排错过程中把环境弄坏了，修了两天，才终于绕回原地，一开始的错误到最后也没修好。 [TOC] ## 从开机说起 **cloud：** 1. `tmux` 进入多窗口 2. `cloudcore` 3. `Ctrl+b`，`c` 创建新窗口 **edge：** 1. `tmux` 2. `edgecore` 3. `Ctrl+b`，`c` **检查是否正常：** 在cloud输入`kubectl get nodes`，如果kubeedge的状态是Ready，就是正常的。 ## 编译边端镜像 原版例程，架构是这样的： Cloud：电脑 → 网页 Edge：树莓派 → 计数器 而我目前的是这样的： Cloud：虚拟机 → 网页 Edge：虚拟机 → 计数器 打算暂时先弄俩虚拟机的，树莓派过两周再弄。现在要把边端的交叉编译选项去掉。 **以下在cloud（192.168.56.103）操作：** 1. ``` cd $GOPATH/src/github.com/kubeedge/examples/kubeedge-counter-demo/counter-mapper vim Makefile ``` 2. 然后把`GOARCH=arm64 go build -o pi-counter-app main.go`前面的`GOARCH=arm64 `去掉，保存。 3. `make` 4. `make docker` > ``` > root@cloud:.../counter-mapper# make docker > docker build . -t kubeedge/kubeedge-pi-counter:v1.0.0 > Sending build context to Docker daemon 7.842MB > Step 1/6 : FROM ubuntu:18.04 > ---> 2c047404e52d > Step 2/6 : LABEL maintainers="KubeEdge Community Developer" > ---> Using cache > ---> 2b12a9bcd11a > Step 3/6 : LABEL description="KubeEdge Raspi Counter App" > ---> Using cache > ---> e8cf16fe535f > Step 4/6 : COPY pi-counter-app /pi-counter-app > ---> c23db0b0c3da > Step 5/6 : ENTRYPOINT ["/pi-counter-app"] > ---> Running in 1b5a42a4f9f1 > Removing intermediate container 1b5a42a4f9f1 > ---> 79c5e9a13935 > Step 6/6 : CMD ["pi-counter-app"] > ---> Running in 090272e8fd18 > Removing intermediate container 090272e8fd18 > ---> 41d033766fcc > Successfully built 41d033766fcc > Successfully tagged kubeedge/kubeedge-pi-counter:v1.0.0 > ``` > > 看一眼镜像： > > ``` >root@cloud:.../counter-mapper# docker images > REPOSITORY TAG IMAGE ID CREATED SIZE > kubeedge/kubeedge-pi-counter v1.0.0 41d033766fcc About a minute ago 71.1MB > kubeedge/kubeedge-counter-app v1.0.0 1d5cbf7e4f13 17 hours ago 90.4MB > pro1515151515/kubeedge-counter-app v1.0.0 1d5cbf7e4f13 17 hours ago 90.4MB > ubuntu 18.04 2c047404e52d 5 days ago 63.3MB > kindest/node v1.19.1 37ddbc9063d2 2 months ago 1.33GB > centos 7.6.1810 f1cb7c7d58b7 20 months ago 202MB > ``` 5. ``` docker tag 41d033766fcc pro1515151515/kubeedge-pi-counter:v1.0.0 docker login --username=pro1515151515 registry.cn-hangzhou.aliyuncs.com docker push pro1515151515/kubeedge-pi-counter:v1.0.0 ``` > ``` > root@cloud:.../counter-mapper# docker push pro1515151515/kubeedge-pi-counter:v1.0.0 > The push refers to repository [docker.io/pro1515151515/kubeedge-pi-counter] > 6421442f68f2: Pushed fe6d8881187d: Mounted from pro1515151515/kubeedge-counter-app > ``` > ``` > 23135df75b44: Mounted from pro1515151515/kubeedge-counter-app > b43408d5f11b: Mounted from pro1515151515/kubeedge-counter-app >v1.0.0: digest: sha256:942a3fcaba89ac7368f2169481b6ec6419474bd45a8f31b54da8765abaaf5c40 size: 1154 > ``` ## 修改crd文件 ### 改设备实例的crd ``` cd $GOPATH/src/github.com/kubeedge/examples/kubeedge-counter-demo/crds vim kubeedge-counter-instance.yaml ``` 将其中的`raspberrypi`改成`kubeedge` ### 改计数器程序的crd ``` cd $GOPATH/src/github.com/kubeedge/examples/kubeedge-counter-demo/crds vim kubeedge-pi-counter-app.yaml ``` 将其中的`image: kubeedge/kubeedge-pi-counter:v1.0.0`替换成`image: pro1515151515/kubeedge-pi-counter:v1.0.0` ## 部署计数器 ``` cd $GOPATH/src/github.com/kubeedge/examples/kubeedge-counter-demo/crds kubectl create -f kubeedge-counter-model.yaml kubectl create -f kubeedge-counter-instance.yaml kubectl create -f kubeedge-pi-counter-app.yaml ``` > ``` > root@cloud:/home/pro/go/src/github.com/kubeedge/examples/kubeedge-counter-demo/crds# kubectl get deployment > NAME READY UP-TO-DATE AVAILABLE AGE > kubeedge-counter-app 1/1 1 1 12h > kubeedge-pi-counter 0/1 1 0 12s > root@cloud:/home/pro/go/src/github.com/kubeedge/examples/kubeedge-counter-demo/crds# kubectl get pods > NAME READY STATUS RESTARTS AGE > kubeedge-counter-app-fc6d947d8-m4459 1/1 Running 1 12h > kubeedge-pi-counter-85b7c977b-s55nn 0/1 ContainerCreating 0 24s > ``` ### 看部署结果 #### 在Cloud上看 ``` kubectl get pods ``` > ``` > root@cloud:.../crds# kubectl get pods > NAME READY STATUS RESTARTS AGE > kubeedge-counter-app-fc6d947d8-m4459 1/1 Running 1 12h > kubeedge-pi-counter-85b7c977b-s55nn 1/1 Running 0 78s > ``` 是Running就是部署好了。按理说kubeedge-pi-counter这个Pod应该是实际部署在Edge的容器里。 #### 在Edge上看 **以下在edge（192.168.56.101）操作：** ``` docker ps ``` > ``` > root@kubeedge:~# docker ps > CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES > 62850924fd09 pro1515151515/kubeedge-pi-counter "/pi-counter-app pi-…" 5 seconds ago Up 4 seconds k8s_kubeedge-pi-counter_kubeedge-pi-counter-85b7c977b-s55nn_default_bbf3d99f-69e2-4650-8ef1-f8dcf22f6233_0 > 077289d0768b kubeedge/pause:3.1 "/pause" About a minute ago Up About a minute k8s_POD_kubeedge-pi-counter-85b7c977b-s55nn_default_bbf3d99f-69e2-4650-8ef1-f8dcf22f6233_0 > 4656e52b5f6a kubeedge/pause:3.1 "/pause" 37 minutes ago Up 37 minutes k8s_POD_kube-proxy-vmbkz_kube-system_afcf9ccf-a18d-467d-9e06-03a3ce2f8354_2 > 5172e7776034 kubeedge/pause:3.1 "/pause" 37 minutes ago Up 37 minutes k8s_POD_kindnet-m8hxp_kube-system_8a6fb1b1-2cda-4e4f-806c-62b092e9393a_2 > ``` 可以看到`pro1515151515/kubeedge-pi-counter`的状态是`Up 4 seconds`，就是在4秒前就启动好了。 ### 从网页上看 内网穿透功能是每次重启都要运行一次。 **以下在cloud（192.168.56.103）操作：** ``` ssh -Nfg -L 0.0.0.0:80:172.18.0.2:80 192.168.56.103 ``` 。。。。现在情况是： 网页能打开，但按钮点了没用。 ## 重新编译部署网页Pod ### 改源码 要改的只有一处，就是k8s主节点的端口号。 备选的有： 127.0.0.1:10000 → WebSocket （v1.0.1） 127.0.0.1:10001 → quic 127.0.0.1:10002 → https 127.0.0.1:10003 → cloudStream_stream（没用到） 127.0.0.1:10004 → cloudStream_tunnel（没用到） 127.0.0.1:44587 → kind集群（v1.0.2） 192.168.56.103:10000 （v1.0.3） https://127.0.0.1:6443 （v1.0.4） http://192.168.56.103:8080 （v1.0.5） config.Insecure=true（v1.0.6） ``` cd $GOPATH/src/github.com/kubeedge/examples/kubeedge-counter-demo/web-controller-app/utils vim kubeclient.go ``` 把`var KubeMaster = "http://127.0.0.1:8080"`的8080改成10000，保存退出。 ### 编译上传镜像 ``` cd $GOPATH/src/github.com/kubeedge/examples/kubeedge-counter-demo/web-controller-app make docker build . -t pro1515151515/kubeedge-counter-app:v1.0.1 docker push pro1515151515/kubeedge-counter-app:v1.0.1 ``` ### 重新部署 #### 删除旧部署 ``` kubectl delete deployment kubeedge-counter-app ``` #### 更新crd文件 ``` cd $GOPATH/src/github.com/kubeedge/examples/kubeedge-counter-demo/crds vim kubeedge-web-controller-app.yaml ``` 将`image: pro1515151515/kubeedge-counter-app:v1.0.0`中的`v1.0.0`改成`v1.0.1` #### 部署 ``` kubectl create -f kubeedge-web-controller-app.yaml ``` #### 查看结果 ``` kubectl get pods kubectl describe pod kubeedge-counter-app ``` > ``` > root@cloud:.../crds# kubectl describe pod kubeedge-counter-app > Name: kubeedge-counter-app-7d477c8757-84529 > Namespace: default > Priority: 0 > Node: kind-control-plane/172.18.0.2 > Start Time: Tue, 01 Dec 2020 02:57:52 +0000 > Labels: k8s-app=kubeedge-counter-app > pod-template-hash=7d477c8757 > Annotations: <none> > Status: Running > IP: 172.18.0.2 > IPs: > IP: 172.18.0.2 > Controlled By: ReplicaSet/kubeedge-counter-app-7d477c8757 > Containers: > kubeedge-counter-app: > Container ID: containerd://2906d4f52582fae7395013255f050a66a1ac85eb1823e201d227f448f587e51c > Image: pro1515151515/kubeedge-counter-app:v1.0.1 > Image ID: docker.io/pro1515151515/kubeedge-counter-app@sha256:27033fc3cbc9475556d3b53014ab5ec0bdba5c7adc06acf382238619bf7a92e5 > Port: <none> > Host Port: <none> > State: Running > Started: Tue, 01 Dec 2020 02:59:15 +0000 > Ready: True > Restart Count: 0 > Environment: <none> > Mounts: > /var/run/secrets/kubernetes.io/serviceaccount from default-token-xnv4t (ro) > Conditions: > Type Status > Initialized True > Ready True > ContainersReady True > PodScheduled True > Volumes: > default-token-xnv4t: Type: Secret (a volume populated by a Secret) > SecretName: default-token-xnv4t > Optional: false > QoS Class: BestEffort > Node-Selectors: node-role.kubernetes.io/master= > Tolerations: node-role.kubernetes.io/master:NoSchedule > node.kubernetes.io/not-ready:NoExecute for 300s > node.kubernetes.io/unreachable:NoExecute for 300s > Events: > Type Reason Age From Message > ---- ------ ---- ---- ------- > Normal Scheduled 2m23s default-scheduler Successfully assigned default/kubeedge-counter-app-7d477c8757-84529 to kind-control-plane > Normal Pulling 2m22s kubelet Pulling image "pro1515151515/kubeedge-counter-app:v1.0.1" > Normal Pulled 61s kubelet Successfully pulled image "pro1515151515/kubeedge-counter-app:v1.0.1" in 1m21.533551874s > Normal Created 60s kubelet Created container kubeedge-counter-app > Normal Started 60s kubelet Started container kubeedge-counter-app > ``` 。。这个还是不行。还是不能用网页打开计数器。 ### 找K8s master IP **以下在cloud（192.168.56.103）操作：** #### 进入kind容器 ``` docker exec -it kind-control-plane /bin/bash ``` 要用net-tools看端口情况。但这个容器里面没有net-tools， #### 安装net-tools ##### 看系统版本 ``` cat /etc/issue ``` > ``` > root@kind-control-plane:/# cat /etc/issue > Ubuntu Groovy Gorilla (development branch) \n \l > ``` > > 到`[ubuntu | 镜像站使用帮助 | 清华大学开源软件镜像站 | Tsinghua Open Source Mirror](https://mirror.tuna.tsinghua.edu.cn/help/ubuntu/) ，选版本20.10。 ##### 改软件源，安装net-tools ``` mv /etc/apt/sources.list /etc/apt/sources.list.bak echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ groovy main restricted universe multiverse" >/etc/apt/sources.list echo "deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ groovy-updates main restricted universe multiverse" >>/etc/apt/sources.list echo "deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ groovy-backports main restricted universe multiverse" >>/etc/apt/sources.list echo "deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ groovy-security main restricted universe multiverse" >>/etc/apt/sources.list apt-get update apt-get install -y net-tools vim ``` #### 看端口情况 ``` netstat -anp ``` >挑了几个开放的端口。 > >``` >tcp6 0 0 :::6443 :::* LISTEN 502/kube-apiserver >tcp6 0 0 :::80 :::* LISTEN 16785/kubeedge-coun >tcp6 0 0 :::10256 :::* LISTEN 892/kube-proxy >tcp6 0 0 :::10250 :::* LISTEN 136/kubelet >``` #### 排错 弄https://127.0.0.1:6443 （v1.0.4），网页按钮还是启动不了计数器。 > ``` > root@cloud:.../crds# kubectl get pods > NAME READY STATUS RESTARTS AGE > kubeedge-counter-app-5689c99c44-7wdq5 1/1 Running 0 2m9s > kubeedge-pi-counter-85b7c977b-s55nn 1/1 Running 0 150m > ``` > > ``` kubectl logs -f kubeedge-counter-app-576dc9fdc8-2jfhv ``` 错误整理： - 127.0.0.1:10000 → WebSocket （v1.0.1） - ``` 2020/12/01 04:49:08 Failed to patch device status &{{[{status {ON map[timestamp:1606 type:string]} {0 map[timestamp:1606 type:string]}}]}} of device counter in namespace default error:Patch "http://127.0.0.1:10000/apis/devices.kubeedge.io/v1alpha1/namespaces/default/devices/counter": dial tcp 127.0.0.1:10000: connect: connection refused ``` - 127.0.0.1:44587 → kind集群（v1.0.2） - ``` 2020/12/01 04:51:29 Failed to patch device status &{{[{status {ON map[timestamp:1606 type:string]} {0 map[timestamp:1606 type:string]}}]}} of device counter in namespace default error:Patch "http://127.0.0.1:44587/apis/devices.kubeedge.io/v1alpha1/namespaces/default/devices/counter": dial tcp 127.0.0.1:44587: connect: connection refused ``` - 192.168.56.103:10000 （v1.0.3） - ``` 2020/12/01 04:54:04 Failed to patch device status &{{[{status {ON map[timestamp:1606 type:string]} {0 map[timestamp:1606 type:string]}}]}} of device counter in namespace default error:Patch "http://192.168.56.103:10000/apis/devices.kubeedge.io/v1alpha1/namespaces/default/devices/counter": EOF ``` - https://127.0.0.1:6443 （v1.0.4） - ``` 2020/12/01 04:37:08 Failed to patch device status &{{[{status {ON map[timestamp:1606 type:string]} {0 map[timestamp:1606 type:string]}}]}} of device counter in namespace default error:Patch "https://127.0.0.1:6443/apis/devices.kubeedge.io/v1alpha1/namespaces/default/devices/counter": x509: certificate signed by unknown authority ``` - 127.0.0.1:10000 → WebSocket （v1.0.1） 127.0.0.1:10001 → quic 127.0.0.1:10002 → https 127.0.0.1:10003 → cloudStream_stream（没用到） 127.0.0.1:10004 → cloudStream_tunnel（没用到） 127.0.0.1:44587 → kind集群（v1.0.2） 192.168.56.103:10000 （v1.0.3） https://127.0.0.1:6443 （v1.0.4） ``` kubectl delete deployment kubeedge-counter-app cd $GOPATH/src/github.com/kubeedge/examples/kubeedge-counter-demo/crds vim kubeedge-web-controller-app.yaml kubectl create -f kubeedge-web-controller-app.yaml kubectl get pods kubectl logs -f kubeedge-counter-app-5689c99c44-qswdl ``` #### 安装证书 ##### 进入Docker容器 ``` docker exec -it kind-control-plane /bin/bash ``` ##### 安装证书 ``` apt-get -qq update apt-get -qq install -y --no-install-recommends ca-certificates curl ``` ##### 重启Docker容器 ``` exit docker restart kind-control-plane ``` #### 继续测试 容器重启以后，需要重启一下cloudcore。 。。。。还是不行。 #### 梳理问题 k8s的master节点确定是安装在`kind-control-plane`这个部署在cloud的Docker容器里了。 最终需要的api-server的IP端口，就是172.18.0.2:6443，是一个kind集群内网里的一个https服务。 现在的问题是，这个https服务没有证书。 两个思路：一是弄一个证书，二是把https改成http，就不需要证书了。 弄证书的话需要域名、外网ip。现在只有内网ip，不好弄证书。所以下面是想办法把api-server改成http服务。 **以下在cloud（192.168.56.103）操作：** 进入kind-control-plane容器： ```' docker exec -it kind-control-plane /bin/bash ``` 修改api-server的配置文件： ``` cd /etc/kubernetes/manifests vim kube-apiserver.yaml ``` 1. 将里面的`- --insecure-port=0`的`0`改成`8080`。 2. 添加`--insecure-bind-address=0.0.0.0` 重启： ``` exit docker restart kind-control-plane ``` > 这个时候已经可以用: curl 172.18.0.2:8080 （HTTP）来访问了。 > 但无法访问到192.168.56.103:8080，需要再加个端口映射。 加8080的端口映射： ``` ssh -Nfg -L 0.0.0.0:8080:172.18.0.2:8080 192.168.56.103 ``` 最后再修改一遍网页Pod，用`http://192.168.56.103:8080`应该就行了。 > ``` > 2020/12/01 07:03:28 Failed to patch device status &{{[{status {ON map[timestamp:1606 type:string]} {0 map[timestamp:1606 type:string]}}]}} of device counter in namespace default > error:the server could not find the requested resource (patch devices.devices.kubeedge.io counter) > ``` 不是edge端MQTT的问题。 >edge： > >``` >root@kubeedge:/# netstat -anp|grep mos >tcp 0 0 0.0.0.0:1883 0.0.0.0:* LISTEN 598/mosquitto >tcp 0 0 127.0.0.1:1883 127.0.0.1:44544 ESTABLISHED 598/mosquitto >tcp 0 0 127.0.0.1:1883 127.0.0.1:44666 ESTABLISHED 598/mosquitto >tcp 0 0 127.0.0.1:1883 127.0.0.1:44668 ESTABLISHED 598/mosquitto >tcp6 0 0 :::1883 :::* LISTEN 598/mosquitto >unix 3 [ ] STREAM CONNECTED 21256 598/mosquitto >``` > > 也不是cloud端的device配置的问题。 > ``` > kubectl describe device counter > ``` > > ``` > root@cloud:~# kubectl describe device counter > Name: counter > Namespace: default > Labels: description=counter > manufacturer=test > Annotations: <none> > API Version: devices.kubeedge.io/v1alpha2 > Kind: Device > Metadata: > Creation Timestamp: 2020-12-01T02:07:58Z > Generation: 1 > Managed Fields: > API Version: devices.kubeedge.io/v1alpha2 > Fields Type: FieldsV1 > ... > ``` 也不是kind容器那边没有设备信息 ``` docker exec -it kind-control-plane /bin/bash kubectl describe device counter ``` > ``` > root@cloud:~# docker exec -it kind-control-plane /bin/bash > root@kind-control-plane:/# kubectl describe device counter > Name: counter > Namespace: default > Labels: description=counter > manufacturer=test > Annotations: <none> > API Version: devices.kubeedge.io/v1alpha2 > Kind: Device > Metadata: > Creation Timestamp: 2020-12-01T02:07:58Z > Generation: 1 > Managed Fields: > API Version: devices.kubeedge.io/v1alpha2 > Fields Type: FieldsV1 > ... > ``` 打算重启看看。 ## 重启 **以下在cloud（192.168.56.103）操作：** 1. tmux 2. cloudcore 3. `ctrl+b` ，`c` 4. 逐行输入： ``` ssh -Nfg -L 0.0.0.0:80:172.18.0.2:80 192.168.56.103 ssh -Nfg -L 0.0.0.0:8080:172.18.0.2:8080 192.168.56.103 ``` 5. kubectl get pods > ``` > root@cloud:~# kubectl get pods > NAME READY STATUS RESTARTS AGE > kubeedge-counter-app-6b7c8755c-4vqqp 1/1 Running 1 31m > kubeedge-pi-counter-85b7c977b-s55nn 1/1 Running 0 5h25m > ``` 好像有点奇怪，kubeedge-counter-app这个Pod是部署在哪的。。 > Edge: > > ``` > root@kubeedge:~# docker ps > CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS > NAMES > ``` 不在Edge上。看一下它的详情： ``` kubectl describe pod kubeedge-counter-app ``` > > > ``` > root@cloud:~# kubectl describe pod kubeedge-counter-app > Name: kubeedge-counter-app-6b7c8755c-4vqqp > Namespace: default > Priority: 0 > Node: kind-control-plane/172.18.0.2 > Start Time: Tue, 01 Dec 2020 07:02:41 +0000 > Labels: k8s-app=kubeedge-counter-app > pod-template-hash=6b7c8755c > Annotations: <none> > Status: Running > IP: 172.18.0.2 > IPs: > IP: 172.18.0.2 > Controlled By: ReplicaSet/kubeedge-counter-app-6b7c8755c > Containers: > kubeedge-counter-app: > Container ID: containerd://40ab51e43d438cde700d3dea2430252733574adeeb6cacb3a428bad1d15f5b63 > Image: pro1515151515/kubeedge-counter-app:v1.0.5 > Image ID: docker.io/pro1515151515/kubeedge-counter-app@sha256:2fb91b13a18b94ebbe218626d260935a7b6286938271cedea9518cfe4c5a9077 > Port: <none> > Host Port: <none> > State: Running > Started: Tue, 01 Dec 2020 07:31:05 +0000 > Last State: Terminated > Reason: Unknown > Exit Code: 255 > Started: Tue, 01 Dec 2020 07:03:14 +0000 > Finished: Tue, 01 Dec 2020 07:30:32 +0000 > Ready: True > Restart Count: 1 > Environment: <none> > Mounts: > /var/run/secrets/kubernetes.io/serviceaccount from default-token-xnv4t (ro) > Conditions: > Type Status > Initialized True > Ready True > ContainersReady True > PodScheduled True > Volumes: > default-token-xnv4t: > Type: Secret (a volume populated by a Secret) > SecretName: default-token-xnv4t > Optional: false > QoS Class: BestEffort > Node-Selectors: node-role.kubernetes.io/master= > Tolerations: node-role.kubernetes.io/master:NoSchedule > node.kubernetes.io/not-ready:NoExecute for 300s > node.kubernetes.io/unreachable:NoExecute for 300s > Events: > Type Reason Age From Message > ---- ------ ---- ---- ------- > Normal Scheduled 35m default-scheduler Successfully assigned default/kubeedge-counter-app-6b7c8755c-4vqqp to kind-control-plane > Normal Pulling 35m kubelet Pulling image "pro1515151515/kubeedge-counter-app:v1.0.5" > Normal Pulled 35m kubelet Successfully pulled image "pro1515151515/kubeedge-counter-app:v1.0.5" in 32.45517705s > Normal Created 35m kubelet Created container kubeedge-counter-app > Normal Started 35m kubelet Started container kubeedge-counter-app > Normal SandboxChanged 7m38s kubelet Pod sandbox changed, it will be killed and re-created. > Normal Pulled 7m37s kubelet Container image "pro1515151515/kubeedge-counter-app:v1.0.5" already present on machine > Normal Created 7m37s kubelet Created container kubeedge-counter-app > Normal Started 7m35s kubelet Started container kubeedge-counter-app > ``` 知道了，edgecore没上线的时候，它就部署到kind集群里了，和网页、k8s-API-server在一块。 **以下在edge（192.168.56.101）操作：** 1. tmux 2. edgecore 3. `ctrl + b`，`c` > ``` > root@kubeedge:~# kubectl get device > The connection to the server localhost:8080 was refused - did you specify the right host or port? > ``` 还有一个可能：edge没连上k8s的master节点。虽然连上kubeedge的了，但没连上k8s的。 ### ~~edge连接k8s的master节点~~ **以下在cloud（192.168.56.103）操作：** ``` kubeadm token create ``` >``` >root@cloud:.../crds# kubeadm token create >W1201 08:43:19.467017 12271 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] >gjx4rl.z9bo8nopdxh3i1b5 >``` **以下在edge（192.168.56.101）操作：** ``` kubeadm join 192.168.56.103:8080 --token=gjx4rl.z9bo8nopdxh3i1b5 --discovery-token-unsafe-skip-ca-verification ``` 算了还是别了。。 ### 新状况 ``` root@cloud:~# kubectl get pods NAME READY STATUS RESTARTS AGE kubeedge-counter-app-6b7c8755c-b42f2 1/1 Running 0 32m kubeedge-pi-counter-85b7c977b-vn6gj 1/1 Running 0 79m root@cloud:~# kubectl logs -f kubeedge-pi-counter-85b7c977b-vn6gj Error from server: Get "https://192.168.56.101:10350/containerLogs/default/kubeedge-pi-counter-85b7c977b-vn6gj/kubeedge-pi-counter?follow=true": dial tcp 192.168.56.101:10350: connect: connection refused ``` 修改： ``` cd $GOPATH/src/github.com/kubeedge/examples/kubeedge-counter-demo/web-controller-app/utils vim crdclient.go ``` 在`config := *cfg`下面加一行：`config.Insecure=true`，保存退出，重新弄个镜像。 。。。。还是不行。 ### 更新kind 可以在 [kindest/node Tags - Docker Hub](https://hub.docker.com/r/kindest/node/tags?page=1&ordering=last_updated) 看到最新的版本号 ``` docker pull kindest/node:v1.19.3 kind delete cluster kind create cluster --image kindest/node:v1.19.3 kubectl cluster-info --context kind-kind ``` > ```bash > root@cloud:~# kind create cluster --image kindest/node:v1.19.3 > Creating cluster "kind" ... > ✓ Ensuring node image (kindest/node:v1.19.3) 🖼 > ✓ Preparing nodes 📦 > ✓ Writing configuration 📜 > ✓ Starting control-plane 🕹️ > ✓ Installing CNI 🔌 > ✓ Installing StorageClass 💾 > Set kubectl context to "kind-kind" > You can now use your cluster with: > > kubectl cluster-info --context kind-kind > > Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂 > root@cloud:~# kubectl cluster-info --context kind-kind > > Kubernetes master is running at https://127.0.0.1:35309 > KubeDNS is running at https://127.0.0.1:35309/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy > > To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. > ``` ### 更新kubeedge ``` rm -rf $GOPATH/src/github.com/kubeedge/kubeedge cd $GOPATH/src/github.com/kubeedge git clone https://github.com.cnpmjs.org/kubeedge/kubeedge.git scp -r kubeedge root@192.168.56.101:/home/pro/go/src/github.com/kubeedge/kubeedge chmod 777 * -R ``` ``` cd $GOPATH/src/github.com/kubeedge/kubeedge make all WHAT=cloudcore cp _output/local/bin/cloudcore /usr/local/bin/cloudcore make all WHAT=keadm cp _output/local/bin/keadm /usr/bin/keadm cp $GOPATH/src/github.com/kubeedge/kubeedge/build/tools/certgen.sh /etc/kubeedge/ export CLOUDCOREIPS="192.168.56.103" /etc/kubeedge/certgen.sh buildSecret /etc/kubeedge/certgen.sh genCertAndKey server scp -r /etc/kubeedge/certs root@192.168.56.101:/etc/kubeedge/ keadm init --kube-config=/root/.kube/config --advertise-address="192.168.56.103" keadm gettoken cd $GOPATH/src/github.com/kubeedge/kubeedge make all WHAT=edgecore cp _output/local/bin/edgecore /usr/bin/edgecore keadm join --cloudcore-ipport=192.168.56.103:10000 --token=8d7e8e45ae1277f4851e2e73fa1fdd1e92f02649f67d55259b4e246ae1965fe7.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MDY5NzU0MDJ9.U6r0F2RSHKuoNYatjnmyQeTpnEduj6Ou7SlOnTHFBPA ssh -Nfg -L 0.0.0.0:80:172.18.0.2:80 192.168.56.103 ssh -Nfg -L 0.0.0.0:8080:172.18.0.2:8080 192.168.56.103 ``` 等会儿。。。最新版本的edgecore老是运行失败。版本是`KubeEdge v1.5.0-beta.0.82+167966f71ee86c-dirty`，不知道最后的dirty是不是运行不了的意思。v1.5.0的git版本号是8b09913e，现在要退回这个版本，重新编译。 ``` cd $GOPATH/src/github.com/kubeedge/kubeedge git reset --hard commit_id 8b0913e ``` 注册设备，换kind集群以后，如果不重新注册设备，就一堆报错 ``` cd $GOPATH/src/github.com/kubeedge/kubeedge/build/crds/devices kubectl apply -f ./devices_v1alpha2_device.yaml kubectl apply -f ./devices_v1alpha2_devicemodel.yaml cd $GOPATH/src/github.com/kubeedge/kubeedge/build/crds/reliablesyncs kubectl apply -f objectsync_v1alpha1.yaml kubectl apply -f cluster_objectsync_v1alpha1.yaml ``` ### 重头来 **云（192.168.56.103）：** ``` kind delete cluster rm -rf ~/.kube rm -rf /etc/kubeedge mkdir ~/.kube mkdir /etc/kubeedge mkdir /etc/kubeedge/config kind create cluster --image kindest/node:v1.19.3 cloudcore --defaultconfig > /etc/kubeedge/config/cloudcore.yaml vim /etc/kubeedge/config/cloudcore.yaml # 改一下IP # 注册设备 cd $GOPATH/src/github.com/kubeedge/kubeedge/build/crds/devices kubectl apply -f ./devices_v1alpha2_device.yaml kubectl apply -f ./devices_v1alpha2_devicemodel.yaml cd $GOPATH/src/github.com/kubeedge/kubeedge/build/crds/reliablesyncs kubectl apply -f objectsync_v1alpha1.yaml kubectl apply -f cluster_objectsync_v1alpha1.yaml # 自签server证书 cp $GOPATH/src/github.com/kubeedge/kubeedge/build/tools/certgen.sh /etc/kubeedge/ export CLOUDCOREIPS="192.168.56.103" /etc/kubeedge/certgen.sh genCertAndKey server # 自签stream证书 mkdir -p /etc/kubernetes/pki docker exec kind-control-plane cat /etc/kubernetes/pki/ca.crt > /etc/kubernetes/pki/ca.crt docker exec kind-control-plane cat /etc/kubernetes/pki/ca.key > /etc/kubernetes/pki/ca.keycd /etc/kubeedge/certgen.sh buildSecret export CLOUDCOREIPS="192.168.56.103" /etc/kubeedge/certgen.sh stream cp /etc/kubeedge/certs/stream.crt /etc/kubeedge/certs/server.crt cp /etc/kubeedge/certs/stream.csr /etc/kubeedge/certs/server.csr cp /etc/kubeedge/certs/stream.key /etc/kubeedge/certs/server.key scp -r /etc/kubeedge/certs root@192.168.56.101:/etc/kubeedge/ scp -r /etc/kubeedge/ca root@192.168.56.101:/etc/kubeedge/ kubectl delete secret casecret -nkubeedge kubectl delete secret cloudcoresecret -nkubeedge kubectl get secret -n kubeedge tokensecret -o=jsonpath='{.data.tokendata}' | base64 -d ``` **边（192.168.56.101）** ``` rm /var/lib/kubeedge/edgecore.db rm -rf ~/.kube rm -rf /etc/kubeedge mkdir ~/.kube mkdir /etc/kubeedge mkdir /etc/kubeedge/config edgecore --defaultconfig > /etc/kubeedge/config/edgecore.yaml vim /etc/kubeedge/config/edgecore.yaml # 改一下IP和token mkdir -p /usr/share/ca-certificates/kubeedge/ cp /etc/kubeedge/ca/rootCA.crt /usr/share/ca-certificates/kubeedge/rootCA.crt cp /etc/kubeedge/ca/streamCA.crt /usr/share/ca-certificates/kubeedge/streamCA.crt cp /etc/kubeedge/certs/edge.crt /usr/share/ca-certificates/kubeedge/edge.crt cp /etc/kubeedge/certs/server.crt /usr/share/ca-certificates/kubeedge/server.crt cp /etc/kubeedge/certs/stream.crt /usr/share/ca-certificates/kubeedge/stream.crt echo "kubeedge/rootCA.crt" >> /etc/ca-certificates.conf echo "kubeedge/streamCA.crt" >> /etc/ca-certificates.conf echo "kubeedge/edge.crt" >> /etc/ca-certificates.conf echo "kubeedge/server.crt" >> /etc/ca-certificates.conf echo "kubeedge/stream.crt" >> /etc/ca-certificates.conf update-ca-certificates token="b20a12207c2244c74b40705b3b9ee4db817643526409137d4a9490e701c9d892.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MDcwMDMxOTZ9.v-UTesL-RM535IXotGjvrKQmjvwK0v7r6NatgaRsNwQ" mkdir -p /etc/kubeedge/{ca,certs} sed -i "s|token: .*|token: ${token}|g" edgecore.yaml ``` ### ~~试试edgecore用容器运行~~ ```` cd $GOPATH/src/github.com/kubeedge/kubeedge/build/edge ./run_daemon.sh prepare ```` ``` ./run_daemon.sh set \ cloudhub=192.168.56.103:10000 \ edgename=edge-node \ edgecore_image="kubeedge/edgecore:latest" \ arch=amd64 \ qemu_arch=x86_64 \ certpath=/etc/kubeedge/certs \ certfile=/etc/kubeedge/certs/edge.crt \ keyfile=/etc/kubeedge/certs/edge.key ``` ### 自签证书 ``` cd /etc/kubeedge rm -rf ca rm -rf certs rm /etc/kubeedge/certgen.sh vim /etc/kubeedge/certgen.sh ``` `esc` ，`:set paste` ，`i`，粘贴入一下内容： ``` #!/usr/bin/env bash set -o errexit readonly caPath=${CA_PATH:-/etc/kubeedge/ca} readonly caSubject=${CA_SUBJECT:-/C=CN/ST=Zhejiang/L=Hangzhou/O=KubeEdge/CN=kubeedge.io} readonly certPath=${CERT_PATH:-/etc/kubeedge/certs} readonly subject=${SUBJECT:-/C=CN/ST=Zhejiang/L=Hangzhou/O=KubeEdge/CN=kubeedge.io} genCA() { openssl genrsa -des3 -out ${caPath}/rootCA.key -passout pass:kubeedge.io 4096 openssl req -x509 -new -nodes -key ${caPath}/rootCA.key -sha256 -days 3650 \ -subj ${subject} -passin pass:kubeedge.io -out ${caPath}/rootCA.crt } ensureCA() { if [ ! -e ${caPath}/rootCA.key ] || [ ! -e ${caPath}/rootCA.crt ]; then genCA fi } ensureFolder() { if [ ! -d ${caPath} ]; then mkdir -p ${caPath} fi if [ ! -d ${certPath} ]; then mkdir -p ${certPath} fi } genCsr() { local name=$1 openssl genrsa -out ${certPath}/${name}.key 2048 openssl req -new -key ${certPath}/${name}.key -subj ${subject} -out ${certPath}/${name}.csr } genCert() { local name=$1 openssl x509 -req -in ${certPath}/${name}.csr -CA ${caPath}/rootCA.crt -CAkey ${caPath}/rootCA.key \ -CAcreateserial -passin pass:kubeedge.io -out ${certPath}/${name}.crt -days 365 -sha256 } genCertAndKey() { ensureFolder ensureCA local name=$1 genCsr $name genCert $name } gen() { ensureFolder ensureCA local name=$1 local ipList=$2 genCsr $name #local name=$1 #openssl x509 -req -in ${certPath}/${name}.csr -CA ${caPath}/rootCA.crt -CAkey ${caPath}/rootCA.key \ #-CAcreateserial -passin pass:kubeedge.io -out ${certPath}/${name}.crt -days 365 -sha256 SUBJECTALTNAME="subjectAltName = IP.1:127.0.0.1" index=1 for ip in ${ipList}; do SUBJECTALTNAME="${SUBJECTALTNAME}," index=$(($index+1)) SUBJECTALTNAME="${SUBJECTALTNAME}IP.${index}:${ip}" done echo $SUBJECTALTNAME > /tmp/server-extfile.cnf # verify openssl x509 -req -in ${certPath}/${name}.csr -CA ${caPath}/rootCA.crt -CAkey ${caPath}/rootCA.key \ -CAcreateserial -passin pass:kubeedge.io -out ${certPath}/${name}.crt -days 5000 -sha256 \ -extfile /tmp/server-extfile.cnf #verify openssl x509 -in ${certPath}/${name}.crt -text -noout } stream() { ensureFolder readonly streamsubject=${SUBJECT:-/C=CN/ST=Zhejiang/L=Hangzhou/O=KubeEdge} readonly STREAM_KEY_FILE=${certPath}/stream.key readonly STREAM_CSR_FILE=${certPath}/stream.csr readonly STREAM_CRT_FILE=${certPath}/stream.crt readonly K8SCA_FILE=/etc/kubernetes/pki/ca.crt readonly K8SCA_KEY_FILE=/etc/kubernetes/pki/ca.key if [ -z ${CLOUDCOREIPS} ]; then echo "You must set CLOUDCOREIPS Env,The environment variable is set to specify the IP addresses of all cloudcore" echo "If there are more than one IP need to be separated with space." exit 1 fi index=1 SUBJECTALTNAME="subjectAltName = IP.1:127.0.0.1" for ip in ${CLOUDCOREIPS}; do SUBJECTALTNAME="${SUBJECTALTNAME}," index=$(($index+1)) SUBJECTALTNAME="${SUBJECTALTNAME}IP.${index}:${ip}" done cp /etc/kubernetes/pki/ca.crt ${caPath}/streamCA.crt echo $SUBJECTALTNAME > /tmp/server-extfile.cnf openssl genrsa -out ${STREAM_KEY_FILE} 2048 openssl req -new -key ${STREAM_KEY_FILE} -subj ${streamsubject} -out ${STREAM_CSR_FILE} # verify openssl req -in ${STREAM_CSR_FILE} -noout -text openssl x509 -req -in ${STREAM_CSR_FILE} -CA ${K8SCA_FILE} -CAkey ${K8SCA_KEY_FILE} -CAcreateserial -out ${STREAM_CRT_FILE} -days 5000 -sha256 -extfile /tmp/server-extfile.cnf #verify openssl x509 -in ${STREAM_CRT_FILE} -text -noout } buildSecret() { local name="edge" genCertAndKey ${name} > /dev/null 2>&1 cat <<EOF apiVersion: v1 kind: Secret metadata: name: cloudcore namespace: kubeedge labels: k8s-app: kubeedge kubeedge: cloudcore stringData: rootCA.crt: | $(pr -T -o 4 ${caPath}/rootCA.crt) edge.crt: | $(pr -T -o 4 ${certPath}/${name}.crt) edge.key: | $(pr -T -o 4 ${certPath}/${name}.key) EOF } $1 $2 $3 ``` 运行以下命令： ``` chmod 755 /etc/kubeedge/certgen.sh /etc/kubeedge/certgen.sh gen server 192.168.56.103 # 运行下面一行命令，相当于运行keadm gettoken kubectl get secret -n kubeedge tokensecret -o=jsonpath='{.data.tokendata}' | base64 -d # 手动给edge签发证书 /etc/kubeedge/certgen.sh buildSecret scp -r /etc/kubeedge/certs root@192.168.56.101:/etc/kubeedge/ ``` 终于好了。 > ``` > root@cloud:/etc/kubeedge/certs# kubectl get nodes > NAME STATUS ROLES AGE VERSION > kind-control-plane Ready master 19h v1.19.3 > kubeedge Ready agent,edge 15s v1.19.3-kubeedge-v1.5.0 > ``` ## 继续排错 > edgecore的报错： > > ``` > W1203 10:40:41.816211 1455 watcher.go:87] Error while processing event ("/sys/fs/cgroup/cpu,cpuacct/system.slice/edgecore.service": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/cpu,cpuacct/system.slice/edgecore.service: no such file or directory > W1203 10:40:41.818268 1455 watcher.go:87] Error while processing event ("/sys/fs/cgroup/blkio/system.slice/edgecore.service": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/blkio/system.slice/edgecore.service: no such file or directory > W1203 10:40:41.818506 1455 watcher.go:87] Error while processing event ("/sys/fs/cgroup/memory/system.slice/edgecore.service": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/system.slice/edgecore.service: no such file or directory > W1203 10:40:41.818574 1455 watcher.go:87] Error while processing event ("/sys/fs/cgroup/devices/system.slice/edgecore.service": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/system.slice/edgecore.service: no such file or directory > W1203 10:40:41.818648 1455 watcher.go:87] Error while processing event ("/sys/fs/cgroup/pids/system.slice/edgecore.service": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/pids/system.slice/edgecore.service: no such file or directory > E1203 10:40:50.943330 1455 dns.go:290] [EdgeMesh] service kubeedge is not found in this cluster > ``` ### 用keadm的debug工具 Edge： ``` root@kubeedge:~# keadm debug check all CPU total: 1 core, Allowed > 1 core CPU usage rate: 0.03, Allowed rate < 0.9 Memory total: 1987.82 MB, Allowed > 256 MB Memory Free total: 1113.69 MB, Allowed > 128 MB Memory usage rate: 0.13, Allowed rate < 0.9 Disk total: 19078.64 MB, Allowed > 1024 MB Disk Free total: 2314.95 MB, Allowed > 512MB Disk usage rate: 0.87, Allowed rate < 0.9 dns resolution success, domain: www.github.com ip: 192.30.255.112 ping 172.17.0.1 success check cloudhubServer 192.168.56.103:10000 success check edgecoreServer 127.0.0.1:10350 success Maximum PIDs: 4194304; Running processes: 107 docker is running |-----------------| |check all succeed| |-----------------| ``` Cloud： ``` root@cloud:~# keadm debug check all CPU total: 1 core, Allowed > 1 core CPU usage rate: 0.17, Allowed rate < 0.9 Memory total: 1987.82 MB, Allowed > 256 MB Memory Free total: 66.46 MB, Allowed > 128 MB Memory usage rate: 0.44, Allowed rate < 0.9 memory check failed |----------------| |check all failed| |----------------| ``` 内存不够。。重启一下试试 ``` root@cloud:~# keadm debug check all CPU total: 1 core, Allowed > 1 core CPU usage rate: 0.16, Allowed rate < 0.9 Memory total: 1987.82 MB, Allowed > 256 MB Memory Free total: 218.80 MB, Allowed > 128 MB Memory usage rate: 0.37, Allowed rate < 0.9 Disk total: 19078.64 MB, Allowed > 1024 MB Disk Free total: 794.83 MB, Allowed > 512MB Disk usage rate: 0.96, Allowed rate < 0.9 disk check failed |----------------| |check all failed| |----------------| root@cloud:~# df -h Filesystem Size Used Avail Use% Mounted on udev 952M 0 952M 0% /dev tmpfs 199M 1.2M 198M 1% /run /dev/mapper/ubuntu--vg-ubuntu--lv 19G 17G 795M 96% / tmpfs 994M 0 994M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 994M 0 994M 0% /sys/fs/cgroup /dev/sda2 976M 292M 618M 33% /boot /dev/loop1 56M 56M 0 100% /snap/core18/1932 /dev/loop0 31M 31M 0 100% /snap/snapd/9721 /dev/loop2 72M 72M 0 100% /snap/lxd/16099 /dev/loop4 55M 55M 0 100% /snap/core18/1880 /dev/loop3 30M 30M 0 100% /snap/snapd/8542 overlay 19G 17G 795M 96% /var/lib/docker/overlay2/d2eb6ed9468e25e7d4d47020869616387a92560383241b826b8fd69b9c70c881/merged tmpfs 199M 0 199M 0% /run/user/0 ``` 硬盘又满了。。。。 #### 清理空间 1. 清理Docker垃圾 ``` docker system prune ``` 清理出了1M的空间，剩余空间还有796M。 2. 清理软件安装包 ``` apt clean ``` 剩余空间现在是867M ，使用率96% 3. 删掉交换分区 ``` swapoff /swap.img rm -f /swap.img ``` 剩余3.2G ，使用率83%，可以了。 内存再加1G。 > ``` > root@cloud:~# keadm debug check all > CPU total: 1 core, Allowed > 1 core > CPU usage rate: 0.36, Allowed rate < 0.9 > Memory total: 2992.82 MB, Allowed > 256 MB > Memory Free total: 1204.18 MB, Allowed > 128 MB > Memory usage rate: 0.26, Allowed rate < 0.9 > Disk total: 19078.64 MB, Allowed > 1024 MB > Disk Free total: 3176.68 MB, Allowed > 512MB > Disk usage rate: 0.83, Allowed rate < 0.9 > dns resolution success, domain: www.github.com ip: 192.30.255.112 > E1203 12:40:54.397884 2449 helper.go:29] Failed to read configfile /etc/kubeedge/config/edgecore.yaml: open /etc/kubeedge/config/edgecore.yaml: no such file or directory > ping 127.0.0.53 success > check edgecoreServer 127.0.0.1:10350 failed, 127.0.0.1:10350 > > |----------------| > |check all failed| > |----------------| > ``` 标签: none 本作品采用 知识共享署名-相同方式共享 4.0 国际许可协议 进行许可。
评论已关闭