Deploy Helm and Tiller on Rasberry PI Cluster
Goal
The main purpose of this exercise is to be able to use Helm on the Rapsberry PI Cluster.
Key Aspects
- The goal is to setup helm and tiller on the Raspberry PI cluster
- Having the golang, glide…and related libraries setup in a PI for compilation is kind of complicated. I started but encounter too many issues (even small), had to install too many compilation related packages on my PI system, hence decided to use an Ubuntu VM to compile and prepare the binaries for image for helm and tiller. This will be usefull to setup CI/CD pipeline with Travis-CI.
Build Tiller executable and Docker image
Cross Compiling from Ubuntu Machine
First check the go setup. Fetch the code
Let’s check the go environment. See Setup GOLANG environment
which go
/usr/bin/go
go version
go version go1.10.1 linux/amd64
export GOPATH=$HOME/src
mkdir -p src/k8s.io
cd src/k8s.io
git clone -b release-2.9 git@github.com:kubernetes/helm.git
cd $HOME/src/k8s.io/helm
make clean bootstrap build-cross dist APP=helm VERSION=2.9 TARGETS=linux/arm
...
go build -o bin/protoc-gen-go ./vendor/github.com/golang/protobuf/protoc-gen-go
CGO_ENABLED=0 gox -parallel=3 -output="_dist/{{.OS}}-{{.Arch}}/{{.Dir}}" -osarch='linux/arm' -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.Version=2.9 -X k8s.io/helm/pkg/version.BuildMetadata= -X k8s.io/helm/pkg/version.GitCommit=20adb27c7c5868466912eebdf6664e7390ebe710 -X k8s.io/helm/pkg/version.GitTreeState=clean -extldflags "-static"' k8s.io/helm/cmd/helm
Number of parallel builds: 3
--> linux/arm: k8s.io/helm/cmd/helm
( \
cd _dist && \
find * -type d -exec cp ../LICENSE {} \; && \
find * -type d -exec cp ../README.md {} \; && \
find * -type d -exec tar -zcf helm-2.9-{}.tar.gz {} \; && \
find * -type d -exec zip -r helm-2.9-{}.zip {} \; \
)
adding: linux-arm/ (stored 0%)
adding: linux-arm/LICENSE (deflated 65%)
adding: linux-arm/README.md (deflated 59%)
adding: linux-arm/helm (deflated 67%)
Let’s transfer the helm binarie to RPI
scp -r _dist/linux-arm/ rpiuser@192.168.1.95:/home/rpiuser/helm_binaries
Cross Compiling and Image cration from Travis-CI
- WIP
Helm
Cross Compiling from Ubuntu Machine
Let’s assume we cloned the code for helm already.
make clean bootstrap build-cross dist APP=tiller VERSION=2.9 TARGETS=linux/arm
...
go build -o bin/protoc-gen-go ./vendor/github.com/golang/protobuf/protoc-gen-go
CGO_ENABLED=0 gox -parallel=3 -output="_dist/{{.OS}}-{{.Arch}}/{{.Dir}}" -osarch='linux/arm' -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.Version=2.9 -X k8s.io/helm/pkg/version.BuildMetadata= -X k8s.io/helm/pkg/version.GitCommit=20adb27c7c5868466912eebdf6664e7390ebe710 -X k8s.io/helm/pkg/version.GitTreeState=clean -extldflags "-static"' k8s.io/helm/cmd/tiller
Number of parallel builds: 3
--> linux/arm: k8s.io/helm/cmd/tiller
( \
cd _dist && \
find * -type d -exec cp ../LICENSE {} \; && \
find * -type d -exec cp ../README.md {} \; && \
find * -type d -exec tar -zcf helm-2.9-{}.tar.gz {} \; && \
find * -type d -exec zip -r helm-2.9-{}.zip {} \; \
)
adding: linux-arm/ (stored 0%)
adding: linux-arm/LICENSE (deflated 65%)
adding: linux-arm/README.md (deflated 59%)
adding: linux-arm/tiller (deflated 67%)
Build the docker image on the remote PI
Create a Dockerfile called rootfs/Dockerfile.arm32v7
FROM debian:jessie-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates \
&& rm -rf /var/lib/apt/lists/*
ENV HOME /tmp
COPY _dist/linux-arm /
EXPOSE 44134
CMD ["/tiller"]
export DHUBREPO=jbrette/tiller-arm32v7
export VERSION=2.9
docker build -t $DHUBREPO:$VERSION -f rootfs/Dockerfile.arm32v7 .
Sending build context to Docker daemon 284.1MB
Step 1/6 : FROM debian:jessie-slim
---> d273fca45b31
Step 2/6 : RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && rm -rf /var/lib/apt/lists/*
---> Running in e51ea5c31ad7
Get:1 http://security.debian.org jessie/updates InRelease [44.9 kB]
Ign http://deb.debian.org jessie InRelease
Get:2 http://deb.debian.org jessie-updates InRelease [145 kB]
Get:3 http://deb.debian.org jessie Release.gpg [2420 B]
Get:4 http://deb.debian.org jessie Release [148 kB]
Get:5 http://security.debian.org jessie/updates/main armel Packages [576 kB]
Get:6 http://deb.debian.org jessie-updates/main armel Packages [23.7 kB]
Get:7 http://deb.debian.org jessie/main armel Packages [8902 kB]
Fetched 9843 kB in 49s (199 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
libssl1.0.0 openssl
The following NEW packages will be installed:
ca-certificates libssl1.0.0 openssl
0 upgraded, 3 newly installed, 0 to remove and 1 not upgraded.
Need to get 1692 kB of archives.
After this operation, 3770 kB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security/ jessie/updates/main ca-certificates all 20141019+deb8u4 [185 kB]
Get:2 http://deb.debian.org/debian/ jessie/main libssl1.0.0 armel 1.0.1t-1+deb8u8 [852 kB]
Get:3 http://deb.debian.org/debian/ jessie/main openssl armel 1.0.1t-1+deb8u8 [655 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 1692 kB in 7s (235 kB/s)
Selecting previously unselected package libssl1.0.0:armel.
(Reading database ... 7451 files and directories currently installed.)
Preparing to unpack .../libssl1.0.0_1.0.1t-1+deb8u8_armel.deb ...
Unpacking libssl1.0.0:armel (1.0.1t-1+deb8u8) ...
Selecting previously unselected package openssl.
Preparing to unpack .../openssl_1.0.1t-1+deb8u8_armel.deb ...
Unpacking openssl (1.0.1t-1+deb8u8) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../ca-certificates_20141019+deb8u4_all.deb ...
Unpacking ca-certificates (20141019+deb8u4) ...
Setting up libssl1.0.0:armel (1.0.1t-1+deb8u8) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/arm-linux-gnueabi/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/arm-linux-gnueabi/perl5/5.20 /usr/share/perl5 /usr/lib/arm-linux-gnueabi/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Setting up openssl (1.0.1t-1+deb8u8) ...
Setting up ca-certificates (20141019+deb8u4) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/arm-linux-gnueabi/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/arm-linux-gnueabi/perl5/5.20 /usr/share/perl5 /usr/lib/arm-linux-gnueabi/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Updating certificates in /etc/ssl/certs... 152 added, 0 removed; done.
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Processing triggers for ca-certificates (20141019+deb8u4) ...
Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
Removing intermediate container e51ea5c31ad7
---> 1dccf46769e7
Step 3/6 : ENV HOME /tmp
---> Running in 45de8c047bd3
Removing intermediate container 45de8c047bd3
---> baa6d8b15164
Step 4/6 : COPY _dist/linux-arm /
---> 59a6cc8bfdbe
Step 5/6 : EXPOSE 44134
---> Running in d2c6d9b3a3bc
Removing intermediate container d2c6d9b3a3bc
---> 28a331217c52
Step 6/6 : CMD ["/tiller"]
---> Running in 9760ea4dc6c5
Removing intermediate container 9760ea4dc6c5
---> 9384b4f39ab3
Successfully built 9384b4f39ab3
Successfully tagged jbrette/tiller-arm32v7:2.9
Cross Compiling and Image cration from Travis-CI
- WIP
Install Tiller and Helm on RPI
Install helm on PI
ssh rpiuser@192.168.1.95
sudo cp /home/rpiuser/helm_binaries/helm /usr/bin/helm
Deploy Tiller POD
First push the tiller docker image to the repo
docker login
docker push jbrette/tiller-arm32v7:2.9
Prepare the POD deployment files
mkdir -p ~/kube-deployments/tiller
cd ~/kube-deployments/tiller
Create the kubectl file for tiller service account
cat tiller-serviceaccount.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
...
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
...
Create the tiller service account
kubectl create -f tiller-serviceaccount.yaml
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
Create manually the tiller deployment file and leverage the tiller service account
helm init --service-account tiller --output yaml > tiller.yaml
Edit the image and replace gcr.io/kubernetes-helm/tiller:2.9 by jbrette/tiller-arm32v7:2.9
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: helm
name: tiller
name: tiller-deploy
namespace: kube-system
spec:
replicas: 1
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: helm
name: tiller
spec:
containers:
- env:
- name: TILLER_NAMESPACE
value: kube-system
- name: TILLER_HISTORY_MAX
value: "0"
image: gcr.io/kubernetes-helm/tiller:2.9
imagePullPolicy: IfNotPresent
livenessProbe:
httpGet:
path: /liveness
port: 44135
initialDelaySeconds: 1
timeoutSeconds: 1
name: tiller
ports:
- containerPort: 44134
name: tiller
- containerPort: 44135
name: http
readinessProbe:
httpGet:
path: /readiness
port: 44135
initialDelaySeconds: 1
timeoutSeconds: 1
resources: {}
serviceAccountName: tiller
status: {}
Deploy tiller in the kubernetes cluster
kubectl create -f tiller.yaml
Check the deployment state
kubectl get all -n kube-system
NAME READY STATUS RESTARTS AGE
pod/tiller-deploy-b59fcc885-66l7s 1/1 Running 0 5m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/tiller-deploy 1 1 1 1 5m
NAME DESIRED CURRENT READY AGE
replicaset.apps/tiller-deploy-b59fcc885 1 1 1 5m
Check the logs
kubectl logs pod/tiller-deploy-b59fcc885-66l7s -n kube-system
[main] 2018/07/15 18:08:10 Starting Tiller 2.9 (tls=false)
[main] 2018/07/15 18:08:10 GRPC listening on :44134
[main] 2018/07/15 18:08:10 Probes listening on :44135
[main] 2018/07/15 18:08:10 Storage driver is ConfigMap
[main] 2018/07/15 18:08:10 Max history per release is 0
Init the Helm client side
helm init --client-only
Creating /home/rpiuser/.helm
Creating /home/rpiuser/.helm/repository
Creating /home/rpiuser/.helm/repository/cache
Creating /home/rpiuser/.helm/repository/local
Creating /home/rpiuser/.helm/plugins
Creating /home/rpiuser/.helm/starters
Creating /home/rpiuser/.helm/cache/archive
Creating /home/rpiuser/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /home/rpiuser/.helm.
Not installing Tiller due to 'client-only' flag having been set
Happy Helming!
Check helm commands are working properly
helm ls
helm search stable/nginx-ingress
helm inspect stable/nginx-ingress
Use helm on RPI
Let’s access the simple helm repo designed for
helm repo add kubeplay 'https://raw.githubusercontent.com/jbrette/kubeplay/arm32v7/helmrepo/'
helm repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com
local http://127.0.0.1:8879/charts
kubeplay https://raw.githubusercontent.com/jbrette/kubeplay/arm32v7/helmrepo/
helm search kubeplay
NAME CHART VERSION APP VERSION DESCRIPTION
kubeplay/kubeplay-arm32v7 0.1.0 0.1.0 A Helm chart for Kubernetes
Install simple hello world helm chart from kubeplay
helm install kubeplay/kubeplay-arm32v7 --name helm-rpi
NAME: helm-rpi
LAST DEPLOYED: Sun Jul 15 19:23:58 2018
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
helm-rpi-kubeplay-arm32v7 NodePort 10.96.168.184 <none> 8005:31439/TCP 2s
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
helm-rpi-kubeplay-arm32v7 1 0 0 0 2s
Check the installation
helm ls
NAME REVISION UPDATED STATUS CHART NAMESPACE
helm-rpi 1 Sun Jul 15 19:23:58 2018 DEPLOYED kubeplay-arm32v7-0.1.0 default
Check with kubectl if the POD is ready
kubectl get all -n default
NAME READY STATUS RESTARTS AGE
pod/helm-rpi-kubeplay-arm32v7-58567c4756-fj968 0/1 Running 4 4m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/helm-rpi-kubeplay-arm32v7 NodePort 10.96.168.184 <none> 8005:31439/TCP 4m
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/helm-rpi-kubeplay-arm32v7 1 1 1 0 4m
NAME DESIRED CURRENT READY AGE
replicaset.apps/helm-rpi-kubeplay-arm32v7-58567c4756 1 1 0 4m
Helm chart is not really consistent. Current image hardcoded to listen on port 80 Let’s use helm upgrade to change the value
helm upgrade --set service.internalPort=80 helm-rpi kubeplay/kubeplay-arm32v7
Release "helm-rpi" has been upgraded. Happy Helming!
LAST DEPLOYED: Sun Jul 15 20:39:17 2018
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
helm-rpi-kubeplay-arm32v7 1 1 0 0 1h
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
helm-rpi-kubeplay-arm32v7-58567c4756-fj968 0/1 Terminating 25 1h
helm-rpi-kubeplay-arm32v7-6cb66496c6-9w97m 0/1 Pending 0 0s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
helm-rpi-kubeplay-arm32v7 NodePort 10.96.168.184 <none> 8005:31439/TCP 1h
The pod seems to now be running properly.
kubectl get all -n default
NAME READY STATUS RESTARTS AGE
pod/helm-rpi-kubeplay-arm32v7-6cb66496c6-9w97m 1/1 Running 0 3m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/helm-rpi-kubeplay-arm32v7 NodePort 10.96.168.184 <none> 8005:31439/TCP 1h
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/helm-rpi-kubeplay-arm32v7 1 1 1 1 1h
NAME DESIRED CURRENT READY AGE
replicaset.apps/helm-rpi-kubeplay-arm32v7-58567c4756 0 0 0 1h
replicaset.apps/helm-rpi-kubeplay-arm32v7-6cb66496c6 1 1 1 3m
Check the POD
$ kubectl describe pod/helm-rpi-kubeplay-arm32v7-6cb66496c6-9w97m
Name: helm-rpi-kubeplay-arm32v7-6cb66496c6-9w97m
Namespace: default
Node: nas-pi/192.168.1.93
Start Time: Sun, 15 Jul 2018 20:39:18 +0000
Labels: app=kubeplay-arm32v7
pod-template-hash=2762205272
release=helm-rpi
Annotations: <none>
Status: Running
IP: 10.244.2.6
Controlled By: ReplicaSet/helm-rpi-kubeplay-arm32v7-6cb66496c6
Containers:
kubeplay-arm32v7:
Container ID: docker://737cd78bebb982c782dae2c07f4604ba71f45bc112e91aa20652e29b25bf5d63
Image: jbrette/kubeplay-arm32v7:0.1.0
Image ID: docker-pullable://jbrette/kubeplay-arm32v7@sha256:86441999035e35514fc647ff9af90bdfc152299636b48b9d3dba9107574f3957
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 15 Jul 2018 20:39:23 +0000
Ready: True
Restart Count: 0
Liveness: http-get http://:80/ delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:80/ delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-c5v7s (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-c5v7s:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-c5v7s
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned default/helm-rpi-kubeplay-arm32v7-6cb66496c6-9w97m to nas-pi
Normal Pulling 10m kubelet, nas-pi pulling image "jbrette/kubeplay-arm32v7:0.1.0"
Normal Pulled 10m kubelet, nas-pi Successfully pulled image "jbrette/kubeplay-arm32v7:0.1.0"
Normal Created 10m kubelet, nas-pi Created container
Normal Started 10m kubelet, nas-pi Started container
Let’s open a browser towards the URL:
Conclusion
- Need to cleanup the helm chart and docker image to ensure consistency of the internal
- Need to improve the NodePort/ClusterIP/LoadBalancer handling.