Deploy Cassandra on Raspberry-PI 3
Goal
- The main goal is to use statefulset and local persistency volume.
- The arm32v7 image are not available on kubernetes example repository
- The arm32v7 images are not available either on the docker hub, probably because Cassandra advises to use 64 bits when PI 3 are still mainly 32 bits OS and that the amount of memory available is limited to 1Gbi.
Build Cassandra for PI 3
Cassandra 1
Based on various Cassandra running on RPI projects
Cassandra 2
This is a fork of Kubernetes example “adapted” to PI 3.
Build process
Since travis cross build is not inplace, let’s use a PI to build
$ git clone git@github.com:jbrette/kube-rpi.git
$ cd kube-rpi/images/cassandra1
$ docker build .
$ docker image list
$ docker tag 6b98a0ccef38 jbrette/cassandra-arm32v7:3.11.1
$ docker push jbrette/cassandra-arm32v7:3.11.1
Deploy
The deployment of the services relies on the “local-storage” class
The service description is identical to Kubernetes tutorial
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
clusterIP: None
ports:
- port: 9042
selector:
app: cassandra
The main difference with official examples:
- Memory reduces to 256M. 1Gi is too big for the PI.
- Use the “local-storage” class which is accessing local partition on the SD card of the PI.
- Change the image to previously build images.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cassandra
labels:
app: cassandra
spec:
serviceName: cassandra
replicas: 3
selector:
matchLabels:
app: cassandra
template:
metadata:
labels:
app: cassandra
spec:
terminationGracePeriodSeconds: 1800
containers:
- name: cassandra
image: jbrette/cassandra-arm32v7:3.11.1
imagePullPolicy: Always
ports:
- containerPort: 7000
name: intra-node
- containerPort: 7001
name: tls-intra-node
- containerPort: 7199
name: jmx
- containerPort: 9042
name: cql
resources:
limits:
cpu: "500m"
memory: 256M
requests:
cpu: "500m"
memory: 256M
securityContext:
capabilities:
add:
- IPC_LOCK
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- nodetool drain
env:
- name: MAX_HEAP_SIZE
value: 128M
- name: HEAP_NEWSIZE
value: 50M
- name: CASSANDRA_SEEDS
value: "cassandra-0.cassandra.default.svc.cluster.local"
- name: CASSANDRA_CLUSTER_NAME
value: "K8Demo"
- name: CASSANDRA_DC
value: "DC1-K8Demo"
- name: CASSANDRA_RACK
value: "Rack1-K8Demo"
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
readinessProbe:
exec:
command:
- /bin/bash
- -c
- /ready-probe.sh
initialDelaySeconds: 15
timeoutSeconds: 5
# These volume mounts are persistent. They are like inline claims,
# but not exactly because the names need to match exactly one of
# the stateful pod volumes.
volumeMounts:
- name: cassandra-data
mountPath: /cassandra_data
# These are converted to volume claims by the controller
# and mounted at the paths mentioned above.
# do not use these in production until ssd GCEPersistentDisk or other ssd pd
volumeClaimTemplates:
- metadata:
name: cassandra-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: local-storage
resources:
requests:
storage: 1Gi
Testing
Still work to be done on the image to get Cassandra operational
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/cassandra-0 1/1 Running 0 2h
pod/cassandra-1 0/1 CrashLoopBackOff 32 2h
pod/cassandra-2 1/1 Running 0 2h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cassandra ClusterIP None <none> 9042/TCP 5h
NAME DESIRED CURRENT AGE
statefulset.apps/cassandra 3 3 2h
Let’s check the PVC
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cassandra-data-cassandra-0 Bound kube-node03-vol07 2Gi RWO local-storage 5h
cassandra-data-cassandra-1 Bound kube-node01-vol04 2Gi RWO local-storage 2h
cassandra-data-cassandra-2 Bound kube-node04-vol06 2Gi RWO local-storage 2h
Cassandra was able to create pvc using the local storage based pv
$ kubectl describe pvc/cassandra-data-cassandra-0
Name: cassandra-data-cassandra-0
Namespace: default
StorageClass: local-storage
Status: Bound
Volume: kube-node03-vol07
Labels: app=cassandra
Annotations: pv.kubernetes.io/bind-completed=yes
pv.kubernetes.io/bound-by-controller=yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2Gi
Access Modes: RWO
Events: <none>
Looks like a bug in the PV creation (using vol1 instead of vol7)
$ kubectl describe pv/kube-node03-vol07
Name: kube-node03-vol07
Labels: <none>
Annotations: pv.kubernetes.io/bound-by-controller=yes
Finalizers: [kubernetes.io/pv-protection]
StorageClass: local-storage
Status: Bound
Claim: default/cassandra-data-cassandra-0
Reclaim Policy: Retain
Access Modes: RWO
Capacity: 2Gi
Node Affinity:
Required Terms:
Term 0: kubernetes.io/hostname in [kube-node03]
Message:
Source:
Type: LocalVolume (a persistent volume backed by local storage on a node)
Path: /mnt/disks/vol1
Events: <none>
Conclusion
- WIP