Integrasi GlusterFS dengan Kubernetes

Dynamic Volume Provisioning adalah konsep dimana kita bisa membuat dan menghapus volume on-demand. Yang gampangnya adalah saat kita membuat PVC, konsep ini akan otomatis membuat volume dan PV. Saat kita menghapus PVC, maka kubernetes akan otomatis menghapus PV dan volumenya.

Konsep ini dapat diterapkan pada GlusterFS. developer (user dari k8s) tidak perlu tau bagaimana internal system dari volumenya. Cukup dengan membuat PVC saja.

GlusterFS Node

  • 1 Block Device (no file system)
  • Ubuntu 16.04
Disk /dev/vda: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
  • Install GlusterFS
apt-get install software-properties-common -y
add-apt-repository ppa:gluster/glusterfs-3.10

apt-get update -y
apt-get install glusterfs-server -y
apt install thin-provisioning-tools -y
  • Rename service glusterFS (diperlukan hanya di ubuntu)
systemctl stop glusterfs-server
mv /etc/init.d/glusterfs-server /etc/init.d/glusterd
systemctl start glusterd
systemctl enable glusterd

gluster peer status
gluster pool list
  • Install Heketi
wget https://github.com/heketi/heketi/releases/download/v8.0.0/heketi-v8.0.0.linux.amd64.tar.gz
tar xzvf heketi-v8.0.0.linux.amd64.tar.gz
cd heketi
cp heketi heketi-cli /usr/local/bin/
cd ..
heketi -v


groupadd -r -g 515 heketi
useradd -r -c "Heketi user" -d /var/lib/heketi -s /bin/false -m -u 515 -g heketi heketi
mkdir -p /var/lib/heketi && chown -R heketi:heketi /var/lib/heketi
mkdir -p /var/log/heketi && chown -R heketi:heketi /var/log/heketi
mkdir -p /etc/heketi
  • Install Heketi Client
wget https://github.com/heketi/heketi/releases/download/v8.0.0/heketi-client-v8.0.0.linux.amd64.tar.gz
mkdir -p /etc/heketi && tar xzvf heketi-client-v8.0.0.linux.amd64.tar.gz -C /etc/heketi
export PATH=$PATH:/etc/heketi/heketi-client/bin
heketi-cli --version
  • Copy Heketi Client ssh public key ke node GlusterFS (IP sendiri karena Heketi dan GlusterFS jadi 1 Node)
ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
chown heketi:heketi /etc/heketi/heketi_key*
ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.150.150.60
  • tambahkan Konfigurasi Heketi
nano /etc/heketi/heketi.json
...
{
	"_port_comment": "Heketi Server Port Number",
	"port": "8080",

	"_jwt": "Private keys for access",
	"jwt": {
		"_admin": "Admin has access to all APIs",
		"admin": {
			"key": "My Secret"
		},
		"_user": "User only has access to /volumes endpoint",
		"user": {
			"key": "My Secret"
		}
	},

	"_use_auth": "Enable JWT authorization. Please enable for deployment",
	"use_auth": true,

	"_glusterfs_comment": "GlusterFS Configuration",
	"glusterfs": {
		"_executor_comment": [
			"Execute plugin. Possible choices: mock, ssh",
			"mock: This setting is used for testing and development.",
			"      It will not send commands to any node.",
			"ssh:  This setting will notify Heketi to ssh to the nodes.",
			"      It will need the values in sshexec to be configured.",
			"kubernetes: Communicate with GlusterFS containers over",
			"            Kubernetes exec api."
		],
		"executor": "ssh",

		"_sshexec_comment": "SSH username and private key file information",
		"sshexec": {
			"keyfile": "/etc/heketi/heketi_key",
			"user": "root",
			"port": "22",
			"fstab": "/etc/fstab"
		}
	}
}
...
  • Buat service Heketi
nano /etc/systemd/system/heketi.service

...
[Unit]
Description=Heketi Server
Requires=network-online.target
After=network-online.target
 
[Service]
Type=simple
User=heketi
Group=heketi
PermissionsStartOnly=true
PIDFile=/run/heketi/heketi.pid
Restart=on-failure
RestartSec=10
WorkingDirectory=/var/lib/heketi
RuntimeDirectory=heketi
RuntimeDirectoryMode=0755
ExecStartPre=[ -f "/run/heketi/heketi.pid" ] && /bin/rm -f /run/heketi/heketi.pid
ExecStart=/usr/local/bin/heketi --config=/etc/heketi/heketi.json
ExecReload=/bin/kill -s HUP $MAINPID
KillSignal=SIGINT
TimeoutStopSec=5
 
[Install]
WantedBy=multi-user.target
...
  • Start Heketi
systemctl daemon-reload
systemctl start heketi.service
systemctl enable heketi.service
systemctl status heketi.service
  • Heketi Topology
nano /etc/heketi/topology.json

...
{
    "clusters": [{
        "nodes": [{
            "node": {
                "hostnames": {
                    "manage": [
                        "zu-glusterfs"
                    ],
                    "storage": [
                        "10.150.150.60"
                    ]
                },
                "zone": 1
            },
            "devices": [
                "/dev/vda"
            ]
        }]
    }]
}
...
  • Setup Heketi Cluster
export HEKETI_CLI_SERVER=http://10.150.150.60:8080
export HEKETI_CLI_USER=admin
export HEKETI_CLI_KEY="My Secret"
heketi-cli topology load --json=/etc/heketi/topology.json

heketi-cli cluster list
heketi-cli node list

All Kubernetes Node

  • Install GlusterFS Client
apt-get install software-properties-common -y
add-apt-repository ppa:gluster/glusterfs-3.10
apt-get update -y
apt-get install glusterfs-client -y

Kubernetes Master / API / Kubectl Node

  • Buat Secret (namespace wordpress, menyesuaikan)
echo -n "My Secret" | base64
nano gf-secret.yaml

...
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: wordpress
type: "kubernetes.io/glusterfs"
data:
  key: PASSWORD_BASE64_ENCODED
...

kubectl apply -f gf-secret.yaml
  • Buat Storage Class (namespace wordpress, menyesuaikan)
nano gf-class.yaml

...
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: gluster-heketi-external
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://10.150.150.60:8080"
  restuser: "admin"
  secretName: "heketi-secret"
  secretNamespace: "wordpress"
  volumetype: none
...

kubectl apply -f gf-class.yaml
  • Test membuat PVC
nano test-pvc.yaml

...
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: test-pvc
 namespace: wordpress
 annotations:
   volume.beta.kubernetes.io/storage-class: gluster-heketi-external
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
 	storage: 2Gi
...

kubectl apply -f test-pvc.yaml

Result

btech@zu-master:~$ kubectl get pvc -n wordpress
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
test-pvc   Bound    pvc-f7da5431-2a05-11e9-9bcb-525400249918   2Gi        RWX            gluster-heketi-external   23m
btech@zu-master:~$ kubectl describe pvc test-pvc -n wordpress
Name:          test-pvc
Namespace:     wordpress
StorageClass:  gluster-heketi-external
Status:        Bound
Volume:        pvc-f7da5431-2a05-11e9-9bcb-525400249918
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-class":"gluster-heketi-ext...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-class: gluster-heketi-external
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Events:
  Type       Reason                 Age   From                         Message
  ----       ------                 ----  ----                         -------
  Normal     ProvisioningSucceeded  23m   persistentvolume-controller  Successfully provisioned volume pvc-f7da5431-2a05-11e9-9bcb-525400249918 using kubernetes.io/glusterfs
Mounted By:  <none>
root@zu-glusterfs:~# heketi-cli volume list
Id:791fe848fe6fb9322b4a157f3289cbb6    Cluster:20dededcafc1a5371c652b1d5f5971a1    Name:vol_791fe848fe6fb9322b4a157f3289cbb6
root@zu-glusterfs:~# heketi-cli volume info 791fe848fe6fb9322b4a157f3289cbb6
Name: vol_791fe848fe6fb9322b4a157f3289cbb6
Size: 2
Volume Id: 791fe848fe6fb9322b4a157f3289cbb6
Cluster Id: 20dededcafc1a5371c652b1d5f5971a1
Mount: 10.150.150.60:vol_791fe848fe6fb9322b4a157f3289cbb6
Mount Options: backup-volfile-servers=
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: none
Snapshot Factor: 1.00
root@zu-glusterfs:~# gluster volume list
vol_791fe848fe6fb9322b4a157f3289cbb6
root@zu-glusterfs:~# gluster volume info
 
Volume Name: vol_791fe848fe6fb9322b4a157f3289cbb6
Type: Distribute
Volume ID: 5308f4d3-6c22-4d18-a352-8cfce17913b4
Status: Started
Snapshot Count: 0
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.150.150.60:/var/lib/heketi/mounts/vg_822ef1975950298b06861afe724a155b/brick_f2246ba37779bbfe3dc6489dfc85705d/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

Comments are closed.