gluster-kubernetes (TODO)

GlusterFS Native Storage Service for Kubernetes

gluster-kubernetes is a project to provide Kubernetes administrators a mechanism to easily deploy GlusterFS as a native storage service onto an existing Kubernetes cluster. Here, GlusterFS is managed and orchestrated like any other app in Kubernetes.

Component Projects

• Kubernetes, the container management system.
• GlusterFS, the scale-out storage system.
• heketi, the RESTful volume management interface for GlusterFS.

Quickstart

This project includes a vagrant setup in the vagrant/ directory to spin up a Kubernetes cluster in VMs.

Pre-Requisites

To run the vagrant setup, you'll need to have the following pre-requisites on your machine:

• 4GB of memory
• 32GB of storage minimum, 112GB recommended
• ansible
• vagrant
• libvirt or VirtualBox

sudo apt-get update
sudo apt-get install software-properties-common -y
sudo apt-add-repository ppa:ansible/ansible -y
sudo apt-get update -y
sudo apt-get install ansible -y

wget https://releases.hashicorp.com/vagrant/2.1.1/vagrant_2.1.1_linux_amd64.zip
unzip vagrant_2.1.1_linux_amd64.zip
echo 'export PATH="/home/hadoop/:$PATH"' >> ~/.bashrc
source ~/.bashrc

sudo apt install virtualbox -y

Spin up the cluster

git clone https://github.com/gluster/gluster-kubernetes.git
cd gluster-kubernetes/vagrant

./up.sh

git clone https://github.com/heketi/heketi
cd heketi/extras/kubernetes

kubectl create -f glusterfs-daemonset.json
daemonset.extensions "glusterfs" created

kubectl get nodes
NAME                                           STATUS    ROLES     AGE       VERSION
ip-172-20-37-51.ap-south-1.compute.internal    Ready     node      6m        v1.9.6
ip-172-20-51-68.ap-south-1.compute.internal    Ready     master    6m        v1.9.6
ip-172-20-56-172.ap-south-1.compute.internal   Ready     node      6m        v1.9.6

kubectl label node ip-172-20-37-51.ap-south-1.compute.internal storagenode=glusterfs
node "ip-172-20-37-51.ap-south-1.compute.internal" labeled

kubectl label node ip-172-20-56-172.ap-south-1.compute.internal storagenode=glusterfs
node "ip-172-20-56-172.ap-south-1.compute.internal" labeled

kubectl create -f heketi-service-account.json
serviceaccount "heketi-service-account" created

kubectl create clusterrolebinding heketi-gluster-admin --clusterrole=edit --serviceaccount=default:heketi-service-account
clusterrolebinding.rbac.authorization.k8s.io "heketi-gluster-admin" created

kubectl create secret generic heketi-config-secret --from-file=./heketi.json
secret "heketi-config-secret" created

kubectl create -f heketi-bootstrap.json
service "deploy-heketi" created
deployment.extensions "deploy-heketi" created

kubectl get pods
NAME                             READY     STATUS    RESTARTS   AGE
deploy-heketi-69bfbd4bbd-7pmfc   1/1       Running   0          42s
glusterfs-bz8dm                  1/1       Running   0          10m
glusterfs-rcn6k                  1/1       Running   0          10m

ubectl port-forward deploy-heketi-69bfbd4bbd-7pmfc :8080
Forwarding from 127.0.0.1:56193 -> 8080
Forwarding from [::1]:56193 -> 8080

curl http://localhost:56193/hello
Hello from Heketi%

export HEKETI_CLI_SERVER=http://localhost:56193/
echo $HEKETI_CLI_SERVER
http://localhost:8080

cat << EOF > topology-sample.json
{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ip-172-20-37-51.ap-south-1.compute.internal"
                            ],
                            "storage": [
                                "35.154.133.103"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        {
                            "name": "/dev/sdb"
                        }
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ip-172-20-56-172.ap-south-1.compute.internal"
                            ],
                            "storage": [
                                "52.66.145.45"
                            ]
                        },
                        "zone": 2
                    },
                    "devices": [
                        {
                            "name": "/dev/sdb"
                        }
                    ]
                }
            ]
        }
    ]
}
EOF

wget https://github.com/heketi/heketi/releases/download/v7.0.0/heketi-client-v7.0.0.darwin.amd64.tar.gz
tar -xvzf heketi-client-v7.0.0.darwin.amd64.tar.gz

cd heketi-client/bin
./heketi-cli topology load --json=topology-sample.json