Trying Vitess in kubernetes

Vitess is a database clustering system for horizontal scaling of MySQL, Vitess has been youtube database since 2011. Some requirement:

  • Kubernetes Cluster
  • Helm
  • Dynamic Storage Provisioning
  • etcd Operator
helm install stable/etcd-operator --name operator
  • mysql client
pt-get install mysql-client
  • vtctlclient
go get vitess.io/vitess/go/cmd/vtctlclient

Installing Vitess

Clone repository

https://github.com/vitessio/vitess

Basically, you can run Vitess by installing via helm chart in directory vitess/helm/vitess, but it will only launch a skeleton cluster without any keyspaces (logical databases). So lets use some example from vitess/example/helm

Starting a single keyspace cluster

keyspaces and databases are essentially the same thing when unsharded. You need to edit 101_initial_cluster.yaml first and add Storage Class configuration:

etcd:
  replicas: 1
  resources:

vtctld:
  serviceType: "NodePort"
  resources:

vtgate:
  serviceType: "NodePort"
  resources:

vttablet:
  mysqlSize: "test"
  resources:
  dataVolumeClaimSpec:
    storageClassName: "gluster-heketi-zufar"
    accessModes: ["ReadWriteOnce"]
    resources:
      requests:
        storage: "2Gi"
  mysqlResources:

vtworker:
  resources:

This file create keyspaces commerce with 3 initial table

helm install ../../helm/vitess -f 101_initial_cluster.yaml
ubuntu@pod20-master:~/vitess/examples/helm$ kubectl get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
vtdataroot-zone1-commerce-0-rdonly-0    Bound    pvc-6f6e16c8-4bd9-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   3m41s
vtdataroot-zone1-commerce-0-replica-0   Bound    pvc-6f7a7ab7-4bd9-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   3m41s
vtdataroot-zone1-commerce-0-replica-1   Bound    pvc-6f82b1e3-4bd9-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   3m41s
ubuntu@pod20-master:~/vitess/examples/helm$ kubectl get pod
NAME                                                            READY   STATUS      RESTARTS   AGE
commerce-apply-schema-initial-2ggvl                             0/1     Completed   0          3m43s
commerce-apply-vschema-initial-dzzp6                            0/1     Completed   0          3m43s
etcd-global-4q4dkznszh                                          1/1     Running     0          3m43s
etcd-zone1-72hjgsdmpc                                           1/1     Running     0          3m43s
vtctld-5dc4466d88-4775b                                         1/1     Running     2          3m43s
vtgate-zone1-9c8589688-8nr99                                    1/1     Running     3          3m43s
zone1-commerce-0-init-shard-master-xg4fs                        0/1     Completed   0          3m43s
zone1-commerce-0-rdonly-0                                       6/6     Running     0          3m43s
zone1-commerce-0-replica-0                                      6/6     Running     0          3m43s
zone1-commerce-0-replica-1                                      6/6     Running     0          3m43s
ubuntu@pod20-master:~/vitess/examples/helm$ kubectl get jobs
NAME                                 COMPLETIONS   DURATION   AGE
commerce-apply-schema-initial        1/1           89s        3m47s
commerce-apply-vschema-initial       1/1           83s        3m47s
zone1-commerce-0-init-shard-master   1/1           85s        3m47s

if you have install mysql client, you can access:

ubuntu@pod20-master:~/vitess/examples/helm$ ./kmysql.sh
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.5.10-Vitess Percona Server (GPL), Release 23, Revision 500fcf5

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show tables;
+--------------------+
| Tables_in_commerce |
+--------------------+
| corder             |
| customer           |
| product            |
+--------------------+
3 rows in set (0.02 sec)

or manually via

ubuntu@pod20-master:~/vitess/examples/helm$ kubectl get svc | grep vtgate-zone1
vtgate-zone1                                             NodePort    10.100.190.111   <none>        15001:31347/TCP,15991:31458/TCP,3306:30114/TCP   19m
ubuntu@pod20-master:~/vitess/examples/helm$ mysql -h 10.100.190.111
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.5.10-Vitess Percona Server (GPL), Release 23, Revision 500fcf5

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show tables
    -> ;
+--------------------+
| Tables_in_commerce |
+--------------------+
| corder             |
| customer           |
| product            |
+--------------------+
3 rows in set (0.01 sec)

lets add some data into table

ubuntu@pod20-master:~/vitess/examples/helm$ ./kmysql.sh < ../common/insert_commerce_data.sql
ubuntu@pod20-master:~/vitess/examples/helm$ mysql -h 10.100.190.111 --table < ../common/select_commerce_data.sql
Using commerce/0
Customer
+-------------+--------------------+
| customer_id | email              |
+-------------+--------------------+
|           1 | alice@domain.com   |
|           2 | bob@domain.com     |
|           3 | charlie@domain.com |
|           4 | dan@domain.com     |
|           5 | eve@domain.com     |
|           6 | alice@domain.com   |
|           7 | bob@domain.com     |
|           8 | charlie@domain.com |
|           9 | dan@domain.com     |
|          10 | eve@domain.com     |
|          11 | alice@domain.com   |
|          12 | bob@domain.com     |
|          13 | charlie@domain.com |
|          14 | dan@domain.com     |
|          15 | eve@domain.com     |
+-------------+--------------------+
Product
+----------+-------------+-------+
| sku      | description | price |
+----------+-------------+-------+
| SKU-1001 | Monitor     |   100 |
| SKU-1002 | Keyboard    |    30 |
+----------+-------------+-------+
COrder
+----------+-------------+----------+-------+
| order_id | customer_id | sku      | price |
+----------+-------------+----------+-------+
|        1 |           1 | SKU-1001 |   100 |
|        2 |           2 | SKU-1002 |    30 |
|        3 |           3 | SKU-1002 |    30 |
|        4 |           4 | SKU-1002 |    30 |
|        5 |           5 | SKU-1002 |    30 |
+----------+-------------+----------+-------+

Vertical Split

Create new keyspace name customer

export release=$(helm ls -q)
helm upgrade $release ../../helm/vitess/ -f 201_customer_keyspace.yaml

create customer tablets (edit the file first and add storage class configuration)

helm upgrade $release ../../helm/vitess/ -f 202_customer_tablets.yaml
ubuntu@pod20-master:~/vitess/examples/helm$ kubectl get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
vtdataroot-zone1-commerce-0-rdonly-0    Bound    pvc-6f6e16c8-4bd9-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   48m
vtdataroot-zone1-commerce-0-replica-0   Bound    pvc-6f7a7ab7-4bd9-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   48m
vtdataroot-zone1-commerce-0-replica-1   Bound    pvc-6f82b1e3-4bd9-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   48m
vtdataroot-zone1-customer-0-rdonly-0    Bound    pvc-33345d5b-4be0-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   15s
vtdataroot-zone1-customer-0-replica-0   Bound    pvc-33202393-4be0-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   15s
vtdataroot-zone1-customer-0-replica-1   Bound    pvc-33222b82-4be0-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   15s

migrating the data by copying from commerce to customer.

helm upgrade $release ../../helm/vitess/ -f 203_vertical_split.yaml
ubuntu@pod20-master:~/vitess/examples/helm$ ./kmysql.sh --table < ../common/select_commerce_data.sql 
Using commerce/0
Customer
+-------------+--------------------+
| customer_id | email              |
+-------------+--------------------+
|           1 | alice@domain.com   |
|           2 | bob@domain.com     |
|           3 | charlie@domain.com |
|           4 | dan@domain.com     |
|           5 | eve@domain.com     |
|           6 | alice@domain.com   |
|           7 | bob@domain.com     |
|           8 | charlie@domain.com |
|           9 | dan@domain.com     |
|          10 | eve@domain.com     |
|          11 | alice@domain.com   |
|          12 | bob@domain.com     |
|          13 | charlie@domain.com |
|          14 | dan@domain.com     |
|          15 | eve@domain.com     |
+-------------+--------------------+
Product
+----------+-------------+-------+
| sku      | description | price |
+----------+-------------+-------+
| SKU-1001 | Monitor     |   100 |
| SKU-1002 | Keyboard    |    30 |
+----------+-------------+-------+
COrder
+----------+-------------+----------+-------+
| order_id | customer_id | sku      | price |
+----------+-------------+----------+-------+
|        1 |           1 | SKU-1001 |   100 |
|        2 |           2 | SKU-1002 |    30 |
|        3 |           3 | SKU-1002 |    30 |
|        4 |           4 | SKU-1002 |    30 |
|        5 |           5 | SKU-1002 |    30 |
+----------+-------------+----------+-------+
ubuntu@pod20-master:~/vitess/examples/helm$ ./kmysql.sh --table < ../common/select_customer0_data.sql 
Using customer/0
Customer
+-------------+--------------------+
| customer_id | email              |
+-------------+--------------------+
|           1 | alice@domain.com   |
|           2 | bob@domain.com     |
|           3 | charlie@domain.com |
|           4 | dan@domain.com     |
|           5 | eve@domain.com     |
|           6 | alice@domain.com   |
|           7 | bob@domain.com     |
|           8 | charlie@domain.com |
|           9 | dan@domain.com     |
|          10 | eve@domain.com     |
|          11 | alice@domain.com   |
|          12 | bob@domain.com     |
|          13 | charlie@domain.com |
|          14 | dan@domain.com     |
|          15 | eve@domain.com     |
+-------------+--------------------+
COrder
+----------+-------------+----------+-------+
| order_id | customer_id | sku      | price |
+----------+-------------+----------+-------+
|        1 |           1 | SKU-1001 |   100 |
|        2 |           2 | SKU-1002 |    30 |
|        3 |           3 | SKU-1002 |    30 |
|        4 |           4 | SKU-1002 |    30 |
|        5 |           5 | SKU-1002 |    30 |
+----------+-------------+----------+-------+

Once you have verified that the customer and corder tables are being continuously updated from commerce, you can cutover the traffic.

Cut Over

This is typically performed in three steps: rdonly, replica and master:

helm upgrade $release ../../helm/vitess/ -f 204_vertical_migrate_replicas.yaml
helm upgrade $release ../../helm/vitess/ -f 205_vertical_migrate_master.yaml

Once this is done, the customer and corder tables are no longer accessible in the commerce keyspace. You can verify this by trying to read from them.

ubuntu@pod20-master:~/vitess/examples/helm$ ./kmysql.sh --table < ../common/select_commerce_data.sql
Using commerce/0
Customer
ERROR 1105 (HY000) at line 4: vtgate: http://vtgate-zone1-9c8589688-8nr99:15001/: target: commerce.0.master, used tablet: zone1-1564760600 (zone1-commerce-0-replica-0.vttablet), vttablet: rpc error: code = FailedPrecondition desc = disallowed due to rule: enforce blacklisted tables (CallerID: userData1)
ubuntu@pod20-master:~/vitess/examples/helm$ 

Clean Up

helm upgrade $release ../../helm/vitess/ -f 206_clean_commerce.yaml
ubuntu@pod20-master:~/vitess/examples/helm$ mysql -h 10.100.190.111
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 17
Server version: 5.5.10-Vitess Percona Server (GPL), Release 23, Revision 500fcf5

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+-----------+
| Databases |
+-----------+
| commerce  |
| customer  |
+-----------+
2 rows in set (0.00 sec)
mysql> use commerce/0;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+--------------------+
| Tables_in_commerce |
+--------------------+
| product            |
+--------------------+
1 row in set (0.00 sec)

mysql> use customer/0;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+--------------------+
| Tables_in_customer |
+--------------------+
| corder             |
| customer           |
+--------------------+
2 rows in set (0.01 sec)

now table is splited in difference keyspace (virtual database)  handle in difference pod and difference pvc

ubuntu@pod20-master:~/vitess/examples/helm$ kubectl get pod
zone1-commerce-0-init-shard-master-xg4fs                        0/1     Completed   0          60m
zone1-commerce-0-rdonly-0                                       6/6     Running     0          60m
zone1-commerce-0-replica-0                                      6/6     Running     0          60m
zone1-commerce-0-replica-1                                      6/6     Running     0          60m
zone1-customer-0-init-shard-master-zrppf                        0/1     Completed   0          11m
zone1-customer-0-rdonly-0                                       6/6     Running     0          11m
zone1-customer-0-replica-0                                      6/6     Running     0          11m
zone1-customer-0-replica-1                                      6/6     Running     0          11m
ubuntu@pod20-master:~/vitess/examples/helm$ kubectl get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
vtdataroot-zone1-commerce-0-rdonly-0    Bound    pvc-6f6e16c8-4bd9-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   61m
vtdataroot-zone1-commerce-0-replica-0   Bound    pvc-6f7a7ab7-4bd9-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   61m
vtdataroot-zone1-commerce-0-replica-1   Bound    pvc-6f82b1e3-4bd9-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   61m
vtdataroot-zone1-customer-0-rdonly-0    Bound    pvc-33345d5b-4be0-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   13m
vtdataroot-zone1-customer-0-replica-0   Bound    pvc-33202393-4be0-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   13m
vtdataroot-zone1-customer-0-replica-1   Bound    pvc-33222b82-4be0-11e9-b6f9-fa163e20bd2f   2Gi        RWO            gluster-heketi-zufar   13m

Comments are closed.