ONOS SONA-CNI Tutorial

SONA-CNI is CNI  for kubernetes design to meet telco requirement, using openvswitch to handle L2 and L3 connectivity and aim provide line-rate performance.

This article uses a different approach from the official documentation about SONA-CNI for kubernetes. The official document use centos and merge the ONOS and master nodes, I try to split the function.

  • Ubuntu 18.04
  • 1 Master Node
  • 2 Worker Node
  • 1 ONOS Node

IP Address Mapping

  • ONOS Node (ens3: 10.100.100.100)
  • Master Node (ens3: 10.100.100.101, ens4: no IP address)
  • Worker0 Node (ens3: 10.100.100.102, ens4: no IP address)
  • Worker1 Node (ens3: 10.100.100.103, ens4: no IP address)

Kubernetes Installation

All Kubernetes Nodes

Install docker, conntrack, and openvswitch package

sudo apt update; sudo apt upgrade -y; sudo apt autoremove -y
sudo apt install -y docker.io; sudo docker version
systemctl enable docker.service

sudo apt install -y conntrack
modprobe nf_conntrack_ipv4
conntrack -L

sudo apt install openvswitch-switch -y
sudo systemctl start openvswitch-switch
sudo systemctl enable openvswitch-switch
sudo systemctl status openvswitch-switch

Enable ovsdb-server in openvswitch

sudo sed -i '/set ovsdb-server \"$DB_FILE\"/a \        set \"$@\" --remote=ptcp:6640' /usr/share/openvswitch/scripts/ovs-ctl
sudo systemctl restart openvswitch-switch

Enable bridged packets will traverse iptables rules and ip forwarding

sudo bash -c 'cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF'
sudo sysctl --system

Bootstraping kubernetes (I am using kubeadm)

sudo kubeadm init --pod-network-cidr=20.20.0.0/16

Delete kube-proxy daemonset

kubectl delete ds kube-proxy -n kube-system

Noted the token

kubeadm token list

Remove iptables rules from kubernetes nodes

sudo iptables -t nat -F
sudo iptables -F
sudo iptables -X
sudo iptables -L

SONA-CNI Installation

All Kubernetes Nodes

install python

sudo apt install python-minimal python-pip -y

Clone SONA-CNI

git clone https://github.com/sonaproject/sona-cni.git && cd sona-cni
sudo pip install -r requirements.txt

Create configuration files, change

  • url_path, API url ONOS for kubernetes networking
  • username, Username ONOS
  • password, Password ONOS
  • external_interface, Interface in kubernetes nodes thats going into router (see the topology)
  • external_gateway_ip, Default gateway of external interface
cat << EOF > etc/sona/sona-cni.conf
# Configuration options for ONOS CNI plugin endpoint
 
[onos]
# (StrOpt) ONOS REST interface URL. This is a mandatory field.
url_path = http://10.100.100.100:8181/onos/k8snetworking
# (StrOpt) Username for authentication. This is a mandatory field.
username = onos
# (StrOpt) Password for authentication. This is a mandatory field.
password = rocks
 
[network]
# (StrOpt) Default overlay network type (VXLAN, GRE, GENEVE). This is an optional field, VXLAN is the default value.
# type = VXLAN
# (StrOpt) Default segment identifier of the network. This is an optional field, 100 is the default value.
# segment_id = 100
# (StrOpt) External uplink interface name. This is a mandatory field.
external_interface = ens4
# (StrOpt) External gateway IP address. This is a mandatory field.
external_gateway_ip = 10.101.101.1
# (StrOpt) Transient network CIDR. This is an optional field. If not specified 172.10.0.0/16 will be used in stread.
# transient_cidr = 172.10.0.0/16
# (StrOpt) Service network CIDR. This is an optional field. If not specified 10.96.0.0/12 will be used instead.
# service_cidr = 10.96.0.0/12
# (StrOpt) Network Maximum Transmission Unit (MTU). This is a mandatory field.
mtu = 1400
EOF

Install SONA-CNI

python setup.py install

Master Node

copy kubeconfig into worker node and ONOS Nodes

ssh root@10.100.100.100 mkdir ~/.kube/
ssh root@10.100.100.102 mkdir ~/.kube/
ssh root@10.100.100.103 mkdir ~/.kube/
scp ~/.kube/config root@10.100.100.100:~/.kube/config
scp ~/.kube/config root@10.100.100.102:~/.kube/config
scp ~/.kube/config root@10.100.100.103:~/.kube/config

All Kubernetes Nodes

Run config-external.py inside sona-cni folder

python config-external.py

Building ONOS

ONOS Node

We using ONOS 1.15, first update and install java

apt update -y
apt upgrade -y
apt install git zip curl unzip python-minimal openjdk-8-jdk -y

Install package for Bezel

sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python3 -y

Create User sdn

adduser sdn
passwd -d sdn
cat <<EOF >> /etc/sudoers
sdn ALL=(ALL) NOPASSWD:ALL
EOF

Change user to sdn

su sdn
cd ~

Install Bazel

wget https://github.com/bazelbuild/bazel/releases/download/0.23.2/bazel-0.23.2-installer-linux-x86_64.sh
chmod +x bazel-0.23.2-installer-linux-x86_64.sh
./bazel-0.23.2-installer-linux-x86_64.sh --user

export PATH="$PATH:$HOME/bin"

Clone & build ONOS

git clone https://gerrit.onosproject.org/onos
cd onos
git checkout onos-1.15

bazel build onos

Copy and extract ONOS

sudo cp bazel-bin/onos.tar.gz /opt/onos.tar.gz
sudo su

cd /opt
tar xzf onos.tar.gz
mv onos-1.15.1-SNAPSHOT onos
chown -R sdn:sdn onos

Create ONOS Options

sudo -u sdn nano /opt/onos/options
...
# running onos with user sdn
export ONOS_USER=sdn
 
# default active drivers and openflow
export ONOS_APPS=drivers,openflow-base,k8s-networking
...

Start ONOS

sudo cp /opt/onos/init/onos.initd /etc/init.d/onos
sudo cp /opt/onos/init/onos.service /etc/systemd/system/

sudo systemctl daemon-reload
sudo systemctl enable onos
sudo systemctl start onos

If you want to access ONOS

ssh-keygen -t rsa
/opt/onos/bin/onos-user-key $USER ~/.ssh/id_rsa.pub

Setup kubernetes in ONOS, take the token and certificate data

cd ~
cat ~/.kube/config

cat << EOF >> api-cfg-k8s.json
{
  "apiConfigs" : [
    {
      "scheme" : "HTTPS",
      "ipAddress" : "10.100.100.101",
      "port" : 6443,
      "token": "iqg7ru.gd63fqx7iwx3a1tw",
      "caCertData": "",
      "clientCertData": "",
      "clientKeyData": ""
    }
  ]
}
EOF

After you fill the token and certificate data, send into ONOS node via API

curl --user onos:rocks -X POST -H "Content-Type: application/json" http://10.100.100.100:8181/onos/k8snode/configure/api -d @api-cfg-k8s.json

All Kubernetes Nodes

config route by running config-route.py in sona-cni directory

sudo python config-route.py

Master Node

trying to create deployment and services

kubectl create deployment nginx --image=nginx
kubectl scale --replicas=5 deployment/nginx
kubectl create service clusterip nginx --tcp=80:80

kubectl create deployment nginx2 --image=nginx
kubectl scale --replicas=2 deployment/nginx2
kubectl create service nodeport nginx2 --tcp=80

ONOS Node

login into ONOS CLI and sync the status

/opt/onos/bin/onos

k8s-sync-states
k8s-sync-rules

 

Comments are closed.