Demystifying ncp workflow

NCP is the container plugin for NSXT, there could be number of articles about NCP architecture already existing, I intend to cover only the areas that lack coverage, example hyperbus internals in upcoming posts. In this post we will see what happens on NSXT side when you create resources on the kubernetes side.

To ramp-up let me give a quick intro about NCP components.

NCP pod:

This is the core piece of NCP solution. It watches k8s events and talks to NSXT MP to facilitate the resource creation. It can run as a pod or as a process(in PKS environments)

Nsx-Node-agent:

Node-agent is an infra pod(this can also run as a process), runs as a daemonset .so each nodevm will get a replica of node-agent pod. As of NCP 2.5 It has 3 containers/components .
  i) nsx-node-agent
  ii)nsx-kube-proxy
  iii) nsx-ovs

nsx-node-agent container takes care of plumbing the network interfaces onto workload pods. nsx-kube-proxy watches service creation events from k8s api and programs OVS with NAT rules to translate cluster IP to pod endpoints.  nsx-ovs manages OVS process  on each nodevm.

ncp-bootstrap:

This pod readies the nodevm by installing CNI plugin and OVS modules. we no longer need to manually install these components.

Lets now look into the following workflow.

i)TeamA wants to create a deployment with 10 pods.
ii) create network policy in such a way that traffic within the same namespace should be allowed, also allow the traffic from TeamB, reject everything else
iii) Expose the application on the pods as a service. The intention is to access the service from N-S as well as E-W.

we will cover i) in this part, rest will follow in the next part.

Multi tenancy in NCP is implemented using namespaces . So each project should create their own namespaces. Lets create one for TeamA.

root@tmz5p2zp18426:~# kubectl create ns team-a

namespace/team-a created

As I mentioned before, NCP watches for events from k8s api. Here it receives a namespace creation event, so essentially it does the below by interacting with NSXT MP.

1. Create a Tier-1(T1) router

2. Create a Logical switch(LS) and uplink it to the T1 router.

3. Uplink the T1 router to T0 router, advertise the connected routes from T1 router

4. Create/Allocate a SNAT ip on T0 from external ip pools.

     - So basically any traffic coming from pod cidr range will be SNAT'ed to this above allocated IP

5. Request/allocate an IP CIDR from the IPAM block of pod network.

     - pods get ip addresses from this CIDR range.

Now lets create a nginx deployment and set replica count to 10 .

root@tmz5p2zp18426:~# kubectl apply -f nginx.yml -n team-a

deployment.extensions/nginx-deployment created

root@tmz5p2zp18426:~# kubectl get pods -n team-a -o wide

NAME                                READY   STATUS    RESTARTS   AGE   IP              NODE                                   NOMINATED NODE   READINESS GATES

nginx-deployment-85d967b876-5k9pr   1/1     Running   0          10s   172.24.109.5    1f7efff5-5bab-4f09-ba14-4e47309f367b   <none>           <none>

nginx-deployment-85d967b876-67hnc   1/1     Running   0          10s   172.24.109.2    7f395889-1126-41f3-a694-fb7c3c0b72a9   <none>           <none>

nginx-deployment-85d967b876-dnxp4   1/1     Running   0          10s   172.24.109.8    fdbb0bef-07dd-4909-a96f-941e789cd681   <none>           <none>

nginx-deployment-85d967b876-glqhf   1/1     Running   0          10s   172.24.109.6    d59b698d-0142-45b3-8905-691af5393499   <none>           <none>

nginx-deployment-85d967b876-jnlvj   1/1     Running   0          10s   172.24.109.4    ff93493d-b66c-4cbe-8c20-2443d35251f5   <none>           <none>

nginx-deployment-85d967b876-md9h4   1/1     Running   0          10s   172.24.109.11   8589bbd4-cdb7-4476-ab8f-925a39e3a755   <none>           <none>

nginx-deployment-85d967b876-q4zqw   1/1     Running   0          10s   172.24.109.7    4cefc703-27e6-4cfc-87e5-5feb74a2ef72   <none>           <none>

nginx-deployment-85d967b876-qm6r8   1/1     Running   0          10s   172.24.109.10   972c87f9-b8cd-4a79-a4d8-41b081286b80   <none>           <none>

nginx-deployment-85d967b876-xvck4   1/1     Running   0          10s   172.24.109.9    d1d76ee2-976d-4337-b529-c6c0aa31ba22   <none>           <none>

nginx-deployment-85d967b876-zxc45   1/1     Running   0          10s   172.24.109.3    4be4bf7a-c852-421c-8e64-a3f4ecc3e616   <none>           <none>

So what happens under the hood when a pod gets created?

NCP pod does the below by interacting with NSX MP. from(a to e)
a)Requests an IP from the Namespace Subnet
b)Request a MAC from the container MAC pool in NSX
c)Assigns a VLAN for the Pod
d)Creates a logical port on the Namespace LS and assigns the IP, MAC and VLAN to the logical port
e)Adds all K8s Pod Labels as Tags to the logical port

f. MP advertises the LP details to CCP.
g. CCP Pushes the relevant config info to nestdb of hypervisor, LCP learns the new config and creates container port .
h. From LCP hyperbus learns the CIF’s Id/IP/MAC/VLAN binding
i.Kubelet sees a new ‘PodSpec’ from the K8s Master and starts a new Pod. It executes the NSX cni plugin binary to do the ‘network wiring’ of the Pod – This call is proxy’ed to the NSX-Node-agent
j.The NSX-Node-agent gets the CIF’s Id/IP/MAC/VLAN binding data from the Hyperbus over the RPC channel.
k.The Node Agent creates the OVS port with the right VLAN, and configures the Pods network interface veth pair to connect to OVS with the received IP/MAC. After this, Kubelet is un-blocked and the Pod creation succeeds