Waiting & BN

Solution: Kubernetes + AWS + peering

Wei -

Introduction

Kubernetes definitely a cool thing, while it replication controller, service and rolling update is extremely useful component and make your deployment easier. However, when it is combine with AWS, it has slightly different with deploying on GKE.

Entrypoint

Setting up Kubernetes on AWS is easy. I recommend clone the kubernetes form github then run kube-up.sh. Of course, you need to export your environment variable before you run kube-up.sh.

My Story

Recently, we are seeking another place to host our service. What we need is a solution to connect to company cooperate network through existing VPC. But according to Amazon's document. This is not going to work. Because transitive connection is not allow in peering connection. However, I have learned that it is still possible if you just want to connect to the service in the VPC you are peering to.

Solution for connection to another VPC

Before starting

Before you start, you need to know why there is a problem while you want to connect to another VPC. Because kubernetes creates a new VPC when you run kube-up.sh. Their starting point is good, it isolate from other service in other VPC so whatever you did in kubernetes is not going to interrupt other service.

Solutions

I can provide two solution.

  1. Create kubernetes on existing VPC by specify VPC id with export VPC_ID=vpc-YOUR_VPC_ID. I won't recommend this unless you know crystal clear what your are doing. The draw back of this is when you run kube-down.sh, it is going to remove cluster. So be careful to use this.

  2. Use peering to connect two VPC.

Prerequisite

Before create connect peering connection, you need to make sure you have no overlap ip range between two VPC. Becuase kubernetes will consider all the ip address like 10.X.X.X is from itself. So when you try to reach ip address, for example, 10.4.32.14, it will redirect to itself and never goes out.

How

I am note sure how to modify ip range after you create kubernetes. but I know how to do it before kube-up.sh.

First you need to export environment variables about ip range. You can reference to here and see the NON_MASQUERADE_CIDR section. For example, you can do

export NON_MASQUERADE_CIDR="172.16.0.0/14"
export SERVICE_CLUSTER_IP_RANGE="172.16.0.0/16"
export DNS_SERVER_IP="172.16.0.10"
export MASTER_IP_RANGE="172.17.0.0/24"
export CLUSTER_IP_RANGE="172.18.0.0/16"

Next step, you can set up the peering connection. You can reference to here.

In short, setting kup peering connection include three steps.

  1. Create peering connection. It will send request to target VPC. Go to target VPC and accept.

  2. Edit route table, you need to update route table for both VPC.

  3. Edit security group to allow connection.

After doing all of these, you should able to reach another VPC.

Conclusion

Kubernetes team really improve a lot on this include set up on other platform. Thanks! Meanwhile, I really want to say "easy! ah?". But I can't, because the complexity of AWS UI elevate the difficulty from O(1) to O(n^2).