When you create a new Kubernetes Cluster in EKS - you still need to do several things until you can start using it. This simple set of steps aims to help people create an EKS cluster in simple steps and configure the most essential services like Log forwarding to Cloudwatch, Cert Manager for automatically vending SSL certs from Letsencrypt, and Ingress controller to route external traffic to k8s services.
Create Kubernetes Cluster on EKS
Install awscli (https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
Install aws-iam-authenticator (https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html)
Run aws configure and setup your root user's access key and secret key.
Install eksctl (https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html)
Create a cluster.yaml file in a nice folder.
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: basic-cluster
region: ap-south-1
cloudWatch:
clusterLogging:
enableTypes: ["*"]
iam:
withOIDC: true
nodeGroups:
- name: ng-1
instanceType: t3.small
desiredCapacity: 2
volumeSize: 80
- name: ng-2
instanceType: t2.small
desiredCapacity: 2
volumeSize: 100
- Create the cluster using eksctl (make sure you have awscli and creds setup already)
eksctl create cluster -f cluster.yaml
- Create kubeconfig to access the cluster
mkdir ~/.kube
eksctl utils write-kubeconfig --cluster=basic-cluster --kubeconfig=~/.kube/config
More info at https://eksctl.io/usage/creating-and-managing-clusters/
Adding vpc-cni addon
vpc-cni addon allows EKS to provision more pods on a node. In the below code, replace the service account ARN with the role from your AWS IAM and change basic-cluster with your cluster name.
eksctl create addon --name vpc-cni --version 1.11.4-eksbuild.1 --cluster basic-cluster \
--service-account-role-arn arn:aws:iam::111122223333:role/AmazonEKSVPCCNIRole --force
More info at https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html
Deploy Nginx ingress controller on the cluster
The following command will create the required resources for Nginx Ingress Controller and also create a LoadBalancer Service. The Load Balancer should create an External Load Balancer on AWS and point it to our ingress controller.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/aws/deploy.yaml
BONUS:
When you deploy nginx and created the load balancer, get the endpoint for the same using kubectl get svc --namespace=nginx-ingress
You can now point your domain to this endpoint using CNAME records (ex - *.k8s.mridulganga.dev -> aaa71bxxxxx-11xxxxx10.us-east-1.elb.amazonaws.com).
To validate the above configuration, you can curl to the endpoint and expect to see a 404 error from the Nginx curl https://xyz.k8s.mridulganga.dev
More info at https://kubernetes.github.io/ingress-nginx/deploy/#aws
Deploy cert-manager & create cluster issuer
Follow the installation instructions on the cert-manager website https://cert-manager.io/docs/installation/
Create ClusterIssuer Save the following content in a file letsencrypt-staging.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: [email protected]
privateKeySecretRef:
name: letsencrypt-staging
solvers:
- http01:
ingress:
class: nginx
Create the resource using kubectl kubectl apply -f letsencrypt-staging.yaml
NOTE: use staging when you are first trying out certs, when you are comfortable with the cert vending process - you can move to letsencrypt-prod by creating a new Issuer using the acme server: https://acme-v02.api.letsencrypt.org/directory. Check the rate limits https://letsencrypt.org/docs/rate-limits/
Enable Cloudwatch log forwarding using Fluent Bit
If you want to forward logs from your pods to cloudwatch, then run the following command to enable that using fluent bit daemons. Replace the variables in the command with your cluster info.
ClusterName=<my-cluster-name>
RegionName=<my-cluster-region>
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/;s/{{region_name}}/'${RegionName}'/;s/{{http_server_toggle}}/"'${FluentBitHttpServer}'"/;s/{{http_server_port}}/"'${FluentBitHttpPort}'"/;s/{{read_from_head}}/"'${FluentBitReadFromHead}'"/;s/{{read_from_tail}}/"'${FluentBitReadFromTail}'"/' | kubectl apply -f -
More info at https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-EKS-quickstart.html
Image Credits - https://www.opensourceforu.com/2018/06/container-orchestration-kubernetes/
