Install Kubernetes Cluster - Adding Monitoring and Service Mesh for Observability (Part 2)-

With the core cluster up and running in part 1, it’s time to bring in some enterprise-grade tools using Helm. In this part, we’ll add monitoring with Prometheus and Grafana, service mesh capabilities with Linkerd.

Clone the repository

First you need to ssh to your control plane node and clone the repo that includes all the files you need for this setup.

git clone https://github.com/fresher-to-uber/k8s-cluster-setup.gitcd k8s-cluster-setup

Install Monitoring with Prometheus and Grafana

Monitoring is a must if you want to see what’s really happening inside your cluster. With Helm, setting up Prometheus and Grafana is as easy as a couple of commands.

First, we need to install Helm

# Ensure correct architectureexport CLI_ARCH=$(dpkg --print-architecture)# Add Helmwget https://get.helm.sh/helm-v3.15.4-linux-$CLI_ARCH.tar.gztar -xf helm-v3.15.4-linux-$CLI_ARCH.tar.gzsudo cp linux-$CLI_ARCH/helm /usr/local/bin/

Then install Prometheus stack with helm charts couldn't be easier

helm repo add prometheus-community https://prometheus-community.github.io/helm-chartshelm repo updatekubectl create ns monitoringhelm install prometheus prometheus-community/kube-prometheus-stack -n monitoring

or simple just run the pre-defined script

$ bash monitoring/prometheus.sh

You can search the helm chart here

Access Grafana

Modify grafana service type

kubectl edit svc prometheus-grafana -n monitoring -o yaml

Verify to make sure the change have applied

kubectl get svc prometheus-grafana -n monitoring

You should see the service type as NodePort and a port mapping like 80:3XXXX/TCP.

Deploy a test resource

---apiVersion: apps/v1kind: Deploymentmetadata:  name: nginx-prometheus-test  labels:    app: nginx-prometheus-testspec:  replicas: 1  selector:    matchLabels:      app: nginx-prometheus-test  template:    metadata:      labels:        app: nginx-prometheus-test    spec:      containers:      - name: nginx        image: nginx:1.21        ports:        - containerPort: 80---apiVersion: v1kind: Servicemetadata:  name: nginx-prometheus-test-service  labels:    app: nginx-prometheus-testspec:  selector:    app: nginx-prometheus-test  ports:    - port: 80      targetPort: 80---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  name: nginx-prometheus-test-monitor  labels:    release: prometheusspec:  selector:    matchLabels:      app: nginx-prometheus-test  endpoints:  - port: http     path: /metrics    interval: 15s

If you have cloned the repo, you can find the file in monitoring/ folder. Then just run the command to apply the resource

kubectl apply -f monitoring/nginx-prometheusp-test.yaml

Access Grafana at http://<node-ip>:<node-port> using the default credentials (admin/prom-operator). Select one of the dashboards provided by prometheus

Install Service Mesh with Linkerd

Run each command below:

curl -sL run.linkerd.io/install | shexport PATH=$PATH:$HOME/.linkerd2/binlinkerd check --prelinkerd install --crds | kubectl apply -f -linkerd install | kubectl apply -f -linkerd checklinkerd viz install --set prometheus.enabled=false --set prometheusUrl=http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090 | kubectl apply -f -linkerd viz check

Observe and make sure the output is success with green check mark.

Because we have our own Prometheus, so we need to configure our Prometheus instance to get Linkerd metrics. You can find the scrape config file in linkerd/ folder.

helm upgrade prometheus prometheus-community/kube-prometheus-stack -n monitoring -f linkerd/prometheus-scrape-configs.yaml

Edit Linkerd Dashboard to allow outside access

kubectl edit deploy web -n linkerd-viz

In spec.template.spec.containers.args section, set -enforced-host to empty

Next, change service type to NodePort, same thing we did to Grafana service

kubectl edit svc web -n linkerd-viz -o yaml

Now we can access Linkerd dashboard at http://<node-ip>:<node-port>.

Inject linkerd to the existing nginx-prometheus-test resource that we created in previous section.

kubectl get deploy nginx-prometheus-test -o yaml | \linkerd inject - | kubectl apply -f -

Now we can observe that the pod is meshed in linkerd dashboard

Conclusion

Look at you, adding all these fancy enhancements to your Kubernetes cluster! Now, it’s monitoring, meshing, and ready to handle whatever comes its way (within reason). Sure, it’s a bit more complex, but it’s also way cooler. So go ahead—give yourself a pat on the back for leveling up your Kubernetes skills!

And again you can find the script on github.