Understanding the Concept of Canary Deployment in Kubernetes Part-1

Troubleshooting and Logging in Distroless Images

Signing the Docker Images using Cosign - Part 4

Comparing distroless vs distrobased vs alpine Docker Image on basis of vulnerability scan

Handson Node Application build on Distroless Image Docker - Part 2

Handson Distroless Installation from Scratch - Part 1

Understanding Distroless Container Images

[Solved] Intermittent / burst logs in the Newrelic / ELK

 

Issue:- 

Although the application was writing the logs continuously and shipper shipping but the logs were missing for a particular period and burst of logs with spikes being observed in the Newrelic ELK.

Error:- 

Following graph shows the actual issue of intermittent or burst of the logs in ELK

Effect:-

Due to the non availability of the logs it was becoming difficult to troubleshoot the issue as the logs were getting delayed and sometimes might be missed out as well.

Resolution:-

Printing the error logs or logs required for troubleshooting helps to overcome this issue.

Explanation:-

More than 1million event logs are getting posted in an hour due to which the Disk would be becoming a bottleneck and burst of events are being pushed into the Newrelic ELK.

Lowering down and printing the error logs or logs required for troubleshooting should help to overcome this issue of intermittent logs in the Newrelic/ELK.

[Solved] panic: unable to load configmap based request-header-client-ca-file: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:kubernetes-infra:metrics-server" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

 

Issue:- 

Metric server is unable to load the configmap "extension-apiserver-authentication" and giving forbidden due to which metricserver does show load with kubectl top command.

Error:- 

  panic: unable to load configmap based request-header-client-ca-file: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:kubernetes-infra:metrics-server" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

Effect:-

Metric server doesn't work and kubectl top command fails 

  Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

Resolution:-

Replace

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  labels:

    k8s-app: metrics-server

  name: system:metrics-server

rules:

- apiGroups:

  - ""

  resources:

  - nodes/metrics

  verbs:

  - get

- apiGroups:

  - ""

  resources:

  - pods

  - nodes

  

  WITH

  

  apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  labels:

    k8s-app: metrics-server

  name: system:metrics-server

rules:

- apiGroups:

  - ""

  resources:

  - nodes/metrics

  verbs:

  - get

- apiGroups:

  - ""

  resources:

  - pods

  - nodes

  - nodes/stats

  - namespaces

  - configmaps

Explanation:-

In the latest version of the metricserver , the yaml has reduced resources on which metrics server can authorise. So we need to add the nodes/stats , namespaces and configmaps which are used by the metric server for working. Because of the above missing resource when it tries to contact configmaps it getting an error after the resources in the yaml it as it able to successfully connect to the configmaps and run the top command.