How to connect an AWS Cluster for tracking K8s Costs

 

Our Application has a unique solution that helps with tracking Kubernetes costs, understanding them better and easily identifying wasted spend.

Our Kubernetes feature assigns pod costs as the maximum between the actual cost and the requirements cost (if it exists).

To show the actual cost, you need to install an agent in each cluster that you want to connect.


As the CloudWatch custom metric is very expensive, we have taken the AWS CloudWatch agent for containers and made a minor adjustment so our agent can now send the container logs to CloudWatch without having to use the CloudWatch custom metric.

Before you install Container Insights on Amazon EKS or Kubernetes, verify the following:

  • You have a functional Amazon EKS or Kubernetes cluster with nodes attached in one of the Regions that supports the Container Insights for Amazon EKS and Kubernetes. For the list of supported Regions, see Using Container Insights.
  • K8s version is 1.22 or higher.
  • You have kubectl installed and running. For more information, see Installing kubectl in the Amazon EKS User Guide.
  • If you're using Kubernetes running on AWS instead of using Amazon EKS, the following prerequisites are also necessary:
    • Be sure that your Kubernetes cluster has enabled role-based access control (RBAC). For more information, see Using RBAC Authorization in the Kubernetes Reference. 
    • Your kubelet has enabled Webhook authorization mode. For more information, see Kubelet authentication/authorization in the Kubernetes Reference.
    • Your container runtime is Docker.
You must also grant IAM permissions to enable your Amazon EKS worker nodes to send metrics and logs to CloudWatch. There are two ways to do this:
  • Attach a policy to the IAM role of your worker nodes. This works for both Amazon EKS clusters and other Kubernetes clusters.
  • Use an IAM role for service accounts for the cluster, and attach the policy to this role. This works only for Amazon EKS clusters.
The first option grants permissions to CloudWatch for the entire node, while using an IAM role for the service account gives CloudWatch access to only the appropriate daemonset pods.
 

Attaching a policy to the IAM role of your worker nodes

Follow the steps below to attach the policy to the IAM role of your worker nodes. This works for both Amazon EKS clusters and Kubernetes clusters outside of Amazon EKS. 

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
  2. Select one of the worker node instances and choose the IAM role in the description.
  3. On the IAM role page, choose Attach policies.
  4. In the list of policies, select the check box next to CloudWatchAgentServerPolicy. If necessary, use the search box to find this policy.
  5. Choose Attach policies.
If you're running a Kubernetes cluster outside Amazon EKS, you might not already have an IAM role attached to your worker nodes. If not, you must first attach an IAM role to the instance and then add the policy as explained in the previous steps.
 
For more information on attaching a role to an instance, see Attaching an IAM Role to an Instance in 
the Amazon EC2 User Guide for Windows Instances.
 
If you're running a Kubernetes cluster outside Amazon EKS and you want to collect EBS volume IDs in the metrics, you must add another policy to the IAM role attached to the instance. Add the following as an inline policy. For more information, see Adding and Removing IAM Identity Permissions in the IAM User Guide. 
{

   "Version": "2012-10-17",
   "Statement": [
       {
           "Action": [
               "ec2:DescribeVolumes"
            ],
           "Resource": "*",
           "Effect": "Allow"
       }
   ]
}
 
To Install the agent just follow the instructions on the following link: https://github.com/pileus-cloud/charts/tree/main/helm-chart-sources/anodot-cost-k8s-cw-agent 
 
After installing the Agent, verify that the installation was successful: access CloudWatch and see that new logs were actually created under the following Log Group: containerinsight/{cluster name}/performance
 
The logs should be created within 30 min from the Agent installation.