Configure Node Exporter Sidecar to Collect Instance Metadata

AWS: Steps for Self Hosted K8s on EC2

Edit node exporter Daemonset to add initContainer to collect instance metadata

Eg:

CODE

kubectl edit daemonset prometheus-prometheus-node-exporter -n prometheus

Sidecar calls the instance metadata API and pushes the data into the .prom file

i.e /textfile_collector/ec2_metadata.prom

CODE

      initContainers:
      - args:
        - |
          mkdir -p /textfile_collector
          METRICS_FILE="/textfile_collector/ec2_instance_metadata.prom"
          # Fetch IMDSv2 Token
           TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
          fetch() {
            curl -s -H "X-aws-ec2-metadata-token: $TOKEN" "http://169.254.169.254/latest/meta-data/$1"
          }
          # Fetch EC2 Metadata using the token
          INSTANCE_ID=$(fetch instance-id)
          INSTANCE_TYPE=$(fetch instance-type)
          AVAILABILITY_ZONE=$(fetch placement/availability-zone)
          REGION=$(echo "$AVAILABILITY_ZONE" | sed 's/.$//')
          LOCAL_HOSTNAME=$(fetch local-hostname)
          PUBLIC_HOSTNAME=$(fetch public-hostname)
          PUBLIC_IPV4=$(fetch public-ipv4)
          RESERVATION_ID=$(fetch reservation-id)
          AZ_ID=$(fetch placement/availability-zone-id)
          # PLACEMENT_GROUP=$(fetch placement/group-name)
          # HOST_ID=$(fetch placement/host-id)
          HOSTNAME=$(fetch hostname)
          esc() {
            echo "$1" | sed 's/\\\\/\\\\\\\\/g; s/\"/\\\\\"/g'
          }
          # Write to Prometheus Textfile Collector
          cat <<EOF > $METRICS_FILE
          ec2_instance_metadata{instance_id="$(esc "$INSTANCE_ID")",instance_type="$(esc "$INSTANCE_TYPE")",region="$(esc "$REGION")",az="$(esc "$AVAILABILITY_ZONE")",az_id="$(esc "$AZ_ID")",hostname="$(esc "$HOSTNAME")",local_hostname="$(esc "$LOCAL_HOSTNAME")",public_hostname="$(esc "$PUBLIC_HOSTNAME")",public_ipv4="$(esc "$PUBLIC_IPV4")",reservation_id="$(esc "$RESERVATION_ID")"} 1
          EOF
        command:
        - /bin/sh
        - -c
        image: alpine/curl:latest
        imagePullPolicy: Always
        name: fetch-ec2-metadata
        volumeMounts:
        - mountPath: /textfile_collector
          name: textfile-collector

and volume to store the file under volumes section.

CODE

      volumes:
      - hostPath:
          path: /var/lib/node_exporter/textfile_collector
          type: DirectoryOrCreate
        name: textfile-collector

The Node Exporter reads the metrics file and exposes the data as metrics, which Prometheus then scrapes (pulls) during its scheduled intervals.

CODE

      containers:
        - name: node-exporter
          image: quay.io/prometheus/node-exporter:latest
          args:
            - "--collector.textfile.directory=/textfile_collector"

Delete the prometheus pod to start the scrapping immediately

CODE

kubectl delete pod prometheus-server-cff59876f-8qfs4 -n prometheus

Verification Steps:

verify if Prometheus has ec2_instance_metadata upon querying

http://{prometheus_url}/api/v1/query?query=ec2_instance_metadata