跳到主要内容

Use Cloud Billing Source to retrieve bills from cloud vendors

· 阅读需 3 分钟

Overview

Billings from cloud vendors allow users to observe the cost of resource.

Vanus obtains bills by using api from various cloud vendors. It uses Elasticsearch for data storage to achieve unified management of bills. It also notifies the team of abnormal expenses via like Slack Channel. aws-billing In this tutorial, you will learn how to use Cloud Billing Source of Vanus to aquire the billing from Cloud Service providers like AWS, and store the data in Elasticsearch.

AWS Billing to Elasticsearch integration

Prerequisites

  • AWS IAM Access Key.

  • AWS permissions ce:GetCostAndUsage for the IAM user.

  • Elasticsearch and Kibana are up and running

  • Go to Vanus Playground :an online K8s environment where Vanus can be deployed.

Step 1: Deploying Vanus

  1. Login Vanus Playground.

  2. Refer to the Quick Start document to complete the Install Vanus & Install vsctl.

  3. Create an eventbus

    ~ # vsctl eventbus create --name aws-billing
    +----------------+-------------+
    | RESULT | EVENTBUS |
    +----------------+-------------+
    | Create Success | aws-billing |
    +----------------+-------------+

Step 2: Deploy the AWS Billing Source

Use command line create AWS Billing Source

  1. Create the config file

    cat << EOF > config.yml
    target: http://192.168.49.2:30002/gateway/aws-billing
    secret:
    access_key_id: AKIAIOSFODNN7***MPLE
    secret_access_key: wJalrXUtnFEMI/K7MDENG/bP**iCYEXAMPLEKEY
    EOF
  2. Start with Docker

    docker run -it --rm --network=host \
    -v ${PWD}:/vanus-connect/config \
    --name source-aws-billing public.ecr.aws/vanus/connector/source-aws-billing

Step 3: Deploy the Elasticsearch Sink

Use the command line of Vanus create the event target: Elasticsearch Sink

  1. Create a yml file named sink-es.yml in the playground with the following command:

    cat << EOF > sink-es.yml
    apiVersion: v1
    kind: Service
    metadata:
    name: sink-es
    namespace: vanus
    spec:
    selector:
    app: sink-es
    type: ClusterIP
    ports:
    - port: 8080
    name: sink-es
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: sink-es
    namespace: vanus
    data:
    config.yml: |-
    port: 8080
    es:
    address: "http://localhost:9200"
    index_name: "vanus_test"
    username: "elastic"
    password: "elastic"
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: sink-es
    namespace: vanus
    labels:
    app: sink-es
    spec:
    selector:
    matchLabels:
    app: sink-es
    replicas: 1
    template:
    metadata:
    labels:
    app: sink-es
    spec:
    containers:
    - name: sink-es
    image: public.ecr.aws/vanus/connector/sink-elasticsearch
    imagePullPolicy: Always
    ports:
    - name: http
    containerPort: 8080
    volumeMounts:
    - name: config
    mountPath: /vanus-connect/config
    volumes:
    - name: config
    configMap:
    name: sink-es
    EOF
  2. Replace the config value with yours.

    es:
    address: "http://localhost:9200"
    index_name: "vanus_test"
    username: "elastic"
    password: "elastic"
  3. Run es sink in kubernetes.

    kubectl apply -f sink-es.yaml

Step 4: Create Subscription

Create Subscription Event, here can do filter before deliver event to sink end, execute the following command:

vsctl subscription create \
--eventbus aws-billing \
--sink 'http://sink-es:8080' \
--filters '[
{
"suffix": {
"source":".billing"
}
}
]'
  • sink Points to the target end of the event delivery.
  • filters The method of filter the events.
    • use keyword suffix to do the postfix match for attribute source

Step 5: Result Checking

Check the Billing Data. Now let's connect to kibana so that we can manage data and perform data analysis.

billing-kibana

Summary

In this tutorial, we learned how to integrate AWS Billing Source and Elasticsearch Sink. Of course, Vanus also have connectors for bills of other cloud services , learn more about Vanus Connect.