Fluent Bit is a data collector service which can be used for collecting data from IoT sensors, or logs from applications running in cloud cluster like Kubernetes, etc.
Fluent Bit with Kibana and Elasticsearch is popularly used for log collection, aggregation and visualization. You can either setup all these 3 services on a Linux server, in a Kubernetes cluster running services within containers or we can even have Fluent Bit running on a Linux machine or Kubernetes cluster(EKS) while Elasticsearch running on AWS.
Why use AWS Elasticsearch?
AWS Elasticsearch is a fully managed Elasticsearch service provided by AWS, where we can configure the number of data nodes requried, number of master nodes required and what type of instance (can be different for both data and master node) to run these nodes and we can start our Elasticsearch cluster.
If you want to use EFK stack for log collection and analysis on production, you should use AWS Elasticsearch service because it's difficult to scale the Elasticsearch service, if you plan to do it yourself.
Rather, have Elasticsearch AWS service, with which Kibana service comes as free plugin, and run a log collection/aggregation service at your end which can be Fluent Bit, Fluentd or Logstash or any other similar service.
When you have Elasticsearch running on AWS, there are two ways to connect it with Fluent Bit service:
By allowing open access for any service to be able to access Elasticsearch service, if the new service is running within the EKS cluster on AWS.
Or, we need to have an AWS IAM role with External Id to access the Elasticsearch service because Fluent Bit service uses the assumeRole API to get access to Elasticsearch running on AWS.
For this introductory tutorial, we will be make our Elasticsearch service to allow open access to the domain, because we have the Fluent Bit running in EKS cluster as Daemonset.
Fluent Bit es Output Plugin
Fluent Bit service provides us with an es output plugin for elasticsearch service to configure Fluent Bit to send output to the configured Elasticsearch service.
Following are the properties that we will be using while configuring Fluent Bit to push data to AWS Elasticsearch service.
[OUTPUT]
Name es
Match *
Host https://vpc-my-elasticsearch-8erHY26sdqudaybp89HCVZey4ifacliy.ap-south-1.es.amazonaws.com
Port 443
AWS_Auth Off
AWS_Region ap-south-1
tls On
Logstash_Format On
Logstash_Prefix my-logs
The additional AWS related property used are:
Host: Here we will provide the VPC endpoint for our AWS Elasticsearch service which would be like https://vpc-my-elasticsearch-8erHY26sdqudaybp89HCVZey4ifacliy.ap-south-1.es.amazonaws.com which you can also use to access your Elasticsearch service.
Port: Port will be 443. Yes, it will not be 9200. AWS Elasticsearch communicates over SSL port.
AWS_Auth: We have set this property to Off because we are not using any kind of authentication to access our AWS Elasticsearch service.
AWS_Region: The region in which you have started your Elasticsearch service. I had mine running in the ap-south-1 region.
tls: We will set this to On.
Other properties available in Fluent Bit are AWS_Role_ARN which provides the AWS IAM Role to assume to put records to your Amazon ES cluster and AWS_External_ID which is the External ID for the AWS IAM Role specified with aws_role_arn.
We will not be using these two properties.
Enable Open Access for Elasticsearch:
Before enabling open access, make sure you have a security group specifying that only the services within the EKS cluster can access the AWS ES cluster.
Go to AWS ES dashboard:
Select your Elasticsearch service and then click on Actions dropdown and choose the option Modify access policy:
And select Allow open access to the domain and you should be good to go.
Once done, your JSON defined access policy should look like the following:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:ap-south-1:98191918321:domain/my-elasticsearch/*"
}
]
}
And with this, our Fluent Bit service should start pushing logs or whatever data you are collecting to the AWS Elasticsearch service. You can see the logs by accessing the Kibana UI which comes along the AWS ES service.
If you face any issues, do share in the comment section below.
You may also like:
