Configure Filebeat for Log Shipping Easily

How to Configure Filebeat for Log Shipping

Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors log files or locations you specify, collects log events, and forwards them to Elasticsearch or Logstash for indexing. In this tutorial, we will cover how to configure Filebeat for efficient log shipping in an ELK (Elasticsearch, Logstash, and Kibana) stack.

Prerequisites

An up-and-running ELK Stack (Official site)
Filebeat installed on the system you want to collect logs from
Basic knowledge of YAML and familiarity with your server’s command line

Step-by-Step Configuration Guide

Step 1: Install Filebeat

First, ensure Filebeat is installed on your server. You can download it from the official Elastic website (Official site).

Step 2: Configuring Filebeat

Edit the filebeat.yml configuration file. You can find this file typically in the /etc/filebeat/ directory.

output.elasticsearch:
  hosts: ["localhost:9200"]
  # Or if using Logstash
output.logstash:
  hosts: ["localhost:5044"]

In the above configuration, replace localhost with the actual IP address of your Elasticsearch or Logstash servers.

Step 3: Define Filebeat Input Configuration

Specify which logs Filebeat will monitor:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

Adjust the paths to match the logs you want to monitor.

Step 4: Setting Paths

Ensure the output path is correctly configured by editing:

path:
  data: /var/lib/filebeat
  logs: /var/log/filebeat

These settings determine how Filebeat manages and stores internal data and its own log files.

Step 5: Test and Validate Configuration

Use the following command to test your configuration:

sudo filebeat test config

If the configuration is valid, you should see Config OK.

Step 6: Start and Enable Filebeat

Finally, start Filebeat and enable it to start on boot:

sudo systemctl start filebeat
sudo systemctl enable filebeat

Troubleshooting Tips

Filebeat not starting: Check the log files in /var/log/filebeat/ for error messages.
Logs not appearing in Elasticsearch: Ensure your output configuration is correct. Double-check that the network is open between Filebeat and your Elasticsearch or Logstash instances.

Summary Checklist

Install Filebeat on the desired server
Configure filebeat.yml with paths and outputs
Test, validate, and start Filebeat

Following these steps allows for efficient log shipping to your ELK stack, enabling centralized logging and monitoring capabilities. For information on installing other components of the Elastic stack, see our guide on Installing Elastic Beats.