Post

Integrating Zeek, Logstash, and Kibana for Real-Time Network Traffic Analysis

By 0xLuc4s
6 min read

In today’s rapidly evolving digital landscape, maintaining robust network security is paramount. Organizations are increasingly facing sophisticated cyber threats that necessitate advanced tools and methodologies for network monitoring and analysis. This is where the integration of Zeek, Logstash, and Kibana becomes invaluable.

Zeek (formerly known as Bro) is a powerful network analysis framework that excels at parsing network traffic and generating insightful logs. It provides a comprehensive view of network activities, making it an essential tool for security monitoring, troubleshooting, and forensic analysis. However, the raw data generated by Zeek can be vast and complex, necessitating an efficient method to process, store, and visualize this information.

Logstash, a dynamic data processing pipeline, serves this purpose by ingesting the logs generated by Zeek, transforming them as needed, and forwarding them to a storage backend like Elasticsearch. This not only streamlines the data processing workflow but also enables the integration of various data sources into a unified analysis platform.

Kibana, a powerful visualization tool, completes this integration by providing a user-friendly interface to explore and visualize the processed data. With Kibana, users can create custom dashboards, perform detailed searches, and analyze network traffic in real-time, thereby gaining actionable insights to enhance network security.

The synergy of Zeek, Logstash, and Kibana creates a comprehensive network monitoring and analysis solution. This integration allows for real-time detection of anomalies, proactive threat hunting, and detailed post-incident investigations. By leveraging these tools, organizations can significantly enhance their network visibility, streamline security operations, and effectively mitigate cyber threats.

Importance of Real-Time Network Traffic Analysis

  1. Enhanced Security Posture:
    • Real-time network traffic analysis enables the early detection of suspicious activities and potential security breaches. By continuously monitoring network traffic, security teams can identify and respond to threats before they escalate into major incidents.
  2. Proactive Threat Hunting:
    • With tools like Zeek, Logstash, and Kibana, organizations can actively search for indicators of compromise (IoCs) and signs of malicious activities within their networks. This proactive approach helps in identifying and mitigating threats that may have evaded traditional security measures.
  3. Comprehensive Visibility:
    • Integrating these tools provides a holistic view of network activities, allowing security teams to monitor all network interactions, detect anomalies, and understand the context of various network events. This visibility is crucial for effective incident response and forensic investigations.
  4. Efficient Data Management:
    • Logstash facilitates the efficient processing and transformation of large volumes of network data. By structuring and normalizing the data, it ensures that the information is readily available for analysis in Elasticsearch, reducing the complexity and time required for data management.
  5. Insightful Visualizations:
    • Kibana’s visualization capabilities enable security teams to create custom dashboards that highlight key metrics and trends. These visualizations provide a clear and intuitive understanding of network activities, making it easier to identify patterns, detect anomalies, and make informed decisions.
  6. Scalability and Flexibility:
    • The combination of Zeek, Logstash, and Kibana is highly scalable, capable of handling large and complex network environments. This flexibility ensures that the solution can grow with the organization’s needs and adapt to evolving security challenges.
  7. Cost-Effective Solution:
    • Utilizing open-source tools like Zeek, Logstash, and Kibana provides a cost-effective approach to network security monitoring. Organizations can leverage these powerful tools without incurring the high costs associated with proprietary solutions.

In conclusion, integrating Zeek, Logstash, and Kibana offers a robust and comprehensive solution for real-time network traffic analysis. This integration not only enhances an organization’s security posture but also provides valuable insights into network activities, enabling proactive threat detection and efficient incident response. By adopting this approach, organizations can stay ahead of cyber threats and ensure the security and integrity of their digital infrastructure.

Step-by-Step Guide

In the subsequent sections, we will walk you through the process of installing, configuring, and integrating Zeek, Logstash, and Kibana. This guide will cover the essential commands, configuration files, and troubleshooting tips to help you set up a seamless network monitoring and analysis platform.

Prerequisites

  • Ubuntu-based system

  • Basic understanding of network traffic analysis

  • Root or sudo access

Step 1: Installing Zeek

Zeek is a powerful network analysis framework that will be used to monitor network traffic.

  1. Download and Install Zeek:
1
2
3

wget -qO - https://download.zeek.org/bro-pub-key.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get install zeek -y
  1. Enable Promiscuous Mode:
1

sudo ip link set <interface> promisc on

Replace <interface> with your network interface, e.g., enp0s3.

  1. Verify Zeek Installation:
1

zeek -v
  1. Configure Zeek to Monitor Network Traffic: Edit /opt/zeek/etc/node.cfg:
1
2
3
4

[zeek]
type=standalone
host=localhost
interface=<interface>
  1. Start Zeek:
1

zeekctl deploy

Step 2: Installing and Configuring Logstash

Logstash processes and forwards logs from Zeek to Elasticsearch.

  1. Install Logstash:
1
2
3

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list
sudo apt-get update
sudo apt-get install logstash -y
  1. Configure Logstash: Create a configuration file /etc/logstash/conf.d/zeek.conf:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

input {
  file {
    path => "/opt/zeek/logs/current/*.log"
    start_position => "beginning"
  }
}

filter {
  if [path] =~ "conn.log" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\t%{GREEDYDATA:uid}\t%{IP:orig_h}\t%{NUMBER:orig_p}\t%{IP:resp_h}\t%{NUMBER:resp_p}\t%{WORD:proto}\t%{GREEDYDATA:service}\t%{NUMBER:duration}\t%{NUMBER:orig_bytes}\t%{NUMBER:resp_bytes}\t%{WORD:conn_state}\t%{WORD:local_orig}\t%{WORD:local_resp}\t%{NUMBER:missed_bytes}\t%{GREEDYDATA:history}\t%{NUMBER:orig_pkts}\t%{NUMBER:orig_ip_bytes}\t%{NUMBER:resp_pkts}\t%{NUMBER:resp_ip_bytes}\t%{GREEDYDATA:tunnel_parents}" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "zeek-logs-%{+YYYY.MM.dd}"
  }
}
  1. Start Logstash:
1
2

sudo systemctl start logstash
sudo systemctl enable logstash

Step 3: Installing and Configuring Kibana

Kibana visualizes the data collected by Zeek and processed by Logstash.

  1. Install Kibana:
1

sudo apt-get install kibana -y
  1. Configure Kibana: Edit /etc/kibana/kibana.yml to point Kibana to Elasticsearch:
1
2

server.port: 5601
elasticsearch.hosts: ["http://localhost:9200"]
  1. Start Kibana:
1
2

sudo systemctl start kibana
sudo systemctl enable kibana

  1. Access Kibana: Open a web browser and go to http://localhost:5601.

  2. Create Index Pattern in Kibana:

    • Navigate to Management > Stack Management > Index Patterns .

  • Click Create Index Pattern and enter zeek-logs-*.

  • Select @timestamp as the Time Filter field name.

Step 4: Viewing Real-Time Logs in Kibana

To view logs in real-time:

  1. Navigate to Discover:
    • Go to the Discover tab in Kibana.

  • Ensure the index pattern zeek-logs-* is selected.

  • Set the time range to Last 15 minutes .

  • Enable Auto-refresh to update the view automatically.

  1. Verify Logs:
    • You should see logs generated by Zeek, such as conn.log, dns.log, http.log, etc.
  • You can filter and search logs to analyze specific events.

Troubleshooting Tips

  • Ensure all services (Zeek, Logstash, and Kibana) are running.

  • Check Logstash logs for any configuration errors:

1

sudo tail -f /var/log/logstash/logstash-plain.log
  • Verify that Zeek is generating logs in /opt/zeek/logs/current/.

By following these steps, you can successfully integrate Zeek, Logstash, and Kibana to monitor and analyze your network traffic in real-time. This setup provides a powerful platform for network security monitoring and incident response.

See more: https://github.com/GlenDSingh/Ensuring-network-security-with-Zeek

blueteam, monitoring
zeek logstash kibana blueteam monitoring
This post is licensed under CC BY 4.0 by the author.