Configure Netflow/IPFIX Collector on Lumu Virtual Appliance

Configure Netflow/IPFIX Collector on Lumu Virtual Appliance

The Lumu Virtual Appliance (VA) can work alongside your current security infrastructure to provide an even better Continuous Compromise Assessment. The Lumu VA offers the option to create Collectors, a seamless way to integrate with metadata collected by Security Information and Event Management (SIEM) technologies. One of the capabilities of the VA Collectors is to gather and process network flow data. To get started with Virtual Appliance Collectors, access our documentation.

The Lumu Netflow collector collects metadata of flows (Netflow/IPFIX) passing over a network device such as a router. Among other malicious behaviors, network flows provide insights into an organization’s devices that are controlled by adversaries and attempting to move laterally.

In addition to collecting Netflow/IPFIX data sent by network hardware, the Lumu VA is able to collect network flows exported by an externally deployed Packetbeat or other solutions. In this guide, we show you how to configure a Lumu VA for network flow collection sent by a network hardware. We also provide guidance for the cases in which you want to have conversation visibility into your network, but your network hardware does not offer network flow synthesis capabilities.

Disclaimer: In this article, we provide sample configurations that include third party solutions. Your organization should internally review and assess to what extent, if any, the scripts and recommendations we provide in this document will be incorporated into your environment.

Get Started

The Lumu Virtual Appliance is only able to ingest Netflow v5, v9 and v10/IPFIX. The Lumu VA does not support other protocols such as sFlow, and sampled versions of Netflow/IPFIX should not be used as it will negatively affect the ability to catch any malicious behavior from the collected data.

The following two scenarios are an overview of the two scenarios in which Lumu VA can collect network flow information:

Scenario 1 - Set up a network device to export Network flow data

Your network devices may be equipped with the capability of generating and exporting network flows information in Netflow or IPFIX format. This is the preferred method, as synthesizing network flows via hardware is more efficient. For this method, you need to create a Virtual Appliance, add a network flow collector in the VA, then configure your device to send the data to the appliance collector.


Network flow collection with flow-enabled network devicesFigure 1 - Network flow collection with flow-enabled network devices.

Scenario 2 - Set up other Network flow devices/software

In case you do not have equipment in your network that offers such capability of network flow information, you still can leverage a third party software and set up a SPAN mirror port at your disposal to collect network flow data using a third party software configured on an additional machine (as illustrated on figure 2) to send it to Lumu to have conversation visibility into your network.

SPAN (Switched Port Analyzer) is a dedicated port that takes a mirrored copy of network traffic to be sent to a specific destination.
SPAN port schemeFigure 2 - SPAN port scheme.

The Lumu VA comes equipped with everything it needs to receive and process flow data captured by externally deployed Netflow/IPFIX solutions. In this scenario, you will create and set up a network flow VA collector and configure third party tools to send the data to the Lumu Virtual Appliance.

Below in this document you are going to find samples of network flow configuration with nProbe, Softflowd and Packetbeat.

Let’s first review the general steps to set up network flow collection using a Lumu Virtual Appliance Collector.

Set Up a Virtual Appliance

In order to send information to Lumu servers from any of the scenarios previously described, you need first to set up a Virtual Appliance.

You can download the version of the Lumu VA that is more convenient for your infrastructure in the Lumu Portal. Once downloaded, follow the documentation instructions to configure the appliance.

Make sure that you follow the sizing guidelines when setting up the virtual appliance with a Netflow collector.

The ability to collect network flow data is available from the version 4.0 of the Lumu Virtual Appliance. In case you already have a Lumu VA set up and running, make sure it has the latest appliance version, simply run the command lumu-appliance upgrade , this check if a new version of the appliance is available. If yes, you can download and install it.

Enable Netflow collection

This step is not required if you are using Packetbeat to collect network flow information.

In order to receive NetFlow/IPFIX data, the Lumu Virtual Appliance needs Filebeat, a log file shipper from Elastic. For more details about Filebeat, consult the Elastic website.

To install and configure Filebeat, go to the Lumu VA and run the following command:

lumu-appliance enable netflow

The Lumu VA will assist you in the download of Filebeat after you accept the Elastic License:

You have selected to enable NetFlow collection. To do so you require Filebeat by Elastic ( https://www.elastic.co/beats/filebeat). Lumu can assist you in downloading, deploying, and configuring Filebeat. By choosing to continue, you are accepting the Elastic License Agreement for Filebeat ( https://github.com/elastic/beats/blob/7.10/licenses/ELASTIC-LICENSE.txt).
Do you want to continue? (y/[n]):

You will be requested to set the port number to listen to network traffic data. This is a positive number below 65536, numbers above 1024 are highly recommended as ports below 1024 are usually used by other applications, and may require different privileges to run. Be sure to select a port that is not used by any other collector. Ports 5044 and 5045 are already in use by the appliance.

Please enter a port in which the appliance will listen netflow:

Create a VA Collector

To add a VA Collector, go to the Lumu Portal and navigate to the Collectors > Virtual Appliance menu, click on the VA in which you want to add the collector, then click on the “Add Collector” button.

Collectors management area on Lumu PortalFigure 3 - Collectors management area on Lumu Portal.

When creating a Virtual Appliance, you are required to provide the following information:

  1. Name : a name for your VA Collector.
  2. Type : the type of metadata you want this VA collector to process (Netflow).
Creating a VA collectorFigure 4 - Creating a VA collector.

Set Up a VA Collector

Once you have created the Netflow collector at the Lumu Portal, go to the Virtual Appliance and refresh the VA collectors settings by running the command lumu-appliance collectors refresh. The newly created collector will be presented:

Enabled collectors

When continuing, you will be prompted to select the device type:

Select device type

The first option applies to NetFlow/IPFIX. Once selected, the following selection will be presented:

Define flows

Reverse and post are referred to the fields where the information about the response flow is going to be expected, reverse as in reverse_octet_delta_count or post as in post_octet_delta_count. Usually, Softflowd sends the information in reverse fields and nProbe sends it in post fields. If you don’t know which is your case, contact our support team to help you do a capture and verify.

For the Packetbeat option, you will be requested to set the port number. Keep in mind that this should be a positive number below 65535, numbers above 1024 are highly recommended as ports below 1024 are usually used by other applications, and may require different privileges to run. Be sure to select a port that is not used by any other collector. Port 5044 is already in use by the appliance.

We recommend reloading the appliance after changing the configuration files to make sure the new settings are applied. 

To reload the VA, run the command lumu-appliance reload

Netflow Configuration

In this section, we show how to leverage third party solutions to collect IP traffic data and have IP conversation visibility into your network. While we provide guidance for nProbe and Softflowd, you can configure other third party tools to listen to the SPAN port, synthesize flow information and send it to Lumu, as long as the data generated follows the NetFlow or IPFIX standards.

Once you decide which tool you want to deploy to synthesize network flows, you have to install it in a physical machine where you set up two ports (network interface controller - NIC), one to listen to the SPAN port and another one to export flows to the Lumu Virtual Appliance.

The next step is to configure your network flow generating device or software to send the information to the port you set when configuring the Lumu Virtual Appliance Collector.

Additionally, you should configure the solution to use one of the supported protocols: NetFlow v5, v9 or v10/IPFIX.

If the device/solution uses a protocol v9 or v10/IPFIX, make sure the following fields are configured:

  1. protocol_identifier
  2. tcp_control_bits
  3. end_reason
  4. flow_duration_milliseconds
  5. source_transport_port
  6. destination_transport_port
  7. octet_delta_count
  8. packet_delta_count
  9. flow_end_sys_up_time
  10. flow_start_sys_up_time
  11. source_ipv4_address or source_ipv6_address
  12. destination_ipv4_address or destination_ipv6_address

If you use the bidirectional version of the protocol, make sure the corresponding fields are also enabled.

Depending on the device you are using, there may be additional steps required to successfully collect network flow data. Consult your vendor documentation for guidance.

Netflow Configuration with nProbe

nProbe is a tool provided by ntop. For detailed information on installing nProbe, search for appropriate instructions on the ntop website.

You can find an example of configuring nProbe as a flow synthesizer for the Lumu Virtual Appliance below. The flag -i should be used with the name of the interface where the SPAN port is connected. The flag -n should be used with the IP of the Lumu VA and the port configured to listen for network traffic data.

Note, we use the template option since the default template used by nProbe doesn’t contain the fields expected by the Lumu Virtual Appliance.

nprobe -i eth0 -n appliance_ip:collector_port -T ‘%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV6_SRC_ADDR %IPV6_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %PROTOCOL %TCP_FLAGS %IP_PROTOCOL_VERSION %FLOW_END_REASON %FLOW_DURATION_MILLISECONDS %LAST_SWITCHED %FIRST_SWITCHED’

To review the appliance logs for any problems, run the following command in the Lumu VA:

lumu-appliance logs

Netflow Configuration with Softflowd

Softflowd is a software implementation of a flow-based network traffic monitor. You have to download the code and follow the instructions to compile and install it.

Below you can find an example of the use with version 9. The flag -i should be used with the name of the interface where the SPAN port is connected. The flag -n should be used with the IP of the appliance and the port configured to listen for IP traffic data.

Softflowd implements Netflow version 5, 9 and 10/IPFIX, but we recommend not to use the bidirectional mode because the source and destination IP can be switched by the implementation.

lumu-appliance logs

To review appliance the logs for any problems, run the following command in the Lumu VA:

lumu-appliance logs

Netflow Configuration with Packetbeat

Packetbeat is a tool that can be configured to listen for IP traffic on a SPAN port and send the exported flows to the Lumu Virtual Appliance. You can also consult Elastic’s official documentation about configuring flows to monitor network traffic.

In this section, we show the configuration procedure based on docker on a Linux environment, for other operating systems, consult the instructions to install docker and docker-compose.

We suggest the following script to install docker and docker-compose in a Linux based environment (make sure to use the last version of the dock composer available):

#!/bin/bash
# install docker
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
rm -f get-docker.sh
# install docker-compose
curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
# add user to the docker group, so that we don't have to use sudo
usermod -aG docker $USER

After running the script, log out and log back in so that your group membership is re-evaluated.

The next step is to create the following files. Replace the interface, IPs and ports with those configured in the Virtual Appliance.

docker-compose.yml
            
  1. version: '3' services: dns_collector: image: docker.elastic.co/beats/packetbeat:7.9.3 network_mode: host cap_add: ['NET_RAW', 'NET_ADMIN'] container_name: lumu_flows_collector hostname: lumu_flows_collector restart: unless-stopped volumes: - "./packetbeat.yml:/usr/share/packetbeat/packetbeat.yml" logging: driver: syslog options: syslog-format: "rfc3164" tag: "lumu_flows_collector"
packetbeat.yml
            
  1. packetbeat.interfaces.device: *replace* packetbeat.interfaces.type: af_packet packetbeat.protocols: - type: icmp ports: true packetbeat.flows: timeout: 30s period: -1s processors: - include_fields: fields: - source - destination - type - event.duration - event.start - network.transport - network.type - flow - icmp output.logstash: hosts: ["appliance_ip:collector_port"] logging: level: info metrics.enabled: false http.enabled: true monitoring.enabled: false

After configuring both files, use docker compose to start Packetbeat with the following command:

docker-compose -f docker-compose.yml up -d

To confirm the Packetbeat configuration is working as expected, run the following command:

tail -f /var/log/syslog | grep lumu_flows_collector

If everything is properly configured, the information will start to reach Lumu Virtual Appliance. To review the appliance logs for any problems, run the following command in the Lumu VA:

lumu-appliance logs
        • Related Articles

        • Lumu Virtual Appliance Collectors

          To get started with Lumu Virtual Appliances, consult our Introduction to Lumu Virtual Appliances article. The Lumu Virtual Appliance (VA) is a pre-configured lightweight virtual machine solution that collects the network metadata of your entire ...
        • Lumu Virtual Appliance Metadata Collection with Logstash

          Some enterprises use the Elastic stack (ELK) to collect, index, and analyze logs from multiple devices. If yours is one of those organizations, deploy a Lumu Virtual Appliance and create collectors that will receive data directly from existing ...
        • Introduction to Lumu Virtual Appliances

          The Lumu Virtual Appliance (VA) is a pre-configured lightweight virtual machine solution that collects the network metadata of your entire enterprise and forwards it to the Lumu cloud with the lowest impact on the network operation. This document ...
        • Lumu Virtual Appliance DNS Packets Collectors Catalog

          Before attempting this type of implementation, we strongly suggest checking out our Lumu Agent for Windows Server , which can act as a DNS server collector and covers the vast majority of the scenarios of this VA implementation (Windows Server 2016 ...
        • Lumu Virtual Appliance DNS Queries Collectors Catalog

          In the following table, you will find a complete list of DNS Queries Collectors available for deployment as part of Lumu's Virtual Appliances. Collector Logo Collect DNS Queries with Lumu VA and Infoblox Collect DNS Queries with Lumu VA and Citrix ...