Custom Collector API integration with Netskope Next Gen Secure Web Gateway (SWG)

Custom Collector API integration with Netskope Next Gen Secure Web Gateway (SWG)

This article shows how to leverage Netskope Next Gen SWG Events API to collect and inject network metadata into Lumu.
Data collection setup from Netskope Events API to Lumu Figure 1 - Data collection setup from Netskope Events API to Lumu.

The script polls data from Netskope’s page and application events.

The Lumu Custom Collector API allows sending network metadata captured from a third-party platform/service/appliance to Lumu for Continuous Compromise Assessment. It can also be used as an alternative for obtaining greater visibility in cases where the enterprise network restricts the use of Virtual Appliances. If you want more context on custom collectors, Custom Collector API .

Requirements

  1. Netskope Next Gen SWG tenant name and API key. The tenant name can be obtained from the Netskope access URL: https://<tenant-name>.goskope.com/
    1. If you are using API v2, please include /api/v2/events/data/page and /api/v2/events/data/application endpoints in the scope of the key. For further reference, please refer to Netskope Platform REST API .
  2. Scripting host with Python v3.9+.
    1. The host must be able to reach both Lumu Custom Collectors endpoints and Netskope REST API endpoints.
  3. Script package.
    1. Contact Get in touch with Lumu  to request the script deployment package.

Add Custom Collector

The first step is to add a custom collector in the Lumu Portal. For this, access the Collector API menu, select the option to add a Custom Collector and select Proxy Log as the collector type.
add a Custom Collector and select Proxy Log as the collector type Figure 2 - add a Custom Collector and select Proxy Log as the collector type.

After creating the custom collector, please copy the client key and Custom Collector ID. For more information about this procedure, please refer to our article  Custom Collector API Specifications .

Deploy the script

Remember to contact the Lumu support team to acquire the script deployment package if you haven’t already done so. 
Script location.

Unpack the deployment package provided by Lumu in your preferred path/folder. Keep in mind this location, as it will be required for further configurations. From this point on, we will refer to this folder as <netskope_lumu_root> .

Install requirements

The file requirements.txt contains the list of prerequisites for this data collector. After deploying the package locally, run the following command from the deployment folder:

[sudo] pip install -r ./requirements.txt

Script details

To use the script, you must locate yourself on the path selected for deployment (<netskope_lumu_root>). Use the following command to show all options available for the package:

python netskope-collect.py --help
netskope-collect.py [options]

Options
Description
-h, --help
Show this help message and exit.
--config CONFIG
Load options from config file.
--client-key CLIENT_KEY

--client_key CLIENT_KEY
Lumu Client Key.
--collector-id COLLECTOR_ID

--collector_id COLLECTOR_ID
Lumu Custom Collector ID.
--proxy-host PROXY_HOST

--proxy_host PROXY_HOST
Proxy host (if required)
--proxy-port PROXY_PORT

--proxy_port PROXY_PORT
Proxy port (if required)
--proxy-user PROXY_USER

--proxy_user PROXY_USER
Proxy user (if required)
--proxy-password PROXY_PASSWORD

--proxy_password PROXY_PASSWORD
Proxy password (if required)
--logging {screen,file}
Logging option (default screen).
--verbose, -v
Verbosity level.
--query QUERY

-q QUERY
Netskope query (if required).
--copy-user, --copy_user
Copy user name to client name.
--tenant TENANT
Netskope tenant.
--key KEY
Netskope API key.
--version VERSION
Netskope API version (default 2).

Usage examples

Task: poll and inject Netskope events into Lumu for the last 5 minutes

Use the following command for polling and injecting all Netskope events the last 5 minutes into a Lumu custom collector:
python netskope-collect.py --tenant --key --version 1 --client-key --collector-id

In the example, we are using Netskope REST API v1. If you want to use version 2 of the Netskope REST API, please remove the --version flag and its value.
After the first run, the script will create a control file called timestamp.stmp . This file will store the last timestamp from the queried events for the next run. This is to avoid repeating the injection of events into the Lumu Custom Collector.

Task: query filtered Netskope events

For filtering events with specific conditions, use the --query argument to define a query. Bear in mind that you need to specify this query using the SkopeIT Query Language .

python netskope-collect.py --tenant --key --version 1 --client-key --collector-id --query “action eq allow”

For this example, the script will query allowed events.
It is recommended to use quotes to close your query to avoid bad syntax issues.

Task: use user name instead of IP address in injected entries

If you want to replace the client IP address with its user name, use the flag --copy_user. All detections shown by the Lumu portal will show the user name instead of its IP address.

python netskope-collect.py --tenant --key --version 1 --client-key --collector-id --copy-user

Task: save log output to file

Use the --logging file argument to store a record of all tasks run. Using this, the entire script output will be redirected to a file named netskope_collect.log in the script root path ( <netskope_lumu_root> ).

python netskope-collect.py --tenant --key --version 1 --client-key --collector-id --logging file

Other tasks

The above samples may be combined according to your needs.

Further considerations

To run the script on a timely basis, consider implementing a Scheduled job in Windows or a cron task in Unix-based systems.

Troubleshooting

To identify failures in the script, please use the -v flag . This will allow you to identify failures in the script execution.
    Still can’t find an answer?
    Send a message to our experts.
    Contact Us

        • Related Articles

        • Custom Collector API

          Some enterprises may already be using defense solutions such as Endpoint Detection and Response (EDR) or network monitoring tools as part of their pipelines to centralize their logs. Lumu gives the option to deploy custom collectors to send your ...

        • Custom Collector API Specifications

          The Custom Collector API is a seamless way to integrate your network infrastructure with Lumu while layering Continuous Compromise Assessment. It allows sending network metadata captured from third-party platforms/services/appliances to Lumu, and it ...

        • Collect Proxy Metadata with Lumu VA and Skyhigh Secure Web Gateway (Cloud)

          The Lumu Virtual Appliance (VA) offers the option to create Collectors, a seamless way to integrate with network metadata of your entire enterprise and forward it to the Lumu cloud with the lowest impact on the network operation. In cases where the ...

        • Custom Collector API Integration With Packetbeat and Logstash

          Some enterprises may already be using Logstash as part of their pipelines to centralize their logs. In such cases, you have the option to deploy custom collectors to seamlessly integrate your network infrastructure with Lumu while layering Continuous ...

        • Using Grouping Rules with Lumu Collectors

          Grouping Rules is a feature shared by some of our collectors that allows you to better organize and categorize the traffic these collectors analyze using your labels. Data Collection collectors are designed to receive traffic in bulk, which is not ...