Custom Collector API Specifications

Custom Collector API Specifications

The Custom Collector API is a seamless way to integrate your network infrastructure with Lumu while layering Continuous Compromise Assessment. It allows sending network metadata captured from third-party platforms/services/appliances to Lumu, and it can also be used as an alternative for obtaining greater visibility in cases where the enterprise network restricts the use of Virtual Appliances.

If you are interested in getting started with Custom Collector API management in the Lumu Portal, consult our documentation.
Custom Collector API general deployment architecture Figure 1 - Custom Collector API general deployment architecture.

This article provides you with Custom Collector API endpoints specifications, requirements, sample JSON, and request parameters.

Access and Authentication

The authentication of the Lumu Customer Collector API is handled by a unique key referred to in this document as {lumu-client-key}. Each custom collector also has a unique ID that is referred to as {collector-id} to perform diagnostic/support tasks smoothly.

The Custom Collector API key (1) and collectors ID (2) are found in the Lumu Portal and are self-managed by company administrators.

Custom Collector API management area - Lumu Portal Figure 2 - Custom Collector API management area - Lumu Portal.

General Syntax

The Lumu Custom Collectors is a REST API restricted to HTTPS and is available to make requests at: https://api.lumu.io

All requests to this endpoint must be sent with the Content-Type: application/json header.

All the post requests to Custom Collector API are identified as traffic of the company by the required query parameters {lumu_client_key} and {collector_id}, as in the following example:

 
  1. https://api.lumu.io/collectors/64c1a892-9ea3-4c7b-b8af-e1f5b2ff6b0c/dns/packets?key=64c1a892-9ea3-4c7b-b8af-e1f5b2ff6b0c.

Send DNS Packets

This method post arrays of DNS packets captured using third-party solutions such as Logstash, Graylog, and other DNS packet capture tools.

 
  1. POST /collectors/{collector-id}/dns/packets?key={lumu-client-key}
Body parameters (example)
 
  1. [ { "id": 69, "timestamp": "2021-01-05T14:37:02.228Z", "client_ip": "192.168.0.103", "client_name": "User9800", "op_code": "QUERY", "response_code": "NOERROR", "question": { "type": "A", "name": "www.example.com", "class": "IN" }, "flags": { "authoritative": false, "recursion_available": true, "truncated_response": false, "checking_disabled": false, "recursion_desired": true, "authentic_data": false }, "answers": [ { "name": "www.example.com", "type": "A", "class": "IN", "ttl": 2549, "data": "69.172.201.153" } ] } ]

Body parameters (details)

Name
Data Type
Description
id
integer
A 16-bit numeric identifier assigned by the software that generates the query, such as Packetbeat, Logstash, Graylog, etc.
timestamp
date-time
The UTC date and time when query request was performed in ISO format, e.g. " 2021-01-07T14:37:02.228Z"
client_ip
string
The dotted-quad formatted or colon-separated IPv4/IPv6 address of the client that initiated the request, e.g. " 192.168.1.16" or "2001:db8:85a3::8a2e:370:7334"
client_name
string
(optional)
The asset identification as you want it to be shown at the Lumu Portal, e.g. the user or the device hostname from which the query originated.
op_code
string
The DNS operation code that specifies the type of query in the message, e.g. "QUERY"
response_code
string
The keyword of the DNS response code, e.g.  " NOERROR"
question.type

string
The type of records being queried, e.g. "A" or " AAAA"
question.name

string
The domain name being queried. Non-printable characters are represented as in C; i.e., as \DDD, where DDD is the octal ASCII character code. Backslashes and quotes are escaped. Tabs, carriage returns, and line feeds are converted to \t, \r, and \n respectively, e.g. " www.example.com "
question.class
string
The class of records being queried,  e.g. " IN"
flags.authoritative
boolean
A flag specifying that the responding server is an authority for the domain name used in the question, e.g. false .
flags.recursion_available
boolean
A boolean flag specifying whether recursive query support is available in the name server, e.g. true
flags.truncated_response
boolean
A flag specifying that only the first 512 bytes of the reply were returned., e.g. false
flags.checking_disabled
boolean
A flag specifying if the client disabled the server signature validation of the query,  e.g. true
flags.recursion_desired
boolean
A flag specifying that the client directs the server to pursue a query recursively. Recursive query support is optional, e.g. false
flags.authentic_data
boolean
A flag specifying that the recursive server considers the response authentic, e.g. true
answers
array
(optional)
Objects with each answer section returned by the server
answers.name
array
The domain name to which this resource record pertains
answers.type
string
The type of data contained in this resource record, e.g. " A"  or " AAAA"
answers.class
string
The class of records being queried, e.g. " IN"
answers.ttl
integer
The time interval in seconds that this resource record may be cached before it should be discarded. This field is a long numeric type where zero values mean that the data should not be cached, e.g. 2549
answers.data
string
The data describing the resource. The meaning of this data depends on the type and class of the resource record, e.g. " 69.172.201.153"

Send DNS Queries

Use this method to post simple DNS queries extracted from DNS servers’ logs. It is a simplified version of the DNS Packets method. While this form of posting DNS data is suitable for Lumu’s Illumination Process, it lacks information that could be useful for understanding more complex adversarial activity.

Note: Only operations of the QUERY type that belongs to the IN DNS Class should be posted through this service.

This method receives an array of DNS queries, which are identified as traffic of the company by the {lumu-client-key} in the Lumu Portal. The path parameter {collector-id} is a unique identifier for each collector to perform diagnostic/support tasks easily.

 
  1. POST /collectors/{collector-id}/dns/queries?key={lumu-client-key}
Body parameters (example)
 
  1. [ { "timestamp": "2021-01-06T14:37:02.228Z", "name": "www.example.com", "client_ip": "192.168.0.103", "client_name": "MACHINE-0987", "type": "A" } ]

Body parameters (details)

Name
Data Type
Description
timestamp
date-time
The UTC date and time when query request was performed in ISO format, e.g. " 2021-01-07T14:37:02.228Z"
name
array
The domain name being queried. Non-printable characters are represented as in C; i.e., as \DDD, where DDD is the octal ASCII character code. Backslashes and quotes are escaped. Tabs, carriage returns, and line feeds are converted to \t, \r, and \n respectively, e.g. "www.example.com "
client_ip
string
The dotted-quad formatted or colon-separated IPv4/IPv6 address of the client that initiated the request, e.g. " 192.168.1.16"  or " 2001:db8:85a3::8a2e:370:7334"
client_name
string
(optional)
The name of the device or the user from which the query was originated
type
string
(optional)
The type of data contained in this resource record, e.g. " A" or " AAAA"

Send Proxy Entries

Use this method to post proxy entries composed of information extracted from the client proxy log. This method receives an array of Proxy entries, which are identified as traffic of the company by the {lumu-client-key}.

 
  1. POST /collectors/{collector-id}/proxy/entries?key={lumu-client-key}
Body parameters (example)
 
  1. [ { "timestamp": "2021-01-07T14:37:02.228Z", "client_ip": "192.168.0.103", "client_name": "optional DeviceName", "user_name": "optional UserName", "remote_ip": "201.98.87.1", "time_taken": 6532, "request": { "uri": { "host": "birazoku.com", "port": 443, "scheme": "https", "path": "/index.html", "query": "name=myname&wlkd=ksi" }, "method": "post", "length": 1034, "body_length": 887, "referrer": "referrer.com", "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X1" }, "response": { "code": 200, "phrase": "Success", "length": 3049, "body_length": 2978 } } ]

Body parameters (details)

Name
Data Type
Description
timestamp
date-time
The UTC date and time when query request was performed in ISO format, e.g. "2021-01-07T14:37:02.228Z"
client_ip
string
A string containing the dotted quad formatted or colon-separated IPv4/IPv6 address of the client that initiated the request, e.g. "192.168.1.16" or "2001:db8:85a3::8a2e:370:7334"
client_name
string
(optional)
The name of the device or the user from which the query was originated
user_name
string
(optional)
The authenticated user in the proxy session
remote_ip
string
(optional)
The dotted quad-formatted or colon-separated IPv4/IPv6 address of the server which response the request, e.g. "192.168.1.16" or "2001:db8:85a3::8a2e:370:7334"
time_taken
integer
(optional)
The time used to complete the request in milliseconds, e.g. 2549
request.uri.host
string
The domain or IP against which the request was made, e.g. "example.com"
request.uri.port
integer 
An integer containing the TCP port used to make the request, e.g. 443
request.uri.scheme
string
The scheme used to perform the request, e.g. "http" or "https"
request.uri.path
string
(optional)
The specified resource that was being requested, default is “/”
request.uri.query
string
(optional)
The parameters used in the request, e.g. "name=myname&wlkd=ksi"
request.method
string
The HTTP method used to run the request, e.g. "post", "get," "put"
request.length
integer
(optional)
The size of the request in bytes, e.g. 1034
request.body_length
integer
(optional)
The size of the body of the request in bytes, e.g. 900
request.referer
string
(optional)
The address of the webpage which is linked to the resource being requested, e.g. "referrer.com"
request.user_agent
string
(optional)
The user agent of the browser that made the request, e.g. "Mozilla/5.0 (Macintosh; Intel Mac OS X1"
response.code
integer 
The HTTP code response for the request, e.g. 200
response.phrase
string
The HTTP string response for the request, e.g. "Success"
response.length
integer
(optional)
The size of the response in bytes, e.g. 1200
response.body_length
integer
(optional)
The size of the body of the response in bytes, e.g. 800

Send Firewall Entries

Use this method to post firewall entries composed of information extracted from the client firewall log. This method receives an array of Firewall entries, which are identified as traffic of the company by the {lumu-client-key} at the Lumu Portal.

  
  1. POST /collectors/{collector-id}/firewall/entries?key={lumu-client-key}
Body parameters (example)
  
  1. [ { "timestamp": "2021-01-06T14:37:02.228Z", "time_taken": 1630, "user_name": "optional UserName", "action": "accept", "protocol": "http", "additional": { "path": "/mhiQtgAAAAE", "query": "key=value&t=846" }, "source": { "ip": "10.0.0.46", "port": 54786, "name": "FW_SGM_DC_01" }, "destination": { "ip": "52.91.3.178", "port": 443, "name": "google.com" }, "sent": { "bytes": 53489, "packets": 183 }, "received": { "bytes": 67385, "packets": 163 } } ]

Body parameters (details)

Name
Data Type
Description
timestamp
date-time
The UTC date and time when query request was performed in ISO format, e.g. "2021-01-07T14:37:02.228Z"
time_taken
integer
(optional)
The time used to complete the request in milliseconds, e.g. 2549
user_name
string
(optional)
The authenticated user in the firewall session
action
string
The action performed by the firewall with the request, being accepted or blocked are common expected scenarios, e.g. "Accept"
protocol
string
The protocol used by the API request, whether is layer seven or four, e.g. "http", "https", "ftp", "udp"
additional.path
string
(optional)
the specified resource the request was requesting, default is “/”
additional.query
string
(optional)
The parameters used in the request, e.g. "name=myname&wlkd=ksi"
source.ip
string
The dotted-quad formatted or colon-separated IPv4/IPv6 address of the client that initiated the request, e.g. "192.168.1.16 or 2001:db8:85a3::8a2e:370:7334"
source.port
integer
An integer containing the TCP port used to make the request, e.g. 54786
source.name
string
(optional)
The name of the device or the user where the query originated. For outgoing connections, this will be shown in the Lumu Portal, e.g. "FW_SGM_DC_01"
destination.ip
string
The dotted-quad formatted or colon-separated IPv4/IPv6 address of the server which response the request, e.g. "192.168.1.16" or "2001:db8:85a3::8a2e:370:7334"
destination.port
integer
An integer containing the TCP port used by the destination to receive the request, e.g. 443
destination.name
string
(optional)
The domain or hostname against which the request was made, e.g. "google.com"
sent.bytes
integer
(optional)
The amount of bytes sent from source to destination during the request, e.g. 51034
sent.packets
integer
(optional)
The amount of packets sent from source to destination during the request., e.g. 134
received.bytes
integer
(optional)
The amount of bytes sent from destination to source during the request, e.g. 51034
received.packets
integer
(optional)
The amount of packets sent from destination to source during the request, e.g. 130

Status Responses

The Lumu Custom Collector API uses standard HTTP features and response codes to indicate the success or failure of API requests

HTTP Code
Description
200/202
OK The request was successfully received, understood, and accepted
400/422
Bad Request/Unprocessable Entity. Client-side error status, verify malformed request syntax, invalid request message parameters, etc.
500
Internal server error. Our application server encountered a problem. Try again later and if the problem persists, please let us know.

Set Up Third-party Solutions With Lumu Custom Collector API

Lumu provides an example of using third-party tools as metadata collectors, check our documentation on Custom Collector API Integration with Packetbeat and Logstash.

        • Related Articles

        • Custom Collector API

          Some enterprises may already be using defense solutions such as Endpoint Detection and Response (EDR) or network monitoring tools as part of their pipelines to centralize their logs. Lumu gives the option to deploy custom collectors to send your ...
        • Custom Collector API Integration With Packetbeat and Logstash

          Some enterprises may already be using Logstash as part of their pipelines to centralize their logs. In such cases, you have the option to deploy custom collectors to seamlessly integrate your network infrastructure with Lumu while layering Continuous ...
        • Custom Collector API integration with Netskope Next Gen Secure Web Gateway (SWG)

          This article shows how to leverage Netskope Next Gen SWG Events API to collect and inject network metadata into Lumu. Figure 1 - Data collection setup from Netskope Events API to Lumu. The script polls data from Netskope’s page and application ...
        • Deploy Collectors with Log Forwarder for Windows

          Log Forwarder is designed to streamline the data collection processes from third party data collection services. While not as optimized as a fully-fledged Virtual Appliance deployment, it is a great alternative for fast and accessible deployment. ...
        • Using Grouping Rules with Lumu Collectors

          Grouping Rules is a feature shared by some of our collectors that allows you to better organize and categorize the traffic these collectors analyze using your labels. Data Collection collectors are designed to receive traffic in bulk, which is not ...