Cortex Data Lake

Overview


Palo Alto Networks Cortex Data Lake provides cloud-based, centralized log storage and aggregation for your on premise, virtual (private cloud and public cloud) firewalls, for Prisma Access, and for cloud-delivered services such as Cortex XDR This integration was integrated and tested with version 2 of Cortex Data Lake


Configure Cortex Data Lake on Demisto


  1. Go to the HUB and select the Demisto v2 app
  2. In the War Room, run the command !GetLicenseID to get the license ID.
  3. Go to Settings > ABOUT > License to get the Customer Name.
  4. Insert the license ID and the Customer Name in the required fields and complete the authentication process in order to get the Authentication Token Registration ID Encryption Key
  5. Navigate to Settings > Integrations > Servers & Services.
  6. Search for Palo Alto Networks Cortex v2.
  7. Click Add instance to create and configure a new integration instance.
    • Name: a textual name for the integration instance.
    • Authentication Token: From the authentication process
    • Registration ID: From the authentication process
    • Encryption Key: From the authentication process
    • proxy: Use system proxy settings
    • insecure: Trust any certificate (not secure)
    • Fetch incidents: Whether to fetch incidents or not
    • first_fetch_timestamp: First fetch time (\<number> \<time unit>, e.g., 12 hours, 7 days, 3 months, 1 year)
    • Severity of events to fetch (Firewall): Select from all,Critical,High,Medium,Low,Informational,Unused
    • Subtype of events to fetch (Firewall): Select from all,attack,url,virus,spyware,vulnerability,file,scan,flood,packet,resource,data,url-content,wildfire,extpcap,wildfire-virus,http-hdr-insert,http-hdr,email-hdr,spyware-dns,spyware-wildfire-dns,spyware-wpc-dns,spyware-custom-dns,spyware-cloud-dns,spyware-raven,spyware-wildfire-raven,spyware-wpc-raven,wpc-virus,sctp
    • Incidents fetched per query: How many incidents will be fetched per query. Caution: high number could create overload. Default is 10.
  8. Click Test to validate the URLs, token, and connection.

In order for the integration to work, the following URLs need to be accessible:

  • For authentication:
    • oproxy.demisto.ninja
    • api.paloaltonetworks.com
  • For API requests, one of the following:
    • US: api.us.cdl.paloaltonetworks.com
    • EU: api.nl.cdl.paloaltonetworks.com

Fetched Incidents Data

Fetches Firewall threat logs as incidents


Commands

You can execute these commands from the Demisto CLI, as part of an automation, or in a playbook. After you successfully execute a command, a DBot message appears in the War Room with the command details. 1. cdl-query-logs 2. cdl-get-critical-threat-logs 3. cdl-get-social-applications 4. cdl-search-by-file-hash 5. cdl-query-traffic-logs 6. cdl-query-threat-logs

1. cdl-query-logs

Runs a query on the Cortex logging service.

Base Command

cdl-query-logs

Input
Argument NameDescriptionRequired
queryA free-text SQL query. For example, query="SELECT * FROM `firewall.traffic` limit 10". There are multiple tables in Loggings, for example: threat, traffic, and so on. Refer to the Cortex Logging service schema reference for the full list.Optional
limitThe number of logs to return. Default is 10Optional
Context Output
PathTypeDescription
CDL.Logging.ActionStringIdentifies the action that the firewall took for the network traffic.
CDL.Logging.AppStringApplication associated with the network traffic.
CDL.Logging.ProtocolStringIP protocol associated with the session.
CDL.Logging.DestinationIPStringOriginal destination IP address.
CDL.Logging.RuleMatchedStringName of the security policy rule that the network traffic matched.
CDL.Logging.CharacteristicOfAppNumberIdentifies the behaviorial characteristic of the application associated with the network traffic.
CDL.Logging.LogSourceNameStringName of the source of the log.
CDL.Logging.IsNatnumberIndicates if the firewall is performing network address translation (NAT) for the logged traffic.
CDL.Logging.NatDestinationPortNumberPost-NAT destination port.
CDL.Logging.NatDestinationStringIf destination NAT performed, the post-NAT destination IP address.
CDL.Logging.NatSourceStringIf source NAT was performed, the post-NAT source IP address.
CDL.Logging.SourceIPStringOriginal source IP address.
CDL.Logging.AppCategoryStringIdentifies the high-level family of the application.
CDL.Logging.SourceLocationStringSource country or internal region for private addresses.
CDL.Logging.DestinationLocationStringDestination country or internal region for private addresses.
CDL.Logging.FileSHA256StringThe binary hash (SHA256) of the file sent for virus analysis.
CDL.Logging.FileNameStringThe name of the infected file.
CDL.Logging.TimeGeneratedDateTime when the log was generated on the firewall's data plane.
Command Example

!cdl-query-logs query="SELECT * FROM `firewall.traffic` limit 1"

Context Example
{
"CDL.Logging": [
{
"Action": "allow",
"App": "smtp",
"Protocol": "tcp",
"DestinationIP": "206.116.22.23",
"RuleMatched": "taplog",
"CharacteristicOfApp": [
"3",
"4",
"5",
"6",
"7",
"8"
],
"LogSourceName": "gw",
"NatDestination": "0.0.0.0",
"NatSource": "0.0.0.0",
"SourceIP": "10.154.1.20",
"AppCategory": "collaboration",
"SourceLocation": "10.0.0.0-10.255.255.255",
"DestinationLocation": "CA",
"TimeGenerated": "2020-03-18T19:36:37"
}
]
}
Human Readable Output

Logs traffic table

ActionAppAppCategoryCharacteristicOfAppDestinationIPDestinationLocationLogSourceNameNatDestinationNatSourceProtocolRuleMatchedSourceIPSourceLocationTimeGenerated
allowsmtpcollaboration3,4,5,6,7,8206.116.22.23CAgw0.0.0.00.0.0.0tcptaplog10.154.1.2010.0.0.0-10.255.255.2552020-03-18T19:36:37

2. cdl-get-critical-threat-logs


Runs a query on the Cortex logging service, according to preset queries.

Base Command

cdl-get-critical-threat-logs

Input
Argument NameDescriptionRequired
start_timeThe query start time. For example, start_time="2018-04-26 00:00:00"Optional
end_timeThe query end time. For example, end_time="2018-04-26 00:00:00"Optional
limitThe number of logs to return. Default is 10Optional
time_rangeFirst log time (\<number> \<time unit>, e.g., 12 hours, 7 days, 3 months, 1 year)Optional
Context Output
PathTypeDescription
CDL.Logging.Threat.SessionIDStringIdentifies the firewall's internal identifier for a specific network session.
CDL.Logging.Threat.ActionStringIdentifies the action that the firewall took for the network traffic.
CDL.Logging.Threat.AppStringApplication associated with the network traffic.
CDL.Logging.Threat.NatStringIndicates whether the firewall is performing network address translation (NAT) for the logged traffic. If it is, this value is 1.
CDL.Logging.Threat.SubcategoryOfAppStringIdentifies the application's subcategory. The subcategoryis related to the application's category, which is identified in category_of_app.
CDL.Logging.Threat.PcapIDStringPacket capture (pcap) ID. This is used to correlate threat pcap files with extended pcaps taken as a part of the session flow. All threat logs will contain either a pcap_id of 0 (no associated pcap) , or an ID referencing the extended pcap file.
CDL.Logging.Threat.NatdstStringIf destination NAT performed, the post-NAT destination IP address. The IP address is an IPv4/IPv6 address in hex format.
CDL.Logging.Threat.FlagsStringBit field which provides details on the session, such as whether the session use IPv6, whether the session was denied due to a URL filtering rule, and/or whether the log corresponds to a transaction within an HTTP proxy session.
CDL.Logging.Threat.DportStringNetwork traffic's destination port. If this value is 0, then the app is using its standard port.
CDL.Logging.Threat.ThreatIDStringNumerical identifier for the threat type. All threats encountered by Palo Alto Networks firewalls are assigned a unique identifier
CDL.Logging.Threat.NatsrcStringIf source NAT was performed, the post-NAT source IP address. The IP address is an IPv4/IPv6 address in hex format.
CDL.Logging.Threat.CategoryOfAppStringIdentifies the managing application, or parent, of the application associated with this network traffic, if any.
CDL.Logging.Threat.SrclocStringSource country or internal region for private addresses. The internal region is a user-defined name for a specific network in the user's enterprise.
CDL.Logging.Threat.DstlocStringDestination country or internal region for private addresses. The internal region is a user-defined name for a specific network in the user's enterprise.
CDL.Logging.Threat.ToStringNetworking zone to which the traffic was sent.
CDL.Logging.Threat.RiskOfAppStringIndicates how risky the application is from a network security perspective. Values range from 1-5, where 5 is the riskiest.
CDL.Logging.Threat.NatsportStringPost-NAT source port.
CDL.Logging.Threat.URLDeniedStringSession was denied due to a URL filtering rule.
CDL.Logging.Threat.CharacteristicOfAppStringIdentifies the behaviorial characteristic of the application associated with the network traffic.
CDL.Logging.Threat.HTTPMethodStringOnly in URL filtering logs. Describes the HTTP Method used in the web request
CDL.Logging.Threat.FromStringThe networking zone from which the traffic originated.
CDL.Logging.Threat.VsysStringVirtual system associated with the network traffic.
CDL.Logging.Threat.ReceiveTimeStringTime the log was received at the management plane.
CDL.Logging.Threat.UsersStringSrcuser or dstuser or srcip (one of).
CDL.Logging.Threat.ProtoStringIP protocol associated with the session.
CDL.Logging.Threat.NatdportStringPost-NAT destination port.
CDL.Logging.Threat.DstStringOriginal destination IP address. The IP address is an IPv4/ IPv6 address in hex format.
CDL.Logging.Threat.RuleStringName of the security policy rule that the network traffic matched.
CDL.Logging.Threat.CategoryOfThreatIDStringThreat category of the detected threat.
CDL.Logging.Threat.DeviceNameStringThe hostname of the firewall that logged the network traffic.
CDL.Logging.Threat.SubtypeStringSubtype of the threat log.
CDL.Logging.Threat.TimeReceivedStringTime the log was received at the management plane.
CDL.Logging.Threat.DirectionStringIndicates the direction of the attack, client-to-server or server-to-client:
CDL.Logging.Threat.MiscStringThe meaning of this field differs according to the log's subtype: Subtype is URL, this field contains the requested URI. Subtype is File, this field contains the file name or file type. Subtype is Virus, this field contains the file name. Subtype is WildFire, this field contains the file name.
CDL.Logging.Threat.SeverityStringSeverity associated with the event.
CDL.Logging.Threat.SrcStringOriginal source IP address. The IP address is an IPv4/IPv6 address in hex format.
CDL.Logging.Threat.TimeGeneratedStringTime the log was generated on the data plane.
CDL.Logging.Threat.SerialStringSerial number of the firewall that generated the log.
CDL.Logging.Threat.VsysIDStringA unique identifier for a virtual system on a Palo Alto Networks firewall.
CDL.Logging.Threat.URLDomainStringThe name of the internet domain that was visited in this session.
CDL.Logging.Threat.CategoryStringFor the URL subtype, this identifies the URL Category. For the WildFire subtype, this identifies the verdict on the file. It is one of ‘malicious’, ‘phishing’, ‘grayware’, or ‘benign’;
CDL.Logging.Threat.SportStringSource port utilized by the session.
CDL.Logging.Threat.IsPhishingBooleanDetected enterprise credential submission by an end user.
IP.AddressStringIP address.
Domain.NameStringThe domain name, for example: "google.com".
File.SHA256StringThe SHA256 hash of the file.
File.NameStringThe full file name (including file extension).
File.TypeStringThe file type, as determined by libmagic (same as displayed in file entries).
Command Example

!cdl-get-critical-threat-logs limit="1" time_range="10 days"

Context Example
{
"CDL.Logging.Threat": [
{
"SessionID": 103986,
"Action": "reset-both",
"App": "imap",
"IsNat": false,
"SubcategoryOfApp": "email",
"PcapID": 0,
"NatDestination": "0.0.0.0",
"Flags": 8192,
"DestinationPort": 143,
"ThreatID": 30663,
"NatSource": "0.0.0.0",
"IsURLDenied": false,
"Users": "10.154.10.88",
"TimeGenerated": "2020-03-18T15:46:10",
"IsPhishing": false,
"AppCategory": "collaboration",
"SourceLocation": "10.0.0.0-10.255.255.255",
"DestinationLocation": "CH",
"ToZone": "TapZone",
"RiskOfApp": 4,
"NatSourcePort": 0,
"CharacteristicOfApp": [
"3",
"4",
"5",
"8"
],
"FromZone": "TapZone",
"Vsys": "vsys1",
"Protocol": "tcp",
"NatDestinationPort": 0,
"DestinationIP": "84.74.104.27",
"SourceIP": "10.154.10.88",
"RuleMatched": "taplog",
"ThreatCategory": "overflow",
"LogSourceName": "gw",
"Subtype": "vulnerability",
"Direction": "server to client",
"FileName": "iZJvnxT27.PpT",
"VendorSeverity": "Critical",
"LogTime": "2020-03-18T15:46:37",
"LogSourceID": "007251000070976",
"VsysID": 1,
"URLDomain": null,
"URLCategory": "any",
"SourcePort": 14484
}
]
}
Human Readable Output

Logs threat table

ActionAppAppCategoryCharacteristicOfAppDestinationIPDestinationLocationDestinationPortDirectionFileNameFlagsFromZoneIsNatIsPhishingIsURLDeniedLogSourceIDLogSourceNameLogTimeNatDestinationNatDestinationPortNatSourceNatSourcePortPcapIDProtocolRiskOfAppRuleMatchedSessionIDSourceIPSourceLocationSourcePortSubcategoryOfAppSubtypeThreatCategoryThreatIDTimeGeneratedToZoneURLCategoryURLDomainUsersVendorSeverityVsysVsysID
reset-bothimapcollaboration3,4,5,884.74.104.27CH143server to clientiZJvnxT27.PpT8192TapZonefalsefalsefalse007251000070976gw2020-03-18T15:46:370.0.0.000.0.0.000tcp4taplog10398610.154.10.8810.0.0.0-10.255.255.25514484emailvulnerabilityoverflow306632020-03-18T15:46:10TapZoneany10.154.10.88Criticalvsys11

3. cdl-get-social-applications


Runs a query on the Cortex logging service, according to preset queries.

Base Command

cdl-get-social-applications

Input
Argument NameDescriptionRequired
start_timeQuery start time. For example, start_time="2018-04-26 00:00:00"Optional
end_timeQuery end time. For example, end_time="2018-04-26 00:00:00"Optional
limitAmount of logs. Default is 10Optional
time_rangeFirst log time (\<number> \<time unit>, e.g., 12 hours, 7 days, 3 months, 1 year)Optional
Context Output
PathTypeDescription
CDL.Logging.Traffic.ActionStringIdentifies the action that the firewall took for the network traffic.
CDL.Logging.Traffic.RiskOfAppStringIndicates how risky the application is from a network security perspective.
CDL.Logging.Traffic.NatSourcePortStringPost-NAT source port.
CDL.Logging.Traffic.SessionIDStringIdentifies the firewall's internal identifier for a specific network session.
CDL.Logging.Traffic.PacketsStringNumber of total packets (transmit and receive) seen for the session.
CDL.Logging.Traffic.CharacteristicOfAppStringIdentifies the behaviorial characteristic of the application associated with the network traffic.
CDL.Logging.Traffic.AppStringApplication associated with the network traffic.
CDL.Logging.Traffic.VsysStringVirtual system associated with the network traffic.
CDL.Logging.Traffic.IsNatStringIndicates whether the firewall is performing network address translation (NAT) for the logged traffic. If it is, this value is 1.
CDL.Logging.Traffic.LogTimedateTime the log was received in Cortex Data Lake.
CDL.Logging.Traffic.SubcategoryOfAppStringIdentifies the application's subcategory. The subcategory is related to the application's category,
CDL.Logging.Traffic.ProtocolStringIP protocol associated with the session.
CDL.Logging.Traffic.NatDestinationPortStringPost-NAT destination port.
CDL.Logging.Traffic.DestinationIPStringOriginal destination IP address.
CDL.Logging.Traffic.NatDestinationStringIf destination NAT performed, the post-NAT destination IP address.
CDL.Logging.Traffic.RuleMatchedStringName of the security policy rule that the network traffic matched.
CDL.Logging.Traffic.DestinationPortStringNetwork traffic's destination port. If this value is 0, then the app is using its standard port.
CDL.Logging.Traffic.TotalTimeElapsedStringTotal time taken for the network session to complete.
CDL.Logging.Traffic.LogSourceNameStringDevice name of the source of the log
CDL.Logging.Traffic.SubtypeStringThe log sub type.
CDL.Logging.Traffic.UsersStringSource/Destination user. If neither is available, source_ip is used.
CDL.Logging.Traffic.TunneledAppStringIs app tunneled.
CDL.Logging.Traffic.IsPhishingStringIndicates whether enterprise credentials were submitted by an end user.
CDL.Logging.Traffic.SessionEndReasonStringThe reason a session terminated.
CDL.Logging.Traffic.NatSourceStringIf source NAT was performed, the post-NAT source IP address.
CDL.Logging.Traffic.SourceIPStringOriginal source IP address.
CDL.Logging.Traffic.SessionStartIPdateTime when the session was established.
CDL.Logging.Traffic.TimeGenerateddateTime when the log was generated on the firewall's data plane.
CDL.Logging.Traffic.AppCategoryStringIdentifies the high-level family of the application.
CDL.Logging.Traffic.SourceLocationStringSource country or internal region for private addresses.
CDL.Logging.Traffic.DestinationLocationStringDestination country or internal region for private addresses.
CDL.Logging.Traffic.LogSourceIDStringD that uniquely identifies the source of the log. If the source is a firewall, this is its serial number.
CDL.Logging.Traffic.TotalBytesStringNumber of total bytes (transmit and receive).
CDL.Logging.Traffic.VsysIDStringA unique identifier for a virtual system on a Palo Alto Networks firewall.
CDL.Logging.Traffic.ToZoneStringNetworking zone to which the traffic was sent.
CDL.Logging.Traffic.URLCategoryStringThe URL category.
CDL.Logging.Traffic.SourcePortStringSource port utilized by the session.
CDL.Logging.Traffic.TunnelStringType of tunnel.
Command Example

!cdl-get-social-applications limit="2" time_range="10 days"

Context Example
{
"CDL.Logging.Traffic": [
{
"Action": "allow",
"RiskOfApp": 4,
"SessionID": 108356,
"Packets": 7,
"CharacteristicOfApp": [
"3",
"4",
"5",
"6",
"8"
],
"App": "facebook-base",
"Vsys": "vsys1",
"LogTime": "2020-03-18T15:54:40",
"SubcategoryOfApp": "social-networking",
"Protocol": "tcp",
"DestinationIP": "131.130.159.25",
"NatDestination": "0.0.0.0",
"RuleMatched": "taplog",
"DestinationPort": 80,
"LogSourceName": "gw",
"Subtype": "start",
"Users": "10.154.230.43",
"TunneledApp": "tunneled-app",
"SessionEndReason": "n-a",
"NatSource": "0.0.0.0",
"SourceIP": "10.154.230.43",
"SessionStartIP": "2020-03-18T15:54:14",
"TimeGenerated": "2020-03-18T15:54:16",
"AppCategory": "collaboration",
"SourceLocation": "10.0.0.0-10.255.255.255",
"DestinationLocation": "AT",
"LogSourceID": "007251000070976",
"TotalBytes": 946,
"VsysID": 1,
"ToZone": "TapZone",
"URLCategory": "social-networking",
"SourcePort": 37252,
"Tunnel": "N/A"
},
{
"Action": "allow",
"RiskOfApp": 4,
"SessionID": 276377,
"Packets": 768,
"CharacteristicOfApp": [
"3",
"4",
"5",
"6",
"8"
],
"App": "facebook-base",
"Vsys": "vsys1",
"LogTime": "2020-03-16T15:54:36",
"SubcategoryOfApp": "social-networking",
"Protocol": "tcp",
"DestinationIP": "213.191.250.86",
"NatDestination": "0.0.0.0",
"RuleMatched": "taplog",
"DestinationPort": 80,
"TotalTimeElapsed": 1,
"LogSourceName": "gw",
"Subtype": "end",
"Users": "10.154.227.21",
"TunneledApp": "tunneled-app",
"SessionEndReason": "tcp-fin",
"NatSource": "0.0.0.0",
"SourceIP": "10.154.227.21",
"SessionStartIP": "2020-03-16T15:53:58",
"TimeGenerated": "2020-03-16T15:54:16",
"AppCategory": "collaboration",
"SourceLocation": "10.0.0.0-10.255.255.255",
"DestinationLocation": "IE",
"LogSourceID": "007251000070976",
"TotalBytes": 384468,
"VsysID": 1,
"ToZone": "TapZone",
"URLCategory": "social-networking",
"SourcePort": 53174,
"Tunnel": "N/A"
}
]
}
Human Readable Output

Logs traffic table

Logs traffic table

ActionAppAppCategoryCharacteristicOfAppDestinationIPDestinationLocationDestinationPortLogSourceIDLogSourceNameLogTimeNatDestinationNatSourcePacketsProtocolRiskOfAppRuleMatchedSessionEndReasonSessionIDSessionStartIPSourceIPSourceLocationSourcePortSubcategoryOfAppSubtypeTimeGeneratedToZoneTotalBytesTunnelTunneledAppURLCategoryUsersVsysVsysID
allowfacebook-basecollaboration3,4,5,6,8131.130.159.25AT80007251000070976gw2020-03-18T15:54:400.0.0.00.0.0.07tcp4taplogn-a1083562020-03-18T15:54:1410.154.230.4310.0.0.0-10.255.255.25537252social-networkingstart2020-03-18T15:54:16TapZone946N/Atunneled-appsocial-networking10.154.230.43vsys11
allowfacebook-basecollaboration3,4,5,6,8213.191.250.86IE80007251000070976gw2020-03-16T15:54:360.0.0.00.0.0.0768tcp4taplogtcp-fin2763772020-03-16T15:53:5810.154.227.2110.0.0.0-10.255.255.25553174social-networkingend2020-03-16T15:54:16TapZone384468N/Atunneled-appsocial-networking10.154.227.21vsys11

4. cdl-search-by-file-hash


Runs a query on the threat table with the query 'SELECT * FROM firewall.threat WHERE file_sha_256 = <file_hash>'

Base Command

cdl-search-by-file-hash

Input
Argument NameDescriptionRequired
start_timeThe query start time. For example, start_time="2018-04-26 00:00:00"Optional
end_timeThe query end time. For example, end_time="2018-04-26 00:00:00"Optional
limitThe number of logs to return. Default is 10.Optional
time_rangeFirst log time (\<number> \<time unit>, e.g., 12 hours, 7 days, 3 months, 1 year)Optional
SHA256The SHA256 hash of the file for the query. For example, SHA256="503ca1a4fc0d48b18c0336f544ba0f0abf305ae3a3f49b3c2b86b8645d6572dc" would return all logs associated with this file.Required
Context Output
PathTypeDescription
CDL.Logging.Threat.SessionIDStringIdentifies the firewall's internal identifier for a specific network session.
CDL.Logging.Threat.ActionStringIdentifies the action that the firewall took for the network traffic.
CDL.Logging.Threat.AppStringApplication associated with the network traffic.
CDL.Logging.Threat.NatStringIndicates whether the firewall is performing network address translation (NAT) for the logged traffic. If it is, this value is 1.
CDL.Logging.Threat.SubcategoryOfAppStringIdentifies the application's subcategory. The subcategoryis related to the application's category, which is identified in category_of_app.
CDL.Logging.Threat.PcapIDStringPacket capture (pcap) ID. This is used to correlate threat pcap files with extended pcaps taken as a part of the session flow. All threat logs will contain either a pcap_id of 0 (no associated pcap) , or an ID referencing the extended pcap file.
CDL.Logging.Threat.NatdstStringIf destination NAT performed, the post-NAT destination IP address. The IP address is an IPv4/IPv6 address in hex format.
CDL.Logging.Threat.FlagsStringBit field which provides details on the session, such as whether the session use IPv6, whether the session was denied due to a URL filtering rule, and/or whether the log corresponds to a transaction within an HTTP proxy session.
CDL.Logging.Threat.DportStringNetwork traffic's destination port. If this value is 0, then the app is using its standard port.
CDL.Logging.Threat.ThreatIDStringNumerical identifier for the threat type. All threats encountered by Palo Alto Networks firewalls are assigned a unique identifier
CDL.Logging.Threat.NatsrcStringIf source NAT was performed, the post-NAT source IP address. The IP address is an IPv4/IPv6 address in hex format.
CDL.Logging.Threat.CategoryOfAppStringIdentifies the managing application, or parent, of the application associated with this network traffic, if any.
CDL.Logging.Threat.SrclocStringSource country or internal region for private addresses. The internal region is a user-defined name for a specific network in the user's enterprise.
CDL.Logging.Threat.DstlocStringDestination country or internal region for private addresses. The internal region is a user-defined name for a specific network in the user's enterprise.
CDL.Logging.Threat.ToStringNetworking zone to which the traffic was sent.
CDL.Logging.Threat.RiskOfAppStringIndicates how risky the application is from a network security perspective. Values range from 1-5, where 5 is the riskiest.
CDL.Logging.Threat.NatsportStringPost-NAT source port.
CDL.Logging.Threat.URLDeniedStringSession was denied due to a URL filtering rule.
CDL.Logging.Threat.CharacteristicOfAppStringIdentifies the behaviorial characteristic of the application associated with the network traffic.
CDL.Logging.Threat.HTTPMethodStringOnly in URL filtering logs. Describes the HTTP Method used in the web request
CDL.Logging.Threat.FromStringThe networking zone from which the traffic originated.
CDL.Logging.Threat.VsysStringVirtual system associated with the network traffic.
CDL.Logging.Threat.ReceiveTimeStringTime the log was received at the management plane.
CDL.Logging.Threat.UsersStringSrcuser or dstuser or srcip (one of).
CDL.Logging.Threat.ProtoStringIP protocol associated with the session.
CDL.Logging.Threat.NatdportStringPost-NAT destination port.
CDL.Logging.Threat.DstStringOriginal destination IP address. The IP address is an IPv4/ IPv6 address in hex format.
CDL.Logging.Threat.RuleStringName of the security policy rule that the network traffic matched.
CDL.Logging.Threat.CategoryOfThreatIDStringThreat category of the detected threat.
CDL.Logging.Threat.DeviceNameStringThe hostname of the firewall that logged the network traffic.
CDL.Logging.Threat.SubtypeStringSubtype of the threat log.
CDL.Logging.Threat.TimeReceivedStringTime the log was received at the management plane.
CDL.Logging.Threat.DirectionStringIndicates the direction of the attack, client-to-server or server-to-client:
CDL.Logging.Threat.MiscStringThe meaning of this field differs according to the log's subtype: Subtype is URL, this field contains the requested URI. Subtype is File, this field contains the file name or file type. Subtype is Virus, this field contains the file name. Subtype is WildFire, this field contains the file name.
CDL.Logging.Threat.SeverityStringSeverity associated with the event.
CDL.Logging.Threat.SrcStringOriginal source IP address. The IP address is an IPv4/IPv6 address in hex format.
CDL.Logging.Threat.TimeGeneratedStringTime the log was generated on the data plane.
CDL.Logging.Threat.SerialStringSerial number of the firewall that generated the log.
CDL.Logging.Threat.VsysIDStringA unique identifier for a virtual system on a Palo Alto Networks firewall.
CDL.Logging.Threat.URLDomainStringThe name of the internet domain that was visited in this session.
CDL.Logging.Threat.CategoryStringFor the URL subtype, this identifies the URL Category. For the WildFire subtype, this identifies the verdict on the file. It is one of ‘malicious’, ‘phishing’, ‘grayware’, or ‘benign’;
CDL.Logging.Threat.SportStringSource port utilized by the session.
CDL.Logging.Threat.IsPhishingBooleanDetected enterprise credential submission by an end user.
IP.AddressStringIP address.
Domain.NameStringThe domain name, for example: "google.com".
File.SHA256StringThe SHA256 hash of the file.
File.NameStringThe full file name (including file extension).
File.TypeStringThe file type, as determined by libmagic (same as displayed in file entries).
Command Example

!cdl-search-by-file-hash SHA256="cbdf1f3cccd949e6e96c425b3d7ccc463b956f002f694472e4d24a12ff2cea4d" limit=1 time_range="10 days"

Context Example
{
"CDL.Logging.Threat": [
{
"SessionID": 784600,
"Action": "block",
"App": "smtp",
"IsNat": false,
"SubcategoryOfApp": "email",
"PcapID": 0,
"NatDestination": "0.0.0.0",
"Flags": 8192,
"DestinationPort": 25,
"ThreatID": 52033,
"NatSource": "0.0.0.0",
"IsURLDenied": false,
"Users": "10.154.246.167",
"TimeGenerated": "2020-03-25T15:42:08",
"IsPhishing": false,
"AppCategory": "collaboration",
"SourceLocation": "10.0.0.0-10.255.255.255",
"DestinationLocation": "US",
"ToZone": "TapZone",
"RiskOfApp": 5,
"NatSourcePort": 0,
"CharacteristicOfApp": [
"3",
"4",
"5",
"6",
"7",
"8"
],
"FromZone": "TapZone",
"Vsys": "vsys1",
"Protocol": "tcp",
"NatDestinationPort": 0,
"DestinationIP": "67.53.137.201",
"SourceIP": "10.154.246.167",
"RuleMatched": "taplog",
"ThreatCategory": "",
"LogSourceName": "gw",
"Subtype": "wildfire",
"Direction": "client to server",
"FileName": "o93yr.ECr",
"VendorSeverity": "Informational",
"LogTime": "2020-03-25T15:42:13",
"LogSourceID": "007251000070976",
"VsysID": 1,
"URLDomain": null,
"URLCategory": "",
"SourcePort": 51819,
"FileSHA256": "cbdf1f3cccd949e6e96c425b3d7ccc463b956f002f694472e4d24a12ff2cea4d"
}
]
}
Human Readable Output

Logs threat table

ActionAppAppCategoryCharacteristicOfAppDestinationIPDestinationLocationDestinationPortDirectionFileNameFileSHA256FlagsFromZoneIsNatIsPhishingIsURLDeniedLogSourceIDLogSourceNameLogTimeNatDestinationNatDestinationPortNatSourceNatSourcePortPcapIDProtocolRiskOfAppRuleMatchedSessionIDSourceIPSourceLocationSourcePortSubcategoryOfAppSubtypeThreatCategoryThreatIDTimeGeneratedToZoneURLCategoryURLDomainUsersVendorSeverityVsysVsysID
blocksmtpcollaboration3,4,5,6,7,867.53.137.201US25client to servero93yr.ECrcbdf1f3cccd949e6e96c425b3d7ccc463b956f002f694472e4d24a12ff2cea4d8192TapZonefalsefalsefalse007251000070976gw2020-03-25T15:42:130.0.0.000.0.0.000tcp5taplog78460010.154.246.16710.0.0.0-10.255.255.25551819emailwildfire520332020-03-25T15:42:08TapZone10.154.246.167Informationalvsys11

5. cdl-query-traffic-logs


Searches the Cortex firewall.traffic table. Traffic logs contain entries for the end of each network session

Base Command

cdl-query-traffic-logs

Input
Argument NameDescriptionRequired
source_ipA source IP address or an array of source IPs addresses for which to search, for example 1.1.1.1,2.2.2.2.Optional
ruleA rule name or an array of rule names to search.Optional
from_zoneA source zone name or an array of source zone names to search.Optional
to_zoneA destination zone name or an array of zone names to search.Optional
source_portSource port utilized by the session. Can be port number or an array of destination port numbers to search. For example '443' or '443,445'Optional
actionAn action name or an array of action names to search.Optional
queryA free-text query for which to search. This forms the WHERE part of the query, for example, !cdl-query-traffic-logs query="source_ip.value LIKE '192.168.1.*' AND dest_ip.value='8.8.8.8' And dest_port=1234"Optional
fieldsThe fields that are selected in the query. Selection can be "all" (same as *) or a comma saparated list of specific fields in the table.Optional
start_timeThe query start time. For example, start_time="2018-04-26 00:00:00"Optional
end_timeThe query end time. For example, end_time="2018-04-26 00:00:00".Optional
time_rangeFirst fetch time (\<number> \<time unit>, e.g., 12 hours, 7 days, 3 months, 1 year)Optional
limitThe number of logs to return. Default is 5.Optional
dest_ipA destination IP address or an array of destination IPs addresses for which to search, for example 1.1.1.1,2.2.2.2.Optional
dest_portDestination port utilized by the session. Can be port number or an array of destination port numbers to search. For example '443' or '443,445'Optional
Context Output
PathTypeDescription
CDL.Logging.Traffic.ActionStringIdentifies the action that the firewall took for the network traffic.
CDL.Logging.Traffic.RiskOfAppStringIndicates how risky the application is from a network security perspective.
CDL.Logging.Traffic.NatSourcePortStringPost-NAT source port.
CDL.Logging.Traffic.SessionIDStringIdentifies the firewall's internal identifier for a specific network session.
CDL.Logging.Traffic.PacketsStringNumber of total packets (transmit and receive) seen for the session.
CDL.Logging.Traffic.CharacteristicOfAppStringIdentifies the behaviorial characteristic of the application associated with the network traffic.
CDL.Logging.Traffic.AppStringApplication associated with the network traffic.
CDL.Logging.Traffic.VsysStringVirtual system associated with the network traffic.
CDL.Logging.Traffic.IsNatStringIndicates whether the firewall is performing network address translation (NAT) for the logged traffic. If it is, this value is 1.
CDL.Logging.Traffic.LogTimedateTime the log was received in Cortex Data Lake.
CDL.Logging.Traffic.SubcategoryOfAppStringIdentifies the application's subcategory. The subcategory is related to the application's category,
CDL.Logging.Traffic.ProtocolStringIP protocol associated with the session.
CDL.Logging.Traffic.NatDestinationPortStringPost-NAT destination port.
CDL.Logging.Traffic.DestinationIPStringOriginal destination IP address.
CDL.Logging.Traffic.NatDestinationStringIf destination NAT performed, the post-NAT destination IP address.
CDL.Logging.Traffic.RuleMatchedStringName of the security policy rule that the network traffic matched.
CDL.Logging.Traffic.DestinationPortStringNetwork traffic's destination port. If this value is 0, then the app is using its standard port.
CDL.Logging.Traffic.TotalTimeElapsedStringTotal time taken for the network session to complete.
CDL.Logging.Traffic.LogSourceNameStringDevice name of the source of the log
CDL.Logging.Traffic.SubtypeStringThe log sub type.
CDL.Logging.Traffic.UsersStringSource/Destination user. If neither is available, source_ip is used.
CDL.Logging.Traffic.TunneledAppStringIs app tunneled.
CDL.Logging.Traffic.IsPhishingStringIndicates whether enterprise credentials were submitted by an end user.
CDL.Logging.Traffic.SessionEndReasonStringThe reason a session terminated.
CDL.Logging.Traffic.NatSourceStringIf source NAT was performed, the post-NAT source IP address.
CDL.Logging.Traffic.SourceIPStringOriginal source IP address.
CDL.Logging.Traffic.SessionStartIPdateTime when the session was established.
CDL.Logging.Traffic.TimeGenerateddateTime when the log was generated on the firewall's data plane.
CDL.Logging.Traffic.AppCategoryStringIdentifies the high-level family of the application.
CDL.Logging.Traffic.SourceLocationStringSource country or internal region for private addresses.
CDL.Logging.Traffic.DestinationLocationStringDestination country or internal region for private addresses.
CDL.Logging.Traffic.LogSourceIDStringD that uniquely identifies the source of the log. If the source is a firewall, this is its serial number.
CDL.Logging.Traffic.TotalBytesStringNumber of total bytes (transmit and receive).
CDL.Logging.Traffic.VsysIDStringA unique identifier for a virtual system on a Palo Alto Networks firewall.
CDL.Logging.Traffic.ToZoneStringNetworking zone to which the traffic was sent.
CDL.Logging.Traffic.URLCategoryStringThe URL category.
CDL.Logging.Traffic.SourcePortStringSource port utilized by the session.
CDL.Logging.Traffic.TunnelStringType of tunnel.
Command Example

!cdl-query-traffic-logs action="allow" fields="vendor_name,log_source,rule_matched,dest_location,log_time" time_range="10 days" limit="5"

Context Example
{
"CDL.Logging.Traffic": [
{
"RuleMatched": "taplog",
"ID": "N2eE+oI3d+esVqaqtVGJv95p4VpTYIihtY50eFi8jgo=",
"DestinationLocation": "TH",
"LogTime": "2020-03-21T16:50:18Z"
},
{
"RuleMatched": "taplog",
"ID": "+zZj7TRjBYRXuSdYrbKAYSjoQDyw4vtNwMhvjlbKGrc=",
"DestinationLocation": "US",
"LogTime": "2020-03-21T16:50:18Z"
},
{
"RuleMatched": "taplog",
"ID": "PetZR587UGE/wOkxgS2b+zF364WTmJ29VnV2gihfJZM=",
"DestinationLocation": "US",
"LogTime": "2020-03-21T16:50:33Z"
},
{
"RuleMatched": "taplog",
"ID": "t6dTRzTObu15RCxw6Nk7SPFXe83uxr06yPMC5Px1p8c=",
"DestinationLocation": "RO",
"LogTime": "2020-03-21T16:50:18Z"
},
{
"RuleMatched": "taplog",
"ID": "X4tXn5Ub82q/DDaCyqcZfSboshpWOu+5xvOSf7ydtrY=",
"DestinationLocation": "CL",
"LogTime": "2020-03-21T16:50:18Z"
}
]
}
Human Readable Output
dest_locationlog_sourcelog_timerule_matchedvendor_name
THfirewall1584809418000000taplogPalo Alto Networks
USfirewall1584809418000000taplogPalo Alto Networks
USfirewall1584809433000000taplogPalo Alto Networks
ROfirewall1584809418000000taplogPalo Alto Networks
CLfirewall1584809418000000taplogPalo Alto Networks

6. cdl-query-threat-logs


Searches the Cortex panw.threat table, which is the threat logs table for PAN-OS/Panorama.

Base Command

cdl-query-threat-logs

Input
Argument NameDescriptionRequired
source_ipOriginal source IP address. Enter an IP address or an array of IP addresses for which to search, for example 1.1.1.1,2.2.2.2.Optional
dest_ipOriginal destination IP address. Enter an IP address or an array of IP addresses for which to search, for example 1.1.1.1,2.2.2.2.Optional
rule_matchedName of the security policy rule that the network traffic matched. Enter a rule name or array of rule names to search.Optional
from_zoneThe networking zone from which the traffic originated. Enter zone or array of zones to search.Optional
to_zoneNetworking zone to which the traffic was sent. Enter zone or array of zones to search.Optional
source_portSource port utilized by the session. Enter a port or array of ports to search.Optional
dest_portNetwork traffic's destination port. Enter a port or array of ports to search.Optional
actionThe action that the firewall took for the network traffic. Enter an action or array of actions to search.Optional
file_sha_256The binary hash (SHA256) of the file. Enter a SHA256 hash or array of SHA256 hashes to search.Optional
file_nameThe name of the file that is blocked. Enter a file name or array of file names to search.Optional
queryFree input query to search. This is the WHERE part of the query. so an example will be !cdl-query-traffic-logs query="source_ip.value LIKE '192.168.1.*' AND dst = '192.168.1.12'"Optional
fieldsThe fields that are selected in the query. Selection can be "all" (same as *) or listing of specific fields in the table. List of fields can be found after viewing all the outputed fields with all.Optional
start_timeThe query start time. For example, start_time="2018-04-26 00:00:00"Optional
end_timeThe query end time. For example, end_time="2018-04-26 00:00:00"Optional
time_rangeFirst fetch time (\<number> \<time unit>, e.g., 12 hours, 7 days, 3 months, 1 year)Optional
limitThe number of logs to return. Default is 5.Optional
Context Output
PathTypeDescription
CDL.Logging.Threat.SessionIDStringIdentifies the firewall's internal identifier for a specific network session.
CDL.Logging.Threat.ActionStringIdentifies the action that the firewall took for the network traffic.
CDL.Logging.Threat.AppStringApplication associated with the network traffic.
CDL.Logging.Threat.NatStringIndicates whether the firewall is performing network address translation (NAT) for the logged traffic. If it is, this value is 1.
CDL.Logging.Threat.SubcategoryOfAppStringIdentifies the application's subcategory. The subcategoryis related to the application's category, which is identified in category_of_app.
CDL.Logging.Threat.PcapIDStringPacket capture (pcap) ID. This is used to correlate threat pcap files with extended pcaps taken as a part of the session flow. All threat logs will contain either a pcap_id of 0 (no associated pcap) , or an ID referencing the extended pcap file.
CDL.Logging.Threat.NatdstStringIf destination NAT performed, the post-NAT destination IP address. The IP address is an IPv4/IPv6 address in hex format.
CDL.Logging.Threat.FlagsStringBit field which provides details on the session, such as whether the session use IPv6, whether the session was denied due to a URL filtering rule, and/or whether the log corresponds to a transaction within an HTTP proxy session.
CDL.Logging.Threat.DportStringNetwork traffic's destination port. If this value is 0, then the app is using its standard port.
CDL.Logging.Threat.ThreatIDStringNumerical identifier for the threat type. All threats encountered by Palo Alto Networks firewalls are assigned a unique identifier
CDL.Logging.Threat.NatsrcStringIf source NAT was performed, the post-NAT source IP address. The IP address is an IPv4/IPv6 address in hex format.
CDL.Logging.Threat.CategoryOfAppStringIdentifies the managing application, or parent, of the application associated with this network traffic, if any.
CDL.Logging.Threat.SrclocStringSource country or internal region for private addresses. The internal region is a user-defined name for a specific network in the user's enterprise.
CDL.Logging.Threat.DstlocStringDestination country or internal region for private addresses. The internal region is a user-defined name for a specific network in the user's enterprise.
CDL.Logging.Threat.ToStringNetworking zone to which the traffic was sent.
CDL.Logging.Threat.RiskOfAppStringIndicates how risky the application is from a network security perspective. Values range from 1-5, where 5 is the riskiest.
CDL.Logging.Threat.NatsportStringPost-NAT source port.
CDL.Logging.Threat.URLDeniedStringSession was denied due to a URL filtering rule.
CDL.Logging.Threat.CharacteristicOfAppStringIdentifies the behaviorial characteristic of the application associated with the network traffic.
CDL.Logging.Threat.HTTPMethodStringOnly in URL filtering logs. Describes the HTTP Method used in the web request
CDL.Logging.Threat.FromStringThe networking zone from which the traffic originated.
CDL.Logging.Threat.VsysStringVirtual system associated with the network traffic.
CDL.Logging.Threat.ReceiveTimeStringTime the log was received at the management plane.
CDL.Logging.Threat.UsersStringSrcuser or dstuser or srcip (one of).
CDL.Logging.Threat.ProtoStringIP protocol associated with the session.
CDL.Logging.Threat.NatdportStringPost-NAT destination port.
CDL.Logging.Threat.DstStringOriginal destination IP address. The IP address is an IPv4/ IPv6 address in hex format.
CDL.Logging.Threat.RuleStringName of the security policy rule that the network traffic matched.
CDL.Logging.Threat.CategoryOfThreatIDStringThreat category of the detected threat.
CDL.Logging.Threat.DeviceNameStringThe hostname of the firewall that logged the network traffic.
CDL.Logging.Threat.SubtypeStringSubtype of the threat log.
CDL.Logging.Threat.TimeReceivedStringTime the log was received at the management plane.
CDL.Logging.Threat.DirectionStringIndicates the direction of the attack, client-to-server or server-to-client:
CDL.Logging.Threat.MiscStringThe meaning of this field differs according to the log's subtype: Subtype is URL, this field contains the requested URI. Subtype is File, this field contains the file name or file type. Subtype is Virus, this field contains the file name. Subtype is WildFire, this field contains the file name.
CDL.Logging.Threat.SeverityStringSeverity associated with the event.
CDL.Logging.Threat.SrcStringOriginal source IP address. The IP address is an IPv4/IPv6 address in hex format.
CDL.Logging.Threat.TimeGeneratedStringTime the log was generated on the data plane.
CDL.Logging.Threat.SerialStringSerial number of the firewall that generated the log.
CDL.Logging.Threat.VsysIDStringA unique identifier for a virtual system on a Palo Alto Networks firewall.
CDL.Logging.Threat.URLDomainStringThe name of the internet domain that was visited in this session.
CDL.Logging.Threat.CategoryStringFor the URL subtype, this identifies the URL Category. For the WildFire subtype, this identifies the verdict on the file. It is one of ‘malicious’, ‘phishing’, ‘grayware’, or ‘benign’;
CDL.Logging.Threat.SportStringSource port utilized by the session.
CDL.Logging.Threat.IsPhishingBooleanDetected enterprise credential submission by an end user.
IP.AddressStringIP address.
Domain.NameStringThe domain name, for example: "google.com".
File.SHA256StringThe SHA256 hash of the file.
File.NameStringThe full file name (including file extension).
File.TypeStringThe file type, as determined by libmagic (same as displayed in file entries).
Command Examples

!cdl-query-threat-logs query="is_packet_capture = true AND severity = \"Critical\"" fields=pcap limit=10 !cdl-query-threat-logs action="allow" fields="vendor_name,log_source,rule_matched,dest_location,log_time" time_range="10 days" limit="1"

Context Example
{
"CDL.Logging.Threat": [
{
"NatDestinationPort": null,
"VsysID": null,
"RuleMatched": "taplog",
"FromZone": null,
"URLDomain": null,
"DestinationLocation": "AE",
"IsPhishing": null,
"URLCategory": "",
"NatSource": "",
"NatSourcePort": null,
"IsURLDenied": null,
"PcapID": null,
"Direction": "",
"Users": null,
"ThreatID": null,
"SessionID": null,
"CharacteristicOfApp": null,
"VendorSeverity": "",
"LogTime": "2020-02-22T16:50:23Z",
"IsNat": null,
"SubcategoryOfApp": null,
"SourceIP": "",
"RiskOfApp": null,
"DestinationIP": "",
"Vsys": null,
"TimeGenerated": null,
"Subtype": "",
"Flags": null,
"ToZone": null,
"Action": "",
"AppCategory": null,
"ThreatCategory": null,
"Protocol": "",
"LogSourceName": null,
"App": null,
"Misc": null,
"DestinationPort": null,
"SourcePort": null,
"NatDestination": "",
"SourceLocation": null,
"LogSourceID": null
}
]
}
Human Readable Output

Logs threat table

dest_locationlog_sourcelog_timerule_matchedvendor_name
AEfirewall1582390223000000taplogPalo Alto Networks

Additional Information


  • In the documented CDL v2, You must now specify the customer's instance ID when you identify the log type that you want to query against. That is, log types must be fully qualified and the instance ID is a part of the fully qualified name: <instanceID>.firewall.traffic However in this integration the instance ID is added automatically to the query so the name firewall.traffic is a valid table name
  • The SQL syntex supported for queries is csql