Retrieving Palo Alto NGFW security events via API. It may seem like NGFW topic is not really related to vulnerability assessment and vulnerability management. In fact, correlation of security events in traffic with vulnerability scan data sometimes may give very interesting results. For example, if we have a Windows desktop host with critical vulnerabilities, it won’t be a big surprise to detect some botnet activity related to this host. Fixing of this hosts should be a high priority task. Moreover, Palo Alto NGFW now supports signatures for vulnerability detection, like Tenable PVS. It’s pretty logical: if you are already searching something in the network traffic, why not to look also for vulnerable software versions in the packet headers?

At the “Monitoring” tab of Palo Alto NGFW GUI web-interface you can see a flow of security events, produced by Palo Alto security rules, standard or custom. With PA query language you may easily filter this events. It is also possible to produce reports. However, the standard reports Palo Alto are not very informative and only represent some statistics of attacks without any additional information. Much more interesting reports you can make using Palo Alto API.

Authentication

For authentication you need a key. Request it using your Palo Alto username and password:

curl -X GET 'https://192.168.56.101/api/?type=keygen&user=username&password=password'

<response status="success">
     <result>
          <key>gJlQWE56987nBxIqyfa62sZeRtYuIo2BgzEA9UOnlZBhU</key>
     </result>
</response>

Requests

Now we are ready to make requests to Palo Alto. What parameters we need?

• Proper search query. Syntax is the same, as for Palo Alto NGFW web interface. For simplicity, in example we will request all security events for the last day. Time value must be greater than ‘2016/12/13 10:00:01’ and less than ‘2016/12/14 10:00:01’.
• Type of security events (log-type). in example it will be “threats” logs; “traffic” and “wildfire” are also supported.
• What should we do if there will be too many events (entries)? We will need to perform multiple queries setting amount of logs we need (nlogs) and the offset (skip). According to Palo Alto manuals 5000 events can be received via API at a one time. In practice, however, in Palo Alto Panorama problems may appear if you will try to get more than 1000 events at a one time. This is a known bug and, as I know, it is not fixed yet. So, be careful!
• Authorization key

The example curl request:

curl --tlsv1 -sS -k --data-urlencode "query=( receive_time geq '2016/12/13 10:00:01' ) and ( receive_time leq '2016/12/14 10:00:01' )" "https://192.168.56.101/api/?type=log&log-type=threat&nlogs=1000&skip=0&key=EUGHSDSJqwdffersdfSADFSDFIjssieHFIOSDIFJOWQ23EFDIJDSFhQEHJSDVJOSDIFJIWEFNDKSVaosiiw232qweufgdyuqpop298e="

<response status="success" code="19">
    <result><msg><line>query job enqueued with jobid 384</line>
    </msg><job>384</job></result>
</response>

Query execution may take quite a long time. At first you will get the task ID (384). You can make multiple get job-id requests until the status will be “FIN” and in answer will be entries in log section.

curl --tlsv1 -sS -k "https://192.168.56.101/api/?type=log&action=get&job-id=384&key=EUGHSDSJqwdffersdfSADFSDFIjssieHFIOSDIFJOWQ23EFDIJDSFhQEHJSDVJOSDIFJIWEFNDKSVaosiiw232qweufgdyuqpop298e="

<response status="success"><result>
  <job>
    <tenq>17:35:05</tenq>
    <tdeq>17:35:05</tdeq>
    <tlast>03:00:00</tlast>
    <status>FIN</status>
    <id>384</id>
    <cached-logs>1152</cached-logs>
  </job>
  <log>
    <logs count="1000" progress="100">
      <entry logid="99124">
        <domain>1</domain>
        <receive_time>2016/12/14 09:50:24</receive_time>
        <serial>001801012243</serial>
        <seqno>8210416</seqno>
        <actionflags>0x8000000000000000</actionflags>
        <type>THREAT</type>
        <subtype>vulnerability</subtype>
        <config_ver>1</config_ver>
        <time_generated>2016/12/14 09:38:45</time_generated>
        <src>xxx.xxx.xxx.xxx</src>
        <dst>xxx.xxx.xxx.xxx</dst>
        <rule>CorporationHQ</rule>
        <srcloc>United States</srcloc>
        <dstloc>xxx.xxx.xxx.xxx-xxx.xxx.xxx.xxx</dstloc>
        <app>smtp</app>
        <vsys>vsys1</vsys>
        <from>trust</from>
        <to>trust</to>
        <inbound_if>ethernet1/5</inbound_if>
        <outbound_if>ethernet1/6</outbound_if>
        <logset>Logs</logset>
        <time_received>2016/12/14 09:38:45</time_received>
        <sessionid>135193</sessionid>
        <repeatcnt>1</repeatcnt>
        <sport>52252</sport>
        <dport>25</dport>
        <natsport>0</natsport>
        <natdport>0</natdport>
        <flags>0x80004000</flags>
        <flag-pcap>yes</flag-pcap>
        <flag-flagged>no</flag-flagged>
        <flag-proxy>no</flag-proxy>
        <flag-url-denied>no</flag-url-denied>
        <flag-nat>no</flag-nat>
        <captive-portal>no</captive-portal>
        <non-std-dport>no</non-std-dport>
        <transaction>no</transaction>
        <pbf-c2s>no</pbf-c2s>
        <pbf-s2c>no</pbf-s2c>
        <temporary-match>no</temporary-match>
        <sym-return>no</sym-return>
        <decrypt-mirror>no</decrypt-mirror>
        <pktlog>1481698125-135115.pcap</pktlog>
        <proto>tcp</proto>
        <action>reset-both</action>
        <cpadding>0</cpadding>
        <dg_hier_level_1>11</dg_hier_level_1>
        <dg_hier_level_2>0</dg_hier_level_2>
        <dg_hier_level_3>0</dg_hier_level_3>
        <dg_hier_level_4>0</dg_hier_level_4>
        <device_name>PA_CorporationHQ</device_name>
        <vsys_id>1</vsys_id>
        <threatid>MAIL: User Login Brute Force Attempt</threatid>
        <tid>40007</tid>
        <reportid>0</reportid>
        <category>any</category>
        <severity>high</severity>
        <direction>client-to-server</direction>
        <url_idx>0</url_idx>
        <padding>0</padding>
        <pcap_id>1201601193257066528</pcap_id>
      </entry>
    [...]
    </logs>
  </log>
</result></response>

Useful queries and log-types:

• Virus

threat: (subtype eq virus)

• Spyware

threat: (subtype eq spyware)

• Brute Force SSH

threat: ( name-of-threatid eq ‘SSH User Authentication Brute Force Attempt’ )

• Brute Force MAIL

threat: ( name-of-threatid eq ‘MAIL: User Login Brute Force Attempt’ )

• Brute Force SIP

threat: ( name-of-threatid eq ‘SIP Bye Message Brute Force Attack’ )

• Botnet C&C activity (custom rule based on IP blacklist)

traffic: (rule eq cnc)

• Malware

wildfire

Multiple appliances and Palo Alto Panorama

When you have several appliances managed by Palo Alto Panorama you can specify the particular appliance with `( device_name eq ‘PA_CorporationHQ’ )`

The final curl request will look like:

curl --tlsv1 -sS -k --data-urlencode "query=( name-of-threatid eq 'MAIL: User Login Brute Force Attempt' ) and ( receive_time geq '2016/12/13 10:00:01' ) and ( receive_time leq '2016/12/14 10:00:01' ) and ( device_name eq 'PA_CorporationHQ' )" "https://192.168.56.101/api/?type=log&log-type=threat&nlogs=1000&skip=0&key=EUGHSDSJqwdffersdfSADFSDFIjssieHFIOSDIFJOWQ23EFDIJDSFhQEHJSDVJOSDIFJIWEFNDKSVaosiiw232qweufgdyuqpop298e="


name-of-threatid

Be careful, threat id names may be changed anytime by Palo Alto without any warning!
Example: “MAIL: User Login Brute-force Attempt” was renamed in “MAIL: User Login Brute Force Attempt”.
Of course this change may break the requests.

What next?

You can group similar security events for the last day and make csv tables like this one for spyware:

Destination;Destination Name;Source;Threat Name;Count
192.168.56.201;corporation\j.smith;178.218.122.66;Locky.JSDownloader.Gen Command and Control Traffic;5
192.168.56.205;corporation\k.bruni;90.126.241.111;nimnul.Gen Command And Control Traffic;1

Furthermore, you can build some dynamic reports. If the same event appears in daily reports regularly, is a good reason to find out if it is a false positive or real issue:

All (2)

New (1)
192.168.56.201;corporation\j.smith;178.218.122.66;Locky.JSDownloader.Gen Command and Control Traffic;5

Already seen (1)
#3 days (1)
#Tue Dec 13 10:03:58 MSK 2016
#Tue Dec 14 10:03:57 MSK 2016
#Thu Dec 15 10:04:39 MSK 2016
192.168.56.205;corporation\k.bruni;90.126.241.111;nimnul.Gen Command And Control Traffic;1

Other ways of integration

Ok, API is great, but what if you want to export all PA data? For example, this data can be used as a source for user behavior analytics. For such a mass exporting, when using of Palo Alto API is impractical, you can try Palo Alto Netflow and Palo Alto Threats export to Syslog server. I followed this official manuals and all works pretty well.