Fetching Incidents

Cortex XSOAR pulls events from third party tools and converts them into incidents using the fetch-incidents command.

This topic provides:

A description of the fetching command and parameters
A description of creating fetched incidents in Cortex XSOAR
An explanation of how missing incidents are fetched with generic lookback methods
Troubleshooting tips

fetch-incidents Command#

The fetch-incidents command is the function that Cortex XSOAR calls every minute to import new incidents. It is triggered by the Fetches incidents parameter in the integration configuration. It is not necessary to configure the fetch-incidents command in the integration settings.

screen shot 2019-01-07 at 15 35 01

Open the fetch-incidents command. Make sure the command is also referenced in the execution block.

def fetch_incidents():
      # your implementation here


if demisto.command() == 'fetch-incidents':
      fetch_incidents()

Last Run#

The demisto.getLastRun() function retrieves the last previous run time.
This helps avoid duplicate incidents by fetching only events that occurred since the last time the function was run.

    # demisto.getLastRun() will returns an obj with the previous run in it.
    last_run = str(demisto.getLastRun())

Note:
The value of demisto.getLastRun() is not stored between CLI command runs; it is only stored in the incident fetch flow on the server side. Therefore, if you print out the value of demisto.getLastRun() in the integration code, its value will appear as {}.

For debugging, you can save the value of demisto.getLastRun() to integration context, which does persist between CLI command runs. Alternatively, you can run the fetch_incidents() command twice within the same command run.

First Run#

When an integration runs for the first time, the last run time is not in the integration context.
To set up the first run properly, use an if statement with a time that is specified in the integration settings.

It is best practice to specify how far back in time to fetch incidents on the first run. This is a configurable parameter in the integration settings.

Query and Parameters#

Queries and parameters enable filtering events.
For example, you may want to import only certain event types into Cortex XSOAR. To do this, you need to query the API for only that specific event type. These are configurable parameters in the integration settings.

Example#

The following example uses the First Run if statement and query.

    # usually there will be some kind of query based on event creation date, 
    # or get all the events with id greater than X id and their status is New
    query = 'status=New'

    day_ago = datetime.now() - timedelta(days=1) 
    start_time = day_ago.time()
    if last_run and 'start_time' in last_run:
        start_time = last_run.get('start_time')

    # execute the query and get the events
    events = query_events(query, start_time)

Fetch Limit#

The Fetch Limit parameter sets the maximum number of incidents to get per fetch command. To maintain an optimal load on Cortex XSOAR we recommend setting a limit of 200 incidents per fetch.
Note: If you enter a larger number or leave Fetch Limit blank, the Test button will fail.

Create an Incident#

Incidents are created by building an array of incident objects. These objects must contain:

The name of the incident
When the incident occurred
The rawJSON key for the incident

Recommended to include:

details - a brief description of the incident.
dbotMirrorId - the ID of the incident in the third-party product (see Fetch History).

Example#

# convert the events to Cortex XSOAR incident 
events = [
  {
      'name': 'event_1',
      'create_time': '2019-10-23T10:11:00Z',
      'event_id': 100
  }
]
    
incidents = []
for event in events:
    incident = {
        'name': event['name'],        # name is required field, must be set
        'occurred': event['create_time'], # must be string of a format ISO8601
        'dbotMirrorId': str(event['event_id']),  # must be a string
        'rawJSON': json.dumps(event)  # the original event, this will allow mapping of the event in the mapping stage. Don't forget to `json.dumps`
    }
    incidents.append(incident)

rawJSON#

The rawJSON key in the incident field enables event mapping. Mapping is how an event gets imported into Cortex XSOAR and enables you to choose which data from the event is to be mapped to Cortex XSOAR fields.

Example#

incident = {
    'name': event['name'],        # name is required field, must be set
    'occurred': '2019-10-23T10:00:00Z', # occurred is optional date - must be string of a format ISO8601
    'rawJSON': json.dumps(event)  # set the original event to rawJSON, this will allow mapping of the event. Don't forget to `json.dumps`
}

Set Last Run#

When the last events are retrieved, you need to save the new last run time to the integration context. This timestamp will be used the next time the fetch-incidents function runs.
Notes:

When setting demisto.setLastRun, the values of the dictionary must be type string.
We recommend using the time of the most recently created incident as the new last run.
If there is an error pulling incidents with fetch-incidents, demisto.setLastRun will not execute.

demisto.setLastRun({
    'start_time': timestamp_to_datestring(last_incident['time'])
})

Send the Incidents to Cortex XSOAR#

When all of the incidents are created, the demisto.incidents() function returns an array of incidents in Cortex XSOAR.
This is similar to the demisto.results() function, but is used exclusively to handle incident objects.

Example#

# this command will create incidents in Cortex XSOAR
demisto.incidents(incidents)

If there are no incidents to return, demisto.incidents() returns an empty list.

# returning an empty list will keep the status as ok but no new incidents are created.
demisto.incidents([])

Fetch History#

In XSOAR versions 6.8 and above, it is possible to observe the results of the last fetch-incidents/fetch-indicators runs using the Fetch History modal. For more details, visit the Fetch History documentation.

When implementing a fetch-incidents command, in some cases we can populate extra data for the following columns in the modal:

Message - In long-running integrations, the info/error message given in demisto.updateModuleHealth() will be displayed in the Message column. Use the is_error boolean argument of this method to determine the message type. For example:
demisto.updateModuleHealth("Could not connect to client.", is_error=True)
The above line will produce a new record in the modal, and its Message value will be Error: Could not connect to client..
Source IDs - If incidents on the third-party product have IDs, it is possible to display them in the Source IDs column by adding the dbotMirrorId field as part of the incident dictionary. For example:
demisto.incidents([
{"name": "This is an incident.", "dbotMirrorId": "123"},
{"name": "This is another incident.", "dbotMirrorId": "124"},
])
The above will produce a new record in the modal, and its Source IDs value will be 123, 124.
Note: the population of the fetch history information occurs before the classification, therefore this field must be defined at the integration code level rather than the classification and mapping level.

Fetch Missing Incidents with Generic Lookback Methods#

This advanced feature uses generic lookback methods for fetching missing incidents because of indexing issues in the 3rd party product. For more information click here.

Troubleshooting#

To troubleshoot fetch-incident, execute !integration_instance_name-fetch debug-mode=true in the Playground to return the incidents.