Skip to main content

Dedup - Generic v3

This Playbook is part of the Common Playbooks Pack.#

Supported versions

Supported Cortex XSOAR versions: 5.5.0 and later.

This playbook identifies duplicate incidents using one of the supported methods. Select one of the following methods to identify duplicate incidents in Cortex XSOAR.

  • ml: Machine learning model, which is trained mostly on phishing incidents.
  • rules: Rules help identify duplicate incidents when the logic is well defined, for example, the same label or custom fields.
  • text: Statistics algorithm that compares text, which is generally useful for phishing incidents. For each method, the playbook will search for the oldest similar incident. when there is a match for a similar incident the playbook will close the current incident and will link it to the older incident.

Dependencies#

This playbook uses the following sub-playbooks, integrations, and scripts.

Sub-playbooks#

This playbook does not use any sub-playbooks.

Integrations#

This playbook does not use any integrations.

Scripts#

  • CloseInvestigationAsDuplicate
  • PhishingDedupPreprocessingRule
  • FindSimilarIncidentsByText
  • FindSimilarIncidents

Commands#

  • linkIncidents

Playbook Inputs#


NameDescriptionDefault ValueRequired
DuplicateMethodSelect a method for identifying duplicate incidents. Can be "ml", "rules", or "text".
'rules' - define specific rules, such as similar incident fields & labels. This method works best if you know the exact logic to find similar incidents.
'text' - text similarity, based on TF-IDF - unique word frequency in the incidents (based on similar incident fields)
'ml' - machine learning model, which was trained on similar phishing incidents. Considers similar labels, incident fields, and indicators.
Required
exsitingIncidentsLookbackUse only with ML Method.
The start date by which to search for duplicated existing incidents. The date format is the same as in the incidents query page. For example, "3 days ago", "2019-01-01T00:00:00 +0200"
7 days agoOptional
statusScopeUse only with ML Method.
Whether to compare the new incident to past closed or non-closed incidents only.
"All" - Default. Compares to all incidents.
"ClosedOnly" - Compares to closed incidents.
"NonClosedOnly" - Compare to open incidents.
Optional
fromPolicyUse only with ML Method.
Whether to take into account the email from field for deduplication.

"TextOnly" - incidents will be considered duplicated based on test similarity only, ignoring the sender's address.

"Exact" - incidents will be considered duplicated if their text is similar, and their sender is the same.

"Domain" - Default. Incidents will be considered duplicated if their text is similar, and their senders' address has the same domain.
Optional
DuplicateThresholdThe similarity threshold by which to consider an incident as a duplicate (0-1), where "1" is a duplicate and "0" is not a duplicate. Use this argument in the ML or text methods.0.7Required
TimeFrameHoursThe time frame (in hours) in which to check for duplicate incident candidates.72Optional
IgnoreClosedIncidentsWhether to ignore closed incidents. Can be "yes" or "no".yesOptional
MaxNumberOfCandidatesThe maximum number of candidates to check for duplication.1000Optional
CloseAsDuplicateWhether to close incidents identified as duplicates. Can be "true" or "false".trueOptional
TimeFieldThe Time field by which to query for past incidents to check for duplicate incident candidates. Values: created, occurred, modifiedcreatedOptional
similarLabelsKeysA comma-separated list of similar label keys. Comma separated value. Also supports allowing X different words between labels, within the following way: label_name:X, where X is the number of words. X can also be '*' for contains. For example: the value "Email/subject:*" will consider email subject similar, if one is substring of the other. Relevant for 'Rules' method.Optional
similarIncidentFieldsFields to compare. Can be label name, incident fields, or custom fields. Comma-separated value. Relevant for 'Text' and 'Rules' methods.name,type,detailsOptional

Playbook Outputs#


PathDescriptionType
isSimilarIncidentFoundWhether a similar incident was found? Can be "true" or "false".boolean
similarIncidentThe similar incident.unknown

Playbook Image#


Dedup - Generic v3