Skip to main content

XSOAR CI/CD

In Cortex XSOAR, you can develop and test your content on other machines, before using it in a production environment. You can do this by using the remote repository feature in the UI or using the CI/CD process.

The CI/CD process#

The CI/CD process helps you develop and maintain content for complex content development using the full functionality of a Git repository, enabling you to run unit tests, code reviews, run test playbooks, etc. Instead of building and maintaining code on a Cortex XSOAR development environment, you can build content from your own repository, build servers, and utilize third party tools (like CircleCI, Jenkins, etc). You can also use version control, undertake code reviews, do lint and validations, use automatic testing, run tests on development machines, etc.

The process is designed for more advanced users who have an understanding of CI/CD concepts and have a number of developers who work on local machines. The remote repository feature in the UI is designed for less complex content (one or two developers working on a local machine).

CI/CD has the following advantages over using the remote repository feature in Cortex XSOAR:

  • Controls the code base with a multi-branch approach. Multiple developers can work on different branches. You can have code reviews, issue pull requests, etc.
  • Use CI/CD solutions (like CircleCI, Jenkins, etc) as part of your testing (including running test playbooks) and deployment. You also have the ability to run automatic testing.

NOTE: To use this process you need to have a Git repository. Github, Gitlab and Bitbucket are currently supported.

The CI/CD process involves the following stages:

For general information about the CI/CD process, see CI/CD FAQs.

Development#

In the development stage, set up the CI/CD process by creating or cloning a Git repository based on the CI/CD template repository. You can then create and maintain your custom content with the built-in functionality of demisto-sdk. The CI/CD process uses demisto-sdk to develop and deploy custom content packs. It downloads, uploads, validates, creates content, migrates content from a Cortex XSOAR server, etc. It also supports content pack structure generation, auto validation of custom content including running linters on the code, and generation of content pack documentation, etc. For a full list of features, commands, and arguments, see demisto-sdk-commands.

Setup the CI/CD Development Process#

Before you begin, download and install the demisto-sdk on your machine.

Ensure that the DEMISTO_BASE_URL and DEMISTO_API_KEY are set to the server environment that you want to use. For example, if you want to upload content directly to the server without using an artifact server you need to point the server to this environment. When working on a branch and you want to download/upload to a development environment, ensure that the demisto-sdk points to the development server.

  1. Create the CI/CD Repository.

    Create or fork a Git repository (such as Github, Gitlab, bitbucket,etc). Use CI/CD template repository as your base structure. The demisto-sdk connects to this Git repository and then automatically connects to Cortex XSOAR servers using the API, to ensure that it validates Cortex XSOAR content. The repository contains the following content:

    ContentDescription
    config.ymlThe CI/CD configuration file (in the .github\workflows folder), which validates the content pack, creates an ID set, runs tests, etc. When you want to deploy your content, you need to update the file with your repository and whether you want to use an artifact server. For more information, see Configuring the config.yml file in Deployment.
    pre-commitWithin the .hooks folder, the pre-commit file uses the Git rebase interactive tool for manual control of your history revision process.
    .vsc codeUsed when using VSC as your IDE.
    build_related_scriptsContains the CI/CD scripts. The Update build_related_scripts/bucket_upload.py script enables you to upload to Google Cloud Storage (artifact server). Before deploying your content, you need to update the name of the bucket list when uploading the Google Cloud Storage. For more information, see Configure the the bucket_upload.py file in Deployment.
    NOTE: If using another storage application such AWS, you need to replace Google Cloud Storage. Contact Customer Support to assist with this.
    The get_modified_packs script enables you to get the latest version of the content pack before merging.
    dev_envs/pytestA folder that contains the conftest.py, which validates python files.
    .demisto-sdk-confThe custom configuration file for the demisto-sdk commands. For more information, see Setting a preset custom command configuration.
    .gitignoreSpecifies intentionally untracked files that Git should ignore.
    .private-repo-settingsSet who can view your repository. For more information, see Setting repository visibility.
    CommonServerPowerShell.ps1Required when using Powershell. For more information, see Powershell code.
    CommonServerPython.pyRequired using Python. For more information, see Common server python.
    README.mdA markdown file that provides a description of the pack.
    demistomock.ps1Enables you to debug Powershell script.
    demistomock.pyEnables you to debug Python Script. For more information about the demistomock files see Debugging using demistomock the demisto-object.
    requirements.txtContains a list of all the project’s dependencies.
    tox.iniThe command-line driven automated testing tool for Python.
    xsoar_config.jsonThe configuration file that defines what packs lists, and jobs will be set up on the machine. Update this when you are ready to deploy. See step 10 below.
  2. (Optional) Set up the repository to work with an IDE.

    For example if working with GitHub via VSC, download Git desktop and then Configure VSC. Ensure that you enable Git in your IDE tool (Code>Preferences>Settings).

  3. Create branches for content.

    Create content on separate branches for content packs. Different developers are able to create content independently, which makes it easier to review content, create pull requests, test the content on a machine, etc.

  4. (Optional) Migrate custom content in Cortex XSOAR to Git Repository.

    You can migrate all custom content or separate items of content. NOTE: Ensure that the DEMISTO_BASE_URL and DEMISTO_API_KEY are set to the server environment that you want to download.

  • Migrate all custom content If you currently use the existing remote repository feature in Cortex XSOAR and now want to use the CI/CD process, download the content from your development machine using the demisto-sdk download command. For example, to download content from your remote repository to your repository created in step 1, run the following command: demisto-sdk download -o address_of_the_pack_folder -i name_of_the_content

    If you have a lot of content, downloading can take a substantial amount of time, as it creates separate folders for each pack. It is recommended to create one folder with a general name and then separate them into branch repositories.

  • Migrate some custom content If you want to migrate some custom content like an integration, run the download command by specifying the content. For example, to add an CICDExample integration to the CICDExample Content Pack, run the following command: demisto-sdk download -o Packs/CICDExample -i CICDExample The CICDExample integration file appears in the Integrations folder, which separates the content into Python and YML files together with a README.md.

    cicd_integration.png

    NOTE: You can download directly from the development server. If downloading Automations and integrations, you need to use the demisto-sdk split command to split the content into the correct format.

  1. Create New Content. In a branch that has been created in step 3 above, create new content, by running the following command: demisto-sdk init

    The init command automatically generates the content pack structure.

    Follow the on-screen instructions by typing the name of the Content Pack, metadata, description, type of Pack, category, author, email address,tags, integration, etc. The Pack appears in the repository with the required folders. You can delete those items that are not needed. In this example, we create a new content pack, called CICDExample.

    pack_example.png

    The content pack contains the following content:

    ContentDescription
    ClassifiersContains the classifiers in JSON format.
    DashboardsDashboards in JSON format.
    GenericDefinitionsContains JSON files that define the generic object that you create.
    NOTE: Although available from version 6.5, it is currently not supported. You can delete this.
    GenericFieldsIncludes subfolders of custom incidents and custom indicator fields.
    NOTE: Although available from version 6.5, it is currently not supported. You can delete this.
    GenericModulesThe context in which the object will be used. Contains the views (pages) displaying the new objects.
    NOTE: Although available from version 6.5, it is currently not supported. You can delete this.
    GenericTypesIncludes subfolders of custom incident and indicator types. Each directory uses the same structure as incident/indicator types.
    NOTE: Although available from version 6.5, it is currently not supported. You can delete this.
    IncidentFieldsIncident fields in JSON format.
    IncidentTypesIncident types in JSON format.
    IndicatorTypesIncident types in JSON format.
    IntegrationsContains YML, PNG, Markdown, and Python files for integrations.
    LayoutsLayouts in JSON format.
    PlaybooksContains YML and Markdown files.
    README.mdInformation about the Content Packs.
    ReportsReports in JSON format.
    ScriptsContains automations in YML, Python and Markdown files.
    TestPlaybooksContains test files.
    WidgetsWidgets in JSON format.
    doc_filesContains images in PNG format.
    pack_metadata.jsonMetadata about the content pack. For example, name, version, description. Every time it changes, you should update the version
  2. Add any additional files that are required.

    You might want to add a Release Notes, secrets-ignore, or pack ignore folders. You can see how the Content hierarchy appears in the Hello World Content Pack.

  1. Generate a README for your integration, script, or playbook.

    Run the following command: Demisto-sdk generate-docs --insecure -e directory For more information, see creating documentation.

  2. (Optional) Run SDK Validate and SDK Lint on your changes

    Run the demisto-sdk format command, which ensures that your new or modified files comply with Cortex XSOAR's standard. To check that your code is valid, run the demisto-sdk validate command and the demisto-sdk lint command on your code.
    Troubleshooting: If you receive many errors from the validate/lint commands, it is not a major issue, as you can control the validations you want to ignore. For example, if you receive the BA111 - Problematic fields names error, or the IN135 or IN145 - Integration params issues, you can add it to the .pack-ignore file in your pack like this:

    [file:some_playbook.yml]
    ignore=BA111
    [file:some_integration.yml]
    ignore=IN135,IN145

    For the DOXXX - Docker Images errors, you can add --no-docker-checks flag to your validate command or add no_docker_checks to the .demisto-sdk-conf repository file like this:

    [validate]
    no-docker-checks=true

Testing/Staging#

In testing/staging proceess, you can do the following:

  • Utilize 3rd party CI/CD Solutions, such as Circle CI, Jenkins, etc.
  • Run validations and automated tests from a testing server as part of a build process.
  • Run test playbooks.

To test Content:#

Before you begin, ensure that you have installed the following:

  • Your CI/CD solution (such as CircleCI).
  • Docker, or applications like Mypy, Flake8, pylint, pytest, etc., for validations and lint commands.
  1. Review the lint and the validate in the demisto-sdk-conf file.

  2. (Optional): Upload the content to a test staging environment.

    Run the following command: Demisto-sdk upload -i Packs/mypack For example, to upload an integration, type demisto-sdk upload -i Packs/CICDExample/Integrations/CICDExample NOTE: Update the DEMISTO_BASE_URL and DEMISTO_API_KEY, if required.

  3. Test the content and run test playbooks, as required.

  4. Open a Pull Request for other developers to review.

    Using your CI/CD solution, it checks the changes - validations, lints, etc. If it requires approval, you have to wait before being able to merge. The validation is done according to the hooks in your repository. When you create the pull request, the CI/CD process runs automatically.

    github_notsuccess.png

After the testing has completed successfully, you are now ready to deploy content.

Deployment#

In the deployment stage, you can do the following:

  • Control both custom and out-of-the-box content packs deployment into production.
  • Deploy via an artifact server or directly to the Cortex XSOAR production server.

By default, deployment occurs when content is pushed from a branch repository to a master repository. Before deployment, you need to consider whether to deploy via an artifact server.

Artifact Server#

An artifact server such as Google Cloud Services, enables you to maintain and control version deployment and rollbacks, so you can keep track of your work. Each content pack version is saved in separate folders in the artifact server. You can use the XSOAR CI/CD Content Pack, to fetch the content from the artifact server to the Cortex XSOAR server. The content pack enables you to monitor, install content packs via a Playbook, incident type, etc.

artifacts_server.png

NOTE: For Multi-tenant you cannot run the playbook in the XSOAR CI/CD Content Pack, as its not possible to create incidents or playbooks on the Main account. Instead you can upload the content via the CI/CD Build.

Although you can choose your own artifact repository, (such as AWS, GCP, Git, FTP server, etc) by default, the CI/CD process uses Google Cloud Storage. You need to install the Google Cloud Storage Content Pack and configure the integration. If using another storage provider, such as AWS you need to install the relevant Content Pack and set up the integrations as necessary.

The main advantage of the artifact server is version control and rollback. You update the xsoar_config.json file with any version change, rather than having to update the repository.

Without an Artifact Server#

You can deploy your content without the need for an artifact server. If you do not use an artifact server, the demisto-sdk upload command, uploads the content pack directly to the Marketplace. When content is pushed to master, the content is automatically deployed to the server.

without_artifacts_server.png

By default the CI/CD process uses an artifact server. You can need to update this in the config.yml file.

Although you do not have the flexibility of version control and rollback, it is relatively easy to set up, as you do not need to configure the artifact server and you do not an integration or set up the XSOAR CI/CD Content Pack. The user can decide whether to install from the Marketplace.

To deploy content:#

  1. Configure the config.yml file.

    When you fork the repository, a config.yml file is included in the repository (github\workflows folder).

    1.1 Update the repository details under steps:

    steps:
    - name: XSOAR CI/CD master checkout
    uses: actions/checkout@v2
    with:
    repository: your/repository
    path: repository
    fetch-depth: 0

    For example:

    repository: demisto/content-ci-cd-template
    path: content-ci-cd-template

    1.2 If not using an artifact server, replace the following section:

    # ========= UPLOAD TO ARTIFACTS SERVER OPTION =========
    # TODO: Upload to the artifacts server of your choice.
    # Create a file with the service account data
    # use the bucket_upload script to upload your packs to google cloud storage
    python $GITHUB_WORKSPACE/repository/build_related_scripts/bucket_upload.py --service_account $GITHUB_WORKSPACE/service_account.json --packs_directory $NEW_PACKS_FOLDER --branch_name $BRANCH_NAME
    # Delete the service account file
    rm $GITHUB_WORKSPACE/service_account.json

    With:

    # ========= UPLOAD DIRECTLY TO YOUR XSOAR MACHINE (WHEN MERGING TO MAIN REPO) =========
    if [ $BRANCH_NAME != master ]; then
    CONFIG_FILE=$(cat xsoar_config.json)
    MARKETPLACE_PACKS_LIST=$(cat $CONFIG_FILE | jq -r '.marketplace_packs')
    # Upload Custom Packs
    demisto-sdk upload --input-config-file /xsoar_config.json
    # Upload MarketPlace Packs
    python3 build_related_scripts/MarketPlaceInstallerFromCICD.py --marketplace-packs-list $MARKETPLACE_PACKS_LIST
  2. (Artifact Server only) Configure the the bucket_upload.py file.


    Update the bucket name, main bucket path and format file before pushing any content from your Git repository. The artifact server is based on Google Cloud Storage.
    If using a different storage provider, you need to update the `bucket_upload.py` file with the required artifact server information. Contact Customer Support if you need to change this.
  3. Configure the xsoar_config.json file.

    The xsoar_config.json file defines how custom packs, content packs, lists, and jobs are set up on the production machine. It consists of the following sections:

    ContentDescription
    custom_packsCustom content packs, which are installed through the build process.
    marketplace_packsMarketplace packs to be installed on the machine.
    listsLists that are created in the machine.
    jobsJobs that are created in the machine.

    3.1 Update the ID and URLs for each custom content pack.

    For a non-artifact server, change the URL to the name of the Pack in the following format (using a local URL): "Url": "Packs\name”

    NOTE: For a non-artifact server, if you want to change the version you need to change it in the branch repository and not in this file. If using an artifact server, update the version in the URL. This enables you to have version control. If there was an error in a content pack you can change the version number to an earlier version in your repository.

    config_json.png

    3.2 (Optional) Update Marketplace Content Packs.

    You can also add any Marketplace Content Packs to update. You have the option to deploy from here with the latest version. If you have a lot of out-of-the-box content, use the demisto-sdk xsoar-config-file --add-all-marketplace-packs, command which automatically adds all the installed out of the box Content Packs to the configuration file.

  4. Check that all the changes are validated and are successful.


    The Git repository should look similar to this:

    github_success.png

    In the Master Repository, ensure that the repository is in a structure similar to this:

    β”œβ”€β”€ .github\workflows
    β”‚ β”œβ”€β”€ config.yml
    β”œβ”€β”€ .hooks
    β”‚ β”œβ”€β”€ (your-hooks-here)
    β”œβ”€β”€ vscode
    β”‚ β”œβ”€β”€ extensions.json
    β”œβ”€β”€ build_releated_scripts
    β”‚ β”œβ”€β”€ bucket_upload.py
    β”‚ β”œβ”€β”€ get_modified_packs.py
    β”œβ”€β”€ dev_envs/pytest
    β”‚ β”œβ”€β”€ conftest.py
    β”œβ”€β”€ Packs
    β”‚ β”œβ”€β”€ Pack1
    β”‚ β”‚ β”œβ”€β”€ IncidentFields
    β”‚ β”‚ β”‚ β”œβ”€β”€ (your-incident-field.json)
    β”‚ β”‚ β”‚ β”œβ”€β”€ …
    β”‚ β”‚ β”œβ”€β”€ IncidentTypes
    β”‚ β”‚ β”‚ β”œβ”€β”€ (your-incident-type.json)
    β”‚ β”‚ β”‚ β”œβ”€β”€ …
    β”‚ β”‚ β”œβ”€β”€ Layouts
    β”‚ β”‚ β”‚ β”œβ”€β”€ (your-layout.json)
    β”‚ β”‚ β”‚ β”œβ”€β”€ …
    β”‚ β”‚ β”œβ”€β”€ Playbooks
    β”‚ β”‚ β”‚ β”œβ”€β”€ (your-playbook.yml)
    β”‚ β”‚ β”‚ β”œβ”€β”€ …
    β”‚ β”‚ β”œβ”€β”€ Scripts
    β”‚ β”‚ β”‚ β”œβ”€β”€ (your-script)
    β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ (your-script.py)
    β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ (your-script.yml)
    β”‚ β”‚ β”‚ β”œβ”€β”€ …
    β”‚ β”‚ β”œβ”€β”€ Integrations
    β”‚ β”‚ β”‚ β”œβ”€β”€ (your-integration)
    β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ (your-integration.py)
    β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ (your-integration.yml)
    β”‚ β”‚ β”‚ β”œβ”€β”€ …
    β”‚ β”‚ β”œβ”€β”€ ReleaseNotes
    β”‚ β”‚ β”‚ β”œβ”€β”€ (1_0_1.md)
    β”‚ β”‚ β”‚ β”œβ”€β”€ (1_0_2.md)
    β”‚ β”‚ β”‚ β”œβ”€β”€ …
    β”‚ β”œβ”€β”€ …
    β”œβ”€β”€ .demisto-sdk-conf
    β”œβ”€β”€ .gitignore
    β”œβ”€β”€ .private-repo-settings
    β”œβ”€β”€ CommonServerPowerShell.ps1
    β”œβ”€β”€ CommonServerPython.py
    β”œβ”€β”€ demistomock.py
    β”œβ”€β”€ demistomock.ps1
    β”œβ”€β”€ README.md
    β”œβ”€β”€ requirements.txt
    β”œβ”€β”€ tox.ini
    β”œβ”€β”€ xsoar_config.json
  5. Merge the changes to the master repository.

    The content is uploaded to the artifacts server or directly to the machine. If using an artifact server, open the folder in the artifact server. The hierarchy should appear similar to this:

    β”œβ”€β”€ builds
    β”‚ β”œβ”€β”€ branch-name
    β”‚ β”‚ β”œβ”€β”€ packs
    β”‚ β”‚ β”‚ β”œβ”€β”€ pack1
    β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ 1.0.0
    β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ pack1.zip
    β”‚ β”‚ β”‚ β”œβ”€β”€ pack2
    β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ 1.0.1
    β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ pack2.zip
    β”‚ β”‚ β”‚ β”œβ”€β”€
    β”‚ β”œβ”€β”€
    β”œβ”€β”€ production
    β”‚ β”œβ”€β”€ packs
    β”‚ β”‚ β”œβ”€β”€ pack1
    β”‚ β”‚ β”‚ β”œβ”€β”€ 1.0.0
    β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ pack1.zip
    β”‚ β”‚ β”‚ β”œβ”€β”€ 1.0.1
    β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ pack1.zip
    β”‚ β”‚ β”‚ β”œβ”€β”€ 1.1.0
    β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ pack1.zip
    β”‚ β”‚ β”‚ β”œβ”€β”€
    β”‚ β”‚ β”œβ”€β”€ pack2
    β”‚ β”‚ β”‚ β”œβ”€β”€ 1.0.0
    β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ pack2.zip
    β”‚ β”‚ β”‚ β”œβ”€β”€ 1.0.1
    β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ pack2.zip
    β”‚ β”‚ β”‚ β”œβ”€β”€ 1.0.2
    β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ pack2.zip
    β”‚ β”‚ β”‚ β”œβ”€β”€
    β”‚ β”‚ β”œβ”€β”€
  6. (Non-artifact server) Install the content to Cortex XSOAR.

    In Cortex XSOAR go to Installed Packs and see your custom content packs.

  7. (Artifact Server) Add the content to Cortex XSOAR. You can either use the XSOAR CI/CD Content Pack or download content manually. If using Google Cloud Services, download the Google Cloud Storage Content Pack and set up the integration instance. If using another storage application you need to download the appropriate Content Pack such as AWS.

    • Download the XSOAR CI/CD Content Pack. The Content Pack includes the Configuration Setup playbook, the configuration setup layout, incident fields, automations, etc. The playbook can run via a job or manually. The playbook fetches the configuration file and loads the contents to the machine. It downloads, and installs the custom content packs and configures lists and jobs if part of the content packs. The XSOAR CI/CD Content Pack uses either Google Cloud Storage or HTTP requests to fetch the content packs. If running a different storage provider, you need to download the integration (such as AWS - S3). You need to either create or duplicate the Configuration Setup incident field and add the provider (such as AWS) as the source. You also need to update the Configuration Set_up playbook by adding a task at the same level as Google Cloud Storage.

    cicd_playbook.png

    • If not using the XSOAR CI/CD Content Pack, install content manually. For example, if using Google Cloud Services integration run the gcd-download-file command.

CI/CD FAQs#

  1. We use Dev/Prod. Do I still need this feature? What is the difference?

    This feature is an alternative for the Dev/Prod environment. Instead of building and maintaining your code on a development Cortex XSOAR platform, you can do so from your own chosen git repository and utilize 3rd party tools like CI/CD infrastructures, build servers, artifact servers and more. This adds to your development and testing process. These multiple powerful tools do not exist with the Cortex XSOAR UI feature like Version control, code review, distributed developing environments, automatic testing, etc.

  2. Do I need to have Github in order to use this pack?

    You need a Git repository, but the choice is yours. We currently support Github/Gitlab/Bitbucket.

  3. How does it work with propagation tags and/or in a multi-tenant environment?

    Once the custom content is pulled to the main account, it will be propagated to the tenants per the propagation labels on the content items.

  4. Can I take all the current marketplace packs and turn it into the configuration format?

    Yes, that is supported.