Security is a critical part of the software development lifecycle, and automating security scans is a best practice for ensuring that vulnerabilities are caught early. In this article, we’ll walk through how to automate security scanning using GitLeaks, Semgrep, and NJSScan, and how to upload the scan results to DefectDojo using a GitLab CI pipeline.

DefectDojo is a powerful open-source security management platform that helps teams manage vulnerabilities and security findings in a centralized location. By integrating it with GitLab CI/CD, you can ensure your pipeline includes automated scanning and streamlined vulnerability management.

Prerequisites

Before diving into the CI pipeline setup, ensure that you have the following in place:

  • DefectDojo instance – Ensure you have an active DefectDojo installation with API access enabled, or try DefectDojo on our demo server for a hands-on experience.
  • DefectDojo API Key – The API key for authenticating to your DefectDojo instance.
  • Engagement ID – The ID for the engagement in DefectDojo where the scan results will be uploaded.
    • Create a product
    • Create a CI/CD engagement for your product
    • Note down the ID of the new engagement
  • GitLab repository – Your project repository should be hosted in GitLab with CI/CD enabled. Refer to my previous article on setting up the Juice Shop repository on GitLab
  • Tools – We will be using the following tools for security scanning:
    • GitLeaks for detecting secrets and sensitive information.
    • Semgrep for finding code vulnerabilities and patterns.
    • NJSScan for scanning JavaScript code for security flaws.

Step-by-Step Setup

Setting Up Environment Variables

To securely manage the credentials and other necessary information for the CI/CD pipeline, I configured environment variables in GitLab. These variables included:

  • DEFECTDOJO_API_IMPORT_SCAN_URL: https://your-defectdojo-instance/api/v2/import-scan/
  • DEFECTDOJO_API_KEY: your-defectdojo-api-key
  • DEFECTDOJO_ENGAGEMENT_ID: your-defectdojo-engagement-id

These variables were added in settings in the GitLab project under CI/CD > Variables.

defectdojo-in-gitLab1

Defining the GitLab CI Pipeline

The following pipeline demonstrates a streamlined CI/CD security scan process using Dockerized tools, showcasing how Docker simplifies CI/CD by enabling each security scan to run in isolated, ready-to-use containers. This approach ensures consistent, repeatable scans and minimal setup, allowing you to focus on automating security checks rather than dealing with dependencies and environment configurations.

Create or update the .gitlab-ci.yml file in your repository. This file defines the CI/CD pipeline that will perform the security scans and upload the results to DefectDojo.

Here’s the complete .gitlab-ci.yml:

stages:
  - security_scan
  - upload_results

variables:
  DEFECTDOJO_API_IMPORT_SCAN_URL: $DEFECTDOJO_API_IMPORT_SCAN_URL 
  DEFECTDOJO_API_KEY: $DEFECTDOJO_API_KEY  
  DEFECTDOJO_ENGAGEMENT_ID: $DEFECTDOJO_ENGAGEMENT_ID  

gitleaks_scan:
  stage: security_scan
  image:
    name: zricethezav/gitleaks
    entrypoint: [""]
  script:
    - gitleaks detect --verbose --source . -f json -r gitleaks_report.json
  artifacts:
    when: always
    paths:
      - gitleaks_report.json
  allow_failure: true

semgrep_scan:
  stage: security_scan
  image: returntocorp/semgrep
  variables:
    SEMGREP_RULES: p/javascript
  script:
    - semgrep ci --json --output semgrep_report.json
  artifacts:
    when: always
    paths:
      - semgrep_report.json
  allow_failure: true

njsscan_scan:
  stage: security_scan
  image: python
  before_script:
    - pip3 install --upgrade njsscan
  script:
    - njsscan --exit-warning . --sarif -o njsscan_report.sarif
  artifacts:
    when: always
    paths:
      - njsscan_report.sarif
  allow_failure: true

upload_to_defectdojo:
  stage: upload_results
  image: python
  when: always
  before_script:
    - pip3 install requests
  script:
    - python3 upload-reports.py gitleaks_report.json
    - python3 upload-reports.py semgrep_report.json
    - python3 upload-reports.py njsscan_report.sarif
  allow_failure: true

Understanding the Pipeline

The pipeline is divided into two main stages:

  • Security Scan: This stage runs the security scanning tools (GitLeaks, Semgrep, NJSScan) to detect vulnerabilities, secrets, and misconfigurations in your code.
  • Upload Results: This stage uploads the scan results to DefectDojo using the DefectDojo API.

Using GitLeaks for Secrets Detection

GitLeaks is a tool that scans your codebase for secrets like API keys, passwords, and tokens. In the pipeline, GitLeaks is configured to run as follows:

gitleaks_scan:
  stage: security_scan
  image:
    name: zricethezav/gitleaks
    entrypoint: [""]
  script:
    - gitleaks detect --verbose --source . -f json -r gitleaks_report.json
  artifacts:
    when: always
    paths:
      - gitleaks_report.json
  allow_failure: true

GitLeaks scans the repository and outputs the results in JSON format, which are stored in an artifact (gitleaks_report.json).

Using Semgrep for Code Vulnerability Detection

Semgrep is used to scan your codebase for security issues and misconfigurations:

semgrep_scan:
  stage: security_scan
  image: returntocorp/semgrep
  variables:
    SEMGREP_RULES: p/javascript
  script:
    - semgrep ci --json --output semgrep_report.json
  artifacts:
    when: always
    paths:
      - semgrep_report.json
  allow_failure: true

It produces a report that is later uploaded to DefectDojo.

Using NJSScan for JavaScript Security Analysis

NJSScan focuses on finding vulnerabilities in JavaScript code:

njsscan_scan:
  stage: security_scan
  image: python
  before_script:
    - pip3 install --upgrade njsscan
  script:
    - njsscan --exit-warning . --sarif -o njsscan_report.sarif
  artifacts:
    when: always
    paths:
      - njsscan_report.sarif
  allow_failure: true

The output is stored in the njsscan_report.json file.

Uploading Scan Results to DefectDojo

The results from each security tool are uploaded to DefectDojo using its REST API. The pipeline leverages a python script to make HTTP POST requests to DefectDojo’s /import-scan endpoint.

Here’s an example for GitLeaks:

upload_to_defectdojo:
  stage: upload_results
  image: python
  when: always
  before_script:
    - pip3 install requests
  script:
    - python3 upload-reports.py gitleaks_report.json
  allow_failure: true

import requests
import sys
import os

DEFECTDOJO_API_IMPORT_SCAN_URL = os.getenv('DEFECTDOJO_API_IMPORT_SCAN_URL')
API_TOKEN = os.getenv('DEFECTDOJO_API_KEY')
ENGAGEMENT_ID = os.getenv('DEFECTDOJO_ENGAGEMENT_ID')
MINIMUM_SEVERITY = 'Low'

# Mapping of file names to scan types
SCAN_TYPE_MAPPING = {
    'gitleaks_report.json': 'Gitleaks Scan',
    'njsscan_report.sarif': 'SARIF',
    'semgrep_report.json': 'Semgrep JSON Report'
}

def get_scan_type(file_name):
    return SCAN_TYPE_MAPPING.get(file_name, 'Unknown Scan Type')

def import_scan_results(file_name, scan_type):
    headers = {
        'Authorization': f'Token {API_TOKEN}'
    }
    data = {
        'active': True,
        'verified': True,
        'scan_type': scan_type,
        'minimum_severity': MINIMUM_SEVERITY,
        'engagement': ENGAGEMENT_ID
    }

    with open(file_name, 'rb') as file:
        files = {'file': file}
        response = requests.post(DEFECTDOJO_API_IMPORT_SCAN_URL, headers=headers, data=data, files=files)

    if response.status_code == 201:
        print('Scan results imported successfully')
    else:
        print(f'Failed to import scan results: {response.content}')

def main():
    if len(sys.argv) < 2:
        print("Usage: python upload-reports.py <file_name>")
        sys.exit(1)

    file_name = sys.argv[1]
    scan_type = get_scan_type(file_name)

    if scan_type == 'Unknown Scan Type':
        print(f"Error: Unsupported file type '{file_name}'")
        sys.exit(1)

    import_scan_results(file_name, scan_type)

if __name__ == '__main__':
    main()

The same process is repeated for Semgrep and NJSScan results, each one specifying the correct scan_type and passing the relevant report file.

Managing Results and Handling Failures

To make the pipeline more flexible, each scan and upload job has allow_failure: true, allowing the pipeline to continue even if one of the security scans fails. This is especially useful when working with security tools that may produce false positives or incomplete results due to configuration or limitations.

defectdojo-in-gitLab2

defectdojo-in-gitLab3

defectdojo-in-gitLab4

Final Thoughts

By integrating security tools like GitLeaks, Semgrep, and NJSScan into your GitLab CI pipeline and uploading results to DefectDojo, you can automate security scanning and vulnerability management in a streamlined manner. This setup ensures that potential security issues are caught early in the development lifecycle and that teams can track and resolve them through DefectDojo.

Start securing your CI/CD pipelines today by implementing these tools and automating the process for continuous security!