gitsearch

% gitsearch
       _ _                           _
  __ _(_) |_ ___  ___  __ _ _ __ ___| |_
 / _` | | __/ __|/ _ \/ _` | '__/ __| '_ \
| (_| | | |_\__ \  __/ (_| | | | (__| | | |
 \__, |_|\__|___/\___|\__,_|_|  \___|_| |_|
 |___/
Search through GitLab repositories for keywords
github.com/henrybravo/gitsearch
----------------------------------------------------------
Author: Henry Bravo | Version: 1.3.1 | Date: June 2025

Required flags (or config file): -url, -token, and -keywords

Config file options:
  -config         : Path to YAML configuration file
  -generate-config: Generate a sample config file (config.yaml)

Optional flags:
  -exclude        : Comma-separated terms to exclude from project names
  -concurrent     : Maximum number of concurrent goroutines (default 10)
  -output         : Output format: 'json' (default) or 'csv'
  -log            : Enable logging to scan_[timestamp].log file
  -skip-ssl       : Only needed for HTTPS URLs with self-signed certificates
  -h              : Show help menu
  -v, -version    : Show version information

Optional branch scanning flags:
  -branch-age     : Skip branches older than N days (default: 90, 0 for all)
  -branch-names   : Branches/patterns to scan (default: main,master,release/*)
                    Supports wildcards: main, release/*, feature/*, dev/*
  -skip-merged    : Skip merged branches (default: true, use -skip-merged=false to include)
  -branch-exclude : Patterns to exclude (e.g. 'temp-*,old-*')

Examples:
  HTTP instance:  -url http://gitlab.local:8929
  HTTPS instance: -url https://gitlab.local -skip-ssl
  GitLab.com:     -url https://gitlab.com/username
  Config file:    -config my-config.yaml
  Config file with overrides: -config my-config.yaml -url https://gitlab.com/different-user

# Search in GitLab.com user repositories
% gitsearch -url https://gitlab.com/username -token YOUR_TOKEN -keywords "api_key,secret"

# Search in self-hosted instance with SSL skip
% gitsearch -url https://gitlab.local -skip-ssl -token YOUR_TOKEN -keywords "password,credential"

# Customize concurrent operations
% gitsearch -url https://gitlab.com/username -token YOUR_TOKEN -keywords "key" -concurrent 1000

# Exclude specific projects
% gitsearch -url https://gitlab.com/username -token YOUR_TOKEN -keywords "api" -exclude "test,sandbox"

# Using Docker
% docker run -it --rm -v $PWD:/data henrybravo/gitsearch:latest -url https://gitlab.com/username -token YOUR_TOKEN -keywords "api_key"

# Using Docker and config file (config.yaml is in $PWD)
% docker run -it --rm -v $PWD:/data henrybravo/gitsearch:latest -config config.yaml

# Using Docker and debugging enabled
% docker run -it --rm -v $PWD:/data henrybravo/gitsearch:latest -config config.yaml -url https://gitlab.com/different-user -token DIFFERENT_TOKEN -debug

% gitsearch -h

Usage of gitsearch:
  -branch-age int
    	Skip branches with no activity for specified days (0 to scan all) (default 90)
  -branch-exclude string
    	Comma-separated patterns to exclude branches (e.g. 'temp-,old-')
  -branch-names string
    	Comma-separated types or patterns of branches to scan (e.g., main,release/*,feature/*) (default "main,master,release/*")
  -concurrent int
    	Maximum number of concurrent goroutines (default 10)
  -config string
    	Path to configuration file (YAML format)
  -debug
    	Enable debug logging to debug_[timestamp].log file
  -exclude string
    	Comma-separated terms to exclude from project names
  -generate-config
    	Generate a sample configuration file named config.yaml
  -keywords string
    	Comma-separated keywords to search for
  -log
    	Enable logging to scan_[timestamp].log file
  -output string
    	Output format: 'json' (default) or 'csv' (default "json")
  -skip-merged
    	Skip merged branches (default true)
  -skip-ssl
    	Skip SSL verification for self-hosted instances
  -token string
    	GitLab API token
  -url string
    	GitLab URL (e.g., https://gitlab.com/username or https://gitlab.company.com)
  -v	Print version information
  -version
    	Print version information

Execution Summary
=================
Scanned URL: http://gitlab.com/henrybravo
Keyword(s): resource,bucket,token
Excluded repo(s): none
Total Duration: 2.97 seconds
Sequential Processing Time: 7.41 seconds  
Parallelization Speedup: 2.49x

Branch Scanning Configuration
=============================
Branch Age Limit: 1000 days
Branch Types: main,master,release/*
Skip Merged Branches: false
Excluded Branch Patterns: none

Scope
=====
Repositories: 4
Projects: 4
Branches: 4
Files: 18
Matches Found: 287
Concurrency: 1000000 goroutines

Performance Details
===================
Slowest Project: tf-azure-container-app (2.76 seconds)
Avg. Time per File: 0.346 seconds

# Generate a sample config file
% gitsearch -generate-config

# Use a config file
% gitsearch -config config.yaml

# Override config file settings with command line arguments
% gitsearch -config config.yaml -url https://gitlab.com/different-user -token NEW_TOKEN

# GitLab connection settings
gitlab:
    # Base URL of the GitLab instance (required)
    base_url: "https://gitlab.com/[user]"
    
    # API token for authentication (required)
    token: "your-token-here"
    
    # Skip SSL verification for self-hosted instances with self-signed certificates
    skip_ssl: false

# Search settings
search:
    # List of keywords to search for (required if no pattern search configured)
    keywords:
        - "password"
        - "api_key"
    
    # Terms to exclude from project names
    exclude_terms:
        - "test"
    
    # Maximum number of concurrent goroutines
    max_goroutines: 10
    
    # Output format: 'json' or 'csv'
    output_format: "csv"
    
    # Scan Log
    scan_log: true

# Branch filtering
branches:
    # Skip branches with no activity for specified days (0 to scan all)
    max_age: 90
    
    # Branch names or patterns to scan
    names:
        - "main"
        - "master"
        - "release/*"
        - "feature/*"

    # Skip merged branches
    skip_merged: true
    
    # Patterns to exclude branches
    excludes:
        - "temp-*"
        - "old-*"

# Pattern definitions
patterns:
  # Single-line pattern with named capture groups
  - name: "aws_secrets"
    description: "Find AWS access keys"
    regex: "(?i)(?Paws_access_key_id|aws_secret_access_key)\\s*=\\s*['\"]?(?P[^'\"\n]+)['\"]?"
    capture_groups:
        - "key_type"
        - "key_value"

  # Pattern for specific files
  - name: "credentials"
    description: "Find credential patterns in any file"
    regex: "(?i)(?:password|secret|key|token)\\s*[:=]\\s*['\"]([^'\"]+)['\"]"
    captures:
    - name: "credential_value"
        group: 1

  # Multi-line pattern with indexed captures
  - name: "terraform_modules"
    description: "Find Terraform module versions"
    file_extensions: [".tf"]
    regex: "module\\s+\"([^\"]+)\"\\s*{[\\s\\S]*?source\\s*=\\s*\"([^\"]+)\"[\\s\\S]*?version\\s*=\\s*\"([^\"]+)\"[\\s\\S]*?}"
    captures:
    - name: "module_name"
        group: 1
    - name: "source"
        group: 2
    - name: "version" 
        group: 3
    multi_line: true
    
  # Multi-line pattern for provider blocks
  - name: "terraform_providers"
    description: "Find Terraform provider versions"
    file_extensions: [".tf"]
    regex: "required_providers\\s*{[\\s\\S]*?([^=\\s]+)\\s*=\\s*{[\\s\\S]*?source\\s*=\\s*\"([^\"]+)\"[\\s\\S]*?version\\s*=\\s*\"([^\"]+)\"[\\s\\S]*?}"
    captures:
      - name: "provider_name"
        group: 1
      - name: "source"
        group: 2
      - name: "version"
        group: 3
    multi_line: true

  # Docker image pattern
  - name: "docker_images"
    description: "Find Docker image references in YAML files"
    file_extensions: [".yml", ".yaml", ".dockerfile", "Dockerfile"]
    regex: "(?m)(?:^\\s*image:\\s*(?:name:\\s*)?\"?([^\"\\n]+)\"?|^\\s*FROM\\s+([^\\s]+))"
    captures:
      - name: "image"
        group: 1
    multi_line: true

  # Environment variables in various formats
  - name: "env_vars"
    description: "Find environment variable definitions"
    file_extensions: [".env", ".yml", ".yaml", ".sh"]
    regex: "(?m)^\\s*(?:export\\s+)?([A-Z][A-Z0-9_]*)\\s*=\\s*['\"]?(.*?)['\"]?\\s*$"
    captures:
    - name: "var_name"
      group: 1
    - name: "var_value"
      group: 2

  # Kubernetes resource pattern
  - name: "kubernetes_resources"
    description: "Find Kubernetes resource definitions"
    file_extensions: [".yml", ".yaml"]
    regex: "(?m)^kind:\\s*([^\\n]+)\\n[\\s\\S]*?^metadata:\\s*\\n[\\s\\S]*?^\\s*name:\\s*([^\\n]+)"
    captures:
      - name: "resource_type"
        group: 1
      - name: "resource_name"
        group: 2
    multi_line: true

 {
    "project": "terraform-azure-container-app",
    "branch": "main",
    "file": "main.tf",
    "line": {
        "number": 9,
        "content": "module \"analytics_workspace\" {\n  source  = \"app.terraform.io/org/analytics-workspace/azurerm\"\n  version = \"0.0.1\"\n\n  analytics_workspace_name = var.azurerm_log_analytics_workspace_name\n  location = var.location\n  resource_group_name = module.resource_group.resource_group_name\n}",
        "keyword": ""
    },
    "pattern": {
        "name": "terraform_modules",
        "captured_data": {
        "module_name": "analytics_workspace",
        "source": "app.terraform.io/org/analytics-workspace/azurerm",
        "version": "0.0.1"
        }
    }
},
{
    "group": "datascience",
    "project": "aws-ds-sandbox",
    "branch": "main",
    "file": "credentials",
    "line": {
        "number": 3,
        "content": "aws_access_key_id = ABCDEFGHIJKLMNOPQRS",
        "keyword": ""
    },
    "pattern": {
        "name": "aws_secrets",
        "captured_data": {
        "key_type": "aws_access_key_id",
        "key_value": "AKIA6U3NHUDCIJK"
        }
    }
},
{
    "group": "sandbox",
    "project": "mysandbox-1",
    "branch": "main",
    "file": ".env",
    "line": {
      "number": 1,
      "content": "export SECRET=bla",
      "keyword": ""
    },
    "pattern": {
      "name": "env_vars",
      "captured_data": {
        "var_name": "SECRET",
        "var_value": "bla"
      }
    }
},