From e87fdd0ab5ad4dc723eff5353c6c3824bbea4864 Mon Sep 17 00:00:00 2001 From: ofir-frd Date: Thu, 17 Jul 2025 15:43:45 +0300 Subject: [PATCH 1/3] docs: add compliance tool documentation and update feature tables --- README.md | 27 +-- docs/docs/index.md | 1 + docs/docs/tools/compliance.md | 300 ++++++++++++++++++++++++++++++++++ docs/docs/tools/index.md | 1 + docs/mkdocs.yml | 1 + 5 files changed, 317 insertions(+), 13 deletions(-) create mode 100644 docs/docs/tools/compliance.md diff --git a/README.md b/README.md index a4cacda2..42b54828 100644 --- a/README.md +++ b/README.md @@ -119,19 +119,20 @@ Here are some advantages of PR-Agent: PR-Agent and Qodo Merge offer comprehensive pull request functionalities integrated with various git providers: -| | | GitHub | GitLab | Bitbucket | Azure DevOps | Gitea | -|---------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|:------:|:------:|:---------:|:------------:|:-----:| -| [TOOLS](https://qodo-merge-docs.qodo.ai/tools/) | [Describe](https://qodo-merge-docs.qodo.ai/tools/describe/) | ✅ | ✅ | ✅ | ✅ | ✅ | -| | [Review](https://qodo-merge-docs.qodo.ai/tools/review/) | ✅ | ✅ | ✅ | ✅ | ✅ | -| | [Improve](https://qodo-merge-docs.qodo.ai/tools/improve/) | ✅ | ✅ | ✅ | ✅ | ✅ | -| | [Ask](https://qodo-merge-docs.qodo.ai/tools/ask/) | ✅ | ✅ | ✅ | ✅ | | -| | ⮑ [Ask on code lines](https://qodo-merge-docs.qodo.ai/tools/ask/#ask-lines) | ✅ | ✅ | | | | -| | [Help Docs](https://qodo-merge-docs.qodo.ai/tools/help_docs/?h=auto#auto-approval) | ✅ | ✅ | ✅ | | | -| | [Update CHANGELOG](https://qodo-merge-docs.qodo.ai/tools/update_changelog/) | ✅ | ✅ | ✅ | ✅ | | -| | [Add Documentation](https://qodo-merge-docs.qodo.ai/tools/documentation/) 💎 | ✅ | ✅ | | | | -| | [Analyze](https://qodo-merge-docs.qodo.ai/tools/analyze/) 💎 | ✅ | ✅ | | | | -| | [Auto-Approve](https://qodo-merge-docs.qodo.ai/tools/improve/?h=auto#auto-approval) 💎 | ✅ | ✅ | ✅ | | | -| | [CI Feedback](https://qodo-merge-docs.qodo.ai/tools/ci_feedback/) 💎 | ✅ | | | | | +| | | GitHub | GitLab | Bitbucket | Azure DevOps | Gitea | +|---------------------------------------------------------|----------------------------------------------------------------------------------------|:------:|:------:|:---------:|:------------:|:-----:| +| [TOOLS](https://qodo-merge-docs.qodo.ai/tools/) | [Describe](https://qodo-merge-docs.qodo.ai/tools/describe/) | ✅ | ✅ | ✅ | ✅ | ✅ | +| | [Review](https://qodo-merge-docs.qodo.ai/tools/review/) | ✅ | ✅ | ✅ | ✅ | ✅ | +| | [Improve](https://qodo-merge-docs.qodo.ai/tools/improve/) | ✅ | ✅ | ✅ | ✅ | ✅ | +| | [Ask](https://qodo-merge-docs.qodo.ai/tools/ask/) | ✅ | ✅ | ✅ | ✅ | | +| | ⮑ [Ask on code lines](https://qodo-merge-docs.qodo.ai/tools/ask/#ask-lines) | ✅ | ✅ | | | | +| | [Help Docs](https://qodo-merge-docs.qodo.ai/tools/help_docs/?h=auto#auto-approval) | ✅ | ✅ | ✅ | | | +| | [Update CHANGELOG](https://qodo-merge-docs.qodo.ai/tools/update_changelog/) | ✅ | ✅ | ✅ | ✅ | | +| | [Add Documentation](https://qodo-merge-docs.qodo.ai/tools/documentation/) 💎 | ✅ | ✅ | | | | +| | [Analyze](https://qodo-merge-docs.qodo.ai/tools/analyze/) 💎 | ✅ | ✅ | | | | +| | [Auto-Approve](https://qodo-merge-docs.qodo.ai/tools/improve/?h=auto#auto-approval) 💎 | ✅ | ✅ | ✅ | | | +| | [CI Feedback](https://qodo-merge-docs.qodo.ai/tools/ci_feedback/) 💎 | ✅ | | | | | +| | [Compliance](https://qodo-merge-docs.qodo.ai/tools/compliance/) 💎 | ✅ | ✅ | ✅ | | | | | [Custom Prompt](https://qodo-merge-docs.qodo.ai/tools/custom_prompt/) 💎 | ✅ | ✅ | ✅ | | | | | [Generate Custom Labels](https://qodo-merge-docs.qodo.ai/tools/custom_labels/) 💎 | ✅ | ✅ | | | | | | [Generate Tests](https://qodo-merge-docs.qodo.ai/tools/test/) 💎 | ✅ | ✅ | | | | diff --git a/docs/docs/index.md b/docs/docs/index.md index f67a8dbb..b4738d80 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -39,6 +39,7 @@ PR-Agent and Qodo Merge offer comprehensive pull request functionalities integra | | [Analyze](https://qodo-merge-docs.qodo.ai/tools/analyze/) 💎 | ✅ | ✅ | | | | | | [Auto-Approve](https://qodo-merge-docs.qodo.ai/tools/improve/?h=auto#auto-approval) 💎 | ✅ | ✅ | ✅ | | | | | [CI Feedback](https://qodo-merge-docs.qodo.ai/tools/ci_feedback/) 💎 | ✅ | | | | | +| | [Compliance](https://qodo-merge-docs.qodo.ai/tools/compliance/) 💎 | ✅ | ✅ | ✅ | | | | | [Custom Prompt](https://qodo-merge-docs.qodo.ai/tools/custom_prompt/) 💎 | ✅ | ✅ | ✅ | | | | | [Generate Custom Labels](https://qodo-merge-docs.qodo.ai/tools/custom_labels/) 💎 | ✅ | ✅ | | | | | | [Generate Tests](https://qodo-merge-docs.qodo.ai/tools/test/) 💎 | ✅ | ✅ | | | | diff --git a/docs/docs/tools/compliance.md b/docs/docs/tools/compliance.md new file mode 100644 index 00000000..4e5ae111 --- /dev/null +++ b/docs/docs/tools/compliance.md @@ -0,0 +1,300 @@ +`Platforms supported: GitHub, GitLab, Bitbucket` + +## Overview + +The `compliance` tool performs comprehensive compliance checks on PR code changes, validating them against security standards, ticket requirements, and custom organizational compliance checklists. + +=== "Fully Compliant" + ![compliance_overview](https://codium.ai/images/pr_agent/compliance_full.png){width=256} + +=== "Partially Compliant" + ![compliance_overview](https://codium.ai/images/pr_agent/compliance_partial.png){width=256} + +___ + +[//]: # (???+ note "The following features are available only for Qodo Merge 💎 users:") + +[//]: # ( - Custom compliance checklists and hierarchical compliance checklists) + +[//]: # ( - Ticket compliance validation with Jira/Linear integration) + +[//]: # ( - Auto-approval based on compliance status) + +[//]: # ( - Compliance labels and automated enforcement) + +## Example Usage + +### Manual Triggering + +Invoke the tool manually by commenting `/compliance` on any PR. The compliance results are presented in a comprehensive table: + +To edit [configurations](#configuration-options) related to the `compliance` tool, use the following template: + +```toml +/compliance --pr_compliance.some_config1=... --pr_compliance.some_config2=... +``` + +For example, you can enable ticket compliance labels by running: + +```toml +/compliance --pr_compliance.enable_ticket_labels=true +``` + +### Automatic Triggering + + +The tool can be triggered automatically every time a new PR is [opened](https://qodo-merge-docs.qodo.ai/usage-guide/automations_and_usage/#github-app-automatic-tools-when-a-new-pr-is-opened), or in a [push](https://qodo-merge-docs.qodo.ai/usage-guide/automations_and_usage/?h=push#github-app-automatic-tools-for-push-actions-commits-to-an-open-pr) event to an existing PR. + +To run the `compliance` tool automatically when a PR is opened, define the following in the [configuration file](https://qodo-merge-docs.qodo.ai/usage-guide/configuration_options/): + +```toml +[github_app] # for example +pr_commands = [ + "/compliance", + ... +] + +``` + +## Compliance Categories + +The compliance tool evaluates three main categories: + + +### 1. Security Compliance + +Scans for security vulnerabilities and potential exploits in the PR code changes: + +- **Verified Security Concerns** 🔴: Clear security vulnerabilities that require immediate attention +- **Possible Security Risks** ⚪: Potential security issues that need human verification +- **No Security Concerns** 🟢: No security vulnerabilities were detected + +Examples of security issues: + +- Exposure of sensitive information (API keys, passwords, secrets) +- SQL injection vulnerabilities +- Cross-site scripting (XSS) risks +- Cross-site request forgery (CSRF) vulnerabilities +- Insecure data handling patterns + + +### 2. Ticket Compliance + +???+ tip "How to set up ticket compliance" + Follow the guide on how to set up [ticket compliance](https://qodo-merge-docs.qodo.ai/core-abilities/fetching_ticket_context/) with Qodo Merge. + +Validates that PR changes fulfill the requirements specified in linked tickets: + +- **Fully Compliant** 🟢: All ticket requirements are satisfied +- **Partially Compliant** 🟡: Some requirements are met, others need attention +- **Not Compliant** 🔴: Clear violations of ticket requirements +- **Requires Verification** ⚪: Requirements that need human review + + +### 3. Custom Compliance + +Validates against an organization-specific compliance checklist: + +- **Fully Compliant** 🟢: All custom compliance are satisfied +- **Not Compliant** 🔴: Violations of custom compliance +- **Requires Verification** ⚪: Compliance that need human assessment + +## Custom Compliance + +### Setting Up Custom Compliance + +Each compliance is defined in a YAML file, as follows: +- `title`: Used to provide a clear name for the compliance +- `compliance_label`: Used to automatically generate labels for non-compliance issues +- `objective`, `success_criteria`, and `failure_criteria`: These fields are used to clearly define what constitutes compliance + +???+ tip "Example of a compliance checklist" + + ```yaml + # pr_compliance_checklist.yaml + pr_compliances: + - title: "Error Handling" + compliance_label: true + objective: "All external API calls must have proper error handling" + success_criteria: "Try-catch blocks around external calls with appropriate logging" + failure_criteria: "External API calls without error handling or logging" + + ... + ``` + +???+ tip "Writing effective compliance checklist" + - Avoid overly complex or subjective compliances that are hard to verify + - Keep compliances focused on security, business requirements, and critical standards + - Use clear, actionable language that developers can understand + - Focus on meaningful compliance requirements, not style preferences + + +### Global Hierarchical Compliance + +Qodo Merge supports hierarchical compliance checklists using a dedicated global configuration repository. + +#### Setting up global hierarchical compliance + +1\. Create a new repository named `pr-agent-settings` in your organization/workspace. + +2\. Build the folder hierarchy in your `pr-agent-settings` repository: + +```bash +pr-agent-settings/ +├── metadata.yaml # Maps repos/folders to compliance paths +└── compliance_standards/ # Root for all compliance definitions + ├── global/ # Global compliance, inherited widely + │ └── pr_compliance_checklist.yaml + ├── groups/ # For groups of repositories + │ ├── frontend_repos/ + │ │ └── pr_compliance_checklist.yaml + │ └── backend_repos/ + │ └── pr_compliance_checklist.yaml + ├── qodo-merge/ # For standalone repositories + │ └── pr_compliance_checklist.yaml + └── qodo-monorepo/ # For monorepo-specific compliance + ├── pr_compliance_checklist.yaml # Root level monorepo compliance + ├── qodo-github/ # Subproject compliance + │ └── pr_compliance_checklist.yaml + └── qodo-gitlab/ # Another subproject + └── pr_compliance_checklist.yaml +``` + +3\. Define the metadata file `metadata.yaml` in the `pr-agent-settings` root: + +```yaml +# Standalone repos +qodo-merge: + pr_compliance_checklist_paths: + - "qodo-merge" + +# Group-associated repos +repo_b: + pr_compliance_checklist_paths: + - "groups/backend_repos" + +# Multi-group repos +repo_c: + pr_compliance_checklist_paths: + - "groups/frontend_repos" + - "groups/backend_repos" + +# Monorepo with subprojects +qodo-monorepo: + pr_compliance_checklist_paths: + - "qodo-monorepo" + monorepo_subprojects: + frontend: + pr_compliance_checklist_paths: + - "qodo-monorepo/qodo-github" + backend: + pr_compliance_checklist_paths: + - "qodo-monorepo/qodo-gitlab" +``` + +4\. Set the following configuration: + +```toml +[pr_compliance] +enable_global_pr_compliance = true +``` + +???- info "Compliance priority and fallback behavior" + + 1\. **Primary**: Global hierarchical compliance checklists from the `pr-agent-settings` repository: + + 1.1 If the repository is mapped in `metadata.yaml`, it uses the specified paths + + 1.2 For monorepos, it automatically collects compliance checklists matching PR file paths + + 1.3 If no mapping exists, it falls back to the global compliance checklists + + 2. **Fallback**: Local repository wiki `pr_compliance_checklist.yaml` file: + + 2.1 Used when global compliance checklists are not found or configured + + 2.2 Acts as a safety net for repositories not yet configured in the global system + + 2.3 Local wiki compliance checklists are completely ignored when compliance checklists are successfully loaded + + +## Configuration Options + +???+ example "General options" + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
extra_instructionsOptional extra instructions for the tool. For example: "focus on the changes in the file X. Ignore changes in ...". Default is empty string.
persistent_commentIf set to true, the compliance comment will be persistent, meaning that every new compliance request will edit the previous one. Default is true.
enable_user_defined_compliance_labelsIf set to true, the tool will add labels for custom compliance violations. Default is true.
enable_estimate_effort_to_reviewIf set to true, the tool will estimate the effort required to review the PR (1-5 scale). Default is true.
enable_todo_scanIf set to true, the tool will scan for TODO comments in the PR code. Default is false.
enable_update_pr_compliance_checkboxIf set to true, the tool will add an update checkbox to refresh compliance status. Default is true.
enable_help_textIf set to true, the tool will display a help text in the comment. Default is false.
+ +???+ example "Security compliance options" + + + + + + + + + + +
enable_security_complianceIf set to true, the tool will check for security vulnerabilities. Default is true.
enable_compliance_labels_securityIf set to true, the tool will add security-related labels to the PR. Default is true.
+ +???+ example "Ticket compliance options" + + + + + + + + + + + + + + +
enable_ticket_labelsIf set to true, the tool will add ticket compliance labels to the PR. Default is false.
enable_no_ticket_labelsIf set to true, the tool will add a label when no ticket is found. Default is false.
check_pr_additional_contentIf set to true, the tool will check if the PR contains content not related to the ticket. Default is false.
+ + +## Usage Tips + +### Blocking PRs Based on Compliance + +!!! tip "" + You can configure CI/CD Actions to prevent merging PRs with specific compliance labels: + + - `Possible security concern` - Block PRs with potential security issues + - `Failed compliance check` - Block PRs that violate custom compliance checklists + + Implement a dedicated [GitHub Action](https://medium.com/sequra-tech/quick-tip-block-pull-request-merge-using-labels-6cc326936221) to enforce these checklists. + + +The compliance tool provides a comprehensive view of PR quality and adherence to organizational standards, helping teams maintain consistent code quality and security practices while ensuring that development work aligns with business requirements. diff --git a/docs/docs/tools/index.md b/docs/docs/tools/index.md index e50e0785..cce65834 100644 --- a/docs/docs/tools/index.md +++ b/docs/docs/tools/index.md @@ -14,6 +14,7 @@ Here is a list of Qodo Merge tools, each with a dedicated page that explains how | **💎 [Add Documentation (`/add_docs`](./documentation.md))** | Generates documentation to methods/functions/classes that changed in the PR | | **💎 [Analyze (`/analyze`](./analyze.md))** | Identify code components that changed in the PR, and enables to interactively generate tests, docs, and code suggestions for each component | | **💎 [CI Feedback (`/checks ci_job`](./ci_feedback.md))** | Automatically generates feedback and analysis for a failed CI job | +| **💎 [Compliance (`/compliance`](./compliance.md))** | Comprehensive compliance checks for security, ticket requirements, and custom organizational rules | | **💎 [Custom Prompt (`/custom_prompt`](./custom_prompt.md))** | Automatically generates custom suggestions for improving the PR code, based on specific guidelines defined by the user | | **💎 [Generate Custom Labels (`/generate_labels`](./custom_labels.md))** | Generates custom labels for the PR, based on specific guidelines defined by the user | | **💎 [Generate Tests (`/test`](./test.md))** | Automatically generates unit tests for a selected component, based on the PR code changes | diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 894de407..746341a6 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -34,6 +34,7 @@ nav: - 💎 Add Documentation: 'tools/documentation.md' - 💎 Analyze: 'tools/analyze.md' - 💎 CI Feedback: 'tools/ci_feedback.md' + - 💎 Compliance: 'tools/compliance.md' - 💎 Custom Prompt: 'tools/custom_prompt.md' - 💎 Generate Labels: 'tools/custom_labels.md' - 💎 Generate Tests: 'tools/test.md' From d7d4b7de8922fb61d682e010ab827b8e00be8ad3 Mon Sep 17 00:00:00 2001 From: ofir-frd Date: Thu, 17 Jul 2025 15:47:31 +0300 Subject: [PATCH 2/3] docs: improve compliance tool overview and remove redundant conclusion --- docs/docs/tools/compliance.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/docs/tools/compliance.md b/docs/docs/tools/compliance.md index 4e5ae111..27c2e70a 100644 --- a/docs/docs/tools/compliance.md +++ b/docs/docs/tools/compliance.md @@ -2,7 +2,7 @@ ## Overview -The `compliance` tool performs comprehensive compliance checks on PR code changes, validating them against security standards, ticket requirements, and custom organizational compliance checklists. +The `compliance` tool performs comprehensive compliance checks on PR code changes, validating them against security standards, ticket requirements, and custom organizational compliance checklists, thereby helping teams maintain consistent code quality and security practices while ensuring that development work aligns with business requirements. === "Fully Compliant" ![compliance_overview](https://codium.ai/images/pr_agent/compliance_full.png){width=256} @@ -296,5 +296,3 @@ enable_global_pr_compliance = true Implement a dedicated [GitHub Action](https://medium.com/sequra-tech/quick-tip-block-pull-request-merge-using-labels-6cc326936221) to enforce these checklists. - -The compliance tool provides a comprehensive view of PR quality and adherence to organizational standards, helping teams maintain consistent code quality and security practices while ensuring that development work aligns with business requirements. From 7efeeb1de845c6ccf05ea237f74a81911de19eea Mon Sep 17 00:00:00 2001 From: ofir-frd Date: Thu, 17 Jul 2025 15:50:30 +0300 Subject: [PATCH 3/3] docs: fix indentation in compliance tool directory structure example --- docs/docs/tools/compliance.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/tools/compliance.md b/docs/docs/tools/compliance.md index 27c2e70a..edbf4e4f 100644 --- a/docs/docs/tools/compliance.md +++ b/docs/docs/tools/compliance.md @@ -149,7 +149,7 @@ pr-agent-settings/ │ ├── frontend_repos/ │ │ └── pr_compliance_checklist.yaml │ └── backend_repos/ - │ └── pr_compliance_checklist.yaml + │ └── pr_compliance_checklist.yaml ├── qodo-merge/ # For standalone repositories │ └── pr_compliance_checklist.yaml └── qodo-monorepo/ # For monorepo-specific compliance