From 83baac975da6fded5302b8358b12654d52e0d353 Mon Sep 17 00:00:00 2001 From: mrT23 Date: Sun, 26 Jan 2025 09:16:52 +0200 Subject: [PATCH 1/5] docs: clarify code suggestion prompt field description for existing code --- pr_agent/settings/pr_code_suggestions_prompts.toml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pr_agent/settings/pr_code_suggestions_prompts.toml b/pr_agent/settings/pr_code_suggestions_prompts.toml index c94cfb6c..6a839a99 100644 --- a/pr_agent/settings/pr_code_suggestions_prompts.toml +++ b/pr_agent/settings/pr_code_suggestions_prompts.toml @@ -30,7 +30,7 @@ __old hunk__ @@ ... @@ def func2(): __new hunk__ unchanged code line4 -+new code line5 removed ++new code line5 added unchanged code line6 ## File: 'src/file2.py' @@ -84,7 +84,7 @@ class CodeSuggestion(BaseModel): relevant_file: str = Field(description="Full path of the relevant file") language: str = Field(description="Programming language used by the relevant file") suggestion_content: str = Field(description="An actionable suggestion to enhance, improve or fix the new code introduced in the PR. Don't present here actual code snippets, just the suggestion. Be short and concise") - existing_code: str = Field(description="A short code snippet from a '__new hunk__' section that the suggestion aims to enhance or fix. Include only complete code lines. Use ellipsis (...) for brevity if needed. This snippet should represent the specific PR code targeted for improvement.") + existing_code: str = Field(description="A short code snippet, from a '__new hunk__' section after the PR changes, that the suggestion aims to enhance or fix. Include only complete code lines. Use ellipsis (...) for brevity if needed. This snippet should represent the specific PR code targeted for improvement.") improved_code: str = Field(description="A refined code snippet that replaces the 'existing_code' snippet after implementing the suggestion.") one_sentence_summary: str = Field(description="A concise, single-sentence overview (up to 6 words) of the suggested improvement. Focus on the 'what'. Be general, and avoid method or variable names.") {%- if not focus_only_on_problems %} From 4d52715d25c7ea6db71a922a3ac226f9174924e7 Mon Sep 17 00:00:00 2001 From: mrT23 Date: Sun, 26 Jan 2025 12:02:27 +0200 Subject: [PATCH 2/5] docs: add auto best practices documentation and configuration --- .../core-abilities/auto_best_practices.md | 64 +++++++++++++++++++ docs/docs/core-abilities/index.md | 1 + docs/docs/tools/improve.md | 39 ++++++++++- docs/mkdocs.yml | 1 + pr_agent/settings/configuration.toml | 9 +++ 5 files changed, 112 insertions(+), 2 deletions(-) create mode 100644 docs/docs/core-abilities/auto_best_practices.md diff --git a/docs/docs/core-abilities/auto_best_practices.md b/docs/docs/core-abilities/auto_best_practices.md new file mode 100644 index 00000000..f2cac762 --- /dev/null +++ b/docs/docs/core-abilities/auto_best_practices.md @@ -0,0 +1,64 @@ +# Auto Best Practices 💎 +`Supported Git Platforms: GitHub` + +## Overview + +### Finding Code Problems - Exploration Phase + +The Qodo Merge `improve` tool identifies potential issues, problems and bugs in Pull Request (PR) code changes. +Rather than focusing on generic concepts like code style, the tool intelligently analyzes code to detect meaningful problems. + +The analysis is intentionally broad to allow discovery of diverse issues - from subtle bugs to architectural concerns. +This open-ended exploration enables the tool to surface any relevant issues without being constrained to predefined categories. + +### Tracking Implemented Suggestions + +Qodo Merge features a novel [tracking system](https://qodo-merge-docs.qodo.ai/tools/improve/#suggestion-tracking) that automatically detects when PR authors implement AI-generated code suggestions. +All accepted suggestions are documented in a repository-specific wiki page called [`.pr_agent_accepted_suggestions`](https://github.com/qodo-ai/pr-agent/wiki/.pr_agent_accepted_suggestions). + +### Learning and Applying Auto Best Practices + +Monthly, Qodo Merge analyzes the collection of accepted suggestions to generate repository-specific best practices, stored in [`.pr_agent_auto_best_practices`](https://github.com/qodo-ai/pr-agent/wiki/.pr_agent_auto_best_practices) wiki file. +These best practices reflect recurring patterns in accepted code improvements. + +The `improve` tool will incorporate these best practices as an additional analysis layer, checking PR code changes against known patterns of previously accepted improvements. +This creates a two-phase analysis: + +1. Open exploration for general code issues +2. Targeted checking against established best practices - exploiting the knowledge gained from past suggestions + +By keeping these phases decoupled, the tool remains free to discover new or unseen issues and problems, while also learning from past experiences. + +When presenting the suggestions generated by the `improve` tool, Qodo Merge will add a dedicated label for each suggestion generated from the auto best practices - 'Learned Best Practice': + +![Auto best practice suggestion](https://www.qodo.ai/images/pr_agent/auto_best_practices.png){width=512} + + +## Auto Best Practices vs Custom Best Practices + +Teams and companies can also manually define their own [custom best practices](https://qodo-merge-docs.qodo.ai/tools/improve/#best-practices) in Qodo Merge to evaluate PR code changes. + +When custom best practices exist, Qodo Merge will still generate an 'auto best practices' wiki file, though it won't use it in the `improve` tool analysis. +However, this auto-generated file can still serve two valuable purposes: + +1. It can help enhance your custom best practices with additional insights derived from suggestions your team found valuable enough to implement +2. It demonstrates effective patterns for writing AI-friendly best practices + +We recommend regularly reviewing the auto best practices file to refine your custom rules, even when using custom best practices. + +## Relevant configurations + +```toml +[auto_best_practices] +# Disable all auto best practices usage or generation +enable_auto_best_practices = true + +# Disable usage of auto best practices file in the 'improve' tool +utilize_auto_best_practices = true + +# Extra instructions to the auto best practices generation prompt +extra_instructions = "" + +# Max number of patterns to be detected +max_patterns = 5 +``` \ No newline at end of file diff --git a/docs/docs/core-abilities/index.md b/docs/docs/core-abilities/index.md index b8895517..a70c04de 100644 --- a/docs/docs/core-abilities/index.md +++ b/docs/docs/core-abilities/index.md @@ -2,6 +2,7 @@ Qodo Merge utilizes a variety of core abilities to provide a comprehensive and efficient code review experience. These abilities include: - [Fetching ticket context](https://qodo-merge-docs.qodo.ai/core-abilities/fetching_ticket_context/) +- [Auto best practices](https://qodo-merge-docs.qodo.ai/core-abilities/auto_best_practices/) - [Local and global metadata](https://qodo-merge-docs.qodo.ai/core-abilities/metadata/) - [Dynamic context](https://qodo-merge-docs.qodo.ai/core-abilities/dynamic_context/) - [Self-reflection](https://qodo-merge-docs.qodo.ai/core-abilities/self_reflection/) diff --git a/docs/docs/tools/improve.md b/docs/docs/tools/improve.md index cfb312a7..58b91a7c 100644 --- a/docs/docs/tools/improve.md +++ b/docs/docs/tools/improve.md @@ -70,7 +70,7 @@ In post-process, Qodo Merge counts the number of suggestions that were implement ## Suggestion tracking 💎 `Platforms supported: GitHub, GitLab` -Qodo Merge employs an novel detection system to automatically [identify](https://qodo-merge-docs.qodo.ai/core-abilities/impact_evaluation/) AI code suggestions that PR authors have accepted and implemented. +Qodo Merge employs a novel detection system to automatically [identify](https://qodo-merge-docs.qodo.ai/core-abilities/impact_evaluation/) AI code suggestions that PR authors have accepted and implemented. Accepted suggestions are also automatically documented in a dedicated wiki page called `.pr_agent_accepted_suggestions`, allowing users to track historical changes, assess the tool's effectiveness, and learn from previously implemented recommendations in the repository. An example [result](https://github.com/Codium-ai/pr-agent/wiki/.pr_agent_accepted_suggestions): @@ -192,7 +192,7 @@ And the label will be: `{organization_name} best practice`. ![best_practice](https://codium.ai/images/pr_agent/org_best_practice.png){width=512} -### How to combine `extra instructions` and `best practices` +### Combining `extra instructions` and `best practices` 💎 The `extra instructions` configuration is more related to the `improve` tool prompt. It can be used, for example, to avoid specific suggestions ("Don't suggest to add try-except block", "Ignore changes in toml files", ...) or to emphasize specific aspects or formats ("Answer in Japanese", "Give only short suggestions", ...) @@ -200,6 +200,41 @@ In contrast, the `best_practices.md` file is a general guideline for the way cod Using a combination of both can help the AI model to provide relevant and tailored suggestions. +### Auto best practices 💎 + +>`Platforms supported: GitHub` + +'Auto best practices' is a novel Qodo Merge capability that: + +1. Identifies recurring patterns from accepted suggestions +2. **Automatically** generates [best practices page](https://github.com/qodo-ai/pr-agent/wiki/.pr_agent_auto_best_practices) based on what your team consistently values +3. Applies these learned patterns to future code reviews + +This creates an automatic feedback loop where the system continuously learns from your team's choices to provide increasingly relevant suggestions. +The system maintains two analysis phases: + +- Open exploration for new issues +- Targeted checking against established best practices + +Note that when a [custom best practices](https://qodo-merge-docs.qodo.ai/tools/improve/#best-practices) exist, Qodo Merge will still generate an 'auto best practices' wiki file, though it won't use it in the `improve` tool. +Learn more about utilizing 'auto best practices' in our [detailed guide](https://qodo-merge-docs.qodo.ai/core-abilities/auto_best_practices/). + +#### Relevant configurations + +```toml +[auto_best_practices] +# Disable all auto best practices usage or generation +enable_auto_best_practices = true + +# Disable usage of auto best practices file in the 'improve' tool +utilize_auto_best_practices = true + +# Extra instructions to the auto best practices generation prompt +extra_instructions = "" + +# Max number of patterns to be detected +max_patterns = 5 +``` ## Usage Tips diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 42ad14df..a817cf54 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -42,6 +42,7 @@ nav: - Core Abilities: - 'core-abilities/index.md' - Fetching ticket context: 'core-abilities/fetching_ticket_context.md' + - Auto best practices: 'core-abilities/auto_best_practices.md' - Local and global metadata: 'core-abilities/metadata.md' - Dynamic context: 'core-abilities/dynamic_context.md' - Self-reflection: 'core-abilities/self_reflection.md' diff --git a/pr_agent/settings/configuration.toml b/pr_agent/settings/configuration.toml index d285b01a..62649d5e 100644 --- a/pr_agent/settings/configuration.toml +++ b/pr_agent/settings/configuration.toml @@ -307,7 +307,16 @@ number_of_results = 5 [lancedb] uri = "./lancedb" + [best_practices] content = "" +organization_name = "" max_lines_allowed = 800 enable_global_best_practices = false + +[auto_best_practices] +enable_auto_best_practices = true # public - general flag to disable all auto best practices usage +utilize_auto_best_practices = true # public - disable usage of auto best practices in the 'improve' tool +extra_instructions = "" # public - extra instructions to the auto best practices generation prompt +content = "" +max_patterns = 5 # max number of patterns to be detected From 2ead9fe4dfb5267ee3688998c36a1ef634ef675b Mon Sep 17 00:00:00 2001 From: mrT23 Date: Sun, 26 Jan 2025 12:07:48 +0200 Subject: [PATCH 3/5] typo --- docs/docs/core-abilities/auto_best_practices.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/core-abilities/auto_best_practices.md b/docs/docs/core-abilities/auto_best_practices.md index f2cac762..db16eaf2 100644 --- a/docs/docs/core-abilities/auto_best_practices.md +++ b/docs/docs/core-abilities/auto_best_practices.md @@ -5,7 +5,7 @@ ### Finding Code Problems - Exploration Phase -The Qodo Merge `improve` tool identifies potential issues, problems and bugs in Pull Request (PR) code changes. +Qodo Merge `improve` tool identifies potential issues, problems and bugs in Pull Request (PR) code changes. Rather than focusing on generic concepts like code style, the tool intelligently analyzes code to detect meaningful problems. The analysis is intentionally broad to allow discovery of diverse issues - from subtle bugs to architectural concerns. From f48c95d1131c4b5fa00b1a874cf62d558f706cdc Mon Sep 17 00:00:00 2001 From: mrT23 Date: Sun, 26 Jan 2025 12:10:30 +0200 Subject: [PATCH 4/5] order --- docs/docs/tools/improve.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/docs/tools/improve.md b/docs/docs/tools/improve.md index 58b91a7c..c9f2bd3c 100644 --- a/docs/docs/tools/improve.md +++ b/docs/docs/tools/improve.md @@ -191,22 +191,13 @@ And the label will be: `{organization_name} best practice`. ![best_practice](https://codium.ai/images/pr_agent/org_best_practice.png){width=512} - -### Combining `extra instructions` and `best practices` 💎 - -The `extra instructions` configuration is more related to the `improve` tool prompt. It can be used, for example, to avoid specific suggestions ("Don't suggest to add try-except block", "Ignore changes in toml files", ...) or to emphasize specific aspects or formats ("Answer in Japanese", "Give only short suggestions", ...) - -In contrast, the `best_practices.md` file is a general guideline for the way code should be written in the repo. - -Using a combination of both can help the AI model to provide relevant and tailored suggestions. - ### Auto best practices 💎 >`Platforms supported: GitHub` 'Auto best practices' is a novel Qodo Merge capability that: -1. Identifies recurring patterns from accepted suggestions +1. Identifies recurring patterns from accepted suggestions 2. **Automatically** generates [best practices page](https://github.com/qodo-ai/pr-agent/wiki/.pr_agent_auto_best_practices) based on what your team consistently values 3. Applies these learned patterns to future code reviews @@ -236,6 +227,16 @@ extra_instructions = "" max_patterns = 5 ``` + +### Combining `extra instructions` and `best practices` 💎 + +The `extra instructions` configuration is more related to the `improve` tool prompt. It can be used, for example, to avoid specific suggestions ("Don't suggest to add try-except block", "Ignore changes in toml files", ...) or to emphasize specific aspects or formats ("Answer in Japanese", "Give only short suggestions", ...) + +In contrast, the `best_practices.md` file is a general guideline for the way code should be written in the repo. + +Using a combination of both can help the AI model to provide relevant and tailored suggestions. + + ## Usage Tips ### Implementing the proposed code suggestions From 63c98d30d9565769d76c2226dc3cf6b1161dccc2 Mon Sep 17 00:00:00 2001 From: mrT23 Date: Sun, 26 Jan 2025 12:57:53 +0200 Subject: [PATCH 5/5] docs: simplify wording and clarify exploration concept in best practices guide --- docs/docs/core-abilities/auto_best_practices.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/docs/core-abilities/auto_best_practices.md b/docs/docs/core-abilities/auto_best_practices.md index db16eaf2..99ab1f97 100644 --- a/docs/docs/core-abilities/auto_best_practices.md +++ b/docs/docs/core-abilities/auto_best_practices.md @@ -8,13 +8,13 @@ Qodo Merge `improve` tool identifies potential issues, problems and bugs in Pull Request (PR) code changes. Rather than focusing on generic concepts like code style, the tool intelligently analyzes code to detect meaningful problems. -The analysis is intentionally broad to allow discovery of diverse issues - from subtle bugs to architectural concerns. -This open-ended exploration enables the tool to surface any relevant issues without being constrained to predefined categories. +The analysis is intentionally broad to allow identifying a wide range of potential issues. +This open-ended _exploration_ enables the tool to surface any relevant issues without being constrained to predefined categories. ### Tracking Implemented Suggestions Qodo Merge features a novel [tracking system](https://qodo-merge-docs.qodo.ai/tools/improve/#suggestion-tracking) that automatically detects when PR authors implement AI-generated code suggestions. -All accepted suggestions are documented in a repository-specific wiki page called [`.pr_agent_accepted_suggestions`](https://github.com/qodo-ai/pr-agent/wiki/.pr_agent_accepted_suggestions). +All accepted suggestions are aggregated in a repository-specific wiki page called [`.pr_agent_accepted_suggestions`](https://github.com/qodo-ai/pr-agent/wiki/.pr_agent_accepted_suggestions). ### Learning and Applying Auto Best Practices