xiaxinhang/pr-agent
1
0
Fork 0
mirror of https://github.com/qodo-ai/pr-agent.git synced 2025-07-13 09:10:38 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #574 from Codium-ai/tr/user_description

Browse Source 
Enhance and reorganize documentation for review and describe tools
This commit is contained in:
Tal
2024-01-07 00:01:30 -08:00
committed by GitHub
parent 6d359fe1d8 94172104f0
commit be4b9c4991
4 changed files with 122 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files

55
docs/DESCRIBE.md
View File

@ -1,8 +1,8 @@
# Describe Tool
## Table of Contents
- [Overview](#overview)
- [Handle custom labels from the Repo's labels page :gem:](#handle-custom-labels-from-the-repos-labels-page-gem)
- [Configuration options](#configuration-options)
- [Handle custom labels from the Repo's labels page :gem:](#handle-custom-labels-from-the-repos-labels-page-gem)
- [Markers template](#markers-template)
- [Usage Tips](#usage-tips)
- [Automation](#automation)
@ -16,31 +16,14 @@ The tool can be triggered automatically every time a new PR is [opened](https://
/describe
```
For example:
___
<kbd><img src=https://codium.ai/images/pr_agent/describe_comment.png width="768"></kbd>
___
<kbd><img src=https://codium.ai/images/pr_agent/describe.png width="768"></kbd>
### Handle custom labels from the Repo's labels page :gem:
> This feature is available only in PR-Agent Pro
You can control the custom labels that will be suggested by the `describe` tool, from the repo's labels page:
* GitHub : go to `https://github.com/{owner}/{repo}/labels` (or click on the "Labels" tab in the issues or PRs page)
* GitLab : go to `https://gitlab.com/{owner}/{repo}/-/labels` (or click on "Manage" -> "Labels" on the left menu)
Now add/edit the custom labels. they should be formatted as follows:
* Label name: The name of the custom label.
* Description: Start the description of with prefix `pr_agent:`, for example: `pr_agent: Description of when AI should suggest this label`.<br>
The description should be comprehensive and detailed, indicating when to add the desired label. For example:
<kbd><img src=https://codium.ai/images/pr_agent/add_native_custom_labels.png width="880"></kbd>
___
### Configuration options
To edit [configurations](./../pr_agent/settings/configuration.toml#L28) related to the describe tool, use the following template:
To edit [configurations](./../pr_agent/settings/configuration.toml#L46) related to the describe tool (`pr_description` section), use the following template:
```
/describe --pr_description.some_config1=... --pr_description.some_config2=...
```
@ -65,6 +48,23 @@ To edit [configurations](./../pr_agent/settings/configuration.toml#L28) related
- `enable_semantic_files_types`: if set to true, "Changes walkthrough" section will be generated. Default is true.
- `collapsible_file_list`: if set to true, the file list in the "Changes walkthrough" section will be collapsible. If set to "adaptive", the file list will be collapsible only if there are more than 8 files. Default is "adaptive".
### Handle custom labels from the Repo's labels page :gem:
> This feature is available only in PR-Agent Pro
You can control the custom labels that will be suggested by the `describe` tool, from the repo's labels page:
* GitHub : go to `https://github.com/{owner}/{repo}/labels` (or click on the "Labels" tab in the issues or PRs page)
* GitLab : go to `https://gitlab.com/{owner}/{repo}/-/labels` (or click on "Manage" -> "Labels" on the left menu)
Now add/edit the custom labels. they should be formatted as follows:
* Label name: The name of the custom label.
* Description: Start the description of with prefix `pr_agent:`, for example: `pr_agent: Description of when AI should suggest this label`.<br>
The description should be comprehensive and detailed, indicating when to add the desired label. For example:
<kbd><img src=https://codium.ai/images/pr_agent/add_native_custom_labels.png width="880"></kbd>
### Markers template
To enable markers, set `pr_description.use_description_markers=true`.
markers enable to easily integrate user's content and auto-generated content, with a template-like mechanism.
@ -97,17 +97,20 @@ The marker `pr_agent:type` will be replaced with the PR type, `pr_agent:summary`
## Usage Tips
1) [Automation](#automation)
2) [Custom labels](#custom-labels)
### Automation
- When you first install the app, the [default mode](https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#github-app-automatic-tools) for the describe tool is:
```
pr_commands = ["/describe --pr_description.add_original_user_description=true"
"--pr_description.keep_original_user_title=true", ...]
```
meaning the `describe` tool will run automatically on every PR, will keep the original title, and will add the original user description above the generated description. <br> This default is quite conservative, and strikes a good balance between automation and control:
meaning the `describe` tool will run automatically on every PR, will keep the original title, and will add the original user description above the generated description.
<br> This default settings aim to strike a good balance between automation and control:
If you want more automation, just give the PR a title, and the tool will auto-write a full description; If you want more control, you can add a detailed description, and the tool will add the complementary description below it.
- For maximal automation, you can change the default mode to:
```
pr_commands = ["/describe --pr_description.add_original_user_description=false"
pr_commands = ["/describe --pr_description.add_original_user_description=false"
"--pr_description.keep_original_user_title=true", ...]
```
so the title will be auto-generated as well.
@ -133,6 +136,6 @@ Examples for custom labels:
- `Dockerfile changes` - pr_agent:The PR contains changes in the Dockerfile
- ...
The list above is eclectic, and aims to give an idea of different possibilities. Define the custom labels that are relevant for your repo and use cases.
The list above is eclectic, and aims to give an idea of different possibilities. Define custom labels that are relevant for your repo and use cases.
Note that Labels are not mutually exclusive, so you can add multiple label categories.
Make sure to provide proper title, and detailed and well-phrased description for each label, so the tool will know when to suggest it.
Make sure to provide proper title, and detailed and well-phrased description for each label, so the tool will know when to suggest it.

113
docs/REVIEW.md
View File

@ -1,44 +1,59 @@
# Review Tool
## Table of Contents
- [Overview](#overview)
- [Configuration options](#configuration-options)
- [Incremental Mode](#incremental-mode)
- [PR Reflection](#pr-reflection)
- [Usage Tips](#usage-tips)
- [General guidelines](#general-guidelines)
- [Code suggestions](#code-suggestions)
- [Automation](#automation)
- [Auto-labels](#auto-labels)
- [Extra instructions](#extra-instructions)
## Overview
The `review` tool scans the PR code changes, and automatically generates a PR review.
It can be invoked manually by commenting on any PR:
The tool can be triggered automatically every time a new PR is [opened](https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#github-app-automatic-tools), or can be invoked manually by commenting on any PR:
```
/review
```
For example:
___
<kbd><img src=https://codium.ai/images/pr_agent/review_comment.png width="768"></kbd>
___
<kbd><img src=https://codium.ai/images/pr_agent/review.png width="768"></kbd>
The `review` tool can also be triggered automatically every time a new PR is opened. See examples for automatic triggers for [GitHub App](https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#github-app-automatic-tools) and [GitHub Action](https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#working-with-github-action)
___
### Configuration options
Under the section 'pr_reviewer', the [configuration file](./../pr_agent/settings/configuration.toml#L16) contains options to customize the 'review' tool:
To edit [configurations](./../pr_agent/settings/configuration.toml#L19) related to the review tool (`pr_reviewer` section), use the following template:
```
/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...
```
#### enable\\disable features
#### General options
- `num_code_suggestions`: number of code suggestions provided by the 'review' tool. Default is 4.
- `inline_code_comments`: if set to true, the tool will publish the code suggestions as comments on the code diff. Default is false.
- `persistent_comment`: if set to true, the review comment will be persistent, meaning that every new review request will edit the previous one. Default is true.
- `extra_instructions`: Optional extra instructions to the tool. For example: "focus on the changes in the file X. Ignore change in ...".
#### Enable\\disable features
- `require_focused_review`: if set to true, the tool will add a section - 'is the PR a focused one'. Default is false.
- `require_score_review`: if set to true, the tool will add a section that scores the PR. Default is false.
- `require_tests_review`: if set to true, the tool will add a section that checks if the PR contains tests. Default is true.
- `require_security_review`: if set to true, the tool will add a section that checks if the PR contains security issues. Default is true.
- `require_estimate_effort_to_review`: if set to true, the tool will add a section that estimates thed effort needed to review the PR. Default is true.
#### general options
- `num_code_suggestions`: number of code suggestions provided by the 'review' tool. Default is 4.
- `inline_code_comments`: if set to true, the tool will publish the code suggestions as comments on the code diff. Default is false.
- `automatic_review`: if set to false, no automatic reviews will be done. Default is true.
- `remove_previous_review_comment`: if set to true, the tool will remove the previous review comment before adding a new one. Default is false.
- `persistent_comment`: if set to true, the review comment will be persistent, meaning that every new review request will edit the previous one. Default is true.
- `extra_instructions`: Optional extra instructions to the tool. For example: "focus on the changes in the file X. Ignore change in ...".
#### SOC2 ticket compliance 💎
This sub-tool checks if the PR description properly contains a ticket to a project management system (e.g., Jira, Asana, Trello, etc.), as required by SOC2 compliance. If not, it will add a label to the PR: "Missing SOC2 ticket".
- `require_soc2_review`: If set to true, the SOC2 ticket checker sub-tool will be enabled. Default is false.
- `soc2_ticket_prompt`: The prompt for the SOC2 ticket review. Default is: `Does the PR description include a link to ticket in a project management system (e.g., Jira, Asana, Trello, etc.) ?`. Edit this field if your compliance requirements are different.
#### review labels
#### Adding PR labels
- `enable_review_labels_security`: if set to true, the tool will publish a 'possible security issue' label if it detects a security issue. Default is true.
- `enable_review_labels_effort`: if set to true, the tool will publish a 'Review effort [1-5]: x' label. Default is false.
- To enable `custom labels`, apply the configuration changes described [here](./GENERATE_CUSTOM_LABELS.md#configuration-changes)
#### Incremental Mode
For an incremental review, which only considers changes since the last PR-Agent review, this can be useful when working on the PR in an iterative manner, and you want to focus on the changes since the last review instead of reviewing the entire PR again, the following command can be used:
### Incremental Mode
Incremental review only considers changes since the last PR-Agent review. This can be useful when working on the PR in an iterative manner, and you want to focus on the changes since the last review instead of reviewing the entire PR again.
For invoking the incremental mode, the following command can be used:
```
/review -i
```
@ -46,7 +61,7 @@ Note that the incremental mode is only available for GitHub.
<kbd><img src=https://codium.ai/images/pr_agent/incremental_review.png width="768"></kbd>
Under the section 'pr_reviewer', the [configuration file](./../pr_agent/settings/configuration.toml#L16) contains options to customize the 'review -i' tool.
Under the section 'pr_reviewer', the [configuration file](./../pr_agent/settings/configuration.toml#L19) contains options to customize the 'review -i' tool.
These configurations can be used to control the rate at which the incremental review tool will create new review comments when invoked automatically, to prevent making too much noise in the PR.
- `minimal_commits_for_incremental_review`: Minimal number of commits since the last review that are required to create incremental review.
If there are less than the specified number of commits since the last review, the tool will not perform any action.
@ -59,8 +74,9 @@ For example, if `minimal_commits_for_incremental_review=2` and `minimal_minutes_
When `require_all_thresholds_for_incremental_review=true` the incremental review __will not__ run, because only 1 out of 2 conditions were met (we have enough commits but the last review is too recent),
but when `require_all_thresholds_for_incremental_review=false` the incremental review __will__ run, because one condition is enough (we have 3 commits which is more than the configured 2).
Default is false - the tool will run as long as at least once conditions is met.
- `remove_previous_review_comment`: if set to true, the tool will remove the previous review comment before adding a new one. Default is false.
#### PR Reflection
### PR Reflection
By invoking:
```
/reflect_and_review
@ -68,16 +84,65 @@ By invoking:
The tool will first ask the author questions about the PR, and will guide the review based on their answers.
<kbd><img src=https://codium.ai/images/pr_agent/reflection_questions.png width="768"></kbd>
___
<kbd><img src=https://codium.ai/images/pr_agent/reflection_answers.png width="768"></kbd>
___
<kbd><img src=https://codium.ai/images/pr_agent/reflection_insights.png width="768"></kbd>
___
#### A note on code suggestions quality
## Usage Tips
1) [General guidelines](#general-guidelines)
2) [Code suggestions](#code-suggestions)
3) [Automation](#automation)
4) [Auto-labels](#auto-labels)
5) [Extra instructions](#extra-instructions)
- With current level of AI for code (GPT-4), mistakes can happen. Not all the suggestions will be perfect, and a user should not accept all of them automatically.
### General guidelines
The `review` tool provides a collection of possible feedbacks about a PR.
It is recommended to review the [Configuration options](#configuration-options) section, and choose the relevant options for your use case.
- Suggestions are not meant to be [simplistic](./../pr_agent/settings/pr_reviewer_prompts.toml#L29). Instead, they aim to give deep feedback and raise questions, ideas and thoughts to the user, who can then use their judgment, experience, and understanding of the code base.
Some of the feature that are disabled by default are quite useful, and should be considered for enabling. For example:
`require_score_review`, `require_soc2_review`, `enable_review_labels_effort`, and more.
- Recommended to use the 'extra_instructions' field to guide the model to suggestions that are more relevant to the specific needs of the project.
On the other hand, if you find one of the enabled features to be irrelevant for your use case, disable it. No default configuration can fit all use cases.
### Code suggestions
The `review` tool provides several type of feedbacks, one of them is code suggestions.
If you are interested **only** in the code suggestions, it is recommended to use the [`improve`](./IMPROVE.md) feature instead, since it dedicated only to code suggestions, and usually gives better results.
Use the `review` tool if you want to get a more comprehensive feedback, which includes code suggestions as well.
### Automation
- When you first install the app, the [default mode](https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#github-app-automatic-tools) for the `review` tool is:
```
pr_commands = ["/review", ...]
```
meaning the `review` tool will run automatically on every PR, with the default configuration.
Edit this field to enable/disable the tool, or to change the used configurations
### Auto-labels
The `review` tool can auto-generate two specific types of labels for a PR:
- a `possible security issue` label if it detects a [security issue](https://github.com/Codium-ai/pr-agent/blob/tr/user_description/pr_agent/settings/pr_reviewer_prompts.toml#L136) (`enable_review_labels_security` flag)
- a `Review effort [1-5]: x` label, where x is the estimated effort to review the PR (`enable_review_labels_effort` flag)
Both modes are useful, and we recommended to enable them.
### Extra instructions
Extra instruction are important.
The `review` tool can be configured with extra instructions, which can be used to guide the model to feedback tailored to the needs of your project.
Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify the relevant sub-tool, and the relevant aspects of the PR that you want to emphasize.
Examples for extra instructions:
```
[pr_reviewer] # /review #
extra_instructions="""
In the code feedback section, emphasize the following:
- Does the code logic covers relevant edge cases?
- Is the code logic clear and easy to understand?
- Is the code logic efficient?
...
"""
```
Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.
- Unlike the 'review' feature, which does a lot of things, the ['improve --extended'](./IMPROVE.md) feature is dedicated only to suggestions, and usually gives better results.

2
pr_agent/settings/configuration.toml
View File

@ -30,7 +30,7 @@ soc2_ticket_prompt="Does the PR description include a link to ticket in a projec
num_code_suggestions=4
inline_code_comments = false
ask_and_reflect=false
automatic_review=true
#automatic_review=true
remove_previous_review_comment=false
persistent_comment=true
extra_instructions = ""

6
pr_agent/tools/pr_reviewer.py
View File

@ -103,9 +103,9 @@ class PRReviewer:
"""
try:
if self.is_auto and not get_settings().pr_reviewer.automatic_review:
get_logger().info(f'Automatic review is disabled {self.pr_url}')
return None
# if self.is_auto and not get_settings().pr_reviewer.automatic_review:
# get_logger().info(f'Automatic review is disabled {self.pr_url}')
# return None
if self.incremental.is_incremental and not self._can_run_incremental_review():
return None