xiaxinhang/pr-agent
1
0
Fork 0
mirror of https://github.com/qodo-ai/pr-agent.git synced 2025-07-04 04:40:38 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1726 from qodo-ai/tr/multi_model_prompt

Browse Source 
Tr/multi model prompt
This commit is contained in:
Tal
2025-04-27 11:15:39 +03:00
committed by GitHub
parent 31e5517833 f505c7ad3c
commit 1b0aa16282
9 changed files with 83 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
docs/docs/tools/improve.md
View File

@ -416,18 +416,18 @@ Qodo Merge uses a dynamic strategy to generate code suggestions based on the siz
#### 1. Chunking large PRs
- Qodo Merge divides large PRs into 'chunks'.
- Each chunk contains up to `pr_code_suggestions.max_context_tokens` tokens (default: 14,000).
- Each chunk contains up to `pr_code_suggestions.max_context_tokens` tokens (default: 24,000).
#### 2. Generating suggestions
- For each chunk, Qodo Merge generates up to `pr_code_suggestions.num_code_suggestions_per_chunk` suggestions (default: 3).
- For each chunk, Qodo Merge generates up to `pr_code_suggestions.num_code_suggestions_per_chunk` suggestions (default: 4).
This approach has two main benefits:
- Scalability: The number of suggestions scales with the PR size, rather than being fixed.
- Quality: By processing smaller chunks, the AI can maintain higher quality suggestions, as larger contexts tend to decrease AI performance.
Note: Chunking is primarily relevant for large PRs. For most PRs (up to 500 lines of code), Qodo Merge will be able to process the entire code in a single call.
Note: Chunking is primarily relevant for large PRs. For most PRs (up to 600 lines of code), Qodo Merge will be able to process the entire code in a single call.
## Configuration options
@ -516,4 +516,5 @@ Note: Chunking is primarily relevant for large PRs. For most PRs (up to 500 line
- **Bug detection:** The suggestions also alert on any _critical bugs_ that may have been identified during the analysis. This provides an additional safety net to catch potential issues before they make it into production. It's perfectly acceptable to implement only the suggestions you find valuable for your specific context.
- **Hierarchy:** Presenting the suggestions in a structured hierarchical table enables the user to _quickly_ understand them, and to decide which ones are relevant and which are not.
- **Customization:** To guide the model to suggestions that are more relevant to the specific needs of your project, we recommend to use the [`extra_instructions`](https://qodo-merge-docs.qodo.ai/tools/improve/#extra-instructions-and-best-practices) and [`best practices`](https://qodo-merge-docs.qodo.ai/tools/improve/#best-practices) fields.
- **Model Selection:** SaaS users can also [choose](https://qodo-merge-docs.qodo.ai/usage-guide/qodo_merge_models/) between different models. For specific programming languages or use cases, some models may perform better than others.
- **Interactive usage:** The interactive [PR chat](https://qodo-merge-docs.qodo.ai/chrome-extension/) also provides an easy way to get more tailored suggestions and feedback from the AI model.

14
docs/docs/usage-guide/qodo_merge_models.md
View File

@ -1,16 +1,15 @@
The default model used by Qodo Merge (March 2025) is Claude Sonnet 3.7.
The default models used by Qodo Merge (April 2025) are a combination of Claude Sonnet 3.7 and Gemini 2.5 Pro.
### Selecting a Specific Model
Users can configure Qodo Merge to use a specific model by editing the [configuration](https://qodo-merge-docs.qodo.ai/usage-guide/configuration_options/) file.
Users can configure Qodo Merge to use only a specific model by editing the [configuration](https://qodo-merge-docs.qodo.ai/usage-guide/configuration_options/) file.
The models supported by Qodo Merge are:
- `claude-3-7-sonnet` (default)
- `claude-3-7-sonnet`
- `o4-mini`
- `gpt-4.1`
- `gemini-2.5-pro`
- `gemini-2.5-flash`
- `deepseek/r1`
To restrict Qodo Merge to using only `o4-mini`, add this setting:
@ -34,13 +33,6 @@ To restrict Qodo Merge to using only `gemini-2.5-pro`, add this setting:
model="gemini-2.5-pro"
```
To restrict Qodo Merge to using only `gemini-2.5-flash`, add this setting:
```toml
[config]
model="gemini-2.5-flash"
```
To restrict Qodo Merge to using only `deepseek-r1` us-hosted, add this setting:

52
pr_agent/algo/git_patch_processing.py
View File

@ -102,20 +102,20 @@ def process_patch_lines(patch_str, original_file_str, patch_extra_lines_before,
lines_before_original = file_original_lines[extended_start1 - 1:start1 - 1]
lines_before_new = file_new_lines[extended_start2 - 1:start2 - 1]
found_header = False
if lines_before_original == lines_before_new: # Making sure no changes from a previous hunk
for i, line, in enumerate(lines_before_original):
if section_header in line:
for i, line in enumerate(lines_before_original):
if section_header in line:
# Update start and size in one line each
extended_start1, extended_start2 = extended_start1 + i, extended_start2 + i
extended_size1, extended_size2 = extended_size1 - i, extended_size2 - i
lines_before_original_dynamic_context = lines_before_original[i:]
lines_before_new_dynamic_context = lines_before_new[i:]
if lines_before_original_dynamic_context == lines_before_new_dynamic_context:
# get_logger().debug(f"found dynamic context match for section header: {section_header}")
found_header = True
# Update start and size in one line each
extended_start1, extended_start2 = extended_start1 + i, extended_start2 + i
extended_size1, extended_size2 = extended_size1 - i, extended_size2 - i
# get_logger().debug(f"Found section header in line {i} before the hunk")
section_header = ''
break
else:
get_logger().debug(f"Extra lines before hunk are different in original and new file - dynamic context",
artifact={"lines_before_original": lines_before_original,
"lines_before_new": lines_before_new})
else:
pass # its ok to be here. We cant apply dynamic context if the lines are different if 'old' and 'new' hunks
break
if not found_header:
# get_logger().debug(f"Section header not found in the extra lines before the hunk")
@ -130,14 +130,26 @@ def process_patch_lines(patch_str, original_file_str, patch_extra_lines_before,
if file_new_lines:
delta_lines_new = [f' {line}' for line in file_new_lines[extended_start2 - 1:start2 - 1]]
if delta_lines_original != delta_lines_new:
get_logger().debug(f"Extra lines before hunk are different in original and new file",
artifact={"delta_lines_original": delta_lines_original,
"delta_lines_new": delta_lines_new})
extended_start1 = start1
extended_size1 = size1
extended_start2 = start2
extended_size2 = size2
delta_lines_original = []
found_mini_match = False
for i in range(len(delta_lines_original)):
if delta_lines_original[i:] == delta_lines_new[i:]:
delta_lines_original = delta_lines_original[i:]
delta_lines_new = delta_lines_new[i:]
extended_start1 += i
extended_size1 -= i
extended_start2 += i
extended_size2 -= i
found_mini_match = True
break
if not found_mini_match:
extended_start1 = start1
extended_size1 = size1
extended_start2 = start2
extended_size2 = size2
delta_lines_original = []
# get_logger().debug(f"Extra lines before hunk are different in original and new file",
# artifact={"delta_lines_original": delta_lines_original,
# "delta_lines_new": delta_lines_new})
# logic to remove section header if its in the extra delta lines (in dynamic context, this is also done)
if section_header and not allow_dynamic_context:

8
pr_agent/algo/pr_processing.py
View File

@ -12,7 +12,7 @@ from pr_agent.algo.git_patch_processing import (
from pr_agent.algo.language_handler import sort_files_by_main_languages
from pr_agent.algo.token_handler import TokenHandler
from pr_agent.algo.types import EDIT_TYPE, FilePatchInfo
from pr_agent.algo.utils import ModelType, clip_tokens, get_max_tokens, get_weak_model
from pr_agent.algo.utils import ModelType, clip_tokens, get_max_tokens, get_model
from pr_agent.config_loader import get_settings
from pr_agent.git_providers.git_provider import GitProvider
from pr_agent.log import get_logger
@ -339,7 +339,11 @@ async def retry_with_fallback_models(f: Callable, model_type: ModelType = ModelT
def _get_all_models(model_type: ModelType = ModelType.REGULAR) -> List[str]:
if model_type == ModelType.WEAK:
model = get_weak_model()
model = get_model('model_weak')
elif model_type == ModelType.REASONING:
model = get_model('model_reasoning')
elif model_type == ModelType.REGULAR:
model = get_settings().config.model
else:
model = get_settings().config.model
fallback_models = get_settings().config.fallback_models

8
pr_agent/algo/utils.py
View File

@ -30,12 +30,13 @@ from pr_agent.config_loader import get_settings, global_settings
from pr_agent.log import get_logger
def get_weak_model() -> str:
if get_settings().get("config.model_weak"):
def get_model(model_type: str = "model_weak") -> str:
if model_type == "model_weak" and get_settings().get("config.model_weak"):
return get_settings().config.model_weak
elif model_type == "model_reasoning" and get_settings().get("config.model_reasoning"):
return get_settings().config.model_reasoning
return get_settings().config.model
class Range(BaseModel):
line_start: int # should be 0-indexed
line_end: int
@ -45,6 +46,7 @@ class Range(BaseModel):
class ModelType(str, Enum):
REGULAR = "regular"
WEAK = "weak"
REASONING = "reasoning"
class PRReviewHeader(str, Enum):
REGULAR = "## PR Reviewer Guide"

10
pr_agent/settings/code_suggestions/pr_code_suggestions_prompts_not_decoupled.toml
View File

@ -55,11 +55,11 @@ Specific guidelines for generating code suggestions:
- DO NOT suggest the following:
- change packages version
- add missing import statement
- declare undefined variable
- declare undefined variable, add missing imports, etc.
- use more specific exception types
{%- endif %}
- When mentioning code elements (variables, names, or files) in your response, surround them with backticks (`). For example: "verify that `user_id` is..."
- Note that you will only see partial code segments that were changed (diff hunks in a PR), and not the entire codebase. Avoid suggestions that might duplicate existing functionality or question the existence of code elements like variables, functions, classes, and import statements, that may be defined elsewhere in the codebase.
- When mentioning code elements (variables, names, or files) in your response, surround them with markdown backticks (`). For example: "verify that `user_id` is..."
- Note that you will only see partial code segments that were changed (diff hunks in a PR code), and not the entire codebase. Avoid suggestions that might duplicate existing functionality of the outer codebase. In addition, the absence of a definition, declaration, import, or initialization for any entity in the PR code is NEVER a basis for a suggestion.
- Also note that if the code ends at an opening brace or statement that begins a new scope (like 'if', 'for', 'try'), don't treat it as incomplete. Instead, acknowledge the visible scope boundary and analyze only the code shown.
{%- if extra_instructions %}
@ -77,8 +77,8 @@ The output must be a YAML object equivalent to type $PRCodeSuggestions, accordin
class CodeSuggestion(BaseModel):
relevant_file: str = Field(description="Full path of the relevant file")
language: str = Field(description="Programming language used by the relevant file")
existing_code: str = Field(description="A short code snippet from the final state of the PR diff that the suggestion will address. Select only the span of code that will be modified - without surrounding unchanged code. Preserve all indentation, newlines, and original formatting. Show the code snippet without the '+'/'-'/' ' prefixes. When providing suggestions for long code sections, shorten the presented code with ellipsis (...) for brevity where possible.")
suggestion_content: str = Field(description="An actionable suggestion to enhance, improve or fix the new code introduced in the PR. Don't present here actual code snippets, just the suggestion. Be short and concise")
existing_code: str = Field(description="A short code snippet, from the final state of the PR diff, that the suggestion will address. Select only the specific span of code that will be modified - without surrounding unchanged code. Preserve all indentation, newlines, and original formatting. Show the code snippet without the '+'/'-'/' ' prefixes. When providing suggestions for long code sections, shorten the presented code with ellipsis (...) for brevity where possible.")
suggestion_content: str = Field(description="An actionable suggestion to enhance, improve or fix the new code introduced in the PR. Use 2-3 short sentences.")
improved_code: str = Field(description="A refined code snippet that replaces the 'existing_code' snippet after implementing the suggestion.")
one_sentence_summary: str = Field(description="A single-sentence overview (up to 6 words) of the suggestion. Focus on the 'what'. Be general, and avoid mentioning method or variable names.")
{%- if not focus_only_on_problems %}

22
pr_agent/settings/code_suggestions/pr_code_suggestions_reflect_prompts.toml
View File

@ -1,26 +1,25 @@
[pr_code_suggestions_reflect_prompt]
system="""You are an AI language model specialized in reviewing and evaluating code suggestions for a Pull Request (PR).
Your task is to analyze a PR code diff and evaluate a set of AI-generated code suggestions. These suggestions aim to address potential bugs and problems, and enhance the new code introduced in the PR.
Your task is to analyze a PR code diff and evaluate the correctness and importance set of AI-generated code suggestions.
In addition to evaluating the suggestion correctness and importance, another sub-task you have is to detect the line numbers in the '__new hunk__' of the PR code diff section that correspond to the 'existing_code' snippet.
Examine each suggestion meticulously, assessing its quality, relevance, and accuracy within the context of PR. Keep in mind that the suggestions may vary in their correctness, accuracy and impact.
Consider the following components of each suggestion:
1. 'one_sentence_summary' - A brief summary of the suggestion's purpose
2. 'suggestion_content' - The detailed suggestion content, explaining the proposed modification
1. 'one_sentence_summary' - A one-liner summary summary of the suggestion's purpose
2. 'suggestion_content' - The suggestion content, explaining the proposed modification
3. 'existing_code' - a code snippet from a __new hunk__ section in the PR code diff that the suggestion addresses
4. 'improved_code' - a code snippet demonstrating how the 'existing_code' should be after the suggestion is applied
Be particularly vigilant for suggestions that:
- Overlook crucial details in the PR
- Overlook crucial details in the PR code
- The 'improved_code' section does not accurately reflect the suggested changes, in relation to the 'existing_code'
- Contradict or ignore parts of the PR's modifications
In such cases, assign the suggestion a score of 0.
Evaluate each valid suggestion by scoring its potential impact on the PR's correctness, quality and functionality.
In addition, you should also detect the line numbers in the '__new hunk__' section that correspond to the 'existing_code' snippet.
Key guidelines for evaluation:
- Thoroughly examine both the suggestion content and the corresponding PR code diff. Be vigilant for potential errors in each suggestion, ensuring they are logically sound, accurate, and directly derived from the PR code diff.
- Extend your review beyond the specifically mentioned code lines to encompass surrounding context, verifying the suggestions' contextual accuracy.
- Extend your review beyond the specifically mentioned code lines to encompass surrounding PR code context, verifying the suggestions' contextual accuracy.
- Validate the 'existing_code' field by confirming it matches or is accurately derived from code lines within a '__new hunk__' section of the PR code diff.
- Ensure the 'improved_code' section accurately reflects the 'existing_code' segment after the suggested modification is applied.
- Apply a nuanced scoring system:
@ -30,13 +29,16 @@ Key guidelines for evaluation:
- Maintain the original order of suggestions in your feedback, corresponding to their input sequence.
Additional scoring considerations:
- If the suggestion is not actionable, and only asks the user to verify or ensure a change, reduce its score by 1-2 points.
- If the suggestion only asks the user to verify or ensure a change done in the PR, it should not receive a score above 7 (and may be lower).
- Error handling or type checking suggestions should not receive a score above 8 (and may be lower).
- If the 'existing_code' snippet is equal to the 'improved_code' snippet, it should not receive a score above 7 (and may be lower).
- Assume each suggestion is independent and is not influenced by the other suggestions.
- Assign a score of 0 to suggestions aiming at:
- Adding docstring, type hints, or comments
- Remove unused imports or variables
- Add missing import statements
- Using more specific exception types.
- Questions the definition, declaration, import, or initialization of any entity in the PR code, that might be done in the outer codebase.
@ -87,7 +89,7 @@ class CodeSuggestionFeedback(BaseModel):
relevant_lines_start: int = Field(description="The relevant line number, from a '__new hunk__' section, where the suggestion starts (inclusive). Should be derived from the added '__new hunk__' line numbers, and correspond to the first line of the relevant 'existing code' snippet.")
relevant_lines_end: int = Field(description="The relevant line number, from a '__new hunk__' section, where the suggestion ends (inclusive). Should be derived from the added '__new hunk__' line numbers, and correspond to the end of the relevant 'existing code' snippet")
suggestion_score: int = Field(description="Evaluate the suggestion and assign a score from 0 to 10. Give 0 if the suggestion is wrong. For valid suggestions, score from 1 (lowest impact/importance) to 10 (highest impact/importance).")
why: str = Field(description="Briefly explain the score given in 1-2 sentences, focusing on the suggestion's impact, relevance, and accuracy.")
why: str = Field(description="Briefly explain the score given in 1-2 short sentences, focusing on the suggestion's impact, relevance, and accuracy. When mentioning code elements (variables, names, or files) in your response, surround them with markdown backticks (`).")
class PRCodeSuggestionsFeedback(BaseModel):
code_suggestions: List[CodeSuggestionFeedback]
@ -118,7 +120,7 @@ user="""You are given a Pull Request (PR) code diff:
======
Below are {{ num_code_suggestions }} AI-generated code suggestions for enhancing the Pull Request:
Below are {{ num_code_suggestions }} AI-generated code suggestions for the Pull Request:
======
{{ suggestion_str|trim }}
======

9
pr_agent/settings/configuration.toml
View File

@ -8,6 +8,7 @@
# models
model="o4-mini"
fallback_models=["gpt-4.1"]
#model_reasoning="o4-mini" # dedictated reasoning model for self-reflection
#model_weak="gpt-4o" # optional, a weaker model to use for some easier tasks
# CLI
git_provider="github"
@ -35,8 +36,8 @@ model_token_count_estimate_factor=0.3 # factor to increase the token count estim
# patch extension logic
patch_extension_skip_types =[".md",".txt"]
allow_dynamic_context=true
max_extra_lines_before_dynamic_context = 8 # will try to include up to 10 extra lines before the hunk in the patch, until we reach an enclosing function or class
patch_extra_lines_before = 3 # Number of extra lines (+3 default ones) to include before each hunk in the patch
max_extra_lines_before_dynamic_context = 10 # will try to include up to 10 extra lines before the hunk in the patch, until we reach an enclosing function or class
patch_extra_lines_before = 5 # Number of extra lines (+3 default ones) to include before each hunk in the patch
patch_extra_lines_after = 1 # Number of extra lines (+3 default ones) to include after each hunk in the patch
secret_provider=""
cli_mode=false
@ -123,7 +124,7 @@ use_conversation_history=true
[pr_code_suggestions] # /improve #
max_context_tokens=16000
max_context_tokens=24000
#
commitable_code_suggestions = false
dual_publishing_score_threshold=-1 # -1 to disable, [0-10] to set the threshold (>=) for publishing a code suggestion both in a table and as commitable
@ -144,7 +145,7 @@ new_score_mechanism_th_high=9
new_score_mechanism_th_medium=7
# params for '/improve --extended' mode
auto_extended_mode=true
num_code_suggestions_per_chunk=3
num_code_suggestions_per_chunk=4
max_number_of_calls = 3
parallel_calls = true

16
pr_agent/tools/pr_code_suggestions.py
View File

@ -19,7 +19,7 @@ from pr_agent.algo.pr_processing import (add_ai_metadata_to_diff_files,
retry_with_fallback_models)
from pr_agent.algo.token_handler import TokenHandler
from pr_agent.algo.utils import (ModelType, load_yaml, replace_code_tags,
show_relevant_configurations, get_max_tokens, clip_tokens)
show_relevant_configurations, get_max_tokens, clip_tokens, get_model)
from pr_agent.config_loader import get_settings
from pr_agent.git_providers import (AzureDevopsProvider, GithubProvider,
GitLabProvider, get_git_provider,
@ -121,7 +121,7 @@ class PRCodeSuggestions:
# if not self.is_extended:
# data = await retry_with_fallback_models(self._prepare_prediction, model_type=ModelType.REGULAR)
# else:
data = await retry_with_fallback_models(self._prepare_prediction_extended, model_type=ModelType.REGULAR)
data = await retry_with_fallback_models(self.prepare_prediction_main, model_type=ModelType.REGULAR)
if not data:
data = {"code_suggestions": []}
self.data = data
@ -416,9 +416,15 @@ class PRCodeSuggestions:
data = self._prepare_pr_code_suggestions(response)
# self-reflect on suggestions (mandatory, since line numbers are generated now here)
model_reflection = get_settings().config.model
model_reflect_with_reasoning = get_model('model_reasoning')
fallbacks = get_settings().config.fallback_models
if model_reflect_with_reasoning == get_settings().config.model and model != get_settings().config.model and fallbacks and model == \
fallbacks[0]:
# we are using a fallback model (should not happen on regular conditions)
get_logger().warning(f"Using the same model for self-reflection as the one used for suggestions")
model_reflect_with_reasoning = model
response_reflect = await self.self_reflect_on_suggestions(data["code_suggestions"],
patches_diff, model=model_reflection)
patches_diff, model=model_reflect_with_reasoning)
if response_reflect:
await self.analyze_self_reflection_response(data, response_reflect)
else:
@ -675,7 +681,7 @@ class PRCodeSuggestions:
get_logger().error(f"Error removing line numbers from patches_diff_list, error: {e}")
return patches_diff_list
async def _prepare_prediction_extended(self, model: str) -> dict:
async def prepare_prediction_main(self, model: str) -> dict:
# get PR diff
if get_settings().pr_code_suggestions.decouple_hunks:
self.patches_diff_list = get_pr_multi_diffs(self.git_provider,