From cbfd250c0cbe9244747e355f59d42e50257157cb Mon Sep 17 00:00:00 2001 From: mrT23 Date: Sun, 15 Sep 2024 14:47:27 +0300 Subject: [PATCH 1/2] self_reflection --- docs/docs/core-abilities/self_reflection.md | 45 ++++++++++++++++++++- 1 file changed, 43 insertions(+), 2 deletions(-) diff --git a/docs/docs/core-abilities/self_reflection.md b/docs/docs/core-abilities/self_reflection.md index 12a24c51..46a04eaa 100644 --- a/docs/docs/core-abilities/self_reflection.md +++ b/docs/docs/core-abilities/self_reflection.md @@ -1,2 +1,43 @@ -## Overview - Self-reflection and suggestion cleaning and re-ranking -TBD \ No newline at end of file + +### Introduction - fast review with hierarchical structure + +Given that not all generated code suggestions will be relevant, it is crucial to enable users to review them in a fast and efficient way, allowing quick identification and filtering of non-applicable ones. + +To achieve this goal, PR-Agent offers a dedicated hierarchical structure when presenting suggestions to users: + +- A "category" section groups suggestions by their category, allowing users to quickly disqualify irrelevant suggestions. +- Each suggestion is first described by a one-line summary, which can be expanded to a full description by clicking on a collapsible. +- Upon expanding a suggestion, the user receives a more comprehensive description, and a code snippet demonstrating the recommendation. + +This hierarchical structure is designed to facilitate rapid review of each suggestion, with users spending an average of ~5-10 seconds per item. + +### Self-reflection and re-ranking + +The AI model is initially tasked with generating suggestions, and outputting them in order of importance. +However, in practice we observe that models often struggle to simultaneously generate high-quality code suggestions and rank them well in a single pass. +Furthermore, the initial set of generated suggestions occasionally contains readily identifiable errors. + +To address these issues, we implemented a "self-reflection" process that refines suggestion ranking and eliminates irrelevant or incorrect proposals. +This process consists of the following steps: + +1. Presenting the generated suggestions to the model in a follow-up call. +2. Instructing the model to score each suggestion and provide a rationale for the assigned score. +3. Utilizing these scores to re-rank the suggestions and filter out incorrect ones (with a score of 0). +4. Optionally, filtering out all suggestions below a user-defined score threshold. + +Note that presenting all generated suggestions simultaneously provides the model with a comprehensive context, enabling it to make more informed decisions compared to evaluating each suggestion individually. + +To conclude, the self-reflection process enables PR-Agent to prioritize suggestions based on their importance, eliminate inaccurate or irrelevant proposals, and optionally exclude suggestions that fall below a specified threshold of significance. + +### Example results + +![self_reflection](https://codium.ai/images/pr_agent/self_reflection1.png){width=768} +![self_reflection](https://codium.ai/images/pr_agent/self_reflection2.png){width=768} + + +### Appendix - relevant configuration options +``` +[pr_code_suggestions] +self_reflect_on_suggestions = true # Enable self-reflection on code suggestions +suggestions_score_threshold = 0 # Filter out suggestions with a score below this threshold (0-10) +``` \ No newline at end of file From be03f83318b244ea893da21f5b560bfe352ca095 Mon Sep 17 00:00:00 2001 From: mrT23 Date: Sun, 15 Sep 2024 14:50:24 +0300 Subject: [PATCH 2/2] self_reflection --- docs/docs/core-abilities/self_reflection.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/docs/core-abilities/self_reflection.md b/docs/docs/core-abilities/self_reflection.md index 46a04eaa..1076da25 100644 --- a/docs/docs/core-abilities/self_reflection.md +++ b/docs/docs/core-abilities/self_reflection.md @@ -5,7 +5,7 @@ Given that not all generated code suggestions will be relevant, it is crucial to To achieve this goal, PR-Agent offers a dedicated hierarchical structure when presenting suggestions to users: -- A "category" section groups suggestions by their category, allowing users to quickly disqualify irrelevant suggestions. +- A "category" section groups suggestions by their category, allowing users to quickly dismiss irrelevant suggestions. - Each suggestion is first described by a one-line summary, which can be expanded to a full description by clicking on a collapsible. - Upon expanding a suggestion, the user receives a more comprehensive description, and a code snippet demonstrating the recommendation. @@ -15,19 +15,20 @@ This hierarchical structure is designed to facilitate rapid review of each sugge The AI model is initially tasked with generating suggestions, and outputting them in order of importance. However, in practice we observe that models often struggle to simultaneously generate high-quality code suggestions and rank them well in a single pass. -Furthermore, the initial set of generated suggestions occasionally contains readily identifiable errors. +Furthermore, the initial set of generated suggestions sometimes contains easily identifiable errors. To address these issues, we implemented a "self-reflection" process that refines suggestion ranking and eliminates irrelevant or incorrect proposals. This process consists of the following steps: 1. Presenting the generated suggestions to the model in a follow-up call. -2. Instructing the model to score each suggestion and provide a rationale for the assigned score. +2. Instructing the model to score each suggestion on a scale of 0-10 and provide a rationale for the assigned score. 3. Utilizing these scores to re-rank the suggestions and filter out incorrect ones (with a score of 0). 4. Optionally, filtering out all suggestions below a user-defined score threshold. Note that presenting all generated suggestions simultaneously provides the model with a comprehensive context, enabling it to make more informed decisions compared to evaluating each suggestion individually. To conclude, the self-reflection process enables PR-Agent to prioritize suggestions based on their importance, eliminate inaccurate or irrelevant proposals, and optionally exclude suggestions that fall below a specified threshold of significance. +This results in a more refined and valuable set of suggestions for the user, saving time and improving the overall experience. ### Example results