Add configuration to request a score for the PR

This can help teams compare the review of the PR agent with that of a human reviewer, and fine-tune a score threshold for automatic approval where they decide the agent's review is satisfactory.

CONFIGURATION.md

@@ -9,6 +9,7 @@ You can select your git_provider with the flag `git_provider` in the `config` se
 You can enable/disable the different PR Reviewer abilities with the following flags (`pr_reviewer` section):
 ```
 require_focused_review=true
+require_score_review=true
 require_tests_review=true
 require_security_review=true
 ```

pr_agent/algo/utils.py

@@ -12,6 +12,7 @@ def convert_to_markdown(output_data: dict) -> str:
     emojis = {
         "Main theme": "🎯",
         "Type of PR": "📌",
+        "Score": "🏅",
         "Relevant tests added": "🧪",
         "Unrelated changes": "⚠️",
         "Focused PR": "✨",

pr_agent/settings/configuration.toml

@@ -6,6 +6,7 @@ verbosity_level=0 # 0,1,2
 
 [pr_reviewer]
 require_focused_review=true
+require_score_review=true
 require_tests_review=true
 require_security_review=true
 num_code_suggestions=3

pr_agent/settings/pr_reviewer_prompts.toml

@@ -20,6 +20,12 @@ You must use the following JSON schema to format your answer:
       "type": "string",
       "enum": ["Bug fix", "Tests", "Bug fix with tests", "Refactoring", "Enhancement", "Documentation", "Other"]
     },
+{%- if require_score %}
+    "Score": {
+      "type": "float",
+      "description": "Rate this PR on a scale of 0-100 (inclusive), where 0 means the worst possible code, and 100 means code of the highest quality without any bugs or performace issues that is ready to be merged immediately and run in production at scale."
+    },
+{%- endif %}
 {%- if require_tests %}
     "Relevant tests added": {
       "type": "string",

pr_agent/tools/pr_reviewer.py

@@ -35,6 +35,7 @@ class PRReviewer:
             "description": self.git_provider.get_pr_description(),
             "language": self.main_language,
             "diff": "",  # empty diff for initial calculation
+            "require_score": settings.pr_reviewer.require_score_review,
             "require_tests": settings.pr_reviewer.require_tests_review,
             "require_security": settings.pr_reviewer.require_security_review,
             "require_focused": settings.pr_reviewer.require_focused_review,