Merge pull request #512 from Codium-ai/trlabeling_files_extended

Refactoring and Enhancement of PR Description Formatting in 'pr_description.py'

.github/workflows/pr-agent-review.yaml

@@ -26,7 +26,7 @@ jobs:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           PINECONE.API_KEY: ${{ secrets.PINECONE_API_KEY }}
           PINECONE.ENVIRONMENT: ${{ secrets.PINECONE_ENVIRONMENT }}
-          GITHUB_ACTION.AUTO_REVIEW: 'true'
-          GITHUB_ACTION.AUTO_IMPROVE: 'true'
+          GITHUB_ACTION.AUTO_REVIEW: true
+          GITHUB_ACTION.AUTO_IMPROVE: true
 
 

INSTALL.md

@@ -157,10 +157,11 @@ jobs:
           OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 ```
-2. Add the following secret to your repository under `Settings > Secrets`:
+2. Add the following secret to your repository under `Settings > Secrets and variables > Actions > New repository secret > Add secret`:
 
 ```
-OPENAI_KEY: <your key>
+Name = OPENAI_KEY
+Secret = <your key>
 ```
 
 The GITHUB_TOKEN secret is automatically created by GitHub.
@@ -205,6 +206,7 @@ Allowing you to automate the review process on your private or public repositori
    - Set the following events:
      - Issue comment
      - Pull request
+     - Push (if you need to enable triggering on PR update)
 
 2. Generate a random secret for your app, and save it for later. For example, you can use:
 
@@ -291,7 +293,8 @@ docker push codiumai/pr-agent:github_app  # Push to your Docker repository
     ```
 4. Create a lambda function that uses the uploaded image. Set the lambda timeout to be at least 3m.
 5. Configure the lambda function to have a Function URL.
-6. Go back to steps 8-9 of [Method 5](#run-as-a-github-app) with the function url as your Webhook URL.
+6. In the environment variables of the Lambda function, specify `AZURE_DEVOPS_CACHE_DIR` to a writable location such as /tmp. (see [link](https://github.com/Codium-ai/pr-agent/pull/450#issuecomment-1840242269))
+7. Go back to steps 8-9 of [Method 5](#run-as-a-github-app) with the function url as your Webhook URL.
     The Webhook URL would look like `https://<LAMBDA_FUNCTION_URL>/api/v1/github_webhooks`
 
 ---
@@ -454,6 +457,6 @@ docker push codiumai/pr-agent:bitbucket_server_webhook  # Push to your Docker re
 Navigate to `Projects` or `Repositories`, `Settings`, `Webhooks`, `Create Webhook`.
 Fill the name and URL, Authentication None select the Pull Request Opened checkbox to receive that event as webhook.
 
-The url should be ends with `/webhook`, example: https://domain.com/webhook
+The URL should end with `/webhook`, for example: https://domain.com/webhook
 
 =======

PR_COMPRESSION.md

@@ -39,4 +39,4 @@ We use [tiktoken](https://github.com/openai/tiktoken) to tokenize the patches af
 4. If we haven't reached the max token length, add the `deleted files` to the prompt until the prompt reaches the max token length (hard stop), skip the rest of the patches.
 
 ### Example
-![](https://codium.ai/images/git_patch_logic.png)
+<kbd><img src=https://codium.ai/images/git_patch_logic.png width="768"></kbd>

README.md

@@ -2,8 +2,13 @@
 
 <div align="center">
 
-<img src="./pics/logo-dark.png#gh-dark-mode-only" width="330"/>
-<img src="./pics/logo-light.png#gh-light-mode-only" width="330"/><br/>
+
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://codium.ai/images/pr_agent/logo-dark.png" width="330">
+  <source media="(prefers-color-scheme: light)" srcset="https://codium.ai/images/pr_agent/logo-light.png" width="330">
+  <img alt="logo">
+</picture>
+<br/>
 Making pull requests less painful with an AI agent
 </div>
 
@@ -174,7 +179,7 @@ There are several ways to use PR-Agent:
 
 The following diagram illustrates PR-Agent tools and their flow:
 
-![PR-Agent Tools](https://www.codium.ai/wp-content/uploads/2023/10/codiumai-diagram-v5.png)
+![PR-Agent Tools](https://codium.ai/images/pr_agent/diagram-v0.9.png)
 
 Check out the [PR Compression strategy](./PR_COMPRESSION.md) page for more details on how we convert a code diff to a manageable LLM prompt
 
@@ -220,9 +225,12 @@ See the [Release notes](./RELEASE_NOTES.md) for updates on the latest changes.
 
 ## Data Privacy
 
-If you use self-host PR-Agent, e.g. via CLI running on your computer, with your OpenAI API key, it is between you and OpenAI. You can read their API data privacy policy here:
+If you use a self-hosted PR-Agent with your OpenAI API key, it is between you and OpenAI. You can read their API data privacy policy here:
 https://openai.com/enterprise-privacy
 
+When using a PR-Agent app hosted by CodiumAI, we will not store any of your data, nor will we used it for training.
+You will also benefit from an OpenAI account with zero data retention.
+
 ## Links
 
 [![Join our Discord community](https://raw.githubusercontent.com/Codium-ai/codiumai-vscode-release/main/media/docs/Joincommunity.png)](https://discord.gg/kG35uSHDBc)

RELEASE_NOTES.md

@@ -1,3 +1,21 @@
+## [Version 0.11] - 2023-12-07
+- codiumai/pr-agent:0.11
+- codiumai/pr-agent:0.11-github_app
+- codiumai/pr-agent:0.11-bitbucket-app
+- codiumai/pr-agent:0.11-gitlab_webhook
+- codiumai/pr-agent:0.11-github_polling
+- codiumai/pr-agent:0.11-github_action
+
+### Added::Algo
+- New section in `/describe` tool - [PR changes walkthrough](https://github.com/Codium-ai/pr-agent/pull/509)
+- Improving PR Agent [prompts](https://github.com/Codium-ai/pr-agent/pull/501)
+- Persistent tools (`/review`, `/describe`) now send an [update message](https://github.com/Codium-ai/pr-agent/pull/499) after finishing
+- Add Amazon Bedrock [support](https://github.com/Codium-ai/pr-agent/pull/483)
+
+### Fixed
+- Update [dependencies](https://github.com/Codium-ai/pr-agent/pull/503) in requirements.txt for Python 3.12
+
+
 ## [Version 0.10] - 2023-11-15
 - codiumai/pr-agent:0.10
 - codiumai/pr-agent:0.10-github_app

Usage.md

@@ -6,6 +6,7 @@
 - [Online usage](#online-usage)
 - [Working with GitHub App](#working-with-github-app)
 - [Working with GitHub Action](#working-with-github-action)
+- [Working with BitBucket App](#working-with-bitbucket-self-hosted-app)
 - [Changing a model](#changing-a-model)
 - [Working with large PRs](#working-with-large-prs)
 - [Appendix - additional configurations walkthrough](#appendix---additional-configurations-walkthrough)
@@ -225,6 +226,18 @@ For example, you can set an environment variable: `pr_description.add_original_u
 add_original_user_description = false
 ```
 
+### Working with BitBucket Self-Hosted App
+Similar to GitHub app, when running PR-Agent from BitBucket App, the default [configuration file](pr_agent/settings/configuration.toml) from a pre-built docker will be initially loaded.
+
+By uploading a local `.pr_agent.toml` file to the root of the repo's main branch, you can edit and customize any configuration parameter. Note that you need to upload `.pr_agent.toml` prior to creating a PR, in order for the configuration to take effect.
+
+For example, if your local `.pr_agent.toml` file contains:
+```
+[pr_reviewer]
+inline_code_comments = true
+```
+
+Each time you invoke a `/review` tool, it will use inline code comments.
 
 ### Changing a model
 
@@ -233,7 +246,7 @@ To use a different model than the default (GPT-4), you need to edit [configurati
 For models and environments not from OPENAI, you might need to provide additional keys and other parameters. See below for instructions.
 
 #### Azure
-To use Azure, set in your .secrets.toml:
+To use Azure, set in your `.secrets.toml` (working from CLI), or in the GitHub `Settings > Secrets and variables` (working from GitHub App or GitHub Action):
 ```
 api_key = "" # your azure api key
 api_type = "azure"
@@ -242,12 +255,11 @@ api_base = ""  # The base URL for your Azure OpenAI resource. e.g. "https://<you
 openai.deployment_id = ""  # The deployment name you chose when you deployed the engine
 ```
 
-and
+and set in your configuration file:
 ```
 [config]
 model="" # the OpenAI model you've deployed on Azure (e.g. gpt-3.5-turbo)
 ```
-in the configuration.toml
 
 #### Huggingface
 
@@ -263,7 +275,7 @@ MAX_TOKENS = {
 e.g.
 MAX_TOKENS={
     ...,
-    "llama2": 4096
+    "ollama/llama2": 4096
 }
 
 
@@ -272,6 +284,8 @@ model = "ollama/llama2"
 
 [ollama] # in .secrets.toml
 api_base = ... # the base url for your huggingface inference endpoint
+# e.g. if running Ollama locally, you may use:
+api_base = "http://localhost:11434/"
 ```
 
 **Inference Endpoints**
@@ -328,6 +342,23 @@ Your [application default credentials](https://cloud.google.com/docs/authenticat
 
 If you do want to set explicit credentials then you can use the `GOOGLE_APPLICATION_CREDENTIALS` environment variable set to a path to a json credentials file.
 
+#### Amazon Bedrock
+
+To use Amazon Bedrock and its foundational models, add the below configuration:
+
+``` 
+[config] # in configuration.toml
+model = "anthropic.claude-v2"
+fallback_models="anthropic.claude-instant-v1"
+
+[aws] # in .secrets.toml
+bedrock_region = "us-east-1"
+```
+
+Note that you have to add access to foundational models before using them. Please refer to [this document](https://docs.aws.amazon.com/bedrock/latest/userguide/setting-up.html) for more details.
+
+AWS session is automatically authenticated from your environment, but you can also explicitly set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.
+
 ### Working with large PRs
 
 The default mode of CodiumAI is to have a single call per tool, using GPT-4, which has a token limit of 8000 tokens.

docs/ADD_DOCUMENTATION.md

@@ -7,8 +7,8 @@ It can be invoked manually by commenting on any PR:
 ```
 For example:
 
-<kbd><img src=./../pics/add_docs_comment.png width="768"></kbd>
-<kbd><img src=./../pics/add_docs.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/add_docs_comment.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/add_docs.png width="768"></kbd>
 
 ### Configuration options
  - `docs_style`: The exact style of the documentation (for python docstring). you can choose between: `google`, `numpy`, `sphinx`, `restructuredtext`, `plain`. Default is `sphinx`.

docs/ASK.md

@@ -7,5 +7,5 @@ It can be invoked manually by commenting on any PR:
 ```
 For example:
 
-<kbd><img src=./../pics/ask_comment.png width="768"></kbd>
-<kbd><img src=./../pics/ask.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/ask_comment.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/ask.png width="768"></kbd>

docs/DESCRIBE.md

@@ -1,15 +1,15 @@
 # Describe Tool
 
-The `describe` tool scans the PR code changes, and automatically generates PR description - title, type, summary, code walkthrough and labels.
+The `describe` tool scans the PR code changes, and automatically generates PR description - title, type, summary, walkthrough and labels.
 It can be invoked manually by commenting on any PR:
 ```
 /describe
 ```
 For example:
 
-<kbd><img src=./../pics/describe_comment.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/describe_comment.png width="768"></kbd>
 
-<kbd><img src=./../pics/describe.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/describe.png width="768"></kbd>
 
 The `describe` tool can also be triggered automatically every time a new PR is opened. See examples for automatic triggers for [GitHub App](https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#github-app-automatic-tools) and [GitHub Action](https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#working-with-github-action)
 
@@ -26,8 +26,14 @@ Under the section 'pr_description', the [configuration file](./../pr_agent/setti
 - `keep_original_user_title`: if set to true, the tool will keep the original PR title, and won't change it. Default is false.
 
 - `extra_instructions`: Optional extra instructions to the tool. For example: "focus on the changes in the file X. Ignore change in ...".
+
 - To enable `custom labels`, apply the configuration changes described [here](./GENERATE_CUSTOM_LABELS.md#configuration-changes)
+
 - `enable_pr_type`: if set to false, it will not show the `PR type` as a text value in the description content. Default is true.
+
+- `final_update_message`: if set to true, it will add a comment message [`PR Description updated to latest commit...`](https://github.com/Codium-ai/pr-agent/pull/499#issuecomment-1837412176) after finishing calling `/describe`. Default is true.
+
+- `enable_semantic_files_types`: if set to true, "PR changes walkthrough" section will be generated. Default is true.
   
 ### Markers template
 
@@ -52,11 +58,11 @@ The marker `pr_agent:summary` will be replaced with the PR summary, and `pr_agen
           pr_description.use_description_markers: 'true'
 ```          
 
-<kbd><img src=./../pics/describe_markers_before.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/describe_markers_before.png width="768"></kbd>
 
 ==>
 
-<kbd><img src=./../pics/describe_markers_after.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/describe_markers_after.png width="768"></kbd>
 
 ##### Configuration params:
 

docs/GENERATE_CUSTOM_LABELS.md

@@ -9,10 +9,10 @@ For example:
 
 If we wish to add detect changes to SQL queries in a given PR, we can add the following custom label along with its description:
 
-<kbd><img src=./../pics/custom_labels_list.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/custom_labels_list.png width="768"></kbd>
 
 When running the `generate_labels` tool on a PR that includes changes in SQL queries, it will automatically suggest the custom label:
-<kbd><img src=./../pics/custom_label_published.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/custom_label_published.png width="768"></kbd>
 
 ### How to enable custom labels
 

docs/IMPROVE.md

@@ -7,8 +7,8 @@ It can be invoked manually by commenting on any PR:
 ```
 For example:
 
-<kbd><img src=./../pics/improve_comment.png width="768"></kbd>
-<kbd><img src=./../pics/improve.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/improve_comment.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/improve.png width="768"></kbd>
 
 The `improve` tool can also be triggered automatically every time a new PR is opened. See examples for automatic triggers for [GitHub App](https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#github-app-automatic-tools) and [GitHub Action](https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#working-with-github-action)
 
@@ -42,11 +42,13 @@ For example:
 
 `/improve --pr_code_suggestions.summarize=true`
 
-<kbd><img src=./../pics/improved_summerize_open.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/improved_summerize_open.png width="768"></kbd>
 
-#### A note on code suggestions quality
+___
 
-- With the current level of AI for code (GPT-4), mistakes can happen. Not all the suggestions will be perfect, and a user should not accept all of them automatically.
+### A note on code suggestions quality
+
+- While the current AI for code is getting better and better (GPT-4), it's not flawless. Not all the suggestions will be perfect, and a user should not accept all of them automatically.
 
 - Suggestions are not meant to be [simplistic](./../pr_agent/settings/pr_code_suggestions_prompts.toml#L34). Instead, they aim to give deep feedback and raise questions, ideas and thoughts to the user, who can then use his judgment, experience, and understanding of the code base.
 

docs/REVIEW.md

@@ -7,8 +7,8 @@ It can be invoked manually by commenting on any PR:
 ```
 For example:
 
-<kbd><img src=./../pics/review_comment.png width="768"></kbd>
-<kbd><img src=./../pics/describe.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/review_comment.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/review.png width="768"></kbd>
 
 The `review` tool can also be triggered automatically every time a new PR is opened. See examples for automatic triggers for [GitHub App](https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#github-app-automatic-tools) and [GitHub Action](https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#working-with-github-action)
 
@@ -40,7 +40,7 @@ For an incremental review, which only considers changes since the last PR-Agent
 ```
 Note that the incremental mode is only available for GitHub.
 
-<kbd><img src=./../pics/incremental_review.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/incremental_review.png width="768"></kbd>
 
 Under the section 'pr_reviewer', the [configuration file](./../pr_agent/settings/configuration.toml#L16) contains options to customize the 'review -i' tool.  
 These configurations can be used to control the rate at which the incremental review tool will create new review comments when invoked automatically, to prevent making too much noise in the PR.
@@ -63,9 +63,9 @@ By invoking:
 ```
 The tool will first ask the author questions about the PR, and will guide the review based on his answers.
 
-<kbd><img src=./../pics/reflection_questions.png width="768"></kbd>
-<kbd><img src=./../pics/reflection_answers.png width="768"></kbd>
-<kbd><img src=./../pics/reflection_insights.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/reflection_questions.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/reflection_answers.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/reflection_insights.png width="768"></kbd>
 
 
 #### A note on code suggestions quality
@@ -76,4 +76,4 @@ The tool will first ask the author questions about the PR, and will guide the re
 
 - Recommended to use the 'extra_instructions' field to guide the model to suggestions that are more relevant to the specific needs of the project.
 
-- Unlike the 'review' feature, which does a lot of things, the ['improve --extended'](./IMPROVE.md) feature is dedicated only to suggestions, and usually gives better results.
+- Unlike the 'review' feature, which does a lot of things, the ['improve --extended'](./IMPROVE.md) feature is dedicated only to suggestions, and usually gives better results.

docs/SIMILAR_ISSUE.md

@@ -6,9 +6,9 @@ It can be invoked manually by commenting on any PR:
 ```
 For example:
 
-<kbd><img src=./../pics/similar_issue_original_issue.png width="768"></kbd>
-<kbd><img src=./../pics/similar_issue_comment.png width="768"></kbd>
-<kbd><img src=./../pics/similar_issue.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/similar_issue_original_issue.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/similar_issue_comment.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/similar_issue.png width="768"></kbd>
 
 Note that to perform retrieval, the `similar_issue` tool indexes all the repo previous issues (once).
 

docs/UPDATE_CHANGELOG.md

@@ -7,8 +7,8 @@ It can be invoked manually by commenting on any PR:
 ```
 For example:
 
-<kbd><img src=./../pics/update_changelog_comment.png width="768"></kbd>
-<kbd><img src=./../pics/update_changelog.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/update_changelog_comment.png width="768"></kbd>
+<kbd><img src=https://codium.ai/images/pr_agent/update_changelog.png width="768"></kbd>
 
 
 ### Configuration options

pr_agent/algo/__init__.py

@@ -18,4 +18,7 @@ MAX_TOKENS = {
     'vertex_ai/codechat-bison-32k': 32000,
     'codechat-bison': 6144,
     'codechat-bison-32k': 32000,
+    'anthropic.claude-v2': 100000,
+    'anthropic.claude-instant-v1': 100000,
+    'anthropic.claude-v1': 100000,
 }

pr_agent/algo/ai_handler.py

@@ -1,5 +1,6 @@
 import os
 
+import boto3
 import litellm
 import openai
 from litellm import acompletion
@@ -24,6 +25,7 @@ class AiHandler:
         Raises a ValueError if the OpenAI key is missing.
         """
         self.azure = False
+        self.aws_bedrock_client = None
 
         if get_settings().get("OPENAI.KEY", None):
             openai.api_key = get_settings().openai.key
@@ -60,6 +62,12 @@ class AiHandler:
             litellm.vertex_location = get_settings().get(
                 "VERTEXAI.VERTEX_LOCATION", None
             )
+        if get_settings().get("AWS.BEDROCK_REGION", None):
+            litellm.AmazonAnthropicConfig.max_tokens_to_sample = 2000
+            self.aws_bedrock_client = boto3.client(
+                service_name="bedrock-runtime",
+                region_name=get_settings().aws.bedrock_region,
+            )
 
     @property
     def deployment_id(self):
@@ -100,13 +108,16 @@ class AiHandler:
             if self.azure:
                 model = 'azure/' + model
             messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
-            response = await acompletion(
-                model=model,
-                deployment_id=deployment_id,
-                messages=messages,
-                temperature=temperature,
-                force_timeout=get_settings().config.ai_timeout
-            )
+            kwargs = {
+                "model": model,
+                "deployment_id": deployment_id,
+                "messages": messages,
+                "temperature": temperature,
+                "force_timeout": get_settings().config.ai_timeout,
+            }
+            if self.aws_bedrock_client:
+                kwargs["aws_bedrock_client"] = self.aws_bedrock_client
+            response = await acompletion(**kwargs)
         except (APIError, Timeout, TryAgain) as e:
             get_logger().error("Error during OpenAI inference: ", e)
             raise

pr_agent/algo/utils.py

@@ -325,7 +325,15 @@ def try_fix_yaml(response_text: str) -> dict:
             break
         except:
             pass
-    return data
+    
+    # thrid fallback - try to remove leading and trailing curly brackets
+    response_text_copy = response_text.strip().rstrip().removeprefix('{').removesuffix('}')
+    try:
+        data = yaml.safe_load(response_text_copy,)
+        get_logger().info(f"Successfully parsed AI prediction after removing curly brackets")
+        return data
+    except:
+        pass
 
 
 def set_custom_labels(variables):
@@ -335,7 +343,7 @@ def set_custom_labels(variables):
     labels = get_settings().custom_labels
     if not labels:
         # set default labels
-        labels = ['Bug fix', 'Tests', 'Bug fix with tests', 'Refactoring', 'Enhancement', 'Documentation', 'Other']
+        labels = ['Bug fix', 'Tests', 'Bug fix with tests', 'Enhancement', 'Documentation', 'Other']
         labels_list = "\n      - ".join(labels) if labels else ""
         labels_list = f"      - {labels_list}" if labels_list else ""
         variables["custom_labels"] = labels_list
@@ -359,7 +367,7 @@ def get_user_labels(current_labels: List[str] = None):
             current_labels = []
         user_labels = []
         for label in current_labels:
-            if label.lower() in ['bug fix', 'tests', 'refactoring', 'enhancement', 'documentation', 'other']:
+            if label.lower() in ['bug fix', 'tests', 'enhancement', 'documentation', 'other']:
                 continue
             if get_settings().config.enable_custom_labels:
                 if label in get_settings().custom_labels:
@@ -375,7 +383,11 @@ def get_user_labels(current_labels: List[str] = None):
 
 def get_max_tokens(model):
     settings = get_settings()
-    max_tokens_model = MAX_TOKENS[model]
+    if model in MAX_TOKENS:
+        max_tokens_model = MAX_TOKENS[model]
+    else:
+        raise Exception(f"MAX_TOKENS must be set for model {model} in ./pr_agent/algo/__init__.py")
+
     if settings.config.max_model_tokens:
         max_tokens_model = min(settings.config.max_model_tokens, max_tokens_model)
         # get_logger().debug(f"limiting max tokens to {max_tokens_model}")

pr_agent/git_providers/bitbucket_provider.py

@@ -229,7 +229,10 @@ class BitbucketProvider(GitProvider):
         return response
 
     def get_line_link(self, relevant_file: str, relevant_line_start: int, relevant_line_end: int = None) -> str:
-        link = f"{self.pr_url}/#L{relevant_file}T{relevant_line_start}"
+        if relevant_line_start == -1:
+            link = f"{self.pr_url}/#L{relevant_file}"
+        else:
+            link = f"{self.pr_url}/#L{relevant_file}T{relevant_line_start}"
         return link
 
     def generate_link_to_relevant_line_number(self, suggestion) -> str:

pr_agent/git_providers/codecommit_provider.py

@@ -61,6 +61,7 @@ class CodeCommitProvider(GitProvider):
         self.pr = None
         self.diff_files = None
         self.git_files = None
+        self.pr_url = pr_url
         if pr_url:
             self.set_pr(pr_url)
 

pr_agent/git_providers/gerrit_provider.py

@@ -192,7 +192,7 @@ class GerritProvider(GitProvider):
         )
         self.repo = Repo(self.repo_path)
         assert self.repo
-
+        self.pr_url = base_url
         self.pr = PullRequestMimic(self.get_pr_title(), self.get_diff_files())
 
     def get_pr_title(self):

pr_agent/git_providers/git_provider.py

@@ -26,6 +26,8 @@ class FilePatchInfo:
     tokens: int = -1
     edit_type: EDIT_TYPE = EDIT_TYPE.UNKNOWN
     old_filename: str = None
+    num_plus_lines: int = -1
+    num_minus_lines: int = -1
 
 
 class GitProvider(ABC):

pr_agent/git_providers/github_provider.py

@@ -34,6 +34,7 @@ class GithubProvider(GitProvider):
         if pr_url and 'pull' in pr_url:
             self.set_pr(pr_url)
             self.last_commit_id = list(self.pr.get_commits())[-1]
+            self.pr_url = self.get_pr_url() # pr_url for github actions can be as api.github.com, so we need to get the url from the pr object
 
     def is_supported(self, capability: str) -> bool:
         return True
@@ -60,6 +61,8 @@ class GithubProvider(GitProvider):
                     get_logger().info(f"Skipping merge commit {commit.commit.message}")
                     continue
                 self.file_set.update({file.filename: file for file in commit.files})
+        else:
+            raise ValueError("No previous review found")
 
     def get_commit_range(self):
         last_review_time = self.previous_review.created_at
@@ -140,8 +143,15 @@ class GithubProvider(GitProvider):
                 else:
                     get_logger().error(f"Unknown edit type: {file.status}")
                     edit_type = EDIT_TYPE.UNKNOWN
+
+                # count number of lines added and removed
+                patch_lines = patch.splitlines(keepends=True)
+                num_plus_lines = len([line for line in patch_lines if line.startswith('+')])
+                num_minus_lines = len([line for line in patch_lines if line.startswith('-')])
                 file_patch_canonical_structure = FilePatchInfo(original_file_content_str, new_file_content_str, patch,
-                                                               file.filename, edit_type=edit_type)
+                                                               file.filename, edit_type=edit_type,
+                                                               num_plus_lines=num_plus_lines,
+                                                               num_minus_lines=num_minus_lines,)
                 diff_files.append(file_patch_canonical_structure)
 
             self.diff_files = diff_files
@@ -439,7 +449,7 @@ class GithubProvider(GitProvider):
     def publish_labels(self, pr_types):
         try:
             label_color_map = {"Bug fix": "1d76db", "Tests": "e99695", "Bug fix with tests": "c5def5",
-                               "Refactoring": "bfdadc", "Enhancement": "bfd4f2", "Documentation": "d4c5f9",
+                               "Enhancement": "bfd4f2", "Documentation": "d4c5f9",
                                "Other": "d1bcf9"}
             post_parameters = []
             for p in pr_types:
@@ -503,7 +513,9 @@ class GithubProvider(GitProvider):
 
     def get_line_link(self, relevant_file: str, relevant_line_start: int, relevant_line_end: int = None) -> str:
         sha_file = hashlib.sha256(relevant_file.encode('utf-8')).hexdigest()
-        if relevant_line_end:
+        if relevant_line_start == -1:
+            link = f"https://github.com/{self.repo}/pull/{self.pr_num}/files#diff-{sha_file}"
+        elif relevant_line_end:
             link = f"https://github.com/{self.repo}/pull/{self.pr_num}/files#diff-{sha_file}R{relevant_line_start}-R{relevant_line_end}"
         else:
             link = f"https://github.com/{self.repo}/pull/{self.pr_num}/files#diff-{sha_file}R{relevant_line_start}"

pr_agent/git_providers/gitlab_provider.py

@@ -37,6 +37,7 @@ class GitLabProvider(GitProvider):
         self.diff_files = None
         self.git_files = None
         self.temp_comments = []
+        self.pr_url = merge_request_url
         self._set_merge_request(merge_request_url)
         self.RE_HUNK_HEADER = re.compile(
             r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@[ ]?(.*)")
@@ -114,12 +115,20 @@ class GitLabProvider(GitProvider):
                 if not patch:
                     patch = load_large_diff(filename, new_file_content_str, original_file_content_str)
 
+
+                # count number of lines added and removed
+                patch_lines = patch.splitlines(keepends=True)
+                num_plus_lines = len([line for line in patch_lines if line.startswith('+')])
+                num_minus_lines = len([line for line in patch_lines if line.startswith('-')])
                 diff_files.append(
                     FilePatchInfo(original_file_content_str, new_file_content_str,
                                   patch=patch,
                                   filename=filename,
                                   edit_type=edit_type,
-                                  old_filename=None if diff['old_path'] == diff['new_path'] else diff['old_path']))
+                                  old_filename=None if diff['old_path'] == diff['new_path'] else diff['old_path'],
+                                  num_plus_lines=num_plus_lines,
+                                  num_minus_lines=num_minus_lines, ))
+
         self.diff_files = diff_files
         return diff_files
 
@@ -423,7 +432,9 @@ class GitLabProvider(GitProvider):
             return ""
 
     def get_line_link(self, relevant_file: str, relevant_line_start: int, relevant_line_end: int = None) -> str:
-        if relevant_line_end:
+        if relevant_line_start == -1:
+            link = f"https://gitlab.com/codiumai/pr-agent/-/blob/{self.mr.source_branch}/{relevant_file}?ref_type=heads"
+        elif relevant_line_end:
             link = f"https://gitlab.com/codiumai/pr-agent/-/blob/{self.mr.source_branch}/{relevant_file}?ref_type=heads#L{relevant_line_start}-L{relevant_line_end}"
         else:
             link = f"https://gitlab.com/codiumai/pr-agent/-/blob/{self.mr.source_branch}/{relevant_file}?ref_type=heads#L{relevant_line_start}"

pr_agent/servers/help.py

@@ -1,19 +1,19 @@
-commands_text = "> **/review**: Request a review of your Pull Request.\n" \
-                "> **/describe**: Update the PR title and description based on the contents of the PR.\n" \
-                "> **/improve [--extended]**: Suggest code improvements. Extended mode provides a higher quality feedback.\n" \
-                "> **/ask \\<QUESTION\\>**: Ask a question about the PR.\n" \
-                "> **/update_changelog**: Update the changelog based on the PR's contents.\n" \
-                "> **/add_docs**: Generate docstring for new components introduced in the PR.\n" \
-                "> **/generate_labels**: Generate labels for the PR based on the PR's contents.\n" \
+commands_text = "> **/review**: Request a review of your Pull Request.   \n" \
+                "> **/describe**: Update the PR title and description based on the contents of the PR.   \n" \
+                "> **/improve [--extended]**: Suggest code improvements. Extended mode provides a higher quality feedback.   \n" \
+                "> **/ask \\<QUESTION\\>**: Ask a question about the PR.   \n" \
+                "> **/update_changelog**: Update the changelog based on the PR's contents.   \n" \
+                "> **/add_docs**: Generate docstring for new components introduced in the PR.   \n" \
+                "> **/generate_labels**: Generate labels for the PR based on the PR's contents.   \n" \
                 "> see the [tools guide](https://github.com/Codium-ai/pr-agent/blob/main/docs/TOOLS_GUIDE.md) for more details.\n\n" \
-                ">To edit any configuration parameter from the [configuration.toml](https://github.com/Codium-ai/pr-agent/blob/main/pr_agent/settings/configuration.toml), add --config_path=new_value.\n" \
-                ">For example: /review --pr_reviewer.extra_instructions=\"focus on the file: ...\" \n" \
-                ">To list the possible configuration parameters, add a **/config** comment.\n" \
+                ">To edit any configuration parameter from the [configuration.toml](https://github.com/Codium-ai/pr-agent/blob/main/pr_agent/settings/configuration.toml), add --config_path=new_value.  \n" \
+                ">For example: /review --pr_reviewer.extra_instructions=\"focus on the file: ...\"    \n" \
+                ">To list the possible configuration parameters, add a **/config** comment.   \n" \
 
 
 def bot_help_text(user: str):
-    return f"> Tag me in a comment '@{user}' and add one of the following commands:\n" + commands_text
+    return f"> Tag me in a comment '@{user}' and add one of the following commands:  \n" + commands_text
 
 
-actions_help_text = "> To invoke the PR-Agent, add a comment using one of the following commands:\n" + \
+actions_help_text = "> To invoke the PR-Agent, add a comment using one of the following commands:  \n" + \
                     commands_text

pr_agent/settings/.secrets_template.toml

@@ -40,6 +40,9 @@ api_base = "" # the base url for your local Llama 2, Code Llama, and other model
 vertex_project = "" # the google cloud platform project name for your vertexai deployment
 vertex_location = "" # the google cloud platform location for your vertexai deployment
 
+[aws]
+bedrock_region = "" # the AWS region to call Bedrock APIs
+
 [github]
 # ---- Set the following only for deployment type == "user"
 user_token = ""  # A GitHub personal access token with 'repo' scope.

pr_agent/settings/configuration.toml

@@ -46,12 +46,16 @@ keep_original_user_title=false
 use_bullet_points=true
 extra_instructions = ""
 enable_pr_type=true
+enable_file_walkthrough=false
+enable_semantic_files_types=true
+final_update_message = true
+
 
 # markers
 use_description_markers=false
 include_generated_by_header=true
 
-#custom_labels = ['Bug fix', 'Tests', 'Bug fix with tests', 'Refactoring', 'Enhancement', 'Documentation', 'Other']
+#custom_labels = ['Bug fix', 'Tests', 'Bug fix with tests', 'Enhancement', 'Documentation', 'Other']
 
 [pr_questions] # /ask #
 

pr_agent/settings/custom_labels.toml

@@ -8,10 +8,8 @@ enable_custom_labels=false
 #description = """Adds or modifies tests"""
 #[custom_labels."Bug fix with tests"]
 #description = """Fixes a bug in the code and adds or modifies tests"""
-#[custom_labels."Refactoring"]
-#description = """Code refactoring without changing functionality"""
 #[custom_labels."Enhancement"]
-#description = """Adds new features or functionality"""
+#description = """Adds new features or modifies existing ones"""
 #[custom_labels."Documentation"]
 #description = """Adds or modifies documentation"""
 #[custom_labels."Other"]

pr_agent/settings/pr_add_docs.toml

@@ -1,22 +1,22 @@
 [pr_add_docs_prompt]
-system="""You are a language model called PR-Code-Documentation Agent, that specializes in generating documentation for code.
-Your task is to generate meaningfull {{ docs_for_language }} to a PR (lines starting with '+').
+system="""You are PR-Doc, a language model that specializes in generating documentation for code components in a Pull Request (PR).
+Your task is to generate {{ docs_for_language }} for code components in the PR Diff.
 
-Example for a PR Diff input:
-'
+
+Example for the PR Diff format:
+======
 ## src/file1.py
 
-@@ -12,3 +12,5 @@ def func1():
+@@ -12,3 +12,4 @@ def func1():
 __new hunk__
-12  code line that already existed in the file...
-13  code line that already existed in the file....
+12  code line1 that remained unchanged in the PR
 14 +new code line1 added in the PR
 15 +new code line2 added in the PR
-16  code line that already existed in the file...
+16  code line2 that remained unchanged in the PR
 __old hunk__
- code line that already existed in the file...
+ code line1 that remained unchanged in the PR
 -code line that was removed in the PR
- code line that already existed in the file...
+ code line2 that remained unchanged in the PR
 
 
 @@ ... @@ def func2():
@@ -28,12 +28,13 @@ __old hunk__
 
 ## src/file2.py
 ...
-'
+======
+
 
 Specific instructions:
-- Try to identify edited/added code components (classes/functions/methods...) that are undocumented. and generate {{ docs_for_language }} for each one.
+- Try to identify edited/added code components (classes/functions/methods...) that are undocumented, and generate {{ docs_for_language }} for each one.
 - If there are documented (any type of {{ language }} documentation) code components in the PR, Don't generate {{ docs_for_language }} for them.
-- Ignore code components that don't appear fully in the '__new hunk__' section. For example. you must see the component header and body,
+- Ignore code components that don't appear fully in the '__new hunk__' section. For example, you must see the component header and body.
 - Make sure the {{ docs_for_language }} starts and ends with standart {{ language }} {{ docs_for_language }} signs.
 - The {{ docs_for_language }} should be in standard format.
 - Provide the exact line number (inclusive) where the {{ docs_for_language }} should be added.
@@ -42,11 +43,12 @@ Specific instructions:
 {%- if extra_instructions %}
 
 Extra instructions from the user:
-'
+======
 {{ extra_instructions }}
-'
+======
 {%- endif %}
 
+
 You must use the following YAML schema to format your answer:
 ```yaml
 Code Documentation:
@@ -99,7 +101,13 @@ Title: '{{ title }}'
 
 Branch: '{{ branch }}'
 
-Description: '{{description}}'
+{%- if description %}
+
+Description:
+======
+{{ description|trim }}
+======
+{%- endif %}
 
 {%- if language %}
 
@@ -108,9 +116,10 @@ Main PR language: '{{language}}'
 
 
 The PR Diff:
-```
-{{- diff|trim }}
-```
+======
+{{ diff|trim }}
+======
+
 
 Response (should be a valid YAML, and nothing else):
 ```yaml

pr_agent/settings/pr_code_suggestions_prompts.toml

@@ -2,21 +2,20 @@
 system="""You are PR-Reviewer, a language model that specializes in suggesting code improvements for a Pull Request (PR).
 Your task is to provide meaningful and actionable code suggestions, to improve the new code presented in a PR diff (lines starting with '+').
 
-Example for a PR Diff input:
-'
+Example for the PR Diff format:
+======
 ## src/file1.py
 
-@@ -12,3 +12,5 @@ def func1():
+@@ -12,3 +12,4 @@ def func1():
 __new hunk__
-12  code line that already existed in the file...
-13  code line that already existed in the file....
+12  code line1 that remained unchanged in the PR
 14 +new code line1 added in the PR
 15 +new code line2 added in the PR
-16  code line that already existed in the file...
+16  code line2 that remained unchanged in the PR
 __old hunk__
- code line that already existed in the file...
+ code line1 that remained unchanged in the PR
 -code line that was removed in the PR
- code line that already existed in the file...
+ code line2 that remained unchanged in the PR
 
 
 @@ ... @@ def func2():
@@ -28,28 +27,29 @@ __old hunk__
 
 ## src/file2.py
 ...
-'
+======
+
 
 Specific instructions:
 - Provide up to {{ num_code_suggestions }} code suggestions. Try to provide diverse and insightful suggestions.
-- Prioritize suggestions that address major problems, issues and bugs in the code.
-  As a second priority, suggestions should focus on best practices, code readability, maintainability, enhancments, performance, and other aspects.
+- Prioritize suggestions that address major problems, issues and bugs in the code. As a second priority, suggestions should focus on best practices, code readability, maintainability, enhancments, performance, and other aspects.
 - Don't suggest to add docstring, type hints, or comments.
 - Suggestions should refer only to code from the '__new hunk__' sections, and focus on new lines of code (lines starting with '+').
 - Avoid making suggestions that have already been implemented in the PR code. For example, if you want to add logs, or change a variable to const, or anything else, make sure it isn't already in the '__new hunk__' code.
 - For each suggestion, make sure to take into consideration also the context, meaning the lines before and after the relevant code.
-- Provide the exact line numbers range (inclusive) for each issue.
+- Provide the exact line numbers range (inclusive) for each suggestion.
 - Assume there is additional relevant code, that is not included in the diff.
 
 
 {%- if extra_instructions %}
 
 Extra instructions from the user:
-'
+======
 {{ extra_instructions }}
-'
+======
 {%- endif %}
 
+
 You must use the following YAML schema to format your answer:
 ```yaml
 Code suggestions:
@@ -116,7 +116,13 @@ Title: '{{title}}'
 
 Branch: '{{branch}}'
 
-Description: '{{description}}'
+{%- if description %}
+
+Description:
+======
+{{ description|trim }}
+======
+{%- endif %}
 
 {%- if language %}
 
@@ -125,9 +131,10 @@ Main PR language: '{{ language }}'
 
 
 The PR Diff:
-```
-{{- diff|trim }}
-```
+======
+{{ diff|trim }}
+======
+
 
 Response (should be a valid YAML, and nothing else):
 ```yaml

pr_agent/settings/pr_custom_labels.toml

@@ -1,5 +1,5 @@
 [pr_custom_labels_prompt]
-system="""You are PR-Reviewer, a language model designed to review a git Pull Request (PR).
+system="""You are PR-Reviewer, a language model designed to review a Git Pull Request (PR).
 Your task is to provide labels that describe the PR content.
 {%- if enable_custom_labels %}
 Thoroughly read the labels name and the provided description, and decide whether the label is relevant to the PR.
@@ -8,14 +8,14 @@ Thoroughly read the labels name and the provided description, and decide whether
 {%- if extra_instructions %}
 
 Extra instructions from the user:
-'
+======
 {{ extra_instructions }}
-'
+======
 {% endif %}
 
 
 The output must be a YAML object equivalent to type $Labels, according to the following Pydantic definitions:
-'
+======
 {%- if enable_custom_labels %}
 
 {{ custom_labels_class }}
@@ -24,7 +24,6 @@ The output must be a YAML object equivalent to type $Labels, according to the fo
 class Label(str, Enum):
     bug_fix = "Bug fix"
     tests = "Tests"
-    refactoring = "Refactoring"
     enhancement = "Enhancement"
     documentation = "Documentation"
     other = "Other"
@@ -32,10 +31,11 @@ class Label(str, Enum):
 
 class Labels(BaseModel):
     labels: List[Label] = Field(min_items=0, description="custom labels that describe the PR. Return the label value, not the name.")
-'
+======
 
 
 Example output:
+
 ```yaml
 labels:
 - ...
@@ -51,7 +51,13 @@ Previous title: '{{title}}'
 
 Branch: '{{ branch }}'
 
-Description: '{{ description }}'
+{%- if description %}
+
+Description:
+======
+{{ description|trim }}
+======
+{%- endif %}
 
 {%- if language %}
 
@@ -59,19 +65,22 @@ Main PR language: '{{ language }}'
 {%- endif %}
 {%- if commit_messages_str %}
 
+
 Commit messages:
-'
-{{ commit_messages_str }}
-'
+======
+{{ commit_messages_str|trim }}
+======
 {%- endif %}
 
 
 The PR Git Diff:
-```
-{{diff}}
-```
+======
+{{ diff|trim }}
+======
+
 Note that lines in the diff body are prefixed with a symbol that represents the type of change: '-' for deletions, '+' for additions, and ' ' (a space) for unchanged lines.
 
+
 Response (should be a valid YAML, and nothing else):
 ```yaml
 """

pr_agent/settings/pr_description_prompts.toml

@@ -1,25 +1,25 @@
 [pr_description_prompt]
-system="""You are PR-Reviewer, a language model designed to review a git Pull Request (PR).
-Your task is to provide a full description for the PR content.
-- Make sure to focus on the new PR code (lines starting with '+').
+system="""You are PR-Reviewer, a language model designed to review a Git Pull Request (PR).
+Your task is to provide a full description for the PR content - title, type, description, and main files walkthrough.
+- Focus on the new PR code (lines starting with '+').
 - Keep in mind that the 'Previous title', 'Previous description' and 'Commit messages' sections may be partial, simplistic, non-informative or out of date. Hence, compare them to the PR diff code, and use them only as a reference.
-- Prioritize the most significant PR changes first, followed by the minor ones.
-- If needed, each YAML output should be in block scalar format ('|-')
+- The generated title and description should prioritize the most significant changes.
+- If needed, each YAML output should be in block scalar indicator ('|-')
+
 {%- if extra_instructions %}
 
 Extra instructions from the user:
-'
+=====
 {{ extra_instructions }}
-'
+=====
 {% endif %}
 
 
 The output must be a YAML object equivalent to type $PRDescription, according to the following Pydantic definitions:
-'
+=====
 class PRType(str, Enum):
     bug_fix = "Bug fix"
     tests = "Tests"
-    refactoring = "Refactoring"
     enhancement = "Enhancement"
     documentation = "Documentation"
     other = "Other"
@@ -30,22 +30,37 @@ class PRType(str, Enum):
 
 {%- endif %}
 
+{%- if enable_file_walkthrough %}
 class FileWalkthrough(BaseModel):
     filename: str = Field(description="the relevant file full path")
-    changes_in_file: str = Field(description="minimal and concise description of the changes in the relevant file")
+    changes_in_file: str = Field(description="minimal and concise summary of the changes in the relevant file")
+{%- endif %}
+
+{%- if enable_semantic_files_types %}
+Class FileDescription(BaseModel):
+    filename: str = Field(description="the relevant file full path")
+    changes_summary: str = Field(description="minimal and concise summary of the changes in the relevant file")
+    label: str = Field(description="a single semantic label that represents a type of code changes that occurred in the File. Possible values (partial list): 'bug fix', 'tests', 'enhancement', 'documentation', 'error handling', 'configuration changes', 'dependencies', 'formatting', 'miscellaneous', ...")
+{%- endif %}
 
 Class PRDescription(BaseModel):
     title: str = Field(description="an informative title for the PR, describing its main theme")
-    type: List[PRType] = Field(description="one or more types that describe the PR type. . Return the label value, not the name.")
-    description: str = Field(description="an informative and concise description of the PR. {%- if use_bullet_points %} Use bullet points. {% endif %}")
+    type: List[PRType] = Field(description="one or more types that describe the PR type. Return the label value, not the name.")
+    description: str = Field(description="an informative and concise description of the PR. {%- if use_bullet_points %} Use bullet points.{% endif %}")
 {%- if enable_custom_labels %}
     labels: List[Label] = Field(min_items=0, description="custom labels that describe the PR. Return the label value, not the name.")
 {%- endif %}
+{%- if enable_file_walkthrough %}
     main_files_walkthrough: List[FileWalkthrough] = Field(max_items=10)
-'
+{%- endif %}
+{%- if enable_semantic_files_types %}
+    pr_files[List[FileDescription]] = Field(max_items=15")
+{%- endif %}
+=====
 
 
 Example output:
+
 ```yaml
 title: |-
   ...
@@ -59,9 +74,21 @@ labels:
 {%- endif %}
 description: |-
   ...
+{%- if enable_file_walkthrough %}
 main_files_walkthrough:
 - ...
 - ...
+{%- endif %}
+{%- if enable_semantic_files_types %}
+pr_files:
+- filename: |
+    ...
+  changes_summary: |
+    ...
+  label: |
+    ...
+...
+{%- endif %}
 ```
 
 Answer should be a valid YAML, and nothing else. Each YAML output MUST be after a newline, with proper indent, and block scalar indicator ('|-')
@@ -74,9 +101,9 @@ Previous title: '{{title}}'
 {%- if description %}
 
 Previous description:
-'
-{{ description }}
-'
+=====
+{{ description|trim }}
+=====
 {%- endif %}
 
 Branch: '{{branch}}'
@@ -87,20 +114,20 @@ Main PR language: '{{ language }}'
 {%- if commit_messages_str %}
 
 Commit messages:
-'
-{{ commit_messages_str }}
-'
+=====
+{{ commit_messages_str|trim }}
+=====
 {%- endif %}
 
 
-The PR Git Diff:
-```
-{{diff}}
-```
-
+The PR Diff:
+=====
+{{ diff|trim }}
+=====
 
 Note that lines in the diff body are prefixed with a symbol that represents the type of change: '-' for deletions, '+' for additions, and ' ' (a space) for unchanged lines.
 
+
 Response (should be a valid YAML, and nothing else):
 ```yaml
 """

pr_agent/settings/pr_information_from_user_prompts.toml

@@ -1,5 +1,5 @@
 [pr_information_from_user_prompt]
-system="""You are PR-Reviewer, a language model designed to review a git Pull Request (PR).
+system="""You are PR-Reviewer, a language model designed to review a Git Pull Request (PR).
 Given the PR Info and the PR Git Diff, generate 3 short questions about the PR code for the PR author.
 The goal of the questions is to help the language model understand the PR better, so the questions should be insightful, informative, non-trivial, and relevant to the PR.
 You should prefer asking yes\\no questions, or multiple choice questions. Also add at least one open-ended question, but make sure they are not too difficult, and can be answered in a sentence or two.
@@ -19,7 +19,13 @@ Title: '{{title}}'
 
 Branch: '{{branch}}'
 
-Description: '{{description}}'
+{%- if description %}
+
+Description:
+======
+{{ description|trim }}
+======
+{%- endif %}
 
 {%- if language %}
 
@@ -27,17 +33,19 @@ Main PR language: '{{ language }}'
 {%- endif %}
 {%- if commit_messages_str %}
 
+
 Commit messages:
-'
-{{commit_messages_str}}
-'
+======
+{{ commit_messages_str|trim }}
+======
 {%- endif %}
 
 
 The PR Git Diff:
-```
-{{diff}}
-```
+======
+{{ diff|trim }}
+======
+
 Note that lines in the diff body are prefixed with a symbol that represents the type of change: '-' for deletions, '+' for additions, and ' ' (a space) for unchanged lines
 
 

pr_agent/settings/pr_questions_prompts.toml

@@ -1,9 +1,9 @@
 [pr_questions_prompt]
-system="""You are PR-Reviewer, a language model designed to review a git Pull Request (PR).
-Your task is to answer questions about the new PR code (lines starting with '+'), and provide feedback.
+system="""You are PR-Reviewer, a language model designed to review a Git Pull Request (PR).
+
+Your goal is to answer questions\\tasks about the new PR code (lines starting with '+'), and provide feedback.
 Be informative, constructive, and give examples. Try to be as specific as possible.
-Don't avoid answering the questions. You must answer the questions, as best as you can, without adding unrelated content.
-Make sure not to repeat modifications already implemented in the new PR code (the '+' lines).
+Don't avoid answering the questions. You must answer the questions, as best as you can, without adding any unrelated content.
 """
 
 user="""PR Info:
@@ -12,32 +12,31 @@ Title: '{{title}}'
 
 Branch: '{{branch}}'
 
-Description: '{{description}}'
+{%- if description %}
+
+Description:
+======
+{{ description|trim }}
+======
+{%- endif %}
 
 {%- if language %}
 
 Main PR language: '{{ language }}'
 {%- endif %}
-{%- if commit_messages_str %}
-
-Commit messages:
-'
-{{ commit_messages_str }}
-'
-{%- endif %}
 
 
 The PR Git Diff:
-```
-{{diff}}
-```
+======
+{{ diff|trim }}
+======
 Note that lines in the diff body are prefixed with a symbol that represents the type of change: '-' for deletions, '+' for additions, and ' ' (a space) for unchanged lines
 
 
 The PR Questions:
-```
-{{ questions }}
-```
+======
+{{ questions|trim }}
+======
 
-Response:
+Response to the PR Questions:
 """

pr_agent/settings/pr_reviewer_prompts.toml

@@ -1,19 +1,19 @@
 [pr_review_prompt]
-system="""You are PR-Reviewer, a language model designed to review a git Pull Request (PR).
+system="""You are PR-Reviewer, a language model designed to review a Git Pull Request (PR).
 Your task is to provide constructive and concise feedback for the PR, and also provide meaningful code suggestions.
 The review should focus on new code added in the PR diff (lines starting with '+')
 
-Example PR Diff input:
-'
+Example PR Diff:
+======
 ## src/file1.py
 
 @@ -12,5 +12,5 @@ def func1():
-code line that already existed in the file...
-code line that already existed in the file....
+code line 1 that remained unchanged in the PR
+code line 2 that remained unchanged in the PR
 -code line that was removed in the PR
-+new code line added in the PR
- code line that already existed in the file...
- code line that already existed in the file...
++code line added in the PR
+code line 3 that remained unchanged in the PR
+
 
 @@ ... @@ def func2():
 ...
@@ -21,10 +21,11 @@ code line that already existed in the file....
 
 ## src/file2.py
 ...
-'
+======
 
 {%- if num_code_suggestions > 0 %}
 
+
 Code suggestions guidelines:
 - Provide up to {{ num_code_suggestions }} code suggestions. Try to provide diverse and insightful suggestions.
 - Focus on important suggestions like fixing code problems, issues and bugs. As a second priority, provide suggestions for meaningful code improvements, like performance, vulnerability, modularity, and best practices.
@@ -36,11 +37,12 @@ Code suggestions guidelines:
 {%- if extra_instructions %}
 
 Extra instructions from the user:
-'
+======
 {{ extra_instructions }}
-'
+======
 {% endif %}
 
+
 You must use the following YAML schema to format your answer:
 ```yaml
 PR Analysis:
@@ -55,7 +57,6 @@ PR Analysis:
     enum:
       - Bug fix
       - Tests
-      - Refactoring
       - Enhancement
       - Documentation
       - Other
@@ -188,9 +189,9 @@ Branch: '{{branch}}'
 {%- if description %}
 
 Description:
-'
-{{description}}
-'
+======
+{{ description|trim }}
+======
 {%- endif %}
 
 {%- if language %}
@@ -200,28 +201,29 @@ Main PR language: '{{ language }}'
 {%- if commit_messages_str %}
 
 Commit messages:
-'
+======
 {{commit_messages_str}}
-'
+======
 {%- endif %}
 
 {%- if question_str %}
-######
+=====
 Here are questions to better understand the PR. Use the answers to provide better feedback.
 
-{{question_str|trim}}
+{{ question_str|trim }}
 
 User answers:
 '
-{{answer_str|trim}}
+{{ answer_str|trim }}
 '
-######
+=====
 {%- endif %}
 
-The PR Git Diff:
-```
-{{diff}}
-```
+
+The PR Diff:
+======
+{{ diff|trim }}
+======
 
 
 Response (should be a valid YAML, and nothing else):

pr_agent/settings/pr_sort_code_suggestions_prompts.toml

@@ -2,10 +2,10 @@
 system="""
 """
 
-user="""You are given a list of code suggestions to improve a git Pull Request (PR):
-'
+user="""You are given a list of code suggestions to improve a Git Pull Request (PR):
+======
 {{ suggestion_str|trim }}
-'
+======
 
 Your task is to sort the code suggestions by their order of importance, and return a list with sorting order.
 The sorting order is a list of pairs, where each pair contains the index of the suggestion in the original list.

pr_agent/settings/pr_update_changelog_prompts.toml

@@ -1,5 +1,5 @@
 [pr_update_changelog_prompt]
-system="""You are a language model called CodiumAI-PR-Changlog-summarizer.
+system="""You are a language model called PR-Changelog-Updater.
 Your task is to update the CHANGELOG.md file of the project, to shortly summarize important changes introduced in this PR (the '+' lines).
 - The output should match the existing CHANGELOG.md format, style and conventions, so it will look like a natural part of the file. For example, if previous changes were summarized in a single line, you should do the same.
 - Don't repeat previous changes. Generate only new content, that is not already in the CHANGELOG.md file.
@@ -8,9 +8,9 @@ Your task is to update the CHANGELOG.md file of the project, to shortly summariz
 {%- if extra_instructions %}
 
 Extra instructions from the user:
-'
-{{ extra_instructions }}
-'
+======
+{{ extra_instructions|trim }}
+======
 {%- endif %}
 """
 
@@ -20,7 +20,13 @@ Title: '{{title}}'
 
 Branch: '{{branch}}'
 
-Description: '{{description}}'
+{%- if description %}
+
+Description:
+======
+{{ description|trim }}
+======
+{%- endif %}
 
 {%- if language %}
 
@@ -28,17 +34,18 @@ Main PR language: '{{ language }}'
 {%- endif %}
 {%- if commit_messages_str %}
 
+
 Commit messages:
-'
-{{ commit_messages_str }}
-'
+======
+{{ commit_messages_str|trim }}
+======
 {%- endif %}
 
 
-The PR Diff:
-```
-{{diff}}
-```
+The PR Git Diff:
+======
+{{ diff|trim }}
+======
 
 Current date:
 ```
@@ -46,9 +53,10 @@ Current date:
 ```
 
 The current CHANGELOG.md:
-```
+======
 {{ changelog_file_str }}
-```
+======
+
 
 Response:
 """

pr_agent/tools/pr_code_suggestions.py

@@ -251,22 +251,36 @@ class PRCodeSuggestions:
     def publish_summarizes_suggestions(self, data: Dict):
         try:
             data_markdown = "## PR Code Suggestions\n\n"
+
+            language_extension_map_org = get_settings().language_extension_map_org
+            extension_to_language = {}
+            for language, extensions in language_extension_map_org.items():
+                for ext in extensions:
+                    extension_to_language[ext] = language
+
             for s in data['Code suggestions']:
-                code_snippet_link = self.git_provider.get_line_link(s['relevant file'], s['relevant lines start'],
-                                                                    s['relevant lines end'])
-                data_markdown += f"\n💡 Suggestion:\n\n**{s['suggestion content']}**\n\n"
-                if code_snippet_link:
-                    data_markdown += f" File: [{s['relevant file']} ({s['relevant lines start']}-{s['relevant lines end']})]({code_snippet_link})\n\n"
-                else:
-                    data_markdown += f"File: {s['relevant file']} ({s['relevant lines start']}-{s['relevant lines end']})\n\n"
-                if self.git_provider.is_supported("gfm_markdown"):
-                    data_markdown += "<details> <summary> Example code:</summary>\n\n"
-                    data_markdown += f"___\n\n"
-                data_markdown += f"Existing code:\n```{self.main_language}\n{s['existing code']}\n```\n"
-                data_markdown += f"Improved code:\n```{self.main_language}\n{s['improved code']}\n```\n"
-                if self.git_provider.is_supported("gfm_markdown"):
-                    data_markdown += "</details>\n"
-                data_markdown += "\n___\n\n"
+                try:
+                    extension_s = s['relevant file'].rsplit('.')[-1]
+                    code_snippet_link = self.git_provider.get_line_link(s['relevant file'], s['relevant lines start'],
+                                                                        s['relevant lines end'])
+                    data_markdown += f"\n💡 Suggestion:\n\n**{s['suggestion content']}**\n\n"
+                    if code_snippet_link:
+                        data_markdown += f" File: [{s['relevant file']} ({s['relevant lines start']}-{s['relevant lines end']})]({code_snippet_link})\n\n"
+                    else:
+                        data_markdown += f"File: {s['relevant file']} ({s['relevant lines start']}-{s['relevant lines end']})\n\n"
+                    if self.git_provider.is_supported("gfm_markdown"):
+                        data_markdown += "<details> <summary> Example code:</summary>\n\n"
+                        data_markdown += f"___\n\n"
+                    language_name = "python"
+                    if extension_s and (extension_s in extension_to_language):
+                        language_name = extension_to_language[extension_s]
+                    data_markdown += f"Existing code:\n```{language_name}\n{s['existing code']}\n```\n"
+                    data_markdown += f"Improved code:\n```{language_name}\n{s['improved code']}\n```\n"
+                    if self.git_provider.is_supported("gfm_markdown"):
+                        data_markdown += "</details>\n"
+                    data_markdown += "\n___\n\n"
+                except Exception as e:
+                    get_logger().error(f"Could not parse suggestion: {s}, error: {e}")
             self.git_provider.publish_comment(data_markdown)
         except Exception as e:
             get_logger().info(f"Failed to publish summarized code suggestions, error: {e}")

pr_agent/tools/pr_description.py

@@ -30,6 +30,11 @@ class PRDescription:
         )
         self.pr_id = self.git_provider.get_pr_id()
 
+        if get_settings().pr_description.enable_semantic_files_types and not self.git_provider.is_supported(
+                "gfm_markdown"):
+            get_logger().debug(f"Disabling semantic files types for {self.pr_id}")
+            get_settings().pr_description.enable_semantic_files_types = False
+
         # Initialize the AI handler
         self.ai_handler = AiHandler()
     
@@ -45,6 +50,8 @@ class PRDescription:
             "commit_messages_str": self.git_provider.get_commit_messages(),
             "enable_custom_labels": get_settings().config.enable_custom_labels,
             "custom_labels_class": "",  # will be filled if necessary in 'set_custom_labels' function
+            "enable_file_walkthrough": get_settings().pr_description.enable_file_walkthrough,
+            "enable_semantic_files_types": get_settings().pr_description.enable_semantic_files_types,
         }
 
         self.user_description = self.git_provider.get_user_description()
@@ -79,6 +86,9 @@ class PRDescription:
             else:
                 return None
 
+            if get_settings().pr_description.enable_semantic_files_types:
+                self._prepare_file_labels()
+
             pr_labels = []
             if get_settings().pr_description.publish_labels:
                 pr_labels = self._prepare_labels()
@@ -98,8 +108,14 @@ class PRDescription:
                     if get_settings().pr_description.publish_labels and self.git_provider.is_supported("get_labels"):
                         current_labels = self.git_provider.get_labels()
                         user_labels = get_user_labels(current_labels)
-
                         self.git_provider.publish_labels(pr_labels + user_labels)
+
+                    if (get_settings().pr_description.final_update_message and
+                            hasattr(self.git_provider, 'pr_url') and self.git_provider.pr_url):
+                        latest_commit_url = self.git_provider.get_latest_commit_url()
+                        if latest_commit_url:
+                            self.git_provider.publish_comment(
+                                f"**[PR Description]({self.git_provider.pr_url})** updated to latest commit ({latest_commit_url})")
                 self.git_provider.remove_initial_comment()
         except Exception as e:
             get_logger().error(f"Error generating PR description {self.pr_id}: {e}")
@@ -251,9 +267,13 @@ class PRDescription:
         # except for the items containing the word 'walkthrough'
         pr_body = ""
         for idx, (key, value) in enumerate(self.data.items()):
-            pr_body += f"## {key}:\n"
+            if key == 'pr_files':
+                value = self.file_label_dict
+                key_publish = "PR changes walkthrough"
+            else:
+                key_publish = key.rstrip(':').replace("_", " ").capitalize()
+            pr_body += f"## {key_publish}\n"
             if 'walkthrough' in key.lower():
-                # for filename, description in value.items():
                 if self.git_provider.is_supported("gfm_markdown"):
                     pr_body += "<details> <summary>files:</summary>\n\n"
                 for file in value:
@@ -261,10 +281,12 @@ class PRDescription:
                     description = file['changes_in_file']
                     pr_body += f'- `{filename}`: {description}\n'
                 if self.git_provider.is_supported("gfm_markdown"):
-                    pr_body +="</details>\n"
+                    pr_body += "</details>\n"
+            elif 'pr_files' in key.lower():
+                pr_body = self.process_pr_files_prediction(pr_body, value)
             else:
                 # if the value is a list, join its items by comma
-                if type(value) == list:
+                if isinstance(value, list):
                     value = ', '.join(v for v in value)
                 pr_body += f"{value}\n"
             if idx < len(self.data) - 1:
@@ -273,4 +295,103 @@ class PRDescription:
         if get_settings().config.verbosity_level >= 2:
             get_logger().info(f"title:\n{title}\n{pr_body}")
 
-        return title, pr_body
+        return title, pr_body
+
+    def _prepare_file_labels(self):
+        self.file_label_dict = {}
+        for file in self.data['pr_files']:
+            try:
+                filename = file['filename'].replace("'", "`").replace('"', '`')
+                changes_summary = file['changes_summary']
+                label = file['label']
+                if label not in self.file_label_dict:
+                    self.file_label_dict[label] = []
+                self.file_label_dict[label].append((filename, changes_summary))
+            except Exception as e:
+                get_logger().error(f"Error preparing file label dict {self.pr_id}: {e}")
+                pass
+
+    def process_pr_files_prediction(self, pr_body, value):
+        if not self.git_provider.is_supported("gfm_markdown"):
+            get_logger().info(f"Disabling semantic files types for {self.pr_id} since gfm_markdown is not supported")
+            return pr_body
+
+        try:
+            pr_body += "<table>"
+            header = f"Relevant files"
+            delta = 65
+            header += "&nbsp; " * delta
+            pr_body += f"""<thead><tr><th></th><th>{header}</th></tr></thead>"""
+            pr_body += """<tbody>"""
+            for semantic_label in value.keys():
+                s_label = semantic_label.strip("'").strip('"')
+                pr_body += f"""<tr><td><strong>{s_label.capitalize()}</strong></td>"""
+                list_tuples = value[semantic_label]
+                pr_body += f"""<td><details><summary>{len(list_tuples)} files</summary><table>"""
+                for filename, file_change_description in list_tuples:
+                    filename = filename.replace("'", "`")
+                    filename_publish = filename.split("/")[-1]
+                    filename_publish = f"{filename_publish}"
+                    if len(filename_publish) < (delta - 5):
+                        filename_publish += "&nbsp; " * ((delta - 5) - len(filename_publish))
+                    diff_plus_minus = ""
+                    diff_files = self.git_provider.diff_files
+                    for f in diff_files:
+                        if f.filename.lower() == filename.lower():
+                            num_plus_lines = f.num_plus_lines
+                            num_minus_lines = f.num_minus_lines
+                            diff_plus_minus += f"+{num_plus_lines}/-{num_minus_lines}"
+                            break
+
+                    # try to add line numbers link to code suggestions
+                    link = ""
+                    if hasattr(self.git_provider, 'get_line_link'):
+                        filename = filename.strip()
+                        link = self.git_provider.get_line_link(filename, relevant_line_start=-1)
+
+                    file_change_description = self._insert_br_after_x_chars(file_change_description, x=(delta - 5))
+                    pr_body += f"""
+<tr>
+  <td>
+    <details>
+      <summary><strong>{filename_publish}</strong></summary>
+      <ul>
+        {filename}<br><br>
+        <strong>{file_change_description}</strong>
+      </ul>
+    </details>
+  </td>
+  <td><a href="{link}"> {diff_plus_minus}</a></td>
+
+</tr>                    
+"""
+                pr_body += """</table></details></td></tr>"""
+            pr_body += """</tr></tbody></table>"""
+
+        except Exception as e:
+            get_logger().error(f"Error processing pr files to markdown {self.pr_id}: {e}")
+            pass
+        return pr_body
+
+    def _insert_br_after_x_chars(self, text, x=70):
+        """
+        Insert <br> into a string after a word that increases its length above x characters.
+        """
+        if len(text) < x:
+            return text
+
+        words = text.split(' ')
+        new_text = ""
+        current_length = 0
+
+        for word in words:
+            # Check if adding this word exceeds x characters
+            if current_length + len(word) > x:
+                new_text += "<br>"  # Insert line break
+                current_length = 0  # Reset counter
+
+            # Add the word to the new text
+            new_text += word + " "
+            current_length += len(word) + 1  # Add 1 for the space
+
+        return new_text.strip()  # Remove trailing space

requirements.txt

@@ -1,25 +1,25 @@
-dynaconf==3.1.12
-fastapi==0.99.0
-PyGithub==1.59.*
-retry==0.9.2
-openai==0.27.8
-Jinja2==3.1.2
-tiktoken==0.4.0
-uvicorn==0.22.0
-python-gitlab==3.15.0
-pytest==7.4.0
-aiohttp==3.8.4
+aiohttp==3.9.1
 atlassian-python-api==3.39.0
-GitPython==3.1.32
-PyYAML==6.0
-starlette-context==0.3.6
-litellm==0.12.5
-boto3==1.28.25
-google-cloud-storage==2.10.0
-ujson==5.8.0
 azure-devops==7.1.0b3
+boto3==1.33.6
+dynaconf==3.2.4
+fastapi==0.99.0
+GitPython==3.1.32
+google-cloud-aiplatform==1.35.0
+google-cloud-storage==2.10.0
+Jinja2==3.1.2
+litellm==0.12.5
+loguru==0.7.2
 msrest==0.7.1
+openai==0.27.8
 pinecone-client
 pinecone-datasets @ git+https://github.com/mrT23/pinecone-datasets.git@main
-loguru==0.7.2
-google-cloud-aiplatform==1.35.0
+pytest==7.4.0
+PyGithub==1.59.*
+PyYAML==6.0.1
+python-gitlab==3.15.0
+retry==0.9.2
+starlette-context==0.3.6
+tiktoken==0.5.2
+ujson==5.8.0
+uvicorn==0.22.0