dockerhub

INSTALL.md Fixed typo in the hyperlink

.github/workflows/pr-agent-review.yaml

@@ -24,4 +24,7 @@ jobs:
           OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
           OPENAI_ORG: ${{ secrets.OPENAI_ORG }} # optional
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          PINECONE.API_KEY: ${{ secrets.PINECONE_API_KEY }}
+          PINECONE.ENVIRONMENT: ${{ secrets.PINECONE_ENVIRONMENT }}
+
 

CONFIGURATION.md

@@ -1,57 +0,0 @@
-## Configuration
-
-The different tools and sub-tools used by CodiumAI PR-Agent are adjustable via the **[configuration file](pr_agent/settings/configuration.toml)**
-
-### Working from CLI
-When running from source (CLI), your local configuration file will be initially used.
-
-Example for invoking the 'review' tools via the CLI: 
-
-```
-python cli.py --pr-url=<pr_url>  review
-```
-In addition to general configurations, the 'review' tool will use parameters from the `[pr_reviewer]` section (every tool has a dedicated section in the configuration file).
-
-Note that you can print results locally, without publishing them, by setting in `configuration.toml`:
-
-```
-[config]
-publish_output=true
-verbosity_level=2
-```
-This is useful for debugging or experimenting with the different tools.
-
-### Working from pre-built repo (GitHub Action/GitHub App/Docker)
-When running PR-Agent from a pre-built repo, the default configuration file will be loaded.
-
-To edit the configuration, you have two options: 
-1. Place a local configuration file in the root of your local repo. The local file will be used instead of the default one.
-2. For online usage, just add `--config_path=<value>` to you command, to edit a specific configuration value.
-For example if you want to edit `pr_reviewer` configurations, you can run:
-```
-/review --pr_reviewer.extra_instructions="..." --pr_reviewer.require_score_review=false ...
-```
-
-Any configuration value in `configuration.toml` file can be similarly edited.
-
-### General configuration parameters
-
-#### Changing a model
-See [here](pr_agent/algo/__init__.py) for the list of available models.
-
-To use Llama2 model, for example, set:
-```
-[config]
-model = "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1"
-[replicate]
-key = ...
-```
-(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
-
-Also review the [AiHandler](pr_agent/algo/ai_handler.py) file for instruction how to set keys for other models.
-
-#### Extra instructions
-All PR-Agent tools have a parameter called `extra_instructions`, that enables to add free-text extra instructions. Example usage:
-```
-/update_changelog --pr_update_changelog.extra_instructions="Make sure to update also the version ..."
-```

Dockerfile.bitbucket_pipeline

@@ -0,0 +1,18 @@
+FROM python:3.10 as base
+
+ENV OPENAI_API_KEY=${OPENAI_API_KEY} \
+    BITBUCKET_BEARER_TOKEN=${BITBUCKET_BEARER_TOKEN} \
+    BITBUCKET_PR_ID=${BITBUCKET_PR_ID} \
+    BITBUCKET_REPO_SLUG=${BITBUCKET_REPO_SLUG} \
+    BITBUCKET_WORKSPACE=${BITBUCKET_WORKSPACE}
+
+
+WORKDIR /app
+ADD pyproject.toml .
+ADD requirements.txt .
+RUN pip install . && rm pyproject.toml requirements.txt
+ENV PYTHONPATH=/app
+ADD pr_agent pr_agent
+ADD bitbucket_pipeline/entrypoint.sh /
+RUN chmod +x /entrypoint.sh
+ENTRYPOINT ["/entrypoint.sh"]

Dockerfile.github_action

@@ -2,7 +2,8 @@ FROM python:3.10 as base
 
 WORKDIR /app
 ADD pyproject.toml .
-RUN pip install . && rm pyproject.toml
+ADD requirements.txt .
+RUN pip install . && rm pyproject.toml requirements.txt
 ENV PYTHONPATH=/app
 ADD pr_agent pr_agent
 ADD github_action/entrypoint.sh /

Dockerfile.github_action_dockerhub

@@ -1 +1 @@
-FROM codiumai/pr-agent:github_action
+FROM codiumai/pr-agent:0.8-github_action

INSTALL.md

@@ -1,17 +1,39 @@
 
 ## Installation
 
+To get started with PR-Agent quickly, you first need to acquire two tokens:
+
+1. An OpenAI key from [here](https://platform.openai.com/), with access to GPT-4.
+2. A GitHub personal access token (classic) with the repo scope.
+
+There are several ways to use PR-Agent:
+
+- [Method 1: Use Docker image (no installation required)](INSTALL.md#method-1-use-docker-image-no-installation-required)
+- [Method 2: Run from source](INSTALL.md#method-2-run-from-source)
+- [Method 3: Run as a GitHub Action](INSTALL.md#method-3-run-as-a-github-action)
+- [Method 4: Run as a polling server](INSTALL.md#method-4-run-as-a-polling-server)
+- [Method 5: Run as a GitHub App](INSTALL.md#method-5-run-as-a-github-app)
+- [Method 6: Deploy as a Lambda Function](INSTALL.md#method-6---deploy-as-a-lambda-function)
+- [Method 7: AWS CodeCommit](INSTALL.md#method-7---aws-codecommit-setup)
+- [Method 8: Run a GitLab webhook server](INSTALL.md#method-8---run-a-gitlab-webhook-server)
+- [Method 9: Run as a Bitbucket Pipeline](INSTALL.md#method-9-run-as-a-bitbucket-pipeline)
 ---
 
-#### Method 1: Use Docker image (no installation required)
+### Method 1: Use Docker image (no installation required)
 
 To request a review for a PR, or ask a question about a PR, you can run directly from the Docker image. Here's how:
 
 1. To request a review for a PR, run the following command:
 
+For GitHub:
 ```
 docker run --rm -it -e OPENAI.KEY=<your key> -e GITHUB.USER_TOKEN=<your token> codiumai/pr-agent --pr_url <pr_url> review
 ```
+For GitLab:
+```
+docker run --rm -it -e OPENAI.KEY=<your key> -e CONFIG.GIT_PROVIDER=gitlab -e GITLAB.PERSONAL_ACCESS_TOKEN=<your token> codiumai/pr-agent --pr_url <pr_url> review
+```
+For other git providers, update CONFIG.GIT_PROVIDER accordingly, and check the `pr_agent/settings/.secrets_template.toml` file for the environment variables expected names and values.
 
 2. To ask a question about a PR, run the following command:
 
@@ -41,77 +63,7 @@ Possible questions you can ask include:
 
 ---
 
-#### Method 2: Run as a GitHub Action
-
-You can use our pre-built Github Action Docker image to run PR-Agent as a Github Action. 
-
-1. Add the following file to your repository under `.github/workflows/pr_agent.yml`:
-
-```yaml
-on:
-  pull_request:
-  issue_comment:
-jobs:
-  pr_agent_job:
-    runs-on: ubuntu-latest
-    permissions:
-      issues: write
-      pull-requests: write
-      contents: write
-    name: Run pr agent on every pull request, respond to user comments
-    steps:
-      - name: PR Agent action step
-        id: pragent
-        uses: Codium-ai/pr-agent@main
-        env:
-          OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-```
-** if you want to pin your action to a specific commit for stability reasons
-```yaml
-on:
-  pull_request:
-  issue_comment:
-
-jobs:
-  pr_agent_job:
-    runs-on: ubuntu-latest
-    permissions:
-      issues: write
-      pull-requests: write
-      contents: write
-    name: Run pr agent on every pull request, respond to user comments
-    steps:
-      - name: PR Agent action step
-        id: pragent
-        uses: Codium-ai/pr-agent@<commit_sha>
-        env:
-          OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-```
-2. Add the following secret to your repository under `Settings > Secrets`:
-
-```
-OPENAI_KEY: <your key>
-```
-
-The GITHUB_TOKEN secret is automatically created by GitHub.
-
-3. Merge this change to your main branch. 
-When you open your next PR, you should see a comment from `github-actions` bot with a review of your PR, and instructions on how to use the rest of the tools.
-
-4. You may configure PR-Agent by adding environment variables under the env section corresponding to any configurable property in the [configuration](./CONFIGURATION.md) file. Some examples:
-```yaml
-      env:
-        # ... previous environment values
-        OPENAI.ORG: "<Your organization name under your OpenAI account>"
-        PR_REVIEWER.REQUIRE_TESTS_REVIEW: "false" # Disable tests review
-        PR_CODE_SUGGESTIONS.NUM_CODE_SUGGESTIONS: 6 # Increase number of code suggestions
-```
-
----
-
-#### Method 3: Run from source
+### Method 2: Run from source
 
 1. Clone this repository:
 
@@ -137,23 +89,85 @@ chmod 600 pr_agent/settings/.secrets.toml
 
 ```
 export PYTHONPATH=[$PYTHONPATH:]<PATH to pr_agent folder>
-python pr_agent/cli.py --pr_url <pr_url> review
-python pr_agent/cli.py --pr_url <pr_url> ask <your question>
-python pr_agent/cli.py --pr_url <pr_url> describe
-python pr_agent/cli.py --pr_url <pr_url> improve
+python pr_agent/cli.py --pr_url <pr_url> /review
+python pr_agent/cli.py --pr_url <pr_url> /ask <your question>
+python pr_agent/cli.py --pr_url <pr_url> /describe
+python pr_agent/cli.py --pr_url <pr_url> /improve
 ```
 
-5. **Debugging LLM API Calls**  
-If you're testing your codium/pr-agent server, and need to see if calls were made successfully + the exact call logs, you can use the [LiteLLM Debugger tool](https://docs.litellm.ai/docs/debugging/hosted_debugging). 
-
-You can do this by setting `litellm_debugger=true` in configuration.toml. Your Logs will be viewable in real-time @ `admin.litellm.ai/<your_email>`. Set your email in the `.secrets.toml` under 'user_email'.
-
-<img src="./pics/debugger.png" width="900"/>
-
-
 ---
 
-#### Method 4: Run as a polling server
+### Method 3: Run as a GitHub Action
+
+You can use our pre-built Github Action Docker image to run PR-Agent as a Github Action. 
+
+1. Add the following file to your repository under `.github/workflows/pr_agent.yml`:
+
+```yaml
+on:
+  pull_request:
+  issue_comment:
+jobs:
+  pr_agent_job:
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      pull-requests: write
+      contents: write
+    name: Run pr agent on every pull request, respond to user comments
+    steps:
+      - name: PR Agent action step
+        id: pragent
+        uses: Codium-ai/pr-agent@main
+        env:
+          OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+```
+** if you want to pin your action to a specific release (v0.7 for example) for stability reasons, use:
+```yaml
+on:
+  pull_request:
+  issue_comment:
+
+jobs:
+  pr_agent_job:
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      pull-requests: write
+      contents: write
+    name: Run pr agent on every pull request, respond to user comments
+    steps:
+      - name: PR Agent action step
+        id: pragent
+        uses: Codium-ai/pr-agent@v0.7
+        env:
+          OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+```
+2. Add the following secret to your repository under `Settings > Secrets`:
+
+```
+OPENAI_KEY: <your key>
+```
+
+The GITHUB_TOKEN secret is automatically created by GitHub.
+
+3. Merge this change to your main branch. 
+When you open your next PR, you should see a comment from `github-actions` bot with a review of your PR, and instructions on how to use the rest of the tools.
+
+4. You may configure PR-Agent by adding environment variables under the env section corresponding to any configurable property in the [configuration](pr_agent/settings/configuration.toml) file. Some examples:
+```yaml
+      env:
+        # ... previous environment values
+        OPENAI.ORG: "<Your organization name under your OpenAI account>"
+        PR_REVIEWER.REQUIRE_TESTS_REVIEW: "false" # Disable tests review
+        PR_CODE_SUGGESTIONS.NUM_CODE_SUGGESTIONS: 6 # Increase number of code suggestions
+```
+
+---
+
+### Method 4: Run as a polling server
 Request reviews by tagging your Github user on a PR
 
 Follow steps 1-3 of method 2.
@@ -165,7 +179,7 @@ python pr_agent/servers/github_polling.py
 
 ---
 
-#### Method 5: Run as a GitHub App
+### Method 5: Run as a GitHub App
 Allowing you to automate the review process on your private or public repositories.
 
 1. Create a GitHub App from the [Github Developer Portal](https://docs.github.com/en/developers/apps/creating-a-github-app).
@@ -245,9 +259,12 @@ docker push codiumai/pr-agent:github_app  # Push to your Docker repository
 
 9. Install the app by navigating to the "Install App" tab and selecting your desired repositories.
 
+> **Note:** When running PR-Agent from GitHub App, the default configuration file (configuration.toml) will be loaded.<br>
+> However, you can override the default tool parameters by uploading a local configuration file<br>
+> For more information please check out [CONFIGURATION.md](Usage.md#working-from-github-app-pre-built-repo)
 ---
 
-####  Deploy as a Lambda Function
+### Method 6 - Deploy as a Lambda Function
 
 1. Follow steps 1-5 of [Method 5](#method-5-run-as-a-github-app).
 2. Build a docker image that can be used as a lambda function
@@ -266,7 +283,7 @@ docker push codiumai/pr-agent:github_app  # Push to your Docker repository
 
 ---
 
-#### AWS CodeCommit Setup
+### Method 7 - AWS CodeCommit Setup
 
 Not all features have been added to CodeCommit yet.  As of right now, CodeCommit has been implemented to run the pr-agent CLI on the command line, using AWS credentials stored in environment variables.  (More features will be added in the future.)  The following is a set of instructions to have pr-agent do a review of your CodeCommit pull request from the command line:
 
@@ -281,7 +298,7 @@ Not all features have been added to CodeCommit yet.  As of right now, CodeCommit
     * Option B: Set `PYTHONPATH` and run the CLI in one command, for example:
         * `PYTHONPATH="/PATH/TO/PROJECTS/pr-agent python pr_agent/cli.py [--ARGS]`
 
-#### AWS CodeCommit IAM Role Example
+##### AWS CodeCommit IAM Role Example
 
 Example IAM permissions to that user to allow access to CodeCommit:
 
@@ -303,7 +320,9 @@ Example IAM permissions to that user to allow access to CodeCommit:
                 "codecommit:Get*",
                 "codecommit:List*",
                 "codecommit:PostComment*",
-                "codecommit:PutCommentReaction"
+                "codecommit:PutCommentReaction",
+                "codecommit:UpdatePullRequestDescription",
+                "codecommit:UpdatePullRequestTitle"                
             ],
             "Resource": "*"
         }
@@ -311,7 +330,7 @@ Example IAM permissions to that user to allow access to CodeCommit:
 }
 ```
 
-#### AWS CodeCommit Access Key and Secret
+##### AWS CodeCommit Access Key and Secret
 
 Example setting the Access Key and Secret using environment variables
 
@@ -321,7 +340,7 @@ export AWS_SECRET_ACCESS_KEY="XXXXXXXXXXXXXXXX"
 export AWS_DEFAULT_REGION="us-east-1"
 ```
 
-#### AWS CodeCommit CLI Example
+##### AWS CodeCommit CLI Example
 
 After you set up AWS CodeCommit using the instructions above, here is an example CLI run that tells pr-agent to **review** a given pull request.
 (Replace your specific PYTHONPATH and PR URL in the example)
@@ -331,3 +350,84 @@ PYTHONPATH="/PATH/TO/PROJECTS/pr-agent" python pr_agent/cli.py \
   --pr_url https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/MY_REPO_NAME/pull-requests/321 \
   review
 ```
+
+---
+
+### Method 8 - Run a GitLab webhook server
+
+1. From the GitLab workspace or group, create an access token. Enable the "api" scope only.
+2. Generate a random secret for your app, and save it for later. For example, you can use:
+
+```
+WEBHOOK_SECRET=$(python -c "import secrets; print(secrets.token_hex(10))")
+```
+3. Follow the instructions to build the Docker image, setup a secrets file and deploy on your own server from [Method 5](#method-5-run-as-a-github-app) steps 4-7.
+4. In the secrets file, fill in the following:
+    - Your OpenAI key.
+    - In the [gitlab] section, fill in personal_access_token and shared_secret. The access token can be a personal access token, or a group or project access token.
+    - Set deployment_type to 'gitlab' in [configuration.toml](./pr_agent/settings/configuration.toml)
+5. Create a webhook in GitLab. Set the URL to the URL of your app's server. Set the secret token to the generated secret from step 2. 
+In the "Trigger" section, check the ‘comments’ and ‘merge request events’ boxes. 
+6. Test your installation by opening a merge request or commenting or a merge request using one of CodiumAI's commands.
+
+
+
+### Method 9: Run as a Bitbucket Pipeline
+
+
+You can use our pre-build Bitbucket-Pipeline docker image to run as Bitbucket-Pipeline.
+
+1. Add the following file in your repository bitbucket_pipelines.yml
+
+```yaml
+  pipelines:
+    pull-requests:
+      '**':
+        - step:
+            name: PR Agent Pipeline
+            caches:
+              - pip
+            image: python:3.8
+            services:
+              - docker
+            script:
+              - git clone https://github.com/Codium-ai/pr-agent.git
+              - cd pr-agent
+              - docker build -t bitbucket_runner:latest -f Dockerfile.bitbucket_pipeline .
+              - docker run -e OPENAI_API_KEY=$OPENAI_API_KEY -e BITBUCKET_BEARER_TOKEN=$BITBUCKET_BEARER_TOKEN -e BITBUCKET_PR_ID=$BITBUCKET_PR_ID -e BITBUCKET_REPO_SLUG=$BITBUCKET_REPO_SLUG -e BITBUCKET_WORKSPACE=$BITBUCKET_WORKSPACE bitbucket_runner:latest
+```
+
+2. Add the following secret to your repository under Repository settings > Pipelines > Repository variables.
+OPENAI_API_KEY: <your key>
+BITBUCKET_BEARER_TOKEN: <your token>
+
+3. To get BITBUCKET_BEARER_TOKEN follow these steps
+  So here is my step by step tutorial
+  i) Insert your workspace name instead of {workspace_name} and go to the following link in order to create an OAuth consumer.
+
+      https://bitbucket.org/{workspace_name}/workspace/settings/api
+
+      set callback URL to http://localhost:8976 (doesn't need to be a real server there)
+      select permissions: repository -> read
+
+  ii) use consumer's Key as a {client_id} and open the following URL in the browser
+
+      https://bitbucket.org/site/oauth2/authorize?client_id={client_id}&response_type=code
+
+  iii)
+      after you press "Grant access" in the browser it will redirect you to
+
+      http://localhost:8976?code=<CODE>
+
+  iv) use the code from the previous step and consumer's Key as a {client_id}, and consumer's Secret as {client_secret}
+
+      curl -X POST -u "{client_id}:{client_secret}" \
+          https://bitbucket.org/site/oauth2/access_token \
+          -d grant_type=authorization_code \
+          -d code={code} \
+
+
+After completing this steps, you just to place this access token in the repository varibles.
+
+
+=======

README.md

@@ -15,106 +15,128 @@ Making pull requests less painful with an AI agent
 </div>
 <div style="text-align:left;">
 
-CodiumAI `PR-Agent` is an open-source tool aiming to help developers review pull requests faster and more efficiently. It automatically analyzes the pull request and can provide several types of feedback:
+CodiumAI `PR-Agent` is an open-source tool aiming to help developers review pull requests faster and more efficiently. It automatically analyzes the pull request and can provide several types of commands:
 
-**Auto-Description**: Automatically generating PR description - title, type, summary, code walkthrough and PR labels.
+‣ **Auto Description (`/describe`)**: Automatically generating [PR description](https://github.com/Codium-ai/pr-agent/pull/229#issue-1860711415) - title, type, summary, code walkthrough and labels.
 \
-**PR Review**: Adjustable feedback about the PR main theme, type, relevant tests, security issues, focus, score, and various suggestions for the PR content.
+‣ **Auto Review (`/review`)**: [Adjustable feedback](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695022908) about the PR main theme, type, relevant tests, security issues, score, and various suggestions for the PR content.
 \
-**Question Answering**: Answering free-text questions about the PR.
+‣ **Question Answering (`/ask ...`)**: Answering [free-text questions](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021332) about the PR.
 \
-**Code Suggestions**: Committable code suggestions for improving the PR.
+‣ **Code Suggestions (`/improve`)**: [Committable code suggestions](https://github.com/Codium-ai/pr-agent/pull/229#discussion_r1306919276) for improving the PR.
 \
-**Update Changelog**: Automatically updating the CHANGELOG.md file with the PR changes.
+‣ **Update Changelog (`/update_changelog`)**: Automatically updating the CHANGELOG.md file with the [PR changes](https://github.com/Codium-ai/pr-agent/pull/168#discussion_r1282077645).
+\
+‣ **Find similar issue (`/similar_issue`)**: Automatically retrieves and presents [similar issues](https://github.com/Alibaba-MIIL/ASL/issues/107).
 
-<h3>Example results:</h2>
+
+See the [usage guide](./Usage.md) for instructions how to run the different tools from [CLI](./Usage.md#working-from-a-local-repo-cli), or by [online usage](./Usage.md#online-usage), as well as additional details on optional commands and configurations.
+
+[Release notes](./RELEASE_NOTES.md)
+
+<h3>Example results:</h3>
 </div>
-<h4>/describe:</h4>
+<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1687561986">/describe:</a></h4>
 <div align="center">
 <p float="center">
 <img src="https://www.codium.ai/images/describe-2.gif" width="800">
 </p>
 </div>
-<h4>/review:</h4>
+
+<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901">/review:</a></h4>
 <div align="center">
 <p float="center">
 <img src="https://www.codium.ai/images/review-2.gif" width="800">
 </p>
 </div>
-<h4>/reflect_and_review:</h4>
-<div align="center">
-<p float="center">
-<img src="https://www.codium.ai/images/reflect_and_review.gif" width="800">
-</p>
-</div>
-<h4>/ask:</h4>
-<div align="center">
-<p float="center">
-<img src="https://www.codium.ai/images/ask-2.gif" width="800">
-</p>
-</div>
-<h4>/improve:</h4>
-<div align="center">
-<p float="center">
-<img src="https://www.codium.ai/images/improve-2.gif" width="800">
-</p>
-</div>
+
+[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/78#issuecomment-1639739496">/reflect_and_review:</a></h4>)
+
+[//]: # (<div align="center">)
+
+[//]: # (<p float="center">)
+
+[//]: # (<img src="https://www.codium.ai/images/reflect_and_review.gif" width="800">)
+
+[//]: # (</p>)
+
+[//]: # (</div>)
+
+[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695020538">/ask:</a></h4>)
+
+[//]: # (<div align="center">)
+
+[//]: # (<p float="center">)
+
+[//]: # (<img src="https://www.codium.ai/images/ask-2.gif" width="800">)
+
+[//]: # (</p>)
+
+[//]: # (</div>)
+
+[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695024952">/improve:</a></h4>)
+
+[//]: # (<div align="center">)
+
+[//]: # (<p float="center">)
+
+[//]: # (<img src="https://www.codium.ai/images/improve-2.gif" width="800">)
+
+[//]: # (</p>)
+
+[//]: # (</div>)
 <div align="left">
 
-
+## Table of Contents
 - [Overview](#overview)
 - [Try it now](#try-it-now)
 - [Installation](#installation)
-- [Configuration](./CONFIGURATION.md)
 - [How it works](#how-it-works)
-- [Why use PR-Agent](#why-use-pr-agent)
+- [Why use PR-Agent?](#why-use-pr-agent)
 - [Roadmap](#roadmap)
-- [Similar projects](#similar-projects)
 </div>
 
 
 ## Overview
 `PR-Agent` offers extensive pull request functionalities across various git providers:
-|       |                                             | GitHub | Gitlab | Bitbucket | CodeCommit |
-|-------|---------------------------------------------|:------:|:------:|:---------:|:----------:|
-| TOOLS | Review                                      |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |   :white_check_mark:    |
-|       | ⮑ Inline review                             |   :white_check_mark:    |   :white_check_mark:    |           |          |
-|       | Ask                                         |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:          |
-|       | Auto-Description                            |   :white_check_mark:    |   :white_check_mark:    |           |          |
-|       | Improve Code                                |   :white_check_mark:    |   :white_check_mark:    |           |          |
-|       | Reflect and Review                          |   :white_check_mark:    |                         |           |          |
-|       | Update CHANGELOG.md                         |   :white_check_mark:    |                         |           |          |
-|       |                                             |        |        |      |      |
-| USAGE | CLI                                         |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |   :white_check_mark:    |
-|       | App / webhook                               |   :white_check_mark:    |   :white_check_mark:    |           |          |
-|       | Tagging bot                                 |   :white_check_mark:    |        |           |          |
-|       | Actions                                     |   :white_check_mark:    |        |           |          |
-|       |                                             |        |        |      |      |
-| CORE  | PR compression                              |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |          |
-|       | Repo language prioritization                |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |          |
-|       | Adaptive and token-aware<br />file patch fitting |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |          |
-|       | Multiple models support |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |          |
-|       | Incremental PR Review |   :white_check_mark:    |      |      |      |
+|       |                                             | GitHub | Gitlab | Bitbucket | CodeCommit | Azure DevOps | Gerrit |
+|-------|---------------------------------------------|:------:|:------:|:---------:|:----------:|:----------:|:----------:|
+| TOOLS | Review                                      |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:    |
+|       | Ask                                         |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:          |   :white_check_mark:          | :white_check_mark: |  :white_check_mark:    |
+|       | Auto-Description                            |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:        |   :white_check_mark:    |   :white_check_mark:    | :white_check_mark:    |
+|       | Improve Code                                |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:        |   :white_check_mark:    |          |    :white_check_mark:    |
+|       | ⮑ Extended                             |   :white_check_mark:    |   :white_check_mark:    |        :white_check_mark:   |   :white_check_mark:    |          | :white_check_mark:    |
+|       | Reflect and Review                          |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:        |          |   :white_check_mark:    |    :white_check_mark:    |
+|       | Update CHANGELOG.md                         |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:        |          |          |       |
+|       | Find similar issue                          |   :white_check_mark:    |                         |                             |          |          |       |
+|       |                                             |        |        |      |      |      |
+| USAGE | CLI                                         |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |   :white_check_mark:    |   :white_check_mark:    |
+|       | App / webhook                               |   :white_check_mark:    |   :white_check_mark:    |           |          |          |
+|       | Tagging bot                                 |   :white_check_mark:    |        |           |          |          |
+|       | Actions                                     |   :white_check_mark:    |        |           |          |          |
+|       | Web server                                  |       |        |           |          |          |  :white_check_mark:   |
+|       |                                             |        |        |      |      |      |
+| CORE  | PR compression                              |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       | :white_check_mark: |   :white_check_mark:       | :white_check_mark:       |
+|       | Repo language prioritization                |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       | :white_check_mark: |   :white_check_mark:       | :white_check_mark:       |
+|       | Adaptive and token-aware<br />file patch fitting |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       | :white_check_mark: |   :white_check_mark:       | :white_check_mark:       |
+|       | Multiple models support |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       | :white_check_mark: |   :white_check_mark:       | :white_check_mark:       |
+|       | Incremental PR Review |   :white_check_mark:    |      |      |      |      |      |
 
-Examples for invoking the different tools via the CLI:
-- **Review**:       python cli.py --pr_url=<pr_url>  review
-- **Describe**:     python cli.py --pr_url=<pr_url>  describe
-- **Improve**:      python cli.py --pr_url=<pr_url>  improve
-- **Ask**:          python cli.py --pr_url=<pr_url>  ask "Write me a poem about this PR"
-- **Reflect**:      python cli.py --pr_url=<pr_url>  reflect
-- **Update Changelog**:      python cli.py --pr_url=<pr_url>  update_changelog
-
-"<pr_url>" is the url of the relevant PR (for example: https://github.com/Codium-ai/pr-agent/pull/50).
-
-In the [configuration](./CONFIGURATION.md) file you can select your git provider (GitHub, Gitlab, Bitbucket), and further configure the different tools.
+Review the **[usage guide](./Usage.md)** section for detailed instructions how to use the different tools, select the relevant git provider (GitHub, Gitlab, Bitbucket,...), and adjust the configuration file to your needs.
 
 ## Try it now
 
-Try GPT-4 powered PR-Agent on your public GitHub repository for free. Just mention `@CodiumAI-Agent` and add the desired command in any PR comment! The agent will generate a response based on your command.
+You can try GPT-4 powered PR-Agent, on your public GitHub repository, instantly. Just mention `@CodiumAI-Agent` and add the desired command in any PR comment. The agent will generate a response based on your command.
+For example, add a comment to any pull request with the following text:
+```
+@CodiumAI-Agent /review
+```
+and the agent will respond with a review of your PR
 
 ![Review generation process](https://www.codium.ai/images/demo-2.gif)
 
-To set up your own PR-Agent, see the [Installation](#installation) section
+
+To set up your own PR-Agent, see the [Installation](#installation) section below.
 
 ---
 
@@ -128,13 +150,16 @@ To get started with PR-Agent quickly, you first need to acquire two tokens:
 There are several ways to use PR-Agent:
 
 - [Method 1: Use Docker image (no installation required)](INSTALL.md#method-1-use-docker-image-no-installation-required)
-- [Method 2: Run as a GitHub Action](INSTALL.md#method-2-run-as-a-github-action)
-- [Method 3: Run from source](INSTALL.md#method-3-run-from-source)
+- [Method 2: Run from source](INSTALL.md#method-2-run-from-source)
+- [Method 3: Run as a GitHub Action](INSTALL.md#method-3-run-as-a-github-action)
 - [Method 4: Run as a polling server](INSTALL.md#method-4-run-as-a-polling-server)
   - Request reviews by tagging your GitHub user on a PR
 - [Method 5: Run as a GitHub App](INSTALL.md#method-5-run-as-a-github-app)
   - Allowing you to automate the review process on your private or public repositories
-
+- [Method 6: Deploy as a Lambda Function](INSTALL.md#method-6---deploy-as-a-lambda-function)
+- [Method 7: AWS CodeCommit](INSTALL.md#method-7---aws-codecommit-setup)
+- [Method 8: Run a GitLab webhook server](INSTALL.md#method-8---run-a-gitlab-webhook-server)
+- [Method 9: Run as a Bitbucket Pipeline](INSTALL.md#method-9-run-as-a-bitbucket-pipeline)
 
 ## How it works
 
@@ -152,7 +177,7 @@ Here are some advantages of PR-Agent:
 
 - We emphasize **real-life practical usage**. Each tool (review, improve, ask, ...) has a single GPT-4 call, no more. We feel that this is critical for realistic team usage - obtaining an answer quickly (~30 seconds) and affordably.
 - Our [PR Compression strategy](./PR_COMPRESSION.md)  is a core ability that enables to effectively tackle both short and long PRs.
-- Our JSON prompting strategy enables to have **modular, customizable tools**. For example, the '/review' tool categories can be controlled via the [configuration](./CONFIGURATION.md) file. Adding additional categories is easy and accessible.
+- Our JSON prompting strategy enables to have **modular, customizable tools**. For example, the '/review' tool categories can be controlled via the [configuration](pr_agent/settings/configuration.toml) file. Adding additional categories is easy and accessible.
 - We support **multiple git providers** (GitHub, Gitlab, Bitbucket, CodeCommit), **multiple ways** to use the tool (CLI, GitHub Action, GitHub App, Docker, ...), and **multiple models** (GPT-4, GPT-3.5, Anthropic, Cohere, Llama2).
 - We are open-source, and welcome contributions from the community.
 
@@ -160,8 +185,9 @@ Here are some advantages of PR-Agent:
 ## Roadmap
 
 - [x] Support additional models, as a replacement for OpenAI (see [here](https://github.com/Codium-ai/pr-agent/pull/172))
-- [ ] Develop additional logic for handling large PRs
+- [x] Develop additional logic for handling large PRs (see [here](https://github.com/Codium-ai/pr-agent/pull/229))
 - [ ] Add additional context to the prompt. For example, repo (or relevant files) summarization, with tools such a [ctags](https://github.com/universal-ctags/ctags)
+- [x] PR-Agent for issues
 - [ ] Adding more tools. Possible directions:
   - [x] PR description
   - [x] Inline code suggestions
@@ -174,8 +200,18 @@ Here are some advantages of PR-Agent:
 
 ## Similar Projects
 
-- [CodiumAI - Meaningful tests for busy devs](https://github.com/Codium-ai/codiumai-vscode-release)
+- [CodiumAI - Meaningful tests for busy devs](https://github.com/Codium-ai/codiumai-vscode-release) (although various capabilities are much more advanced in the CodiumAI IDE plugins)
 - [Aider - GPT powered coding in your terminal](https://github.com/paul-gauthier/aider)
 - [openai-pr-reviewer](https://github.com/coderabbitai/openai-pr-reviewer)
 - [CodeReview BOT](https://github.com/anc95/ChatGPT-CodeReview)
 - [AI-Maintainer](https://github.com/merwanehamadi/AI-Maintainer)
+  
+## Links
+
+[![Join our Discord community](https://raw.githubusercontent.com/Codium-ai/codiumai-vscode-release/main/media/docs/Joincommunity.png)](https://discord.gg/kG35uSHDBc)
+
+- Discord community: https://discord.gg/kG35uSHDBc
+- CodiumAI site: https://codium.ai
+- Blog: https://www.codium.ai/blog/
+- Troubleshooting: https://www.codium.ai/blog/technical-faq-and-troubleshooting/
+- Support: support@codium.ai

RELEASE_NOTES.md

@@ -0,0 +1,41 @@
+## [Version 0.8] - 2023-09-27
+- codiumai/pr-agent:0.8
+- codiumai/pr-agent:0.8-github_app
+- codiumai/pr-agent:0.8-bitbucket-app
+- codiumai/pr-agent:0.8-gitlab_webhook
+- codiumai/pr-agent:0.8-github_polling
+- codiumai/pr-agent:0.8-github_action
+
+### Added::Algo
+- GitHub Action: Can control which tools will run automatically when a new PR is created. (see usage guide: https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#working-with-github-action)
+- Code suggestion tool: Will try to avoid an 'add comments' suggestion  (see https://github.com/Codium-ai/pr-agent/pull/327)
+
+### Fixed
+- Gitlab: Fixed a bug of improper usage of pr_id
+
+
+## [Version 0.7] - 2023-09-20
+
+### Docker Tags
+- codiumai/pr-agent:0.7
+- codiumai/pr-agent:0.7-github_app
+- codiumai/pr-agent:0.7-bitbucket-app
+- codiumai/pr-agent:0.7-gitlab_webhook
+- codiumai/pr-agent:0.7-github_polling
+- codiumai/pr-agent:0.7-github_action
+ 
+### Added::Algo
+- New tool /similar_issue - Currently on GitHub app and CLI: indexes the issues in the repo, find the most similar issues to the target issue.
+- Describe markers: Empower the /describe tool with a templating capability (see more details in https://github.com/Codium-ai/pr-agent/pull/273).
+- New feature in the /review tool - added an estimated effort estimation to the review (https://github.com/Codium-ai/pr-agent/pull/306).
+
+### Added::Infrastructure
+- Implementation of a GitLab webhook.
+- Implementation of a BitBucket app.
+
+### Fixed
+- Protection against no code suggestions generated.
+- Resilience to repositories where the languages cannot be automatically detected.
+
+
+

Usage.md

@@ -0,0 +1,286 @@
+## Usage guide
+
+### Table of Contents
+- [Introduction](#introduction)
+- [Working from a local repo (CLI)](#working-from-a-local-repo-cli)
+- [Online usage](#online-usage)
+- [Working with GitHub App](#working-with-github-app)
+- [Working with GitHub Action](#working-with-github-action)
+- [Appendix - additional configurations walkthrough](#appendix---additional-configurations-walkthrough)
+
+### Introduction
+
+There are 3 basic ways to invoke CodiumAI PR-Agent:
+1. Locally running a CLI command
+2. Online usage - by [commenting](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901) on a PR
+3. Enabling PR-Agent tools to run automatically when a new PR is opened
+
+See the [installation guide](/INSTALL.md) for instructions on how to setup your own PR-Agent.
+
+Specifically, CLI commands can be issued by invoking a pre-built [docker image](/INSTALL.md#running-from-source), or by invoking a [locally cloned repo](INSTALL.md#method-2-run-from-source).
+
+For online usage, you will need to setup either a [GitHub App](INSTALL.md#method-5-run-as-a-github-app), or a [GitHub Action](INSTALL.md#method-3-run-as-a-github-action).
+GitHub App and GitHub Action also enable to run PR-Agent specific tool automatically when a new PR is opened.
+
+
+#### The configuration file
+The different tools and sub-tools used by CodiumAI PR-Agent are adjustable via the **[configuration file](pr_agent/settings/configuration.toml)**.
+In addition to general configuration options, each tool has its own configurations. For example, the `review` tool will use parameters from the [pr_reviewer](/pr_agent/settings/configuration.toml#L16) section in the configuration file.
+
+**git provider:**
+The [git_provider](pr_agent/settings/configuration.toml#L4) field in the configuration file determines the GIT provider that will be used by PR-Agent. Currently, the following providers are supported:
+`
+"github", "gitlab", "azure", "codecommit", "local"
+`
+
+[//]: # (** online usage:**)
+
+[//]: # (Options that are available in the configuration file can be specified at run time when calling actions. Two examples:)
+
+[//]: # (```)
+
+[//]: # (- /review --pr_reviewer.extra_instructions="focus on the file: ...")
+
+[//]: # (- /describe --pr_description.add_original_user_description=false -pr_description.extra_instructions="make sure to mention: ...")
+
+[//]: # (```)
+
+### Working from a local repo (CLI)
+When running from your local repo (CLI), your local configuration file will be used.
+
+Examples for invoking the different tools via the CLI:
+
+- **Review**:       `python cli.py --pr_url=<pr_url>  review`
+- **Describe**:     `python cli.py --pr_url=<pr_url>  describe`
+- **Improve**:      `python cli.py --pr_url=<pr_url>  improve`
+- **Ask**:          `python cli.py --pr_url=<pr_url>  ask "Write me a poem about this PR"`
+- **Reflect**:      `python cli.py --pr_url=<pr_url>  reflect`
+- **Update Changelog**:      `python cli.py --pr_url=<pr_url>  update_changelog`
+
+`<pr_url>` is the url of the relevant PR (for example: https://github.com/Codium-ai/pr-agent/pull/50).
+
+**Notes:**
+
+(1) in addition to editing your local configuration file, you can also change any configuration value by adding it to the command line:
+```
+python cli.py --pr_url=<pr_url>  /review --pr_reviewer.extra_instructions="focus on the file: ..."
+```
+
+(2) You can print results locally, without publishing them, by setting in `configuration.toml`:
+```
+[config]
+publish_output=true
+verbosity_level=2
+```
+This is useful for debugging or experimenting with the different tools.
+
+
+### Online usage
+
+Online usage means invoking PR-Agent tools by [comments](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901) on a PR.
+Commands for invoking the different tools via comments:
+
+- **Review**:       `/review`
+- **Describe**:     `/describe`
+- **Improve**:      `/improve`
+- **Ask**:          `/ask "..."`
+- **Reflect**:      `/reflect`
+- **Update Changelog**:      `/update_changelog`
+
+
+To edit a specific configuration value, just add `--config_path=<value>` to any command.
+For example if you want to edit the `review` tool configurations, you can run:
+```
+/review --pr_reviewer.extra_instructions="..." --pr_reviewer.require_score_review=false
+```
+Any configuration value in [configuration file](pr_agent/settings/configuration.toml) file can be similarly edited.
+
+
+### Working with GitHub App
+When running PR-Agent from [GitHub App](INSTALL.md#method-5-run-as-a-github-app), the default configurations from a pre-built repo will be initially loaded.
+
+#### GitHub app automatic tools
+The [github_app](pr_agent/settings/configuration.toml#L56) section defines GitHub app specific configurations. 
+An important parameter is `pr_commands`, which is a list of tools that will be **run automatically when a new PR is opened**:
+```
+[github_app]
+pr_commands = [
+    "/describe --pr_description.add_original_user_description=true --pr_description.keep_original_user_title=true",
+    "/auto_review",
+]
+```
+This means that when a new PR is opened, PR-Agent will run the `describe` and `auto_review` tools.
+For the describe tool, the `add_original_user_description` and `keep_original_user_title` parameters will be set to true.
+
+However, you can override the default tool parameters by uploading a local configuration file called `.pr_agent.toml` to the root of your repo.
+For example, if your local `.pr_agent.toml` file contains:
+```
+[pr_description]
+add_original_user_description = false
+keep_original_user_title = false
+```
+When a new PR is opened, PR-Agent will run the `describe` tool with the above parameters.
+
+Note that a local `.pr_agent.toml` file enables you to edit and customize the default parameters of any tool, not just the ones that are run automatically.
+
+#### Editing the prompts
+The prompts for the various PR-Agent tools are defined in the `pr_agent/settings` folder.
+
+In practice, the prompts are loaded and stored as a standard setting object. 
+Hence, editing them is similar to editing any other configuration value - just place the relevant key in `.pr_agent.toml`file, and override the default value.
+
+For example, if you want to edit the prompts of the [describe](./pr_agent/settings/pr_description_prompts.toml) tool, you can add the following to your `.pr_agent.toml` file:
+```
+[pr_description_prompt]
+system="""
+...
+"""
+user="""
+...
+"""
+```
+Note that the new prompt will need to generate an output compatible with the relevant [post-process function](./pr_agent/tools/pr_description.py#L137).
+
+### Working with GitHub Action
+You can configure settings in GitHub action by adding environment variables under the env section in `.github/workflows/pr_agent.yml` file. Some examples:
+```yaml
+      env:
+        # ... previous environment values
+        OPENAI.ORG: "<Your organization name under your OpenAI account>"
+        PR_REVIEWER.REQUIRE_TESTS_REVIEW: "false" # Disable tests review
+        PR_CODE_SUGGESTIONS.NUM_CODE_SUGGESTIONS: 6 # Increase number of code suggestions
+        github_action.auto_review: "true" # Enable auto review
+        github_action.auto_describe: "true" # Enable auto describe
+        github_action.auto_improve: "false" # Disable auto improve      
+```
+specifically, `github_action.auto_review`, `github_action.auto_describe` and `github_action.auto_improve` are used to enable/disable automatic tools that run when a new PR is opened.
+
+if not set, the default option is that only the `review` tool will run automatically when a new PR is opened.
+
+
+### Appendix - additional configurations walkthrough
+
+#### Changing a model
+See [here](pr_agent/algo/__init__.py) for the list of available models.
+
+#### Azure
+To use Azure, set: 
+```
+api_key = "" # your azure api key
+api_type = "azure"
+api_version = '2023-05-15'  # Check Azure documentation for the current API version
+api_base = ""  # The base URL for your Azure OpenAI resource. e.g. "https://<your resource name>.openai.azure.com"
+deployment_id = ""  # The deployment name you chose when you deployed the engine
+```
+in your .secrets.toml
+
+and 
+```
+[config]
+model="" # the OpenAI model you've deployed on Azure (e.g. gpt-3.5-turbo)
+```
+in the configuration.toml 
+
+#### Huggingface
+
+**Local**  
+You can run Huggingface models locally through either [VLLM](https://docs.litellm.ai/docs/providers/vllm) or [Ollama](https://docs.litellm.ai/docs/providers/ollama)
+
+E.g. to use a new Huggingface model locally via Ollama, set:
+```
+[__init__.py]
+MAX_TOKENS = {
+    "model-name-on-ollama": <max_tokens>
+}
+e.g.
+MAX_TOKENS={
+    ...,
+    "llama2": 4096
+}
+
+
+[config] # in configuration.toml
+model = "ollama/llama2"
+
+[ollama] # in .secrets.toml
+api_base = ... # the base url for your huggingface inference endpoint 
+```
+
+**Inference Endpoints**
+
+To use a new model with Huggingface Inference Endpoints, for example, set:
+```
+[__init__.py]
+MAX_TOKENS = {
+    "model-name-on-huggingface": <max_tokens>
+}
+e.g.
+MAX_TOKENS={
+    ...,
+    "meta-llama/Llama-2-7b-chat-hf": 4096
+}
+[config] # in configuration.toml
+model = "huggingface/meta-llama/Llama-2-7b-chat-hf"
+
+[huggingface] # in .secrets.toml
+key = ... # your huggingface api key
+api_base = ... # the base url for your huggingface inference endpoint 
+```
+(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
+
+#### Replicate
+
+To use Llama2 model with Replicate, for example, set:
+```
+[config] # in configuration.toml
+model = "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1"
+[replicate] # in .secrets.toml
+key = ...
+```
+(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
+
+
+Also review the [AiHandler](pr_agent/algo/ai_handler.py) file for instruction how to set keys for other models.
+
+#### Extra instructions
+All PR-Agent tools have a parameter called `extra_instructions`, that enables to add free-text extra instructions. Example usage:
+```
+/update_changelog --pr_update_changelog.extra_instructions="Make sure to update also the version ..."
+```
+
+#### Azure DevOps provider
+To use Azure DevOps provider use the following settings in configuration.toml:
+```
+[config]
+git_provider="azure"
+use_repo_settings_file=false
+```
+
+And use the following settings (you have to replace the values) in .secrets.toml:
+```
+[azure_devops]
+org = "https://dev.azure.com/YOUR_ORGANIZATION/"
+pat = "YOUR_PAT_TOKEN"
+```
+
+#### Similar issue tool
+
+[Example usage](https://github.com/Alibaba-MIIL/ASL/issues/107)
+
+<img src=./pics/similar_issue_tool.png width="768">
+
+To enable usage of the '**similar issue**' tool, you need to set the following keys in `.secrets.toml` (or in the relevant environment variables):
+```
+[pinecone]
+api_key = "..."
+environment = "..."
+```
+These parameters can be obtained by registering to [Pinecone](https://app.pinecone.io/?sessionType=signup/).
+
+- To invoke the 'similar issue' tool from **CLI**, run:
+`python3 cli.py --issue_url=... similar_issue`
+
+- To invoke the 'similar' issue tool via online usage, [comment](https://github.com/Codium-ai/pr-agent/issues/178#issuecomment-1716934893) on a PR:
+`/similar_issue`
+
+- You can also enable the 'similar issue' tool to run automatically when a new issue is opened, by adding it to the [pr_commands list in the github_app section](https://github.com/Codium-ai/pr-agent/blob/main/pr_agent/settings/configuration.toml#L66)

bitbucket_pipeline/entrypoint.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+python /app/pr_agent/servers/bitbucket_pipeline_runner.py

docker/Dockerfile

@@ -2,7 +2,8 @@ FROM python:3.10 as base
 
 WORKDIR /app
 ADD pyproject.toml .
-RUN pip install . && rm pyproject.toml
+ADD requirements.txt .
+RUN pip install . && rm pyproject.toml requirements.txt
 ENV PYTHONPATH=/app
 
 FROM base as github_app
@@ -17,6 +18,10 @@ FROM base as github_polling
 ADD pr_agent pr_agent
 CMD ["python", "pr_agent/servers/github_polling.py"]
 
+FROM base as gitlab_webhook
+ADD pr_agent pr_agent
+CMD ["python", "pr_agent/servers/gitlab_webhook.py"]
+
 FROM base as test
 ADD requirements-dev.txt .
 RUN pip install -r requirements-dev.txt && rm requirements-dev.txt

pr_agent/agent/pr_agent.py

@@ -9,6 +9,7 @@ from pr_agent.git_providers import get_git_provider
 from pr_agent.tools.pr_code_suggestions import PRCodeSuggestions
 from pr_agent.tools.pr_description import PRDescription
 from pr_agent.tools.pr_information_from_user import PRInformationFromUser
+from pr_agent.tools.pr_similar_issue import PRSimilarIssue
 from pr_agent.tools.pr_questions import PRQuestions
 from pr_agent.tools.pr_reviewer import PRReviewer
 from pr_agent.tools.pr_update_changelog import PRUpdateChangelog
@@ -30,6 +31,7 @@ command2class = {
     "update_changelog": PRUpdateChangelog,
     "config": PRConfig,
     "settings": PRConfig,
+    "similar_issue": PRSimilarIssue,
 }
 
 commands = list(command2class.keys())

pr_agent/algo/__init__.py

@@ -1,4 +1,5 @@
 MAX_TOKENS = {
+    'text-embedding-ada-002': 8000,
     'gpt-3.5-turbo': 4000,
     'gpt-3.5-turbo-0613': 4000,
     'gpt-3.5-turbo-0301': 4000,
@@ -11,4 +12,5 @@ MAX_TOKENS = {
     'claude-2': 100000,
     'command-nightly': 4096,
     'replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1': 4096,
+    'meta-llama/Llama-2-7b-chat-hf': 4096
 }

pr_agent/algo/ai_handler.py

@@ -1,13 +1,12 @@
 import logging
+import os
 
 import litellm
 import openai
 from litellm import acompletion
 from openai.error import APIError, RateLimitError, Timeout, TryAgain
 from retry import retry
-
 from pr_agent.config_loader import get_settings
-
 OPENAI_RETRIES = 5
 
 
@@ -26,7 +25,11 @@ class AiHandler:
         try:
             openai.api_key = get_settings().openai.key
             litellm.openai_key = get_settings().openai.key
-            litellm.debugger = get_settings().config.litellm_debugger
+            if get_settings().get("litellm.use_client"):
+                litellm_token = get_settings().get("litellm.LITELLM_TOKEN")
+                assert litellm_token, "LITELLM_TOKEN is required"
+                os.environ["LITELLM_TOKEN"] = litellm_token
+                litellm.use_client = True
             self.azure = False
             if get_settings().get("OPENAI.ORG", None):
                 litellm.organization = get_settings().openai.org
@@ -48,6 +51,8 @@ class AiHandler:
                 litellm.replicate_key = get_settings().replicate.key
             if get_settings().get("HUGGINGFACE.KEY", None):
                 litellm.huggingface_key = get_settings().huggingface.key
+                if get_settings().get("HUGGINGFACE.API_BASE", None):
+                    litellm.api_base = get_settings().huggingface.api_base
         except AttributeError as e:
             raise ValueError("OpenAI key is required") from e
 
@@ -87,8 +92,6 @@ class AiHandler:
                     f"Generating completion with {model}"
                     f"{(' from deployment ' + deployment_id) if deployment_id else ''}"
                 )
-            if self.azure:
-                model = self.azure + "/" + model
             response = await acompletion(
                 model=model,
                 deployment_id=deployment_id,
@@ -97,6 +100,7 @@ class AiHandler:
                     {"role": "user", "content": user}
                 ],
                 temperature=temperature,
+                azure=self.azure,
                 force_timeout=get_settings().config.ai_timeout
             )
         except (APIError, Timeout, TryAgain) as e:

pr_agent/algo/git_patch_processing.py

@@ -40,12 +40,16 @@ def extend_patch(original_file_str, patch_str, num_lines) -> str:
                         extended_patch_lines.extend(
                             original_lines[start1 + size1 - 1:start1 + size1 - 1 + num_lines])
 
+                    res = list(match.groups())
+                    for i in range(len(res)):
+                        if res[i] is None:
+                            res[i] = 0
                     try:
-                        start1, size1, start2, size2 = map(int, match.groups()[:4])
+                        start1, size1, start2, size2 = map(int, res[:4])
                     except:  # '@@ -0,0 +1 @@' case
-                        start1, size1, size2 = map(int, match.groups()[:3])
+                        start1, size1, size2 = map(int, res[:3])
                         start2 = 0
-                    section_header = match.groups()[4]
+                    section_header = res[4]
                     extended_start1 = max(1, start1 - num_lines)
                     extended_size1 = size1 + (start1 - extended_start1) + num_lines
                     extended_start2 = max(1, start2 - num_lines)
@@ -207,10 +211,15 @@ __old hunk__
                 old_content_lines = []
             if match:
                 prev_header_line = header_line
+
+            res = list(match.groups())
+            for i in range(len(res)):
+                if res[i] is None:
+                    res[i] = 0
             try:
-                start1, size1, start2, size2 = map(int, match.groups()[:4])
+                start1, size1, start2, size2 = map(int, res[:4])
             except: # '@@ -0,0 +1 @@' case
-                start1, size1, size2 = map(int, match.groups()[:3])
+                start1, size1, size2 = map(int, res[:3])
                 start2 = 0
 
         elif line.startswith('+'):

pr_agent/algo/language_handler.py

@@ -42,6 +42,11 @@ def sort_files_by_main_languages(languages: Dict, files: list):
     files_sorted = []
     rest_files = {}
 
+    # if no languages detected, put all files in the "Other" category
+    if not languages:
+        files_sorted = [({"language": "Other", "files": list(files_filtered)})]
+        return files_sorted
+
     main_extensions_flat = []
     for ext in main_extensions:
         main_extensions_flat.extend(ext)

pr_agent/algo/token_handler.py

@@ -21,7 +21,7 @@ class TokenHandler:
       method.
     """
 
-    def __init__(self, pr, vars: dict, system, user):
+    def __init__(self, pr=None, vars: dict = {}, system="", user=""):
         """
         Initializes the TokenHandler object.
 
@@ -32,7 +32,8 @@ class TokenHandler:
         - user: The user string.
         """
         self.encoder = get_token_encoder()
-        self.prompt_tokens = self._get_system_user_tokens(pr, self.encoder, vars, system, user)
+        if pr is not None:
+            self.prompt_tokens = self._get_system_user_tokens(pr, self.encoder, vars, system, user)
 
     def _get_system_user_tokens(self, pr, encoder, vars: dict, system, user):
         """

pr_agent/algo/utils.py

@@ -20,7 +20,7 @@ def get_setting(key: str) -> Any:
     except Exception:
         return global_settings.get(key, None)
 
-def convert_to_markdown(output_data: dict) -> str:
+def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
     """
     Convert a dictionary of data into markdown format.
     Args:
@@ -42,6 +42,7 @@ def convert_to_markdown(output_data: dict) -> str:
         "General suggestions": "💡",
         "Insights from user's answers": "📝",
         "Code feedback": "🤖",
+        "Estimated effort to review [1-5]": "⏱️",
     }
 
     for key, value in output_data.items():
@@ -49,11 +50,14 @@ def convert_to_markdown(output_data: dict) -> str:
             continue
         if isinstance(value, dict):
             markdown_text += f"## {key}\n\n"
-            markdown_text += convert_to_markdown(value)
+            markdown_text += convert_to_markdown(value, gfm_supported)
         elif isinstance(value, list):
             emoji = emojis.get(key, "")
             if key.lower() == 'code feedback':
-                markdown_text += f"\n\n- **<details><summary> { emoji } Code feedback:**</summary>\n\n"
+                if gfm_supported:
+                    markdown_text += f"\n\n- **<details><summary> { emoji } Code feedback:**</summary>\n\n"
+                else:
+                    markdown_text += f"\n\n- **{emoji} Code feedback:**\n\n"
             else:
                 markdown_text += f"- {emoji} **{key}:**\n\n"
             for item in value:
@@ -62,7 +66,10 @@ def convert_to_markdown(output_data: dict) -> str:
                 elif item:
                     markdown_text += f"  - {item}\n"
             if key.lower() == 'code feedback':
-                markdown_text += "</details>\n\n"
+                if gfm_supported:
+                    markdown_text += "</details>\n\n"
+                else:
+                    markdown_text += "\n\n"
         elif value != 'n/a':
             emoji = emojis.get(key, "")
             markdown_text += f"- {emoji} **{key}:** {value}\n"
@@ -276,7 +283,7 @@ def _fix_key_value(key: str, value: str):
 def load_yaml(review_text: str) -> dict:
     review_text = review_text.removeprefix('```yaml').rstrip('`')
     try:
-        data = yaml.load(review_text, Loader=yaml.SafeLoader)
+        data = yaml.safe_load(review_text)
     except Exception as e:
         logging.error(f"Failed to parse AI prediction: {e}")
         data = try_fix_yaml(review_text)

pr_agent/cli.py

@@ -17,6 +17,7 @@ For example:
 - cli.py --pr_url=... improve
 - cli.py --pr_url=... ask "write me a poem about this PR"
 - cli.py --pr_url=... reflect
+- cli.py --issue_url=... similar_issue
 
 Supported commands:
 -review / review_pr - Add a review that includes a summary of the PR and specific suggestions for improvement.
@@ -37,14 +38,22 @@ Configuration:
 To edit any configuration parameter from 'configuration.toml', just add -config_path=<value>.
 For example: 'python cli.py --pr_url=... review --pr_reviewer.extra_instructions="focus on the file: ..."'
 """)
-    parser.add_argument('--pr_url', type=str, help='The URL of the PR to review', required=True)
+    parser.add_argument('--pr_url', type=str, help='The URL of the PR to review', default=None)
+    parser.add_argument('--issue_url', type=str, help='The URL of the Issue to review', default=None)
     parser.add_argument('command', type=str, help='The', choices=commands, default='review')
     parser.add_argument('rest', nargs=argparse.REMAINDER, default=[])
     args = parser.parse_args(inargs)
+    if not args.pr_url and not args.issue_url:
+        parser.print_help()
+        return
+
     logging.basicConfig(level=os.environ.get("LOGLEVEL", "INFO"))
     command = args.command.lower()
     get_settings().set("CONFIG.CLI_MODE", True)
-    result = asyncio.run(PRAgent().handle_request(args.pr_url, command + " " + " ".join(args.rest)))
+    if args.issue_url:
+        result = asyncio.run(PRAgent().handle_request(args.issue_url, command + " " + " ".join(args.rest)))
+    else:
+        result = asyncio.run(PRAgent().handle_request(args.pr_url, command + " " + " ".join(args.rest)))
     if not result:
         parser.print_help()
 

pr_agent/git_providers/__init__.py

@@ -4,13 +4,18 @@ from pr_agent.git_providers.codecommit_provider import CodeCommitProvider
 from pr_agent.git_providers.github_provider import GithubProvider
 from pr_agent.git_providers.gitlab_provider import GitLabProvider
 from pr_agent.git_providers.local_git_provider import LocalGitProvider
+from pr_agent.git_providers.azuredevops_provider import AzureDevopsProvider
+from pr_agent.git_providers.gerrit_provider import GerritProvider
+
 
 _GIT_PROVIDERS = {
     'github': GithubProvider,
     'gitlab': GitLabProvider,
     'bitbucket': BitbucketProvider,
+    'azure': AzureDevopsProvider,
     'codecommit': CodeCommitProvider,
-    'local' : LocalGitProvider
+    'local' : LocalGitProvider,
+    'gerrit': GerritProvider,
 }
 
 def get_git_provider():

pr_agent/git_providers/azuredevops_provider.py

@@ -0,0 +1,270 @@
+import json
+import logging
+from typing import Optional, Tuple
+from urllib.parse import urlparse
+
+import os
+
+AZURE_DEVOPS_AVAILABLE = True
+try:
+    from msrest.authentication import BasicAuthentication
+    from azure.devops.connection import Connection
+    from azure.devops.v7_1.git.models import Comment, CommentThread, GitVersionDescriptor, GitPullRequest
+except ImportError:
+    AZURE_DEVOPS_AVAILABLE = False
+
+from ..algo.pr_processing import clip_tokens
+from ..config_loader import get_settings
+from ..algo.utils import load_large_diff
+from ..algo.language_handler import is_valid_file
+from .git_provider import EDIT_TYPE, FilePatchInfo
+
+
+class AzureDevopsProvider:
+    def __init__(self, pr_url: Optional[str] = None, incremental: Optional[bool] = False):
+        if not AZURE_DEVOPS_AVAILABLE:
+            raise ImportError("Azure DevOps provider is not available. Please install the required dependencies.")
+
+        self.azure_devops_client = self._get_azure_devops_client()
+
+        self.workspace_slug = None
+        self.repo_slug = None
+        self.repo = None
+        self.pr_num = None
+        self.pr = None
+        self.temp_comments = []
+        self.incremental = incremental
+        if pr_url:
+            self.set_pr(pr_url)
+
+    def is_supported(self, capability: str) -> bool:
+        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels',
+                          'remove_initial_comment', 'gfm_markdown']:
+            return False
+        return True
+
+    def set_pr(self, pr_url: str):
+        self.workspace_slug, self.repo_slug, self.pr_num = self._parse_pr_url(pr_url)
+        self.pr = self._get_pr()
+
+    def get_repo_settings(self):
+        try:
+            contents = self.azure_devops_client.get_item_content(repository_id=self.repo_slug,
+                                                                 project=self.workspace_slug, download=False,
+                                                                 include_content_metadata=False, include_content=True,
+                                                                 path=".pr_agent.toml")
+            return contents
+        except Exception as e:
+            logging.exception("get repo settings error")
+            return ""
+
+    def get_files(self):
+        files = []
+        for i in self.azure_devops_client.get_pull_request_commits(project=self.workspace_slug,
+                                                                   repository_id=self.repo_slug,
+                                                                   pull_request_id=self.pr_num):
+
+            changes_obj = self.azure_devops_client.get_changes(project=self.workspace_slug,
+                                                               repository_id=self.repo_slug, commit_id=i.commit_id)
+
+            for c in changes_obj.changes:
+                files.append(c['item']['path'])
+        return list(set(files))
+
+    def get_diff_files(self) -> list[FilePatchInfo]:
+        try:
+            base_sha = self.pr.last_merge_target_commit
+            head_sha = self.pr.last_merge_source_commit
+
+            commits = self.azure_devops_client.get_pull_request_commits(project=self.workspace_slug,
+                                                                        repository_id=self.repo_slug,
+                                                                        pull_request_id=self.pr_num)
+
+            diff_files = []
+            diffs = []
+            diff_types = {}
+
+            for c in commits:
+                changes_obj = self.azure_devops_client.get_changes(project=self.workspace_slug,
+                                                                   repository_id=self.repo_slug, commit_id=c.commit_id)
+                for i in changes_obj.changes:
+                    diffs.append(i['item']['path'])
+                    diff_types[i['item']['path']] = i['changeType']
+
+            diffs = list(set(diffs))
+
+            for file in diffs:
+                if not is_valid_file(file):
+                    continue
+
+                version = GitVersionDescriptor(version=head_sha.commit_id, version_type='commit')
+                new_file_content_str = self.azure_devops_client.get_item(repository_id=self.repo_slug,
+                                                                         path=file,
+                                                                         project=self.workspace_slug,
+                                                                         version_descriptor=version,
+                                                                         download=False,
+                                                                         include_content=True)
+
+                new_file_content_str = new_file_content_str.content
+
+                edit_type = EDIT_TYPE.MODIFIED
+                if diff_types[file] == 'add':
+                    edit_type = EDIT_TYPE.ADDED
+                elif diff_types[file] == 'delete':
+                    edit_type = EDIT_TYPE.DELETED
+                elif diff_types[file] == 'rename':
+                    edit_type = EDIT_TYPE.RENAMED
+
+                version = GitVersionDescriptor(version=base_sha.commit_id, version_type='commit')
+                original_file_content_str = self.azure_devops_client.get_item(repository_id=self.repo_slug,
+                                                                              path=file,
+                                                                              project=self.workspace_slug,
+                                                                              version_descriptor=version,
+                                                                              download=False,
+                                                                              include_content=True)
+                original_file_content_str = original_file_content_str.content
+
+                patch = load_large_diff(file, new_file_content_str, original_file_content_str)
+
+                diff_files.append(FilePatchInfo(original_file_content_str, new_file_content_str,
+                                                patch=patch,
+                                                filename=file,
+                                                edit_type=edit_type))
+
+            self.diff_files = diff_files
+            return diff_files
+        except Exception as e:
+            print(f"Error: {str(e)}")
+            return []
+
+    def publish_comment(self, pr_comment: str, is_temporary: bool = False):
+        comment = Comment(content=pr_comment)
+        thread = CommentThread(comments=[comment])
+        thread_response = self.azure_devops_client.create_thread(comment_thread=thread, project=self.workspace_slug,
+                                                                 repository_id=self.repo_slug,
+                                                                 pull_request_id=self.pr_num)
+        if is_temporary:
+            self.temp_comments.append({'thread_id': thread_response.id, 'comment_id': comment.id})
+
+    def publish_description(self, pr_title: str, pr_body: str):
+        try:
+            updated_pr = GitPullRequest()
+            updated_pr.title = pr_title
+            updated_pr.description = pr_body
+            self.azure_devops_client.update_pull_request(project=self.workspace_slug,
+                                                         repository_id=self.repo_slug,
+                                                         pull_request_id=self.pr_num,
+                                                         git_pull_request_to_update=updated_pr)
+        except Exception as e:
+            logging.exception(f"Could not update pull request {self.pr_num} description: {e}")
+
+    def remove_initial_comment(self):
+        return ""  # not implemented yet
+
+    def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
+        raise NotImplementedError("Azure DevOps provider does not support publishing inline comment yet")
+
+    def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
+        raise NotImplementedError("Azure DevOps provider does not support creating inline comments yet")
+
+    def publish_inline_comments(self, comments: list[dict]):
+        raise NotImplementedError("Azure DevOps provider does not support publishing inline comments yet")
+
+    def get_title(self):
+        return self.pr.title
+
+    def get_languages(self):
+        languages = []
+        files = self.azure_devops_client.get_items(project=self.workspace_slug, repository_id=self.repo_slug,
+                                                   recursion_level="Full", include_content_metadata=True,
+                                                   include_links=False, download=False)
+        for f in files:
+            if f.git_object_type == 'blob':
+                file_name, file_extension = os.path.splitext(f.path)
+                languages.append(file_extension[1:])
+
+        extension_counts = {}
+        for ext in languages:
+            if ext != '':
+                extension_counts[ext] = extension_counts.get(ext, 0) + 1
+
+        total_extensions = sum(extension_counts.values())
+
+        extension_percentages = {ext: (count / total_extensions) * 100 for ext, count in extension_counts.items()}
+
+        return extension_percentages
+
+    def get_pr_branch(self):
+        pr_info = self.azure_devops_client.get_pull_request_by_id(project=self.workspace_slug,
+                                                                  pull_request_id=self.pr_num)
+        source_branch = pr_info.source_ref_name.split('/')[-1]
+        return source_branch
+
+    def get_pr_description(self):
+        max_tokens = get_settings().get("CONFIG.MAX_DESCRIPTION_TOKENS", None)
+        if max_tokens:
+            return clip_tokens(self.pr.description, max_tokens)
+        return self.pr.description
+
+    def get_user_id(self):
+        return 0
+
+    def get_issue_comments(self):
+        raise NotImplementedError("Azure DevOps provider does not support issue comments yet")
+
+    def add_eyes_reaction(self, issue_comment_id: int) -> Optional[int]:
+        return True
+
+    def remove_reaction(self, issue_comment_id: int, reaction_id: int) -> bool:
+        return True
+
+    def get_issue_comments(self):
+        raise NotImplementedError("Azure DevOps provider does not support issue comments yet")
+
+    @staticmethod
+    def _parse_pr_url(pr_url: str) -> Tuple[str, int]:
+        parsed_url = urlparse(pr_url)
+
+        if 'azure.com' not in parsed_url.netloc:
+            raise ValueError("The provided URL is not a valid Azure DevOps URL")
+
+        path_parts = parsed_url.path.strip('/').split('/')
+
+        if len(path_parts) < 6 or path_parts[4] != 'pullrequest':
+            raise ValueError("The provided URL does not appear to be a Azure DevOps PR URL")
+
+        workspace_slug = path_parts[1]
+        repo_slug = path_parts[3]
+        try:
+            pr_number = int(path_parts[5])
+        except ValueError as e:
+            raise ValueError("Unable to convert PR number to integer") from e
+
+        return workspace_slug, repo_slug, pr_number
+
+    def _get_azure_devops_client(self):
+        try:
+            pat = get_settings().azure_devops.pat
+            org = get_settings().azure_devops.org
+        except AttributeError as e:
+            raise ValueError(
+                "Azure DevOps PAT token is required ") from e
+
+        credentials = BasicAuthentication('', pat)
+        azure_devops_connection = Connection(base_url=org, creds=credentials)
+        azure_devops_client = azure_devops_connection.clients.get_git_client()
+
+        return azure_devops_client
+
+    def _get_repo(self):
+        if self.repo is None:
+            self.repo = self.azure_devops_client.get_repository(project=self.workspace_slug,
+                                                                repository_id=self.repo_slug)
+        return self.repo
+
+    def _get_pr(self):
+        self.pr = self.azure_devops_client.get_pull_request_by_id(pull_request_id=self.pr_num, project=self.workspace_slug)
+        return self.pr
+
+    def get_commit_messages(self):
+        return ""  # not implemented yet

pr_agent/git_providers/bitbucket_provider.py

@@ -7,6 +7,7 @@ import requests
 from atlassian.bitbucket import Cloud
 from starlette_context import context
 
+from ..algo.pr_processing import clip_tokens, find_line_number_of_relevant_line_in_file
 from ..config_loader import get_settings
 from .git_provider import FilePatchInfo, GitProvider
 
@@ -35,9 +36,8 @@ class BitbucketProvider(GitProvider):
         self.incremental = incremental
         if pr_url:
             self.set_pr(pr_url)
-        self.bitbucket_comment_api_url = self.pr._BitbucketBase__data["links"][
-            "comments"
-        ]["href"]
+        self.bitbucket_comment_api_url = self.pr._BitbucketBase__data["links"]["comments"]["href"]
+        self.bitbucket_pull_request_api_url = self.pr._BitbucketBase__data["links"]['self']['href']
 
     def get_repo_settings(self):
         try:
@@ -101,12 +101,7 @@ class BitbucketProvider(GitProvider):
             return False
 
     def is_supported(self, capability: str) -> bool:
-        if capability in [
-            "get_issue_comments",
-            "create_inline_comment",
-            "publish_inline_comments",
-            "get_labels",
-        ]:
+        if capability in ['get_issue_comments', 'publish_inline_comments', 'get_labels', 'gfm_markdown']:
             return False
         return True
 
@@ -151,17 +146,30 @@ class BitbucketProvider(GitProvider):
         except Exception as e:
             logging.exception(f"Failed to remove temp comments, error: {e}")
 
-    def publish_inline_comment(
-        self, comment: str, from_line: int, to_line: int, file: str
-    ):
-        payload = json.dumps(
-            {
-                "content": {
-                    "raw": comment,
-                },
-                "inline": {"to": from_line, "path": file},
-            }
-        )
+
+    # funtion to create_inline_comment
+    def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
+        position, absolute_position = find_line_number_of_relevant_line_in_file(self.get_diff_files(), relevant_file.strip('`'), relevant_line_in_file)
+        if position == -1:
+            if get_settings().config.verbosity_level >= 2:
+                logging.info(f"Could not find position for {relevant_file} {relevant_line_in_file}")
+            subject_type = "FILE"
+        else:
+            subject_type = "LINE"
+        path = relevant_file.strip()
+        return dict(body=body, path=path, position=absolute_position) if subject_type == "LINE" else {}
+
+
+    def publish_inline_comment(self, comment: str, from_line: int, file: str):
+        payload = json.dumps( {
+            "content": {
+                "raw": comment,
+            },
+            "inline": {
+                "to": from_line,
+                "path": file
+            },
+        })
         response = requests.request(
             "POST", self.bitbucket_comment_api_url, data=payload, headers=self.headers
         )
@@ -169,9 +177,7 @@ class BitbucketProvider(GitProvider):
 
     def publish_inline_comments(self, comments: list[dict]):
         for comment in comments:
-            self.publish_inline_comment(
-                comment["body"], comment["start_line"], comment["line"], comment["path"]
-            )
+            self.publish_inline_comment(comment['body'], comment['start_line'], comment['path'])
 
     def get_title(self):
         return self.pr.title
@@ -239,15 +245,21 @@ class BitbucketProvider(GitProvider):
     def get_commit_messages(self):
         return ""  # not implemented yet
     
-    def publish_description(self, pr_title: str, pr_body: str):
-        pass
-    def create_inline_comment(
-        self, body: str, relevant_file: str, relevant_line_in_file: str
-    ):
-        pass
-
-    def publish_labels(self, labels):
+    # bitbucket does not support labels
+    def publish_description(self, pr_title: str, description: str):
+        payload = json.dumps({
+            "description": description,
+            "title": pr_title
+
+            })
+
+        response = requests.request("PUT", self.bitbucket_pull_request_api_url, headers=self.headers, data=payload)
+        return response
+
+    # bitbucket does not support labels
+    def publish_labels(self, pr_types: list):
         pass
     
+    # bitbucket does not support labels
     def get_labels(self):
         pass

pr_agent/git_providers/codecommit_client.py

@@ -54,17 +54,22 @@ class CodeCommitClient:
     def __init__(self):
         self.boto_client = None
 
+    def is_supported(self, capability: str) -> bool:
+        if capability in ["gfm_markdown"]:
+            return False
+        return True
+
     def _connect_boto_client(self):
         try:
             self.boto_client = boto3.client("codecommit")
         except Exception as e:
-            raise ValueError(f"Failed to connect to AWS CodeCommit: {e}")
+            raise ValueError(f"Failed to connect to AWS CodeCommit: {e}") from e
 
     def get_differences(self, repo_name: int, destination_commit: str, source_commit: str):
         """
         Get the differences between two commits in CodeCommit.
 
-        Parameters:
+        Args:
         - repo_name: Name of the repository
         - destination_commit: Commit hash you want to merge into (the "before" hash) (usually on the main or master branch)
         - source_commit: Commit hash of the code you are adding (the "after" branch)
@@ -73,8 +78,8 @@ class CodeCommitClient:
         - List of CodeCommitDifferencesResponse objects
 
         Boto3 Documentation:
-        aws codecommit get-differences
-        https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_differences.html
+        - aws codecommit get-differences
+        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_differences.html
         """
         if self.boto_client is None:
             self._connect_boto_client()
@@ -90,7 +95,11 @@ class CodeCommitClient:
             ):
                 differences.extend(page.get("differences", []))
         except botocore.exceptions.ClientError as e:
-            raise ValueError(f"Failed to retrieve differences from CodeCommit PR #{self.pr_num}") from e
+            if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
+                raise ValueError(f"CodeCommit cannot retrieve differences: Repository does not exist: {repo_name}") from e
+            raise ValueError(f"CodeCommit cannot retrieve differences for {source_commit}..{destination_commit}") from e
+        except Exception as e:
+            raise ValueError(f"CodeCommit cannot retrieve differences for {source_commit}..{destination_commit}") from e
 
         output = []
         for json in differences:
@@ -101,7 +110,7 @@ class CodeCommitClient:
         """
         Retrieve a file from CodeCommit.
 
-        Parameters:
+        Args:
         - repo_name: Name of the repository
         - file_path: Path to the file you are retrieving
         - sha_hash: Commit hash of the file you are retrieving
@@ -110,8 +119,8 @@ class CodeCommitClient:
         - File contents
 
         Boto3 Documentation:
-        aws codecommit get_file
-        https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_file.html
+        - aws codecommit get_file
+        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_file.html
         """
         if not file_path:
             return ""
@@ -122,6 +131,8 @@ class CodeCommitClient:
         try:
             response = self.boto_client.get_file(repositoryName=repo_name, commitSpecifier=sha_hash, filePath=file_path)
         except botocore.exceptions.ClientError as e:
+            if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
+                raise ValueError(f"CodeCommit cannot retrieve PR: Repository does not exist: {repo_name}") from e
             # if the file does not exist, but is flagged as optional, then return an empty string
             if optional and e.response["Error"]["Code"] == 'FileDoesNotExistException':
                 return ""
@@ -133,19 +144,20 @@ class CodeCommitClient:
 
         return response.get("fileContent", "")
 
-    def get_pr(self, pr_number: int):
+    def get_pr(self, repo_name: str, pr_number: int):
         """
         Get a information about a CodeCommit PR.
 
-        Parameters:
+        Args:
+        - repo_name: Name of the repository
         - pr_number: The PR number you are requesting
 
         Returns:
         - CodeCommitPullRequestResponse object
 
         Boto3 Documentation:
-        aws codecommit get_pull_request
-        https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_pull_request.html
+        - aws codecommit get_pull_request
+        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_pull_request.html
         """
         if self.boto_client is None:
             self._connect_boto_client()
@@ -155,6 +167,8 @@ class CodeCommitClient:
         except botocore.exceptions.ClientError as e:
             if e.response["Error"]["Code"] == 'PullRequestDoesNotExistException':
                 raise ValueError(f"CodeCommit cannot retrieve PR: PR number does not exist: {pr_number}") from e
+            if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
+                raise ValueError(f"CodeCommit cannot retrieve PR: Repository does not exist: {repo_name}") from e
             raise ValueError(f"CodeCommit cannot retrieve PR: {pr_number}: boto client error") from e
         except Exception as e:
             raise ValueError(f"CodeCommit cannot retrieve PR: {pr_number}") from e
@@ -164,35 +178,95 @@ class CodeCommitClient:
 
         return CodeCommitPullRequestResponse(response.get("pullRequest", {}))
 
-    def publish_comment(self, repo_name: str, pr_number: int, destination_commit: str, source_commit: str, comment: str):
+    def publish_description(self, pr_number: int, pr_title: str, pr_body: str):
         """
-        Publish a comment to a pull request
+        Set the title and description on a pull request
 
-        Parameters:
-        - repo_name: name of the repository
-        - pr_number: number of the pull request
-        - destination_commit: The commit hash you want to merge into (the "before" hash) (usually on the main or master branch)
-        - source_commit: The commit hash of the code you are adding (the "after" branch)
-        - pr_comment: comment
+        Args:
+        - pr_number: the AWS CodeCommit pull request number
+        - pr_title: title of the pull request
+        - pr_body: body of the pull request
 
         Returns:
         - None
 
         Boto3 Documentation:
-        aws codecommit post_comment_for_pull_request
-        https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_pull_request.html
+        - aws codecommit update_pull_request_title
+        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/update_pull_request_title.html
+        - aws codecommit update_pull_request_description
+        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/update_pull_request_description.html
         """
         if self.boto_client is None:
             self._connect_boto_client()
 
         try:
-            self.boto_client.post_comment_for_pull_request(
-                pullRequestId=str(pr_number),
-                repositoryName=repo_name,
-                beforeCommitId=destination_commit,
-                afterCommitId=source_commit,
-                content=comment,
-            )
+            self.boto_client.update_pull_request_title(pullRequestId=str(pr_number), title=pr_title)
+            self.boto_client.update_pull_request_description(pullRequestId=str(pr_number), description=pr_body)
+        except botocore.exceptions.ClientError as e:
+            if e.response["Error"]["Code"] == 'PullRequestDoesNotExistException':
+                raise ValueError(f"PR number does not exist: {pr_number}") from e
+            if e.response["Error"]["Code"] == 'InvalidTitleException':
+                raise ValueError(f"Invalid title for PR number: {pr_number}") from e
+            if e.response["Error"]["Code"] == 'InvalidDescriptionException':
+                raise ValueError(f"Invalid description for PR number: {pr_number}") from e
+            if e.response["Error"]["Code"] == 'PullRequestAlreadyClosedException':
+                raise ValueError(f"PR is already closed: PR number: {pr_number}") from e
+            raise ValueError(f"Boto3 client error calling publish_description") from e
+        except Exception as e:
+            raise ValueError(f"Error calling publish_description") from e
+
+    def publish_comment(self, repo_name: str, pr_number: int, destination_commit: str, source_commit: str, comment: str, annotation_file: str = None, annotation_line: int = None):
+        """
+        Publish a comment to a pull request
+
+        Args:
+        - repo_name: name of the repository
+        - pr_number: number of the pull request
+        - destination_commit: The commit hash you want to merge into (the "before" hash) (usually on the main or master branch)
+        - source_commit: The commit hash of the code you are adding (the "after" branch)
+        - comment: The comment you want to publish
+        - annotation_file: The file you want to annotate (optional)
+        - annotation_line: The line number you want to annotate (optional)
+
+        Comment annotations for CodeCommit are different than GitHub.
+        CodeCommit only designates the starting line number for the comment.
+        It does not support the ending line number to highlight a range of lines.
+
+        Returns:
+        - None
+
+        Boto3 Documentation:
+        - aws codecommit post_comment_for_pull_request
+        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_pull_request.html
+        """
+        if self.boto_client is None:
+            self._connect_boto_client()
+
+        try:
+            # If the comment has code annotations,
+            # then set the file path and line number in the location dictionary
+            if annotation_file and annotation_line:
+                self.boto_client.post_comment_for_pull_request(
+                    pullRequestId=str(pr_number),
+                    repositoryName=repo_name,
+                    beforeCommitId=destination_commit,
+                    afterCommitId=source_commit,
+                    content=comment,
+                    location={
+                        "filePath": annotation_file,
+                        "filePosition": annotation_line,
+                        "relativeFileVersion": "AFTER",
+                    },
+                )
+            else:
+                # The comment does not have code annotations
+                self.boto_client.post_comment_for_pull_request(
+                    pullRequestId=str(pr_number),
+                    repositoryName=repo_name,
+                    beforeCommitId=destination_commit,
+                    afterCommitId=source_commit,
+                    content=comment,
+                )
         except botocore.exceptions.ClientError as e:
             if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
                 raise ValueError(f"Repository does not exist: {repo_name}") from e

pr_agent/git_providers/codecommit_provider.py

@@ -1,5 +1,6 @@
 import logging
 import os
+import re
 from collections import Counter
 from typing import List, Optional, Tuple
 from urllib.parse import urlparse
@@ -73,6 +74,7 @@ class CodeCommitProvider(GitProvider):
             "create_inline_comment",
             "publish_inline_comments",
             "get_labels",
+            "gfm_markdown"
         ]:
             return False
         return True
@@ -153,26 +155,63 @@ class CodeCommitProvider(GitProvider):
         return self.diff_files
 
     def publish_description(self, pr_title: str, pr_body: str):
-        return ""  # not implemented yet
+        try:
+            self.codecommit_client.publish_description(
+                pr_number=self.pr_num,
+                pr_title=pr_title,
+                pr_body=CodeCommitProvider._add_additional_newlines(pr_body),
+            )
+        except Exception as e:
+            raise ValueError(f"CodeCommit Cannot publish description for PR: {self.pr_num}") from e    
 
     def publish_comment(self, pr_comment: str, is_temporary: bool = False):
         if is_temporary:
             logging.info(pr_comment)
             return
 
+        pr_comment = CodeCommitProvider._remove_markdown_html(pr_comment)
+        pr_comment = CodeCommitProvider._add_additional_newlines(pr_comment)
+
         try:
             self.codecommit_client.publish_comment(
                 repo_name=self.repo_name,
-                pr_number=str(self.pr_num),
+                pr_number=self.pr_num,
                 destination_commit=self.pr.destination_commit,
                 source_commit=self.pr.source_commit,
                 comment=pr_comment,
             )
         except Exception as e:
-            raise ValueError(f"CodeCommit Cannot post comment for PR: {self.pr_num}") from e
+            raise ValueError(f"CodeCommit Cannot publish comment for PR: {self.pr_num}") from e
 
     def publish_code_suggestions(self, code_suggestions: list) -> bool:
-        return [""]  # not implemented yet
+        counter = 1
+        for suggestion in code_suggestions:
+            # Verify that each suggestion has the required keys
+            if not all(key in suggestion for key in ["body", "relevant_file", "relevant_lines_start"]):
+                logging.warning(f"Skipping code suggestion #{counter}: Each suggestion must have 'body', 'relevant_file', 'relevant_lines_start' keys")
+                continue
+       
+            # Publish the code suggestion to CodeCommit
+            try:
+                logging.debug(f"Code Suggestion #{counter} in file: {suggestion['relevant_file']}: {suggestion['relevant_lines_start']}")
+                self.codecommit_client.publish_comment(
+                    repo_name=self.repo_name,
+                    pr_number=self.pr_num,
+                    destination_commit=self.pr.destination_commit,
+                    source_commit=self.pr.source_commit,
+                    comment=suggestion["body"],
+                    annotation_file=suggestion["relevant_file"],
+                    annotation_line=suggestion["relevant_lines_start"],
+                )
+            except Exception as e:
+                raise ValueError(f"CodeCommit Cannot publish code suggestions for PR: {self.pr_num}") from e
+            
+            counter += 1
+
+        # The calling function passes in a list of code suggestions, and this function publishes each suggestion one at a time.
+        # If we were to return False here, the calling function will attempt to publish the same list of code suggestions again, one at a time.
+        # Since this function publishes the suggestions one at a time anyway, we always return True here to avoid the retry.        
+        return True
 
     def publish_labels(self, labels):
         return [""]  # not implemented yet
@@ -184,6 +223,7 @@ class CodeCommitProvider(GitProvider):
         return ""  # not implemented yet
 
     def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
+        # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_compared_commit.html
         raise NotImplementedError("CodeCommit provider does not support publishing inline comments yet")
 
     def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
@@ -193,14 +233,26 @@ class CodeCommitProvider(GitProvider):
         raise NotImplementedError("CodeCommit provider does not support publishing inline comments yet")
 
     def get_title(self):
-        return self.pr.get("title", "")
+        return self.pr.title
+
+    def get_pr_id(self):
+        """
+        Returns the PR ID in the format: "repo_name/pr_number".
+        Note: This is an internal identifier for PR-Agent, 
+        and is not the same as the CodeCommit PR identifier.
+        """
+        try:
+            pr_id = f"{self.repo_name}/{self.pr_num}"
+            return pr_id
+        except:
+            return ""
         
     def get_languages(self):
         """
         Returns a dictionary of languages, containing the percentage of each language used in the PR.
 
         Returns:
-            dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR.
+        - dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR.
         """
         commit_files = self.get_files()
         filenames = [ item.filename for item in commit_files ]
@@ -244,18 +296,29 @@ class CodeCommitProvider(GitProvider):
         return self.codecommit_client.get_file(self.repo_name, settings_filename, self.pr.source_commit, optional=True)
 
     def add_eyes_reaction(self, issue_comment_id: int) -> Optional[int]:
+        logging.info("CodeCommit provider does not support eyes reaction yet")
         return True
 
     def remove_reaction(self, issue_comment_id: int, reaction_id: int) -> bool:
+        logging.info("CodeCommit provider does not support removing reactions yet")
         return True
 
     @staticmethod
     def _parse_pr_url(pr_url: str) -> Tuple[str, int]:
+        """
+        Parse the CodeCommit PR URL and return the repository name and PR number.
+
+        Args:
+        - pr_url: the full AWS CodeCommit pull request URL
+
+        Returns:
+        - Tuple[str, int]: A tuple containing the repository name and PR number.
+        """
         # Example PR URL:
         # https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/__MY_REPO__/pull-requests/123456"
         parsed_url = urlparse(pr_url)
 
-        if "us-east-1.console.aws.amazon.com" not in parsed_url.netloc:
+        if not CodeCommitProvider._is_valid_codecommit_hostname(parsed_url.netloc):
             raise ValueError(f"The provided URL is not a valid CodeCommit URL: {pr_url}")
 
         path_parts = parsed_url.path.strip("/").split("/")
@@ -278,17 +341,33 @@ class CodeCommitProvider(GitProvider):
 
         return repo_name, pr_number
 
+    @staticmethod
+    def _is_valid_codecommit_hostname(hostname: str) -> bool:
+        """
+        Check if the provided hostname is a valid AWS CodeCommit hostname.
+
+        This is not an exhaustive check of AWS region names, 
+        but instead uses a regex to check for matching AWS region patterns.
+
+        Args:
+        - hostname: the hostname to check
+
+        Returns:
+        - bool: True if the hostname is valid, False otherwise.
+        """
+        return re.match(r"^[a-z]{2}-(gov-)?[a-z]+-\d\.console\.aws\.amazon\.com$", hostname) is not None
+
     def _get_pr(self):
-        response = self.codecommit_client.get_pr(self.pr_num)
+        response = self.codecommit_client.get_pr(self.repo_name, self.pr_num)
 
         if len(response.targets) == 0:
             raise ValueError(f"No files found in CodeCommit PR: {self.pr_num}")
 
-        # TODO: implement support for multiple commits in one CodeCommit PR
-        #       for now, we are only using the first commit in the PR
+        # TODO: implement support for multiple targets in one CodeCommit PR
+        #       for now, we are only using the first target in the PR
         if len(response.targets) > 1:
             logging.warning(
-                "Multiple commits in one PR is not supported for CodeCommit yet. Continuing, using the first commit only..."
+                "Multiple targets in one PR is not supported for CodeCommit yet. Continuing, using the first target only..."
             )
 
         # Return our object that mimics PullRequest class from the PyGithub library
@@ -306,13 +385,52 @@ class CodeCommitProvider(GitProvider):
         return ""  # not implemented yet
 
     @staticmethod
-    def _get_edit_type(codecommit_change_type):
+    def _add_additional_newlines(body: str) -> str:
+        """
+        Replace single newlines in a PR body with double newlines.
+
+        CodeCommit Markdown does not seem to render as well as GitHub Markdown,
+        so we add additional newlines to the PR body to make it more readable in CodeCommit.
+
+        Args:
+        - body: the PR body
+
+        Returns:
+        - str: the PR body with the double newlines added
+        """
+        return re.sub(r'(?<!\n)\n(?!\n)', '\n\n', body)
+
+    @staticmethod
+    def _remove_markdown_html(comment: str) -> str:
+        """
+        Remove the HTML tags from a PR comment.
+
+        CodeCommit Markdown does not seem to render as well as GitHub Markdown,
+        so we remove the HTML tags from the PR comment to make it more readable in CodeCommit.
+
+        Args:
+        - comment: the PR comment
+
+        Returns:
+        - str: the PR comment with the HTML tags removed
+        """
+        comment = comment.replace("<details>", "")
+        comment = comment.replace("</details>", "")
+        comment = comment.replace("<summary>", "")
+        comment = comment.replace("</summary>", "")
+        return comment
+
+    @staticmethod
+    def _get_edit_type(codecommit_change_type: str):
         """
         Convert the CodeCommit change type string to the EDIT_TYPE enum.
         The CodeCommit change type string is returned from the get_differences SDK method.
 
+        Args:
+        - codecommit_change_type: the CodeCommit change type string
+
         Returns:
-            An EDIT_TYPE enum representing the modified, added, deleted, or renamed file in the PR diff.
+        - An EDIT_TYPE enum representing the modified, added, deleted, or renamed file in the PR diff.
         """
         t = codecommit_change_type.upper()
         edit_type = None
@@ -333,6 +451,12 @@ class CodeCommitProvider(GitProvider):
         The returned extensions will include the dot "." prefix,
         to accommodate for the dots in the existing language_extension_map settings.
         Filenames with no extension will return an empty string for the extension.
+
+        Args:
+        - filenames: a list of filenames
+
+        Returns:
+        - list: A list of file extensions, including the dot "." prefix.
         """
         extensions = []
         for filename in filenames:
@@ -349,6 +473,12 @@ class CodeCommitProvider(GitProvider):
         Return a dictionary containing the programming language name (as the key),
         and the percentage that language is used (as the value),
         given a list of file extensions.
+
+        Args:
+        - extensions: a list of file extensions
+
+        Returns:
+        - dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR.
         """
         total_files = len(extensions)
         if total_files == 0:

pr_agent/git_providers/gerrit_provider.py

@@ -0,0 +1,403 @@
+import json
+import logging
+import os
+import pathlib
+import shutil
+import subprocess
+import uuid
+from collections import Counter, namedtuple
+from pathlib import Path
+from tempfile import mkdtemp, NamedTemporaryFile
+
+import requests
+import urllib3.util
+from git import Repo
+
+from pr_agent.config_loader import get_settings
+from pr_agent.git_providers.git_provider import GitProvider, FilePatchInfo, \
+    EDIT_TYPE
+from pr_agent.git_providers.local_git_provider import PullRequestMimic
+
+logger = logging.getLogger(__name__)
+
+
+def _call(*command, **kwargs) -> (int, str, str):
+    res = subprocess.run(
+        command,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+        check=True,
+        **kwargs,
+    )
+    return res.stdout.decode()
+
+
+def clone(url, directory):
+    logger.info("Cloning %s to %s", url, directory)
+    stdout = _call('git', 'clone', "--depth", "1", url, directory)
+    logger.info(stdout)
+
+
+def fetch(url, refspec, cwd):
+    logger.info("Fetching %s %s", url, refspec)
+    stdout = _call(
+        'git', 'fetch', '--depth', '2', url, refspec,
+        cwd=cwd
+    )
+    logger.info(stdout)
+
+
+def checkout(cwd):
+    logger.info("Checking out")
+    stdout = _call('git', 'checkout', "FETCH_HEAD", cwd=cwd)
+    logger.info(stdout)
+
+
+def show(*args, cwd=None):
+    logger.info("Show")
+    return _call('git', 'show', *args, cwd=cwd)
+
+
+def diff(*args, cwd=None):
+    logger.info("Diff")
+    patch = _call('git', 'diff', *args, cwd=cwd)
+    if not patch:
+        logger.warning("No changes found")
+        return
+    return patch
+
+
+def reset_local_changes(cwd):
+    logger.info("Reset local changes")
+    _call('git', 'checkout', "--force", cwd=cwd)
+
+
+def add_comment(url: urllib3.util.Url, refspec, message):
+    *_, patchset, changenum = refspec.rsplit("/")
+    message = "'" + message.replace("'", "'\"'\"'") + "'"
+    return _call(
+        "ssh",
+        "-p", str(url.port),
+        f"{url.auth}@{url.host}",
+        "gerrit", "review",
+        "--message", message,
+        # "--code-review", score,
+        f"{patchset},{changenum}",
+    )
+
+
+def list_comments(url: urllib3.util.Url, refspec):
+    *_, patchset, _ = refspec.rsplit("/")
+    stdout = _call(
+        "ssh",
+        "-p", str(url.port),
+        f"{url.auth}@{url.host}",
+        "gerrit", "query",
+        "--comments",
+        "--current-patch-set", patchset,
+        "--format", "JSON",
+    )
+    change_set, *_ = stdout.splitlines()
+    return json.loads(change_set)["currentPatchSet"]["comments"]
+
+
+def prepare_repo(url: urllib3.util.Url, project, refspec):
+    repo_url = (f"{url.scheme}://{url.auth}@{url.host}:{url.port}/{project}")
+
+    directory = pathlib.Path(mkdtemp())
+    clone(repo_url, directory),
+    fetch(repo_url, refspec, cwd=directory)
+    checkout(cwd=directory)
+    return directory
+
+
+def adopt_to_gerrit_message(message):
+    lines = message.splitlines()
+    buf = []
+    for line in lines:
+        # remove markdown formatting
+        line = (line.replace("*", "")
+                .replace("``", "`")
+                .replace("<details>", "")
+                .replace("</details>", "")
+                .replace("<summary>", "")
+                .replace("</summary>", ""))
+
+        line = line.strip()
+        if line.startswith('#'):
+            buf.append("\n" +
+                       line.replace('#', '').removesuffix(":").strip() +
+                       ":")
+            continue
+        elif line.startswith('-'):
+            buf.append(line.removeprefix('-').strip())
+            continue
+        else:
+            buf.append(line)
+    return "\n".join(buf).strip()
+
+
+def add_suggestion(src_filename, context: str, start, end: int):
+    with (
+        NamedTemporaryFile("w", delete=False) as tmp,
+        open(src_filename, "r") as src
+    ):
+        lines = src.readlines()
+        tmp.writelines(lines[:start - 1])
+        if context:
+            tmp.write(context)
+        tmp.writelines(lines[end:])
+
+    shutil.copy(tmp.name, src_filename)
+    os.remove(tmp.name)
+
+
+def upload_patch(patch, path):
+    patch_server_endpoint = get_settings().get(
+        'gerrit.patch_server_endpoint')
+    patch_server_token = get_settings().get(
+        'gerrit.patch_server_token')
+
+    response = requests.post(
+        patch_server_endpoint,
+        json={
+            "content": patch,
+            "path": path,
+        },
+        headers={
+            "Content-Type": "application/json",
+            "Authorization": f"Bearer {patch_server_token}",
+        }
+    )
+    response.raise_for_status()
+    patch_server_endpoint = patch_server_endpoint.rstrip("/")
+    return patch_server_endpoint + "/" + path
+
+
+class GerritProvider(GitProvider):
+
+    def __init__(self, key: str, incremental=False):
+        self.project, self.refspec = key.split(':')
+        assert self.project, "Project name is required"
+        assert self.refspec, "Refspec is required"
+        base_url = get_settings().get('gerrit.url')
+        assert base_url, "Gerrit URL is required"
+        user = get_settings().get('gerrit.user')
+        assert user, "Gerrit user is required"
+
+        parsed = urllib3.util.parse_url(base_url)
+        self.parsed_url = urllib3.util.parse_url(
+            f"{parsed.scheme}://{user}@{parsed.host}:{parsed.port}"
+        )
+
+        self.repo_path = prepare_repo(
+            self.parsed_url, self.project, self.refspec
+        )
+        self.repo = Repo(self.repo_path)
+        assert self.repo
+
+        self.pr = PullRequestMimic(self.get_pr_title(), self.get_diff_files())
+
+    def get_pr_title(self):
+        """
+        Substitutes the branch-name as the PR-mimic title.
+        """
+        return self.repo.branches[0].name
+
+    def get_issue_comments(self):
+        comments = list_comments(self.parsed_url, self.refspec)
+        Comments = namedtuple('Comments', ['reversed'])
+        Comment = namedtuple('Comment', ['body'])
+        return Comments([Comment(c['message']) for c in reversed(comments)])
+
+    def get_labels(self):
+        raise NotImplementedError(
+            'Getting labels is not implemented for the gerrit provider')
+
+    def add_eyes_reaction(self, issue_comment_id: int):
+        raise NotImplementedError(
+            'Adding reactions is not implemented for the gerrit provider')
+
+    def remove_reaction(self, issue_comment_id: int, reaction_id: int):
+        raise NotImplementedError(
+            'Removing reactions is not implemented for the gerrit provider')
+
+    def get_commit_messages(self):
+        return [self.repo.head.commit.message]
+
+    def get_repo_settings(self):
+        try:
+            with open(self.repo_path / ".pr_agent.toml", 'rb') as f:
+                contents = f.read()
+            return contents
+        except OSError:
+            return b""
+
+    def get_diff_files(self) -> list[FilePatchInfo]:
+        diffs = self.repo.head.commit.diff(
+            self.repo.head.commit.parents[0],  # previous commit
+            create_patch=True,
+            R=True
+        )
+
+        diff_files = []
+        for diff_item in diffs:
+            if diff_item.a_blob is not None:
+                original_file_content_str = (
+                    diff_item.a_blob.data_stream.read().decode('utf-8')
+                )
+            else:
+                original_file_content_str = ""  # empty file
+            if diff_item.b_blob is not None:
+                new_file_content_str = diff_item.b_blob.data_stream.read(). \
+                    decode('utf-8')
+            else:
+                new_file_content_str = ""  # empty file
+            edit_type = EDIT_TYPE.MODIFIED
+            if diff_item.new_file:
+                edit_type = EDIT_TYPE.ADDED
+            elif diff_item.deleted_file:
+                edit_type = EDIT_TYPE.DELETED
+            elif diff_item.renamed_file:
+                edit_type = EDIT_TYPE.RENAMED
+            diff_files.append(
+                FilePatchInfo(
+                    original_file_content_str,
+                    new_file_content_str,
+                    diff_item.diff.decode('utf-8'),
+                    diff_item.b_path,
+                    edit_type=edit_type,
+                    old_filename=None
+                    if diff_item.a_path == diff_item.b_path
+                    else diff_item.a_path
+                )
+            )
+        self.diff_files = diff_files
+        return diff_files
+
+    def get_files(self):
+        diff_index = self.repo.head.commit.diff(
+            self.repo.head.commit.parents[0],  # previous commit
+            R=True
+        )
+        # Get the list of changed files
+        diff_files = [item.a_path for item in diff_index]
+        return diff_files
+
+    def get_languages(self):
+        """
+        Calculate percentage of languages in repository. Used for hunk
+        prioritisation.
+        """
+        # Get all files in repository
+        filepaths = [Path(item.path) for item in
+                     self.repo.tree().traverse() if item.type == 'blob']
+        # Identify language by file extension and count
+        lang_count = Counter(
+            ext.lstrip('.') for filepath in filepaths for ext in
+            [filepath.suffix.lower()])
+        # Convert counts to percentages
+        total_files = len(filepaths)
+        lang_percentage = {lang: count / total_files * 100 for lang, count
+                           in lang_count.items()}
+        return lang_percentage
+
+    def get_pr_description_full(self):
+        return self.repo.head.commit.message
+
+    def get_user_id(self):
+        return self.repo.head.commit.author.email
+
+    def is_supported(self, capability: str) -> bool:
+        if capability in [
+            # 'get_issue_comments',
+            'create_inline_comment',
+            'publish_inline_comments',
+            'get_labels',
+            'gfm_markdown'
+        ]:
+            return False
+        return True
+
+    def split_suggestion(self, msg) -> tuple[str, str]:
+        is_code_context = False
+        description = []
+        context = []
+        for line in msg.splitlines():
+            if line.startswith('```suggestion'):
+                is_code_context = True
+                continue
+            if line.startswith('```'):
+                is_code_context = False
+                continue
+            if is_code_context:
+                context.append(line)
+            else:
+                description.append(
+                    line.replace('*', '')
+                )
+
+        return (
+            '\n'.join(description),
+            '\n'.join(context) + '\n' if context else ''
+        )
+
+    def publish_code_suggestions(self, code_suggestions: list):
+        msg = []
+        for suggestion in code_suggestions:
+            description, code = self.split_suggestion(suggestion['body'])
+            add_suggestion(
+                pathlib.Path(self.repo_path) / suggestion["relevant_file"],
+                code,
+                suggestion["relevant_lines_start"],
+                suggestion["relevant_lines_end"],
+            )
+            patch = diff(cwd=self.repo_path)
+            patch_id = uuid.uuid4().hex[0:4]
+            path = "/".join(["codium-ai", self.refspec, patch_id])
+            full_path = upload_patch(patch, path)
+            reset_local_changes(self.repo_path)
+            msg.append(f'* {description}\n{full_path}')
+
+        if msg:
+            add_comment(self.parsed_url, self.refspec, "\n".join(msg))
+            return True
+
+    def publish_comment(self, pr_comment: str, is_temporary: bool = False):
+        if not is_temporary:
+            msg = adopt_to_gerrit_message(pr_comment)
+            add_comment(self.parsed_url, self.refspec, msg)
+
+    def publish_description(self, pr_title: str, pr_body: str):
+        msg = adopt_to_gerrit_message(pr_body)
+        add_comment(self.parsed_url, self.refspec, pr_title + '\n' + msg)
+
+    def publish_inline_comments(self, comments: list[dict]):
+        raise NotImplementedError(
+            'Publishing inline comments is not implemented for the gerrit '
+            'provider')
+
+    def publish_inline_comment(self, body: str, relevant_file: str,
+                               relevant_line_in_file: str):
+        raise NotImplementedError(
+            'Publishing inline comments is not implemented for the gerrit '
+            'provider')
+
+    def create_inline_comment(self, body: str, relevant_file: str,
+                              relevant_line_in_file: str):
+        raise NotImplementedError(
+            'Creating inline comments is not implemented for the gerrit '
+            'provider')
+
+    def publish_labels(self, labels):
+        # Not applicable to the local git provider,
+        # but required by the interface
+        pass
+
+    def remove_initial_comment(self):
+        # remove repo, cloned in previous steps
+        # shutil.rmtree(self.repo_path)
+        pass
+
+    def get_pr_branch(self):
+        return self.repo.head

pr_agent/git_providers/git_provider.py

@@ -1,3 +1,4 @@
+import logging
 from abc import ABC, abstractmethod
 from dataclasses import dataclass
 
@@ -85,11 +86,11 @@ class GitProvider(ABC):
     def get_pr_description_full(self) -> str:
         pass
 
-    def get_pr_description(self) -> str:
+    def get_pr_description(self, *, full: bool = True) -> str:
         from pr_agent.config_loader import get_settings
         from pr_agent.algo.pr_processing import clip_tokens
         max_tokens = get_settings().get("CONFIG.MAX_DESCRIPTION_TOKENS", None)
-        description = self.get_pr_description_full()
+        description = self.get_pr_description_full() if full else self.get_user_description()
         if max_tokens:
             return clip_tokens(description, max_tokens)
         return description
@@ -126,17 +127,26 @@ class GitProvider(ABC):
     def get_commit_messages(self):
         pass
 
+    def get_pr_id(self):
+        return ""
+
 def get_main_pr_language(languages, files) -> str:
     """
     Get the main language of the commit. Return an empty string if cannot determine.
     """
     main_language_str = ""
+    if not languages:
+        logging.info("No languages detected")
+        return main_language_str
+
     try:
         top_language = max(languages, key=languages.get).lower()
 
         # validate that the specific commit uses the main language
         extension_list = []
         for file in files:
+            if isinstance(file, str):
+                file = FilePatchInfo(base_file=None, head_file=None, patch=None, filename=file)
             extension_list.append(file.filename.rsplit('.')[-1])
 
         # get the most common extension
@@ -158,10 +168,11 @@ def get_main_pr_language(languages, files) -> str:
                 most_common_extension == 'scala' and top_language == 'scala' or \
                 most_common_extension == 'kt' and top_language == 'kotlin' or \
                 most_common_extension == 'pl' and top_language == 'perl' or \
-                most_common_extension == 'swift' and top_language == 'swift':
+                most_common_extension == top_language:
             main_language_str = top_language
 
-    except Exception:
+    except Exception as e:
+        logging.exception(e)
         pass
 
     return main_language_str

pr_agent/git_providers/github_provider.py

@@ -32,7 +32,7 @@ class GithubProvider(GitProvider):
         self.diff_files = None
         self.git_files = None
         self.incremental = incremental
-        if pr_url:
+        if pr_url and 'pull' in pr_url:
             self.set_pr(pr_url)
             self.last_commit_id = list(self.pr.get_commits())[-1]
 
@@ -239,9 +239,10 @@ class GithubProvider(GitProvider):
     def get_user_id(self):
         if not self.github_user_id:
             try:
-                self.github_user_id = self.github_client.get_user().login
+                self.github_user_id = self.github_client.get_user().raw_data['login']
             except Exception as e:
-                logging.exception(f"Failed to get user id, error: {e}")
+                self.github_user_id = ""
+                # logging.exception(f"Failed to get user id, error: {e}")
         return self.github_user_id
 
     def get_notifications(self, since: datetime):
@@ -309,6 +310,35 @@ class GithubProvider(GitProvider):
 
         return repo_name, pr_number
 
+    @staticmethod
+    def _parse_issue_url(issue_url: str) -> Tuple[str, int]:
+        parsed_url = urlparse(issue_url)
+
+        if 'github.com' not in parsed_url.netloc:
+            raise ValueError("The provided URL is not a valid GitHub URL")
+
+        path_parts = parsed_url.path.strip('/').split('/')
+        if 'api.github.com' in parsed_url.netloc:
+            if len(path_parts) < 5 or path_parts[3] != 'issues':
+                raise ValueError("The provided URL does not appear to be a GitHub ISSUE URL")
+            repo_name = '/'.join(path_parts[1:3])
+            try:
+                issue_number = int(path_parts[4])
+            except ValueError as e:
+                raise ValueError("Unable to convert issue number to integer") from e
+            return repo_name, issue_number
+
+        if len(path_parts) < 4 or path_parts[2] != 'issues':
+            raise ValueError("The provided URL does not appear to be a GitHub PR issue")
+
+        repo_name = '/'.join(path_parts[:2])
+        try:
+            issue_number = int(path_parts[3])
+        except ValueError as e:
+            raise ValueError("Unable to convert issue number to integer") from e
+
+        return repo_name, issue_number
+
     def _get_github_client(self):
         deployment_type = get_settings().get("GITHUB.DEPLOYMENT_TYPE", "user")
 
@@ -417,3 +447,10 @@ class GithubProvider(GitProvider):
                 logging.info(f"Failed adding line link, error: {e}")
 
         return ""
+
+    def get_pr_id(self):
+        try:
+            pr_id = f"{self.repo}/{self.pr_num}"
+            return pr_id
+        except:
+            return ""

pr_agent/git_providers/gitlab_provider.py

@@ -43,7 +43,7 @@ class GitLabProvider(GitProvider):
         self.incremental = incremental
 
     def is_supported(self, capability: str) -> bool:
-        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments']:
+        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'gfm_markdown']:
             return False
         return True
 
@@ -379,3 +379,10 @@ class GitLabProvider(GitProvider):
         if max_tokens:
             commit_messages_str = clip_tokens(commit_messages_str, max_tokens)
         return commit_messages_str
+
+    def get_pr_id(self):
+        try:
+            pr_id = self.mr.web_url
+            return pr_id
+        except:
+            return ""

pr_agent/git_providers/local_git_provider.py

@@ -56,7 +56,8 @@ class LocalGitProvider(GitProvider):
             raise KeyError(f'Branch: {self.target_branch_name} does not exist')
 
     def is_supported(self, capability: str) -> bool:
-        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels']:
+        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels',
+                          'gfm_markdown']:
             return False
         return True
 

pr_agent/servers/bitbucket_pipeline_runner.py

@@ -0,0 +1,34 @@
+import os
+from pr_agent.agent.pr_agent import PRAgent
+from pr_agent.config_loader import get_settings
+from pr_agent.tools.pr_reviewer import PRReviewer
+import asyncio
+
+async def run_action():
+    try:
+        pull_request_id = os.environ.get("BITBUCKET_PR_ID", '')
+        slug = os.environ.get("BITBUCKET_REPO_SLUG", '')
+        workspace = os.environ.get("BITBUCKET_WORKSPACE", '')
+        bearer_token = os.environ.get('BITBUCKET_BEARER_TOKEN', None)
+        OPENAI_KEY = os.environ.get('OPENAI_API_KEY') or os.environ.get('OPENAI.KEY')
+        OPENAI_ORG = os.environ.get('OPENAI_ORG') or os.environ.get('OPENAI.ORG')
+        # Check if required environment variables are set
+        if not bearer_token:
+            print("BITBUCKET_BEARER_TOKEN not set")
+            return
+        
+        if not OPENAI_KEY:
+            print("OPENAI_KEY not set")
+            return
+        # Set the environment variables in the settings
+        get_settings().set("BITBUCKET.BEARER_TOKEN", bearer_token)
+        get_settings().set("OPENAI.KEY", OPENAI_KEY)
+        if OPENAI_ORG:
+            get_settings().set("OPENAI.ORG", OPENAI_ORG)
+        if pull_request_id and slug and workspace:
+            pr_url = f"https://bitbucket.org/{workspace}/{slug}/pull-requests/{pull_request_id}"
+            await PRReviewer(pr_url).run()
+    except Exception as e:
+        print(f"An error occurred: {e}")
+if __name__ == "__main__":
+    asyncio.run(run_action())

pr_agent/servers/gerrit_server.py

@@ -0,0 +1,78 @@
+import copy
+import logging
+import sys
+from enum import Enum
+from json import JSONDecodeError
+
+import uvicorn
+from fastapi import APIRouter, FastAPI, HTTPException
+from pydantic import BaseModel
+from starlette.middleware import Middleware
+from starlette_context import context
+from starlette_context.middleware import RawContextMiddleware
+
+from pr_agent.agent.pr_agent import PRAgent
+from pr_agent.config_loader import global_settings, get_settings
+
+logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
+router = APIRouter()
+
+
+class Action(str, Enum):
+    review = "review"
+    describe = "describe"
+    ask = "ask"
+    improve = "improve"
+    reflect = "reflect"
+    answer = "answer"
+
+
+class Item(BaseModel):
+    refspec: str
+    project: str
+    msg: str
+
+
+@router.post("/api/v1/gerrit/{action}")
+async def handle_gerrit_request(action: Action, item: Item):
+    logging.debug("Received a Gerrit request")
+    context["settings"] = copy.deepcopy(global_settings)
+
+    if action == Action.ask:
+        if not item.msg:
+            return HTTPException(
+                status_code=400,
+                detail="msg is required for ask command"
+            )
+    await PRAgent().handle_request(
+        f"{item.project}:{item.refspec}",
+        f"/{item.msg.strip()}"
+    )
+
+
+async def get_body(request):
+    try:
+        body = await request.json()
+    except JSONDecodeError as e:
+        logging.error("Error parsing request body", e)
+        return {}
+    return body
+
+
+@router.get("/")
+async def root():
+    return {"status": "ok"}
+
+
+def start():
+    # to prevent adding help messages with the output
+    get_settings().set("CONFIG.CLI_MODE", True)
+    middleware = [Middleware(RawContextMiddleware)]
+    app = FastAPI(middleware=middleware)
+    app.include_router(router)
+
+    uvicorn.run(app, host="0.0.0.0", port=3000)
+
+
+if __name__ == '__main__':
+    start()

pr_agent/servers/github_action_runner.py

@@ -5,6 +5,8 @@ import os
 from pr_agent.agent.pr_agent import PRAgent
 from pr_agent.config_loader import get_settings
 from pr_agent.git_providers import get_git_provider
+from pr_agent.tools.pr_code_suggestions import PRCodeSuggestions
+from pr_agent.tools.pr_description import PRDescription
 from pr_agent.tools.pr_reviewer import PRReviewer
 
 
@@ -12,8 +14,8 @@ async def run_action():
     # Get environment variables
     GITHUB_EVENT_NAME = os.environ.get('GITHUB_EVENT_NAME')
     GITHUB_EVENT_PATH = os.environ.get('GITHUB_EVENT_PATH')
-    OPENAI_KEY = os.environ.get('OPENAI_KEY')
-    OPENAI_ORG = os.environ.get('OPENAI_ORG')
+    OPENAI_KEY = os.environ.get('OPENAI_KEY') or os.environ.get('OPENAI.KEY')
+    OPENAI_ORG = os.environ.get('OPENAI_ORG') or os.environ.get('OPENAI.ORG')
     GITHUB_TOKEN = os.environ.get('GITHUB_TOKEN')
     get_settings().set("CONFIG.PUBLISH_OUTPUT_PROGRESS", False)
 
@@ -53,7 +55,15 @@ async def run_action():
         if action in ["opened", "reopened"]:
             pr_url = event_payload.get("pull_request", {}).get("url")
             if pr_url:
-                await PRReviewer(pr_url).run()
+                auto_review = os.environ.get('github_action.auto_review', None)
+                if auto_review is None or (isinstance(auto_review, str) and auto_review.lower() == 'true'):
+                    await PRReviewer(pr_url).run()
+                auto_describe = os.environ.get('github_action.auto_describe', None)
+                if isinstance(auto_describe, str) and auto_describe.lower() == 'true':
+                    await PRDescription(pr_url).run()
+                auto_improve = os.environ.get('github_action.auto_improve', None)
+                if isinstance(auto_improve, str) and auto_improve.lower() == 'true':
+                    await PRCodeSuggestions(pr_url).run()
 
     # Handle issue comment event
     elif GITHUB_EVENT_NAME == "issue_comment":
@@ -61,12 +71,21 @@ async def run_action():
         if action in ["created", "edited"]:
             comment_body = event_payload.get("comment", {}).get("body")
             if comment_body:
-                pr_url = event_payload.get("issue", {}).get("pull_request", {}).get("url")
-                if pr_url:
+                is_pr = False
+                # check if issue is pull request
+                if event_payload.get("issue", {}).get("pull_request"):
+                    url = event_payload.get("issue", {}).get("pull_request", {}).get("url")
+                    is_pr = True
+                else:
+                    url = event_payload.get("issue", {}).get("url")
+                if url:
                     body = comment_body.strip().lower()
                     comment_id = event_payload.get("comment", {}).get("id")
-                    provider = get_git_provider()(pr_url=pr_url)
-                    await PRAgent().handle_request(pr_url, body, notify=lambda: provider.add_eyes_reaction(comment_id))
+                    provider = get_git_provider()(pr_url=url)
+                    if is_pr:
+                        await PRAgent().handle_request(url, body, notify=lambda: provider.add_eyes_reaction(comment_id))
+                    else:
+                        await PRAgent().handle_request(url, body)
 
 
 if __name__ == '__main__':

pr_agent/servers/github_app.py

@@ -12,6 +12,7 @@ from starlette_context import context
 from starlette_context.middleware import RawContextMiddleware
 
 from pr_agent.agent.pr_agent import PRAgent
+from pr_agent.algo.utils import update_settings_from_args
 from pr_agent.config_loader import get_settings, global_settings
 from pr_agent.git_providers import get_git_provider
 from pr_agent.servers.utils import verify_signature
@@ -97,6 +98,7 @@ async def handle_request(body: Dict[str, Any], event: str):
             api_url = body["comment"]["pull_request_url"]
         else:
             return {}
+        logging.info(body)
         logging.info(f"Handling comment because of event={event} and action={action}")
         comment_id = body.get("comment", {}).get("id")
         provider = get_git_provider()(pr_url=api_url)
@@ -123,8 +125,14 @@ async def handle_request(body: Dict[str, Any], event: str):
                     return {}
             logging.info(f"Performing review because of event={event} and action={action}")
             for command in get_settings().github_app.pr_commands:
-                logging.info(f"Performing command: {command}")
-                await agent.handle_request(api_url, command)
+                split_command = command.split(" ")
+                command = split_command[0]
+                args = split_command[1:]
+                other_args = update_settings_from_args(args)
+                new_command = ' '.join([command] + other_args)
+                logging.info(body)
+                logging.info(f"Performing command: {new_command}")
+                await agent.handle_request(api_url, new_command)
 
     logging.info("event or action does not require handling")
     return {}

pr_agent/servers/gitlab_webhook.py

@@ -1,21 +1,51 @@
+import copy
+import json
 import logging
+import sys
 
 import uvicorn
 from fastapi import APIRouter, FastAPI, Request, status
 from fastapi.encoders import jsonable_encoder
 from fastapi.responses import JSONResponse
 from starlette.background import BackgroundTasks
+from starlette.middleware import Middleware
+from starlette_context import context
+from starlette_context.middleware import RawContextMiddleware
 
 from pr_agent.agent.pr_agent import PRAgent
-from pr_agent.config_loader import get_settings
+from pr_agent.config_loader import get_settings, global_settings
+from pr_agent.secret_providers import get_secret_provider
 
-app = FastAPI()
+logging.basicConfig(stream=sys.stdout, level=logging.INFO)
 router = APIRouter()
 
+secret_provider = get_secret_provider() if get_settings().get("CONFIG.SECRET_PROVIDER") else None
+
 
 @router.post("/webhook")
 async def gitlab_webhook(background_tasks: BackgroundTasks, request: Request):
+    if request.headers.get("X-Gitlab-Token") and secret_provider:
+        request_token = request.headers.get("X-Gitlab-Token")
+        secret = secret_provider.get_secret(request_token)
+        try:
+            secret_dict = json.loads(secret)
+            gitlab_token = secret_dict["gitlab_token"]
+            context["settings"] = copy.deepcopy(global_settings)
+            context["settings"].gitlab.personal_access_token = gitlab_token
+        except Exception as e:
+            logging.error(f"Failed to validate secret {request_token}: {e}")
+            return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
+    elif get_settings().get("GITLAB.SHARED_SECRET"):
+        secret = get_settings().get("GITLAB.SHARED_SECRET")
+        if not request.headers.get("X-Gitlab-Token") == secret:
+            return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
+    else:
+        return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
+    gitlab_token = get_settings().get("GITLAB.PERSONAL_ACCESS_TOKEN", None)
+    if not gitlab_token:
+        return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
     data = await request.json()
+    logging.info(json.dumps(data))
     if data.get('object_kind') == 'merge_request' and data['object_attributes'].get('action') in ['open', 'reopen']:
         logging.info(f"A merge request has been opened: {data['object_attributes'].get('title')}")
         url = data['object_attributes'].get('url')
@@ -28,16 +58,18 @@ async def gitlab_webhook(background_tasks: BackgroundTasks, request: Request):
             background_tasks.add_task(PRAgent().handle_request, url, body)
     return JSONResponse(status_code=status.HTTP_200_OK, content=jsonable_encoder({"message": "success"}))
 
+
+@router.get("/")
+async def root():
+    return {"status": "ok"}
+
 def start():
     gitlab_url = get_settings().get("GITLAB.URL", None)
     if not gitlab_url:
         raise ValueError("GITLAB.URL is not set")
-    gitlab_token = get_settings().get("GITLAB.PERSONAL_ACCESS_TOKEN", None)
-    if not gitlab_token:
-        raise ValueError("GITLAB.PERSONAL_ACCESS_TOKEN is not set")
     get_settings().config.git_provider = "gitlab"
-
-    app = FastAPI()
+    middleware = [Middleware(RawContextMiddleware)]
+    app = FastAPI(middleware=middleware)
     app.include_router(router)
 
     uvicorn.run(app, host="0.0.0.0", port=3000)

pr_agent/settings/.secrets_template.toml

@@ -16,6 +16,10 @@ key = ""  # Acquire through https://platform.openai.com
 #deployment_id = ""  # The deployment name you chose when you deployed the engine
 #fallback_deployments = []  # For each fallback model specified in configuration.toml in the [config] section, specify the appropriate deployment_id
 
+[pinecone]
+api_key = "..."
+environment = "gcp-starter"
+
 [anthropic]
 key = "" # Optional, uncomment if you want to use Anthropic. Acquire through https://www.anthropic.com/
 
@@ -24,6 +28,14 @@ key = "" # Optional, uncomment if you want to use Cohere. Acquire through https:
 
 [replicate]
 key = "" # Optional, uncomment if you want to use Replicate. Acquire through https://replicate.com/
+
+[huggingface]
+key = "" # Optional, uncomment if you want to use Huggingface Inference API. Acquire through https://huggingface.co/docs/api-inference/quicktour
+api_base = "" # the base url for your huggingface inference endpoint 
+
+[ollama]
+api_base = "" # the base url for your huggingface inference endpoint 
+
 [github]
 # ---- Set the following only for deployment type == "user"
 user_token = ""  # A GitHub personal access token with 'repo' scope.
@@ -43,5 +55,12 @@ webhook_secret = "<WEBHOOK SECRET>"  # Optional, may be commented out.
 personal_access_token = ""
 
 [bitbucket]
-# Bitbucket personal bearer token
+# For Bitbucket personal/repository bearer token
 bearer_token = ""
+
+# For Bitbucket app
+app_key = ""
+base_url = ""
+
+[litellm]
+LITELLM_TOKEN = "" # see https://docs.litellm.ai/docs/debugging/hosted_debugging for details and instructions on how to get a token

pr_agent/settings/configuration.toml

@@ -10,14 +10,15 @@ use_repo_settings_file=true
 ai_timeout=180
 max_description_tokens = 500
 max_commits_tokens = 500
-litellm_debugger=false
 secret_provider="google_cloud_storage"
+cli_mode=false
 
 [pr_reviewer] # /review #
 require_focused_review=false
 require_score_review=false
 require_tests_review=true
 require_security_review=true
+require_estimate_effort_to_review=true
 num_code_suggestions=4
 inline_code_comments = false
 ask_and_reflect=false
@@ -25,10 +26,14 @@ automatic_review=true
 extra_instructions = ""
 
 [pr_description] # /describe #
+publish_labels=true
 publish_description_as_comment=false
 add_original_user_description=false
 keep_original_user_title=false
 extra_instructions = ""
+# markers
+use_description_markers=false
+include_generated_by_header=true
 
 [pr_questions] # /ask #
 
@@ -53,6 +58,11 @@ extra_instructions = ""
 deployment_type = "user"
 ratelimit_retries = 5
 
+[github_action]
+# auto_review = true    # set as env var in .github/workflows/pr-agent.yaml
+# auto_describe = true  # set as env var in .github/workflows/pr-agent.yaml
+# auto_improve = true   # set as env var in .github/workflows/pr-agent.yaml
+
 [github_app]
 # these toggles allows running the github app from custom deployments
 bot_user = "github-actions[bot]"
@@ -85,3 +95,26 @@ polling_interval_seconds = 30
 # LocalGitProvider settings - uncomment to use paths other than default
 # description_path= "path/to/description.md"
 # review_path= "path/to/review.md"
+
+[gerrit]
+# endpoint to the gerrit service
+# url = "ssh://gerrit.example.com:29418"
+# user for gerrit authentication
+# user = "ai-reviewer"
+# patch server where patches will be saved
+# patch_server_endpoint = "http://127.0.0.1:5000/patch"
+# token to authenticate in the patch server
+# patch_server_token = ""
+
+[litellm]
+#use_client = false
+
+[pr_similar_issue]
+skip_comments = false
+force_update_dataset = false
+max_issues_to_scan = 500
+
+[pinecone]
+# fill and place in .secrets.toml
+#api_key = ...
+# environment = "gcp-starter"

pr_agent/settings/pr_code_suggestions_prompts.toml

@@ -1,8 +1,8 @@
 [pr_code_suggestions_prompt]
-system="""You are a language model called PR-Code-Reviewer.
-Your task is to provide meaningful actionable code suggestions, to improve the new code presented in a PR.
+system="""You are a language model called PR-Code-Reviewer, that specializes in suggesting code improvements for Pull Request (PR).
+Your task is to provide meaningful and actionable code suggestions, to improve the new code presented in a PR.
 
-Example PR Diff input:
+Example for a PR Diff input:
 '
 ## src/file1.py
 
@@ -10,8 +10,8 @@ Example PR Diff input:
 __new hunk__
 12  code line that already existed in the file...
 13  code line that already existed in the file....
-14 +new code line added in the PR
-15  code line that already existed in the file...
+14 +new code line1 added in the PR
+15 +new code line2 added in the PR
 16  code line that already existed in the file...
 __old hunk__
  code line that already existed in the file...
@@ -31,13 +31,17 @@ __old hunk__
 '
 
 Specific instructions:
-- Focus on important suggestions like fixing code problems, issues and bugs. As a second priority, provide suggestions for meaningful code improvements, like performance, vulnerability, modularity, and best practices.
-- Suggestions should refer only to code from the '__new hunk__' sections, and focus on new lines of code (lines starting with '+').
-- Provide the exact line number range (inclusive) for each issue.
-- Assume there is additional relevant code, that is not included in the diff.
 - Provide up to {{ num_code_suggestions }} code suggestions.
-- Avoid making suggestions that have already been implemented in the PR code. For example, if you want to add logs, or change a variable to const, or anything else, make sure it isn't already in the '__new hunk__' code.
-- Don't suggest to add docstring or type hints.
+- Prioritize suggestions that address major problems, issues and bugs in the code.
+  As a second priority, suggestions should focus on best practices, code readability, maintainability, enhancments, performance, and other aspects.
+  Don't suggest to add docstring, type hints, or comments.
+  Try to provide diverse and insightful suggestions.
+- Suggestions should refer only to code from the '__new hunk__' sections, and focus on new lines of code (lines starting with '+').
+  Avoid making suggestions that have already been implemented in the PR code. For example, if you want to add logs, or change a variable to const, or anything else, make sure it isn't already in the '__new hunk__' code.
+  For each suggestion, make sure to take into consideration also the context, meaning the lines before and after the relevant code.
+- Provide the exact line numbers range (inclusive) for each issue.
+- Assume there is additional relevant code, that is not included in the diff.
+
 
 {%- if extra_instructions %}
 
@@ -45,63 +49,82 @@ Extra instructions from the user:
 {{ extra_instructions }}
 {%- endif %}
 
-You must use the following JSON schema to format your answer:
-```json
-{
-    "Code suggestions": {
-      "type": "array",
-      "minItems": 1,
-      "maxItems": {{ num_code_suggestions }},
-      "uniqueItems": "true",
-      "items": {
-        "relevant file": {
-          "type": "string",
-          "description": "the relevant file full path"
-        },
-        "suggestion content": {
-          "type": "string",
-          "description": "a concrete suggestion for meaningfully improving the new PR code (lines from the '__new hunk__' sections, starting with '+')."
-        },
-        "existing code": {
-          "type": "string",
-          "description": "a code snippet showing the relevant code lines from a '__new hunk__' section. It must be continuous, correctly formatted and indented, and without line numbers."
-        },
-        "relevant lines": {
-          "type": "string",
-          "description": "the relevant lines from a '__new hunk__' section, in the format of 'start_line-end_line'. For example: '10-15'. They should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above."
-        },
-        "improved code": {
-          "type": "string",
-          "description": "a new code snippet that can be used to replace the relevant lines in '__new hunk__' code. Replacement suggestions should be complete, correctly formatted and indented, and without line numbers."
-        }
-      }
-    }
-}
+You must use the following YAML schema to format your answer:
+```yaml
+Code suggestions:
+  type: array
+  minItems: 1
+  maxItems: {{ num_code_suggestions }}
+  uniqueItems: true
+  items:
+    relevant file:
+      type: string
+      description: the relevant file full path
+    suggestion content:
+      type: string
+      description: |-
+        a concrete suggestion for meaningfully improving the new PR code.
+    existing code:
+      type: string
+      description: |-
+        a code snippet showing the relevant code lines from a '__new hunk__' section.
+        It must be contiguous, correctly formatted and indented, and without line numbers.
+    relevant lines start:
+      type: integer
+      description: |-
+        The relevant line number from a '__new hunk__' section where the suggestion starts (inclusive).
+        Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above.
+    relevant lines end:
+      type: integer
+      description: |-
+        The relevant line number from a '__new hunk__' section where the suggestion ends (inclusive).
+        Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above.
+    improved code:
+      type: string
+      description: |-
+        a new code snippet that can be used to replace the relevant lines in '__new hunk__' code.
+        Replacement suggestions should be complete, correctly formatted and indented, and without line numbers.
 ```
 
-Don't output line numbers in the 'improved code' snippets.
+Example output:
+```yaml
+Code suggestions:
+  - relevant file: |-
+        src/file1.py
+    suggestion content: |-
+        Add a docstring to func1()
+    existing code: |-
+        def func1():
+    relevant lines start: 12
+    relevant lines end: 12
+    improved code: |-
+        ...
+```
+
+
+Each YAML output MUST be after a newline, indented, with block scalar indicator ('|-').
 Don't repeat the prompt in the answer, and avoid outputting the 'type' and 'description' fields.
 """
 
 user="""PR Info:
-Title: '{{title}}'
-Branch: '{{branch}}'
-Description: '{{description}}'
-{%- if language %}
-Main language: {{language}}
-{%- endif %}
-{%- if commit_messages_str %}
 
-Commit messages:
-{{commit_messages_str}}
+Title: '{{title}}'
+
+Branch: '{{branch}}'
+
+Description: '{{description}}'
+
+{%- if language %}
+
+Main language: {{language}}
 {%- endif %}
 
 
 The PR Diff:
 ```
-{{diff}}
+{{- diff|trim }}
 ```
 
-Response (should be a valid JSON, and nothing else):
-```json
+Response (should be a valid YAML, and nothing else):
+```yaml
 """

pr_agent/settings/pr_reviewer_prompts.toml

@@ -22,13 +22,13 @@ code line that already existed in the file....
 ...
 '
 
-Thre review should focus on new code added in the PR (lines starting with '+'), and not on code that already existed in the file (lines starting with '-', or without prefix).
+The review should focus on new code added in the PR (lines starting with '+'), and not on code that already existed in the file (lines starting with '-', or without prefix).
 
 {%- if num_code_suggestions > 0 %}
 - Provide up to {{ num_code_suggestions }} code suggestions.
 - Focus on important suggestions like fixing code problems, issues and bugs. As a second priority, provide suggestions for meaningful code improvements, like performance, vulnerability, modularity, and best practices.
 - Avoid making suggestions that have already been implemented in the PR code. For example, if you want to add logs, or change a variable to const, or anything else, make sure it isn't already in the PR code.
-- Don't suggest to add docstring or type hints.
+- Don't suggest to add docstring, type hints, or comments.
 - Suggestions should focus on improving the new code added in the PR (lines starting with '+')
 {%- endif %}
 
@@ -85,6 +85,14 @@ PR Analysis:
       code diff changes are too scattered, then the PR is not focused. Explain
       your answer shortly.
 {%- endif %}
+{%- if require_estimate_effort_to_review %}
+  Estimated effort to review [1-5]:
+    type: string
+    description: >-
+      Estimate, on a scale of 1-5 (inclusive), the time and effort required to review this PR by an experienced and knowledgeable developer. 1 means short and easy review , 5 means long and hard review.
+      Take into account the size, complexity, quality, and the needed changes of the PR code diff.
+      Explain your answer shortly (1-2 sentences).
+{%- endif %}
 PR Feedback:
   General suggestions:
     type: string

pr_agent/tools/pr_code_suggestions.py

@@ -1,16 +1,13 @@
 import copy
-import json
 import logging
 import textwrap
-from typing import List
-
-import yaml
+from typing import List, Dict
 from jinja2 import Environment, StrictUndefined
 
 from pr_agent.algo.ai_handler import AiHandler
 from pr_agent.algo.pr_processing import get_pr_diff, retry_with_fallback_models, get_pr_multi_diffs
 from pr_agent.algo.token_handler import TokenHandler
-from pr_agent.algo.utils import try_fix_json
+from pr_agent.algo.utils import load_yaml
 from pr_agent.config_loader import get_settings
 from pr_agent.git_providers import BitbucketProvider, get_git_provider
 from pr_agent.git_providers.git_provider import get_main_pr_language
@@ -25,7 +22,10 @@ class PRCodeSuggestions:
         )
 
         # extended mode
-        self.is_extended = any(["extended" in arg for arg in args])
+        try:
+            self.is_extended = any(["extended" in arg for arg in args])
+        except:
+            self.is_extended = False
         if self.is_extended:
             num_code_suggestions = get_settings().pr_code_suggestions.num_code_suggestions_per_chunk
         else:
@@ -51,27 +51,33 @@ class PRCodeSuggestions:
                                           get_settings().pr_code_suggestions_prompt.user)
 
     async def run(self):
-        logging.info('Generating code suggestions for PR...')
-        if get_settings().config.publish_output:
-            self.git_provider.publish_comment("Preparing review...", is_temporary=True)
+        try:
+            logging.info('Generating code suggestions for PR...')
+            if get_settings().config.publish_output:
+                self.git_provider.publish_comment("Preparing review...", is_temporary=True)
 
-        logging.info('Preparing PR review...')
-        if not self.is_extended:
-            await retry_with_fallback_models(self._prepare_prediction)
-            data = self._prepare_pr_code_suggestions()
-        else:
-            data = await retry_with_fallback_models(self._prepare_prediction_extended)
+            logging.info('Preparing PR review...')
+            if not self.is_extended:
+                await retry_with_fallback_models(self._prepare_prediction)
+                data = self._prepare_pr_code_suggestions()
+            else:
+                data = await retry_with_fallback_models(self._prepare_prediction_extended)
+            if (not data) or (not 'Code suggestions' in data):
+                logging.info('No code suggestions found for PR.')
+                return
 
-        if (not self.is_extended and get_settings().pr_code_suggestions.rank_suggestions) or \
-                (self.is_extended and get_settings().pr_code_suggestions.rank_extended_suggestions):
-            logging.info('Ranking Suggestions...')
-            data['Code suggestions'] = await self.rank_suggestions(data['Code suggestions'])
+            if (not self.is_extended and get_settings().pr_code_suggestions.rank_suggestions) or \
+                    (self.is_extended and get_settings().pr_code_suggestions.rank_extended_suggestions):
+                logging.info('Ranking Suggestions...')
+                data['Code suggestions'] = await self.rank_suggestions(data['Code suggestions'])
 
-        if get_settings().config.publish_output:
-            logging.info('Pushing PR review...')
-            self.git_provider.remove_initial_comment()
-            logging.info('Pushing inline code suggestions...')
-            self.push_inline_code_suggestions(data)
+            if get_settings().config.publish_output:
+                logging.info('Pushing PR review...')
+                self.git_provider.remove_initial_comment()
+                logging.info('Pushing inline code suggestions...')
+                self.push_inline_code_suggestions(data)
+        except Exception as e:
+            logging.error(f"Failed to generate code suggestions for PR, error: {e}")
 
     async def _prepare_prediction(self, model: str):
         logging.info('Getting PR diff...')
@@ -98,14 +104,11 @@ class PRCodeSuggestions:
 
         return response
 
-    def _prepare_pr_code_suggestions(self) -> str:
+    def _prepare_pr_code_suggestions(self) -> Dict:
         review = self.prediction.strip()
-        try:
-            data = json.loads(review)
-        except json.decoder.JSONDecodeError:
-            if get_settings().config.verbosity_level >= 2:
-                logging.info(f"Could not parse json response: {review}")
-            data = try_fix_json(review, code_suggestions=True)
+        data = load_yaml(review)
+        if isinstance(data, list):
+            data = {'Code suggestions': data}
         return data
 
     def push_inline_code_suggestions(self, data):
@@ -119,11 +122,8 @@ class PRCodeSuggestions:
                 if get_settings().config.verbosity_level >= 2:
                     logging.info(f"suggestion: {d}")
                 relevant_file = d['relevant file'].strip()
-                relevant_lines_str = d['relevant lines'].strip()
-                if ',' in relevant_lines_str:  # handling 'relevant lines': '181, 190' or '178-184, 188-194'
-                    relevant_lines_str = relevant_lines_str.split(',')[0]
-                relevant_lines_start = int(relevant_lines_str.split('-')[0])  # absolute position
-                relevant_lines_end = int(relevant_lines_str.split('-')[-1])
+                relevant_lines_start = int(d['relevant lines start'])  # absolute position
+                relevant_lines_end = int(d['relevant lines end'])
                 content = d['suggestion content']
                 new_code_snippet = d['improved code']
 
@@ -227,7 +227,7 @@ class PRCodeSuggestions:
             response, finish_reason = await self.ai_handler.chat_completion(model=model, system=system_prompt,
                                                                             user=user_prompt)
 
-            sort_order = yaml.safe_load(response)
+            sort_order = load_yaml(response)
             for s in sort_order['Sort Order']:
                 suggestion_number = s['suggestion number']
                 importance_order = s['importance order']

pr_agent/tools/pr_description.py

@@ -1,5 +1,6 @@
 import copy
 import json
+import re
 import logging
 from typing import List, Tuple
 
@@ -28,6 +29,7 @@ class PRDescription:
         self.main_pr_language = get_main_pr_language(
             self.git_provider.get_languages(), self.git_provider.get_files()
         )
+        self.pr_id = self.git_provider.get_pr_id()
 
         # Initialize the AI handler
         self.ai_handler = AiHandler()
@@ -36,7 +38,7 @@ class PRDescription:
         self.vars = {
             "title": self.git_provider.pr.title,
             "branch": self.git_provider.get_pr_branch(),
-            "description": self.git_provider.get_pr_description(),
+            "description": self.git_provider.get_pr_description(full=False),
             "language": self.main_pr_language,
             "diff": "",  # empty diff for initial calculation
             "extra_instructions": get_settings().pr_description.extra_instructions,
@@ -61,27 +63,44 @@ class PRDescription:
         """
         Generates a PR description using an AI model and publishes it to the PR.
         """
-        logging.info('Generating a PR description...')
-        if get_settings().config.publish_output:
-            self.git_provider.publish_comment("Preparing pr description...", is_temporary=True)
 
-        await retry_with_fallback_models(self._prepare_prediction)
+        try:
+            logging.info(f"Generating a PR description {self.pr_id}")
+            if get_settings().config.publish_output:
+                self.git_provider.publish_comment("Preparing PR description...", is_temporary=True)
 
-        logging.info('Preparing answer...')
-        pr_title, pr_body, pr_types, markdown_text = self._prepare_pr_answer()
+            await retry_with_fallback_models(self._prepare_prediction)
 
-        if get_settings().config.publish_output:
-            logging.info('Pushing answer...')
-            if get_settings().pr_description.publish_description_as_comment:
-                self.git_provider.publish_comment(markdown_text)
+            logging.info(f"Preparing answer {self.pr_id}")
+            if self.prediction:
+                self._prepare_data()
             else:
-                self.git_provider.publish_description(pr_title, pr_body)
-                if self.git_provider.is_supported("get_labels"):
-                    current_labels = self.git_provider.get_labels()
-                    if current_labels is None:
-                        current_labels = []
-                    self.git_provider.publish_labels(pr_types + current_labels)
-            self.git_provider.remove_initial_comment()
+                return None
+
+            pr_labels = []
+            if get_settings().pr_description.publish_labels:
+                pr_labels = self._prepare_labels()
+
+            if get_settings().pr_description.use_description_markers:
+                pr_title, pr_body = self._prepare_pr_answer_with_markers()
+            else:
+                pr_title, pr_body,  = self._prepare_pr_answer()
+            full_markdown_description = f"## Title\n\n{pr_title}\n\n___\n{pr_body}"
+
+            if get_settings().config.publish_output:
+                logging.info(f"Pushing answer {self.pr_id}")
+                if get_settings().pr_description.publish_description_as_comment:
+                    self.git_provider.publish_comment(full_markdown_description)
+                else:
+                    self.git_provider.publish_description(pr_title, pr_body)
+                    if get_settings().pr_description.publish_labels and self.git_provider.is_supported("get_labels"):
+                        current_labels = self.git_provider.get_labels()
+                        if current_labels is None:
+                            current_labels = []
+                        self.git_provider.publish_labels(pr_labels + current_labels)
+                self.git_provider.remove_initial_comment()
+        except Exception as e:
+            logging.error(f"Error generating PR description {self.pr_id}: {e}")
         
         return ""
 
@@ -99,9 +118,12 @@ class PRDescription:
             Any exceptions raised by the 'get_pr_diff' and '_get_prediction' functions.
 
         """
-        logging.info('Getting PR diff...')
+        if get_settings().pr_description.use_description_markers and 'pr_agent:' not in self.user_description:
+            return None
+
+        logging.info(f"Getting PR diff {self.pr_id}")
         self.patches_diff = get_pr_diff(self.git_provider, self.token_handler, model)
-        logging.info('Getting AI prediction...')
+        logging.info(f"Getting AI prediction {self.pr_id}")
         self.prediction = await self._get_prediction(model)
 
     async def _get_prediction(self, model: str) -> str:
@@ -134,34 +156,71 @@ class PRDescription:
 
         return response
 
-    def _prepare_pr_answer(self) -> Tuple[str, str, List[str], str]:
+
+    def _prepare_data(self):
+        # Load the AI prediction data into a dictionary
+        self.data = load_yaml(self.prediction.strip())
+
+        if get_settings().pr_description.add_original_user_description and self.user_description:
+            self.data["User Description"] = self.user_description
+
+
+    def _prepare_labels(self) -> List[str]:
+        pr_types = []
+
+        # If the 'PR Type' key is present in the dictionary, split its value by comma and assign it to 'pr_types'
+        if 'PR Type' in self.data:
+            if type(self.data['PR Type']) == list:
+                pr_types = self.data['PR Type']
+            elif type(self.data['PR Type']) == str:
+                pr_types = self.data['PR Type'].split(',')
+
+        return pr_types
+
+    def _prepare_pr_answer_with_markers(self) -> Tuple[str, str]:
+        logging.info(f"Using description marker replacements {self.pr_id}")
+        title = self.vars["title"]
+        body = self.user_description
+        if get_settings().pr_description.include_generated_by_header:
+            ai_header = f"### 🤖 Generated by PR Agent at {self.git_provider.last_commit_id.sha}\n\n"
+        else:
+            ai_header = ""
+
+        ai_summary = self.data.get('PR Description')
+        if ai_summary and not re.search(r'<!--\s*pr_agent:summary\s*-->', body):
+            summary = f"{ai_header}{ai_summary}"
+            body = body.replace('pr_agent:summary', summary)
+
+        if not re.search(r'<!--\s*pr_agent:walkthrough\s*-->', body):
+            ai_walkthrough = self.data.get('PR Main Files Walkthrough')
+            if ai_walkthrough:
+                walkthrough = str(ai_header)
+                for file in ai_walkthrough:
+                    filename = file['filename'].replace("'", "`")
+                    description = file['changes in file'].replace("'", "`")
+                    walkthrough += f'- `{filename}`: {description}\n'
+
+                body = body.replace('pr_agent:walkthrough', walkthrough)
+
+        return title, body
+
+    def _prepare_pr_answer(self) -> Tuple[str, str]:
         """
         Prepare the PR description based on the AI prediction data.
 
         Returns:
         - title: a string containing the PR title.
-        - pr_body: a string containing the PR body in a markdown format.
-        - pr_types: a list of strings containing the PR types.
-        - markdown_text: a string containing the AI prediction data in a markdown format. used for publishing a comment
+        - pr_body: a string containing the PR description body in a markdown format.
         """
-        # Load the AI prediction data into a dictionary
-        data = load_yaml(self.prediction.strip())
 
-        if get_settings().pr_description.add_original_user_description and self.user_description:
-            data["User Description"] = self.user_description
-
-        # Initialization
-        pr_types = []
-
-        # If the 'PR Type' key is present in the dictionary, split its value by comma and assign it to 'pr_types'
-        if 'PR Type' in data:
-            if type(data['PR Type']) == list:
-                pr_types = data['PR Type']
-            elif type(data['PR Type']) == str:
-                pr_types = data['PR Type'].split(',')
+        # Iterate over the dictionary items and append the key and value to 'markdown_text' in a markdown format
+        markdown_text = ""
+        for key, value in self.data.items():
+            markdown_text += f"## {key}\n\n"
+            markdown_text += f"{value}\n\n"
 
         # Remove the 'PR Title' key from the dictionary
-        ai_title = data.pop('PR Title')
+        ai_title = self.data.pop('PR Title', self.vars["title"])
         if get_settings().pr_description.keep_original_user_title:
             # Assign the original PR title to the 'title' variable
             title = self.vars["title"]
@@ -172,25 +231,27 @@ class PRDescription:
         # Iterate over the remaining dictionary items and append the key and value to 'pr_body' in a markdown format,
         # except for the items containing the word 'walkthrough'
         pr_body = ""
-        for idx, (key, value) in enumerate(data.items()):
+        for idx, (key, value) in enumerate(self.data.items()):
             pr_body += f"## {key}:\n"
             if 'walkthrough' in key.lower():
                 # for filename, description in value.items():
+                if self.git_provider.is_supported("gfm_markdown"):
+                    pr_body += "<details> <summary>files:</summary>\n\n"
                 for file in value:
                     filename = file['filename'].replace("'", "`")
                     description = file['changes in file']
                     pr_body += f'`{filename}`: {description}\n'
+                if self.git_provider.is_supported("gfm_markdown"):
+                    pr_body +="</details>\n"
             else:
                 # if the value is a list, join its items by comma
                 if type(value) == list:
                     value = ', '.join(v for v in value)
                 pr_body += f"{value}\n"
-            if idx < len(data) - 1:
+            if idx < len(self.data) - 1:
                 pr_body += "\n___\n"
 
-        markdown_text = f"## Title\n\n{title}\n\n___\n{pr_body}"
-
         if get_settings().config.verbosity_level >= 2:
             logging.info(f"title:\n{title}\n{pr_body}")
 
-        return title, pr_body, pr_types, markdown_text
+        return title, pr_body

pr_agent/tools/pr_reviewer.py

@@ -59,6 +59,7 @@ class PRReviewer:
             "require_tests": get_settings().pr_reviewer.require_tests_review,
             "require_security": get_settings().pr_reviewer.require_security_review,
             "require_focused": get_settings().pr_reviewer.require_focused_review,
+            "require_estimate_effort_to_review": get_settings().pr_reviewer.require_estimate_effort_to_review,
             'num_code_suggestions': get_settings().pr_reviewer.num_code_suggestions,
             'question_str': question_str,
             'answer_str': answer_str,
@@ -94,28 +95,32 @@ class PRReviewer:
         """
         Review the pull request and generate feedback.
         """
-        if self.is_auto and not get_settings().pr_reviewer.automatic_review:
-            logging.info(f'Automatic review is disabled {self.pr_url}')
-            return None
 
-        logging.info(f'Reviewing PR: {self.pr_url} ...')
+        try:
+            if self.is_auto and not get_settings().pr_reviewer.automatic_review:
+                logging.info(f'Automatic review is disabled {self.pr_url}')
+                return None
 
-        if get_settings().config.publish_output:
-            self.git_provider.publish_comment("Preparing review...", is_temporary=True)
+            logging.info(f'Reviewing PR: {self.pr_url} ...')
 
-        await retry_with_fallback_models(self._prepare_prediction)
+            if get_settings().config.publish_output:
+                self.git_provider.publish_comment("Preparing review...", is_temporary=True)
 
-        logging.info('Preparing PR review...')
-        pr_comment = self._prepare_pr_review()
+            await retry_with_fallback_models(self._prepare_prediction)
 
-        if get_settings().config.publish_output:
-            logging.info('Pushing PR review...')
-            self.git_provider.publish_comment(pr_comment)
-            self.git_provider.remove_initial_comment()
+            logging.info('Preparing PR review...')
+            pr_comment = self._prepare_pr_review()
 
-            if get_settings().pr_reviewer.inline_code_comments:
-                logging.info('Pushing inline code comments...')
-                self._publish_inline_code_comments()
+            if get_settings().config.publish_output:
+                logging.info('Pushing PR review...')
+                self.git_provider.publish_comment(pr_comment)
+                self.git_provider.remove_initial_comment()
+
+                if get_settings().pr_reviewer.inline_code_comments:
+                    logging.info('Pushing inline code comments...')
+                    self._publish_inline_code_comments()
+        except Exception as e:
+            logging.error(f"Failed to review PR: {e}")
 
     async def _prepare_prediction(self, model: str) -> None:
         """
@@ -214,7 +219,7 @@ class PRReviewer:
                 "⏮️ Review for commits since previous PR-Agent review": f"Starting from commit {last_commit_url}"}})
             data.move_to_end('Incremental PR Review', last=False)
 
-        markdown_text = convert_to_markdown(data)
+        markdown_text = convert_to_markdown(data, self.git_provider.is_supported("gfm_markdown"))
         user = self.git_provider.get_user_id()
 
         # Add help text if not in CLI mode
@@ -266,7 +271,7 @@ class PRReviewer:
                 self.git_provider.publish_inline_comment(content, relevant_file, relevant_line_in_file)
 
         if comments:
-            self.git_provider.publish_inline_comments(comments)
+                self.git_provider.publish_inline_comments(comments)
 
     def _get_user_answers(self) -> Tuple[str, str]:
         """

pr_agent/tools/pr_similar_issue.py

@@ -0,0 +1,276 @@
+import copy
+import json
+import logging
+from enum import Enum
+from typing import List, Tuple
+import pinecone
+import openai
+import pandas as pd
+from pydantic import BaseModel, Field
+
+from pr_agent.algo import MAX_TOKENS
+from pr_agent.algo.token_handler import TokenHandler
+from pr_agent.config_loader import get_settings
+from pr_agent.git_providers import get_git_provider
+from pinecone_datasets import Dataset, DatasetMetadata
+
+MODEL = "text-embedding-ada-002"
+
+
+class PRSimilarIssue:
+    def __init__(self, issue_url: str, args: list = None):
+        if get_settings().config.git_provider != "github":
+            raise Exception("Only github is supported for similar issue tool")
+
+        self.cli_mode = get_settings().CONFIG.CLI_MODE
+        self.max_issues_to_scan = get_settings().pr_similar_issue.max_issues_to_scan
+        self.issue_url = issue_url
+        self.git_provider = get_git_provider()()
+        repo_name, issue_number = self.git_provider._parse_issue_url(issue_url.split('=')[-1])
+        self.git_provider.repo = repo_name
+        self.git_provider.repo_obj = self.git_provider.github_client.get_repo(repo_name)
+        self.token_handler = TokenHandler()
+        repo_obj = self.git_provider.repo_obj
+        repo_name_for_index = self.repo_name_for_index = repo_obj.full_name.lower().replace('/', '-').replace('_/', '-')
+        index_name = self.index_name = "codium-ai-pr-agent-issues"
+
+        # assuming pinecone api key and environment are set in secrets file
+        try:
+            api_key = get_settings().pinecone.api_key
+            environment = get_settings().pinecone.environment
+        except Exception:
+            if not self.cli_mode:
+                repo_name, original_issue_number = self.git_provider._parse_issue_url(self.issue_url.split('=')[-1])
+                issue_main = self.git_provider.repo_obj.get_issue(original_issue_number)
+                issue_main.create_comment("Please set pinecone api key and environment in secrets file")
+            raise Exception("Please set pinecone api key and environment in secrets file")
+
+        # check if index exists, and if repo is already indexed
+        run_from_scratch = False
+        upsert = True
+        pinecone.init(api_key=api_key, environment=environment)
+        if not index_name in pinecone.list_indexes():
+            run_from_scratch = True
+            upsert = False
+        else:
+            if get_settings().pr_similar_issue.force_update_dataset:
+                upsert = True
+            else:
+                pinecone_index = pinecone.Index(index_name=index_name)
+                res = pinecone_index.fetch([f"example_issue_{repo_name_for_index}"]).to_dict()
+                if res["vectors"]:
+                    upsert = False
+
+        if run_from_scratch or upsert:  # index the entire repo
+            logging.info('Indexing the entire repo...')
+
+            logging.info('Getting issues...')
+            issues = list(repo_obj.get_issues(state='all'))
+            logging.info('Done')
+            self._update_index_with_issues(issues, repo_name_for_index, upsert=upsert)
+        else:  # update index if needed
+            pinecone_index = pinecone.Index(index_name=index_name)
+            issues_to_update = []
+            issues_paginated_list = repo_obj.get_issues(state='all')
+            counter = 1
+            for issue in issues_paginated_list:
+                if issue.pull_request:
+                    continue
+                issue_str, comments, number = self._process_issue(issue)
+                issue_key = f"issue_{number}"
+                id = issue_key + "." + "issue"
+                res = pinecone_index.fetch([id]).to_dict()
+                is_new_issue = True
+                for vector in res["vectors"].values():
+                    if vector['metadata']['repo'] == repo_name_for_index:
+                        is_new_issue = False
+                        break
+                if is_new_issue:
+                    counter += 1
+                    issues_to_update.append(issue)
+                else:
+                    break
+
+            if issues_to_update:
+                logging.info(f'Updating index with {counter} new issues...')
+                self._update_index_with_issues(issues_to_update, repo_name_for_index, upsert=True)
+            else:
+                logging.info('No new issues to update')
+
+    async def run(self):
+        logging.info('Getting issue...')
+        repo_name, original_issue_number = self.git_provider._parse_issue_url(self.issue_url.split('=')[-1])
+        issue_main = self.git_provider.repo_obj.get_issue(original_issue_number)
+        issue_str, comments, number = self._process_issue(issue_main)
+        openai.api_key = get_settings().openai.key
+        logging.info('Done')
+
+        logging.info('Querying...')
+        res = openai.Embedding.create(input=[issue_str], engine=MODEL)
+        embeds = [record['embedding'] for record in res['data']]
+        pinecone_index = pinecone.Index(index_name=self.index_name)
+        res = pinecone_index.query(embeds[0],
+                                   top_k=5,
+                                   filter={"repo": self.repo_name_for_index},
+                                   include_metadata=True).to_dict()
+        relevant_issues_number_list = []
+        relevant_comment_number_list = []
+        score_list = []
+        for r in res['matches']:
+            issue_number = int(r["id"].split('.')[0].split('_')[-1])
+            if original_issue_number == issue_number:
+                continue
+            if issue_number not in relevant_issues_number_list:
+                relevant_issues_number_list.append(issue_number)
+            if 'comment' in r["id"]:
+                relevant_comment_number_list.append(int(r["id"].split('.')[1].split('_')[-1]))
+            else:
+                relevant_comment_number_list.append(-1)
+            score_list.append(str("{:.2f}".format(r['score'])))
+        logging.info('Done')
+
+        logging.info('Publishing response...')
+        similar_issues_str = "### Similar Issues\n___\n\n"
+        for i, issue_number_similar in enumerate(relevant_issues_number_list):
+            issue = self.git_provider.repo_obj.get_issue(issue_number_similar)
+            title = issue.title
+            url = issue.html_url
+            if relevant_comment_number_list[i] != -1:
+                url = list(issue.get_comments())[relevant_comment_number_list[i]].html_url
+            similar_issues_str += f"{i + 1}. **[{title}]({url})** (score={score_list[i]})\n\n"
+        if get_settings().config.publish_output:
+            response = issue_main.create_comment(similar_issues_str)
+        logging.info(similar_issues_str)
+        logging.info('Done')
+
+    def _process_issue(self, issue):
+        header = issue.title
+        body = issue.body
+        number = issue.number
+        if get_settings().pr_similar_issue.skip_comments:
+            comments = []
+        else:
+            comments = list(issue.get_comments())
+        issue_str = f"Issue Header: \"{header}\"\n\nIssue Body:\n{body}"
+        return issue_str, comments, number
+
+    def _update_index_with_issues(self, issues_list, repo_name_for_index, upsert=False):
+        logging.info('Processing issues...')
+        corpus = Corpus()
+        example_issue_record = Record(
+            id=f"example_issue_{repo_name_for_index}",
+            text="example_issue",
+            metadata=Metadata(repo=repo_name_for_index)
+        )
+        corpus.append(example_issue_record)
+
+        counter = 0
+        for issue in issues_list:
+            if issue.pull_request:
+                continue
+
+            counter += 1
+            if counter % 100 == 0:
+                logging.info(f"Scanned {counter} issues")
+            if counter >= self.max_issues_to_scan:
+                logging.info(f"Scanned {self.max_issues_to_scan} issues, stopping")
+                break
+
+            issue_str, comments, number = self._process_issue(issue)
+            issue_key = f"issue_{number}"
+            username = issue.user.login
+            created_at = str(issue.created_at)
+            if len(issue_str) < 8000 or \
+                    self.token_handler.count_tokens(issue_str) < MAX_TOKENS[MODEL]:  # fast reject first
+                issue_record = Record(
+                    id=issue_key + "." + "issue",
+                    text=issue_str,
+                    metadata=Metadata(repo=repo_name_for_index,
+                                      username=username,
+                                      created_at=created_at,
+                                      level=IssueLevel.ISSUE)
+                )
+                corpus.append(issue_record)
+                if comments:
+                    for j, comment in enumerate(comments):
+                        comment_body = comment.body
+                        num_words_comment = len(comment_body.split())
+                        if num_words_comment < 10 or not isinstance(comment_body, str):
+                            continue
+
+                        if len(comment_body) < 8000 or \
+                                self.token_handler.count_tokens(comment_body) < MAX_TOKENS[MODEL]:
+                            comment_record = Record(
+                                id=issue_key + ".comment_" + str(j + 1),
+                                text=comment_body,
+                                metadata=Metadata(repo=repo_name_for_index,
+                                                  username=username,  # use issue username for all comments
+                                                  created_at=created_at,
+                                                  level=IssueLevel.COMMENT)
+                            )
+                            corpus.append(comment_record)
+        df = pd.DataFrame(corpus.dict()["documents"])
+        logging.info('Done')
+
+        logging.info('Embedding...')
+        openai.api_key = get_settings().openai.key
+        list_to_encode = list(df["text"].values)
+        try:
+            res = openai.Embedding.create(input=list_to_encode, engine=MODEL)
+            embeds = [record['embedding'] for record in res['data']]
+        except:
+            embeds = []
+            logging.error('Failed to embed entire list, embedding one by one...')
+            for i, text in enumerate(list_to_encode):
+                try:
+                    res = openai.Embedding.create(input=[text], engine=MODEL)
+                    embeds.append(res['data'][0]['embedding'])
+                except:
+                    embeds.append([0] * 1536)
+        df["values"] = embeds
+        meta = DatasetMetadata.empty()
+        meta.dense_model.dimension = len(embeds[0])
+        ds = Dataset.from_pandas(df, meta)
+        logging.info('Done')
+
+        api_key = get_settings().pinecone.api_key
+        environment = get_settings().pinecone.environment
+        if not upsert:
+            logging.info('Creating index from scratch...')
+            ds.to_pinecone_index(self.index_name, api_key=api_key, environment=environment)
+        else:
+            logging.info('Upserting index...')
+            namespace = ""
+            batch_size: int = 100
+            concurrency: int = 10
+            pinecone.init(api_key=api_key, environment=environment)
+            ds._upsert_to_index(self.index_name, namespace, batch_size, concurrency)
+        logging.info('Done')
+
+
+class IssueLevel(str, Enum):
+    ISSUE = "issue"
+    COMMENT = "comment"
+
+
+class Metadata(BaseModel):
+    repo: str
+    username: str = Field(default="@codium")
+    created_at: str = Field(default="01-01-1970 00:00:00.00000")
+    level: IssueLevel = Field(default=IssueLevel.ISSUE)
+
+    class Config:
+        use_enum_values = True
+
+
+class Record(BaseModel):
+    id: str
+    text: str
+    metadata: Metadata
+
+
+class Corpus(BaseModel):
+    documents: List[Record] = Field(default=[])
+
+    def append(self, r: Record):
+        self.documents.append(r)

pr_agent/tools/pr_update_changelog.py

@@ -46,7 +46,7 @@ class PRUpdateChangelog:
                                           get_settings().pr_update_changelog_prompt.user)
 
     async def run(self):
-        assert type(self.git_provider) == GithubProvider, "Currently only Github is supported"
+        # assert type(self.git_provider) == GithubProvider, "Currently only Github is supported"
 
         logging.info('Updating the changelog...')
         if get_settings().config.publish_output:

pyproject.toml

@@ -26,28 +26,10 @@ classifiers = [
   "Operating System :: Independent",
   "Programming Language :: Python :: 3",
 ]
+dynamic = ["dependencies"]
 
-dependencies = [
-  "dynaconf==3.1.12",
-  "fastapi==0.99.0",
-  "PyGithub==1.59.*",
-  "retry==0.9.2",
-  "openai==0.27.8",
-  "Jinja2==3.1.2",
-  "tiktoken==0.4.0",
-  "uvicorn==0.22.0",
-  "python-gitlab==3.15.0",
-  "pytest~=7.4.0",
-  "aiohttp~=3.8.4",
-  "atlassian-python-api==3.39.0",
-  "GitPython~=3.1.32",
-  "starlette-context==0.3.6",
-  "litellm~=0.1.445",
-  "PyYAML==6.0",
-  "boto3~=1.28.25",
-  "google-cloud-storage==2.10.0",
-  "ujson==5.8.0"
-]
+[tool.setuptools.dynamic]
+dependencies = {file = ["requirements.txt"]}
 
 [project.urls]
 "Homepage" = "https://github.com/Codium-ai/pr-agent"

requirements.txt

@@ -7,13 +7,17 @@ Jinja2==3.1.2
 tiktoken==0.4.0
 uvicorn==0.22.0
 python-gitlab==3.15.0
-pytest~=7.4.0
-aiohttp~=3.8.4
+pytest==7.4.0
+aiohttp==3.8.4
 atlassian-python-api==3.39.0
-GitPython~=3.1.32
+GitPython==3.1.32
 PyYAML==6.0
 starlette-context==0.3.6
-litellm~=0.1.445
-boto3~=1.28.25
+litellm~=0.1.574
+boto3==1.28.25
 google-cloud-storage==2.10.0
 ujson==5.8.0
+azure-devops==7.1.0b3
+msrest==0.7.1
+pinecone-client
+pinecone-datasets @ git+https://github.com/mrT23/pinecone-datasets.git@main

tests/unittest/test_codecommit_client.py

@@ -110,7 +110,7 @@ class TestCodeCommitProvider:
         # Mock the response from the AWS client for get_pull_request method
         api.boto_client.get_pull_request.return_value = {
             "pullRequest": {
-                "pullRequestId": "3",
+                "pullRequestId": "321",
                 "title": "My PR",
                 "description": "My PR description",
                 "pullRequestTargets": [
@@ -125,7 +125,7 @@ class TestCodeCommitProvider:
             }
         }
 
-        pr = api.get_pr(321)
+        pr = api.get_pr("my_test_repo", 321)
 
         assert pr.title == "My PR"
         assert pr.description == "My PR description"

tests/unittest/test_codecommit_provider.py

@@ -1,6 +1,8 @@
 import pytest
+from unittest.mock import patch
 from pr_agent.git_providers.codecommit_provider import CodeCommitFile
 from pr_agent.git_providers.codecommit_provider import CodeCommitProvider
+from pr_agent.git_providers.codecommit_provider import PullRequestCCMimic
 from pr_agent.git_providers.git_provider import EDIT_TYPE
 
 
@@ -25,12 +27,64 @@ class TestCodeCommitFile:
 
 
 class TestCodeCommitProvider:
+    def test_get_title(self):
+        # Test that the get_title() function returns the PR title
+        with patch.object(CodeCommitProvider, "__init__", lambda x, y: None):
+            provider = CodeCommitProvider(None)
+            provider.pr = PullRequestCCMimic("My Test PR Title", [])
+            assert provider.get_title() == "My Test PR Title"
+
+    def test_get_pr_id(self):
+        # Test that the get_pr_id() function returns the correct ID
+        with patch.object(CodeCommitProvider, "__init__", lambda x, y: None):
+            provider = CodeCommitProvider(None)
+            provider.repo_name = "my_test_repo"
+            provider.pr_num = 321
+            assert provider.get_pr_id() == "my_test_repo/321"
+
     def test_parse_pr_url(self):
+        # Test that the _parse_pr_url() function can extract the repo name and PR number from a CodeCommit URL
         url = "https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/my_test_repo/pull-requests/321"
         repo_name, pr_number = CodeCommitProvider._parse_pr_url(url)
         assert repo_name == "my_test_repo"
         assert pr_number == 321
 
+    def test_is_valid_codecommit_hostname(self):
+        # Test the various AWS regions
+        assert CodeCommitProvider._is_valid_codecommit_hostname("af-south-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-east-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-2.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-3.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-south-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-south-2.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-2.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-3.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-4.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("ca-central-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-central-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-central-2.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-north-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-south-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-south-2.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-2.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-3.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("il-central-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("me-central-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("me-south-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("sa-east-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("us-east-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("us-east-2.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("us-gov-east-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("us-gov-west-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("us-west-1.console.aws.amazon.com")
+        assert CodeCommitProvider._is_valid_codecommit_hostname("us-west-2.console.aws.amazon.com")
+        # Test non-AWS regions
+        assert not CodeCommitProvider._is_valid_codecommit_hostname("no-such-region.console.aws.amazon.com")
+        assert not CodeCommitProvider._is_valid_codecommit_hostname("console.aws.amazon.com")
+
     # Test that an error is raised when an invalid CodeCommit URL is provided to the set_pr() method of the CodeCommitProvider class.
     # Generated by CodiumAI
     def test_invalid_codecommit_url(self):
@@ -106,6 +160,7 @@ class TestCodeCommitProvider:
         assert percentages == {}
 
     def test_get_edit_type(self):
+        # Test that the _get_edit_type() function can convert a CodeCommit letter to an EDIT_TYPE enum
         assert CodeCommitProvider._get_edit_type("A") == EDIT_TYPE.ADDED
         assert CodeCommitProvider._get_edit_type("D") == EDIT_TYPE.DELETED
         assert CodeCommitProvider._get_edit_type("M") == EDIT_TYPE.MODIFIED
@@ -117,3 +172,18 @@ class TestCodeCommitProvider:
         assert CodeCommitProvider._get_edit_type("r") == EDIT_TYPE.RENAMED
 
         assert CodeCommitProvider._get_edit_type("X") is None
+
+    def test_add_additional_newlines(self):
+        # a short string to test adding double newlines
+        input = "abc\ndef\n\n___\nghi\njkl\nmno\n\npqr\n"
+        expect = "abc\n\ndef\n\n___\n\nghi\n\njkl\n\nmno\n\npqr\n\n"
+        assert CodeCommitProvider._add_additional_newlines(input) == expect
+        # a test example from a real PR
+        input = "## PR Type:\nEnhancement\n\n___\n## PR Description:\nThis PR introduces a new feature to the script, allowing users to filter servers by name.\n\n___\n## PR Main Files Walkthrough:\n`foo`: The foo script has been updated to include a new command line option `-f` or `--filter`.\n`bar`: The bar script has been updated to list stopped servers.\n"
+        expect = "## PR Type:\n\nEnhancement\n\n___\n\n## PR Description:\n\nThis PR introduces a new feature to the script, allowing users to filter servers by name.\n\n___\n\n## PR Main Files Walkthrough:\n\n`foo`: The foo script has been updated to include a new command line option `-f` or `--filter`.\n\n`bar`: The bar script has been updated to list stopped servers.\n\n"
+        assert CodeCommitProvider._add_additional_newlines(input) == expect
+
+    def test_remove_markdown_html(self):
+        input = "## PR Feedback\n<details><summary>Code feedback:</summary>\nfile foo\n</summary>\n"
+        expect = "## PR Feedback\nCode feedback:\nfile foo\n\n"
+        assert CodeCommitProvider._remove_markdown_html(input) == expect

tests/unittest/test_language_handler.py

@@ -61,7 +61,7 @@ class TestSortFilesByMainLanguages:
             type('', (object,), {'filename': 'file1.py'})(),
             type('', (object,), {'filename': 'file2.java'})()
         ]
-        expected_output = [{'language': 'Other', 'files': []}]
+        expected_output = [{'language': 'Other', 'files': files}]
         assert sort_files_by_main_languages(languages, files) == expected_output
 
     # Tests that function handles empty files list