Add in memory provider

.github/workflows/pr-agent-review.yaml

@@ -24,7 +24,4 @@ jobs:
           OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
           OPENAI_ORG: ${{ secrets.OPENAI_ORG }} # optional
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          PINECONE.API_KEY: ${{ secrets.PINECONE_API_KEY }}
-          PINECONE.ENVIRONMENT: ${{ secrets.PINECONE_ENVIRONMENT }}
-
 

CONFIGURATION.md

@@ -0,0 +1,57 @@
+## Configuration
+
+The different tools and sub-tools used by CodiumAI PR-Agent are adjustable via the **[configuration file](pr_agent/settings/configuration.toml)**
+
+### Working from CLI
+When running from source (CLI), your local configuration file will be initially used.
+
+Example for invoking the 'review' tools via the CLI: 
+
+```
+python cli.py --pr-url=<pr_url>  review
+```
+In addition to general configurations, the 'review' tool will use parameters from the `[pr_reviewer]` section (every tool has a dedicated section in the configuration file).
+
+Note that you can print results locally, without publishing them, by setting in `configuration.toml`:
+
+```
+[config]
+publish_output=true
+verbosity_level=2
+```
+This is useful for debugging or experimenting with the different tools.
+
+### Working from pre-built repo (GitHub Action/GitHub App/Docker)
+When running PR-Agent from a pre-built repo, the default configuration file will be loaded.
+
+To edit the configuration, you have two options: 
+1. Place a local configuration file in the root of your local repo. The local file will be used instead of the default one.
+2. For online usage, just add `--config_path=<value>` to you command, to edit a specific configuration value.
+For example if you want to edit `pr_reviewer` configurations, you can run:
+```
+/review --pr_reviewer.extra_instructions="..." --pr_reviewer.require_score_review=false ...
+```
+
+Any configuration value in `configuration.toml` file can be similarly edited.
+
+### General configuration parameters
+
+#### Changing a model
+See [here](pr_agent/algo/__init__.py) for the list of available models.
+
+To use Llama2 model, for example, set:
+```
+[config]
+model = "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1"
+[replicate]
+key = ...
+```
+(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
+
+Also review the [AiHandler](pr_agent/algo/ai_handler.py) file for instruction how to set keys for other models.
+
+#### Extra instructions
+All PR-Agent tools have a parameter called `extra_instructions`, that enables to add free-text extra instructions. Example usage:
+```
+/update_changelog --pr_update_changelog.extra_instructions="Make sure to update also the version ..."
+```

Dockerfile.github_action

@@ -2,8 +2,7 @@ FROM python:3.10 as base
 
 WORKDIR /app
 ADD pyproject.toml .
-ADD requirements.txt .
+RUN pip install . && rm pyproject.toml
-RUN pip install . && rm pyproject.toml requirements.txt
 ENV PYTHONPATH=/app
 ADD pr_agent pr_agent
 ADD github_action/entrypoint.sh /

Dockerfile.github_action_dockerhub

@@ -1 +1 @@
-FROM codiumai/pr-agent:0.7-github_action
+FROM codiumai/pr-agent:github_action

INSTALL.md

@@ -9,13 +9,12 @@ To get started with PR-Agent quickly, you first need to acquire two tokens:
 There are several ways to use PR-Agent:
 
 - [Method 1: Use Docker image (no installation required)](INSTALL.md#method-1-use-docker-image-no-installation-required)
-- [Method 2: Run from source](INSTALL.md#method-2-run-from-source)
+- [Method 2: Run as a GitHub Action](INSTALL.md#method-2-run-as-a-github-action)
-- [Method 3: Run as a GitHub Action](INSTALL.md#method-3-run-as-a-github-action)
+- [Method 3: Run from source](INSTALL.md#method-3-run-from-source)
 - [Method 4: Run as a polling server](INSTALL.md#method-4-run-as-a-polling-server)
 - [Method 5: Run as a GitHub App](INSTALL.md#method-5-run-as-a-github-app)
 - [Method 6: Deploy as a Lambda Function](INSTALL.md#method-6---deploy-as-a-lambda-function)
 - [Method 7: AWS CodeCommit](INSTALL.md#method-7---aws-codecommit-setup)
-- [Method 8: Run a GitLab webhook server](INSTALL.md#method-8---run-a-gitlab-webhook-server)
 ---
 
 ### Method 1: Use Docker image (no installation required)
@@ -24,15 +23,9 @@ To request a review for a PR, or ask a question about a PR, you can run directly
 
 1. To request a review for a PR, run the following command:
 
-For GitHub:
 ```
 docker run --rm -it -e OPENAI.KEY=<your key> -e GITHUB.USER_TOKEN=<your token> codiumai/pr-agent --pr_url <pr_url> review
 ```
-For GitLab:
-```
-docker run --rm -it -e OPENAI.KEY=<your key> -e CONFIG.GIT_PROVIDER=gitlab -e GITLAB.PERSONAL_ACCESS_TOKEN=<your token> codiumai/pr-agent --pr_url <pr_url> review
-```
-For other git providers, update CONFIG.GIT_PROVIDER accordingly, and check the `pr_agent/settings/.secrets_template.toml` file for the environment variables expected names and values.
 
 2. To ask a question about a PR, run the following command:
 
@@ -62,41 +55,7 @@ Possible questions you can ask include:
 
 ---
 
-### Method 2: Run from source
+### Method 2: Run as a GitHub Action
-
-1. Clone this repository:
-
-```
-git clone https://github.com/Codium-ai/pr-agent.git
-```
-
-2. Install the requirements in your favorite virtual environment:
-
-```
-pip install -r requirements.txt
-```
-
-3. Copy the secrets template file and fill in your OpenAI key and your GitHub user token:
-
-```
-cp pr_agent/settings/.secrets_template.toml pr_agent/settings/.secrets.toml
-chmod 600 pr_agent/settings/.secrets.toml
-# Edit .secrets.toml file
-```
-
-4. Add the pr_agent folder to your PYTHONPATH, then run the cli.py script:
-
-```
-export PYTHONPATH=[$PYTHONPATH:]<PATH to pr_agent folder>
-python pr_agent/cli.py --pr_url <pr_url> /review
-python pr_agent/cli.py --pr_url <pr_url> /ask <your question>
-python pr_agent/cli.py --pr_url <pr_url> /describe
-python pr_agent/cli.py --pr_url <pr_url> /improve
-```
-
----
-
-### Method 3: Run as a GitHub Action
 
 You can use our pre-built Github Action Docker image to run PR-Agent as a Github Action. 
 
@@ -155,7 +114,7 @@ The GITHUB_TOKEN secret is automatically created by GitHub.
 3. Merge this change to your main branch. 
 When you open your next PR, you should see a comment from `github-actions` bot with a review of your PR, and instructions on how to use the rest of the tools.
 
-4. You may configure PR-Agent by adding environment variables under the env section corresponding to any configurable property in the [configuration](./Usage.md) file. Some examples:
+4. You may configure PR-Agent by adding environment variables under the env section corresponding to any configurable property in the [configuration](./CONFIGURATION.md) file. Some examples:
 ```yaml
       env:
         # ... previous environment values
@@ -166,6 +125,40 @@ When you open your next PR, you should see a comment from `github-actions` bot w
 
 ---
 
+### Method 3: Run from source
+
+1. Clone this repository:
+
+```
+git clone https://github.com/Codium-ai/pr-agent.git
+```
+
+2. Install the requirements in your favorite virtual environment:
+
+```
+pip install -r requirements.txt
+```
+
+3. Copy the secrets template file and fill in your OpenAI key and your GitHub user token:
+
+```
+cp pr_agent/settings/.secrets_template.toml pr_agent/settings/.secrets.toml
+chmod 600 pr_agent/settings/.secrets.toml
+# Edit .secrets.toml file
+```
+
+4. Add the pr_agent folder to your PYTHONPATH, then run the cli.py script:
+
+```
+export PYTHONPATH=[$PYTHONPATH:]<PATH to pr_agent folder>
+python pr_agent/cli.py --pr_url <pr_url> review
+python pr_agent/cli.py --pr_url <pr_url> ask <your question>
+python pr_agent/cli.py --pr_url <pr_url> describe
+python pr_agent/cli.py --pr_url <pr_url> improve
+```
+
+---
+
 ### Method 4: Run as a polling server
 Request reviews by tagging your Github user on a PR
 
@@ -258,9 +251,6 @@ docker push codiumai/pr-agent:github_app  # Push to your Docker repository
 
 9. Install the app by navigating to the "Install App" tab and selecting your desired repositories.
 
-> **Note:** When running PR-Agent from GitHub App, the default configuration file (configuration.toml) will be loaded.<br>
-> However, you can override the default tool parameters by uploading a local configuration file<br>
-> For more information please check out [CONFIGURATION.md](Usage.md#working-from-github-app-pre-built-repo)
 ---
 
 ### Method 6 - Deploy as a Lambda Function
@@ -319,9 +309,7 @@ Example IAM permissions to that user to allow access to CodeCommit:
                 "codecommit:Get*",
                 "codecommit:List*",
                 "codecommit:PostComment*",
-                "codecommit:PutCommentReaction",
+                "codecommit:PutCommentReaction"
-                "codecommit:UpdatePullRequestDescription",
-                "codecommit:UpdatePullRequestTitle"                
             ],
             "Resource": "*"
         }
@@ -350,24 +338,9 @@ PYTHONPATH="/PATH/TO/PROJECTS/pr-agent" python pr_agent/cli.py \
   review
 ```
 
----
+### Appendix - **Debugging LLM API Calls**  
+If you're testing your codium/pr-agent server, and need to see if calls were made successfully + the exact call logs, you can use the [LiteLLM Debugger tool](https://docs.litellm.ai/docs/debugging/hosted_debugging). 
 
-### Method 8 - Run a GitLab webhook server
+You can do this by setting `litellm_debugger=true` in configuration.toml. Your Logs will be viewable in real-time @ `admin.litellm.ai/<your_email>`. Set your email in the `.secrets.toml` under 'user_email'.
 
-1. From the GitLab workspace or group, create an access token. Enable the "api" scope only.
+<img src="./pics/debugger.png" width="800"/>
-2. Generate a random secret for your app, and save it for later. For example, you can use:
-
-```
-WEBHOOK_SECRET=$(python -c "import secrets; print(secrets.token_hex(10))")
-```
-3. Follow the instructions to build the Docker image, setup a secrets file and deploy on your own server from [Method 5](#method-5-run-as-a-github-app) steps 4-7.
-4. In the secrets file, fill in the following:
-    - Your OpenAI key.
-    - In the [gitlab] section, fill in personal_access_token and shared_secret. The access token can be a personal access token, or a group or project access token.
-    - Set deployment_type to 'gitlab' in [configuration.toml](./pr_agent/settings/configuration.toml)
-5. Create a webhook in GitLab. Set the URL to the URL of your app's server. Set the secret token to the generated secret from step 2. 
-In the "Trigger" section, check the ‘comments’ and ‘merge request events’ boxes. 
-6. Test your installation by opening a merge request or commenting or a merge request using one of CodiumAI's commands.
-
-
-=======

MANIFEST.in

@@ -0,0 +1 @@
+recursive-include pr_agent/settings/ *.toml

README.md

@@ -15,24 +15,19 @@ Making pull requests less painful with an AI agent
 </div>
 <div style="text-align:left;">
 
-CodiumAI `PR-Agent` is an open-source tool aiming to help developers review pull requests faster and more efficiently. It automatically analyzes the pull request and can provide several types of commands:
+CodiumAI `PR-Agent` is an open-source tool aiming to help developers review pull requests faster and more efficiently. It automatically analyzes the pull request and can provide several types of PR feedback:
 
-‣ **Auto Description (`/describe`)**: Automatically generating [PR description](https://github.com/Codium-ai/pr-agent/pull/229#issue-1860711415) - title, type, summary, code walkthrough and labels.
+**Auto-Description**: Automatically generating [PR description](https://github.com/Codium-ai/pr-agent/pull/229#issue-1860711415) - title, type, summary, code walkthrough and labels.
 \
-‣ **Auto Review (`/review`)**: [Adjustable feedback](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695022908) about the PR main theme, type, relevant tests, security issues, score, and various suggestions for the PR content.
+**Auto Review**: [Adjustable feedback](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695022908) about the PR main theme, type, relevant tests, security issues, score, and various suggestions for the PR content.
 \
-‣ **Question Answering (`/ask ...`)**: Answering [free-text questions](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021332) about the PR.
+**Question Answering**: Answering [free-text questions](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021332) about the PR.
 \
-‣ **Code Suggestions (`/improve`)**: [Committable code suggestions](https://github.com/Codium-ai/pr-agent/pull/229#discussion_r1306919276) for improving the PR.
+**Code Suggestions**: [Committable code suggestions](https://github.com/Codium-ai/pr-agent/pull/229#discussion_r1306919276) for improving the PR.
 \
-‣ **Update Changelog (`/update_changelog`)**: Automatically updating the CHANGELOG.md file with the [PR changes](https://github.com/Codium-ai/pr-agent/pull/168#discussion_r1282077645).
+**Update Changelog**: Automatically updating the CHANGELOG.md file with the [PR changes](https://github.com/Codium-ai/pr-agent/pull/168#discussion_r1282077645).
-\
-‣ **Find similar issue (`/similar_issue`)**: Automatically retrieves and presents [similar issues](https://github.com/Alibaba-MIIL/ASL/issues/107).
 
-
+<h3>Example results:</h2>
-See the [usage guide](./Usage.md) for instructions how to run the different tools from [CLI](./Usage.md#working-from-a-local-repo-cli), or by [online usage](./Usage.md#online-usage), as well as additional details on optional commands and configurations.
-
-<h3>Example results:</h3>
 </div>
 <h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1687561986">/describe:</a></h4>
 <div align="center">
@@ -40,101 +35,87 @@ See the [usage guide](./Usage.md) for instructions how to run the different tool
 <img src="https://www.codium.ai/images/describe-2.gif" width="800">
 </p>
 </div>
-
 <h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901">/review:</a></h4>
 <div align="center">
 <p float="center">
 <img src="https://www.codium.ai/images/review-2.gif" width="800">
 </p>
 </div>
-
+<h4><a href="https://github.com/Codium-ai/pr-agent/pull/78#issuecomment-1639739496">/reflect_and_review:</a></h4>
-[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/78#issuecomment-1639739496">/reflect_and_review:</a></h4>)
+<div align="center">
-
+<p float="center">
-[//]: # (<div align="center">)
+<img src="https://www.codium.ai/images/reflect_and_review.gif" width="800">
-
+</p>
-[//]: # (<p float="center">)
+</div>
-
+<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695020538">/ask:</a></h4>
-[//]: # (<img src="https://www.codium.ai/images/reflect_and_review.gif" width="800">)
+<div align="center">
-
+<p float="center">
-[//]: # (</p>)
+<img src="https://www.codium.ai/images/ask-2.gif" width="800">
-
+</p>
-[//]: # (</div>)
+</div>
-
+<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695024952">/improve:</a></h4>
-[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695020538">/ask:</a></h4>)
+<div align="center">
-
+<p float="center">
-[//]: # (<div align="center">)
+<img src="https://www.codium.ai/images/improve-2.gif" width="800">
-
+</p>
-[//]: # (<p float="center">)
+</div>
-
-[//]: # (<img src="https://www.codium.ai/images/ask-2.gif" width="800">)
-
-[//]: # (</p>)
-
-[//]: # (</div>)
-
-[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695024952">/improve:</a></h4>)
-
-[//]: # (<div align="center">)
-
-[//]: # (<p float="center">)
-
-[//]: # (<img src="https://www.codium.ai/images/improve-2.gif" width="800">)
-
-[//]: # (</p>)
-
-[//]: # (</div>)
 <div align="left">
 
-## Table of Contents
+
 - [Overview](#overview)
 - [Try it now](#try-it-now)
 - [Installation](#installation)
+- [Configuration](./CONFIGURATION.md)
 - [How it works](#how-it-works)
-- [Why use PR-Agent?](#why-use-pr-agent)
+- [Why use PR-Agent](#why-use-pr-agent)
 - [Roadmap](#roadmap)
+- [Similar projects](#similar-projects)
 </div>
 
 
 ## Overview
 `PR-Agent` offers extensive pull request functionalities across various git providers:
-|       |                                             | GitHub | Gitlab | Bitbucket | CodeCommit | Azure DevOps | Gerrit |
+|       |                                             | GitHub | Gitlab | Bitbucket | CodeCommit |
-|-------|---------------------------------------------|:------:|:------:|:---------:|:----------:|:----------:|:----------:|
+|-------|---------------------------------------------|:------:|:------:|:---------:|:----------:|
-| TOOLS | Review                                      |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:    |
+| TOOLS | Review                                      |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |   :white_check_mark:    |
-|       | Ask                                         |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:          |   :white_check_mark:          | :white_check_mark: |  :white_check_mark:    |
+|       | ⮑ Inline review                             |   :white_check_mark:    |   :white_check_mark:    |           |          |
-|       | Auto-Description                            |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:        |   :white_check_mark:    |   :white_check_mark:    | :white_check_mark:    |
+|       | Ask                                         |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:          |
-|       | Improve Code                                |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:        |   :white_check_mark:    |          |    :white_check_mark:    |
+|       | Auto-Description                            |   :white_check_mark:    |   :white_check_mark:    |           |          |
-|       | ⮑ Extended                             |   :white_check_mark:    |   :white_check_mark:    |        :white_check_mark:   |   :white_check_mark:    |          | :white_check_mark:    |
+|       | Improve Code                                |   :white_check_mark:    |   :white_check_mark:    |           |          |
-|       | Reflect and Review                          |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:        |          |   :white_check_mark:    |    :white_check_mark:    |
+|       | ⮑ Extended                             |   :white_check_mark:    |   :white_check_mark:    |           |          |
-|       | Update CHANGELOG.md                         |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:        |          |          |       |
+|       | Reflect and Review                          |   :white_check_mark:    |                         |           |          |
-|       | Find similar issue                          |   :white_check_mark:    |                         |                             |          |          |       |
+|       | Update CHANGELOG.md                         |   :white_check_mark:    |                         |           |          |
-|       |                                             |        |        |      |      |      |
+|       |                                             |        |        |      |      |
-| USAGE | CLI                                         |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |   :white_check_mark:    |   :white_check_mark:    |
+| USAGE | CLI                                         |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |   :white_check_mark:    |
-|       | App / webhook                               |   :white_check_mark:    |   :white_check_mark:    |           |          |          |
+|       | App / webhook                               |   :white_check_mark:    |   :white_check_mark:    |           |          |
-|       | Tagging bot                                 |   :white_check_mark:    |        |           |          |          |
+|       | Tagging bot                                 |   :white_check_mark:    |        |           |          |
-|       | Actions                                     |   :white_check_mark:    |        |           |          |          |
+|       | Actions                                     |   :white_check_mark:    |        |           |          |
-|       | Web server                                  |       |        |           |          |          |  :white_check_mark:   |
+|       |                                             |        |        |      |      |
-|       |                                             |        |        |      |      |      |
+| CORE  | PR compression                              |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |          |
-| CORE  | PR compression                              |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       | :white_check_mark: |   :white_check_mark:       | :white_check_mark:       |
+|       | Repo language prioritization                |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |          |
-|       | Repo language prioritization                |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       | :white_check_mark: |   :white_check_mark:       | :white_check_mark:       |
+|       | Adaptive and token-aware<br />file patch fitting |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |          |
-|       | Adaptive and token-aware<br />file patch fitting |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       | :white_check_mark: |   :white_check_mark:       | :white_check_mark:       |
+|       | Multiple models support |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |          |
-|       | Multiple models support |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       | :white_check_mark: |   :white_check_mark:       | :white_check_mark:       |
+|       | Incremental PR Review |   :white_check_mark:    |      |      |      |
-|       | Incremental PR Review |   :white_check_mark:    |      |      |      |      |      |
 
-Review the **[usage guide](./Usage.md)** section for detailed instructions how to use the different tools, select the relevant git provider (GitHub, Gitlab, Bitbucket,...), and adjust the configuration file to your needs.
+Examples for invoking the different tools via the CLI:
+- **Review**:       python cli.py --pr_url=<pr_url>  review
+- **Describe**:     python cli.py --pr_url=<pr_url>  describe
+- **Improve**:      python cli.py --pr_url=<pr_url>  improve
+- **Ask**:          python cli.py --pr_url=<pr_url>  ask "Write me a poem about this PR"
+- **Reflect**:      python cli.py --pr_url=<pr_url>  reflect
+- **Update Changelog**:      python cli.py --pr_url=<pr_url>  update_changelog
+
+"<pr_url>" is the url of the relevant PR (for example: https://github.com/Codium-ai/pr-agent/pull/50).
+
+In the [configuration](./CONFIGURATION.md) file you can select your git provider (GitHub, Gitlab, Bitbucket), and further configure the different tools.
 
 ## Try it now
 
-You can try GPT-4 powered PR-Agent, on your public GitHub repository, instantly. Just mention `@CodiumAI-Agent` and add the desired command in any PR comment. The agent will generate a response based on your command.
+Try GPT-4 powered PR-Agent on your public GitHub repository for free. Just mention `@CodiumAI-Agent` and add the desired command in any PR comment! The agent will generate a response based on your command.
-For example, add a comment to any pull request with the following text:
-```
-@CodiumAI-Agent /review
-```
-and the agent will respond with a review of your PR
 
 ![Review generation process](https://www.codium.ai/images/demo-2.gif)
 
-
+To set up your own PR-Agent, see the [Installation](#installation) section
-To set up your own PR-Agent, see the [Installation](#installation) section below.
 
 ---
 
@@ -148,15 +129,14 @@ To get started with PR-Agent quickly, you first need to acquire two tokens:
 There are several ways to use PR-Agent:
 
 - [Method 1: Use Docker image (no installation required)](INSTALL.md#method-1-use-docker-image-no-installation-required)
-- [Method 2: Run from source](INSTALL.md#method-2-run-from-source)
+- [Method 2: Run as a GitHub Action](INSTALL.md#method-2-run-as-a-github-action)
-- [Method 3: Run as a GitHub Action](INSTALL.md#method-3-run-as-a-github-action)
+- [Method 3: Run from source](INSTALL.md#method-3-run-from-source)
 - [Method 4: Run as a polling server](INSTALL.md#method-4-run-as-a-polling-server)
   - Request reviews by tagging your GitHub user on a PR
 - [Method 5: Run as a GitHub App](INSTALL.md#method-5-run-as-a-github-app)
   - Allowing you to automate the review process on your private or public repositories
 - [Method 6: Deploy as a Lambda Function](INSTALL.md#method-6---deploy-as-a-lambda-function)
 - [Method 7: AWS CodeCommit](INSTALL.md#method-7---aws-codecommit-setup)
-- [Method 8: Run a GitLab webhook server](INSTALL.md#method-8---run-a-gitlab-webhook-server)
 
 ## How it works
 
@@ -174,7 +154,7 @@ Here are some advantages of PR-Agent:
 
 - We emphasize **real-life practical usage**. Each tool (review, improve, ask, ...) has a single GPT-4 call, no more. We feel that this is critical for realistic team usage - obtaining an answer quickly (~30 seconds) and affordably.
 - Our [PR Compression strategy](./PR_COMPRESSION.md)  is a core ability that enables to effectively tackle both short and long PRs.
-- Our JSON prompting strategy enables to have **modular, customizable tools**. For example, the '/review' tool categories can be controlled via the [configuration](pr_agent/settings/configuration.toml) file. Adding additional categories is easy and accessible.
+- Our JSON prompting strategy enables to have **modular, customizable tools**. For example, the '/review' tool categories can be controlled via the [configuration](./CONFIGURATION.md) file. Adding additional categories is easy and accessible.
 - We support **multiple git providers** (GitHub, Gitlab, Bitbucket, CodeCommit), **multiple ways** to use the tool (CLI, GitHub Action, GitHub App, Docker, ...), and **multiple models** (GPT-4, GPT-3.5, Anthropic, Cohere, Llama2).
 - We are open-source, and welcome contributions from the community.
 
@@ -184,7 +164,7 @@ Here are some advantages of PR-Agent:
 - [x] Support additional models, as a replacement for OpenAI (see [here](https://github.com/Codium-ai/pr-agent/pull/172))
 - [x] Develop additional logic for handling large PRs (see [here](https://github.com/Codium-ai/pr-agent/pull/229))
 - [ ] Add additional context to the prompt. For example, repo (or relevant files) summarization, with tools such a [ctags](https://github.com/universal-ctags/ctags)
-- [x] PR-Agent for issues
+- [ ] PR-Agent for issues, and just for pull requests
 - [ ] Adding more tools. Possible directions:
   - [x] PR description
   - [x] Inline code suggestions
@@ -197,18 +177,8 @@ Here are some advantages of PR-Agent:
 
 ## Similar Projects
 
-- [CodiumAI - Meaningful tests for busy devs](https://github.com/Codium-ai/codiumai-vscode-release) (although various capabilities are much more advanced in the CodiumAI IDE plugins)
+- [CodiumAI - Meaningful tests for busy devs](https://github.com/Codium-ai/codiumai-vscode-release)
 - [Aider - GPT powered coding in your terminal](https://github.com/paul-gauthier/aider)
 - [openai-pr-reviewer](https://github.com/coderabbitai/openai-pr-reviewer)
 - [CodeReview BOT](https://github.com/anc95/ChatGPT-CodeReview)
 - [AI-Maintainer](https://github.com/merwanehamadi/AI-Maintainer)
-  
-## Links
-
-[![Join our Discord community](https://raw.githubusercontent.com/Codium-ai/codiumai-vscode-release/main/media/docs/Joincommunity.png)](https://discord.gg/kG35uSHDBc)
-
-- Discord community: https://discord.gg/kG35uSHDBc
-- CodiumAI site: https://codium.ai
-- Blog: https://www.codium.ai/blog/
-- Troubleshooting: https://www.codium.ai/blog/technical-faq-and-troubleshooting/
-- Support: support@codium.ai

RELEASE_NOTES.md

@@ -1,25 +0,0 @@
-## [Version 0.7] - 2023-09-20
-
-### Docker Tags
-- codiumai/pr-agent:0.7
-- codiumai/pr-agent:0.7-github_app
-- codiumai/pr-agent:0.7-bitbucket-app
-- codiumai/pr-agent:0.7-gitlab_webhook
-- codiumai/pr-agent:0.7-github_polling
-- codiumai/pr-agent:0.7-github_action
- 
-### Added::Algo
-- New tool /similar_issue - Currently on GitHub app and CLI: indexes the issues in the repo, find the most similar issues to the target issue.
-- Describe markers: Empower the /describe tool with a templating capability (see more details in https://github.com/Codium-ai/pr-agent/pull/273).
-- New feature in the /review tool - added an estimated effort estimation to the review (https://github.com/Codium-ai/pr-agent/pull/306).
-
-### Added::Infrastructure
-- Implementation of a GitLab webhook.
-- Implementation of a BitBucket app.
-
-### Fixed
-- Protection against no code suggestions generated.
-- Resilience to repositories where the languages cannot be automatically detected.
-
-
-

Usage.md

@@ -1,272 +0,0 @@
-## Usage guide
-
-### Table of Contents
-- [Introduction](#introduction)
-- [Working from a local repo (CLI)](#working-from-a-local-repo-cli)
-- [Online usage](#online-usage)
-- [Working with GitHub App](#working-with-github-app)
-- [Working with GitHub Action](#working-with-github-action)
-- [Appendix - additional configurations walkthrough](#appendix---additional-configurations-walkthrough)
-
-### Introduction
-
-There are 3 basic ways to invoke CodiumAI PR-Agent:
-1. Locally running a CLI command
-2. Online usage - by [commenting](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901) on a PR
-3. Enabling PR-Agent tools to run automatically when a new PR is opened
-
-See the [installation guide](/INSTALL.md) for instructions on how to setup your own PR-Agent.
-
-Specifically, CLI commands can be issued by invoking a pre-built [docker image](/INSTALL.md#running-from-source), or by invoking a [locally cloned repo](INSTALL.md#method-2-run-from-source).
-
-For online usage, you will need to setup either a [GitHub App](INSTALL.md#method-5-run-as-a-github-app), or a [GitHub Action](INSTALL.md#method-3-run-as-a-github-action).
-GitHub App and GitHub Action also enable to run PR-Agent specific tool automatically when a new PR is opened.
-
-
-#### The configuration file
-The different tools and sub-tools used by CodiumAI PR-Agent are adjustable via the **[configuration file](pr_agent/settings/configuration.toml)**.
-In addition to general configuration options, each tool has its own configurations. For example, the `review` tool will use parameters from the [pr_reviewer](/pr_agent/settings/configuration.toml#L16) section in the configuration file.
-
-**git provider:**
-The [git_provider](pr_agent/settings/configuration.toml#L4) field in the configuration file determines the GIT provider that will be used by PR-Agent. Currently, the following providers are supported:
-`
-"github", "gitlab", "azure", "codecommit", "local"
-`
-
-[//]: # (** online usage:**)
-
-[//]: # (Options that are available in the configuration file can be specified at run time when calling actions. Two examples:)
-
-[//]: # (```)
-
-[//]: # (- /review --pr_reviewer.extra_instructions="focus on the file: ...")
-
-[//]: # (- /describe --pr_description.add_original_user_description=false -pr_description.extra_instructions="make sure to mention: ...")
-
-[//]: # (```)
-
-### Working from a local repo (CLI)
-When running from your local repo (CLI), your local configuration file will be used.
-
-Examples for invoking the different tools via the CLI:
-
-- **Review**:       `python cli.py --pr_url=<pr_url>  review`
-- **Describe**:     `python cli.py --pr_url=<pr_url>  describe`
-- **Improve**:      `python cli.py --pr_url=<pr_url>  improve`
-- **Ask**:          `python cli.py --pr_url=<pr_url>  ask "Write me a poem about this PR"`
-- **Reflect**:      `python cli.py --pr_url=<pr_url>  reflect`
-- **Update Changelog**:      `python cli.py --pr_url=<pr_url>  update_changelog`
-
-`<pr_url>` is the url of the relevant PR (for example: https://github.com/Codium-ai/pr-agent/pull/50).
-
-**Notes:**
-
-(1) in addition to editing your local configuration file, you can also change any configuration value by adding it to the command line:
-```
-python cli.py --pr_url=<pr_url>  /review --pr_reviewer.extra_instructions="focus on the file: ..."
-```
-
-(2) You can print results locally, without publishing them, by setting in `configuration.toml`:
-```
-[config]
-publish_output=true
-verbosity_level=2
-```
-This is useful for debugging or experimenting with the different tools.
-
-
-### Online usage
-
-Online usage means invoking PR-Agent tools by [comments](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901) on a PR.
-Commands for invoking the different tools via comments:
-
-- **Review**:       `/review`
-- **Describe**:     `/describe`
-- **Improve**:      `/improve`
-- **Ask**:          `/ask "..."`
-- **Reflect**:      `/reflect`
-- **Update Changelog**:      `/update_changelog`
-
-
-To edit a specific configuration value, just add `--config_path=<value>` to any command.
-For example if you want to edit the `review` tool configurations, you can run:
-```
-/review --pr_reviewer.extra_instructions="..." --pr_reviewer.require_score_review=false
-```
-Any configuration value in [configuration file](pr_agent/settings/configuration.toml) file can be similarly edited.
-
-
-### Working with GitHub App
-When running PR-Agent from [GitHub App](INSTALL.md#method-5-run-as-a-github-app), the default configurations from a pre-built repo will be initially loaded.
-
-#### GitHub app automatic tools
-The [github_app](pr_agent/settings/configuration.toml#L56) section defines GitHub app specific configurations. 
-An important parameter is `pr_commands`, which is a list of tools that will be **run automatically when a new PR is opened**:
-```
-[github_app]
-pr_commands = [
-    "/describe --pr_description.add_original_user_description=true --pr_description.keep_original_user_title=true",
-    "/auto_review",
-]
-```
-This means that when a new PR is opened, PR-Agent will run the `describe` and `auto_review` tools.
-For the describe tool, the `add_original_user_description` and `keep_original_user_title` parameters will be set to true.
-
-However, you can override the default tool parameters by uploading a local configuration file called `.pr_agent.toml` to the root of your repo.
-For example, if your local `.pr_agent.toml` file contains:
-```
-[pr_description]
-add_original_user_description = false
-keep_original_user_title = false
-```
-When a new PR is opened, PR-Agent will run the `describe` tool with the above parameters.
-
-Note that a local `.pr_agent.toml` file enables you to edit and customize the default parameters of any tool, not just the ones that are run automatically.
-
-#### Editing the prompts
-The prompts for the various PR-Agent tools are defined in the `pr_agent/settings` folder.
-
-In practice, the prompts are loaded and stored as a standard setting object. 
-Hence, editing them is similar to editing any other configuration value - just place the relevant key in `.pr_agent.toml`file, and override the default value.
-
-For example, if you want to edit the prompts of the [describe](./pr_agent/settings/pr_description_prompts.toml) tool, you can add the following to your `.pr_agent.toml` file:
-```
-[pr_description_prompt]
-system="""
-...
-"""
-user="""
-...
-"""
-```
-Note that the new prompt will need to generate an output compatible with the relevant [post-process function](./pr_agent/tools/pr_description.py#L137).
-
-### Working with GitHub Action
-TBD
-
-### Appendix - additional configurations walkthrough
-
-#### Changing a model
-See [here](pr_agent/algo/__init__.py) for the list of available models.
-
-#### Azure
-To use Azure, set: 
-```
-api_key = "" # your azure api key
-api_type = "azure"
-api_version = '2023-05-15'  # Check Azure documentation for the current API version
-api_base = ""  # The base URL for your Azure OpenAI resource. e.g. "https://<your resource name>.openai.azure.com"
-deployment_id = ""  # The deployment name you chose when you deployed the engine
-```
-in your .secrets.toml
-
-and 
-```
-[config]
-model="" # the OpenAI model you've deployed on Azure (e.g. gpt-3.5-turbo)
-```
-in the configuration.toml 
-
-#### Huggingface
-
-**Local**  
-You can run Huggingface models locally through either [VLLM](https://docs.litellm.ai/docs/providers/vllm) or [Ollama](https://docs.litellm.ai/docs/providers/ollama)
-
-E.g. to use a new Huggingface model locally via Ollama, set:
-```
-[__init__.py]
-MAX_TOKENS = {
-    "model-name-on-ollama": <max_tokens>
-}
-e.g.
-MAX_TOKENS={
-    ...,
-    "llama2": 4096
-}
-
-
-[config] # in configuration.toml
-model = "ollama/llama2"
-
-[ollama] # in .secrets.toml
-api_base = ... # the base url for your huggingface inference endpoint 
-```
-
-**Inference Endpoints**
-
-To use a new model with Huggingface Inference Endpoints, for example, set:
-```
-[__init__.py]
-MAX_TOKENS = {
-    "model-name-on-huggingface": <max_tokens>
-}
-e.g.
-MAX_TOKENS={
-    ...,
-    "meta-llama/Llama-2-7b-chat-hf": 4096
-}
-[config] # in configuration.toml
-model = "huggingface/meta-llama/Llama-2-7b-chat-hf"
-
-[huggingface] # in .secrets.toml
-key = ... # your huggingface api key
-api_base = ... # the base url for your huggingface inference endpoint 
-```
-(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
-
-#### Replicate
-
-To use Llama2 model with Replicate, for example, set:
-```
-[config] # in configuration.toml
-model = "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1"
-[replicate] # in .secrets.toml
-key = ...
-```
-(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
-
-
-Also review the [AiHandler](pr_agent/algo/ai_handler.py) file for instruction how to set keys for other models.
-
-#### Extra instructions
-All PR-Agent tools have a parameter called `extra_instructions`, that enables to add free-text extra instructions. Example usage:
-```
-/update_changelog --pr_update_changelog.extra_instructions="Make sure to update also the version ..."
-```
-
-#### Azure DevOps provider
-To use Azure DevOps provider use the following settings in configuration.toml:
-```
-[config]
-git_provider="azure"
-use_repo_settings_file=false
-```
-
-And use the following settings (you have to replace the values) in .secrets.toml:
-```
-[azure_devops]
-org = "https://dev.azure.com/YOUR_ORGANIZATION/"
-pat = "YOUR_PAT_TOKEN"
-```
-
-#### Similar issue tool
-
-[Example usage](https://github.com/Alibaba-MIIL/ASL/issues/107)
-
-<img src=./pics/similar_issue_tool.png width="768">
-
-To enable usage of the '**similar issue**' tool, you need to set the following keys in `.secrets.toml` (or in the relevant environment variables):
-```
-[pinecone]
-api_key = "..."
-environment = "..."
-```
-These parameters can be obtained by registering to [Pinecone](https://app.pinecone.io/?sessionType=signup/).
-
-- To invoke the 'similar issue' tool from **CLI**, run:
-`python3 cli.py --issue_url=... similar_issue`
-
-- To invoke the 'similar' issue tool via online usage, [comment](https://github.com/Codium-ai/pr-agent/issues/178#issuecomment-1716934893) on a PR:
-`/similar_issue`
-
-- You can also enable the 'similar issue' tool to run automatically when a new issue is opened, by adding it to the [pr_commands list in the github_app section](https://github.com/Codium-ai/pr-agent/blob/main/pr_agent/settings/configuration.toml#L66)

docker/Dockerfile

@@ -18,10 +18,6 @@ FROM base as github_polling
 ADD pr_agent pr_agent
 CMD ["python", "pr_agent/servers/github_polling.py"]
 
-FROM base as gitlab_webhook
-ADD pr_agent pr_agent
-CMD ["python", "pr_agent/servers/gitlab_webhook.py"]
-
 FROM base as test
 ADD requirements-dev.txt .
 RUN pip install -r requirements-dev.txt && rm requirements-dev.txt

pr_agent/agent/pr_agent.py

@@ -9,7 +9,6 @@ from pr_agent.git_providers import get_git_provider
 from pr_agent.tools.pr_code_suggestions import PRCodeSuggestions
 from pr_agent.tools.pr_description import PRDescription
 from pr_agent.tools.pr_information_from_user import PRInformationFromUser
-from pr_agent.tools.pr_similar_issue import PRSimilarIssue
 from pr_agent.tools.pr_questions import PRQuestions
 from pr_agent.tools.pr_reviewer import PRReviewer
 from pr_agent.tools.pr_update_changelog import PRUpdateChangelog
@@ -31,7 +30,6 @@ command2class = {
     "update_changelog": PRUpdateChangelog,
     "config": PRConfig,
     "settings": PRConfig,
-    "similar_issue": PRSimilarIssue,
 }
 
 commands = list(command2class.keys())

pr_agent/algo/__init__.py

@@ -1,5 +1,4 @@
 MAX_TOKENS = {
-    'text-embedding-ada-002': 8000,
     'gpt-3.5-turbo': 4000,
     'gpt-3.5-turbo-0613': 4000,
     'gpt-3.5-turbo-0301': 4000,
@@ -12,5 +11,4 @@ MAX_TOKENS = {
     'claude-2': 100000,
     'command-nightly': 4096,
     'replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1': 4096,
-    'meta-llama/Llama-2-7b-chat-hf': 4096
 }

pr_agent/algo/ai_handler.py

@@ -1,12 +1,13 @@
 import logging
-import os
 
 import litellm
 import openai
 from litellm import acompletion
 from openai.error import APIError, RateLimitError, Timeout, TryAgain
 from retry import retry
+
 from pr_agent.config_loader import get_settings
+
 OPENAI_RETRIES = 5
 
 
@@ -25,11 +26,7 @@ class AiHandler:
         try:
             openai.api_key = get_settings().openai.key
             litellm.openai_key = get_settings().openai.key
-            if get_settings().get("litellm.use_client"):
+            litellm.debugger = get_settings().config.litellm_debugger
-                litellm_token = get_settings().get("litellm.LITELLM_TOKEN")
-                assert litellm_token, "LITELLM_TOKEN is required"
-                os.environ["LITELLM_TOKEN"] = litellm_token
-                litellm.use_client = True
             self.azure = False
             if get_settings().get("OPENAI.ORG", None):
                 litellm.organization = get_settings().openai.org
@@ -51,8 +48,6 @@ class AiHandler:
                 litellm.replicate_key = get_settings().replicate.key
             if get_settings().get("HUGGINGFACE.KEY", None):
                 litellm.huggingface_key = get_settings().huggingface.key
-                if get_settings().get("HUGGINGFACE.API_BASE", None):
-                    litellm.api_base = get_settings().huggingface.api_base
         except AttributeError as e:
             raise ValueError("OpenAI key is required") from e
 

pr_agent/algo/language_handler.py

@@ -42,11 +42,6 @@ def sort_files_by_main_languages(languages: Dict, files: list):
     files_sorted = []
     rest_files = {}
 
-    # if no languages detected, put all files in the "Other" category
-    if not languages:
-        files_sorted = [({"language": "Other", "files": list(files_filtered)})]
-        return files_sorted
-
     main_extensions_flat = []
     for ext in main_extensions:
         main_extensions_flat.extend(ext)

pr_agent/algo/pr_processing.py

@@ -1,19 +1,17 @@
 from __future__ import annotations
 
-import difflib
 import logging
-import re
 import traceback
-from typing import Any, Callable, List, Tuple
+from typing import Callable, List, Tuple
 
 from github import RateLimitExceededException
 
 from pr_agent.algo import MAX_TOKENS
 from pr_agent.algo.git_patch_processing import convert_to_hunks_with_lines_numbers, extend_patch, handle_patch_deletions
 from pr_agent.algo.language_handler import sort_files_by_main_languages
-from pr_agent.algo.token_handler import TokenHandler, get_token_encoder
+from pr_agent.algo.token_handler import TokenHandler
 from pr_agent.config_loader import get_settings
-from pr_agent.git_providers.git_provider import FilePatchInfo, GitProvider
+from pr_agent.git_providers.git_provider import GitProvider
 
 DELETED_FILES_ = "Deleted files:\n"
 
@@ -247,99 +245,6 @@ def _get_all_deployments(all_models: List[str]) -> List[str]:
     return all_deployments
 
 
-def find_line_number_of_relevant_line_in_file(diff_files: List[FilePatchInfo],
-                                              relevant_file: str,
-                                              relevant_line_in_file: str) -> Tuple[int, int]:
-    """
-    Find the line number and absolute position of a relevant line in a file.
-
-    Args:
-        diff_files (List[FilePatchInfo]): A list of FilePatchInfo objects representing the patches of files.
-        relevant_file (str): The name of the file where the relevant line is located.
-        relevant_line_in_file (str): The content of the relevant line.
-
-    Returns:
-        Tuple[int, int]: A tuple containing the line number and absolute position of the relevant line in the file.
-    """
-    position = -1
-    absolute_position = -1
-    re_hunk_header = re.compile(
-        r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@[ ]?(.*)")
-
-    for file in diff_files:
-        if file.filename.strip() == relevant_file:
-            patch = file.patch
-            patch_lines = patch.splitlines()
-
-            # try to find the line in the patch using difflib, with some margin of error
-            matches_difflib: list[str | Any] = difflib.get_close_matches(relevant_line_in_file,
-                                                                         patch_lines, n=3, cutoff=0.93)
-            if len(matches_difflib) == 1 and matches_difflib[0].startswith('+'):
-                relevant_line_in_file = matches_difflib[0]
-
-            delta = 0
-            start1, size1, start2, size2 = 0, 0, 0, 0
-            for i, line in enumerate(patch_lines):
-                if line.startswith('@@'):
-                    delta = 0
-                    match = re_hunk_header.match(line)
-                    start1, size1, start2, size2 = map(int, match.groups()[:4])
-                elif not line.startswith('-'):
-                    delta += 1
-
-                if relevant_line_in_file in line and line[0] != '-':
-                    position = i
-                    absolute_position = start2 + delta - 1
-                    break
-
-            if position == -1 and relevant_line_in_file[0] == '+':
-                no_plus_line = relevant_line_in_file[1:].lstrip()
-                for i, line in enumerate(patch_lines):
-                    if line.startswith('@@'):
-                        delta = 0
-                        match = re_hunk_header.match(line)
-                        start1, size1, start2, size2 = map(int, match.groups()[:4])
-                    elif not line.startswith('-'):
-                        delta += 1
-
-                    if no_plus_line in line and line[0] != '-':
-                        # The model might add a '+' to the beginning of the relevant_line_in_file even if originally
-                        # it's a context line
-                        position = i
-                        absolute_position = start2 + delta - 1
-                        break
-    return position, absolute_position
-
-
-def clip_tokens(text: str, max_tokens: int) -> str:
-    """
-    Clip the number of tokens in a string to a maximum number of tokens.
-
-    Args:
-        text (str): The string to clip.
-        max_tokens (int): The maximum number of tokens allowed in the string.
-
-    Returns:
-        str: The clipped string.
-    """
-    if not text:
-        return text
-
-    try:
-        encoder = get_token_encoder()
-        num_input_tokens = len(encoder.encode(text))
-        if num_input_tokens <= max_tokens:
-            return text
-        num_chars = len(text)
-        chars_per_token = num_chars / num_input_tokens
-        num_output_chars = int(chars_per_token * max_tokens)
-        clipped_text = text[:num_output_chars]
-        return clipped_text
-    except Exception as e:
-        logging.warning(f"Failed to clip tokens: {e}")
-        return text
-
-
 def get_pr_multi_diffs(git_provider: GitProvider,
                        token_handler: TokenHandler,
                        model: str,

pr_agent/algo/token_handler.py

@@ -21,7 +21,7 @@ class TokenHandler:
       method.
     """
 
-    def __init__(self, pr=None, vars: dict = {}, system="", user=""):
+    def __init__(self, vars: dict, system, user):
         """
         Initializes the TokenHandler object.
 
@@ -32,10 +32,9 @@ class TokenHandler:
         - user: The user string.
         """
         self.encoder = get_token_encoder()
-        if pr is not None:
+        self.prompt_tokens = self._get_system_user_tokens(self.encoder, vars, system, user)
-            self.prompt_tokens = self._get_system_user_tokens(pr, self.encoder, vars, system, user)
 
-    def _get_system_user_tokens(self, pr, encoder, vars: dict, system, user):
+    def _get_system_user_tokens(self, encoder, vars: dict, system, user):
         """
         Calculates the number of tokens in the system and user strings.
 

pr_agent/algo/utils.py

@@ -5,14 +5,24 @@ import json
 import logging
 import re
 import textwrap
+from dataclasses import dataclass
 from datetime import datetime
-from typing import Any, List
+from enum import Enum
+from typing import Any, List, Tuple, Optional
 
 import yaml
 from starlette_context import context
+
+from pr_agent.algo.token_handler import get_token_encoder
 from pr_agent.config_loader import get_settings, global_settings
 
 
+class EDIT_TYPE(Enum):
+    ADDED = 1
+    DELETED = 2
+    MODIFIED = 3
+    RENAMED = 4
+
 def get_setting(key: str) -> Any:
     try:
         key = key.upper()
@@ -20,7 +30,7 @@ def get_setting(key: str) -> Any:
     except Exception:
         return global_settings.get(key, None)
 
-def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
+def convert_to_markdown(output_data: dict) -> str:
     """
     Convert a dictionary of data into markdown format.
     Args:
@@ -42,7 +52,6 @@ def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
         "General suggestions": "💡",
         "Insights from user's answers": "📝",
         "Code feedback": "🤖",
-        "Estimated effort to review [1-5]": "⏱️",
     }
 
     for key, value in output_data.items():
@@ -50,14 +59,11 @@ def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
             continue
         if isinstance(value, dict):
             markdown_text += f"## {key}\n\n"
-            markdown_text += convert_to_markdown(value, gfm_supported)
+            markdown_text += convert_to_markdown(value)
         elif isinstance(value, list):
             emoji = emojis.get(key, "")
             if key.lower() == 'code feedback':
-                if gfm_supported:
                 markdown_text += f"\n\n- **<details><summary> { emoji } Code feedback:**</summary>\n\n"
-                else:
-                    markdown_text += f"\n\n- **{emoji} Code feedback:**\n\n"
             else:
                 markdown_text += f"- {emoji} **{key}:**\n\n"
             for item in value:
@@ -66,10 +72,7 @@ def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
                 elif item:
                     markdown_text += f"  - {item}\n"
             if key.lower() == 'code feedback':
-                if gfm_supported:
                 markdown_text += "</details>\n\n"
-                else:
-                    markdown_text += "\n\n"
         elif value != 'n/a':
             emoji = emojis.get(key, "")
             markdown_text += f"- {emoji} **{key}:** {value}\n"
@@ -301,3 +304,108 @@ def try_fix_yaml(review_text: str) -> dict:
         except:
             pass
     return data
+
+def clip_tokens(text: str, max_tokens: int) -> str:
+    """
+    Clip the number of tokens in a string to a maximum number of tokens.
+
+    Args:
+        text (str): The string to clip.
+        max_tokens (int): The maximum number of tokens allowed in the string.
+
+    Returns:
+        str: The clipped string.
+    """
+    if not text:
+        return text
+
+    try:
+        encoder = get_token_encoder()
+        num_input_tokens = len(encoder.encode(text))
+        if num_input_tokens <= max_tokens:
+            return text
+        num_chars = len(text)
+        chars_per_token = num_chars / num_input_tokens
+        num_output_chars = int(chars_per_token * max_tokens)
+        clipped_text = text[:num_output_chars]
+        return clipped_text
+    except Exception as e:
+        logging.warning(f"Failed to clip tokens: {e}")
+        return text
+
+
+@dataclass
+class FilePatchInfo:
+    base_file: str
+    head_file: str
+    patch: str
+    filename: str
+    tokens: int = -1
+    edit_type: EDIT_TYPE = EDIT_TYPE.MODIFIED
+    old_filename: str = None
+    language: Optional[str] = None
+
+
+def find_line_number_of_relevant_line_in_file(diff_files: List[FilePatchInfo],
+                                              relevant_file: str,
+                                              relevant_line_in_file: str) -> Tuple[int, int]:
+    """
+    Find the line number and absolute position of a relevant line in a file.
+
+    Args:
+        diff_files (List[FilePatchInfo]): A list of FilePatchInfo objects representing the patches of files.
+        relevant_file (str): The name of the file where the relevant line is located.
+        relevant_line_in_file (str): The content of the relevant line.
+
+    Returns:
+        Tuple[int, int]: A tuple containing the line number and absolute position of the relevant line in the file.
+    """
+    position = -1
+    absolute_position = -1
+    re_hunk_header = re.compile(
+        r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@[ ]?(.*)")
+
+    for file in diff_files:
+        if file.filename.strip() == relevant_file:
+            patch = file.patch
+            patch_lines = patch.splitlines()
+
+            # try to find the line in the patch using difflib, with some margin of error
+            matches_difflib: list[str | Any] = difflib.get_close_matches(relevant_line_in_file,
+                                                                         patch_lines, n=3, cutoff=0.93)
+            if len(matches_difflib) == 1 and matches_difflib[0].startswith('+'):
+                relevant_line_in_file = matches_difflib[0]
+
+            delta = 0
+            start1, size1, start2, size2 = 0, 0, 0, 0
+            for i, line in enumerate(patch_lines):
+                if line.startswith('@@'):
+                    delta = 0
+                    match = re_hunk_header.match(line)
+                    start1, size1, start2, size2 = map(int, match.groups()[:4])
+                elif not line.startswith('-'):
+                    delta += 1
+
+                if relevant_line_in_file in line and line[0] != '-':
+                    position = i
+                    absolute_position = start2 + delta - 1
+                    break
+
+            if position == -1 and relevant_line_in_file[0] == '+':
+                no_plus_line = relevant_line_in_file[1:].lstrip()
+                for i, line in enumerate(patch_lines):
+                    if line.startswith('@@'):
+                        delta = 0
+                        match = re_hunk_header.match(line)
+                        start1, size1, start2, size2 = map(int, match.groups()[:4])
+                    elif not line.startswith('-'):
+                        delta += 1
+
+                    if no_plus_line in line and line[0] != '-':
+                        # The model might add a '+' to the beginning of the relevant_line_in_file even if originally
+                        # it's a context line
+                        position = i
+                        absolute_position = start2 + delta - 1
+                        break
+    return position, absolute_position
+

pr_agent/cli.py

@@ -17,7 +17,6 @@ For example:
 - cli.py --pr_url=... improve
 - cli.py --pr_url=... ask "write me a poem about this PR"
 - cli.py --pr_url=... reflect
-- cli.py --issue_url=... similar_issue
 
 Supported commands:
 -review / review_pr - Add a review that includes a summary of the PR and specific suggestions for improvement.
@@ -38,21 +37,13 @@ Configuration:
 To edit any configuration parameter from 'configuration.toml', just add -config_path=<value>.
 For example: 'python cli.py --pr_url=... review --pr_reviewer.extra_instructions="focus on the file: ..."'
 """)
-    parser.add_argument('--pr_url', type=str, help='The URL of the PR to review', default=None)
+    parser.add_argument('--pr_url', type=str, help='The URL of the PR to review', required=True)
-    parser.add_argument('--issue_url', type=str, help='The URL of the Issue to review', default=None)
     parser.add_argument('command', type=str, help='The', choices=commands, default='review')
     parser.add_argument('rest', nargs=argparse.REMAINDER, default=[])
     args = parser.parse_args(inargs)
-    if not args.pr_url and not args.issue_url:
-        parser.print_help()
-        return
-
     logging.basicConfig(level=os.environ.get("LOGLEVEL", "INFO"))
     command = args.command.lower()
     get_settings().set("CONFIG.CLI_MODE", True)
-    if args.issue_url:
-        result = asyncio.run(PRAgent().handle_request(args.issue_url, command + " " + " ".join(args.rest)))
-    else:
     result = asyncio.run(PRAgent().handle_request(args.pr_url, command + " " + " ".join(args.rest)))
     if not result:
         parser.print_help()

pr_agent/git_providers/__init__.py

@@ -5,8 +5,6 @@ from pr_agent.git_providers.github_provider import GithubProvider
 from pr_agent.git_providers.gitlab_provider import GitLabProvider
 from pr_agent.git_providers.local_git_provider import LocalGitProvider
 from pr_agent.git_providers.azuredevops_provider import AzureDevopsProvider
-from pr_agent.git_providers.gerrit_provider import GerritProvider
-
 
 _GIT_PROVIDERS = {
     'github': GithubProvider,
@@ -14,8 +12,7 @@ _GIT_PROVIDERS = {
     'bitbucket': BitbucketProvider,
     'azure': AzureDevopsProvider,
     'codecommit': CodeCommitProvider,
-    'local' : LocalGitProvider,
+    'local' : LocalGitProvider
-    'gerrit': GerritProvider,
 }
 
 def get_git_provider():

pr_agent/git_providers/azuredevops_provider.py

@@ -13,11 +13,9 @@ try:
 except ImportError:
     AZURE_DEVOPS_AVAILABLE = False
 
-from ..algo.pr_processing import clip_tokens
 from ..config_loader import get_settings
-from ..algo.utils import load_large_diff
+from ..algo.utils import load_large_diff, FilePatchInfo, EDIT_TYPE, clip_tokens
 from ..algo.language_handler import is_valid_file
-from .git_provider import EDIT_TYPE, FilePatchInfo
 
 
 class AzureDevopsProvider:
@@ -38,8 +36,7 @@ class AzureDevopsProvider:
             self.set_pr(pr_url)
 
     def is_supported(self, capability: str) -> bool:
-        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels',
+        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels', 'remove_initial_comment']:
-                          'remove_initial_comment', 'gfm_markdown']:
             return False
         return True
 

pr_agent/git_providers/bitbucket_provider.py

@@ -7,9 +7,9 @@ import requests
 from atlassian.bitbucket import Cloud
 from starlette_context import context
 
-from ..algo.pr_processing import clip_tokens, find_line_number_of_relevant_line_in_file
 from ..config_loader import get_settings
-from .git_provider import FilePatchInfo, GitProvider
+from .git_provider import GitProvider
+from ..algo.utils import FilePatchInfo
 
 
 class BitbucketProvider(GitProvider):
@@ -36,8 +36,9 @@ class BitbucketProvider(GitProvider):
         self.incremental = incremental
         if pr_url:
             self.set_pr(pr_url)
-        self.bitbucket_comment_api_url = self.pr._BitbucketBase__data["links"]["comments"]["href"]
+        self.bitbucket_comment_api_url = self.pr._BitbucketBase__data["links"][
-        self.bitbucket_pull_request_api_url = self.pr._BitbucketBase__data["links"]['self']['href']
+            "comments"
+        ]["href"]
 
     def get_repo_settings(self):
         try:
@@ -101,7 +102,12 @@ class BitbucketProvider(GitProvider):
             return False
 
     def is_supported(self, capability: str) -> bool:
-        if capability in ['get_issue_comments', 'publish_inline_comments', 'get_labels', 'gfm_markdown']:
+        if capability in [
+            "get_issue_comments",
+            "create_inline_comment",
+            "publish_inline_comments",
+            "get_labels",
+        ]:
             return False
         return True
 
@@ -146,30 +152,17 @@ class BitbucketProvider(GitProvider):
         except Exception as e:
             logging.exception(f"Failed to remove temp comments, error: {e}")
 
-
+    def publish_inline_comment(
-    # funtion to create_inline_comment
+        self, comment: str, from_line: int, to_line: int, file: str
-    def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
+    ):
-        position, absolute_position = find_line_number_of_relevant_line_in_file(self.get_diff_files(), relevant_file.strip('`'), relevant_line_in_file)
+        payload = json.dumps(
-        if position == -1:
+            {
-            if get_settings().config.verbosity_level >= 2:
-                logging.info(f"Could not find position for {relevant_file} {relevant_line_in_file}")
-            subject_type = "FILE"
-        else:
-            subject_type = "LINE"
-        path = relevant_file.strip()
-        return dict(body=body, path=path, position=absolute_position) if subject_type == "LINE" else {}
-
-
-    def publish_inline_comment(self, comment: str, from_line: int, file: str):
-        payload = json.dumps( {
                 "content": {
                     "raw": comment,
                 },
-            "inline": {
+                "inline": {"to": from_line, "path": file},
-                "to": from_line,
+            }
-                "path": file
+        )
-            },
-        })
         response = requests.request(
             "POST", self.bitbucket_comment_api_url, data=payload, headers=self.headers
         )
@@ -177,7 +170,9 @@ class BitbucketProvider(GitProvider):
 
     def publish_inline_comments(self, comments: list[dict]):
         for comment in comments:
-            self.publish_inline_comment(comment['body'], comment['start_line'], comment['path'])
+            self.publish_inline_comment(
+                comment["body"], comment["start_line"], comment["line"], comment["path"]
+            )
 
     def get_title(self):
         return self.pr.title
@@ -245,21 +240,15 @@ class BitbucketProvider(GitProvider):
     def get_commit_messages(self):
         return ""  # not implemented yet
 
-    # bitbucket does not support labels
+    def publish_description(self, pr_title: str, pr_body: str):
-    def publish_description(self, pr_title: str, description: str):
+        pass
-        payload = json.dumps({
+    def create_inline_comment(
-            "description": description,
+        self, body: str, relevant_file: str, relevant_line_in_file: str
-            "title": pr_title
+    ):
-
+        pass
-            })
+
-
+    def publish_labels(self, labels):
-        response = requests.request("PUT", self.bitbucket_pull_request_api_url, headers=self.headers, data=payload)
-        return response
-
-    # bitbucket does not support labels
-    def publish_labels(self, pr_types: list):
         pass
 
-    # bitbucket does not support labels
     def get_labels(self):
         pass

pr_agent/git_providers/codecommit_client.py

@@ -1,6 +1,9 @@
+try:  # Allow this module to be imported without requiring boto3
     import boto3
     import botocore
-
+except ModuleNotFoundError:
+    boto3 = None
+    botocore = None
 
 class CodeCommitDifferencesResponse:
     """
@@ -54,22 +57,17 @@ class CodeCommitClient:
     def __init__(self):
         self.boto_client = None
 
-    def is_supported(self, capability: str) -> bool:
-        if capability in ["gfm_markdown"]:
-            return False
-        return True
-
     def _connect_boto_client(self):
         try:
             self.boto_client = boto3.client("codecommit")
         except Exception as e:
-            raise ValueError(f"Failed to connect to AWS CodeCommit: {e}") from e
+            raise ValueError(f"Failed to connect to AWS CodeCommit: {e}")
 
     def get_differences(self, repo_name: int, destination_commit: str, source_commit: str):
         """
         Get the differences between two commits in CodeCommit.
 
-        Args:
+        Parameters:
         - repo_name: Name of the repository
         - destination_commit: Commit hash you want to merge into (the "before" hash) (usually on the main or master branch)
         - source_commit: Commit hash of the code you are adding (the "after" branch)
@@ -78,8 +76,8 @@ class CodeCommitClient:
         - List of CodeCommitDifferencesResponse objects
 
         Boto3 Documentation:
-        - aws codecommit get-differences
+        aws codecommit get-differences
-        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_differences.html
+        https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_differences.html
         """
         if self.boto_client is None:
             self._connect_boto_client()
@@ -95,11 +93,7 @@ class CodeCommitClient:
             ):
                 differences.extend(page.get("differences", []))
         except botocore.exceptions.ClientError as e:
-            if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
+            raise ValueError(f"Failed to retrieve differences from CodeCommit PR #{self.pr_num}") from e
-                raise ValueError(f"CodeCommit cannot retrieve differences: Repository does not exist: {repo_name}") from e
-            raise ValueError(f"CodeCommit cannot retrieve differences for {source_commit}..{destination_commit}") from e
-        except Exception as e:
-            raise ValueError(f"CodeCommit cannot retrieve differences for {source_commit}..{destination_commit}") from e
 
         output = []
         for json in differences:
@@ -110,7 +104,7 @@ class CodeCommitClient:
         """
         Retrieve a file from CodeCommit.
 
-        Args:
+        Parameters:
         - repo_name: Name of the repository
         - file_path: Path to the file you are retrieving
         - sha_hash: Commit hash of the file you are retrieving
@@ -119,8 +113,8 @@ class CodeCommitClient:
         - File contents
 
         Boto3 Documentation:
-        - aws codecommit get_file
+        aws codecommit get_file
-        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_file.html
+        https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_file.html
         """
         if not file_path:
             return ""
@@ -131,8 +125,6 @@ class CodeCommitClient:
         try:
             response = self.boto_client.get_file(repositoryName=repo_name, commitSpecifier=sha_hash, filePath=file_path)
         except botocore.exceptions.ClientError as e:
-            if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
-                raise ValueError(f"CodeCommit cannot retrieve PR: Repository does not exist: {repo_name}") from e
             # if the file does not exist, but is flagged as optional, then return an empty string
             if optional and e.response["Error"]["Code"] == 'FileDoesNotExistException':
                 return ""
@@ -144,20 +136,19 @@ class CodeCommitClient:
 
         return response.get("fileContent", "")
 
-    def get_pr(self, repo_name: str, pr_number: int):
+    def get_pr(self, pr_number: int):
         """
         Get a information about a CodeCommit PR.
 
-        Args:
+        Parameters:
-        - repo_name: Name of the repository
         - pr_number: The PR number you are requesting
 
         Returns:
         - CodeCommitPullRequestResponse object
 
         Boto3 Documentation:
-        - aws codecommit get_pull_request
+        aws codecommit get_pull_request
-        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_pull_request.html
+        https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_pull_request.html
         """
         if self.boto_client is None:
             self._connect_boto_client()
@@ -167,8 +158,6 @@ class CodeCommitClient:
         except botocore.exceptions.ClientError as e:
             if e.response["Error"]["Code"] == 'PullRequestDoesNotExistException':
                 raise ValueError(f"CodeCommit cannot retrieve PR: PR number does not exist: {pr_number}") from e
-            if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
-                raise ValueError(f"CodeCommit cannot retrieve PR: Repository does not exist: {repo_name}") from e
             raise ValueError(f"CodeCommit cannot retrieve PR: {pr_number}: boto client error") from e
         except Exception as e:
             raise ValueError(f"CodeCommit cannot retrieve PR: {pr_number}") from e
@@ -178,88 +167,28 @@ class CodeCommitClient:
 
         return CodeCommitPullRequestResponse(response.get("pullRequest", {}))
 
-    def publish_description(self, pr_number: int, pr_title: str, pr_body: str):
+    def publish_comment(self, repo_name: str, pr_number: int, destination_commit: str, source_commit: str, comment: str):
-        """
-        Set the title and description on a pull request
-
-        Args:
-        - pr_number: the AWS CodeCommit pull request number
-        - pr_title: title of the pull request
-        - pr_body: body of the pull request
-
-        Returns:
-        - None
-
-        Boto3 Documentation:
-        - aws codecommit update_pull_request_title
-        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/update_pull_request_title.html
-        - aws codecommit update_pull_request_description
-        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/update_pull_request_description.html
-        """
-        if self.boto_client is None:
-            self._connect_boto_client()
-
-        try:
-            self.boto_client.update_pull_request_title(pullRequestId=str(pr_number), title=pr_title)
-            self.boto_client.update_pull_request_description(pullRequestId=str(pr_number), description=pr_body)
-        except botocore.exceptions.ClientError as e:
-            if e.response["Error"]["Code"] == 'PullRequestDoesNotExistException':
-                raise ValueError(f"PR number does not exist: {pr_number}") from e
-            if e.response["Error"]["Code"] == 'InvalidTitleException':
-                raise ValueError(f"Invalid title for PR number: {pr_number}") from e
-            if e.response["Error"]["Code"] == 'InvalidDescriptionException':
-                raise ValueError(f"Invalid description for PR number: {pr_number}") from e
-            if e.response["Error"]["Code"] == 'PullRequestAlreadyClosedException':
-                raise ValueError(f"PR is already closed: PR number: {pr_number}") from e
-            raise ValueError(f"Boto3 client error calling publish_description") from e
-        except Exception as e:
-            raise ValueError(f"Error calling publish_description") from e
-
-    def publish_comment(self, repo_name: str, pr_number: int, destination_commit: str, source_commit: str, comment: str, annotation_file: str = None, annotation_line: int = None):
         """
         Publish a comment to a pull request
 
-        Args:
+        Parameters:
         - repo_name: name of the repository
         - pr_number: number of the pull request
         - destination_commit: The commit hash you want to merge into (the "before" hash) (usually on the main or master branch)
         - source_commit: The commit hash of the code you are adding (the "after" branch)
-        - comment: The comment you want to publish
+        - pr_comment: comment
-        - annotation_file: The file you want to annotate (optional)
-        - annotation_line: The line number you want to annotate (optional)
-
-        Comment annotations for CodeCommit are different than GitHub.
-        CodeCommit only designates the starting line number for the comment.
-        It does not support the ending line number to highlight a range of lines.
 
         Returns:
         - None
 
         Boto3 Documentation:
-        - aws codecommit post_comment_for_pull_request
+        aws codecommit post_comment_for_pull_request
-        - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_pull_request.html
+        https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_pull_request.html
         """
         if self.boto_client is None:
             self._connect_boto_client()
 
         try:
-            # If the comment has code annotations,
-            # then set the file path and line number in the location dictionary
-            if annotation_file and annotation_line:
-                self.boto_client.post_comment_for_pull_request(
-                    pullRequestId=str(pr_number),
-                    repositoryName=repo_name,
-                    beforeCommitId=destination_commit,
-                    afterCommitId=source_commit,
-                    content=comment,
-                    location={
-                        "filePath": annotation_file,
-                        "filePosition": annotation_line,
-                        "relativeFileVersion": "AFTER",
-                    },
-                )
-            else:
-                # The comment does not have code annotations
             self.boto_client.post_comment_for_pull_request(
                 pullRequestId=str(pr_number),
                 repositoryName=repo_name,

pr_agent/git_providers/codecommit_provider.py

@@ -1,17 +1,15 @@
 import logging
 import os
-import re
 from collections import Counter
 from typing import List, Optional, Tuple
 from urllib.parse import urlparse
 
-from ..algo.language_handler import is_valid_file, language_extension_map
-from ..algo.pr_processing import clip_tokens
-from ..algo.utils import load_large_diff
-from ..config_loader import get_settings
-from .git_provider import EDIT_TYPE, FilePatchInfo, GitProvider, IncrementalPR
 from pr_agent.git_providers.codecommit_client import CodeCommitClient
 
+from ..algo.language_handler import is_valid_file, language_extension_map
+from ..algo.utils import EDIT_TYPE, FilePatchInfo, load_large_diff
+from .git_provider import GitProvider
+
 
 class PullRequestCCMimic:
     """
@@ -74,7 +72,6 @@ class CodeCommitProvider(GitProvider):
             "create_inline_comment",
             "publish_inline_comments",
             "get_labels",
-            "gfm_markdown"
         ]:
             return False
         return True
@@ -155,63 +152,26 @@ class CodeCommitProvider(GitProvider):
         return self.diff_files
 
     def publish_description(self, pr_title: str, pr_body: str):
-        try:
+        return ""  # not implemented yet
-            self.codecommit_client.publish_description(
-                pr_number=self.pr_num,
-                pr_title=pr_title,
-                pr_body=CodeCommitProvider._add_additional_newlines(pr_body),
-            )
-        except Exception as e:
-            raise ValueError(f"CodeCommit Cannot publish description for PR: {self.pr_num}") from e    
 
     def publish_comment(self, pr_comment: str, is_temporary: bool = False):
         if is_temporary:
             logging.info(pr_comment)
             return
 
-        pr_comment = CodeCommitProvider._remove_markdown_html(pr_comment)
-        pr_comment = CodeCommitProvider._add_additional_newlines(pr_comment)
-
         try:
             self.codecommit_client.publish_comment(
                 repo_name=self.repo_name,
-                pr_number=self.pr_num,
+                pr_number=str(self.pr_num),
                 destination_commit=self.pr.destination_commit,
                 source_commit=self.pr.source_commit,
                 comment=pr_comment,
             )
         except Exception as e:
-            raise ValueError(f"CodeCommit Cannot publish comment for PR: {self.pr_num}") from e
+            raise ValueError(f"CodeCommit Cannot post comment for PR: {self.pr_num}") from e
 
     def publish_code_suggestions(self, code_suggestions: list) -> bool:
-        counter = 1
+        return [""]  # not implemented yet
-        for suggestion in code_suggestions:
-            # Verify that each suggestion has the required keys
-            if not all(key in suggestion for key in ["body", "relevant_file", "relevant_lines_start"]):
-                logging.warning(f"Skipping code suggestion #{counter}: Each suggestion must have 'body', 'relevant_file', 'relevant_lines_start' keys")
-                continue
-       
-            # Publish the code suggestion to CodeCommit
-            try:
-                logging.debug(f"Code Suggestion #{counter} in file: {suggestion['relevant_file']}: {suggestion['relevant_lines_start']}")
-                self.codecommit_client.publish_comment(
-                    repo_name=self.repo_name,
-                    pr_number=self.pr_num,
-                    destination_commit=self.pr.destination_commit,
-                    source_commit=self.pr.source_commit,
-                    comment=suggestion["body"],
-                    annotation_file=suggestion["relevant_file"],
-                    annotation_line=suggestion["relevant_lines_start"],
-                )
-            except Exception as e:
-                raise ValueError(f"CodeCommit Cannot publish code suggestions for PR: {self.pr_num}") from e
-            
-            counter += 1
-
-        # The calling function passes in a list of code suggestions, and this function publishes each suggestion one at a time.
-        # If we were to return False here, the calling function will attempt to publish the same list of code suggestions again, one at a time.
-        # Since this function publishes the suggestions one at a time anyway, we always return True here to avoid the retry.        
-        return True
 
     def publish_labels(self, labels):
         return [""]  # not implemented yet
@@ -223,7 +183,6 @@ class CodeCommitProvider(GitProvider):
         return ""  # not implemented yet
 
     def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
-        # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_compared_commit.html
         raise NotImplementedError("CodeCommit provider does not support publishing inline comments yet")
 
     def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
@@ -240,7 +199,7 @@ class CodeCommitProvider(GitProvider):
         Returns a dictionary of languages, containing the percentage of each language used in the PR.
 
         Returns:
-        - dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR.
+            dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR.
         """
         commit_files = self.get_files()
         filenames = [ item.filename for item in commit_files ]
@@ -284,29 +243,18 @@ class CodeCommitProvider(GitProvider):
         return self.codecommit_client.get_file(self.repo_name, settings_filename, self.pr.source_commit, optional=True)
 
     def add_eyes_reaction(self, issue_comment_id: int) -> Optional[int]:
-        logging.info("CodeCommit provider does not support eyes reaction yet")
         return True
 
     def remove_reaction(self, issue_comment_id: int, reaction_id: int) -> bool:
-        logging.info("CodeCommit provider does not support removing reactions yet")
         return True
 
     @staticmethod
     def _parse_pr_url(pr_url: str) -> Tuple[str, int]:
-        """
-        Parse the CodeCommit PR URL and return the repository name and PR number.
-
-        Args:
-        - pr_url: the full AWS CodeCommit pull request URL
-
-        Returns:
-        - Tuple[str, int]: A tuple containing the repository name and PR number.
-        """
         # Example PR URL:
         # https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/__MY_REPO__/pull-requests/123456"
         parsed_url = urlparse(pr_url)
 
-        if not CodeCommitProvider._is_valid_codecommit_hostname(parsed_url.netloc):
+        if "us-east-1.console.aws.amazon.com" not in parsed_url.netloc:
             raise ValueError(f"The provided URL is not a valid CodeCommit URL: {pr_url}")
 
         path_parts = parsed_url.path.strip("/").split("/")
@@ -329,33 +277,17 @@ class CodeCommitProvider(GitProvider):
 
         return repo_name, pr_number
 
-    @staticmethod
-    def _is_valid_codecommit_hostname(hostname: str) -> bool:
-        """
-        Check if the provided hostname is a valid AWS CodeCommit hostname.
-
-        This is not an exhaustive check of AWS region names, 
-        but instead uses a regex to check for matching AWS region patterns.
-
-        Args:
-        - hostname: the hostname to check
-
-        Returns:
-        - bool: True if the hostname is valid, False otherwise.
-        """
-        return re.match(r"^[a-z]{2}-(gov-)?[a-z]+-\d\.console\.aws\.amazon\.com$", hostname) is not None
-
     def _get_pr(self):
-        response = self.codecommit_client.get_pr(self.repo_name, self.pr_num)
+        response = self.codecommit_client.get_pr(self.pr_num)
 
         if len(response.targets) == 0:
             raise ValueError(f"No files found in CodeCommit PR: {self.pr_num}")
 
-        # TODO: implement support for multiple targets in one CodeCommit PR
+        # TODO: implement support for multiple commits in one CodeCommit PR
-        #       for now, we are only using the first target in the PR
+        #       for now, we are only using the first commit in the PR
         if len(response.targets) > 1:
             logging.warning(
-                "Multiple targets in one PR is not supported for CodeCommit yet. Continuing, using the first target only..."
+                "Multiple commits in one PR is not supported for CodeCommit yet. Continuing, using the first commit only..."
             )
 
         # Return our object that mimics PullRequest class from the PyGithub library
@@ -373,52 +305,13 @@ class CodeCommitProvider(GitProvider):
         return ""  # not implemented yet
 
     @staticmethod
-    def _add_additional_newlines(body: str) -> str:
+    def _get_edit_type(codecommit_change_type):
-        """
-        Replace single newlines in a PR body with double newlines.
-
-        CodeCommit Markdown does not seem to render as well as GitHub Markdown,
-        so we add additional newlines to the PR body to make it more readable in CodeCommit.
-
-        Args:
-        - body: the PR body
-
-        Returns:
-        - str: the PR body with the double newlines added
-        """
-        return re.sub(r'(?<!\n)\n(?!\n)', '\n\n', body)
-
-    @staticmethod
-    def _remove_markdown_html(comment: str) -> str:
-        """
-        Remove the HTML tags from a PR comment.
-
-        CodeCommit Markdown does not seem to render as well as GitHub Markdown,
-        so we remove the HTML tags from the PR comment to make it more readable in CodeCommit.
-
-        Args:
-        - comment: the PR comment
-
-        Returns:
-        - str: the PR comment with the HTML tags removed
-        """
-        comment = comment.replace("<details>", "")
-        comment = comment.replace("</details>", "")
-        comment = comment.replace("<summary>", "")
-        comment = comment.replace("</summary>", "")
-        return comment
-
-    @staticmethod
-    def _get_edit_type(codecommit_change_type: str):
         """
         Convert the CodeCommit change type string to the EDIT_TYPE enum.
         The CodeCommit change type string is returned from the get_differences SDK method.
 
-        Args:
-        - codecommit_change_type: the CodeCommit change type string
-
         Returns:
-        - An EDIT_TYPE enum representing the modified, added, deleted, or renamed file in the PR diff.
+            An EDIT_TYPE enum representing the modified, added, deleted, or renamed file in the PR diff.
         """
         t = codecommit_change_type.upper()
         edit_type = None
@@ -439,12 +332,6 @@ class CodeCommitProvider(GitProvider):
         The returned extensions will include the dot "." prefix,
         to accommodate for the dots in the existing language_extension_map settings.
         Filenames with no extension will return an empty string for the extension.
-
-        Args:
-        - filenames: a list of filenames
-
-        Returns:
-        - list: A list of file extensions, including the dot "." prefix.
         """
         extensions = []
         for filename in filenames:
@@ -461,12 +348,6 @@ class CodeCommitProvider(GitProvider):
         Return a dictionary containing the programming language name (as the key),
         and the percentage that language is used (as the value),
         given a list of file extensions.
-
-        Args:
-        - extensions: a list of file extensions
-
-        Returns:
-        - dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR.
         """
         total_files = len(extensions)
         if total_files == 0:

pr_agent/git_providers/gerrit_provider.py

@@ -1,403 +0,0 @@
-import json
-import logging
-import os
-import pathlib
-import shutil
-import subprocess
-import uuid
-from collections import Counter, namedtuple
-from pathlib import Path
-from tempfile import mkdtemp, NamedTemporaryFile
-
-import requests
-import urllib3.util
-from git import Repo
-
-from pr_agent.config_loader import get_settings
-from pr_agent.git_providers.git_provider import GitProvider, FilePatchInfo, \
-    EDIT_TYPE
-from pr_agent.git_providers.local_git_provider import PullRequestMimic
-
-logger = logging.getLogger(__name__)
-
-
-def _call(*command, **kwargs) -> (int, str, str):
-    res = subprocess.run(
-        command,
-        stdout=subprocess.PIPE,
-        stderr=subprocess.PIPE,
-        check=True,
-        **kwargs,
-    )
-    return res.stdout.decode()
-
-
-def clone(url, directory):
-    logger.info("Cloning %s to %s", url, directory)
-    stdout = _call('git', 'clone', "--depth", "1", url, directory)
-    logger.info(stdout)
-
-
-def fetch(url, refspec, cwd):
-    logger.info("Fetching %s %s", url, refspec)
-    stdout = _call(
-        'git', 'fetch', '--depth', '2', url, refspec,
-        cwd=cwd
-    )
-    logger.info(stdout)
-
-
-def checkout(cwd):
-    logger.info("Checking out")
-    stdout = _call('git', 'checkout', "FETCH_HEAD", cwd=cwd)
-    logger.info(stdout)
-
-
-def show(*args, cwd=None):
-    logger.info("Show")
-    return _call('git', 'show', *args, cwd=cwd)
-
-
-def diff(*args, cwd=None):
-    logger.info("Diff")
-    patch = _call('git', 'diff', *args, cwd=cwd)
-    if not patch:
-        logger.warning("No changes found")
-        return
-    return patch
-
-
-def reset_local_changes(cwd):
-    logger.info("Reset local changes")
-    _call('git', 'checkout', "--force", cwd=cwd)
-
-
-def add_comment(url: urllib3.util.Url, refspec, message):
-    *_, patchset, changenum = refspec.rsplit("/")
-    message = "'" + message.replace("'", "'\"'\"'") + "'"
-    return _call(
-        "ssh",
-        "-p", str(url.port),
-        f"{url.auth}@{url.host}",
-        "gerrit", "review",
-        "--message", message,
-        # "--code-review", score,
-        f"{patchset},{changenum}",
-    )
-
-
-def list_comments(url: urllib3.util.Url, refspec):
-    *_, patchset, _ = refspec.rsplit("/")
-    stdout = _call(
-        "ssh",
-        "-p", str(url.port),
-        f"{url.auth}@{url.host}",
-        "gerrit", "query",
-        "--comments",
-        "--current-patch-set", patchset,
-        "--format", "JSON",
-    )
-    change_set, *_ = stdout.splitlines()
-    return json.loads(change_set)["currentPatchSet"]["comments"]
-
-
-def prepare_repo(url: urllib3.util.Url, project, refspec):
-    repo_url = (f"{url.scheme}://{url.auth}@{url.host}:{url.port}/{project}")
-
-    directory = pathlib.Path(mkdtemp())
-    clone(repo_url, directory),
-    fetch(repo_url, refspec, cwd=directory)
-    checkout(cwd=directory)
-    return directory
-
-
-def adopt_to_gerrit_message(message):
-    lines = message.splitlines()
-    buf = []
-    for line in lines:
-        # remove markdown formatting
-        line = (line.replace("*", "")
-                .replace("``", "`")
-                .replace("<details>", "")
-                .replace("</details>", "")
-                .replace("<summary>", "")
-                .replace("</summary>", ""))
-
-        line = line.strip()
-        if line.startswith('#'):
-            buf.append("\n" +
-                       line.replace('#', '').removesuffix(":").strip() +
-                       ":")
-            continue
-        elif line.startswith('-'):
-            buf.append(line.removeprefix('-').strip())
-            continue
-        else:
-            buf.append(line)
-    return "\n".join(buf).strip()
-
-
-def add_suggestion(src_filename, context: str, start, end: int):
-    with (
-        NamedTemporaryFile("w", delete=False) as tmp,
-        open(src_filename, "r") as src
-    ):
-        lines = src.readlines()
-        tmp.writelines(lines[:start - 1])
-        if context:
-            tmp.write(context)
-        tmp.writelines(lines[end:])
-
-    shutil.copy(tmp.name, src_filename)
-    os.remove(tmp.name)
-
-
-def upload_patch(patch, path):
-    patch_server_endpoint = get_settings().get(
-        'gerrit.patch_server_endpoint')
-    patch_server_token = get_settings().get(
-        'gerrit.patch_server_token')
-
-    response = requests.post(
-        patch_server_endpoint,
-        json={
-            "content": patch,
-            "path": path,
-        },
-        headers={
-            "Content-Type": "application/json",
-            "Authorization": f"Bearer {patch_server_token}",
-        }
-    )
-    response.raise_for_status()
-    patch_server_endpoint = patch_server_endpoint.rstrip("/")
-    return patch_server_endpoint + "/" + path
-
-
-class GerritProvider(GitProvider):
-
-    def __init__(self, key: str, incremental=False):
-        self.project, self.refspec = key.split(':')
-        assert self.project, "Project name is required"
-        assert self.refspec, "Refspec is required"
-        base_url = get_settings().get('gerrit.url')
-        assert base_url, "Gerrit URL is required"
-        user = get_settings().get('gerrit.user')
-        assert user, "Gerrit user is required"
-
-        parsed = urllib3.util.parse_url(base_url)
-        self.parsed_url = urllib3.util.parse_url(
-            f"{parsed.scheme}://{user}@{parsed.host}:{parsed.port}"
-        )
-
-        self.repo_path = prepare_repo(
-            self.parsed_url, self.project, self.refspec
-        )
-        self.repo = Repo(self.repo_path)
-        assert self.repo
-
-        self.pr = PullRequestMimic(self.get_pr_title(), self.get_diff_files())
-
-    def get_pr_title(self):
-        """
-        Substitutes the branch-name as the PR-mimic title.
-        """
-        return self.repo.branches[0].name
-
-    def get_issue_comments(self):
-        comments = list_comments(self.parsed_url, self.refspec)
-        Comments = namedtuple('Comments', ['reversed'])
-        Comment = namedtuple('Comment', ['body'])
-        return Comments([Comment(c['message']) for c in reversed(comments)])
-
-    def get_labels(self):
-        raise NotImplementedError(
-            'Getting labels is not implemented for the gerrit provider')
-
-    def add_eyes_reaction(self, issue_comment_id: int):
-        raise NotImplementedError(
-            'Adding reactions is not implemented for the gerrit provider')
-
-    def remove_reaction(self, issue_comment_id: int, reaction_id: int):
-        raise NotImplementedError(
-            'Removing reactions is not implemented for the gerrit provider')
-
-    def get_commit_messages(self):
-        return [self.repo.head.commit.message]
-
-    def get_repo_settings(self):
-        try:
-            with open(self.repo_path / ".pr_agent.toml", 'rb') as f:
-                contents = f.read()
-            return contents
-        except OSError:
-            return b""
-
-    def get_diff_files(self) -> list[FilePatchInfo]:
-        diffs = self.repo.head.commit.diff(
-            self.repo.head.commit.parents[0],  # previous commit
-            create_patch=True,
-            R=True
-        )
-
-        diff_files = []
-        for diff_item in diffs:
-            if diff_item.a_blob is not None:
-                original_file_content_str = (
-                    diff_item.a_blob.data_stream.read().decode('utf-8')
-                )
-            else:
-                original_file_content_str = ""  # empty file
-            if diff_item.b_blob is not None:
-                new_file_content_str = diff_item.b_blob.data_stream.read(). \
-                    decode('utf-8')
-            else:
-                new_file_content_str = ""  # empty file
-            edit_type = EDIT_TYPE.MODIFIED
-            if diff_item.new_file:
-                edit_type = EDIT_TYPE.ADDED
-            elif diff_item.deleted_file:
-                edit_type = EDIT_TYPE.DELETED
-            elif diff_item.renamed_file:
-                edit_type = EDIT_TYPE.RENAMED
-            diff_files.append(
-                FilePatchInfo(
-                    original_file_content_str,
-                    new_file_content_str,
-                    diff_item.diff.decode('utf-8'),
-                    diff_item.b_path,
-                    edit_type=edit_type,
-                    old_filename=None
-                    if diff_item.a_path == diff_item.b_path
-                    else diff_item.a_path
-                )
-            )
-        self.diff_files = diff_files
-        return diff_files
-
-    def get_files(self):
-        diff_index = self.repo.head.commit.diff(
-            self.repo.head.commit.parents[0],  # previous commit
-            R=True
-        )
-        # Get the list of changed files
-        diff_files = [item.a_path for item in diff_index]
-        return diff_files
-
-    def get_languages(self):
-        """
-        Calculate percentage of languages in repository. Used for hunk
-        prioritisation.
-        """
-        # Get all files in repository
-        filepaths = [Path(item.path) for item in
-                     self.repo.tree().traverse() if item.type == 'blob']
-        # Identify language by file extension and count
-        lang_count = Counter(
-            ext.lstrip('.') for filepath in filepaths for ext in
-            [filepath.suffix.lower()])
-        # Convert counts to percentages
-        total_files = len(filepaths)
-        lang_percentage = {lang: count / total_files * 100 for lang, count
-                           in lang_count.items()}
-        return lang_percentage
-
-    def get_pr_description_full(self):
-        return self.repo.head.commit.message
-
-    def get_user_id(self):
-        return self.repo.head.commit.author.email
-
-    def is_supported(self, capability: str) -> bool:
-        if capability in [
-            # 'get_issue_comments',
-            'create_inline_comment',
-            'publish_inline_comments',
-            'get_labels',
-            'gfm_markdown'
-        ]:
-            return False
-        return True
-
-    def split_suggestion(self, msg) -> tuple[str, str]:
-        is_code_context = False
-        description = []
-        context = []
-        for line in msg.splitlines():
-            if line.startswith('```suggestion'):
-                is_code_context = True
-                continue
-            if line.startswith('```'):
-                is_code_context = False
-                continue
-            if is_code_context:
-                context.append(line)
-            else:
-                description.append(
-                    line.replace('*', '')
-                )
-
-        return (
-            '\n'.join(description),
-            '\n'.join(context) + '\n' if context else ''
-        )
-
-    def publish_code_suggestions(self, code_suggestions: list):
-        msg = []
-        for suggestion in code_suggestions:
-            description, code = self.split_suggestion(suggestion['body'])
-            add_suggestion(
-                pathlib.Path(self.repo_path) / suggestion["relevant_file"],
-                code,
-                suggestion["relevant_lines_start"],
-                suggestion["relevant_lines_end"],
-            )
-            patch = diff(cwd=self.repo_path)
-            patch_id = uuid.uuid4().hex[0:4]
-            path = "/".join(["codium-ai", self.refspec, patch_id])
-            full_path = upload_patch(patch, path)
-            reset_local_changes(self.repo_path)
-            msg.append(f'* {description}\n{full_path}')
-
-        if msg:
-            add_comment(self.parsed_url, self.refspec, "\n".join(msg))
-            return True
-
-    def publish_comment(self, pr_comment: str, is_temporary: bool = False):
-        if not is_temporary:
-            msg = adopt_to_gerrit_message(pr_comment)
-            add_comment(self.parsed_url, self.refspec, msg)
-
-    def publish_description(self, pr_title: str, pr_body: str):
-        msg = adopt_to_gerrit_message(pr_body)
-        add_comment(self.parsed_url, self.refspec, pr_title + '\n' + msg)
-
-    def publish_inline_comments(self, comments: list[dict]):
-        raise NotImplementedError(
-            'Publishing inline comments is not implemented for the gerrit '
-            'provider')
-
-    def publish_inline_comment(self, body: str, relevant_file: str,
-                               relevant_line_in_file: str):
-        raise NotImplementedError(
-            'Publishing inline comments is not implemented for the gerrit '
-            'provider')
-
-    def create_inline_comment(self, body: str, relevant_file: str,
-                              relevant_line_in_file: str):
-        raise NotImplementedError(
-            'Creating inline comments is not implemented for the gerrit '
-            'provider')
-
-    def publish_labels(self, labels):
-        # Not applicable to the local git provider,
-        # but required by the interface
-        pass
-
-    def remove_initial_comment(self):
-        # remove repo, cloned in previous steps
-        # shutil.rmtree(self.repo_path)
-        pass
-
-    def get_pr_branch(self):
-        return self.repo.head

pr_agent/git_providers/git_provider.py

@@ -1,28 +1,10 @@
 import logging
 from abc import ABC, abstractmethod
-from dataclasses import dataclass
 
 # enum EDIT_TYPE (ADDED, DELETED, MODIFIED, RENAMED)
-from enum import Enum
 from typing import Optional
 
-
+from pr_agent.algo.utils import FilePatchInfo
-class EDIT_TYPE(Enum):
-    ADDED = 1
-    DELETED = 2
-    MODIFIED = 3
-    RENAMED = 4
-
-
-@dataclass
-class FilePatchInfo:
-    base_file: str
-    head_file: str
-    patch: str
-    filename: str
-    tokens: int = -1
-    edit_type: EDIT_TYPE = EDIT_TYPE.MODIFIED
-    old_filename: str = None
 
 
 class GitProvider(ABC):
@@ -86,11 +68,11 @@ class GitProvider(ABC):
     def get_pr_description_full(self) -> str:
         pass
 
-    def get_pr_description(self, *, full: bool = True) -> str:
+    def get_pr_description(self) -> str:
         from pr_agent.config_loader import get_settings
-        from pr_agent.algo.pr_processing import clip_tokens
+        from pr_agent.algo.utils import clip_tokens
         max_tokens = get_settings().get("CONFIG.MAX_DESCRIPTION_TOKENS", None)
-        description = self.get_pr_description_full() if full else self.get_user_description()
+        description = self.get_pr_description_full()
         if max_tokens:
             return clip_tokens(description, max_tokens)
         return description
@@ -132,10 +114,6 @@ def get_main_pr_language(languages, files) -> str:
     Get the main language of the commit. Return an empty string if cannot determine.
     """
     main_language_str = ""
-    if not languages:
-        logging.info("No languages detected")
-        return main_language_str
-
     try:
         top_language = max(languages, key=languages.get).lower()
 
@@ -165,6 +143,7 @@ def get_main_pr_language(languages, files) -> str:
                 most_common_extension == 'scala' and top_language == 'scala' or \
                 most_common_extension == 'kt' and top_language == 'kotlin' or \
                 most_common_extension == 'pl' and top_language == 'perl' or \
+                most_common_extension == 'swift' and top_language == 'swift' or \
                 most_common_extension == top_language:
             main_language_str = top_language
 

pr_agent/git_providers/github_provider.py

@@ -9,10 +9,9 @@ from github import AppAuthentication, Auth, Github, GithubException, Reaction
 from retry import retry
 from starlette_context import context
 
-from .git_provider import FilePatchInfo, GitProvider, IncrementalPR
+from .git_provider import GitProvider, IncrementalPR
 from ..algo.language_handler import is_valid_file
-from ..algo.utils import load_large_diff
+from ..algo.utils import load_large_diff, clip_tokens, find_line_number_of_relevant_line_in_file, FilePatchInfo
-from ..algo.pr_processing import find_line_number_of_relevant_line_in_file, clip_tokens
 from ..config_loader import get_settings
 from ..servers.utils import RateLimitExceeded
 
@@ -32,7 +31,7 @@ class GithubProvider(GitProvider):
         self.diff_files = None
         self.git_files = None
         self.incremental = incremental
-        if pr_url and 'pull' in pr_url:
+        if pr_url:
             self.set_pr(pr_url)
             self.last_commit_id = list(self.pr.get_commits())[-1]
 
@@ -309,35 +308,6 @@ class GithubProvider(GitProvider):
 
         return repo_name, pr_number
 
-    @staticmethod
-    def _parse_issue_url(issue_url: str) -> Tuple[str, int]:
-        parsed_url = urlparse(issue_url)
-
-        if 'github.com' not in parsed_url.netloc:
-            raise ValueError("The provided URL is not a valid GitHub URL")
-
-        path_parts = parsed_url.path.strip('/').split('/')
-        if 'api.github.com' in parsed_url.netloc:
-            if len(path_parts) < 5 or path_parts[3] != 'issues':
-                raise ValueError("The provided URL does not appear to be a GitHub ISSUE URL")
-            repo_name = '/'.join(path_parts[1:3])
-            try:
-                issue_number = int(path_parts[4])
-            except ValueError as e:
-                raise ValueError("Unable to convert issue number to integer") from e
-            return repo_name, issue_number
-
-        if len(path_parts) < 4 or path_parts[2] != 'issues':
-            raise ValueError("The provided URL does not appear to be a GitHub PR issue")
-
-        repo_name = '/'.join(path_parts[:2])
-        try:
-            issue_number = int(path_parts[3])
-        except ValueError as e:
-            raise ValueError("Unable to convert issue number to integer") from e
-
-        return repo_name, issue_number
-
     def _get_github_client(self):
         deployment_type = get_settings().get("GITHUB.DEPLOYMENT_TYPE", "user")
 

pr_agent/git_providers/gitlab_provider.py

@@ -7,10 +7,9 @@ import gitlab
 from gitlab import GitlabGetError
 
 from ..algo.language_handler import is_valid_file
-from ..algo.pr_processing import clip_tokens
+from ..algo.utils import load_large_diff, clip_tokens, EDIT_TYPE, FilePatchInfo
-from ..algo.utils import load_large_diff
 from ..config_loader import get_settings
-from .git_provider import EDIT_TYPE, FilePatchInfo, GitProvider
+from .git_provider import GitProvider
 
 logger = logging.getLogger()
 
@@ -43,7 +42,7 @@ class GitLabProvider(GitProvider):
         self.incremental = incremental
 
     def is_supported(self, capability: str) -> bool:
-        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'gfm_markdown']:
+        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments']:
             return False
         return True
 

pr_agent/git_providers/in_memory_provider.py

@@ -0,0 +1,79 @@
+import itertools
+from collections import Counter
+from typing import List, Optional
+
+from pr_agent.algo.utils import FilePatchInfo
+from pr_agent.git_providers.git_provider import GitProvider
+
+
+class InMemoryProvider(GitProvider):
+    def __init__(self, head_branch: str, target_branch: str, files: List[FilePatchInfo]):
+        self.head_branch = head_branch
+        self.target_branch = target_branch
+        self.files = files
+
+    def is_supported(self, capability: str) -> bool:
+        pass
+
+    def get_files(self) -> list[FilePatchInfo]:
+        return self.files
+
+    def get_diff_files(self) -> list[FilePatchInfo]:
+        return self.get_files()
+
+    def publish_description(self, pr_title: str, pr_body: str):
+        pass
+
+    def publish_comment(self, pr_comment: str, is_temporary: bool = False):
+        pass
+
+    def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
+        pass
+
+    def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
+        pass
+
+    def publish_inline_comments(self, comments: list[dict]):
+        pass
+
+    def publish_code_suggestions(self, code_suggestions: list) -> bool:
+        pass
+
+    def publish_labels(self, labels):
+        pass
+
+    def get_labels(self):
+        pass
+
+    def remove_initial_comment(self):
+        pass
+
+    def get_languages(self):
+        language_count = Counter(file.language for file in self.files)
+        return dict(language_count)
+
+    def get_pr_branch(self):
+        pass
+
+    def get_user_id(self):
+        pass
+
+    def get_pr_description_full(self) -> str:
+        pass
+
+    def get_issue_comments(self):
+        pass
+
+    def get_repo_settings(self):
+        pass
+
+    def add_eyes_reaction(self, issue_comment_id: int) -> Optional[int]:
+        pass
+
+    def remove_reaction(self, issue_comment_id: int, reaction_id: int) -> bool:
+        pass
+
+    def get_commit_messages(self):
+        pass
+
+

pr_agent/git_providers/local_git_provider.py

@@ -6,7 +6,8 @@ from typing import List
 from git import Repo
 
 from pr_agent.config_loader import _find_repository_root, get_settings
-from pr_agent.git_providers.git_provider import EDIT_TYPE, FilePatchInfo, GitProvider
+from pr_agent.git_providers.git_provider import GitProvider
+from pr_agent.algo.utils import EDIT_TYPE, FilePatchInfo
 
 
 class PullRequestMimic:
@@ -56,8 +57,7 @@ class LocalGitProvider(GitProvider):
             raise KeyError(f'Branch: {self.target_branch_name} does not exist')
 
     def is_supported(self, capability: str) -> bool:
-        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels',
+        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels']:
-                          'gfm_markdown']:
             return False
         return True
 

pr_agent/servers/gerrit_server.py

@@ -1,78 +0,0 @@
-import copy
-import logging
-import sys
-from enum import Enum
-from json import JSONDecodeError
-
-import uvicorn
-from fastapi import APIRouter, FastAPI, HTTPException
-from pydantic import BaseModel
-from starlette.middleware import Middleware
-from starlette_context import context
-from starlette_context.middleware import RawContextMiddleware
-
-from pr_agent.agent.pr_agent import PRAgent
-from pr_agent.config_loader import global_settings, get_settings
-
-logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
-router = APIRouter()
-
-
-class Action(str, Enum):
-    review = "review"
-    describe = "describe"
-    ask = "ask"
-    improve = "improve"
-    reflect = "reflect"
-    answer = "answer"
-
-
-class Item(BaseModel):
-    refspec: str
-    project: str
-    msg: str
-
-
-@router.post("/api/v1/gerrit/{action}")
-async def handle_gerrit_request(action: Action, item: Item):
-    logging.debug("Received a Gerrit request")
-    context["settings"] = copy.deepcopy(global_settings)
-
-    if action == Action.ask:
-        if not item.msg:
-            return HTTPException(
-                status_code=400,
-                detail="msg is required for ask command"
-            )
-    await PRAgent().handle_request(
-        f"{item.project}:{item.refspec}",
-        f"/{item.msg.strip()}"
-    )
-
-
-async def get_body(request):
-    try:
-        body = await request.json()
-    except JSONDecodeError as e:
-        logging.error("Error parsing request body", e)
-        return {}
-    return body
-
-
-@router.get("/")
-async def root():
-    return {"status": "ok"}
-
-
-def start():
-    # to prevent adding help messages with the output
-    get_settings().set("CONFIG.CLI_MODE", True)
-    middleware = [Middleware(RawContextMiddleware)]
-    app = FastAPI(middleware=middleware)
-    app.include_router(router)
-
-    uvicorn.run(app, host="0.0.0.0", port=3000)
-
-
-if __name__ == '__main__':
-    start()

pr_agent/servers/github_action_runner.py

@@ -12,8 +12,8 @@ async def run_action():
     # Get environment variables
     GITHUB_EVENT_NAME = os.environ.get('GITHUB_EVENT_NAME')
     GITHUB_EVENT_PATH = os.environ.get('GITHUB_EVENT_PATH')
-    OPENAI_KEY = os.environ.get('OPENAI_KEY') or os.environ.get('OPENAI.KEY')
+    OPENAI_KEY = os.environ.get('OPENAI_KEY')
-    OPENAI_ORG = os.environ.get('OPENAI_ORG') or os.environ.get('OPENAI.ORG')
+    OPENAI_ORG = os.environ.get('OPENAI_ORG')
     GITHUB_TOKEN = os.environ.get('GITHUB_TOKEN')
     get_settings().set("CONFIG.PUBLISH_OUTPUT_PROGRESS", False)
 
@@ -61,21 +61,12 @@ async def run_action():
         if action in ["created", "edited"]:
             comment_body = event_payload.get("comment", {}).get("body")
             if comment_body:
-                is_pr = False
+                pr_url = event_payload.get("issue", {}).get("pull_request", {}).get("url")
-                # check if issue is pull request
+                if pr_url:
-                if event_payload.get("issue", {}).get("pull_request"):
-                    url = event_payload.get("issue", {}).get("pull_request", {}).get("url")
-                    is_pr = True
-                else:
-                    url = event_payload.get("issue", {}).get("url")
-                if url:
                     body = comment_body.strip().lower()
                     comment_id = event_payload.get("comment", {}).get("id")
-                    provider = get_git_provider()(pr_url=url)
+                    provider = get_git_provider()(pr_url=pr_url)
-                    if is_pr:
+                    await PRAgent().handle_request(pr_url, body, notify=lambda: provider.add_eyes_reaction(comment_id))
-                        await PRAgent().handle_request(url, body, notify=lambda: provider.add_eyes_reaction(comment_id))
-                    else:
-                        await PRAgent().handle_request(url, body)
 
 
 if __name__ == '__main__':

pr_agent/servers/github_app.py

@@ -12,7 +12,6 @@ from starlette_context import context
 from starlette_context.middleware import RawContextMiddleware
 
 from pr_agent.agent.pr_agent import PRAgent
-from pr_agent.algo.utils import update_settings_from_args
 from pr_agent.config_loader import get_settings, global_settings
 from pr_agent.git_providers import get_git_provider
 from pr_agent.servers.utils import verify_signature
@@ -98,7 +97,6 @@ async def handle_request(body: Dict[str, Any], event: str):
             api_url = body["comment"]["pull_request_url"]
         else:
             return {}
-        logging.info(body)
         logging.info(f"Handling comment because of event={event} and action={action}")
         comment_id = body.get("comment", {}).get("id")
         provider = get_git_provider()(pr_url=api_url)
@@ -125,14 +123,8 @@ async def handle_request(body: Dict[str, Any], event: str):
                     return {}
             logging.info(f"Performing review because of event={event} and action={action}")
             for command in get_settings().github_app.pr_commands:
-                split_command = command.split(" ")
+                logging.info(f"Performing command: {command}")
-                command = split_command[0]
+                await agent.handle_request(api_url, command)
-                args = split_command[1:]
-                other_args = update_settings_from_args(args)
-                new_command = ' '.join([command] + other_args)
-                logging.info(body)
-                logging.info(f"Performing command: {new_command}")
-                await agent.handle_request(api_url, new_command)
 
     logging.info("event or action does not require handling")
     return {}

pr_agent/servers/gitlab_webhook.py

@@ -1,51 +1,21 @@
-import copy
-import json
 import logging
-import sys
 
 import uvicorn
 from fastapi import APIRouter, FastAPI, Request, status
 from fastapi.encoders import jsonable_encoder
 from fastapi.responses import JSONResponse
 from starlette.background import BackgroundTasks
-from starlette.middleware import Middleware
-from starlette_context import context
-from starlette_context.middleware import RawContextMiddleware
 
 from pr_agent.agent.pr_agent import PRAgent
-from pr_agent.config_loader import get_settings, global_settings
+from pr_agent.config_loader import get_settings
-from pr_agent.secret_providers import get_secret_provider
 
-logging.basicConfig(stream=sys.stdout, level=logging.INFO)
+app = FastAPI()
 router = APIRouter()
 
-secret_provider = get_secret_provider() if get_settings().get("CONFIG.SECRET_PROVIDER") else None
-
 
 @router.post("/webhook")
 async def gitlab_webhook(background_tasks: BackgroundTasks, request: Request):
-    if request.headers.get("X-Gitlab-Token") and secret_provider:
-        request_token = request.headers.get("X-Gitlab-Token")
-        secret = secret_provider.get_secret(request_token)
-        try:
-            secret_dict = json.loads(secret)
-            gitlab_token = secret_dict["gitlab_token"]
-            context["settings"] = copy.deepcopy(global_settings)
-            context["settings"].gitlab.personal_access_token = gitlab_token
-        except Exception as e:
-            logging.error(f"Failed to validate secret {request_token}: {e}")
-            return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
-    elif get_settings().get("GITLAB.SHARED_SECRET"):
-        secret = get_settings().get("GITLAB.SHARED_SECRET")
-        if not request.headers.get("X-Gitlab-Token") == secret:
-            return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
-    else:
-        return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
-    gitlab_token = get_settings().get("GITLAB.PERSONAL_ACCESS_TOKEN", None)
-    if not gitlab_token:
-        return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
     data = await request.json()
-    logging.info(json.dumps(data))
     if data.get('object_kind') == 'merge_request' and data['object_attributes'].get('action') in ['open', 'reopen']:
         logging.info(f"A merge request has been opened: {data['object_attributes'].get('title')}")
         url = data['object_attributes'].get('url')
@@ -58,18 +28,16 @@ async def gitlab_webhook(background_tasks: BackgroundTasks, request: Request):
             background_tasks.add_task(PRAgent().handle_request, url, body)
     return JSONResponse(status_code=status.HTTP_200_OK, content=jsonable_encoder({"message": "success"}))
 
-
-@router.get("/")
-async def root():
-    return {"status": "ok"}
-
 def start():
     gitlab_url = get_settings().get("GITLAB.URL", None)
     if not gitlab_url:
         raise ValueError("GITLAB.URL is not set")
+    gitlab_token = get_settings().get("GITLAB.PERSONAL_ACCESS_TOKEN", None)
+    if not gitlab_token:
+        raise ValueError("GITLAB.PERSONAL_ACCESS_TOKEN is not set")
     get_settings().config.git_provider = "gitlab"
-    middleware = [Middleware(RawContextMiddleware)]
+
-    app = FastAPI(middleware=middleware)
+    app = FastAPI()
     app.include_router(router)
 
     uvicorn.run(app, host="0.0.0.0", port=3000)

pr_agent/settings/.secrets_template.toml

@@ -16,10 +16,6 @@ key = ""  # Acquire through https://platform.openai.com
 #deployment_id = ""  # The deployment name you chose when you deployed the engine
 #fallback_deployments = []  # For each fallback model specified in configuration.toml in the [config] section, specify the appropriate deployment_id
 
-[pinecone]
-api_key = "..."
-environment = "gcp-starter"
-
 [anthropic]
 key = "" # Optional, uncomment if you want to use Anthropic. Acquire through https://www.anthropic.com/
 
@@ -28,14 +24,6 @@ key = "" # Optional, uncomment if you want to use Cohere. Acquire through https:
 
 [replicate]
 key = "" # Optional, uncomment if you want to use Replicate. Acquire through https://replicate.com/
-
-[huggingface]
-key = "" # Optional, uncomment if you want to use Huggingface Inference API. Acquire through https://huggingface.co/docs/api-inference/quicktour
-api_base = "" # the base url for your huggingface inference endpoint 
-
-[ollama]
-api_base = "" # the base url for your huggingface inference endpoint 
-
 [github]
 # ---- Set the following only for deployment type == "user"
 user_token = ""  # A GitHub personal access token with 'repo' scope.
@@ -55,12 +43,5 @@ webhook_secret = "<WEBHOOK SECRET>"  # Optional, may be commented out.
 personal_access_token = ""
 
 [bitbucket]
-# For Bitbucket personal/repository bearer token
+# Bitbucket personal bearer token
 bearer_token = ""
-
-# For Bitbucket app
-app_key = ""
-base_url = ""
-
-[litellm]
-LITELLM_TOKEN = "" # see https://docs.litellm.ai/docs/debugging/hosted_debugging for details and instructions on how to get a token

pr_agent/settings/configuration.toml

@@ -10,15 +10,14 @@ use_repo_settings_file=true
 ai_timeout=180
 max_description_tokens = 500
 max_commits_tokens = 500
+litellm_debugger=false
 secret_provider="google_cloud_storage"
-cli_mode=false
 
 [pr_reviewer] # /review #
 require_focused_review=false
 require_score_review=false
 require_tests_review=true
 require_security_review=true
-require_estimate_effort_to_review=true
 num_code_suggestions=4
 inline_code_comments = false
 ask_and_reflect=false
@@ -26,14 +25,10 @@ automatic_review=true
 extra_instructions = ""
 
 [pr_description] # /describe #
-publish_labels=true
 publish_description_as_comment=false
 add_original_user_description=false
 keep_original_user_title=false
 extra_instructions = ""
-# markers
-use_description_markers=false
-include_generated_by_header=true
 
 [pr_questions] # /ask #
 
@@ -90,26 +85,3 @@ polling_interval_seconds = 30
 # LocalGitProvider settings - uncomment to use paths other than default
 # description_path= "path/to/description.md"
 # review_path= "path/to/review.md"
-
-[gerrit]
-# endpoint to the gerrit service
-# url = "ssh://gerrit.example.com:29418"
-# user for gerrit authentication
-# user = "ai-reviewer"
-# patch server where patches will be saved
-# patch_server_endpoint = "http://127.0.0.1:5000/patch"
-# token to authenticate in the patch server
-# patch_server_token = ""
-
-[litellm]
-#use_client = false
-
-[pr_similar_issue]
-skip_comments = false
-force_update_dataset = false
-max_issues_to_scan = 500
-
-[pinecone]
-# fill and place in .secrets.toml
-#api_key = ...
-# environment = "gcp-starter"

pr_agent/settings/pr_code_suggestions_prompts.toml

@@ -68,17 +68,12 @@ Code suggestions:
       type: string
       description: |-
         a code snippet showing the relevant code lines from a '__new hunk__' section.
-        It must be contiguous, correctly formatted and indented, and without line numbers.
+        It must be continuous, correctly formatted and indented, and without line numbers.
-    relevant lines start:
+    relevant lines:
-      type: integer
+      type: string
       description: |-
-        The relevant line number from a '__new hunk__' section where the suggestion starts (inclusive).
+        the relevant lines from a '__new hunk__' section, in the format of 'start_line-end_line'.
-        Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above.
+        For example: '10-15'. They should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above.
-    relevant lines end:
-      type: integer
-      description: |-
-        The relevant line number from a '__new hunk__' section where the suggestion ends (inclusive).
-        Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above.
     improved code:
       type: string
       description: |-
@@ -95,8 +90,7 @@ Code suggestions:
         Add a docstring to func1()
     existing code: |-
         def func1():
-    relevant lines start: 12
+    relevant lines: '12-12'
-    relevant lines end: 12
     improved code: |-
         ...
 ```

pr_agent/settings/pr_reviewer_prompts.toml

@@ -85,14 +85,6 @@ PR Analysis:
       code diff changes are too scattered, then the PR is not focused. Explain
       your answer shortly.
 {%- endif %}
-{%- if require_estimate_effort_to_review %}
-  Estimated effort to review [1-5]:
-    type: string
-    description: >-
-      Estimate, on a scale of 1-5 (inclusive), the time and effort required to review this PR by an experienced and knowledgeable developer. 1 means short and easy review , 5 means long and hard review.
-      Take into account the size, complexity, quality, and the needed changes of the PR code diff.
-      Explain your answer shortly (1-2 sentences).
-{%- endif %}
 PR Feedback:
   General suggestions:
     type: string

pr_agent/tools/pr_code_suggestions.py

@@ -42,13 +42,10 @@ class PRCodeSuggestions:
             "extra_instructions": get_settings().pr_code_suggestions.extra_instructions,
             "commit_messages_str": self.git_provider.get_commit_messages(),
         }
-        self.token_handler = TokenHandler(self.git_provider.pr,
+        self.token_handler = TokenHandler(self.vars, get_settings().pr_code_suggestions_prompt.system,
-                                          self.vars,
-                                          get_settings().pr_code_suggestions_prompt.system,
                                           get_settings().pr_code_suggestions_prompt.user)
 
     async def run(self):
-        try:
         logging.info('Generating code suggestions for PR...')
         if get_settings().config.publish_output:
             self.git_provider.publish_comment("Preparing review...", is_temporary=True)
@@ -59,9 +56,6 @@ class PRCodeSuggestions:
             data = self._prepare_pr_code_suggestions()
         else:
             data = await retry_with_fallback_models(self._prepare_prediction_extended)
-            if (not data) or (not 'Code suggestions' in data):
-                logging.info('No code suggestions found for PR.')
-                return
 
         if (not self.is_extended and get_settings().pr_code_suggestions.rank_suggestions) or \
                 (self.is_extended and get_settings().pr_code_suggestions.rank_extended_suggestions):
@@ -73,8 +67,6 @@ class PRCodeSuggestions:
             self.git_provider.remove_initial_comment()
             logging.info('Pushing inline code suggestions...')
             self.push_inline_code_suggestions(data)
-        except Exception as e:
-            logging.error(f"Failed to generate code suggestions for PR, error: {e}")
 
     async def _prepare_prediction(self, model: str):
         logging.info('Getting PR diff...')
@@ -119,8 +111,11 @@ class PRCodeSuggestions:
                 if get_settings().config.verbosity_level >= 2:
                     logging.info(f"suggestion: {d}")
                 relevant_file = d['relevant file'].strip()
-                relevant_lines_start = int(d['relevant lines start'])  # absolute position
+                relevant_lines_str = d['relevant lines'].strip()
-                relevant_lines_end = int(d['relevant lines end'])
+                if ',' in relevant_lines_str:  # handling 'relevant lines': '181, 190' or '178-184, 188-194'
+                    relevant_lines_str = relevant_lines_str.split(',')[0]
+                relevant_lines_start = int(relevant_lines_str.split('-')[0])  # absolute position
+                relevant_lines_end = int(relevant_lines_str.split('-')[-1])
                 content = d['suggestion content']
                 new_code_snippet = d['improved code']
 

pr_agent/tools/pr_description.py

@@ -1,6 +1,5 @@
 import copy
 import json
-import re
 import logging
 from typing import List, Tuple
 
@@ -29,7 +28,6 @@ class PRDescription:
         self.main_pr_language = get_main_pr_language(
             self.git_provider.get_languages(), self.git_provider.get_files()
         )
-        self.pr_id = f"{self.git_provider.repo}/{self.git_provider.pr_num}"
 
         # Initialize the AI handler
         self.ai_handler = AiHandler()
@@ -38,7 +36,7 @@ class PRDescription:
         self.vars = {
             "title": self.git_provider.pr.title,
             "branch": self.git_provider.get_pr_branch(),
-            "description": self.git_provider.get_pr_description(full=False),
+            "description": self.git_provider.get_pr_description(),
             "language": self.main_pr_language,
             "diff": "",  # empty diff for initial calculation
             "extra_instructions": get_settings().pr_description.extra_instructions,
@@ -48,12 +46,8 @@ class PRDescription:
         self.user_description = self.git_provider.get_user_description()
     
         # Initialize the token handler
-        self.token_handler = TokenHandler(
+        self.token_handler = TokenHandler(self.vars, get_settings().pr_description_prompt.system,
-            self.git_provider.pr,
+                                          get_settings().pr_description_prompt.user)
-            self.vars,
-            get_settings().pr_description_prompt.system,
-            get_settings().pr_description_prompt.user,
-        )
     
         # Initialize patches_diff and prediction attributes
         self.patches_diff = None
@@ -63,44 +57,27 @@ class PRDescription:
         """
         Generates a PR description using an AI model and publishes it to the PR.
         """
-
+        logging.info('Generating a PR description...')
-        try:
-            logging.info(f"Generating a PR description {self.pr_id}")
         if get_settings().config.publish_output:
             self.git_provider.publish_comment("Preparing pr description...", is_temporary=True)
         
         await retry_with_fallback_models(self._prepare_prediction)
         
-            logging.info(f"Preparing answer {self.pr_id}")
+        logging.info('Preparing answer...')
-            if self.prediction:
+        pr_title, pr_body, pr_types, markdown_text = self._prepare_pr_answer()
-                self._prepare_data()
-            else:
-                return None
-
-            pr_labels = []
-            if get_settings().pr_description.publish_labels:
-                pr_labels = self._prepare_labels()
-
-            if get_settings().pr_description.use_description_markers:
-                pr_title, pr_body = self._prepare_pr_answer_with_markers()
-            else:
-                pr_title, pr_body,  = self._prepare_pr_answer()
-            full_markdown_description = f"## Title\n\n{pr_title}\n\n___\n{pr_body}"
         
         if get_settings().config.publish_output:
-                logging.info(f"Pushing answer {self.pr_id}")
+            logging.info('Pushing answer...')
             if get_settings().pr_description.publish_description_as_comment:
-                    self.git_provider.publish_comment(full_markdown_description)
+                self.git_provider.publish_comment(markdown_text)
             else:
                 self.git_provider.publish_description(pr_title, pr_body)
-                    if get_settings().pr_description.publish_labels and self.git_provider.is_supported("get_labels"):
+                if self.git_provider.is_supported("get_labels"):
                     current_labels = self.git_provider.get_labels()
                     if current_labels is None:
                         current_labels = []
-                        self.git_provider.publish_labels(pr_labels + current_labels)
+                    self.git_provider.publish_labels(pr_types + current_labels)
             self.git_provider.remove_initial_comment()
-        except Exception as e:
-            logging.error(f"Error generating PR description {self.pr_id}: {e}")
         
         return ""
 
@@ -118,12 +95,9 @@ class PRDescription:
             Any exceptions raised by the 'get_pr_diff' and '_get_prediction' functions.
 
         """
-        if get_settings().pr_description.use_description_markers and 'pr_agent:' not in self.user_description:
+        logging.info('Getting PR diff...')
-            return None
-
-        logging.info(f"Getting PR diff {self.pr_id}")
         self.patches_diff = get_pr_diff(self.git_provider, self.token_handler, model)
-        logging.info(f"Getting AI prediction {self.pr_id}")
+        logging.info('Getting AI prediction...')
         self.prediction = await self._get_prediction(model)
 
     async def _get_prediction(self, model: str) -> str:
@@ -156,71 +130,34 @@ class PRDescription:
 
         return response
 
-
+    def _prepare_pr_answer(self) -> Tuple[str, str, List[str], str]:
-    def _prepare_data(self):
-        # Load the AI prediction data into a dictionary
-        self.data = load_yaml(self.prediction.strip())
-
-        if get_settings().pr_description.add_original_user_description and self.user_description:
-            self.data["User Description"] = self.user_description
-
-
-    def _prepare_labels(self) -> List[str]:
-        pr_types = []
-
-        # If the 'PR Type' key is present in the dictionary, split its value by comma and assign it to 'pr_types'
-        if 'PR Type' in self.data:
-            if type(self.data['PR Type']) == list:
-                pr_types = self.data['PR Type']
-            elif type(self.data['PR Type']) == str:
-                pr_types = self.data['PR Type'].split(',')
-
-        return pr_types
-
-    def _prepare_pr_answer_with_markers(self) -> Tuple[str, str]:
-        logging.info(f"Using description marker replacements {self.pr_id}")
-        title = self.vars["title"]
-        body = self.user_description
-        if get_settings().pr_description.include_generated_by_header:
-            ai_header = f"### 🤖 Generated by PR Agent at {self.git_provider.last_commit_id.sha}\n\n"
-        else:
-            ai_header = ""
-
-        ai_summary = self.data.get('PR Description')
-        if ai_summary and not re.search(r'<!--\s*pr_agent:summary\s*-->', body):
-            summary = f"{ai_header}{ai_summary}"
-            body = body.replace('pr_agent:summary', summary)
-
-        if not re.search(r'<!--\s*pr_agent:walkthrough\s*-->', body):
-            ai_walkthrough = self.data.get('PR Main Files Walkthrough')
-            if ai_walkthrough:
-                walkthrough = str(ai_header)
-                for file in ai_walkthrough:
-                    filename = file['filename'].replace("'", "`")
-                    description = file['changes in file'].replace("'", "`")
-                    walkthrough += f'- `{filename}`: {description}\n'
-
-                body = body.replace('pr_agent:walkthrough', walkthrough)
-
-        return title, body
-
-    def _prepare_pr_answer(self) -> Tuple[str, str]:
         """
         Prepare the PR description based on the AI prediction data.
 
         Returns:
         - title: a string containing the PR title.
-        - pr_body: a string containing the PR description body in a markdown format.
+        - pr_body: a string containing the PR body in a markdown format.
+        - pr_types: a list of strings containing the PR types.
+        - markdown_text: a string containing the AI prediction data in a markdown format. used for publishing a comment
         """
+        # Load the AI prediction data into a dictionary
+        data = load_yaml(self.prediction.strip())
 
-        # Iterate over the dictionary items and append the key and value to 'markdown_text' in a markdown format
+        if get_settings().pr_description.add_original_user_description and self.user_description:
-        markdown_text = ""
+            data["User Description"] = self.user_description
-        for key, value in self.data.items():
+
-            markdown_text += f"## {key}\n\n"
+        # Initialization
-            markdown_text += f"{value}\n\n"
+        pr_types = []
+
+        # If the 'PR Type' key is present in the dictionary, split its value by comma and assign it to 'pr_types'
+        if 'PR Type' in data:
+            if type(data['PR Type']) == list:
+                pr_types = data['PR Type']
+            elif type(data['PR Type']) == str:
+                pr_types = data['PR Type'].split(',')
 
         # Remove the 'PR Title' key from the dictionary
-        ai_title = self.data.pop('PR Title', self.vars["title"])
+        ai_title = data.pop('PR Title')
         if get_settings().pr_description.keep_original_user_title:
             # Assign the original PR title to the 'title' variable
             title = self.vars["title"]
@@ -231,27 +168,25 @@ class PRDescription:
         # Iterate over the remaining dictionary items and append the key and value to 'pr_body' in a markdown format,
         # except for the items containing the word 'walkthrough'
         pr_body = ""
-        for idx, (key, value) in enumerate(self.data.items()):
+        for idx, (key, value) in enumerate(data.items()):
             pr_body += f"## {key}:\n"
             if 'walkthrough' in key.lower():
                 # for filename, description in value.items():
-                if self.git_provider.is_supported("gfm_markdown"):
-                    pr_body += "<details> <summary>files:</summary>\n\n"
                 for file in value:
                     filename = file['filename'].replace("'", "`")
                     description = file['changes in file']
                     pr_body += f'`{filename}`: {description}\n'
-                if self.git_provider.is_supported("gfm_markdown"):
-                    pr_body +="</details>\n"
             else:
                 # if the value is a list, join its items by comma
                 if type(value) == list:
                     value = ', '.join(v for v in value)
                 pr_body += f"{value}\n"
-            if idx < len(self.data) - 1:
+            if idx < len(data) - 1:
                 pr_body += "\n___\n"
 
+        markdown_text = f"## Title\n\n{title}\n\n___\n{pr_body}"
+
         if get_settings().config.verbosity_level >= 2:
             logging.info(f"title:\n{title}\n{pr_body}")
 
-        return title, pr_body
+        return title, pr_body, pr_types, markdown_text

pr_agent/tools/pr_information_from_user.py

@@ -26,9 +26,7 @@ class PRInformationFromUser:
             "diff": "",  # empty diff for initial calculation
             "commit_messages_str": self.git_provider.get_commit_messages(),
         }
-        self.token_handler = TokenHandler(self.git_provider.pr,
+        self.token_handler = TokenHandler(self.vars, get_settings().pr_information_from_user_prompt.system,
-                                          self.vars,
-                                          get_settings().pr_information_from_user_prompt.system,
                                           get_settings().pr_information_from_user_prompt.user)
         self.patches_diff = None
         self.prediction = None

pr_agent/tools/pr_questions.py

@@ -29,9 +29,7 @@ class PRQuestions:
             "questions": self.question_str,
             "commit_messages_str": self.git_provider.get_commit_messages(),
         }
-        self.token_handler = TokenHandler(self.git_provider.pr,
+        self.token_handler = TokenHandler(self.vars, get_settings().pr_questions_prompt.system,
-                                          self.vars,
-                                          get_settings().pr_questions_prompt.system,
                                           get_settings().pr_questions_prompt.user)
         self.patches_diff = None
         self.prediction = None

pr_agent/tools/pr_reviewer.py

@@ -9,8 +9,7 @@ from jinja2 import Environment, StrictUndefined
 from yaml import SafeLoader
 
 from pr_agent.algo.ai_handler import AiHandler
-from pr_agent.algo.pr_processing import get_pr_diff, retry_with_fallback_models, \
+from pr_agent.algo.pr_processing import get_pr_diff, retry_with_fallback_models
-    find_line_number_of_relevant_line_in_file, clip_tokens
 from pr_agent.algo.token_handler import TokenHandler
 from pr_agent.algo.utils import convert_to_markdown, try_fix_json, try_fix_yaml, load_yaml
 from pr_agent.config_loader import get_settings
@@ -59,7 +58,6 @@ class PRReviewer:
             "require_tests": get_settings().pr_reviewer.require_tests_review,
             "require_security": get_settings().pr_reviewer.require_security_review,
             "require_focused": get_settings().pr_reviewer.require_focused_review,
-            "require_estimate_effort_to_review": get_settings().pr_reviewer.require_estimate_effort_to_review,
             'num_code_suggestions': get_settings().pr_reviewer.num_code_suggestions,
             'question_str': question_str,
             'answer_str': answer_str,
@@ -67,12 +65,8 @@ class PRReviewer:
             "commit_messages_str": self.git_provider.get_commit_messages(),
         }
 
-        self.token_handler = TokenHandler(
+        self.token_handler = TokenHandler(self.vars, get_settings().pr_review_prompt.system,
-            self.git_provider.pr,
+                                          get_settings().pr_review_prompt.user)
-            self.vars,
-            get_settings().pr_review_prompt.system,
-            get_settings().pr_review_prompt.user
-        )
 
     def parse_args(self, args: List[str]) -> None:
         """
@@ -95,8 +89,6 @@ class PRReviewer:
         """
         Review the pull request and generate feedback.
         """
-
-        try:
         if self.is_auto and not get_settings().pr_reviewer.automatic_review:
             logging.info(f'Automatic review is disabled {self.pr_url}')
             return None
@@ -119,8 +111,6 @@ class PRReviewer:
             if get_settings().pr_reviewer.inline_code_comments:
                 logging.info('Pushing inline code comments...')
                 self._publish_inline_code_comments()
-        except Exception as e:
-            logging.error(f"Failed to review PR: {e}")
 
     async def _prepare_prediction(self, model: str) -> None:
         """
@@ -219,11 +209,11 @@ class PRReviewer:
                 "⏮️ Review for commits since previous PR-Agent review": f"Starting from commit {last_commit_url}"}})
             data.move_to_end('Incremental PR Review', last=False)
 
-        markdown_text = convert_to_markdown(data, self.git_provider.is_supported("gfm_markdown"))
+        markdown_text = convert_to_markdown(data)
         user = self.git_provider.get_user_id()
 
-        # Add help text if not in CLI mode
+        # Add help text if not in CLI§ mode
-        if not get_settings().get("CONFIG.CLI_MODE", False):
+        if not get_settings().get("CONFIG.CLI§_MODE", False):
             markdown_text += "\n### How to use\n"
             if user and '[bot]' not in user:
                 markdown_text += bot_help_text(user)

pr_agent/tools/pr_similar_issue.py

@@ -1,276 +0,0 @@
-import copy
-import json
-import logging
-from enum import Enum
-from typing import List, Tuple
-import pinecone
-import openai
-import pandas as pd
-from pydantic import BaseModel, Field
-
-from pr_agent.algo import MAX_TOKENS
-from pr_agent.algo.token_handler import TokenHandler
-from pr_agent.config_loader import get_settings
-from pr_agent.git_providers import get_git_provider
-from pinecone_datasets import Dataset, DatasetMetadata
-
-MODEL = "text-embedding-ada-002"
-
-
-class PRSimilarIssue:
-    def __init__(self, issue_url: str, args: list = None):
-        if get_settings().config.git_provider != "github":
-            raise Exception("Only github is supported for similar issue tool")
-
-        self.cli_mode = get_settings().CONFIG.CLI_MODE
-        self.max_issues_to_scan = get_settings().pr_similar_issue.max_issues_to_scan
-        self.issue_url = issue_url
-        self.git_provider = get_git_provider()()
-        repo_name, issue_number = self.git_provider._parse_issue_url(issue_url.split('=')[-1])
-        self.git_provider.repo = repo_name
-        self.git_provider.repo_obj = self.git_provider.github_client.get_repo(repo_name)
-        self.token_handler = TokenHandler()
-        repo_obj = self.git_provider.repo_obj
-        repo_name_for_index = self.repo_name_for_index = repo_obj.full_name.lower().replace('/', '-').replace('_/', '-')
-        index_name = self.index_name = "codium-ai-pr-agent-issues"
-
-        # assuming pinecone api key and environment are set in secrets file
-        try:
-            api_key = get_settings().pinecone.api_key
-            environment = get_settings().pinecone.environment
-        except Exception:
-            if not self.cli_mode:
-                repo_name, original_issue_number = self.git_provider._parse_issue_url(self.issue_url.split('=')[-1])
-                issue_main = self.git_provider.repo_obj.get_issue(original_issue_number)
-                issue_main.create_comment("Please set pinecone api key and environment in secrets file")
-            raise Exception("Please set pinecone api key and environment in secrets file")
-
-        # check if index exists, and if repo is already indexed
-        run_from_scratch = False
-        upsert = True
-        pinecone.init(api_key=api_key, environment=environment)
-        if not index_name in pinecone.list_indexes():
-            run_from_scratch = True
-            upsert = False
-        else:
-            if get_settings().pr_similar_issue.force_update_dataset:
-                upsert = True
-            else:
-                pinecone_index = pinecone.Index(index_name=index_name)
-                res = pinecone_index.fetch([f"example_issue_{repo_name_for_index}"]).to_dict()
-                if res["vectors"]:
-                    upsert = False
-
-        if run_from_scratch or upsert:  # index the entire repo
-            logging.info('Indexing the entire repo...')
-
-            logging.info('Getting issues...')
-            issues = list(repo_obj.get_issues(state='all'))
-            logging.info('Done')
-            self._update_index_with_issues(issues, repo_name_for_index, upsert=upsert)
-        else:  # update index if needed
-            pinecone_index = pinecone.Index(index_name=index_name)
-            issues_to_update = []
-            issues_paginated_list = repo_obj.get_issues(state='all')
-            counter = 1
-            for issue in issues_paginated_list:
-                if issue.pull_request:
-                    continue
-                issue_str, comments, number = self._process_issue(issue)
-                issue_key = f"issue_{number}"
-                id = issue_key + "." + "issue"
-                res = pinecone_index.fetch([id]).to_dict()
-                is_new_issue = True
-                for vector in res["vectors"].values():
-                    if vector['metadata']['repo'] == repo_name_for_index:
-                        is_new_issue = False
-                        break
-                if is_new_issue:
-                    counter += 1
-                    issues_to_update.append(issue)
-                else:
-                    break
-
-            if issues_to_update:
-                logging.info(f'Updating index with {counter} new issues...')
-                self._update_index_with_issues(issues_to_update, repo_name_for_index, upsert=True)
-            else:
-                logging.info('No new issues to update')
-
-    async def run(self):
-        logging.info('Getting issue...')
-        repo_name, original_issue_number = self.git_provider._parse_issue_url(self.issue_url.split('=')[-1])
-        issue_main = self.git_provider.repo_obj.get_issue(original_issue_number)
-        issue_str, comments, number = self._process_issue(issue_main)
-        openai.api_key = get_settings().openai.key
-        logging.info('Done')
-
-        logging.info('Querying...')
-        res = openai.Embedding.create(input=[issue_str], engine=MODEL)
-        embeds = [record['embedding'] for record in res['data']]
-        pinecone_index = pinecone.Index(index_name=self.index_name)
-        res = pinecone_index.query(embeds[0],
-                                   top_k=5,
-                                   filter={"repo": self.repo_name_for_index},
-                                   include_metadata=True).to_dict()
-        relevant_issues_number_list = []
-        relevant_comment_number_list = []
-        score_list = []
-        for r in res['matches']:
-            issue_number = int(r["id"].split('.')[0].split('_')[-1])
-            if original_issue_number == issue_number:
-                continue
-            if issue_number not in relevant_issues_number_list:
-                relevant_issues_number_list.append(issue_number)
-            if 'comment' in r["id"]:
-                relevant_comment_number_list.append(int(r["id"].split('.')[1].split('_')[-1]))
-            else:
-                relevant_comment_number_list.append(-1)
-            score_list.append(str("{:.2f}".format(r['score'])))
-        logging.info('Done')
-
-        logging.info('Publishing response...')
-        similar_issues_str = "### Similar Issues\n___\n\n"
-        for i, issue_number_similar in enumerate(relevant_issues_number_list):
-            issue = self.git_provider.repo_obj.get_issue(issue_number_similar)
-            title = issue.title
-            url = issue.html_url
-            if relevant_comment_number_list[i] != -1:
-                url = list(issue.get_comments())[relevant_comment_number_list[i]].html_url
-            similar_issues_str += f"{i + 1}. **[{title}]({url})** (score={score_list[i]})\n\n"
-        if get_settings().config.publish_output:
-            response = issue_main.create_comment(similar_issues_str)
-        logging.info(similar_issues_str)
-        logging.info('Done')
-
-    def _process_issue(self, issue):
-        header = issue.title
-        body = issue.body
-        number = issue.number
-        if get_settings().pr_similar_issue.skip_comments:
-            comments = []
-        else:
-            comments = list(issue.get_comments())
-        issue_str = f"Issue Header: \"{header}\"\n\nIssue Body:\n{body}"
-        return issue_str, comments, number
-
-    def _update_index_with_issues(self, issues_list, repo_name_for_index, upsert=False):
-        logging.info('Processing issues...')
-        corpus = Corpus()
-        example_issue_record = Record(
-            id=f"example_issue_{repo_name_for_index}",
-            text="example_issue",
-            metadata=Metadata(repo=repo_name_for_index)
-        )
-        corpus.append(example_issue_record)
-
-        counter = 0
-        for issue in issues_list:
-            if issue.pull_request:
-                continue
-
-            counter += 1
-            if counter % 100 == 0:
-                logging.info(f"Scanned {counter} issues")
-            if counter >= self.max_issues_to_scan:
-                logging.info(f"Scanned {self.max_issues_to_scan} issues, stopping")
-                break
-
-            issue_str, comments, number = self._process_issue(issue)
-            issue_key = f"issue_{number}"
-            username = issue.user.login
-            created_at = str(issue.created_at)
-            if len(issue_str) < 8000 or \
-                    self.token_handler.count_tokens(issue_str) < MAX_TOKENS[MODEL]:  # fast reject first
-                issue_record = Record(
-                    id=issue_key + "." + "issue",
-                    text=issue_str,
-                    metadata=Metadata(repo=repo_name_for_index,
-                                      username=username,
-                                      created_at=created_at,
-                                      level=IssueLevel.ISSUE)
-                )
-                corpus.append(issue_record)
-                if comments:
-                    for j, comment in enumerate(comments):
-                        comment_body = comment.body
-                        num_words_comment = len(comment_body.split())
-                        if num_words_comment < 10 or not isinstance(comment_body, str):
-                            continue
-
-                        if len(comment_body) < 8000 or \
-                                self.token_handler.count_tokens(comment_body) < MAX_TOKENS[MODEL]:
-                            comment_record = Record(
-                                id=issue_key + ".comment_" + str(j + 1),
-                                text=comment_body,
-                                metadata=Metadata(repo=repo_name_for_index,
-                                                  username=username,  # use issue username for all comments
-                                                  created_at=created_at,
-                                                  level=IssueLevel.COMMENT)
-                            )
-                            corpus.append(comment_record)
-        df = pd.DataFrame(corpus.dict()["documents"])
-        logging.info('Done')
-
-        logging.info('Embedding...')
-        openai.api_key = get_settings().openai.key
-        list_to_encode = list(df["text"].values)
-        try:
-            res = openai.Embedding.create(input=list_to_encode, engine=MODEL)
-            embeds = [record['embedding'] for record in res['data']]
-        except:
-            embeds = []
-            logging.error('Failed to embed entire list, embedding one by one...')
-            for i, text in enumerate(list_to_encode):
-                try:
-                    res = openai.Embedding.create(input=[text], engine=MODEL)
-                    embeds.append(res['data'][0]['embedding'])
-                except:
-                    embeds.append([0] * 1536)
-        df["values"] = embeds
-        meta = DatasetMetadata.empty()
-        meta.dense_model.dimension = len(embeds[0])
-        ds = Dataset.from_pandas(df, meta)
-        logging.info('Done')
-
-        api_key = get_settings().pinecone.api_key
-        environment = get_settings().pinecone.environment
-        if not upsert:
-            logging.info('Creating index from scratch...')
-            ds.to_pinecone_index(self.index_name, api_key=api_key, environment=environment)
-        else:
-            logging.info('Upserting index...')
-            namespace = ""
-            batch_size: int = 100
-            concurrency: int = 10
-            pinecone.init(api_key=api_key, environment=environment)
-            ds._upsert_to_index(self.index_name, namespace, batch_size, concurrency)
-        logging.info('Done')
-
-
-class IssueLevel(str, Enum):
-    ISSUE = "issue"
-    COMMENT = "comment"
-
-
-class Metadata(BaseModel):
-    repo: str
-    username: str = Field(default="@codium")
-    created_at: str = Field(default="01-01-1970 00:00:00.00000")
-    level: IssueLevel = Field(default=IssueLevel.ISSUE)
-
-    class Config:
-        use_enum_values = True
-
-
-class Record(BaseModel):
-    id: str
-    text: str
-    metadata: Metadata
-
-
-class Corpus(BaseModel):
-    documents: List[Record] = Field(default=[])
-
-    def append(self, r: Record):
-        self.documents.append(r)

pr_agent/tools/pr_update_changelog.py

@@ -40,13 +40,11 @@ class PRUpdateChangelog:
             "extra_instructions": get_settings().pr_update_changelog.extra_instructions,
             "commit_messages_str": self.git_provider.get_commit_messages(),
         }
-        self.token_handler = TokenHandler(self.git_provider.pr,
+        self.token_handler = TokenHandler(self.vars, get_settings().pr_update_changelog_prompt.system,
-                                          self.vars,
-                                          get_settings().pr_update_changelog_prompt.system,
                                           get_settings().pr_update_changelog_prompt.user)
 
     async def run(self):
-        # assert type(self.git_provider) == GithubProvider, "Currently only Github is supported"
+        assert type(self.git_provider) == GithubProvider, "Currently only Github is supported"
 
         logging.info('Updating the changelog...')
         if get_settings().config.publish_output:

pyproject.toml

@@ -35,12 +35,12 @@ dependencies = {file = ["requirements.txt"]}
 "Homepage" = "https://github.com/Codium-ai/pr-agent"
 
 [tool.setuptools]
-include-package-data = false
+include-package-data = true
 license-files = ["LICENSE"]
 
 [tool.setuptools.packages.find]
 where = ["."]
-include = ["pr_agent"]
+include = ["pr_agent", "pr_agent.*"]
 
 [project.scripts]
 pr-agent = "pr_agent.cli:run"

requirements.txt

@@ -1,23 +1,19 @@
-dynaconf==3.1.12
+dynaconf~=3.1.12
-fastapi==0.99.0
+fastapi~=0.103.0
-PyGithub==1.59.*
+PyGithub~=1.59.0
-retry==0.9.2
+retry~=0.9.2
-openai==0.27.8
+openai~=0.27.8
-Jinja2==3.1.2
+Jinja2~=3.1.2
-tiktoken==0.4.0
+tiktoken~=0.4.0
-uvicorn==0.22.0
+uvicorn~=0.22.0
-python-gitlab==3.15.0
+python-gitlab~=3.15.0
-pytest==7.4.0
+pytest~=7.4.0
-aiohttp==3.8.4
+aiohttp~=3.8.4
-atlassian-python-api==3.39.0
+atlassian-python-api~=3.39.0
-GitPython==3.1.32
+GitPython~=3.1.32
-PyYAML==6.0
+PyYAML~=6.0
-starlette-context==0.3.6
+starlette-context~=0.3.6
-litellm~=0.1.574
+litellm~=0.1.445
-boto3==1.28.25
+boto3~=1.28.25
-google-cloud-storage==2.10.0
+google-cloud-storage~=2.10.0
-ujson==5.8.0
+ujson~=5.8.0
-azure-devops==7.1.0b3
-msrest==0.7.1
-pinecone-client
-pinecone-datasets @ git+https://github.com/mrT23/pinecone-datasets.git@main

tests/unittest/test_codecommit_client.py

@@ -125,7 +125,7 @@ class TestCodeCommitProvider:
             }
         }
 
-        pr = api.get_pr("my_test_repo", 321)
+        pr = api.get_pr(321)
 
         assert pr.title == "My PR"
         assert pr.description == "My PR description"

tests/unittest/test_codecommit_provider.py

@@ -1,7 +1,7 @@
 import pytest
 from pr_agent.git_providers.codecommit_provider import CodeCommitFile
 from pr_agent.git_providers.codecommit_provider import CodeCommitProvider
-from pr_agent.git_providers.git_provider import EDIT_TYPE
+from pr_agent.algo.utils import EDIT_TYPE
 
 
 class TestCodeCommitFile:
@@ -26,48 +26,11 @@ class TestCodeCommitFile:
 
 class TestCodeCommitProvider:
     def test_parse_pr_url(self):
-        # Test that the _parse_pr_url() function can extract the repo name and PR number from a CodeCommit URL
         url = "https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/my_test_repo/pull-requests/321"
         repo_name, pr_number = CodeCommitProvider._parse_pr_url(url)
         assert repo_name == "my_test_repo"
         assert pr_number == 321
 
-    def test_is_valid_codecommit_hostname(self):
-        # Test the various AWS regions
-        assert CodeCommitProvider._is_valid_codecommit_hostname("af-south-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-east-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-2.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-3.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-south-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-south-2.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-2.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-3.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-4.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("ca-central-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-central-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-central-2.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-north-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-south-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-south-2.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-2.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-3.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("il-central-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("me-central-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("me-south-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("sa-east-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("us-east-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("us-east-2.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("us-gov-east-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("us-gov-west-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("us-west-1.console.aws.amazon.com")
-        assert CodeCommitProvider._is_valid_codecommit_hostname("us-west-2.console.aws.amazon.com")
-        # Test non-AWS regions
-        assert not CodeCommitProvider._is_valid_codecommit_hostname("no-such-region.console.aws.amazon.com")
-        assert not CodeCommitProvider._is_valid_codecommit_hostname("console.aws.amazon.com")
-
     # Test that an error is raised when an invalid CodeCommit URL is provided to the set_pr() method of the CodeCommitProvider class.
     # Generated by CodiumAI
     def test_invalid_codecommit_url(self):
@@ -143,7 +106,6 @@ class TestCodeCommitProvider:
         assert percentages == {}
 
     def test_get_edit_type(self):
-        # Test that the _get_edit_type() function can convert a CodeCommit letter to an EDIT_TYPE enum
         assert CodeCommitProvider._get_edit_type("A") == EDIT_TYPE.ADDED
         assert CodeCommitProvider._get_edit_type("D") == EDIT_TYPE.DELETED
         assert CodeCommitProvider._get_edit_type("M") == EDIT_TYPE.MODIFIED
@@ -155,18 +117,3 @@ class TestCodeCommitProvider:
         assert CodeCommitProvider._get_edit_type("r") == EDIT_TYPE.RENAMED
 
         assert CodeCommitProvider._get_edit_type("X") is None
-
-    def test_add_additional_newlines(self):
-        # a short string to test adding double newlines
-        input = "abc\ndef\n\n___\nghi\njkl\nmno\n\npqr\n"
-        expect = "abc\n\ndef\n\n___\n\nghi\n\njkl\n\nmno\n\npqr\n\n"
-        assert CodeCommitProvider._add_additional_newlines(input) == expect
-        # a test example from a real PR
-        input = "## PR Type:\nEnhancement\n\n___\n## PR Description:\nThis PR introduces a new feature to the script, allowing users to filter servers by name.\n\n___\n## PR Main Files Walkthrough:\n`foo`: The foo script has been updated to include a new command line option `-f` or `--filter`.\n`bar`: The bar script has been updated to list stopped servers.\n"
-        expect = "## PR Type:\n\nEnhancement\n\n___\n\n## PR Description:\n\nThis PR introduces a new feature to the script, allowing users to filter servers by name.\n\n___\n\n## PR Main Files Walkthrough:\n\n`foo`: The foo script has been updated to include a new command line option `-f` or `--filter`.\n\n`bar`: The bar script has been updated to list stopped servers.\n\n"
-        assert CodeCommitProvider._add_additional_newlines(input) == expect
-
-    def test_remove_markdown_html(self):
-        input = "## PR Feedback\n<details><summary>Code feedback:</summary>\nfile foo\n</summary>\n"
-        expect = "## PR Feedback\nCode feedback:\nfile foo\n\n"
-        assert CodeCommitProvider._remove_markdown_html(input) == expect

tests/unittest/test_find_line_number_of_relevant_line_in_file.py

@@ -1,8 +1,6 @@
 
 # Generated by CodiumAI
-from pr_agent.git_providers.git_provider import FilePatchInfo
+from pr_agent.algo.utils import FilePatchInfo, find_line_number_of_relevant_line_in_file
-from pr_agent.algo.pr_processing import find_line_number_of_relevant_line_in_file
-
 
 import pytest
 

tests/unittest/test_language_handler.py

@@ -61,7 +61,7 @@ class TestSortFilesByMainLanguages:
             type('', (object,), {'filename': 'file1.py'})(),
             type('', (object,), {'filename': 'file2.java'})()
         ]
-        expected_output = [{'language': 'Other', 'files': files}]
+        expected_output = [{'language': 'Other', 'files': []}]
         assert sort_files_by_main_languages(languages, files) == expected_output
 
     # Tests that function handles empty files list