Compare commits

..

9 Commits
v0.7 ... pre_pr

52 changed files with 633 additions and 2140 deletions

View File

@ -24,7 +24,4 @@ jobs:
OPENAI_KEY: ${{ secrets.OPENAI_KEY }} OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
OPENAI_ORG: ${{ secrets.OPENAI_ORG }} # optional OPENAI_ORG: ${{ secrets.OPENAI_ORG }} # optional
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PINECONE.API_KEY: ${{ secrets.PINECONE_API_KEY }}
PINECONE.ENVIRONMENT: ${{ secrets.PINECONE_ENVIRONMENT }}

57
CONFIGURATION.md Normal file
View File

@ -0,0 +1,57 @@
## Configuration
The different tools and sub-tools used by CodiumAI PR-Agent are adjustable via the **[configuration file](pr_agent/settings/configuration.toml)**
### Working from CLI
When running from source (CLI), your local configuration file will be initially used.
Example for invoking the 'review' tools via the CLI:
```
python cli.py --pr-url=<pr_url> review
```
In addition to general configurations, the 'review' tool will use parameters from the `[pr_reviewer]` section (every tool has a dedicated section in the configuration file).
Note that you can print results locally, without publishing them, by setting in `configuration.toml`:
```
[config]
publish_output=true
verbosity_level=2
```
This is useful for debugging or experimenting with the different tools.
### Working from pre-built repo (GitHub Action/GitHub App/Docker)
When running PR-Agent from a pre-built repo, the default configuration file will be loaded.
To edit the configuration, you have two options:
1. Place a local configuration file in the root of your local repo. The local file will be used instead of the default one.
2. For online usage, just add `--config_path=<value>` to you command, to edit a specific configuration value.
For example if you want to edit `pr_reviewer` configurations, you can run:
```
/review --pr_reviewer.extra_instructions="..." --pr_reviewer.require_score_review=false ...
```
Any configuration value in `configuration.toml` file can be similarly edited.
### General configuration parameters
#### Changing a model
See [here](pr_agent/algo/__init__.py) for the list of available models.
To use Llama2 model, for example, set:
```
[config]
model = "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1"
[replicate]
key = ...
```
(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
Also review the [AiHandler](pr_agent/algo/ai_handler.py) file for instruction how to set keys for other models.
#### Extra instructions
All PR-Agent tools have a parameter called `extra_instructions`, that enables to add free-text extra instructions. Example usage:
```
/update_changelog --pr_update_changelog.extra_instructions="Make sure to update also the version ..."
```

View File

@ -2,8 +2,7 @@ FROM python:3.10 as base
WORKDIR /app WORKDIR /app
ADD pyproject.toml . ADD pyproject.toml .
ADD requirements.txt . RUN pip install . && rm pyproject.toml
RUN pip install . && rm pyproject.toml requirements.txt
ENV PYTHONPATH=/app ENV PYTHONPATH=/app
ADD pr_agent pr_agent ADD pr_agent pr_agent
ADD github_action/entrypoint.sh / ADD github_action/entrypoint.sh /

View File

@ -1 +1 @@
FROM codiumai/pr-agent:0.7-github_action FROM codiumai/pr-agent:github_action

View File

@ -9,13 +9,12 @@ To get started with PR-Agent quickly, you first need to acquire two tokens:
There are several ways to use PR-Agent: There are several ways to use PR-Agent:
- [Method 1: Use Docker image (no installation required)](INSTALL.md#method-1-use-docker-image-no-installation-required) - [Method 1: Use Docker image (no installation required)](INSTALL.md#method-1-use-docker-image-no-installation-required)
- [Method 2: Run from source](INSTALL.md#method-2-run-from-source) - [Method 2: Run as a GitHub Action](INSTALL.md#method-2-run-as-a-github-action)
- [Method 3: Run as a GitHub Action](INSTALL.md#method-3-run-as-a-github-action) - [Method 3: Run from source](INSTALL.md#method-3-run-from-source)
- [Method 4: Run as a polling server](INSTALL.md#method-4-run-as-a-polling-server) - [Method 4: Run as a polling server](INSTALL.md#method-4-run-as-a-polling-server)
- [Method 5: Run as a GitHub App](INSTALL.md#method-5-run-as-a-github-app) - [Method 5: Run as a GitHub App](INSTALL.md#method-5-run-as-a-github-app)
- [Method 6: Deploy as a Lambda Function](INSTALL.md#method-6---deploy-as-a-lambda-function) - [Method 6: Deploy as a Lambda Function](INSTALL.md#method-6---deploy-as-a-lambda-function)
- [Method 7: AWS CodeCommit](INSTALL.md#method-7---aws-codecommit-setup) - [Method 7: AWS CodeCommit](INSTALL.md#method-7---aws-codecommit-setup)
- [Method 8: Run a GitLab webhook server](INSTALL.md#method-8---run-a-gitlab-webhook-server)
--- ---
### Method 1: Use Docker image (no installation required) ### Method 1: Use Docker image (no installation required)
@ -24,15 +23,9 @@ To request a review for a PR, or ask a question about a PR, you can run directly
1. To request a review for a PR, run the following command: 1. To request a review for a PR, run the following command:
For GitHub:
``` ```
docker run --rm -it -e OPENAI.KEY=<your key> -e GITHUB.USER_TOKEN=<your token> codiumai/pr-agent --pr_url <pr_url> review docker run --rm -it -e OPENAI.KEY=<your key> -e GITHUB.USER_TOKEN=<your token> codiumai/pr-agent --pr_url <pr_url> review
``` ```
For GitLab:
```
docker run --rm -it -e OPENAI.KEY=<your key> -e CONFIG.GIT_PROVIDER=gitlab -e GITLAB.PERSONAL_ACCESS_TOKEN=<your token> codiumai/pr-agent --pr_url <pr_url> review
```
For other git providers, update CONFIG.GIT_PROVIDER accordingly, and check the `pr_agent/settings/.secrets_template.toml` file for the environment variables expected names and values.
2. To ask a question about a PR, run the following command: 2. To ask a question about a PR, run the following command:
@ -62,41 +55,7 @@ Possible questions you can ask include:
--- ---
### Method 2: Run from source ### Method 2: Run as a GitHub Action
1. Clone this repository:
```
git clone https://github.com/Codium-ai/pr-agent.git
```
2. Install the requirements in your favorite virtual environment:
```
pip install -r requirements.txt
```
3. Copy the secrets template file and fill in your OpenAI key and your GitHub user token:
```
cp pr_agent/settings/.secrets_template.toml pr_agent/settings/.secrets.toml
chmod 600 pr_agent/settings/.secrets.toml
# Edit .secrets.toml file
```
4. Add the pr_agent folder to your PYTHONPATH, then run the cli.py script:
```
export PYTHONPATH=[$PYTHONPATH:]<PATH to pr_agent folder>
python pr_agent/cli.py --pr_url <pr_url> /review
python pr_agent/cli.py --pr_url <pr_url> /ask <your question>
python pr_agent/cli.py --pr_url <pr_url> /describe
python pr_agent/cli.py --pr_url <pr_url> /improve
```
---
### Method 3: Run as a GitHub Action
You can use our pre-built Github Action Docker image to run PR-Agent as a Github Action. You can use our pre-built Github Action Docker image to run PR-Agent as a Github Action.
@ -155,7 +114,7 @@ The GITHUB_TOKEN secret is automatically created by GitHub.
3. Merge this change to your main branch. 3. Merge this change to your main branch.
When you open your next PR, you should see a comment from `github-actions` bot with a review of your PR, and instructions on how to use the rest of the tools. When you open your next PR, you should see a comment from `github-actions` bot with a review of your PR, and instructions on how to use the rest of the tools.
4. You may configure PR-Agent by adding environment variables under the env section corresponding to any configurable property in the [configuration](./Usage.md) file. Some examples: 4. You may configure PR-Agent by adding environment variables under the env section corresponding to any configurable property in the [configuration](./CONFIGURATION.md) file. Some examples:
```yaml ```yaml
env: env:
# ... previous environment values # ... previous environment values
@ -166,6 +125,40 @@ When you open your next PR, you should see a comment from `github-actions` bot w
--- ---
### Method 3: Run from source
1. Clone this repository:
```
git clone https://github.com/Codium-ai/pr-agent.git
```
2. Install the requirements in your favorite virtual environment:
```
pip install -r requirements.txt
```
3. Copy the secrets template file and fill in your OpenAI key and your GitHub user token:
```
cp pr_agent/settings/.secrets_template.toml pr_agent/settings/.secrets.toml
chmod 600 pr_agent/settings/.secrets.toml
# Edit .secrets.toml file
```
4. Add the pr_agent folder to your PYTHONPATH, then run the cli.py script:
```
export PYTHONPATH=[$PYTHONPATH:]<PATH to pr_agent folder>
python pr_agent/cli.py --pr_url <pr_url> review
python pr_agent/cli.py --pr_url <pr_url> ask <your question>
python pr_agent/cli.py --pr_url <pr_url> describe
python pr_agent/cli.py --pr_url <pr_url> improve
```
---
### Method 4: Run as a polling server ### Method 4: Run as a polling server
Request reviews by tagging your Github user on a PR Request reviews by tagging your Github user on a PR
@ -258,9 +251,6 @@ docker push codiumai/pr-agent:github_app # Push to your Docker repository
9. Install the app by navigating to the "Install App" tab and selecting your desired repositories. 9. Install the app by navigating to the "Install App" tab and selecting your desired repositories.
> **Note:** When running PR-Agent from GitHub App, the default configuration file (configuration.toml) will be loaded.<br>
> However, you can override the default tool parameters by uploading a local configuration file<br>
> For more information please check out [CONFIGURATION.md](Usage.md#working-from-github-app-pre-built-repo)
--- ---
### Method 6 - Deploy as a Lambda Function ### Method 6 - Deploy as a Lambda Function
@ -319,9 +309,7 @@ Example IAM permissions to that user to allow access to CodeCommit:
"codecommit:Get*", "codecommit:Get*",
"codecommit:List*", "codecommit:List*",
"codecommit:PostComment*", "codecommit:PostComment*",
"codecommit:PutCommentReaction", "codecommit:PutCommentReaction"
"codecommit:UpdatePullRequestDescription",
"codecommit:UpdatePullRequestTitle"
], ],
"Resource": "*" "Resource": "*"
} }
@ -350,24 +338,9 @@ PYTHONPATH="/PATH/TO/PROJECTS/pr-agent" python pr_agent/cli.py \
review review
``` ```
--- ### Appendix - **Debugging LLM API Calls**
If you're testing your codium/pr-agent server, and need to see if calls were made successfully + the exact call logs, you can use the [LiteLLM Debugger tool](https://docs.litellm.ai/docs/debugging/hosted_debugging).
### Method 8 - Run a GitLab webhook server You can do this by setting `litellm_debugger=true` in configuration.toml. Your Logs will be viewable in real-time @ `admin.litellm.ai/<your_email>`. Set your email in the `.secrets.toml` under 'user_email'.
1. From the GitLab workspace or group, create an access token. Enable the "api" scope only. <img src="./pics/debugger.png" width="800"/>
2. Generate a random secret for your app, and save it for later. For example, you can use:
```
WEBHOOK_SECRET=$(python -c "import secrets; print(secrets.token_hex(10))")
```
3. Follow the instructions to build the Docker image, setup a secrets file and deploy on your own server from [Method 5](#method-5-run-as-a-github-app) steps 4-7.
4. In the secrets file, fill in the following:
- Your OpenAI key.
- In the [gitlab] section, fill in personal_access_token and shared_secret. The access token can be a personal access token, or a group or project access token.
- Set deployment_type to 'gitlab' in [configuration.toml](./pr_agent/settings/configuration.toml)
5. Create a webhook in GitLab. Set the URL to the URL of your app's server. Set the secret token to the generated secret from step 2.
In the "Trigger" section, check the comments and merge request events boxes.
6. Test your installation by opening a merge request or commenting or a merge request using one of CodiumAI's commands.
=======

1
MANIFEST.in Normal file
View File

@ -0,0 +1 @@
recursive-include pr_agent/settings/ *.toml

166
README.md
View File

@ -15,24 +15,19 @@ Making pull requests less painful with an AI agent
</div> </div>
<div style="text-align:left;"> <div style="text-align:left;">
CodiumAI `PR-Agent` is an open-source tool aiming to help developers review pull requests faster and more efficiently. It automatically analyzes the pull request and can provide several types of commands: CodiumAI `PR-Agent` is an open-source tool aiming to help developers review pull requests faster and more efficiently. It automatically analyzes the pull request and can provide several types of PR feedback:
**Auto Description (`/describe`)**: Automatically generating [PR description](https://github.com/Codium-ai/pr-agent/pull/229#issue-1860711415) - title, type, summary, code walkthrough and labels. **Auto-Description**: Automatically generating [PR description](https://github.com/Codium-ai/pr-agent/pull/229#issue-1860711415) - title, type, summary, code walkthrough and labels.
\ \
**Auto Review (`/review`)**: [Adjustable feedback](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695022908) about the PR main theme, type, relevant tests, security issues, score, and various suggestions for the PR content. **Auto Review**: [Adjustable feedback](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695022908) about the PR main theme, type, relevant tests, security issues, score, and various suggestions for the PR content.
\ \
**Question Answering (`/ask ...`)**: Answering [free-text questions](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021332) about the PR. **Question Answering**: Answering [free-text questions](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021332) about the PR.
\ \
**Code Suggestions (`/improve`)**: [Committable code suggestions](https://github.com/Codium-ai/pr-agent/pull/229#discussion_r1306919276) for improving the PR. **Code Suggestions**: [Committable code suggestions](https://github.com/Codium-ai/pr-agent/pull/229#discussion_r1306919276) for improving the PR.
\ \
**Update Changelog (`/update_changelog`)**: Automatically updating the CHANGELOG.md file with the [PR changes](https://github.com/Codium-ai/pr-agent/pull/168#discussion_r1282077645). **Update Changelog**: Automatically updating the CHANGELOG.md file with the [PR changes](https://github.com/Codium-ai/pr-agent/pull/168#discussion_r1282077645).
\
**Find similar issue (`/similar_issue`)**: Automatically retrieves and presents [similar issues](https://github.com/Alibaba-MIIL/ASL/issues/107).
<h3>Example results:</h2>
See the [usage guide](./Usage.md) for instructions how to run the different tools from [CLI](./Usage.md#working-from-a-local-repo-cli), or by [online usage](./Usage.md#online-usage), as well as additional details on optional commands and configurations.
<h3>Example results:</h3>
</div> </div>
<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1687561986">/describe:</a></h4> <h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1687561986">/describe:</a></h4>
<div align="center"> <div align="center">
@ -40,101 +35,87 @@ See the [usage guide](./Usage.md) for instructions how to run the different tool
<img src="https://www.codium.ai/images/describe-2.gif" width="800"> <img src="https://www.codium.ai/images/describe-2.gif" width="800">
</p> </p>
</div> </div>
<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901">/review:</a></h4> <h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901">/review:</a></h4>
<div align="center"> <div align="center">
<p float="center"> <p float="center">
<img src="https://www.codium.ai/images/review-2.gif" width="800"> <img src="https://www.codium.ai/images/review-2.gif" width="800">
</p> </p>
</div> </div>
<h4><a href="https://github.com/Codium-ai/pr-agent/pull/78#issuecomment-1639739496">/reflect_and_review:</a></h4>
[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/78#issuecomment-1639739496">/reflect_and_review:</a></h4>) <div align="center">
<p float="center">
[//]: # (<div align="center">) <img src="https://www.codium.ai/images/reflect_and_review.gif" width="800">
</p>
[//]: # (<p float="center">) </div>
<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695020538">/ask:</a></h4>
[//]: # (<img src="https://www.codium.ai/images/reflect_and_review.gif" width="800">) <div align="center">
<p float="center">
[//]: # (</p>) <img src="https://www.codium.ai/images/ask-2.gif" width="800">
</p>
[//]: # (</div>) </div>
<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695024952">/improve:</a></h4>
[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695020538">/ask:</a></h4>) <div align="center">
<p float="center">
[//]: # (<div align="center">) <img src="https://www.codium.ai/images/improve-2.gif" width="800">
</p>
[//]: # (<p float="center">) </div>
[//]: # (<img src="https://www.codium.ai/images/ask-2.gif" width="800">)
[//]: # (</p>)
[//]: # (</div>)
[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695024952">/improve:</a></h4>)
[//]: # (<div align="center">)
[//]: # (<p float="center">)
[//]: # (<img src="https://www.codium.ai/images/improve-2.gif" width="800">)
[//]: # (</p>)
[//]: # (</div>)
<div align="left"> <div align="left">
## Table of Contents
- [Overview](#overview) - [Overview](#overview)
- [Try it now](#try-it-now) - [Try it now](#try-it-now)
- [Installation](#installation) - [Installation](#installation)
- [Configuration](./CONFIGURATION.md)
- [How it works](#how-it-works) - [How it works](#how-it-works)
- [Why use PR-Agent?](#why-use-pr-agent) - [Why use PR-Agent](#why-use-pr-agent)
- [Roadmap](#roadmap) - [Roadmap](#roadmap)
- [Similar projects](#similar-projects)
</div> </div>
## Overview ## Overview
`PR-Agent` offers extensive pull request functionalities across various git providers: `PR-Agent` offers extensive pull request functionalities across various git providers:
| | | GitHub | Gitlab | Bitbucket | CodeCommit | Azure DevOps | Gerrit | | | | GitHub | Gitlab | Bitbucket | CodeCommit |
|-------|---------------------------------------------|:------:|:------:|:---------:|:----------:|:----------:|:----------:| |-------|---------------------------------------------|:------:|:------:|:---------:|:----------:|
| TOOLS | Review | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | TOOLS | Review | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | Ask | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | ⮑ Inline review | :white_check_mark: | :white_check_mark: | | |
| | Auto-Description | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | Ask | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | Improve Code | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | :white_check_mark: | | | Auto-Description | :white_check_mark: | :white_check_mark: | | |
| | ⮑ Extended | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | :white_check_mark: | | | Improve Code | :white_check_mark: | :white_check_mark: | | |
| | Reflect and Review | :white_check_mark: | :white_check_mark: | :white_check_mark: | | :white_check_mark: | :white_check_mark: | | | ⮑ Extended | :white_check_mark: | :white_check_mark: | | |
| | Update CHANGELOG.md | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | | | | Reflect and Review | :white_check_mark: | | | |
| | Find similar issue | :white_check_mark: | | | | | | | | Update CHANGELOG.md | :white_check_mark: | | | |
| | | | | | | | | | | | | | |
| USAGE | CLI | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | USAGE | CLI | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | App / webhook | :white_check_mark: | :white_check_mark: | | | | | | App / webhook | :white_check_mark: | :white_check_mark: | | |
| | Tagging bot | :white_check_mark: | | | | | | | Tagging bot | :white_check_mark: | | | |
| | Actions | :white_check_mark: | | | | | | | Actions | :white_check_mark: | | | |
| | Web server | | | | | | :white_check_mark: | | | | | | | |
| | | | | | | | | CORE | PR compression | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| CORE | PR compression | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | Repo language prioritization | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| | Repo language prioritization | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | Adaptive and token-aware<br />file patch fitting | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| | Adaptive and token-aware<br />file patch fitting | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | Multiple models support | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| | Multiple models support | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | Incremental PR Review | :white_check_mark: | | | |
| | Incremental PR Review | :white_check_mark: | | | | | |
Review the **[usage guide](./Usage.md)** section for detailed instructions how to use the different tools, select the relevant git provider (GitHub, Gitlab, Bitbucket,...), and adjust the configuration file to your needs. Examples for invoking the different tools via the CLI:
- **Review**: python cli.py --pr_url=<pr_url> review
- **Describe**: python cli.py --pr_url=<pr_url> describe
- **Improve**: python cli.py --pr_url=<pr_url> improve
- **Ask**: python cli.py --pr_url=<pr_url> ask "Write me a poem about this PR"
- **Reflect**: python cli.py --pr_url=<pr_url> reflect
- **Update Changelog**: python cli.py --pr_url=<pr_url> update_changelog
"<pr_url>" is the url of the relevant PR (for example: https://github.com/Codium-ai/pr-agent/pull/50).
In the [configuration](./CONFIGURATION.md) file you can select your git provider (GitHub, Gitlab, Bitbucket), and further configure the different tools.
## Try it now ## Try it now
You can try GPT-4 powered PR-Agent, on your public GitHub repository, instantly. Just mention `@CodiumAI-Agent` and add the desired command in any PR comment. The agent will generate a response based on your command. Try GPT-4 powered PR-Agent on your public GitHub repository for free. Just mention `@CodiumAI-Agent` and add the desired command in any PR comment! The agent will generate a response based on your command.
For example, add a comment to any pull request with the following text:
```
@CodiumAI-Agent /review
```
and the agent will respond with a review of your PR
![Review generation process](https://www.codium.ai/images/demo-2.gif) ![Review generation process](https://www.codium.ai/images/demo-2.gif)
To set up your own PR-Agent, see the [Installation](#installation) section
To set up your own PR-Agent, see the [Installation](#installation) section below.
--- ---
@ -148,15 +129,14 @@ To get started with PR-Agent quickly, you first need to acquire two tokens:
There are several ways to use PR-Agent: There are several ways to use PR-Agent:
- [Method 1: Use Docker image (no installation required)](INSTALL.md#method-1-use-docker-image-no-installation-required) - [Method 1: Use Docker image (no installation required)](INSTALL.md#method-1-use-docker-image-no-installation-required)
- [Method 2: Run from source](INSTALL.md#method-2-run-from-source) - [Method 2: Run as a GitHub Action](INSTALL.md#method-2-run-as-a-github-action)
- [Method 3: Run as a GitHub Action](INSTALL.md#method-3-run-as-a-github-action) - [Method 3: Run from source](INSTALL.md#method-3-run-from-source)
- [Method 4: Run as a polling server](INSTALL.md#method-4-run-as-a-polling-server) - [Method 4: Run as a polling server](INSTALL.md#method-4-run-as-a-polling-server)
- Request reviews by tagging your GitHub user on a PR - Request reviews by tagging your GitHub user on a PR
- [Method 5: Run as a GitHub App](INSTALL.md#method-5-run-as-a-github-app) - [Method 5: Run as a GitHub App](INSTALL.md#method-5-run-as-a-github-app)
- Allowing you to automate the review process on your private or public repositories - Allowing you to automate the review process on your private or public repositories
- [Method 6: Deploy as a Lambda Function](INSTALL.md#method-6---deploy-as-a-lambda-function) - [Method 6: Deploy as a Lambda Function](INSTALL.md#method-6---deploy-as-a-lambda-function)
- [Method 7: AWS CodeCommit](INSTALL.md#method-7---aws-codecommit-setup) - [Method 7: AWS CodeCommit](INSTALL.md#method-7---aws-codecommit-setup)
- [Method 8: Run a GitLab webhook server](INSTALL.md#method-8---run-a-gitlab-webhook-server)
## How it works ## How it works
@ -174,7 +154,7 @@ Here are some advantages of PR-Agent:
- We emphasize **real-life practical usage**. Each tool (review, improve, ask, ...) has a single GPT-4 call, no more. We feel that this is critical for realistic team usage - obtaining an answer quickly (~30 seconds) and affordably. - We emphasize **real-life practical usage**. Each tool (review, improve, ask, ...) has a single GPT-4 call, no more. We feel that this is critical for realistic team usage - obtaining an answer quickly (~30 seconds) and affordably.
- Our [PR Compression strategy](./PR_COMPRESSION.md) is a core ability that enables to effectively tackle both short and long PRs. - Our [PR Compression strategy](./PR_COMPRESSION.md) is a core ability that enables to effectively tackle both short and long PRs.
- Our JSON prompting strategy enables to have **modular, customizable tools**. For example, the '/review' tool categories can be controlled via the [configuration](pr_agent/settings/configuration.toml) file. Adding additional categories is easy and accessible. - Our JSON prompting strategy enables to have **modular, customizable tools**. For example, the '/review' tool categories can be controlled via the [configuration](./CONFIGURATION.md) file. Adding additional categories is easy and accessible.
- We support **multiple git providers** (GitHub, Gitlab, Bitbucket, CodeCommit), **multiple ways** to use the tool (CLI, GitHub Action, GitHub App, Docker, ...), and **multiple models** (GPT-4, GPT-3.5, Anthropic, Cohere, Llama2). - We support **multiple git providers** (GitHub, Gitlab, Bitbucket, CodeCommit), **multiple ways** to use the tool (CLI, GitHub Action, GitHub App, Docker, ...), and **multiple models** (GPT-4, GPT-3.5, Anthropic, Cohere, Llama2).
- We are open-source, and welcome contributions from the community. - We are open-source, and welcome contributions from the community.
@ -184,7 +164,7 @@ Here are some advantages of PR-Agent:
- [x] Support additional models, as a replacement for OpenAI (see [here](https://github.com/Codium-ai/pr-agent/pull/172)) - [x] Support additional models, as a replacement for OpenAI (see [here](https://github.com/Codium-ai/pr-agent/pull/172))
- [x] Develop additional logic for handling large PRs (see [here](https://github.com/Codium-ai/pr-agent/pull/229)) - [x] Develop additional logic for handling large PRs (see [here](https://github.com/Codium-ai/pr-agent/pull/229))
- [ ] Add additional context to the prompt. For example, repo (or relevant files) summarization, with tools such a [ctags](https://github.com/universal-ctags/ctags) - [ ] Add additional context to the prompt. For example, repo (or relevant files) summarization, with tools such a [ctags](https://github.com/universal-ctags/ctags)
- [x] PR-Agent for issues - [ ] PR-Agent for issues, and just for pull requests
- [ ] Adding more tools. Possible directions: - [ ] Adding more tools. Possible directions:
- [x] PR description - [x] PR description
- [x] Inline code suggestions - [x] Inline code suggestions
@ -197,18 +177,8 @@ Here are some advantages of PR-Agent:
## Similar Projects ## Similar Projects
- [CodiumAI - Meaningful tests for busy devs](https://github.com/Codium-ai/codiumai-vscode-release) (although various capabilities are much more advanced in the CodiumAI IDE plugins) - [CodiumAI - Meaningful tests for busy devs](https://github.com/Codium-ai/codiumai-vscode-release)
- [Aider - GPT powered coding in your terminal](https://github.com/paul-gauthier/aider) - [Aider - GPT powered coding in your terminal](https://github.com/paul-gauthier/aider)
- [openai-pr-reviewer](https://github.com/coderabbitai/openai-pr-reviewer) - [openai-pr-reviewer](https://github.com/coderabbitai/openai-pr-reviewer)
- [CodeReview BOT](https://github.com/anc95/ChatGPT-CodeReview) - [CodeReview BOT](https://github.com/anc95/ChatGPT-CodeReview)
- [AI-Maintainer](https://github.com/merwanehamadi/AI-Maintainer) - [AI-Maintainer](https://github.com/merwanehamadi/AI-Maintainer)
## Links
[![Join our Discord community](https://raw.githubusercontent.com/Codium-ai/codiumai-vscode-release/main/media/docs/Joincommunity.png)](https://discord.gg/kG35uSHDBc)
- Discord community: https://discord.gg/kG35uSHDBc
- CodiumAI site: https://codium.ai
- Blog: https://www.codium.ai/blog/
- Troubleshooting: https://www.codium.ai/blog/technical-faq-and-troubleshooting/
- Support: support@codium.ai

View File

@ -1,25 +0,0 @@
## [Version 0.7] - 2023-09-20
### Docker Tags
- codiumai/pr-agent:0.7
- codiumai/pr-agent:0.7-github_app
- codiumai/pr-agent:0.7-bitbucket-app
- codiumai/pr-agent:0.7-gitlab_webhook
- codiumai/pr-agent:0.7-github_polling
- codiumai/pr-agent:0.7-github_action
### Added::Algo
- New tool /similar_issue - Currently on GitHub app and CLI: indexes the issues in the repo, find the most similar issues to the target issue.
- Describe markers: Empower the /describe tool with a templating capability (see more details in https://github.com/Codium-ai/pr-agent/pull/273).
- New feature in the /review tool - added an estimated effort estimation to the review (https://github.com/Codium-ai/pr-agent/pull/306).
### Added::Infrastructure
- Implementation of a GitLab webhook.
- Implementation of a BitBucket app.
### Fixed
- Protection against no code suggestions generated.
- Resilience to repositories where the languages cannot be automatically detected.

272
Usage.md
View File

@ -1,272 +0,0 @@
## Usage guide
### Table of Contents
- [Introduction](#introduction)
- [Working from a local repo (CLI)](#working-from-a-local-repo-cli)
- [Online usage](#online-usage)
- [Working with GitHub App](#working-with-github-app)
- [Working with GitHub Action](#working-with-github-action)
- [Appendix - additional configurations walkthrough](#appendix---additional-configurations-walkthrough)
### Introduction
There are 3 basic ways to invoke CodiumAI PR-Agent:
1. Locally running a CLI command
2. Online usage - by [commenting](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901) on a PR
3. Enabling PR-Agent tools to run automatically when a new PR is opened
See the [installation guide](/INSTALL.md) for instructions on how to setup your own PR-Agent.
Specifically, CLI commands can be issued by invoking a pre-built [docker image](/INSTALL.md#running-from-source), or by invoking a [locally cloned repo](INSTALL.md#method-2-run-from-source).
For online usage, you will need to setup either a [GitHub App](INSTALL.md#method-5-run-as-a-github-app), or a [GitHub Action](INSTALL.md#method-3-run-as-a-github-action).
GitHub App and GitHub Action also enable to run PR-Agent specific tool automatically when a new PR is opened.
#### The configuration file
The different tools and sub-tools used by CodiumAI PR-Agent are adjustable via the **[configuration file](pr_agent/settings/configuration.toml)**.
In addition to general configuration options, each tool has its own configurations. For example, the `review` tool will use parameters from the [pr_reviewer](/pr_agent/settings/configuration.toml#L16) section in the configuration file.
**git provider:**
The [git_provider](pr_agent/settings/configuration.toml#L4) field in the configuration file determines the GIT provider that will be used by PR-Agent. Currently, the following providers are supported:
`
"github", "gitlab", "azure", "codecommit", "local"
`
[//]: # (** online usage:**)
[//]: # (Options that are available in the configuration file can be specified at run time when calling actions. Two examples:)
[//]: # (```)
[//]: # (- /review --pr_reviewer.extra_instructions="focus on the file: ...")
[//]: # (- /describe --pr_description.add_original_user_description=false -pr_description.extra_instructions="make sure to mention: ...")
[//]: # (```)
### Working from a local repo (CLI)
When running from your local repo (CLI), your local configuration file will be used.
Examples for invoking the different tools via the CLI:
- **Review**: `python cli.py --pr_url=<pr_url> review`
- **Describe**: `python cli.py --pr_url=<pr_url> describe`
- **Improve**: `python cli.py --pr_url=<pr_url> improve`
- **Ask**: `python cli.py --pr_url=<pr_url> ask "Write me a poem about this PR"`
- **Reflect**: `python cli.py --pr_url=<pr_url> reflect`
- **Update Changelog**: `python cli.py --pr_url=<pr_url> update_changelog`
`<pr_url>` is the url of the relevant PR (for example: https://github.com/Codium-ai/pr-agent/pull/50).
**Notes:**
(1) in addition to editing your local configuration file, you can also change any configuration value by adding it to the command line:
```
python cli.py --pr_url=<pr_url> /review --pr_reviewer.extra_instructions="focus on the file: ..."
```
(2) You can print results locally, without publishing them, by setting in `configuration.toml`:
```
[config]
publish_output=true
verbosity_level=2
```
This is useful for debugging or experimenting with the different tools.
### Online usage
Online usage means invoking PR-Agent tools by [comments](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901) on a PR.
Commands for invoking the different tools via comments:
- **Review**: `/review`
- **Describe**: `/describe`
- **Improve**: `/improve`
- **Ask**: `/ask "..."`
- **Reflect**: `/reflect`
- **Update Changelog**: `/update_changelog`
To edit a specific configuration value, just add `--config_path=<value>` to any command.
For example if you want to edit the `review` tool configurations, you can run:
```
/review --pr_reviewer.extra_instructions="..." --pr_reviewer.require_score_review=false
```
Any configuration value in [configuration file](pr_agent/settings/configuration.toml) file can be similarly edited.
### Working with GitHub App
When running PR-Agent from [GitHub App](INSTALL.md#method-5-run-as-a-github-app), the default configurations from a pre-built repo will be initially loaded.
#### GitHub app automatic tools
The [github_app](pr_agent/settings/configuration.toml#L56) section defines GitHub app specific configurations.
An important parameter is `pr_commands`, which is a list of tools that will be **run automatically when a new PR is opened**:
```
[github_app]
pr_commands = [
"/describe --pr_description.add_original_user_description=true --pr_description.keep_original_user_title=true",
"/auto_review",
]
```
This means that when a new PR is opened, PR-Agent will run the `describe` and `auto_review` tools.
For the describe tool, the `add_original_user_description` and `keep_original_user_title` parameters will be set to true.
However, you can override the default tool parameters by uploading a local configuration file called `.pr_agent.toml` to the root of your repo.
For example, if your local `.pr_agent.toml` file contains:
```
[pr_description]
add_original_user_description = false
keep_original_user_title = false
```
When a new PR is opened, PR-Agent will run the `describe` tool with the above parameters.
Note that a local `.pr_agent.toml` file enables you to edit and customize the default parameters of any tool, not just the ones that are run automatically.
#### Editing the prompts
The prompts for the various PR-Agent tools are defined in the `pr_agent/settings` folder.
In practice, the prompts are loaded and stored as a standard setting object.
Hence, editing them is similar to editing any other configuration value - just place the relevant key in `.pr_agent.toml`file, and override the default value.
For example, if you want to edit the prompts of the [describe](./pr_agent/settings/pr_description_prompts.toml) tool, you can add the following to your `.pr_agent.toml` file:
```
[pr_description_prompt]
system="""
...
"""
user="""
...
"""
```
Note that the new prompt will need to generate an output compatible with the relevant [post-process function](./pr_agent/tools/pr_description.py#L137).
### Working with GitHub Action
TBD
### Appendix - additional configurations walkthrough
#### Changing a model
See [here](pr_agent/algo/__init__.py) for the list of available models.
#### Azure
To use Azure, set:
```
api_key = "" # your azure api key
api_type = "azure"
api_version = '2023-05-15' # Check Azure documentation for the current API version
api_base = "" # The base URL for your Azure OpenAI resource. e.g. "https://<your resource name>.openai.azure.com"
deployment_id = "" # The deployment name you chose when you deployed the engine
```
in your .secrets.toml
and
```
[config]
model="" # the OpenAI model you've deployed on Azure (e.g. gpt-3.5-turbo)
```
in the configuration.toml
#### Huggingface
**Local**
You can run Huggingface models locally through either [VLLM](https://docs.litellm.ai/docs/providers/vllm) or [Ollama](https://docs.litellm.ai/docs/providers/ollama)
E.g. to use a new Huggingface model locally via Ollama, set:
```
[__init__.py]
MAX_TOKENS = {
"model-name-on-ollama": <max_tokens>
}
e.g.
MAX_TOKENS={
...,
"llama2": 4096
}
[config] # in configuration.toml
model = "ollama/llama2"
[ollama] # in .secrets.toml
api_base = ... # the base url for your huggingface inference endpoint
```
**Inference Endpoints**
To use a new model with Huggingface Inference Endpoints, for example, set:
```
[__init__.py]
MAX_TOKENS = {
"model-name-on-huggingface": <max_tokens>
}
e.g.
MAX_TOKENS={
...,
"meta-llama/Llama-2-7b-chat-hf": 4096
}
[config] # in configuration.toml
model = "huggingface/meta-llama/Llama-2-7b-chat-hf"
[huggingface] # in .secrets.toml
key = ... # your huggingface api key
api_base = ... # the base url for your huggingface inference endpoint
```
(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
#### Replicate
To use Llama2 model with Replicate, for example, set:
```
[config] # in configuration.toml
model = "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1"
[replicate] # in .secrets.toml
key = ...
```
(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
Also review the [AiHandler](pr_agent/algo/ai_handler.py) file for instruction how to set keys for other models.
#### Extra instructions
All PR-Agent tools have a parameter called `extra_instructions`, that enables to add free-text extra instructions. Example usage:
```
/update_changelog --pr_update_changelog.extra_instructions="Make sure to update also the version ..."
```
#### Azure DevOps provider
To use Azure DevOps provider use the following settings in configuration.toml:
```
[config]
git_provider="azure"
use_repo_settings_file=false
```
And use the following settings (you have to replace the values) in .secrets.toml:
```
[azure_devops]
org = "https://dev.azure.com/YOUR_ORGANIZATION/"
pat = "YOUR_PAT_TOKEN"
```
#### Similar issue tool
[Example usage](https://github.com/Alibaba-MIIL/ASL/issues/107)
<img src=./pics/similar_issue_tool.png width="768">
To enable usage of the '**similar issue**' tool, you need to set the following keys in `.secrets.toml` (or in the relevant environment variables):
```
[pinecone]
api_key = "..."
environment = "..."
```
These parameters can be obtained by registering to [Pinecone](https://app.pinecone.io/?sessionType=signup/).
- To invoke the 'similar issue' tool from **CLI**, run:
`python3 cli.py --issue_url=... similar_issue`
- To invoke the 'similar' issue tool via online usage, [comment](https://github.com/Codium-ai/pr-agent/issues/178#issuecomment-1716934893) on a PR:
`/similar_issue`
- You can also enable the 'similar issue' tool to run automatically when a new issue is opened, by adding it to the [pr_commands list in the github_app section](https://github.com/Codium-ai/pr-agent/blob/main/pr_agent/settings/configuration.toml#L66)

View File

@ -18,10 +18,6 @@ FROM base as github_polling
ADD pr_agent pr_agent ADD pr_agent pr_agent
CMD ["python", "pr_agent/servers/github_polling.py"] CMD ["python", "pr_agent/servers/github_polling.py"]
FROM base as gitlab_webhook
ADD pr_agent pr_agent
CMD ["python", "pr_agent/servers/gitlab_webhook.py"]
FROM base as test FROM base as test
ADD requirements-dev.txt . ADD requirements-dev.txt .
RUN pip install -r requirements-dev.txt && rm requirements-dev.txt RUN pip install -r requirements-dev.txt && rm requirements-dev.txt

BIN
pics/debugger.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 534 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 210 KiB

View File

@ -9,7 +9,6 @@ from pr_agent.git_providers import get_git_provider
from pr_agent.tools.pr_code_suggestions import PRCodeSuggestions from pr_agent.tools.pr_code_suggestions import PRCodeSuggestions
from pr_agent.tools.pr_description import PRDescription from pr_agent.tools.pr_description import PRDescription
from pr_agent.tools.pr_information_from_user import PRInformationFromUser from pr_agent.tools.pr_information_from_user import PRInformationFromUser
from pr_agent.tools.pr_similar_issue import PRSimilarIssue
from pr_agent.tools.pr_questions import PRQuestions from pr_agent.tools.pr_questions import PRQuestions
from pr_agent.tools.pr_reviewer import PRReviewer from pr_agent.tools.pr_reviewer import PRReviewer
from pr_agent.tools.pr_update_changelog import PRUpdateChangelog from pr_agent.tools.pr_update_changelog import PRUpdateChangelog
@ -31,7 +30,6 @@ command2class = {
"update_changelog": PRUpdateChangelog, "update_changelog": PRUpdateChangelog,
"config": PRConfig, "config": PRConfig,
"settings": PRConfig, "settings": PRConfig,
"similar_issue": PRSimilarIssue,
} }
commands = list(command2class.keys()) commands = list(command2class.keys())

View File

@ -1,5 +1,4 @@
MAX_TOKENS = { MAX_TOKENS = {
'text-embedding-ada-002': 8000,
'gpt-3.5-turbo': 4000, 'gpt-3.5-turbo': 4000,
'gpt-3.5-turbo-0613': 4000, 'gpt-3.5-turbo-0613': 4000,
'gpt-3.5-turbo-0301': 4000, 'gpt-3.5-turbo-0301': 4000,
@ -12,5 +11,4 @@ MAX_TOKENS = {
'claude-2': 100000, 'claude-2': 100000,
'command-nightly': 4096, 'command-nightly': 4096,
'replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1': 4096, 'replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1': 4096,
'meta-llama/Llama-2-7b-chat-hf': 4096
} }

View File

@ -1,12 +1,13 @@
import logging import logging
import os
import litellm import litellm
import openai import openai
from litellm import acompletion from litellm import acompletion
from openai.error import APIError, RateLimitError, Timeout, TryAgain from openai.error import APIError, RateLimitError, Timeout, TryAgain
from retry import retry from retry import retry
from pr_agent.config_loader import get_settings from pr_agent.config_loader import get_settings
OPENAI_RETRIES = 5 OPENAI_RETRIES = 5
@ -25,11 +26,7 @@ class AiHandler:
try: try:
openai.api_key = get_settings().openai.key openai.api_key = get_settings().openai.key
litellm.openai_key = get_settings().openai.key litellm.openai_key = get_settings().openai.key
if get_settings().get("litellm.use_client"): litellm.debugger = get_settings().config.litellm_debugger
litellm_token = get_settings().get("litellm.LITELLM_TOKEN")
assert litellm_token, "LITELLM_TOKEN is required"
os.environ["LITELLM_TOKEN"] = litellm_token
litellm.use_client = True
self.azure = False self.azure = False
if get_settings().get("OPENAI.ORG", None): if get_settings().get("OPENAI.ORG", None):
litellm.organization = get_settings().openai.org litellm.organization = get_settings().openai.org
@ -51,8 +48,6 @@ class AiHandler:
litellm.replicate_key = get_settings().replicate.key litellm.replicate_key = get_settings().replicate.key
if get_settings().get("HUGGINGFACE.KEY", None): if get_settings().get("HUGGINGFACE.KEY", None):
litellm.huggingface_key = get_settings().huggingface.key litellm.huggingface_key = get_settings().huggingface.key
if get_settings().get("HUGGINGFACE.API_BASE", None):
litellm.api_base = get_settings().huggingface.api_base
except AttributeError as e: except AttributeError as e:
raise ValueError("OpenAI key is required") from e raise ValueError("OpenAI key is required") from e

View File

@ -42,11 +42,6 @@ def sort_files_by_main_languages(languages: Dict, files: list):
files_sorted = [] files_sorted = []
rest_files = {} rest_files = {}
# if no languages detected, put all files in the "Other" category
if not languages:
files_sorted = [({"language": "Other", "files": list(files_filtered)})]
return files_sorted
main_extensions_flat = [] main_extensions_flat = []
for ext in main_extensions: for ext in main_extensions:
main_extensions_flat.extend(ext) main_extensions_flat.extend(ext)

View File

@ -1,19 +1,17 @@
from __future__ import annotations from __future__ import annotations
import difflib
import logging import logging
import re
import traceback import traceback
from typing import Any, Callable, List, Tuple from typing import Callable, List, Tuple
from github import RateLimitExceededException from github import RateLimitExceededException
from pr_agent.algo import MAX_TOKENS from pr_agent.algo import MAX_TOKENS
from pr_agent.algo.git_patch_processing import convert_to_hunks_with_lines_numbers, extend_patch, handle_patch_deletions from pr_agent.algo.git_patch_processing import convert_to_hunks_with_lines_numbers, extend_patch, handle_patch_deletions
from pr_agent.algo.language_handler import sort_files_by_main_languages from pr_agent.algo.language_handler import sort_files_by_main_languages
from pr_agent.algo.token_handler import TokenHandler, get_token_encoder from pr_agent.algo.token_handler import TokenHandler
from pr_agent.config_loader import get_settings from pr_agent.config_loader import get_settings
from pr_agent.git_providers.git_provider import FilePatchInfo, GitProvider from pr_agent.git_providers.git_provider import GitProvider
DELETED_FILES_ = "Deleted files:\n" DELETED_FILES_ = "Deleted files:\n"
@ -247,99 +245,6 @@ def _get_all_deployments(all_models: List[str]) -> List[str]:
return all_deployments return all_deployments
def find_line_number_of_relevant_line_in_file(diff_files: List[FilePatchInfo],
relevant_file: str,
relevant_line_in_file: str) -> Tuple[int, int]:
"""
Find the line number and absolute position of a relevant line in a file.
Args:
diff_files (List[FilePatchInfo]): A list of FilePatchInfo objects representing the patches of files.
relevant_file (str): The name of the file where the relevant line is located.
relevant_line_in_file (str): The content of the relevant line.
Returns:
Tuple[int, int]: A tuple containing the line number and absolute position of the relevant line in the file.
"""
position = -1
absolute_position = -1
re_hunk_header = re.compile(
r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@[ ]?(.*)")
for file in diff_files:
if file.filename.strip() == relevant_file:
patch = file.patch
patch_lines = patch.splitlines()
# try to find the line in the patch using difflib, with some margin of error
matches_difflib: list[str | Any] = difflib.get_close_matches(relevant_line_in_file,
patch_lines, n=3, cutoff=0.93)
if len(matches_difflib) == 1 and matches_difflib[0].startswith('+'):
relevant_line_in_file = matches_difflib[0]
delta = 0
start1, size1, start2, size2 = 0, 0, 0, 0
for i, line in enumerate(patch_lines):
if line.startswith('@@'):
delta = 0
match = re_hunk_header.match(line)
start1, size1, start2, size2 = map(int, match.groups()[:4])
elif not line.startswith('-'):
delta += 1
if relevant_line_in_file in line and line[0] != '-':
position = i
absolute_position = start2 + delta - 1
break
if position == -1 and relevant_line_in_file[0] == '+':
no_plus_line = relevant_line_in_file[1:].lstrip()
for i, line in enumerate(patch_lines):
if line.startswith('@@'):
delta = 0
match = re_hunk_header.match(line)
start1, size1, start2, size2 = map(int, match.groups()[:4])
elif not line.startswith('-'):
delta += 1
if no_plus_line in line and line[0] != '-':
# The model might add a '+' to the beginning of the relevant_line_in_file even if originally
# it's a context line
position = i
absolute_position = start2 + delta - 1
break
return position, absolute_position
def clip_tokens(text: str, max_tokens: int) -> str:
"""
Clip the number of tokens in a string to a maximum number of tokens.
Args:
text (str): The string to clip.
max_tokens (int): The maximum number of tokens allowed in the string.
Returns:
str: The clipped string.
"""
if not text:
return text
try:
encoder = get_token_encoder()
num_input_tokens = len(encoder.encode(text))
if num_input_tokens <= max_tokens:
return text
num_chars = len(text)
chars_per_token = num_chars / num_input_tokens
num_output_chars = int(chars_per_token * max_tokens)
clipped_text = text[:num_output_chars]
return clipped_text
except Exception as e:
logging.warning(f"Failed to clip tokens: {e}")
return text
def get_pr_multi_diffs(git_provider: GitProvider, def get_pr_multi_diffs(git_provider: GitProvider,
token_handler: TokenHandler, token_handler: TokenHandler,
model: str, model: str,

View File

@ -21,7 +21,7 @@ class TokenHandler:
method. method.
""" """
def __init__(self, pr=None, vars: dict = {}, system="", user=""): def __init__(self, vars: dict, system, user):
""" """
Initializes the TokenHandler object. Initializes the TokenHandler object.
@ -32,10 +32,9 @@ class TokenHandler:
- user: The user string. - user: The user string.
""" """
self.encoder = get_token_encoder() self.encoder = get_token_encoder()
if pr is not None: self.prompt_tokens = self._get_system_user_tokens(self.encoder, vars, system, user)
self.prompt_tokens = self._get_system_user_tokens(pr, self.encoder, vars, system, user)
def _get_system_user_tokens(self, pr, encoder, vars: dict, system, user): def _get_system_user_tokens(self, encoder, vars: dict, system, user):
""" """
Calculates the number of tokens in the system and user strings. Calculates the number of tokens in the system and user strings.

View File

@ -5,14 +5,24 @@ import json
import logging import logging
import re import re
import textwrap import textwrap
from dataclasses import dataclass
from datetime import datetime from datetime import datetime
from typing import Any, List from enum import Enum
from typing import Any, List, Tuple, Optional
import yaml import yaml
from starlette_context import context from starlette_context import context
from pr_agent.algo.token_handler import get_token_encoder
from pr_agent.config_loader import get_settings, global_settings from pr_agent.config_loader import get_settings, global_settings
class EDIT_TYPE(Enum):
ADDED = 1
DELETED = 2
MODIFIED = 3
RENAMED = 4
def get_setting(key: str) -> Any: def get_setting(key: str) -> Any:
try: try:
key = key.upper() key = key.upper()
@ -20,7 +30,7 @@ def get_setting(key: str) -> Any:
except Exception: except Exception:
return global_settings.get(key, None) return global_settings.get(key, None)
def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str: def convert_to_markdown(output_data: dict) -> str:
""" """
Convert a dictionary of data into markdown format. Convert a dictionary of data into markdown format.
Args: Args:
@ -42,7 +52,6 @@ def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
"General suggestions": "💡", "General suggestions": "💡",
"Insights from user's answers": "📝", "Insights from user's answers": "📝",
"Code feedback": "🤖", "Code feedback": "🤖",
"Estimated effort to review [1-5]": "⏱️",
} }
for key, value in output_data.items(): for key, value in output_data.items():
@ -50,14 +59,11 @@ def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
continue continue
if isinstance(value, dict): if isinstance(value, dict):
markdown_text += f"## {key}\n\n" markdown_text += f"## {key}\n\n"
markdown_text += convert_to_markdown(value, gfm_supported) markdown_text += convert_to_markdown(value)
elif isinstance(value, list): elif isinstance(value, list):
emoji = emojis.get(key, "") emoji = emojis.get(key, "")
if key.lower() == 'code feedback': if key.lower() == 'code feedback':
if gfm_supported:
markdown_text += f"\n\n- **<details><summary> { emoji } Code feedback:**</summary>\n\n" markdown_text += f"\n\n- **<details><summary> { emoji } Code feedback:**</summary>\n\n"
else:
markdown_text += f"\n\n- **{emoji} Code feedback:**\n\n"
else: else:
markdown_text += f"- {emoji} **{key}:**\n\n" markdown_text += f"- {emoji} **{key}:**\n\n"
for item in value: for item in value:
@ -66,10 +72,7 @@ def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
elif item: elif item:
markdown_text += f" - {item}\n" markdown_text += f" - {item}\n"
if key.lower() == 'code feedback': if key.lower() == 'code feedback':
if gfm_supported:
markdown_text += "</details>\n\n" markdown_text += "</details>\n\n"
else:
markdown_text += "\n\n"
elif value != 'n/a': elif value != 'n/a':
emoji = emojis.get(key, "") emoji = emojis.get(key, "")
markdown_text += f"- {emoji} **{key}:** {value}\n" markdown_text += f"- {emoji} **{key}:** {value}\n"
@ -301,3 +304,108 @@ def try_fix_yaml(review_text: str) -> dict:
except: except:
pass pass
return data return data
def clip_tokens(text: str, max_tokens: int) -> str:
"""
Clip the number of tokens in a string to a maximum number of tokens.
Args:
text (str): The string to clip.
max_tokens (int): The maximum number of tokens allowed in the string.
Returns:
str: The clipped string.
"""
if not text:
return text
try:
encoder = get_token_encoder()
num_input_tokens = len(encoder.encode(text))
if num_input_tokens <= max_tokens:
return text
num_chars = len(text)
chars_per_token = num_chars / num_input_tokens
num_output_chars = int(chars_per_token * max_tokens)
clipped_text = text[:num_output_chars]
return clipped_text
except Exception as e:
logging.warning(f"Failed to clip tokens: {e}")
return text
@dataclass
class FilePatchInfo:
base_file: str
head_file: str
patch: str
filename: str
tokens: int = -1
edit_type: EDIT_TYPE = EDIT_TYPE.MODIFIED
old_filename: str = None
language: Optional[str] = None
def find_line_number_of_relevant_line_in_file(diff_files: List[FilePatchInfo],
relevant_file: str,
relevant_line_in_file: str) -> Tuple[int, int]:
"""
Find the line number and absolute position of a relevant line in a file.
Args:
diff_files (List[FilePatchInfo]): A list of FilePatchInfo objects representing the patches of files.
relevant_file (str): The name of the file where the relevant line is located.
relevant_line_in_file (str): The content of the relevant line.
Returns:
Tuple[int, int]: A tuple containing the line number and absolute position of the relevant line in the file.
"""
position = -1
absolute_position = -1
re_hunk_header = re.compile(
r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@[ ]?(.*)")
for file in diff_files:
if file.filename.strip() == relevant_file:
patch = file.patch
patch_lines = patch.splitlines()
# try to find the line in the patch using difflib, with some margin of error
matches_difflib: list[str | Any] = difflib.get_close_matches(relevant_line_in_file,
patch_lines, n=3, cutoff=0.93)
if len(matches_difflib) == 1 and matches_difflib[0].startswith('+'):
relevant_line_in_file = matches_difflib[0]
delta = 0
start1, size1, start2, size2 = 0, 0, 0, 0
for i, line in enumerate(patch_lines):
if line.startswith('@@'):
delta = 0
match = re_hunk_header.match(line)
start1, size1, start2, size2 = map(int, match.groups()[:4])
elif not line.startswith('-'):
delta += 1
if relevant_line_in_file in line and line[0] != '-':
position = i
absolute_position = start2 + delta - 1
break
if position == -1 and relevant_line_in_file[0] == '+':
no_plus_line = relevant_line_in_file[1:].lstrip()
for i, line in enumerate(patch_lines):
if line.startswith('@@'):
delta = 0
match = re_hunk_header.match(line)
start1, size1, start2, size2 = map(int, match.groups()[:4])
elif not line.startswith('-'):
delta += 1
if no_plus_line in line and line[0] != '-':
# The model might add a '+' to the beginning of the relevant_line_in_file even if originally
# it's a context line
position = i
absolute_position = start2 + delta - 1
break
return position, absolute_position

View File

@ -17,7 +17,6 @@ For example:
- cli.py --pr_url=... improve - cli.py --pr_url=... improve
- cli.py --pr_url=... ask "write me a poem about this PR" - cli.py --pr_url=... ask "write me a poem about this PR"
- cli.py --pr_url=... reflect - cli.py --pr_url=... reflect
- cli.py --issue_url=... similar_issue
Supported commands: Supported commands:
-review / review_pr - Add a review that includes a summary of the PR and specific suggestions for improvement. -review / review_pr - Add a review that includes a summary of the PR and specific suggestions for improvement.
@ -38,21 +37,13 @@ Configuration:
To edit any configuration parameter from 'configuration.toml', just add -config_path=<value>. To edit any configuration parameter from 'configuration.toml', just add -config_path=<value>.
For example: 'python cli.py --pr_url=... review --pr_reviewer.extra_instructions="focus on the file: ..."' For example: 'python cli.py --pr_url=... review --pr_reviewer.extra_instructions="focus on the file: ..."'
""") """)
parser.add_argument('--pr_url', type=str, help='The URL of the PR to review', default=None) parser.add_argument('--pr_url', type=str, help='The URL of the PR to review', required=True)
parser.add_argument('--issue_url', type=str, help='The URL of the Issue to review', default=None)
parser.add_argument('command', type=str, help='The', choices=commands, default='review') parser.add_argument('command', type=str, help='The', choices=commands, default='review')
parser.add_argument('rest', nargs=argparse.REMAINDER, default=[]) parser.add_argument('rest', nargs=argparse.REMAINDER, default=[])
args = parser.parse_args(inargs) args = parser.parse_args(inargs)
if not args.pr_url and not args.issue_url:
parser.print_help()
return
logging.basicConfig(level=os.environ.get("LOGLEVEL", "INFO")) logging.basicConfig(level=os.environ.get("LOGLEVEL", "INFO"))
command = args.command.lower() command = args.command.lower()
get_settings().set("CONFIG.CLI_MODE", True) get_settings().set("CONFIG.CLI_MODE", True)
if args.issue_url:
result = asyncio.run(PRAgent().handle_request(args.issue_url, command + " " + " ".join(args.rest)))
else:
result = asyncio.run(PRAgent().handle_request(args.pr_url, command + " " + " ".join(args.rest))) result = asyncio.run(PRAgent().handle_request(args.pr_url, command + " " + " ".join(args.rest)))
if not result: if not result:
parser.print_help() parser.print_help()

View File

@ -5,8 +5,6 @@ from pr_agent.git_providers.github_provider import GithubProvider
from pr_agent.git_providers.gitlab_provider import GitLabProvider from pr_agent.git_providers.gitlab_provider import GitLabProvider
from pr_agent.git_providers.local_git_provider import LocalGitProvider from pr_agent.git_providers.local_git_provider import LocalGitProvider
from pr_agent.git_providers.azuredevops_provider import AzureDevopsProvider from pr_agent.git_providers.azuredevops_provider import AzureDevopsProvider
from pr_agent.git_providers.gerrit_provider import GerritProvider
_GIT_PROVIDERS = { _GIT_PROVIDERS = {
'github': GithubProvider, 'github': GithubProvider,
@ -14,8 +12,7 @@ _GIT_PROVIDERS = {
'bitbucket': BitbucketProvider, 'bitbucket': BitbucketProvider,
'azure': AzureDevopsProvider, 'azure': AzureDevopsProvider,
'codecommit': CodeCommitProvider, 'codecommit': CodeCommitProvider,
'local' : LocalGitProvider, 'local' : LocalGitProvider
'gerrit': GerritProvider,
} }
def get_git_provider(): def get_git_provider():

View File

@ -13,11 +13,9 @@ try:
except ImportError: except ImportError:
AZURE_DEVOPS_AVAILABLE = False AZURE_DEVOPS_AVAILABLE = False
from ..algo.pr_processing import clip_tokens
from ..config_loader import get_settings from ..config_loader import get_settings
from ..algo.utils import load_large_diff from ..algo.utils import load_large_diff, FilePatchInfo, EDIT_TYPE, clip_tokens
from ..algo.language_handler import is_valid_file from ..algo.language_handler import is_valid_file
from .git_provider import EDIT_TYPE, FilePatchInfo
class AzureDevopsProvider: class AzureDevopsProvider:
@ -38,8 +36,7 @@ class AzureDevopsProvider:
self.set_pr(pr_url) self.set_pr(pr_url)
def is_supported(self, capability: str) -> bool: def is_supported(self, capability: str) -> bool:
if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels', if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels', 'remove_initial_comment']:
'remove_initial_comment', 'gfm_markdown']:
return False return False
return True return True

View File

@ -7,9 +7,9 @@ import requests
from atlassian.bitbucket import Cloud from atlassian.bitbucket import Cloud
from starlette_context import context from starlette_context import context
from ..algo.pr_processing import clip_tokens, find_line_number_of_relevant_line_in_file
from ..config_loader import get_settings from ..config_loader import get_settings
from .git_provider import FilePatchInfo, GitProvider from .git_provider import GitProvider
from ..algo.utils import FilePatchInfo
class BitbucketProvider(GitProvider): class BitbucketProvider(GitProvider):
@ -36,8 +36,9 @@ class BitbucketProvider(GitProvider):
self.incremental = incremental self.incremental = incremental
if pr_url: if pr_url:
self.set_pr(pr_url) self.set_pr(pr_url)
self.bitbucket_comment_api_url = self.pr._BitbucketBase__data["links"]["comments"]["href"] self.bitbucket_comment_api_url = self.pr._BitbucketBase__data["links"][
self.bitbucket_pull_request_api_url = self.pr._BitbucketBase__data["links"]['self']['href'] "comments"
]["href"]
def get_repo_settings(self): def get_repo_settings(self):
try: try:
@ -101,7 +102,12 @@ class BitbucketProvider(GitProvider):
return False return False
def is_supported(self, capability: str) -> bool: def is_supported(self, capability: str) -> bool:
if capability in ['get_issue_comments', 'publish_inline_comments', 'get_labels', 'gfm_markdown']: if capability in [
"get_issue_comments",
"create_inline_comment",
"publish_inline_comments",
"get_labels",
]:
return False return False
return True return True
@ -146,30 +152,17 @@ class BitbucketProvider(GitProvider):
except Exception as e: except Exception as e:
logging.exception(f"Failed to remove temp comments, error: {e}") logging.exception(f"Failed to remove temp comments, error: {e}")
def publish_inline_comment(
# funtion to create_inline_comment self, comment: str, from_line: int, to_line: int, file: str
def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str): ):
position, absolute_position = find_line_number_of_relevant_line_in_file(self.get_diff_files(), relevant_file.strip('`'), relevant_line_in_file) payload = json.dumps(
if position == -1: {
if get_settings().config.verbosity_level >= 2:
logging.info(f"Could not find position for {relevant_file} {relevant_line_in_file}")
subject_type = "FILE"
else:
subject_type = "LINE"
path = relevant_file.strip()
return dict(body=body, path=path, position=absolute_position) if subject_type == "LINE" else {}
def publish_inline_comment(self, comment: str, from_line: int, file: str):
payload = json.dumps( {
"content": { "content": {
"raw": comment, "raw": comment,
}, },
"inline": { "inline": {"to": from_line, "path": file},
"to": from_line, }
"path": file )
},
})
response = requests.request( response = requests.request(
"POST", self.bitbucket_comment_api_url, data=payload, headers=self.headers "POST", self.bitbucket_comment_api_url, data=payload, headers=self.headers
) )
@ -177,7 +170,9 @@ class BitbucketProvider(GitProvider):
def publish_inline_comments(self, comments: list[dict]): def publish_inline_comments(self, comments: list[dict]):
for comment in comments: for comment in comments:
self.publish_inline_comment(comment['body'], comment['start_line'], comment['path']) self.publish_inline_comment(
comment["body"], comment["start_line"], comment["line"], comment["path"]
)
def get_title(self): def get_title(self):
return self.pr.title return self.pr.title
@ -245,21 +240,15 @@ class BitbucketProvider(GitProvider):
def get_commit_messages(self): def get_commit_messages(self):
return "" # not implemented yet return "" # not implemented yet
# bitbucket does not support labels def publish_description(self, pr_title: str, pr_body: str):
def publish_description(self, pr_title: str, description: str): pass
payload = json.dumps({ def create_inline_comment(
"description": description, self, body: str, relevant_file: str, relevant_line_in_file: str
"title": pr_title ):
pass
})
def publish_labels(self, labels):
response = requests.request("PUT", self.bitbucket_pull_request_api_url, headers=self.headers, data=payload)
return response
# bitbucket does not support labels
def publish_labels(self, pr_types: list):
pass pass
# bitbucket does not support labels
def get_labels(self): def get_labels(self):
pass pass

View File

@ -1,6 +1,9 @@
try: # Allow this module to be imported without requiring boto3
import boto3 import boto3
import botocore import botocore
except ModuleNotFoundError:
boto3 = None
botocore = None
class CodeCommitDifferencesResponse: class CodeCommitDifferencesResponse:
""" """
@ -54,22 +57,17 @@ class CodeCommitClient:
def __init__(self): def __init__(self):
self.boto_client = None self.boto_client = None
def is_supported(self, capability: str) -> bool:
if capability in ["gfm_markdown"]:
return False
return True
def _connect_boto_client(self): def _connect_boto_client(self):
try: try:
self.boto_client = boto3.client("codecommit") self.boto_client = boto3.client("codecommit")
except Exception as e: except Exception as e:
raise ValueError(f"Failed to connect to AWS CodeCommit: {e}") from e raise ValueError(f"Failed to connect to AWS CodeCommit: {e}")
def get_differences(self, repo_name: int, destination_commit: str, source_commit: str): def get_differences(self, repo_name: int, destination_commit: str, source_commit: str):
""" """
Get the differences between two commits in CodeCommit. Get the differences between two commits in CodeCommit.
Args: Parameters:
- repo_name: Name of the repository - repo_name: Name of the repository
- destination_commit: Commit hash you want to merge into (the "before" hash) (usually on the main or master branch) - destination_commit: Commit hash you want to merge into (the "before" hash) (usually on the main or master branch)
- source_commit: Commit hash of the code you are adding (the "after" branch) - source_commit: Commit hash of the code you are adding (the "after" branch)
@ -78,8 +76,8 @@ class CodeCommitClient:
- List of CodeCommitDifferencesResponse objects - List of CodeCommitDifferencesResponse objects
Boto3 Documentation: Boto3 Documentation:
- aws codecommit get-differences aws codecommit get-differences
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_differences.html https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_differences.html
""" """
if self.boto_client is None: if self.boto_client is None:
self._connect_boto_client() self._connect_boto_client()
@ -95,11 +93,7 @@ class CodeCommitClient:
): ):
differences.extend(page.get("differences", [])) differences.extend(page.get("differences", []))
except botocore.exceptions.ClientError as e: except botocore.exceptions.ClientError as e:
if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException': raise ValueError(f"Failed to retrieve differences from CodeCommit PR #{self.pr_num}") from e
raise ValueError(f"CodeCommit cannot retrieve differences: Repository does not exist: {repo_name}") from e
raise ValueError(f"CodeCommit cannot retrieve differences for {source_commit}..{destination_commit}") from e
except Exception as e:
raise ValueError(f"CodeCommit cannot retrieve differences for {source_commit}..{destination_commit}") from e
output = [] output = []
for json in differences: for json in differences:
@ -110,7 +104,7 @@ class CodeCommitClient:
""" """
Retrieve a file from CodeCommit. Retrieve a file from CodeCommit.
Args: Parameters:
- repo_name: Name of the repository - repo_name: Name of the repository
- file_path: Path to the file you are retrieving - file_path: Path to the file you are retrieving
- sha_hash: Commit hash of the file you are retrieving - sha_hash: Commit hash of the file you are retrieving
@ -119,8 +113,8 @@ class CodeCommitClient:
- File contents - File contents
Boto3 Documentation: Boto3 Documentation:
- aws codecommit get_file aws codecommit get_file
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_file.html https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_file.html
""" """
if not file_path: if not file_path:
return "" return ""
@ -131,8 +125,6 @@ class CodeCommitClient:
try: try:
response = self.boto_client.get_file(repositoryName=repo_name, commitSpecifier=sha_hash, filePath=file_path) response = self.boto_client.get_file(repositoryName=repo_name, commitSpecifier=sha_hash, filePath=file_path)
except botocore.exceptions.ClientError as e: except botocore.exceptions.ClientError as e:
if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
raise ValueError(f"CodeCommit cannot retrieve PR: Repository does not exist: {repo_name}") from e
# if the file does not exist, but is flagged as optional, then return an empty string # if the file does not exist, but is flagged as optional, then return an empty string
if optional and e.response["Error"]["Code"] == 'FileDoesNotExistException': if optional and e.response["Error"]["Code"] == 'FileDoesNotExistException':
return "" return ""
@ -144,20 +136,19 @@ class CodeCommitClient:
return response.get("fileContent", "") return response.get("fileContent", "")
def get_pr(self, repo_name: str, pr_number: int): def get_pr(self, pr_number: int):
""" """
Get a information about a CodeCommit PR. Get a information about a CodeCommit PR.
Args: Parameters:
- repo_name: Name of the repository
- pr_number: The PR number you are requesting - pr_number: The PR number you are requesting
Returns: Returns:
- CodeCommitPullRequestResponse object - CodeCommitPullRequestResponse object
Boto3 Documentation: Boto3 Documentation:
- aws codecommit get_pull_request aws codecommit get_pull_request
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_pull_request.html https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_pull_request.html
""" """
if self.boto_client is None: if self.boto_client is None:
self._connect_boto_client() self._connect_boto_client()
@ -167,8 +158,6 @@ class CodeCommitClient:
except botocore.exceptions.ClientError as e: except botocore.exceptions.ClientError as e:
if e.response["Error"]["Code"] == 'PullRequestDoesNotExistException': if e.response["Error"]["Code"] == 'PullRequestDoesNotExistException':
raise ValueError(f"CodeCommit cannot retrieve PR: PR number does not exist: {pr_number}") from e raise ValueError(f"CodeCommit cannot retrieve PR: PR number does not exist: {pr_number}") from e
if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
raise ValueError(f"CodeCommit cannot retrieve PR: Repository does not exist: {repo_name}") from e
raise ValueError(f"CodeCommit cannot retrieve PR: {pr_number}: boto client error") from e raise ValueError(f"CodeCommit cannot retrieve PR: {pr_number}: boto client error") from e
except Exception as e: except Exception as e:
raise ValueError(f"CodeCommit cannot retrieve PR: {pr_number}") from e raise ValueError(f"CodeCommit cannot retrieve PR: {pr_number}") from e
@ -178,88 +167,28 @@ class CodeCommitClient:
return CodeCommitPullRequestResponse(response.get("pullRequest", {})) return CodeCommitPullRequestResponse(response.get("pullRequest", {}))
def publish_description(self, pr_number: int, pr_title: str, pr_body: str): def publish_comment(self, repo_name: str, pr_number: int, destination_commit: str, source_commit: str, comment: str):
"""
Set the title and description on a pull request
Args:
- pr_number: the AWS CodeCommit pull request number
- pr_title: title of the pull request
- pr_body: body of the pull request
Returns:
- None
Boto3 Documentation:
- aws codecommit update_pull_request_title
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/update_pull_request_title.html
- aws codecommit update_pull_request_description
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/update_pull_request_description.html
"""
if self.boto_client is None:
self._connect_boto_client()
try:
self.boto_client.update_pull_request_title(pullRequestId=str(pr_number), title=pr_title)
self.boto_client.update_pull_request_description(pullRequestId=str(pr_number), description=pr_body)
except botocore.exceptions.ClientError as e:
if e.response["Error"]["Code"] == 'PullRequestDoesNotExistException':
raise ValueError(f"PR number does not exist: {pr_number}") from e
if e.response["Error"]["Code"] == 'InvalidTitleException':
raise ValueError(f"Invalid title for PR number: {pr_number}") from e
if e.response["Error"]["Code"] == 'InvalidDescriptionException':
raise ValueError(f"Invalid description for PR number: {pr_number}") from e
if e.response["Error"]["Code"] == 'PullRequestAlreadyClosedException':
raise ValueError(f"PR is already closed: PR number: {pr_number}") from e
raise ValueError(f"Boto3 client error calling publish_description") from e
except Exception as e:
raise ValueError(f"Error calling publish_description") from e
def publish_comment(self, repo_name: str, pr_number: int, destination_commit: str, source_commit: str, comment: str, annotation_file: str = None, annotation_line: int = None):
""" """
Publish a comment to a pull request Publish a comment to a pull request
Args: Parameters:
- repo_name: name of the repository - repo_name: name of the repository
- pr_number: number of the pull request - pr_number: number of the pull request
- destination_commit: The commit hash you want to merge into (the "before" hash) (usually on the main or master branch) - destination_commit: The commit hash you want to merge into (the "before" hash) (usually on the main or master branch)
- source_commit: The commit hash of the code you are adding (the "after" branch) - source_commit: The commit hash of the code you are adding (the "after" branch)
- comment: The comment you want to publish - pr_comment: comment
- annotation_file: The file you want to annotate (optional)
- annotation_line: The line number you want to annotate (optional)
Comment annotations for CodeCommit are different than GitHub.
CodeCommit only designates the starting line number for the comment.
It does not support the ending line number to highlight a range of lines.
Returns: Returns:
- None - None
Boto3 Documentation: Boto3 Documentation:
- aws codecommit post_comment_for_pull_request aws codecommit post_comment_for_pull_request
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_pull_request.html https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_pull_request.html
""" """
if self.boto_client is None: if self.boto_client is None:
self._connect_boto_client() self._connect_boto_client()
try: try:
# If the comment has code annotations,
# then set the file path and line number in the location dictionary
if annotation_file and annotation_line:
self.boto_client.post_comment_for_pull_request(
pullRequestId=str(pr_number),
repositoryName=repo_name,
beforeCommitId=destination_commit,
afterCommitId=source_commit,
content=comment,
location={
"filePath": annotation_file,
"filePosition": annotation_line,
"relativeFileVersion": "AFTER",
},
)
else:
# The comment does not have code annotations
self.boto_client.post_comment_for_pull_request( self.boto_client.post_comment_for_pull_request(
pullRequestId=str(pr_number), pullRequestId=str(pr_number),
repositoryName=repo_name, repositoryName=repo_name,

View File

@ -1,17 +1,15 @@
import logging import logging
import os import os
import re
from collections import Counter from collections import Counter
from typing import List, Optional, Tuple from typing import List, Optional, Tuple
from urllib.parse import urlparse from urllib.parse import urlparse
from ..algo.language_handler import is_valid_file, language_extension_map
from ..algo.pr_processing import clip_tokens
from ..algo.utils import load_large_diff
from ..config_loader import get_settings
from .git_provider import EDIT_TYPE, FilePatchInfo, GitProvider, IncrementalPR
from pr_agent.git_providers.codecommit_client import CodeCommitClient from pr_agent.git_providers.codecommit_client import CodeCommitClient
from ..algo.language_handler import is_valid_file, language_extension_map
from ..algo.utils import EDIT_TYPE, FilePatchInfo, load_large_diff
from .git_provider import GitProvider
class PullRequestCCMimic: class PullRequestCCMimic:
""" """
@ -74,7 +72,6 @@ class CodeCommitProvider(GitProvider):
"create_inline_comment", "create_inline_comment",
"publish_inline_comments", "publish_inline_comments",
"get_labels", "get_labels",
"gfm_markdown"
]: ]:
return False return False
return True return True
@ -155,63 +152,26 @@ class CodeCommitProvider(GitProvider):
return self.diff_files return self.diff_files
def publish_description(self, pr_title: str, pr_body: str): def publish_description(self, pr_title: str, pr_body: str):
try: return "" # not implemented yet
self.codecommit_client.publish_description(
pr_number=self.pr_num,
pr_title=pr_title,
pr_body=CodeCommitProvider._add_additional_newlines(pr_body),
)
except Exception as e:
raise ValueError(f"CodeCommit Cannot publish description for PR: {self.pr_num}") from e
def publish_comment(self, pr_comment: str, is_temporary: bool = False): def publish_comment(self, pr_comment: str, is_temporary: bool = False):
if is_temporary: if is_temporary:
logging.info(pr_comment) logging.info(pr_comment)
return return
pr_comment = CodeCommitProvider._remove_markdown_html(pr_comment)
pr_comment = CodeCommitProvider._add_additional_newlines(pr_comment)
try: try:
self.codecommit_client.publish_comment( self.codecommit_client.publish_comment(
repo_name=self.repo_name, repo_name=self.repo_name,
pr_number=self.pr_num, pr_number=str(self.pr_num),
destination_commit=self.pr.destination_commit, destination_commit=self.pr.destination_commit,
source_commit=self.pr.source_commit, source_commit=self.pr.source_commit,
comment=pr_comment, comment=pr_comment,
) )
except Exception as e: except Exception as e:
raise ValueError(f"CodeCommit Cannot publish comment for PR: {self.pr_num}") from e raise ValueError(f"CodeCommit Cannot post comment for PR: {self.pr_num}") from e
def publish_code_suggestions(self, code_suggestions: list) -> bool: def publish_code_suggestions(self, code_suggestions: list) -> bool:
counter = 1 return [""] # not implemented yet
for suggestion in code_suggestions:
# Verify that each suggestion has the required keys
if not all(key in suggestion for key in ["body", "relevant_file", "relevant_lines_start"]):
logging.warning(f"Skipping code suggestion #{counter}: Each suggestion must have 'body', 'relevant_file', 'relevant_lines_start' keys")
continue
# Publish the code suggestion to CodeCommit
try:
logging.debug(f"Code Suggestion #{counter} in file: {suggestion['relevant_file']}: {suggestion['relevant_lines_start']}")
self.codecommit_client.publish_comment(
repo_name=self.repo_name,
pr_number=self.pr_num,
destination_commit=self.pr.destination_commit,
source_commit=self.pr.source_commit,
comment=suggestion["body"],
annotation_file=suggestion["relevant_file"],
annotation_line=suggestion["relevant_lines_start"],
)
except Exception as e:
raise ValueError(f"CodeCommit Cannot publish code suggestions for PR: {self.pr_num}") from e
counter += 1
# The calling function passes in a list of code suggestions, and this function publishes each suggestion one at a time.
# If we were to return False here, the calling function will attempt to publish the same list of code suggestions again, one at a time.
# Since this function publishes the suggestions one at a time anyway, we always return True here to avoid the retry.
return True
def publish_labels(self, labels): def publish_labels(self, labels):
return [""] # not implemented yet return [""] # not implemented yet
@ -223,7 +183,6 @@ class CodeCommitProvider(GitProvider):
return "" # not implemented yet return "" # not implemented yet
def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str): def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_compared_commit.html
raise NotImplementedError("CodeCommit provider does not support publishing inline comments yet") raise NotImplementedError("CodeCommit provider does not support publishing inline comments yet")
def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str): def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
@ -240,7 +199,7 @@ class CodeCommitProvider(GitProvider):
Returns a dictionary of languages, containing the percentage of each language used in the PR. Returns a dictionary of languages, containing the percentage of each language used in the PR.
Returns: Returns:
- dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR. dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR.
""" """
commit_files = self.get_files() commit_files = self.get_files()
filenames = [ item.filename for item in commit_files ] filenames = [ item.filename for item in commit_files ]
@ -284,29 +243,18 @@ class CodeCommitProvider(GitProvider):
return self.codecommit_client.get_file(self.repo_name, settings_filename, self.pr.source_commit, optional=True) return self.codecommit_client.get_file(self.repo_name, settings_filename, self.pr.source_commit, optional=True)
def add_eyes_reaction(self, issue_comment_id: int) -> Optional[int]: def add_eyes_reaction(self, issue_comment_id: int) -> Optional[int]:
logging.info("CodeCommit provider does not support eyes reaction yet")
return True return True
def remove_reaction(self, issue_comment_id: int, reaction_id: int) -> bool: def remove_reaction(self, issue_comment_id: int, reaction_id: int) -> bool:
logging.info("CodeCommit provider does not support removing reactions yet")
return True return True
@staticmethod @staticmethod
def _parse_pr_url(pr_url: str) -> Tuple[str, int]: def _parse_pr_url(pr_url: str) -> Tuple[str, int]:
"""
Parse the CodeCommit PR URL and return the repository name and PR number.
Args:
- pr_url: the full AWS CodeCommit pull request URL
Returns:
- Tuple[str, int]: A tuple containing the repository name and PR number.
"""
# Example PR URL: # Example PR URL:
# https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/__MY_REPO__/pull-requests/123456" # https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/__MY_REPO__/pull-requests/123456"
parsed_url = urlparse(pr_url) parsed_url = urlparse(pr_url)
if not CodeCommitProvider._is_valid_codecommit_hostname(parsed_url.netloc): if "us-east-1.console.aws.amazon.com" not in parsed_url.netloc:
raise ValueError(f"The provided URL is not a valid CodeCommit URL: {pr_url}") raise ValueError(f"The provided URL is not a valid CodeCommit URL: {pr_url}")
path_parts = parsed_url.path.strip("/").split("/") path_parts = parsed_url.path.strip("/").split("/")
@ -329,33 +277,17 @@ class CodeCommitProvider(GitProvider):
return repo_name, pr_number return repo_name, pr_number
@staticmethod
def _is_valid_codecommit_hostname(hostname: str) -> bool:
"""
Check if the provided hostname is a valid AWS CodeCommit hostname.
This is not an exhaustive check of AWS region names,
but instead uses a regex to check for matching AWS region patterns.
Args:
- hostname: the hostname to check
Returns:
- bool: True if the hostname is valid, False otherwise.
"""
return re.match(r"^[a-z]{2}-(gov-)?[a-z]+-\d\.console\.aws\.amazon\.com$", hostname) is not None
def _get_pr(self): def _get_pr(self):
response = self.codecommit_client.get_pr(self.repo_name, self.pr_num) response = self.codecommit_client.get_pr(self.pr_num)
if len(response.targets) == 0: if len(response.targets) == 0:
raise ValueError(f"No files found in CodeCommit PR: {self.pr_num}") raise ValueError(f"No files found in CodeCommit PR: {self.pr_num}")
# TODO: implement support for multiple targets in one CodeCommit PR # TODO: implement support for multiple commits in one CodeCommit PR
# for now, we are only using the first target in the PR # for now, we are only using the first commit in the PR
if len(response.targets) > 1: if len(response.targets) > 1:
logging.warning( logging.warning(
"Multiple targets in one PR is not supported for CodeCommit yet. Continuing, using the first target only..." "Multiple commits in one PR is not supported for CodeCommit yet. Continuing, using the first commit only..."
) )
# Return our object that mimics PullRequest class from the PyGithub library # Return our object that mimics PullRequest class from the PyGithub library
@ -373,52 +305,13 @@ class CodeCommitProvider(GitProvider):
return "" # not implemented yet return "" # not implemented yet
@staticmethod @staticmethod
def _add_additional_newlines(body: str) -> str: def _get_edit_type(codecommit_change_type):
"""
Replace single newlines in a PR body with double newlines.
CodeCommit Markdown does not seem to render as well as GitHub Markdown,
so we add additional newlines to the PR body to make it more readable in CodeCommit.
Args:
- body: the PR body
Returns:
- str: the PR body with the double newlines added
"""
return re.sub(r'(?<!\n)\n(?!\n)', '\n\n', body)
@staticmethod
def _remove_markdown_html(comment: str) -> str:
"""
Remove the HTML tags from a PR comment.
CodeCommit Markdown does not seem to render as well as GitHub Markdown,
so we remove the HTML tags from the PR comment to make it more readable in CodeCommit.
Args:
- comment: the PR comment
Returns:
- str: the PR comment with the HTML tags removed
"""
comment = comment.replace("<details>", "")
comment = comment.replace("</details>", "")
comment = comment.replace("<summary>", "")
comment = comment.replace("</summary>", "")
return comment
@staticmethod
def _get_edit_type(codecommit_change_type: str):
""" """
Convert the CodeCommit change type string to the EDIT_TYPE enum. Convert the CodeCommit change type string to the EDIT_TYPE enum.
The CodeCommit change type string is returned from the get_differences SDK method. The CodeCommit change type string is returned from the get_differences SDK method.
Args:
- codecommit_change_type: the CodeCommit change type string
Returns: Returns:
- An EDIT_TYPE enum representing the modified, added, deleted, or renamed file in the PR diff. An EDIT_TYPE enum representing the modified, added, deleted, or renamed file in the PR diff.
""" """
t = codecommit_change_type.upper() t = codecommit_change_type.upper()
edit_type = None edit_type = None
@ -439,12 +332,6 @@ class CodeCommitProvider(GitProvider):
The returned extensions will include the dot "." prefix, The returned extensions will include the dot "." prefix,
to accommodate for the dots in the existing language_extension_map settings. to accommodate for the dots in the existing language_extension_map settings.
Filenames with no extension will return an empty string for the extension. Filenames with no extension will return an empty string for the extension.
Args:
- filenames: a list of filenames
Returns:
- list: A list of file extensions, including the dot "." prefix.
""" """
extensions = [] extensions = []
for filename in filenames: for filename in filenames:
@ -461,12 +348,6 @@ class CodeCommitProvider(GitProvider):
Return a dictionary containing the programming language name (as the key), Return a dictionary containing the programming language name (as the key),
and the percentage that language is used (as the value), and the percentage that language is used (as the value),
given a list of file extensions. given a list of file extensions.
Args:
- extensions: a list of file extensions
Returns:
- dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR.
""" """
total_files = len(extensions) total_files = len(extensions)
if total_files == 0: if total_files == 0:

View File

@ -1,403 +0,0 @@
import json
import logging
import os
import pathlib
import shutil
import subprocess
import uuid
from collections import Counter, namedtuple
from pathlib import Path
from tempfile import mkdtemp, NamedTemporaryFile
import requests
import urllib3.util
from git import Repo
from pr_agent.config_loader import get_settings
from pr_agent.git_providers.git_provider import GitProvider, FilePatchInfo, \
EDIT_TYPE
from pr_agent.git_providers.local_git_provider import PullRequestMimic
logger = logging.getLogger(__name__)
def _call(*command, **kwargs) -> (int, str, str):
res = subprocess.run(
command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
check=True,
**kwargs,
)
return res.stdout.decode()
def clone(url, directory):
logger.info("Cloning %s to %s", url, directory)
stdout = _call('git', 'clone', "--depth", "1", url, directory)
logger.info(stdout)
def fetch(url, refspec, cwd):
logger.info("Fetching %s %s", url, refspec)
stdout = _call(
'git', 'fetch', '--depth', '2', url, refspec,
cwd=cwd
)
logger.info(stdout)
def checkout(cwd):
logger.info("Checking out")
stdout = _call('git', 'checkout', "FETCH_HEAD", cwd=cwd)
logger.info(stdout)
def show(*args, cwd=None):
logger.info("Show")
return _call('git', 'show', *args, cwd=cwd)
def diff(*args, cwd=None):
logger.info("Diff")
patch = _call('git', 'diff', *args, cwd=cwd)
if not patch:
logger.warning("No changes found")
return
return patch
def reset_local_changes(cwd):
logger.info("Reset local changes")
_call('git', 'checkout', "--force", cwd=cwd)
def add_comment(url: urllib3.util.Url, refspec, message):
*_, patchset, changenum = refspec.rsplit("/")
message = "'" + message.replace("'", "'\"'\"'") + "'"
return _call(
"ssh",
"-p", str(url.port),
f"{url.auth}@{url.host}",
"gerrit", "review",
"--message", message,
# "--code-review", score,
f"{patchset},{changenum}",
)
def list_comments(url: urllib3.util.Url, refspec):
*_, patchset, _ = refspec.rsplit("/")
stdout = _call(
"ssh",
"-p", str(url.port),
f"{url.auth}@{url.host}",
"gerrit", "query",
"--comments",
"--current-patch-set", patchset,
"--format", "JSON",
)
change_set, *_ = stdout.splitlines()
return json.loads(change_set)["currentPatchSet"]["comments"]
def prepare_repo(url: urllib3.util.Url, project, refspec):
repo_url = (f"{url.scheme}://{url.auth}@{url.host}:{url.port}/{project}")
directory = pathlib.Path(mkdtemp())
clone(repo_url, directory),
fetch(repo_url, refspec, cwd=directory)
checkout(cwd=directory)
return directory
def adopt_to_gerrit_message(message):
lines = message.splitlines()
buf = []
for line in lines:
# remove markdown formatting
line = (line.replace("*", "")
.replace("``", "`")
.replace("<details>", "")
.replace("</details>", "")
.replace("<summary>", "")
.replace("</summary>", ""))
line = line.strip()
if line.startswith('#'):
buf.append("\n" +
line.replace('#', '').removesuffix(":").strip() +
":")
continue
elif line.startswith('-'):
buf.append(line.removeprefix('-').strip())
continue
else:
buf.append(line)
return "\n".join(buf).strip()
def add_suggestion(src_filename, context: str, start, end: int):
with (
NamedTemporaryFile("w", delete=False) as tmp,
open(src_filename, "r") as src
):
lines = src.readlines()
tmp.writelines(lines[:start - 1])
if context:
tmp.write(context)
tmp.writelines(lines[end:])
shutil.copy(tmp.name, src_filename)
os.remove(tmp.name)
def upload_patch(patch, path):
patch_server_endpoint = get_settings().get(
'gerrit.patch_server_endpoint')
patch_server_token = get_settings().get(
'gerrit.patch_server_token')
response = requests.post(
patch_server_endpoint,
json={
"content": patch,
"path": path,
},
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {patch_server_token}",
}
)
response.raise_for_status()
patch_server_endpoint = patch_server_endpoint.rstrip("/")
return patch_server_endpoint + "/" + path
class GerritProvider(GitProvider):
def __init__(self, key: str, incremental=False):
self.project, self.refspec = key.split(':')
assert self.project, "Project name is required"
assert self.refspec, "Refspec is required"
base_url = get_settings().get('gerrit.url')
assert base_url, "Gerrit URL is required"
user = get_settings().get('gerrit.user')
assert user, "Gerrit user is required"
parsed = urllib3.util.parse_url(base_url)
self.parsed_url = urllib3.util.parse_url(
f"{parsed.scheme}://{user}@{parsed.host}:{parsed.port}"
)
self.repo_path = prepare_repo(
self.parsed_url, self.project, self.refspec
)
self.repo = Repo(self.repo_path)
assert self.repo
self.pr = PullRequestMimic(self.get_pr_title(), self.get_diff_files())
def get_pr_title(self):
"""
Substitutes the branch-name as the PR-mimic title.
"""
return self.repo.branches[0].name
def get_issue_comments(self):
comments = list_comments(self.parsed_url, self.refspec)
Comments = namedtuple('Comments', ['reversed'])
Comment = namedtuple('Comment', ['body'])
return Comments([Comment(c['message']) for c in reversed(comments)])
def get_labels(self):
raise NotImplementedError(
'Getting labels is not implemented for the gerrit provider')
def add_eyes_reaction(self, issue_comment_id: int):
raise NotImplementedError(
'Adding reactions is not implemented for the gerrit provider')
def remove_reaction(self, issue_comment_id: int, reaction_id: int):
raise NotImplementedError(
'Removing reactions is not implemented for the gerrit provider')
def get_commit_messages(self):
return [self.repo.head.commit.message]
def get_repo_settings(self):
try:
with open(self.repo_path / ".pr_agent.toml", 'rb') as f:
contents = f.read()
return contents
except OSError:
return b""
def get_diff_files(self) -> list[FilePatchInfo]:
diffs = self.repo.head.commit.diff(
self.repo.head.commit.parents[0], # previous commit
create_patch=True,
R=True
)
diff_files = []
for diff_item in diffs:
if diff_item.a_blob is not None:
original_file_content_str = (
diff_item.a_blob.data_stream.read().decode('utf-8')
)
else:
original_file_content_str = "" # empty file
if diff_item.b_blob is not None:
new_file_content_str = diff_item.b_blob.data_stream.read(). \
decode('utf-8')
else:
new_file_content_str = "" # empty file
edit_type = EDIT_TYPE.MODIFIED
if diff_item.new_file:
edit_type = EDIT_TYPE.ADDED
elif diff_item.deleted_file:
edit_type = EDIT_TYPE.DELETED
elif diff_item.renamed_file:
edit_type = EDIT_TYPE.RENAMED
diff_files.append(
FilePatchInfo(
original_file_content_str,
new_file_content_str,
diff_item.diff.decode('utf-8'),
diff_item.b_path,
edit_type=edit_type,
old_filename=None
if diff_item.a_path == diff_item.b_path
else diff_item.a_path
)
)
self.diff_files = diff_files
return diff_files
def get_files(self):
diff_index = self.repo.head.commit.diff(
self.repo.head.commit.parents[0], # previous commit
R=True
)
# Get the list of changed files
diff_files = [item.a_path for item in diff_index]
return diff_files
def get_languages(self):
"""
Calculate percentage of languages in repository. Used for hunk
prioritisation.
"""
# Get all files in repository
filepaths = [Path(item.path) for item in
self.repo.tree().traverse() if item.type == 'blob']
# Identify language by file extension and count
lang_count = Counter(
ext.lstrip('.') for filepath in filepaths for ext in
[filepath.suffix.lower()])
# Convert counts to percentages
total_files = len(filepaths)
lang_percentage = {lang: count / total_files * 100 for lang, count
in lang_count.items()}
return lang_percentage
def get_pr_description_full(self):
return self.repo.head.commit.message
def get_user_id(self):
return self.repo.head.commit.author.email
def is_supported(self, capability: str) -> bool:
if capability in [
# 'get_issue_comments',
'create_inline_comment',
'publish_inline_comments',
'get_labels',
'gfm_markdown'
]:
return False
return True
def split_suggestion(self, msg) -> tuple[str, str]:
is_code_context = False
description = []
context = []
for line in msg.splitlines():
if line.startswith('```suggestion'):
is_code_context = True
continue
if line.startswith('```'):
is_code_context = False
continue
if is_code_context:
context.append(line)
else:
description.append(
line.replace('*', '')
)
return (
'\n'.join(description),
'\n'.join(context) + '\n' if context else ''
)
def publish_code_suggestions(self, code_suggestions: list):
msg = []
for suggestion in code_suggestions:
description, code = self.split_suggestion(suggestion['body'])
add_suggestion(
pathlib.Path(self.repo_path) / suggestion["relevant_file"],
code,
suggestion["relevant_lines_start"],
suggestion["relevant_lines_end"],
)
patch = diff(cwd=self.repo_path)
patch_id = uuid.uuid4().hex[0:4]
path = "/".join(["codium-ai", self.refspec, patch_id])
full_path = upload_patch(patch, path)
reset_local_changes(self.repo_path)
msg.append(f'* {description}\n{full_path}')
if msg:
add_comment(self.parsed_url, self.refspec, "\n".join(msg))
return True
def publish_comment(self, pr_comment: str, is_temporary: bool = False):
if not is_temporary:
msg = adopt_to_gerrit_message(pr_comment)
add_comment(self.parsed_url, self.refspec, msg)
def publish_description(self, pr_title: str, pr_body: str):
msg = adopt_to_gerrit_message(pr_body)
add_comment(self.parsed_url, self.refspec, pr_title + '\n' + msg)
def publish_inline_comments(self, comments: list[dict]):
raise NotImplementedError(
'Publishing inline comments is not implemented for the gerrit '
'provider')
def publish_inline_comment(self, body: str, relevant_file: str,
relevant_line_in_file: str):
raise NotImplementedError(
'Publishing inline comments is not implemented for the gerrit '
'provider')
def create_inline_comment(self, body: str, relevant_file: str,
relevant_line_in_file: str):
raise NotImplementedError(
'Creating inline comments is not implemented for the gerrit '
'provider')
def publish_labels(self, labels):
# Not applicable to the local git provider,
# but required by the interface
pass
def remove_initial_comment(self):
# remove repo, cloned in previous steps
# shutil.rmtree(self.repo_path)
pass
def get_pr_branch(self):
return self.repo.head

View File

@ -1,28 +1,10 @@
import logging import logging
from abc import ABC, abstractmethod from abc import ABC, abstractmethod
from dataclasses import dataclass
# enum EDIT_TYPE (ADDED, DELETED, MODIFIED, RENAMED) # enum EDIT_TYPE (ADDED, DELETED, MODIFIED, RENAMED)
from enum import Enum
from typing import Optional from typing import Optional
from pr_agent.algo.utils import FilePatchInfo
class EDIT_TYPE(Enum):
ADDED = 1
DELETED = 2
MODIFIED = 3
RENAMED = 4
@dataclass
class FilePatchInfo:
base_file: str
head_file: str
patch: str
filename: str
tokens: int = -1
edit_type: EDIT_TYPE = EDIT_TYPE.MODIFIED
old_filename: str = None
class GitProvider(ABC): class GitProvider(ABC):
@ -86,11 +68,11 @@ class GitProvider(ABC):
def get_pr_description_full(self) -> str: def get_pr_description_full(self) -> str:
pass pass
def get_pr_description(self, *, full: bool = True) -> str: def get_pr_description(self) -> str:
from pr_agent.config_loader import get_settings from pr_agent.config_loader import get_settings
from pr_agent.algo.pr_processing import clip_tokens from pr_agent.algo.utils import clip_tokens
max_tokens = get_settings().get("CONFIG.MAX_DESCRIPTION_TOKENS", None) max_tokens = get_settings().get("CONFIG.MAX_DESCRIPTION_TOKENS", None)
description = self.get_pr_description_full() if full else self.get_user_description() description = self.get_pr_description_full()
if max_tokens: if max_tokens:
return clip_tokens(description, max_tokens) return clip_tokens(description, max_tokens)
return description return description
@ -132,10 +114,6 @@ def get_main_pr_language(languages, files) -> str:
Get the main language of the commit. Return an empty string if cannot determine. Get the main language of the commit. Return an empty string if cannot determine.
""" """
main_language_str = "" main_language_str = ""
if not languages:
logging.info("No languages detected")
return main_language_str
try: try:
top_language = max(languages, key=languages.get).lower() top_language = max(languages, key=languages.get).lower()
@ -165,6 +143,7 @@ def get_main_pr_language(languages, files) -> str:
most_common_extension == 'scala' and top_language == 'scala' or \ most_common_extension == 'scala' and top_language == 'scala' or \
most_common_extension == 'kt' and top_language == 'kotlin' or \ most_common_extension == 'kt' and top_language == 'kotlin' or \
most_common_extension == 'pl' and top_language == 'perl' or \ most_common_extension == 'pl' and top_language == 'perl' or \
most_common_extension == 'swift' and top_language == 'swift' or \
most_common_extension == top_language: most_common_extension == top_language:
main_language_str = top_language main_language_str = top_language

View File

@ -9,10 +9,9 @@ from github import AppAuthentication, Auth, Github, GithubException, Reaction
from retry import retry from retry import retry
from starlette_context import context from starlette_context import context
from .git_provider import FilePatchInfo, GitProvider, IncrementalPR from .git_provider import GitProvider, IncrementalPR
from ..algo.language_handler import is_valid_file from ..algo.language_handler import is_valid_file
from ..algo.utils import load_large_diff from ..algo.utils import load_large_diff, clip_tokens, find_line_number_of_relevant_line_in_file, FilePatchInfo
from ..algo.pr_processing import find_line_number_of_relevant_line_in_file, clip_tokens
from ..config_loader import get_settings from ..config_loader import get_settings
from ..servers.utils import RateLimitExceeded from ..servers.utils import RateLimitExceeded
@ -32,7 +31,7 @@ class GithubProvider(GitProvider):
self.diff_files = None self.diff_files = None
self.git_files = None self.git_files = None
self.incremental = incremental self.incremental = incremental
if pr_url and 'pull' in pr_url: if pr_url:
self.set_pr(pr_url) self.set_pr(pr_url)
self.last_commit_id = list(self.pr.get_commits())[-1] self.last_commit_id = list(self.pr.get_commits())[-1]
@ -309,35 +308,6 @@ class GithubProvider(GitProvider):
return repo_name, pr_number return repo_name, pr_number
@staticmethod
def _parse_issue_url(issue_url: str) -> Tuple[str, int]:
parsed_url = urlparse(issue_url)
if 'github.com' not in parsed_url.netloc:
raise ValueError("The provided URL is not a valid GitHub URL")
path_parts = parsed_url.path.strip('/').split('/')
if 'api.github.com' in parsed_url.netloc:
if len(path_parts) < 5 or path_parts[3] != 'issues':
raise ValueError("The provided URL does not appear to be a GitHub ISSUE URL")
repo_name = '/'.join(path_parts[1:3])
try:
issue_number = int(path_parts[4])
except ValueError as e:
raise ValueError("Unable to convert issue number to integer") from e
return repo_name, issue_number
if len(path_parts) < 4 or path_parts[2] != 'issues':
raise ValueError("The provided URL does not appear to be a GitHub PR issue")
repo_name = '/'.join(path_parts[:2])
try:
issue_number = int(path_parts[3])
except ValueError as e:
raise ValueError("Unable to convert issue number to integer") from e
return repo_name, issue_number
def _get_github_client(self): def _get_github_client(self):
deployment_type = get_settings().get("GITHUB.DEPLOYMENT_TYPE", "user") deployment_type = get_settings().get("GITHUB.DEPLOYMENT_TYPE", "user")

View File

@ -7,10 +7,9 @@ import gitlab
from gitlab import GitlabGetError from gitlab import GitlabGetError
from ..algo.language_handler import is_valid_file from ..algo.language_handler import is_valid_file
from ..algo.pr_processing import clip_tokens from ..algo.utils import load_large_diff, clip_tokens, EDIT_TYPE, FilePatchInfo
from ..algo.utils import load_large_diff
from ..config_loader import get_settings from ..config_loader import get_settings
from .git_provider import EDIT_TYPE, FilePatchInfo, GitProvider from .git_provider import GitProvider
logger = logging.getLogger() logger = logging.getLogger()
@ -43,7 +42,7 @@ class GitLabProvider(GitProvider):
self.incremental = incremental self.incremental = incremental
def is_supported(self, capability: str) -> bool: def is_supported(self, capability: str) -> bool:
if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'gfm_markdown']: if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments']:
return False return False
return True return True

View File

@ -0,0 +1,79 @@
import itertools
from collections import Counter
from typing import List, Optional
from pr_agent.algo.utils import FilePatchInfo
from pr_agent.git_providers.git_provider import GitProvider
class InMemoryProvider(GitProvider):
def __init__(self, head_branch: str, target_branch: str, files: List[FilePatchInfo]):
self.head_branch = head_branch
self.target_branch = target_branch
self.files = files
def is_supported(self, capability: str) -> bool:
pass
def get_files(self) -> list[FilePatchInfo]:
return self.files
def get_diff_files(self) -> list[FilePatchInfo]:
return self.get_files()
def publish_description(self, pr_title: str, pr_body: str):
pass
def publish_comment(self, pr_comment: str, is_temporary: bool = False):
pass
def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
pass
def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
pass
def publish_inline_comments(self, comments: list[dict]):
pass
def publish_code_suggestions(self, code_suggestions: list) -> bool:
pass
def publish_labels(self, labels):
pass
def get_labels(self):
pass
def remove_initial_comment(self):
pass
def get_languages(self):
language_count = Counter(file.language for file in self.files)
return dict(language_count)
def get_pr_branch(self):
pass
def get_user_id(self):
pass
def get_pr_description_full(self) -> str:
pass
def get_issue_comments(self):
pass
def get_repo_settings(self):
pass
def add_eyes_reaction(self, issue_comment_id: int) -> Optional[int]:
pass
def remove_reaction(self, issue_comment_id: int, reaction_id: int) -> bool:
pass
def get_commit_messages(self):
pass

View File

@ -6,7 +6,8 @@ from typing import List
from git import Repo from git import Repo
from pr_agent.config_loader import _find_repository_root, get_settings from pr_agent.config_loader import _find_repository_root, get_settings
from pr_agent.git_providers.git_provider import EDIT_TYPE, FilePatchInfo, GitProvider from pr_agent.git_providers.git_provider import GitProvider
from pr_agent.algo.utils import EDIT_TYPE, FilePatchInfo
class PullRequestMimic: class PullRequestMimic:
@ -56,8 +57,7 @@ class LocalGitProvider(GitProvider):
raise KeyError(f'Branch: {self.target_branch_name} does not exist') raise KeyError(f'Branch: {self.target_branch_name} does not exist')
def is_supported(self, capability: str) -> bool: def is_supported(self, capability: str) -> bool:
if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels', if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels']:
'gfm_markdown']:
return False return False
return True return True

View File

@ -1,78 +0,0 @@
import copy
import logging
import sys
from enum import Enum
from json import JSONDecodeError
import uvicorn
from fastapi import APIRouter, FastAPI, HTTPException
from pydantic import BaseModel
from starlette.middleware import Middleware
from starlette_context import context
from starlette_context.middleware import RawContextMiddleware
from pr_agent.agent.pr_agent import PRAgent
from pr_agent.config_loader import global_settings, get_settings
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
router = APIRouter()
class Action(str, Enum):
review = "review"
describe = "describe"
ask = "ask"
improve = "improve"
reflect = "reflect"
answer = "answer"
class Item(BaseModel):
refspec: str
project: str
msg: str
@router.post("/api/v1/gerrit/{action}")
async def handle_gerrit_request(action: Action, item: Item):
logging.debug("Received a Gerrit request")
context["settings"] = copy.deepcopy(global_settings)
if action == Action.ask:
if not item.msg:
return HTTPException(
status_code=400,
detail="msg is required for ask command"
)
await PRAgent().handle_request(
f"{item.project}:{item.refspec}",
f"/{item.msg.strip()}"
)
async def get_body(request):
try:
body = await request.json()
except JSONDecodeError as e:
logging.error("Error parsing request body", e)
return {}
return body
@router.get("/")
async def root():
return {"status": "ok"}
def start():
# to prevent adding help messages with the output
get_settings().set("CONFIG.CLI_MODE", True)
middleware = [Middleware(RawContextMiddleware)]
app = FastAPI(middleware=middleware)
app.include_router(router)
uvicorn.run(app, host="0.0.0.0", port=3000)
if __name__ == '__main__':
start()

View File

@ -12,8 +12,8 @@ async def run_action():
# Get environment variables # Get environment variables
GITHUB_EVENT_NAME = os.environ.get('GITHUB_EVENT_NAME') GITHUB_EVENT_NAME = os.environ.get('GITHUB_EVENT_NAME')
GITHUB_EVENT_PATH = os.environ.get('GITHUB_EVENT_PATH') GITHUB_EVENT_PATH = os.environ.get('GITHUB_EVENT_PATH')
OPENAI_KEY = os.environ.get('OPENAI_KEY') or os.environ.get('OPENAI.KEY') OPENAI_KEY = os.environ.get('OPENAI_KEY')
OPENAI_ORG = os.environ.get('OPENAI_ORG') or os.environ.get('OPENAI.ORG') OPENAI_ORG = os.environ.get('OPENAI_ORG')
GITHUB_TOKEN = os.environ.get('GITHUB_TOKEN') GITHUB_TOKEN = os.environ.get('GITHUB_TOKEN')
get_settings().set("CONFIG.PUBLISH_OUTPUT_PROGRESS", False) get_settings().set("CONFIG.PUBLISH_OUTPUT_PROGRESS", False)
@ -61,21 +61,12 @@ async def run_action():
if action in ["created", "edited"]: if action in ["created", "edited"]:
comment_body = event_payload.get("comment", {}).get("body") comment_body = event_payload.get("comment", {}).get("body")
if comment_body: if comment_body:
is_pr = False pr_url = event_payload.get("issue", {}).get("pull_request", {}).get("url")
# check if issue is pull request if pr_url:
if event_payload.get("issue", {}).get("pull_request"):
url = event_payload.get("issue", {}).get("pull_request", {}).get("url")
is_pr = True
else:
url = event_payload.get("issue", {}).get("url")
if url:
body = comment_body.strip().lower() body = comment_body.strip().lower()
comment_id = event_payload.get("comment", {}).get("id") comment_id = event_payload.get("comment", {}).get("id")
provider = get_git_provider()(pr_url=url) provider = get_git_provider()(pr_url=pr_url)
if is_pr: await PRAgent().handle_request(pr_url, body, notify=lambda: provider.add_eyes_reaction(comment_id))
await PRAgent().handle_request(url, body, notify=lambda: provider.add_eyes_reaction(comment_id))
else:
await PRAgent().handle_request(url, body)
if __name__ == '__main__': if __name__ == '__main__':

View File

@ -12,7 +12,6 @@ from starlette_context import context
from starlette_context.middleware import RawContextMiddleware from starlette_context.middleware import RawContextMiddleware
from pr_agent.agent.pr_agent import PRAgent from pr_agent.agent.pr_agent import PRAgent
from pr_agent.algo.utils import update_settings_from_args
from pr_agent.config_loader import get_settings, global_settings from pr_agent.config_loader import get_settings, global_settings
from pr_agent.git_providers import get_git_provider from pr_agent.git_providers import get_git_provider
from pr_agent.servers.utils import verify_signature from pr_agent.servers.utils import verify_signature
@ -98,7 +97,6 @@ async def handle_request(body: Dict[str, Any], event: str):
api_url = body["comment"]["pull_request_url"] api_url = body["comment"]["pull_request_url"]
else: else:
return {} return {}
logging.info(body)
logging.info(f"Handling comment because of event={event} and action={action}") logging.info(f"Handling comment because of event={event} and action={action}")
comment_id = body.get("comment", {}).get("id") comment_id = body.get("comment", {}).get("id")
provider = get_git_provider()(pr_url=api_url) provider = get_git_provider()(pr_url=api_url)
@ -125,14 +123,8 @@ async def handle_request(body: Dict[str, Any], event: str):
return {} return {}
logging.info(f"Performing review because of event={event} and action={action}") logging.info(f"Performing review because of event={event} and action={action}")
for command in get_settings().github_app.pr_commands: for command in get_settings().github_app.pr_commands:
split_command = command.split(" ") logging.info(f"Performing command: {command}")
command = split_command[0] await agent.handle_request(api_url, command)
args = split_command[1:]
other_args = update_settings_from_args(args)
new_command = ' '.join([command] + other_args)
logging.info(body)
logging.info(f"Performing command: {new_command}")
await agent.handle_request(api_url, new_command)
logging.info("event or action does not require handling") logging.info("event or action does not require handling")
return {} return {}

View File

@ -1,51 +1,21 @@
import copy
import json
import logging import logging
import sys
import uvicorn import uvicorn
from fastapi import APIRouter, FastAPI, Request, status from fastapi import APIRouter, FastAPI, Request, status
from fastapi.encoders import jsonable_encoder from fastapi.encoders import jsonable_encoder
from fastapi.responses import JSONResponse from fastapi.responses import JSONResponse
from starlette.background import BackgroundTasks from starlette.background import BackgroundTasks
from starlette.middleware import Middleware
from starlette_context import context
from starlette_context.middleware import RawContextMiddleware
from pr_agent.agent.pr_agent import PRAgent from pr_agent.agent.pr_agent import PRAgent
from pr_agent.config_loader import get_settings, global_settings from pr_agent.config_loader import get_settings
from pr_agent.secret_providers import get_secret_provider
logging.basicConfig(stream=sys.stdout, level=logging.INFO) app = FastAPI()
router = APIRouter() router = APIRouter()
secret_provider = get_secret_provider() if get_settings().get("CONFIG.SECRET_PROVIDER") else None
@router.post("/webhook") @router.post("/webhook")
async def gitlab_webhook(background_tasks: BackgroundTasks, request: Request): async def gitlab_webhook(background_tasks: BackgroundTasks, request: Request):
if request.headers.get("X-Gitlab-Token") and secret_provider:
request_token = request.headers.get("X-Gitlab-Token")
secret = secret_provider.get_secret(request_token)
try:
secret_dict = json.loads(secret)
gitlab_token = secret_dict["gitlab_token"]
context["settings"] = copy.deepcopy(global_settings)
context["settings"].gitlab.personal_access_token = gitlab_token
except Exception as e:
logging.error(f"Failed to validate secret {request_token}: {e}")
return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
elif get_settings().get("GITLAB.SHARED_SECRET"):
secret = get_settings().get("GITLAB.SHARED_SECRET")
if not request.headers.get("X-Gitlab-Token") == secret:
return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
else:
return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
gitlab_token = get_settings().get("GITLAB.PERSONAL_ACCESS_TOKEN", None)
if not gitlab_token:
return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
data = await request.json() data = await request.json()
logging.info(json.dumps(data))
if data.get('object_kind') == 'merge_request' and data['object_attributes'].get('action') in ['open', 'reopen']: if data.get('object_kind') == 'merge_request' and data['object_attributes'].get('action') in ['open', 'reopen']:
logging.info(f"A merge request has been opened: {data['object_attributes'].get('title')}") logging.info(f"A merge request has been opened: {data['object_attributes'].get('title')}")
url = data['object_attributes'].get('url') url = data['object_attributes'].get('url')
@ -58,18 +28,16 @@ async def gitlab_webhook(background_tasks: BackgroundTasks, request: Request):
background_tasks.add_task(PRAgent().handle_request, url, body) background_tasks.add_task(PRAgent().handle_request, url, body)
return JSONResponse(status_code=status.HTTP_200_OK, content=jsonable_encoder({"message": "success"})) return JSONResponse(status_code=status.HTTP_200_OK, content=jsonable_encoder({"message": "success"}))
@router.get("/")
async def root():
return {"status": "ok"}
def start(): def start():
gitlab_url = get_settings().get("GITLAB.URL", None) gitlab_url = get_settings().get("GITLAB.URL", None)
if not gitlab_url: if not gitlab_url:
raise ValueError("GITLAB.URL is not set") raise ValueError("GITLAB.URL is not set")
gitlab_token = get_settings().get("GITLAB.PERSONAL_ACCESS_TOKEN", None)
if not gitlab_token:
raise ValueError("GITLAB.PERSONAL_ACCESS_TOKEN is not set")
get_settings().config.git_provider = "gitlab" get_settings().config.git_provider = "gitlab"
middleware = [Middleware(RawContextMiddleware)]
app = FastAPI(middleware=middleware) app = FastAPI()
app.include_router(router) app.include_router(router)
uvicorn.run(app, host="0.0.0.0", port=3000) uvicorn.run(app, host="0.0.0.0", port=3000)

View File

@ -16,10 +16,6 @@ key = "" # Acquire through https://platform.openai.com
#deployment_id = "" # The deployment name you chose when you deployed the engine #deployment_id = "" # The deployment name you chose when you deployed the engine
#fallback_deployments = [] # For each fallback model specified in configuration.toml in the [config] section, specify the appropriate deployment_id #fallback_deployments = [] # For each fallback model specified in configuration.toml in the [config] section, specify the appropriate deployment_id
[pinecone]
api_key = "..."
environment = "gcp-starter"
[anthropic] [anthropic]
key = "" # Optional, uncomment if you want to use Anthropic. Acquire through https://www.anthropic.com/ key = "" # Optional, uncomment if you want to use Anthropic. Acquire through https://www.anthropic.com/
@ -28,14 +24,6 @@ key = "" # Optional, uncomment if you want to use Cohere. Acquire through https:
[replicate] [replicate]
key = "" # Optional, uncomment if you want to use Replicate. Acquire through https://replicate.com/ key = "" # Optional, uncomment if you want to use Replicate. Acquire through https://replicate.com/
[huggingface]
key = "" # Optional, uncomment if you want to use Huggingface Inference API. Acquire through https://huggingface.co/docs/api-inference/quicktour
api_base = "" # the base url for your huggingface inference endpoint
[ollama]
api_base = "" # the base url for your huggingface inference endpoint
[github] [github]
# ---- Set the following only for deployment type == "user" # ---- Set the following only for deployment type == "user"
user_token = "" # A GitHub personal access token with 'repo' scope. user_token = "" # A GitHub personal access token with 'repo' scope.
@ -55,12 +43,5 @@ webhook_secret = "<WEBHOOK SECRET>" # Optional, may be commented out.
personal_access_token = "" personal_access_token = ""
[bitbucket] [bitbucket]
# For Bitbucket personal/repository bearer token # Bitbucket personal bearer token
bearer_token = "" bearer_token = ""
# For Bitbucket app
app_key = ""
base_url = ""
[litellm]
LITELLM_TOKEN = "" # see https://docs.litellm.ai/docs/debugging/hosted_debugging for details and instructions on how to get a token

View File

@ -10,15 +10,14 @@ use_repo_settings_file=true
ai_timeout=180 ai_timeout=180
max_description_tokens = 500 max_description_tokens = 500
max_commits_tokens = 500 max_commits_tokens = 500
litellm_debugger=false
secret_provider="google_cloud_storage" secret_provider="google_cloud_storage"
cli_mode=false
[pr_reviewer] # /review # [pr_reviewer] # /review #
require_focused_review=false require_focused_review=false
require_score_review=false require_score_review=false
require_tests_review=true require_tests_review=true
require_security_review=true require_security_review=true
require_estimate_effort_to_review=true
num_code_suggestions=4 num_code_suggestions=4
inline_code_comments = false inline_code_comments = false
ask_and_reflect=false ask_and_reflect=false
@ -26,14 +25,10 @@ automatic_review=true
extra_instructions = "" extra_instructions = ""
[pr_description] # /describe # [pr_description] # /describe #
publish_labels=true
publish_description_as_comment=false publish_description_as_comment=false
add_original_user_description=false add_original_user_description=false
keep_original_user_title=false keep_original_user_title=false
extra_instructions = "" extra_instructions = ""
# markers
use_description_markers=false
include_generated_by_header=true
[pr_questions] # /ask # [pr_questions] # /ask #
@ -90,26 +85,3 @@ polling_interval_seconds = 30
# LocalGitProvider settings - uncomment to use paths other than default # LocalGitProvider settings - uncomment to use paths other than default
# description_path= "path/to/description.md" # description_path= "path/to/description.md"
# review_path= "path/to/review.md" # review_path= "path/to/review.md"
[gerrit]
# endpoint to the gerrit service
# url = "ssh://gerrit.example.com:29418"
# user for gerrit authentication
# user = "ai-reviewer"
# patch server where patches will be saved
# patch_server_endpoint = "http://127.0.0.1:5000/patch"
# token to authenticate in the patch server
# patch_server_token = ""
[litellm]
#use_client = false
[pr_similar_issue]
skip_comments = false
force_update_dataset = false
max_issues_to_scan = 500
[pinecone]
# fill and place in .secrets.toml
#api_key = ...
# environment = "gcp-starter"

View File

@ -68,17 +68,12 @@ Code suggestions:
type: string type: string
description: |- description: |-
a code snippet showing the relevant code lines from a '__new hunk__' section. a code snippet showing the relevant code lines from a '__new hunk__' section.
It must be contiguous, correctly formatted and indented, and without line numbers. It must be continuous, correctly formatted and indented, and without line numbers.
relevant lines start: relevant lines:
type: integer type: string
description: |- description: |-
The relevant line number from a '__new hunk__' section where the suggestion starts (inclusive). the relevant lines from a '__new hunk__' section, in the format of 'start_line-end_line'.
Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above. For example: '10-15'. They should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above.
relevant lines end:
type: integer
description: |-
The relevant line number from a '__new hunk__' section where the suggestion ends (inclusive).
Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above.
improved code: improved code:
type: string type: string
description: |- description: |-
@ -95,8 +90,7 @@ Code suggestions:
Add a docstring to func1() Add a docstring to func1()
existing code: |- existing code: |-
def func1(): def func1():
relevant lines start: 12 relevant lines: '12-12'
relevant lines end: 12
improved code: |- improved code: |-
... ...
``` ```

View File

@ -85,14 +85,6 @@ PR Analysis:
code diff changes are too scattered, then the PR is not focused. Explain code diff changes are too scattered, then the PR is not focused. Explain
your answer shortly. your answer shortly.
{%- endif %} {%- endif %}
{%- if require_estimate_effort_to_review %}
Estimated effort to review [1-5]:
type: string
description: >-
Estimate, on a scale of 1-5 (inclusive), the time and effort required to review this PR by an experienced and knowledgeable developer. 1 means short and easy review , 5 means long and hard review.
Take into account the size, complexity, quality, and the needed changes of the PR code diff.
Explain your answer shortly (1-2 sentences).
{%- endif %}
PR Feedback: PR Feedback:
General suggestions: General suggestions:
type: string type: string

View File

@ -42,13 +42,10 @@ class PRCodeSuggestions:
"extra_instructions": get_settings().pr_code_suggestions.extra_instructions, "extra_instructions": get_settings().pr_code_suggestions.extra_instructions,
"commit_messages_str": self.git_provider.get_commit_messages(), "commit_messages_str": self.git_provider.get_commit_messages(),
} }
self.token_handler = TokenHandler(self.git_provider.pr, self.token_handler = TokenHandler(self.vars, get_settings().pr_code_suggestions_prompt.system,
self.vars,
get_settings().pr_code_suggestions_prompt.system,
get_settings().pr_code_suggestions_prompt.user) get_settings().pr_code_suggestions_prompt.user)
async def run(self): async def run(self):
try:
logging.info('Generating code suggestions for PR...') logging.info('Generating code suggestions for PR...')
if get_settings().config.publish_output: if get_settings().config.publish_output:
self.git_provider.publish_comment("Preparing review...", is_temporary=True) self.git_provider.publish_comment("Preparing review...", is_temporary=True)
@ -59,9 +56,6 @@ class PRCodeSuggestions:
data = self._prepare_pr_code_suggestions() data = self._prepare_pr_code_suggestions()
else: else:
data = await retry_with_fallback_models(self._prepare_prediction_extended) data = await retry_with_fallback_models(self._prepare_prediction_extended)
if (not data) or (not 'Code suggestions' in data):
logging.info('No code suggestions found for PR.')
return
if (not self.is_extended and get_settings().pr_code_suggestions.rank_suggestions) or \ if (not self.is_extended and get_settings().pr_code_suggestions.rank_suggestions) or \
(self.is_extended and get_settings().pr_code_suggestions.rank_extended_suggestions): (self.is_extended and get_settings().pr_code_suggestions.rank_extended_suggestions):
@ -73,8 +67,6 @@ class PRCodeSuggestions:
self.git_provider.remove_initial_comment() self.git_provider.remove_initial_comment()
logging.info('Pushing inline code suggestions...') logging.info('Pushing inline code suggestions...')
self.push_inline_code_suggestions(data) self.push_inline_code_suggestions(data)
except Exception as e:
logging.error(f"Failed to generate code suggestions for PR, error: {e}")
async def _prepare_prediction(self, model: str): async def _prepare_prediction(self, model: str):
logging.info('Getting PR diff...') logging.info('Getting PR diff...')
@ -119,8 +111,11 @@ class PRCodeSuggestions:
if get_settings().config.verbosity_level >= 2: if get_settings().config.verbosity_level >= 2:
logging.info(f"suggestion: {d}") logging.info(f"suggestion: {d}")
relevant_file = d['relevant file'].strip() relevant_file = d['relevant file'].strip()
relevant_lines_start = int(d['relevant lines start']) # absolute position relevant_lines_str = d['relevant lines'].strip()
relevant_lines_end = int(d['relevant lines end']) if ',' in relevant_lines_str: # handling 'relevant lines': '181, 190' or '178-184, 188-194'
relevant_lines_str = relevant_lines_str.split(',')[0]
relevant_lines_start = int(relevant_lines_str.split('-')[0]) # absolute position
relevant_lines_end = int(relevant_lines_str.split('-')[-1])
content = d['suggestion content'] content = d['suggestion content']
new_code_snippet = d['improved code'] new_code_snippet = d['improved code']

View File

@ -1,6 +1,5 @@
import copy import copy
import json import json
import re
import logging import logging
from typing import List, Tuple from typing import List, Tuple
@ -29,7 +28,6 @@ class PRDescription:
self.main_pr_language = get_main_pr_language( self.main_pr_language = get_main_pr_language(
self.git_provider.get_languages(), self.git_provider.get_files() self.git_provider.get_languages(), self.git_provider.get_files()
) )
self.pr_id = f"{self.git_provider.repo}/{self.git_provider.pr_num}"
# Initialize the AI handler # Initialize the AI handler
self.ai_handler = AiHandler() self.ai_handler = AiHandler()
@ -38,7 +36,7 @@ class PRDescription:
self.vars = { self.vars = {
"title": self.git_provider.pr.title, "title": self.git_provider.pr.title,
"branch": self.git_provider.get_pr_branch(), "branch": self.git_provider.get_pr_branch(),
"description": self.git_provider.get_pr_description(full=False), "description": self.git_provider.get_pr_description(),
"language": self.main_pr_language, "language": self.main_pr_language,
"diff": "", # empty diff for initial calculation "diff": "", # empty diff for initial calculation
"extra_instructions": get_settings().pr_description.extra_instructions, "extra_instructions": get_settings().pr_description.extra_instructions,
@ -48,12 +46,8 @@ class PRDescription:
self.user_description = self.git_provider.get_user_description() self.user_description = self.git_provider.get_user_description()
# Initialize the token handler # Initialize the token handler
self.token_handler = TokenHandler( self.token_handler = TokenHandler(self.vars, get_settings().pr_description_prompt.system,
self.git_provider.pr, get_settings().pr_description_prompt.user)
self.vars,
get_settings().pr_description_prompt.system,
get_settings().pr_description_prompt.user,
)
# Initialize patches_diff and prediction attributes # Initialize patches_diff and prediction attributes
self.patches_diff = None self.patches_diff = None
@ -63,44 +57,27 @@ class PRDescription:
""" """
Generates a PR description using an AI model and publishes it to the PR. Generates a PR description using an AI model and publishes it to the PR.
""" """
logging.info('Generating a PR description...')
try:
logging.info(f"Generating a PR description {self.pr_id}")
if get_settings().config.publish_output: if get_settings().config.publish_output:
self.git_provider.publish_comment("Preparing pr description...", is_temporary=True) self.git_provider.publish_comment("Preparing pr description...", is_temporary=True)
await retry_with_fallback_models(self._prepare_prediction) await retry_with_fallback_models(self._prepare_prediction)
logging.info(f"Preparing answer {self.pr_id}") logging.info('Preparing answer...')
if self.prediction: pr_title, pr_body, pr_types, markdown_text = self._prepare_pr_answer()
self._prepare_data()
else:
return None
pr_labels = []
if get_settings().pr_description.publish_labels:
pr_labels = self._prepare_labels()
if get_settings().pr_description.use_description_markers:
pr_title, pr_body = self._prepare_pr_answer_with_markers()
else:
pr_title, pr_body, = self._prepare_pr_answer()
full_markdown_description = f"## Title\n\n{pr_title}\n\n___\n{pr_body}"
if get_settings().config.publish_output: if get_settings().config.publish_output:
logging.info(f"Pushing answer {self.pr_id}") logging.info('Pushing answer...')
if get_settings().pr_description.publish_description_as_comment: if get_settings().pr_description.publish_description_as_comment:
self.git_provider.publish_comment(full_markdown_description) self.git_provider.publish_comment(markdown_text)
else: else:
self.git_provider.publish_description(pr_title, pr_body) self.git_provider.publish_description(pr_title, pr_body)
if get_settings().pr_description.publish_labels and self.git_provider.is_supported("get_labels"): if self.git_provider.is_supported("get_labels"):
current_labels = self.git_provider.get_labels() current_labels = self.git_provider.get_labels()
if current_labels is None: if current_labels is None:
current_labels = [] current_labels = []
self.git_provider.publish_labels(pr_labels + current_labels) self.git_provider.publish_labels(pr_types + current_labels)
self.git_provider.remove_initial_comment() self.git_provider.remove_initial_comment()
except Exception as e:
logging.error(f"Error generating PR description {self.pr_id}: {e}")
return "" return ""
@ -118,12 +95,9 @@ class PRDescription:
Any exceptions raised by the 'get_pr_diff' and '_get_prediction' functions. Any exceptions raised by the 'get_pr_diff' and '_get_prediction' functions.
""" """
if get_settings().pr_description.use_description_markers and 'pr_agent:' not in self.user_description: logging.info('Getting PR diff...')
return None
logging.info(f"Getting PR diff {self.pr_id}")
self.patches_diff = get_pr_diff(self.git_provider, self.token_handler, model) self.patches_diff = get_pr_diff(self.git_provider, self.token_handler, model)
logging.info(f"Getting AI prediction {self.pr_id}") logging.info('Getting AI prediction...')
self.prediction = await self._get_prediction(model) self.prediction = await self._get_prediction(model)
async def _get_prediction(self, model: str) -> str: async def _get_prediction(self, model: str) -> str:
@ -156,71 +130,34 @@ class PRDescription:
return response return response
def _prepare_pr_answer(self) -> Tuple[str, str, List[str], str]:
def _prepare_data(self):
# Load the AI prediction data into a dictionary
self.data = load_yaml(self.prediction.strip())
if get_settings().pr_description.add_original_user_description and self.user_description:
self.data["User Description"] = self.user_description
def _prepare_labels(self) -> List[str]:
pr_types = []
# If the 'PR Type' key is present in the dictionary, split its value by comma and assign it to 'pr_types'
if 'PR Type' in self.data:
if type(self.data['PR Type']) == list:
pr_types = self.data['PR Type']
elif type(self.data['PR Type']) == str:
pr_types = self.data['PR Type'].split(',')
return pr_types
def _prepare_pr_answer_with_markers(self) -> Tuple[str, str]:
logging.info(f"Using description marker replacements {self.pr_id}")
title = self.vars["title"]
body = self.user_description
if get_settings().pr_description.include_generated_by_header:
ai_header = f"### 🤖 Generated by PR Agent at {self.git_provider.last_commit_id.sha}\n\n"
else:
ai_header = ""
ai_summary = self.data.get('PR Description')
if ai_summary and not re.search(r'<!--\s*pr_agent:summary\s*-->', body):
summary = f"{ai_header}{ai_summary}"
body = body.replace('pr_agent:summary', summary)
if not re.search(r'<!--\s*pr_agent:walkthrough\s*-->', body):
ai_walkthrough = self.data.get('PR Main Files Walkthrough')
if ai_walkthrough:
walkthrough = str(ai_header)
for file in ai_walkthrough:
filename = file['filename'].replace("'", "`")
description = file['changes in file'].replace("'", "`")
walkthrough += f'- `{filename}`: {description}\n'
body = body.replace('pr_agent:walkthrough', walkthrough)
return title, body
def _prepare_pr_answer(self) -> Tuple[str, str]:
""" """
Prepare the PR description based on the AI prediction data. Prepare the PR description based on the AI prediction data.
Returns: Returns:
- title: a string containing the PR title. - title: a string containing the PR title.
- pr_body: a string containing the PR description body in a markdown format. - pr_body: a string containing the PR body in a markdown format.
- pr_types: a list of strings containing the PR types.
- markdown_text: a string containing the AI prediction data in a markdown format. used for publishing a comment
""" """
# Load the AI prediction data into a dictionary
data = load_yaml(self.prediction.strip())
# Iterate over the dictionary items and append the key and value to 'markdown_text' in a markdown format if get_settings().pr_description.add_original_user_description and self.user_description:
markdown_text = "" data["User Description"] = self.user_description
for key, value in self.data.items():
markdown_text += f"## {key}\n\n" # Initialization
markdown_text += f"{value}\n\n" pr_types = []
# If the 'PR Type' key is present in the dictionary, split its value by comma and assign it to 'pr_types'
if 'PR Type' in data:
if type(data['PR Type']) == list:
pr_types = data['PR Type']
elif type(data['PR Type']) == str:
pr_types = data['PR Type'].split(',')
# Remove the 'PR Title' key from the dictionary # Remove the 'PR Title' key from the dictionary
ai_title = self.data.pop('PR Title', self.vars["title"]) ai_title = data.pop('PR Title')
if get_settings().pr_description.keep_original_user_title: if get_settings().pr_description.keep_original_user_title:
# Assign the original PR title to the 'title' variable # Assign the original PR title to the 'title' variable
title = self.vars["title"] title = self.vars["title"]
@ -231,27 +168,25 @@ class PRDescription:
# Iterate over the remaining dictionary items and append the key and value to 'pr_body' in a markdown format, # Iterate over the remaining dictionary items and append the key and value to 'pr_body' in a markdown format,
# except for the items containing the word 'walkthrough' # except for the items containing the word 'walkthrough'
pr_body = "" pr_body = ""
for idx, (key, value) in enumerate(self.data.items()): for idx, (key, value) in enumerate(data.items()):
pr_body += f"## {key}:\n" pr_body += f"## {key}:\n"
if 'walkthrough' in key.lower(): if 'walkthrough' in key.lower():
# for filename, description in value.items(): # for filename, description in value.items():
if self.git_provider.is_supported("gfm_markdown"):
pr_body += "<details> <summary>files:</summary>\n\n"
for file in value: for file in value:
filename = file['filename'].replace("'", "`") filename = file['filename'].replace("'", "`")
description = file['changes in file'] description = file['changes in file']
pr_body += f'`{filename}`: {description}\n' pr_body += f'`{filename}`: {description}\n'
if self.git_provider.is_supported("gfm_markdown"):
pr_body +="</details>\n"
else: else:
# if the value is a list, join its items by comma # if the value is a list, join its items by comma
if type(value) == list: if type(value) == list:
value = ', '.join(v for v in value) value = ', '.join(v for v in value)
pr_body += f"{value}\n" pr_body += f"{value}\n"
if idx < len(self.data) - 1: if idx < len(data) - 1:
pr_body += "\n___\n" pr_body += "\n___\n"
markdown_text = f"## Title\n\n{title}\n\n___\n{pr_body}"
if get_settings().config.verbosity_level >= 2: if get_settings().config.verbosity_level >= 2:
logging.info(f"title:\n{title}\n{pr_body}") logging.info(f"title:\n{title}\n{pr_body}")
return title, pr_body return title, pr_body, pr_types, markdown_text

View File

@ -26,9 +26,7 @@ class PRInformationFromUser:
"diff": "", # empty diff for initial calculation "diff": "", # empty diff for initial calculation
"commit_messages_str": self.git_provider.get_commit_messages(), "commit_messages_str": self.git_provider.get_commit_messages(),
} }
self.token_handler = TokenHandler(self.git_provider.pr, self.token_handler = TokenHandler(self.vars, get_settings().pr_information_from_user_prompt.system,
self.vars,
get_settings().pr_information_from_user_prompt.system,
get_settings().pr_information_from_user_prompt.user) get_settings().pr_information_from_user_prompt.user)
self.patches_diff = None self.patches_diff = None
self.prediction = None self.prediction = None

View File

@ -29,9 +29,7 @@ class PRQuestions:
"questions": self.question_str, "questions": self.question_str,
"commit_messages_str": self.git_provider.get_commit_messages(), "commit_messages_str": self.git_provider.get_commit_messages(),
} }
self.token_handler = TokenHandler(self.git_provider.pr, self.token_handler = TokenHandler(self.vars, get_settings().pr_questions_prompt.system,
self.vars,
get_settings().pr_questions_prompt.system,
get_settings().pr_questions_prompt.user) get_settings().pr_questions_prompt.user)
self.patches_diff = None self.patches_diff = None
self.prediction = None self.prediction = None

View File

@ -9,8 +9,7 @@ from jinja2 import Environment, StrictUndefined
from yaml import SafeLoader from yaml import SafeLoader
from pr_agent.algo.ai_handler import AiHandler from pr_agent.algo.ai_handler import AiHandler
from pr_agent.algo.pr_processing import get_pr_diff, retry_with_fallback_models, \ from pr_agent.algo.pr_processing import get_pr_diff, retry_with_fallback_models
find_line_number_of_relevant_line_in_file, clip_tokens
from pr_agent.algo.token_handler import TokenHandler from pr_agent.algo.token_handler import TokenHandler
from pr_agent.algo.utils import convert_to_markdown, try_fix_json, try_fix_yaml, load_yaml from pr_agent.algo.utils import convert_to_markdown, try_fix_json, try_fix_yaml, load_yaml
from pr_agent.config_loader import get_settings from pr_agent.config_loader import get_settings
@ -59,7 +58,6 @@ class PRReviewer:
"require_tests": get_settings().pr_reviewer.require_tests_review, "require_tests": get_settings().pr_reviewer.require_tests_review,
"require_security": get_settings().pr_reviewer.require_security_review, "require_security": get_settings().pr_reviewer.require_security_review,
"require_focused": get_settings().pr_reviewer.require_focused_review, "require_focused": get_settings().pr_reviewer.require_focused_review,
"require_estimate_effort_to_review": get_settings().pr_reviewer.require_estimate_effort_to_review,
'num_code_suggestions': get_settings().pr_reviewer.num_code_suggestions, 'num_code_suggestions': get_settings().pr_reviewer.num_code_suggestions,
'question_str': question_str, 'question_str': question_str,
'answer_str': answer_str, 'answer_str': answer_str,
@ -67,12 +65,8 @@ class PRReviewer:
"commit_messages_str": self.git_provider.get_commit_messages(), "commit_messages_str": self.git_provider.get_commit_messages(),
} }
self.token_handler = TokenHandler( self.token_handler = TokenHandler(self.vars, get_settings().pr_review_prompt.system,
self.git_provider.pr, get_settings().pr_review_prompt.user)
self.vars,
get_settings().pr_review_prompt.system,
get_settings().pr_review_prompt.user
)
def parse_args(self, args: List[str]) -> None: def parse_args(self, args: List[str]) -> None:
""" """
@ -95,8 +89,6 @@ class PRReviewer:
""" """
Review the pull request and generate feedback. Review the pull request and generate feedback.
""" """
try:
if self.is_auto and not get_settings().pr_reviewer.automatic_review: if self.is_auto and not get_settings().pr_reviewer.automatic_review:
logging.info(f'Automatic review is disabled {self.pr_url}') logging.info(f'Automatic review is disabled {self.pr_url}')
return None return None
@ -119,8 +111,6 @@ class PRReviewer:
if get_settings().pr_reviewer.inline_code_comments: if get_settings().pr_reviewer.inline_code_comments:
logging.info('Pushing inline code comments...') logging.info('Pushing inline code comments...')
self._publish_inline_code_comments() self._publish_inline_code_comments()
except Exception as e:
logging.error(f"Failed to review PR: {e}")
async def _prepare_prediction(self, model: str) -> None: async def _prepare_prediction(self, model: str) -> None:
""" """
@ -219,11 +209,11 @@ class PRReviewer:
"⏮️ Review for commits since previous PR-Agent review": f"Starting from commit {last_commit_url}"}}) "⏮️ Review for commits since previous PR-Agent review": f"Starting from commit {last_commit_url}"}})
data.move_to_end('Incremental PR Review', last=False) data.move_to_end('Incremental PR Review', last=False)
markdown_text = convert_to_markdown(data, self.git_provider.is_supported("gfm_markdown")) markdown_text = convert_to_markdown(data)
user = self.git_provider.get_user_id() user = self.git_provider.get_user_id()
# Add help text if not in CLI mode # Add help text if not in CLI§ mode
if not get_settings().get("CONFIG.CLI_MODE", False): if not get_settings().get("CONFIG.CLI§_MODE", False):
markdown_text += "\n### How to use\n" markdown_text += "\n### How to use\n"
if user and '[bot]' not in user: if user and '[bot]' not in user:
markdown_text += bot_help_text(user) markdown_text += bot_help_text(user)

View File

@ -1,276 +0,0 @@
import copy
import json
import logging
from enum import Enum
from typing import List, Tuple
import pinecone
import openai
import pandas as pd
from pydantic import BaseModel, Field
from pr_agent.algo import MAX_TOKENS
from pr_agent.algo.token_handler import TokenHandler
from pr_agent.config_loader import get_settings
from pr_agent.git_providers import get_git_provider
from pinecone_datasets import Dataset, DatasetMetadata
MODEL = "text-embedding-ada-002"
class PRSimilarIssue:
def __init__(self, issue_url: str, args: list = None):
if get_settings().config.git_provider != "github":
raise Exception("Only github is supported for similar issue tool")
self.cli_mode = get_settings().CONFIG.CLI_MODE
self.max_issues_to_scan = get_settings().pr_similar_issue.max_issues_to_scan
self.issue_url = issue_url
self.git_provider = get_git_provider()()
repo_name, issue_number = self.git_provider._parse_issue_url(issue_url.split('=')[-1])
self.git_provider.repo = repo_name
self.git_provider.repo_obj = self.git_provider.github_client.get_repo(repo_name)
self.token_handler = TokenHandler()
repo_obj = self.git_provider.repo_obj
repo_name_for_index = self.repo_name_for_index = repo_obj.full_name.lower().replace('/', '-').replace('_/', '-')
index_name = self.index_name = "codium-ai-pr-agent-issues"
# assuming pinecone api key and environment are set in secrets file
try:
api_key = get_settings().pinecone.api_key
environment = get_settings().pinecone.environment
except Exception:
if not self.cli_mode:
repo_name, original_issue_number = self.git_provider._parse_issue_url(self.issue_url.split('=')[-1])
issue_main = self.git_provider.repo_obj.get_issue(original_issue_number)
issue_main.create_comment("Please set pinecone api key and environment in secrets file")
raise Exception("Please set pinecone api key and environment in secrets file")
# check if index exists, and if repo is already indexed
run_from_scratch = False
upsert = True
pinecone.init(api_key=api_key, environment=environment)
if not index_name in pinecone.list_indexes():
run_from_scratch = True
upsert = False
else:
if get_settings().pr_similar_issue.force_update_dataset:
upsert = True
else:
pinecone_index = pinecone.Index(index_name=index_name)
res = pinecone_index.fetch([f"example_issue_{repo_name_for_index}"]).to_dict()
if res["vectors"]:
upsert = False
if run_from_scratch or upsert: # index the entire repo
logging.info('Indexing the entire repo...')
logging.info('Getting issues...')
issues = list(repo_obj.get_issues(state='all'))
logging.info('Done')
self._update_index_with_issues(issues, repo_name_for_index, upsert=upsert)
else: # update index if needed
pinecone_index = pinecone.Index(index_name=index_name)
issues_to_update = []
issues_paginated_list = repo_obj.get_issues(state='all')
counter = 1
for issue in issues_paginated_list:
if issue.pull_request:
continue
issue_str, comments, number = self._process_issue(issue)
issue_key = f"issue_{number}"
id = issue_key + "." + "issue"
res = pinecone_index.fetch([id]).to_dict()
is_new_issue = True
for vector in res["vectors"].values():
if vector['metadata']['repo'] == repo_name_for_index:
is_new_issue = False
break
if is_new_issue:
counter += 1
issues_to_update.append(issue)
else:
break
if issues_to_update:
logging.info(f'Updating index with {counter} new issues...')
self._update_index_with_issues(issues_to_update, repo_name_for_index, upsert=True)
else:
logging.info('No new issues to update')
async def run(self):
logging.info('Getting issue...')
repo_name, original_issue_number = self.git_provider._parse_issue_url(self.issue_url.split('=')[-1])
issue_main = self.git_provider.repo_obj.get_issue(original_issue_number)
issue_str, comments, number = self._process_issue(issue_main)
openai.api_key = get_settings().openai.key
logging.info('Done')
logging.info('Querying...')
res = openai.Embedding.create(input=[issue_str], engine=MODEL)
embeds = [record['embedding'] for record in res['data']]
pinecone_index = pinecone.Index(index_name=self.index_name)
res = pinecone_index.query(embeds[0],
top_k=5,
filter={"repo": self.repo_name_for_index},
include_metadata=True).to_dict()
relevant_issues_number_list = []
relevant_comment_number_list = []
score_list = []
for r in res['matches']:
issue_number = int(r["id"].split('.')[0].split('_')[-1])
if original_issue_number == issue_number:
continue
if issue_number not in relevant_issues_number_list:
relevant_issues_number_list.append(issue_number)
if 'comment' in r["id"]:
relevant_comment_number_list.append(int(r["id"].split('.')[1].split('_')[-1]))
else:
relevant_comment_number_list.append(-1)
score_list.append(str("{:.2f}".format(r['score'])))
logging.info('Done')
logging.info('Publishing response...')
similar_issues_str = "### Similar Issues\n___\n\n"
for i, issue_number_similar in enumerate(relevant_issues_number_list):
issue = self.git_provider.repo_obj.get_issue(issue_number_similar)
title = issue.title
url = issue.html_url
if relevant_comment_number_list[i] != -1:
url = list(issue.get_comments())[relevant_comment_number_list[i]].html_url
similar_issues_str += f"{i + 1}. **[{title}]({url})** (score={score_list[i]})\n\n"
if get_settings().config.publish_output:
response = issue_main.create_comment(similar_issues_str)
logging.info(similar_issues_str)
logging.info('Done')
def _process_issue(self, issue):
header = issue.title
body = issue.body
number = issue.number
if get_settings().pr_similar_issue.skip_comments:
comments = []
else:
comments = list(issue.get_comments())
issue_str = f"Issue Header: \"{header}\"\n\nIssue Body:\n{body}"
return issue_str, comments, number
def _update_index_with_issues(self, issues_list, repo_name_for_index, upsert=False):
logging.info('Processing issues...')
corpus = Corpus()
example_issue_record = Record(
id=f"example_issue_{repo_name_for_index}",
text="example_issue",
metadata=Metadata(repo=repo_name_for_index)
)
corpus.append(example_issue_record)
counter = 0
for issue in issues_list:
if issue.pull_request:
continue
counter += 1
if counter % 100 == 0:
logging.info(f"Scanned {counter} issues")
if counter >= self.max_issues_to_scan:
logging.info(f"Scanned {self.max_issues_to_scan} issues, stopping")
break
issue_str, comments, number = self._process_issue(issue)
issue_key = f"issue_{number}"
username = issue.user.login
created_at = str(issue.created_at)
if len(issue_str) < 8000 or \
self.token_handler.count_tokens(issue_str) < MAX_TOKENS[MODEL]: # fast reject first
issue_record = Record(
id=issue_key + "." + "issue",
text=issue_str,
metadata=Metadata(repo=repo_name_for_index,
username=username,
created_at=created_at,
level=IssueLevel.ISSUE)
)
corpus.append(issue_record)
if comments:
for j, comment in enumerate(comments):
comment_body = comment.body
num_words_comment = len(comment_body.split())
if num_words_comment < 10 or not isinstance(comment_body, str):
continue
if len(comment_body) < 8000 or \
self.token_handler.count_tokens(comment_body) < MAX_TOKENS[MODEL]:
comment_record = Record(
id=issue_key + ".comment_" + str(j + 1),
text=comment_body,
metadata=Metadata(repo=repo_name_for_index,
username=username, # use issue username for all comments
created_at=created_at,
level=IssueLevel.COMMENT)
)
corpus.append(comment_record)
df = pd.DataFrame(corpus.dict()["documents"])
logging.info('Done')
logging.info('Embedding...')
openai.api_key = get_settings().openai.key
list_to_encode = list(df["text"].values)
try:
res = openai.Embedding.create(input=list_to_encode, engine=MODEL)
embeds = [record['embedding'] for record in res['data']]
except:
embeds = []
logging.error('Failed to embed entire list, embedding one by one...')
for i, text in enumerate(list_to_encode):
try:
res = openai.Embedding.create(input=[text], engine=MODEL)
embeds.append(res['data'][0]['embedding'])
except:
embeds.append([0] * 1536)
df["values"] = embeds
meta = DatasetMetadata.empty()
meta.dense_model.dimension = len(embeds[0])
ds = Dataset.from_pandas(df, meta)
logging.info('Done')
api_key = get_settings().pinecone.api_key
environment = get_settings().pinecone.environment
if not upsert:
logging.info('Creating index from scratch...')
ds.to_pinecone_index(self.index_name, api_key=api_key, environment=environment)
else:
logging.info('Upserting index...')
namespace = ""
batch_size: int = 100
concurrency: int = 10
pinecone.init(api_key=api_key, environment=environment)
ds._upsert_to_index(self.index_name, namespace, batch_size, concurrency)
logging.info('Done')
class IssueLevel(str, Enum):
ISSUE = "issue"
COMMENT = "comment"
class Metadata(BaseModel):
repo: str
username: str = Field(default="@codium")
created_at: str = Field(default="01-01-1970 00:00:00.00000")
level: IssueLevel = Field(default=IssueLevel.ISSUE)
class Config:
use_enum_values = True
class Record(BaseModel):
id: str
text: str
metadata: Metadata
class Corpus(BaseModel):
documents: List[Record] = Field(default=[])
def append(self, r: Record):
self.documents.append(r)

View File

@ -40,13 +40,11 @@ class PRUpdateChangelog:
"extra_instructions": get_settings().pr_update_changelog.extra_instructions, "extra_instructions": get_settings().pr_update_changelog.extra_instructions,
"commit_messages_str": self.git_provider.get_commit_messages(), "commit_messages_str": self.git_provider.get_commit_messages(),
} }
self.token_handler = TokenHandler(self.git_provider.pr, self.token_handler = TokenHandler(self.vars, get_settings().pr_update_changelog_prompt.system,
self.vars,
get_settings().pr_update_changelog_prompt.system,
get_settings().pr_update_changelog_prompt.user) get_settings().pr_update_changelog_prompt.user)
async def run(self): async def run(self):
# assert type(self.git_provider) == GithubProvider, "Currently only Github is supported" assert type(self.git_provider) == GithubProvider, "Currently only Github is supported"
logging.info('Updating the changelog...') logging.info('Updating the changelog...')
if get_settings().config.publish_output: if get_settings().config.publish_output:

View File

@ -35,12 +35,12 @@ dependencies = {file = ["requirements.txt"]}
"Homepage" = "https://github.com/Codium-ai/pr-agent" "Homepage" = "https://github.com/Codium-ai/pr-agent"
[tool.setuptools] [tool.setuptools]
include-package-data = false include-package-data = true
license-files = ["LICENSE"] license-files = ["LICENSE"]
[tool.setuptools.packages.find] [tool.setuptools.packages.find]
where = ["."] where = ["."]
include = ["pr_agent"] include = ["pr_agent", "pr_agent.*"]
[project.scripts] [project.scripts]
pr-agent = "pr_agent.cli:run" pr-agent = "pr_agent.cli:run"

View File

@ -1,23 +1,19 @@
dynaconf==3.1.12 dynaconf~=3.1.12
fastapi==0.99.0 fastapi~=0.103.0
PyGithub==1.59.* PyGithub~=1.59.0
retry==0.9.2 retry~=0.9.2
openai==0.27.8 openai~=0.27.8
Jinja2==3.1.2 Jinja2~=3.1.2
tiktoken==0.4.0 tiktoken~=0.4.0
uvicorn==0.22.0 uvicorn~=0.22.0
python-gitlab==3.15.0 python-gitlab~=3.15.0
pytest==7.4.0 pytest~=7.4.0
aiohttp==3.8.4 aiohttp~=3.8.4
atlassian-python-api==3.39.0 atlassian-python-api~=3.39.0
GitPython==3.1.32 GitPython~=3.1.32
PyYAML==6.0 PyYAML~=6.0
starlette-context==0.3.6 starlette-context~=0.3.6
litellm~=0.1.574 litellm~=0.1.445
boto3==1.28.25 boto3~=1.28.25
google-cloud-storage==2.10.0 google-cloud-storage~=2.10.0
ujson==5.8.0 ujson~=5.8.0
azure-devops==7.1.0b3
msrest==0.7.1
pinecone-client
pinecone-datasets @ git+https://github.com/mrT23/pinecone-datasets.git@main

View File

@ -125,7 +125,7 @@ class TestCodeCommitProvider:
} }
} }
pr = api.get_pr("my_test_repo", 321) pr = api.get_pr(321)
assert pr.title == "My PR" assert pr.title == "My PR"
assert pr.description == "My PR description" assert pr.description == "My PR description"

View File

@ -1,7 +1,7 @@
import pytest import pytest
from pr_agent.git_providers.codecommit_provider import CodeCommitFile from pr_agent.git_providers.codecommit_provider import CodeCommitFile
from pr_agent.git_providers.codecommit_provider import CodeCommitProvider from pr_agent.git_providers.codecommit_provider import CodeCommitProvider
from pr_agent.git_providers.git_provider import EDIT_TYPE from pr_agent.algo.utils import EDIT_TYPE
class TestCodeCommitFile: class TestCodeCommitFile:
@ -26,48 +26,11 @@ class TestCodeCommitFile:
class TestCodeCommitProvider: class TestCodeCommitProvider:
def test_parse_pr_url(self): def test_parse_pr_url(self):
# Test that the _parse_pr_url() function can extract the repo name and PR number from a CodeCommit URL
url = "https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/my_test_repo/pull-requests/321" url = "https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/my_test_repo/pull-requests/321"
repo_name, pr_number = CodeCommitProvider._parse_pr_url(url) repo_name, pr_number = CodeCommitProvider._parse_pr_url(url)
assert repo_name == "my_test_repo" assert repo_name == "my_test_repo"
assert pr_number == 321 assert pr_number == 321
def test_is_valid_codecommit_hostname(self):
# Test the various AWS regions
assert CodeCommitProvider._is_valid_codecommit_hostname("af-south-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-east-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-3.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-south-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-south-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-3.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-4.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ca-central-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-central-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-central-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-north-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-south-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-south-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-3.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("il-central-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("me-central-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("me-south-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("sa-east-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-east-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-east-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-gov-east-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-gov-west-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-west-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-west-2.console.aws.amazon.com")
# Test non-AWS regions
assert not CodeCommitProvider._is_valid_codecommit_hostname("no-such-region.console.aws.amazon.com")
assert not CodeCommitProvider._is_valid_codecommit_hostname("console.aws.amazon.com")
# Test that an error is raised when an invalid CodeCommit URL is provided to the set_pr() method of the CodeCommitProvider class. # Test that an error is raised when an invalid CodeCommit URL is provided to the set_pr() method of the CodeCommitProvider class.
# Generated by CodiumAI # Generated by CodiumAI
def test_invalid_codecommit_url(self): def test_invalid_codecommit_url(self):
@ -143,7 +106,6 @@ class TestCodeCommitProvider:
assert percentages == {} assert percentages == {}
def test_get_edit_type(self): def test_get_edit_type(self):
# Test that the _get_edit_type() function can convert a CodeCommit letter to an EDIT_TYPE enum
assert CodeCommitProvider._get_edit_type("A") == EDIT_TYPE.ADDED assert CodeCommitProvider._get_edit_type("A") == EDIT_TYPE.ADDED
assert CodeCommitProvider._get_edit_type("D") == EDIT_TYPE.DELETED assert CodeCommitProvider._get_edit_type("D") == EDIT_TYPE.DELETED
assert CodeCommitProvider._get_edit_type("M") == EDIT_TYPE.MODIFIED assert CodeCommitProvider._get_edit_type("M") == EDIT_TYPE.MODIFIED
@ -155,18 +117,3 @@ class TestCodeCommitProvider:
assert CodeCommitProvider._get_edit_type("r") == EDIT_TYPE.RENAMED assert CodeCommitProvider._get_edit_type("r") == EDIT_TYPE.RENAMED
assert CodeCommitProvider._get_edit_type("X") is None assert CodeCommitProvider._get_edit_type("X") is None
def test_add_additional_newlines(self):
# a short string to test adding double newlines
input = "abc\ndef\n\n___\nghi\njkl\nmno\n\npqr\n"
expect = "abc\n\ndef\n\n___\n\nghi\n\njkl\n\nmno\n\npqr\n\n"
assert CodeCommitProvider._add_additional_newlines(input) == expect
# a test example from a real PR
input = "## PR Type:\nEnhancement\n\n___\n## PR Description:\nThis PR introduces a new feature to the script, allowing users to filter servers by name.\n\n___\n## PR Main Files Walkthrough:\n`foo`: The foo script has been updated to include a new command line option `-f` or `--filter`.\n`bar`: The bar script has been updated to list stopped servers.\n"
expect = "## PR Type:\n\nEnhancement\n\n___\n\n## PR Description:\n\nThis PR introduces a new feature to the script, allowing users to filter servers by name.\n\n___\n\n## PR Main Files Walkthrough:\n\n`foo`: The foo script has been updated to include a new command line option `-f` or `--filter`.\n\n`bar`: The bar script has been updated to list stopped servers.\n\n"
assert CodeCommitProvider._add_additional_newlines(input) == expect
def test_remove_markdown_html(self):
input = "## PR Feedback\n<details><summary>Code feedback:</summary>\nfile foo\n</summary>\n"
expect = "## PR Feedback\nCode feedback:\nfile foo\n\n"
assert CodeCommitProvider._remove_markdown_html(input) == expect

View File

@ -1,8 +1,6 @@
# Generated by CodiumAI # Generated by CodiumAI
from pr_agent.git_providers.git_provider import FilePatchInfo from pr_agent.algo.utils import FilePatchInfo, find_line_number_of_relevant_line_in_file
from pr_agent.algo.pr_processing import find_line_number_of_relevant_line_in_file
import pytest import pytest

View File

@ -61,7 +61,7 @@ class TestSortFilesByMainLanguages:
type('', (object,), {'filename': 'file1.py'})(), type('', (object,), {'filename': 'file1.py'})(),
type('', (object,), {'filename': 'file2.java'})() type('', (object,), {'filename': 'file2.java'})()
] ]
expected_output = [{'language': 'Other', 'files': files}] expected_output = [{'language': 'Other', 'files': []}]
assert sort_files_by_main_languages(languages, files) == expected_output assert sort_files_by_main_languages(languages, files) == expected_output
# Tests that function handles empty files list # Tests that function handles empty files list