Compare commits

..

9 Commits
v0.8 ... pre_pr

56 changed files with 650 additions and 2378 deletions

View File

@ -24,7 +24,4 @@ jobs:
OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
OPENAI_ORG: ${{ secrets.OPENAI_ORG }} # optional
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PINECONE.API_KEY: ${{ secrets.PINECONE_API_KEY }}
PINECONE.ENVIRONMENT: ${{ secrets.PINECONE_ENVIRONMENT }}

57
CONFIGURATION.md Normal file
View File

@ -0,0 +1,57 @@
## Configuration
The different tools and sub-tools used by CodiumAI PR-Agent are adjustable via the **[configuration file](pr_agent/settings/configuration.toml)**
### Working from CLI
When running from source (CLI), your local configuration file will be initially used.
Example for invoking the 'review' tools via the CLI:
```
python cli.py --pr-url=<pr_url> review
```
In addition to general configurations, the 'review' tool will use parameters from the `[pr_reviewer]` section (every tool has a dedicated section in the configuration file).
Note that you can print results locally, without publishing them, by setting in `configuration.toml`:
```
[config]
publish_output=true
verbosity_level=2
```
This is useful for debugging or experimenting with the different tools.
### Working from pre-built repo (GitHub Action/GitHub App/Docker)
When running PR-Agent from a pre-built repo, the default configuration file will be loaded.
To edit the configuration, you have two options:
1. Place a local configuration file in the root of your local repo. The local file will be used instead of the default one.
2. For online usage, just add `--config_path=<value>` to you command, to edit a specific configuration value.
For example if you want to edit `pr_reviewer` configurations, you can run:
```
/review --pr_reviewer.extra_instructions="..." --pr_reviewer.require_score_review=false ...
```
Any configuration value in `configuration.toml` file can be similarly edited.
### General configuration parameters
#### Changing a model
See [here](pr_agent/algo/__init__.py) for the list of available models.
To use Llama2 model, for example, set:
```
[config]
model = "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1"
[replicate]
key = ...
```
(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
Also review the [AiHandler](pr_agent/algo/ai_handler.py) file for instruction how to set keys for other models.
#### Extra instructions
All PR-Agent tools have a parameter called `extra_instructions`, that enables to add free-text extra instructions. Example usage:
```
/update_changelog --pr_update_changelog.extra_instructions="Make sure to update also the version ..."
```

View File

@ -1,18 +0,0 @@
FROM python:3.10 as base
ENV OPENAI_API_KEY=${OPENAI_API_KEY} \
BITBUCKET_BEARER_TOKEN=${BITBUCKET_BEARER_TOKEN} \
BITBUCKET_PR_ID=${BITBUCKET_PR_ID} \
BITBUCKET_REPO_SLUG=${BITBUCKET_REPO_SLUG} \
BITBUCKET_WORKSPACE=${BITBUCKET_WORKSPACE}
WORKDIR /app
ADD pyproject.toml .
ADD requirements.txt .
RUN pip install . && rm pyproject.toml requirements.txt
ENV PYTHONPATH=/app
ADD pr_agent pr_agent
ADD bitbucket_pipeline/entrypoint.sh /
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

View File

@ -2,8 +2,7 @@ FROM python:3.10 as base
WORKDIR /app
ADD pyproject.toml .
ADD requirements.txt .
RUN pip install . && rm pyproject.toml requirements.txt
RUN pip install . && rm pyproject.toml
ENV PYTHONPATH=/app
ADD pr_agent pr_agent
ADD github_action/entrypoint.sh /

View File

@ -1 +1 @@
FROM codiumai/pr-agent:0.8-github_action
FROM codiumai/pr-agent:github_action

View File

@ -9,14 +9,12 @@ To get started with PR-Agent quickly, you first need to acquire two tokens:
There are several ways to use PR-Agent:
- [Method 1: Use Docker image (no installation required)](INSTALL.md#method-1-use-docker-image-no-installation-required)
- [Method 2: Run from source](INSTALL.md#method-2-run-from-source)
- [Method 3: Run as a GitHub Action](INSTALL.md#method-3-run-as-a-github-action)
- [Method 2: Run as a GitHub Action](INSTALL.md#method-2-run-as-a-github-action)
- [Method 3: Run from source](INSTALL.md#method-3-run-from-source)
- [Method 4: Run as a polling server](INSTALL.md#method-4-run-as-a-polling-server)
- [Method 5: Run as a GitHub App](INSTALL.md#method-5-run-as-a-github-app)
- [Method 6: Deploy as a Lambda Function](INSTALL.md#method-6---deploy-as-a-lambda-function)
- [Method 7: AWS CodeCommit](INSTALL.md#method-7---aws-codecommit-setup)
- [Method 8: Run a GitLab webhook server](INSTALL.md#method-8---run-a-gitlab-webhook-server)
- [Method 9: Run as a Bitbucket Pipeline](INSTALL.md#method-9-run-as-a-bitbucket-pipeline)
---
### Method 1: Use Docker image (no installation required)
@ -25,15 +23,9 @@ To request a review for a PR, or ask a question about a PR, you can run directly
1. To request a review for a PR, run the following command:
For GitHub:
```
docker run --rm -it -e OPENAI.KEY=<your key> -e GITHUB.USER_TOKEN=<your token> codiumai/pr-agent --pr_url <pr_url> review
```
For GitLab:
```
docker run --rm -it -e OPENAI.KEY=<your key> -e CONFIG.GIT_PROVIDER=gitlab -e GITLAB.PERSONAL_ACCESS_TOKEN=<your token> codiumai/pr-agent --pr_url <pr_url> review
```
For other git providers, update CONFIG.GIT_PROVIDER accordingly, and check the `pr_agent/settings/.secrets_template.toml` file for the environment variables expected names and values.
2. To ask a question about a PR, run the following command:
@ -63,41 +55,7 @@ Possible questions you can ask include:
---
### Method 2: Run from source
1. Clone this repository:
```
git clone https://github.com/Codium-ai/pr-agent.git
```
2. Install the requirements in your favorite virtual environment:
```
pip install -r requirements.txt
```
3. Copy the secrets template file and fill in your OpenAI key and your GitHub user token:
```
cp pr_agent/settings/.secrets_template.toml pr_agent/settings/.secrets.toml
chmod 600 pr_agent/settings/.secrets.toml
# Edit .secrets.toml file
```
4. Add the pr_agent folder to your PYTHONPATH, then run the cli.py script:
```
export PYTHONPATH=[$PYTHONPATH:]<PATH to pr_agent folder>
python pr_agent/cli.py --pr_url <pr_url> /review
python pr_agent/cli.py --pr_url <pr_url> /ask <your question>
python pr_agent/cli.py --pr_url <pr_url> /describe
python pr_agent/cli.py --pr_url <pr_url> /improve
```
---
### Method 3: Run as a GitHub Action
### Method 2: Run as a GitHub Action
You can use our pre-built Github Action Docker image to run PR-Agent as a Github Action.
@ -123,7 +81,7 @@ jobs:
OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```
** if you want to pin your action to a specific release (v0.7 for example) for stability reasons, use:
** if you want to pin your action to a specific commit for stability reasons
```yaml
on:
pull_request:
@ -140,7 +98,7 @@ jobs:
steps:
- name: PR Agent action step
id: pragent
uses: Codium-ai/pr-agent@v0.7
uses: Codium-ai/pr-agent@<commit_sha>
env:
OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@ -156,7 +114,7 @@ The GITHUB_TOKEN secret is automatically created by GitHub.
3. Merge this change to your main branch.
When you open your next PR, you should see a comment from `github-actions` bot with a review of your PR, and instructions on how to use the rest of the tools.
4. You may configure PR-Agent by adding environment variables under the env section corresponding to any configurable property in the [configuration](pr_agent/settings/configuration.toml) file. Some examples:
4. You may configure PR-Agent by adding environment variables under the env section corresponding to any configurable property in the [configuration](./CONFIGURATION.md) file. Some examples:
```yaml
env:
# ... previous environment values
@ -167,6 +125,40 @@ When you open your next PR, you should see a comment from `github-actions` bot w
---
### Method 3: Run from source
1. Clone this repository:
```
git clone https://github.com/Codium-ai/pr-agent.git
```
2. Install the requirements in your favorite virtual environment:
```
pip install -r requirements.txt
```
3. Copy the secrets template file and fill in your OpenAI key and your GitHub user token:
```
cp pr_agent/settings/.secrets_template.toml pr_agent/settings/.secrets.toml
chmod 600 pr_agent/settings/.secrets.toml
# Edit .secrets.toml file
```
4. Add the pr_agent folder to your PYTHONPATH, then run the cli.py script:
```
export PYTHONPATH=[$PYTHONPATH:]<PATH to pr_agent folder>
python pr_agent/cli.py --pr_url <pr_url> review
python pr_agent/cli.py --pr_url <pr_url> ask <your question>
python pr_agent/cli.py --pr_url <pr_url> describe
python pr_agent/cli.py --pr_url <pr_url> improve
```
---
### Method 4: Run as a polling server
Request reviews by tagging your Github user on a PR
@ -259,9 +251,6 @@ docker push codiumai/pr-agent:github_app # Push to your Docker repository
9. Install the app by navigating to the "Install App" tab and selecting your desired repositories.
> **Note:** When running PR-Agent from GitHub App, the default configuration file (configuration.toml) will be loaded.<br>
> However, you can override the default tool parameters by uploading a local configuration file<br>
> For more information please check out [CONFIGURATION.md](Usage.md#working-from-github-app-pre-built-repo)
---
### Method 6 - Deploy as a Lambda Function
@ -320,9 +309,7 @@ Example IAM permissions to that user to allow access to CodeCommit:
"codecommit:Get*",
"codecommit:List*",
"codecommit:PostComment*",
"codecommit:PutCommentReaction",
"codecommit:UpdatePullRequestDescription",
"codecommit:UpdatePullRequestTitle"
"codecommit:PutCommentReaction"
],
"Resource": "*"
}
@ -351,83 +338,9 @@ PYTHONPATH="/PATH/TO/PROJECTS/pr-agent" python pr_agent/cli.py \
review
```
---
### Appendix - **Debugging LLM API Calls**
If you're testing your codium/pr-agent server, and need to see if calls were made successfully + the exact call logs, you can use the [LiteLLM Debugger tool](https://docs.litellm.ai/docs/debugging/hosted_debugging).
### Method 8 - Run a GitLab webhook server
You can do this by setting `litellm_debugger=true` in configuration.toml. Your Logs will be viewable in real-time @ `admin.litellm.ai/<your_email>`. Set your email in the `.secrets.toml` under 'user_email'.
1. From the GitLab workspace or group, create an access token. Enable the "api" scope only.
2. Generate a random secret for your app, and save it for later. For example, you can use:
```
WEBHOOK_SECRET=$(python -c "import secrets; print(secrets.token_hex(10))")
```
3. Follow the instructions to build the Docker image, setup a secrets file and deploy on your own server from [Method 5](#method-5-run-as-a-github-app) steps 4-7.
4. In the secrets file, fill in the following:
- Your OpenAI key.
- In the [gitlab] section, fill in personal_access_token and shared_secret. The access token can be a personal access token, or a group or project access token.
- Set deployment_type to 'gitlab' in [configuration.toml](./pr_agent/settings/configuration.toml)
5. Create a webhook in GitLab. Set the URL to the URL of your app's server. Set the secret token to the generated secret from step 2.
In the "Trigger" section, check the comments and merge request events boxes.
6. Test your installation by opening a merge request or commenting or a merge request using one of CodiumAI's commands.
### Method 9: Run as a Bitbucket Pipeline
You can use our pre-build Bitbucket-Pipeline docker image to run as Bitbucket-Pipeline.
1. Add the following file in your repository bitbucket_pipelines.yml
```yaml
pipelines:
pull-requests:
'**':
- step:
name: PR Agent Pipeline
caches:
- pip
image: python:3.8
services:
- docker
script:
- git clone https://github.com/Codium-ai/pr-agent.git
- cd pr-agent
- docker build -t bitbucket_runner:latest -f Dockerfile.bitbucket_pipeline .
- docker run -e OPENAI_API_KEY=$OPENAI_API_KEY -e BITBUCKET_BEARER_TOKEN=$BITBUCKET_BEARER_TOKEN -e BITBUCKET_PR_ID=$BITBUCKET_PR_ID -e BITBUCKET_REPO_SLUG=$BITBUCKET_REPO_SLUG -e BITBUCKET_WORKSPACE=$BITBUCKET_WORKSPACE bitbucket_runner:latest
```
2. Add the following secret to your repository under Repository settings > Pipelines > Repository variables.
OPENAI_API_KEY: <your key>
BITBUCKET_BEARER_TOKEN: <your token>
3. To get BITBUCKET_BEARER_TOKEN follow these steps
So here is my step by step tutorial
i) Insert your workspace name instead of {workspace_name} and go to the following link in order to create an OAuth consumer.
https://bitbucket.org/{workspace_name}/workspace/settings/api
set callback URL to http://localhost:8976 (doesn't need to be a real server there)
select permissions: repository -> read
ii) use consumer's Key as a {client_id} and open the following URL in the browser
https://bitbucket.org/site/oauth2/authorize?client_id={client_id}&response_type=code
iii)
after you press "Grant access" in the browser it will redirect you to
http://localhost:8976?code=<CODE>
iv) use the code from the previous step and consumer's Key as a {client_id}, and consumer's Secret as {client_secret}
curl -X POST -u "{client_id}:{client_secret}" \
https://bitbucket.org/site/oauth2/access_token \
-d grant_type=authorization_code \
-d code={code} \
After completing this steps, you just to place this access token in the repository varibles.
=======
<img src="./pics/debugger.png" width="800"/>

1
MANIFEST.in Normal file
View File

@ -0,0 +1 @@
recursive-include pr_agent/settings/ *.toml

169
README.md
View File

@ -15,26 +15,19 @@ Making pull requests less painful with an AI agent
</div>
<div style="text-align:left;">
CodiumAI `PR-Agent` is an open-source tool aiming to help developers review pull requests faster and more efficiently. It automatically analyzes the pull request and can provide several types of commands:
CodiumAI `PR-Agent` is an open-source tool aiming to help developers review pull requests faster and more efficiently. It automatically analyzes the pull request and can provide several types of PR feedback:
**Auto Description (`/describe`)**: Automatically generating [PR description](https://github.com/Codium-ai/pr-agent/pull/229#issue-1860711415) - title, type, summary, code walkthrough and labels.
**Auto-Description**: Automatically generating [PR description](https://github.com/Codium-ai/pr-agent/pull/229#issue-1860711415) - title, type, summary, code walkthrough and labels.
\
**Auto Review (`/review`)**: [Adjustable feedback](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695022908) about the PR main theme, type, relevant tests, security issues, score, and various suggestions for the PR content.
**Auto Review**: [Adjustable feedback](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695022908) about the PR main theme, type, relevant tests, security issues, score, and various suggestions for the PR content.
\
**Question Answering (`/ask ...`)**: Answering [free-text questions](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021332) about the PR.
**Question Answering**: Answering [free-text questions](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021332) about the PR.
\
**Code Suggestions (`/improve`)**: [Committable code suggestions](https://github.com/Codium-ai/pr-agent/pull/229#discussion_r1306919276) for improving the PR.
**Code Suggestions**: [Committable code suggestions](https://github.com/Codium-ai/pr-agent/pull/229#discussion_r1306919276) for improving the PR.
\
**Update Changelog (`/update_changelog`)**: Automatically updating the CHANGELOG.md file with the [PR changes](https://github.com/Codium-ai/pr-agent/pull/168#discussion_r1282077645).
\
**Find similar issue (`/similar_issue`)**: Automatically retrieves and presents [similar issues](https://github.com/Alibaba-MIIL/ASL/issues/107).
**Update Changelog**: Automatically updating the CHANGELOG.md file with the [PR changes](https://github.com/Codium-ai/pr-agent/pull/168#discussion_r1282077645).
See the [usage guide](./Usage.md) for instructions how to run the different tools from [CLI](./Usage.md#working-from-a-local-repo-cli), or by [online usage](./Usage.md#online-usage), as well as additional details on optional commands and configurations.
[Release notes](./RELEASE_NOTES.md)
<h3>Example results:</h3>
<h3>Example results:</h2>
</div>
<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1687561986">/describe:</a></h4>
<div align="center">
@ -42,101 +35,87 @@ See the [usage guide](./Usage.md) for instructions how to run the different tool
<img src="https://www.codium.ai/images/describe-2.gif" width="800">
</p>
</div>
<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901">/review:</a></h4>
<div align="center">
<p float="center">
<img src="https://www.codium.ai/images/review-2.gif" width="800">
</p>
</div>
[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/78#issuecomment-1639739496">/reflect_and_review:</a></h4>)
[//]: # (<div align="center">)
[//]: # (<p float="center">)
[//]: # (<img src="https://www.codium.ai/images/reflect_and_review.gif" width="800">)
[//]: # (</p>)
[//]: # (</div>)
[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695020538">/ask:</a></h4>)
[//]: # (<div align="center">)
[//]: # (<p float="center">)
[//]: # (<img src="https://www.codium.ai/images/ask-2.gif" width="800">)
[//]: # (</p>)
[//]: # (</div>)
[//]: # (<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695024952">/improve:</a></h4>)
[//]: # (<div align="center">)
[//]: # (<p float="center">)
[//]: # (<img src="https://www.codium.ai/images/improve-2.gif" width="800">)
[//]: # (</p>)
[//]: # (</div>)
<h4><a href="https://github.com/Codium-ai/pr-agent/pull/78#issuecomment-1639739496">/reflect_and_review:</a></h4>
<div align="center">
<p float="center">
<img src="https://www.codium.ai/images/reflect_and_review.gif" width="800">
</p>
</div>
<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695020538">/ask:</a></h4>
<div align="center">
<p float="center">
<img src="https://www.codium.ai/images/ask-2.gif" width="800">
</p>
</div>
<h4><a href="https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695024952">/improve:</a></h4>
<div align="center">
<p float="center">
<img src="https://www.codium.ai/images/improve-2.gif" width="800">
</p>
</div>
<div align="left">
## Table of Contents
- [Overview](#overview)
- [Try it now](#try-it-now)
- [Installation](#installation)
- [Configuration](./CONFIGURATION.md)
- [How it works](#how-it-works)
- [Why use PR-Agent?](#why-use-pr-agent)
- [Why use PR-Agent](#why-use-pr-agent)
- [Roadmap](#roadmap)
- [Similar projects](#similar-projects)
</div>
## Overview
`PR-Agent` offers extensive pull request functionalities across various git providers:
| | | GitHub | Gitlab | Bitbucket | CodeCommit | Azure DevOps | Gerrit |
|-------|---------------------------------------------|:------:|:------:|:---------:|:----------:|:----------:|:----------:|
| TOOLS | Review | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | Ask | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | Auto-Description | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | Improve Code | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | :white_check_mark: |
| | ⮑ Extended | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | :white_check_mark: |
| | Reflect and Review | :white_check_mark: | :white_check_mark: | :white_check_mark: | | :white_check_mark: | :white_check_mark: |
| | Update CHANGELOG.md | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | |
| | Find similar issue | :white_check_mark: | | | | | |
| | | | | | | |
| USAGE | CLI | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | App / webhook | :white_check_mark: | :white_check_mark: | | | |
| | Tagging bot | :white_check_mark: | | | | |
| | Actions | :white_check_mark: | | | | |
| | Web server | | | | | | :white_check_mark: |
| | | | | | | |
| CORE | PR compression | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | Repo language prioritization | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | Adaptive and token-aware<br />file patch fitting | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | Multiple models support | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | Incremental PR Review | :white_check_mark: | | | | | |
| | | GitHub | Gitlab | Bitbucket | CodeCommit |
|-------|---------------------------------------------|:------:|:------:|:---------:|:----------:|
| TOOLS | Review | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | ⮑ Inline review | :white_check_mark: | :white_check_mark: | | |
| | Ask | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | Auto-Description | :white_check_mark: | :white_check_mark: | | |
| | Improve Code | :white_check_mark: | :white_check_mark: | | |
| | ⮑ Extended | :white_check_mark: | :white_check_mark: | | |
| | Reflect and Review | :white_check_mark: | | | |
| | Update CHANGELOG.md | :white_check_mark: | | | |
| | | | | | |
| USAGE | CLI | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| | App / webhook | :white_check_mark: | :white_check_mark: | | |
| | Tagging bot | :white_check_mark: | | | |
| | Actions | :white_check_mark: | | | |
| | | | | | |
| CORE | PR compression | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| | Repo language prioritization | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| | Adaptive and token-aware<br />file patch fitting | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| | Multiple models support | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| | Incremental PR Review | :white_check_mark: | | | |
Review the **[usage guide](./Usage.md)** section for detailed instructions how to use the different tools, select the relevant git provider (GitHub, Gitlab, Bitbucket,...), and adjust the configuration file to your needs.
Examples for invoking the different tools via the CLI:
- **Review**: python cli.py --pr_url=<pr_url> review
- **Describe**: python cli.py --pr_url=<pr_url> describe
- **Improve**: python cli.py --pr_url=<pr_url> improve
- **Ask**: python cli.py --pr_url=<pr_url> ask "Write me a poem about this PR"
- **Reflect**: python cli.py --pr_url=<pr_url> reflect
- **Update Changelog**: python cli.py --pr_url=<pr_url> update_changelog
"<pr_url>" is the url of the relevant PR (for example: https://github.com/Codium-ai/pr-agent/pull/50).
In the [configuration](./CONFIGURATION.md) file you can select your git provider (GitHub, Gitlab, Bitbucket), and further configure the different tools.
## Try it now
You can try GPT-4 powered PR-Agent, on your public GitHub repository, instantly. Just mention `@CodiumAI-Agent` and add the desired command in any PR comment. The agent will generate a response based on your command.
For example, add a comment to any pull request with the following text:
```
@CodiumAI-Agent /review
```
and the agent will respond with a review of your PR
Try GPT-4 powered PR-Agent on your public GitHub repository for free. Just mention `@CodiumAI-Agent` and add the desired command in any PR comment! The agent will generate a response based on your command.
![Review generation process](https://www.codium.ai/images/demo-2.gif)
To set up your own PR-Agent, see the [Installation](#installation) section below.
To set up your own PR-Agent, see the [Installation](#installation) section
---
@ -150,16 +129,14 @@ To get started with PR-Agent quickly, you first need to acquire two tokens:
There are several ways to use PR-Agent:
- [Method 1: Use Docker image (no installation required)](INSTALL.md#method-1-use-docker-image-no-installation-required)
- [Method 2: Run from source](INSTALL.md#method-2-run-from-source)
- [Method 3: Run as a GitHub Action](INSTALL.md#method-3-run-as-a-github-action)
- [Method 2: Run as a GitHub Action](INSTALL.md#method-2-run-as-a-github-action)
- [Method 3: Run from source](INSTALL.md#method-3-run-from-source)
- [Method 4: Run as a polling server](INSTALL.md#method-4-run-as-a-polling-server)
- Request reviews by tagging your GitHub user on a PR
- [Method 5: Run as a GitHub App](INSTALL.md#method-5-run-as-a-github-app)
- Allowing you to automate the review process on your private or public repositories
- [Method 6: Deploy as a Lambda Function](INSTALL.md#method-6---deploy-as-a-lambda-function)
- [Method 7: AWS CodeCommit](INSTALL.md#method-7---aws-codecommit-setup)
- [Method 8: Run a GitLab webhook server](INSTALL.md#method-8---run-a-gitlab-webhook-server)
- [Method 9: Run as a Bitbucket Pipeline](INSTALL.md#method-9-run-as-a-bitbucket-pipeline)
## How it works
@ -177,7 +154,7 @@ Here are some advantages of PR-Agent:
- We emphasize **real-life practical usage**. Each tool (review, improve, ask, ...) has a single GPT-4 call, no more. We feel that this is critical for realistic team usage - obtaining an answer quickly (~30 seconds) and affordably.
- Our [PR Compression strategy](./PR_COMPRESSION.md) is a core ability that enables to effectively tackle both short and long PRs.
- Our JSON prompting strategy enables to have **modular, customizable tools**. For example, the '/review' tool categories can be controlled via the [configuration](pr_agent/settings/configuration.toml) file. Adding additional categories is easy and accessible.
- Our JSON prompting strategy enables to have **modular, customizable tools**. For example, the '/review' tool categories can be controlled via the [configuration](./CONFIGURATION.md) file. Adding additional categories is easy and accessible.
- We support **multiple git providers** (GitHub, Gitlab, Bitbucket, CodeCommit), **multiple ways** to use the tool (CLI, GitHub Action, GitHub App, Docker, ...), and **multiple models** (GPT-4, GPT-3.5, Anthropic, Cohere, Llama2).
- We are open-source, and welcome contributions from the community.
@ -187,7 +164,7 @@ Here are some advantages of PR-Agent:
- [x] Support additional models, as a replacement for OpenAI (see [here](https://github.com/Codium-ai/pr-agent/pull/172))
- [x] Develop additional logic for handling large PRs (see [here](https://github.com/Codium-ai/pr-agent/pull/229))
- [ ] Add additional context to the prompt. For example, repo (or relevant files) summarization, with tools such a [ctags](https://github.com/universal-ctags/ctags)
- [x] PR-Agent for issues
- [ ] PR-Agent for issues, and just for pull requests
- [ ] Adding more tools. Possible directions:
- [x] PR description
- [x] Inline code suggestions
@ -200,18 +177,8 @@ Here are some advantages of PR-Agent:
## Similar Projects
- [CodiumAI - Meaningful tests for busy devs](https://github.com/Codium-ai/codiumai-vscode-release) (although various capabilities are much more advanced in the CodiumAI IDE plugins)
- [CodiumAI - Meaningful tests for busy devs](https://github.com/Codium-ai/codiumai-vscode-release)
- [Aider - GPT powered coding in your terminal](https://github.com/paul-gauthier/aider)
- [openai-pr-reviewer](https://github.com/coderabbitai/openai-pr-reviewer)
- [CodeReview BOT](https://github.com/anc95/ChatGPT-CodeReview)
- [AI-Maintainer](https://github.com/merwanehamadi/AI-Maintainer)
## Links
[![Join our Discord community](https://raw.githubusercontent.com/Codium-ai/codiumai-vscode-release/main/media/docs/Joincommunity.png)](https://discord.gg/kG35uSHDBc)
- Discord community: https://discord.gg/kG35uSHDBc
- CodiumAI site: https://codium.ai
- Blog: https://www.codium.ai/blog/
- Troubleshooting: https://www.codium.ai/blog/technical-faq-and-troubleshooting/
- Support: support@codium.ai

View File

@ -1,41 +0,0 @@
## [Version 0.8] - 2023-09-27
- codiumai/pr-agent:0.8
- codiumai/pr-agent:0.8-github_app
- codiumai/pr-agent:0.8-bitbucket-app
- codiumai/pr-agent:0.8-gitlab_webhook
- codiumai/pr-agent:0.8-github_polling
- codiumai/pr-agent:0.8-github_action
### Added::Algo
- GitHub Action: Can control which tools will run automatically when a new PR is created. (see usage guide: https://github.com/Codium-ai/pr-agent/blob/main/Usage.md#working-with-github-action)
- Code suggestion tool: Will try to avoid an 'add comments' suggestion (see https://github.com/Codium-ai/pr-agent/pull/327)
### Fixed
- Gitlab: Fixed a bug of improper usage of pr_id
## [Version 0.7] - 2023-09-20
### Docker Tags
- codiumai/pr-agent:0.7
- codiumai/pr-agent:0.7-github_app
- codiumai/pr-agent:0.7-bitbucket-app
- codiumai/pr-agent:0.7-gitlab_webhook
- codiumai/pr-agent:0.7-github_polling
- codiumai/pr-agent:0.7-github_action
### Added::Algo
- New tool /similar_issue - Currently on GitHub app and CLI: indexes the issues in the repo, find the most similar issues to the target issue.
- Describe markers: Empower the /describe tool with a templating capability (see more details in https://github.com/Codium-ai/pr-agent/pull/273).
- New feature in the /review tool - added an estimated effort estimation to the review (https://github.com/Codium-ai/pr-agent/pull/306).
### Added::Infrastructure
- Implementation of a GitLab webhook.
- Implementation of a BitBucket app.
### Fixed
- Protection against no code suggestions generated.
- Resilience to repositories where the languages cannot be automatically detected.

286
Usage.md
View File

@ -1,286 +0,0 @@
## Usage guide
### Table of Contents
- [Introduction](#introduction)
- [Working from a local repo (CLI)](#working-from-a-local-repo-cli)
- [Online usage](#online-usage)
- [Working with GitHub App](#working-with-github-app)
- [Working with GitHub Action](#working-with-github-action)
- [Appendix - additional configurations walkthrough](#appendix---additional-configurations-walkthrough)
### Introduction
There are 3 basic ways to invoke CodiumAI PR-Agent:
1. Locally running a CLI command
2. Online usage - by [commenting](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901) on a PR
3. Enabling PR-Agent tools to run automatically when a new PR is opened
See the [installation guide](/INSTALL.md) for instructions on how to setup your own PR-Agent.
Specifically, CLI commands can be issued by invoking a pre-built [docker image](/INSTALL.md#running-from-source), or by invoking a [locally cloned repo](INSTALL.md#method-2-run-from-source).
For online usage, you will need to setup either a [GitHub App](INSTALL.md#method-5-run-as-a-github-app), or a [GitHub Action](INSTALL.md#method-3-run-as-a-github-action).
GitHub App and GitHub Action also enable to run PR-Agent specific tool automatically when a new PR is opened.
#### The configuration file
The different tools and sub-tools used by CodiumAI PR-Agent are adjustable via the **[configuration file](pr_agent/settings/configuration.toml)**.
In addition to general configuration options, each tool has its own configurations. For example, the `review` tool will use parameters from the [pr_reviewer](/pr_agent/settings/configuration.toml#L16) section in the configuration file.
**git provider:**
The [git_provider](pr_agent/settings/configuration.toml#L4) field in the configuration file determines the GIT provider that will be used by PR-Agent. Currently, the following providers are supported:
`
"github", "gitlab", "azure", "codecommit", "local"
`
[//]: # (** online usage:**)
[//]: # (Options that are available in the configuration file can be specified at run time when calling actions. Two examples:)
[//]: # (```)
[//]: # (- /review --pr_reviewer.extra_instructions="focus on the file: ...")
[//]: # (- /describe --pr_description.add_original_user_description=false -pr_description.extra_instructions="make sure to mention: ...")
[//]: # (```)
### Working from a local repo (CLI)
When running from your local repo (CLI), your local configuration file will be used.
Examples for invoking the different tools via the CLI:
- **Review**: `python cli.py --pr_url=<pr_url> review`
- **Describe**: `python cli.py --pr_url=<pr_url> describe`
- **Improve**: `python cli.py --pr_url=<pr_url> improve`
- **Ask**: `python cli.py --pr_url=<pr_url> ask "Write me a poem about this PR"`
- **Reflect**: `python cli.py --pr_url=<pr_url> reflect`
- **Update Changelog**: `python cli.py --pr_url=<pr_url> update_changelog`
`<pr_url>` is the url of the relevant PR (for example: https://github.com/Codium-ai/pr-agent/pull/50).
**Notes:**
(1) in addition to editing your local configuration file, you can also change any configuration value by adding it to the command line:
```
python cli.py --pr_url=<pr_url> /review --pr_reviewer.extra_instructions="focus on the file: ..."
```
(2) You can print results locally, without publishing them, by setting in `configuration.toml`:
```
[config]
publish_output=true
verbosity_level=2
```
This is useful for debugging or experimenting with the different tools.
### Online usage
Online usage means invoking PR-Agent tools by [comments](https://github.com/Codium-ai/pr-agent/pull/229#issuecomment-1695021901) on a PR.
Commands for invoking the different tools via comments:
- **Review**: `/review`
- **Describe**: `/describe`
- **Improve**: `/improve`
- **Ask**: `/ask "..."`
- **Reflect**: `/reflect`
- **Update Changelog**: `/update_changelog`
To edit a specific configuration value, just add `--config_path=<value>` to any command.
For example if you want to edit the `review` tool configurations, you can run:
```
/review --pr_reviewer.extra_instructions="..." --pr_reviewer.require_score_review=false
```
Any configuration value in [configuration file](pr_agent/settings/configuration.toml) file can be similarly edited.
### Working with GitHub App
When running PR-Agent from [GitHub App](INSTALL.md#method-5-run-as-a-github-app), the default configurations from a pre-built repo will be initially loaded.
#### GitHub app automatic tools
The [github_app](pr_agent/settings/configuration.toml#L56) section defines GitHub app specific configurations.
An important parameter is `pr_commands`, which is a list of tools that will be **run automatically when a new PR is opened**:
```
[github_app]
pr_commands = [
"/describe --pr_description.add_original_user_description=true --pr_description.keep_original_user_title=true",
"/auto_review",
]
```
This means that when a new PR is opened, PR-Agent will run the `describe` and `auto_review` tools.
For the describe tool, the `add_original_user_description` and `keep_original_user_title` parameters will be set to true.
However, you can override the default tool parameters by uploading a local configuration file called `.pr_agent.toml` to the root of your repo.
For example, if your local `.pr_agent.toml` file contains:
```
[pr_description]
add_original_user_description = false
keep_original_user_title = false
```
When a new PR is opened, PR-Agent will run the `describe` tool with the above parameters.
Note that a local `.pr_agent.toml` file enables you to edit and customize the default parameters of any tool, not just the ones that are run automatically.
#### Editing the prompts
The prompts for the various PR-Agent tools are defined in the `pr_agent/settings` folder.
In practice, the prompts are loaded and stored as a standard setting object.
Hence, editing them is similar to editing any other configuration value - just place the relevant key in `.pr_agent.toml`file, and override the default value.
For example, if you want to edit the prompts of the [describe](./pr_agent/settings/pr_description_prompts.toml) tool, you can add the following to your `.pr_agent.toml` file:
```
[pr_description_prompt]
system="""
...
"""
user="""
...
"""
```
Note that the new prompt will need to generate an output compatible with the relevant [post-process function](./pr_agent/tools/pr_description.py#L137).
### Working with GitHub Action
You can configure settings in GitHub action by adding environment variables under the env section in `.github/workflows/pr_agent.yml` file. Some examples:
```yaml
env:
# ... previous environment values
OPENAI.ORG: "<Your organization name under your OpenAI account>"
PR_REVIEWER.REQUIRE_TESTS_REVIEW: "false" # Disable tests review
PR_CODE_SUGGESTIONS.NUM_CODE_SUGGESTIONS: 6 # Increase number of code suggestions
github_action.auto_review: "true" # Enable auto review
github_action.auto_describe: "true" # Enable auto describe
github_action.auto_improve: "false" # Disable auto improve
```
specifically, `github_action.auto_review`, `github_action.auto_describe` and `github_action.auto_improve` are used to enable/disable automatic tools that run when a new PR is opened.
if not set, the default option is that only the `review` tool will run automatically when a new PR is opened.
### Appendix - additional configurations walkthrough
#### Changing a model
See [here](pr_agent/algo/__init__.py) for the list of available models.
#### Azure
To use Azure, set:
```
api_key = "" # your azure api key
api_type = "azure"
api_version = '2023-05-15' # Check Azure documentation for the current API version
api_base = "" # The base URL for your Azure OpenAI resource. e.g. "https://<your resource name>.openai.azure.com"
deployment_id = "" # The deployment name you chose when you deployed the engine
```
in your .secrets.toml
and
```
[config]
model="" # the OpenAI model you've deployed on Azure (e.g. gpt-3.5-turbo)
```
in the configuration.toml
#### Huggingface
**Local**
You can run Huggingface models locally through either [VLLM](https://docs.litellm.ai/docs/providers/vllm) or [Ollama](https://docs.litellm.ai/docs/providers/ollama)
E.g. to use a new Huggingface model locally via Ollama, set:
```
[__init__.py]
MAX_TOKENS = {
"model-name-on-ollama": <max_tokens>
}
e.g.
MAX_TOKENS={
...,
"llama2": 4096
}
[config] # in configuration.toml
model = "ollama/llama2"
[ollama] # in .secrets.toml
api_base = ... # the base url for your huggingface inference endpoint
```
**Inference Endpoints**
To use a new model with Huggingface Inference Endpoints, for example, set:
```
[__init__.py]
MAX_TOKENS = {
"model-name-on-huggingface": <max_tokens>
}
e.g.
MAX_TOKENS={
...,
"meta-llama/Llama-2-7b-chat-hf": 4096
}
[config] # in configuration.toml
model = "huggingface/meta-llama/Llama-2-7b-chat-hf"
[huggingface] # in .secrets.toml
key = ... # your huggingface api key
api_base = ... # the base url for your huggingface inference endpoint
```
(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
#### Replicate
To use Llama2 model with Replicate, for example, set:
```
[config] # in configuration.toml
model = "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1"
[replicate] # in .secrets.toml
key = ...
```
(you can obtain a Llama2 key from [here](https://replicate.com/replicate/llama-2-70b-chat/api))
Also review the [AiHandler](pr_agent/algo/ai_handler.py) file for instruction how to set keys for other models.
#### Extra instructions
All PR-Agent tools have a parameter called `extra_instructions`, that enables to add free-text extra instructions. Example usage:
```
/update_changelog --pr_update_changelog.extra_instructions="Make sure to update also the version ..."
```
#### Azure DevOps provider
To use Azure DevOps provider use the following settings in configuration.toml:
```
[config]
git_provider="azure"
use_repo_settings_file=false
```
And use the following settings (you have to replace the values) in .secrets.toml:
```
[azure_devops]
org = "https://dev.azure.com/YOUR_ORGANIZATION/"
pat = "YOUR_PAT_TOKEN"
```
#### Similar issue tool
[Example usage](https://github.com/Alibaba-MIIL/ASL/issues/107)
<img src=./pics/similar_issue_tool.png width="768">
To enable usage of the '**similar issue**' tool, you need to set the following keys in `.secrets.toml` (or in the relevant environment variables):
```
[pinecone]
api_key = "..."
environment = "..."
```
These parameters can be obtained by registering to [Pinecone](https://app.pinecone.io/?sessionType=signup/).
- To invoke the 'similar issue' tool from **CLI**, run:
`python3 cli.py --issue_url=... similar_issue`
- To invoke the 'similar' issue tool via online usage, [comment](https://github.com/Codium-ai/pr-agent/issues/178#issuecomment-1716934893) on a PR:
`/similar_issue`
- You can also enable the 'similar issue' tool to run automatically when a new issue is opened, by adding it to the [pr_commands list in the github_app section](https://github.com/Codium-ai/pr-agent/blob/main/pr_agent/settings/configuration.toml#L66)

View File

@ -1,2 +0,0 @@
#!/bin/bash
python /app/pr_agent/servers/bitbucket_pipeline_runner.py

View File

@ -18,10 +18,6 @@ FROM base as github_polling
ADD pr_agent pr_agent
CMD ["python", "pr_agent/servers/github_polling.py"]
FROM base as gitlab_webhook
ADD pr_agent pr_agent
CMD ["python", "pr_agent/servers/gitlab_webhook.py"]
FROM base as test
ADD requirements-dev.txt .
RUN pip install -r requirements-dev.txt && rm requirements-dev.txt

BIN
pics/debugger.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 534 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 210 KiB

View File

@ -9,7 +9,6 @@ from pr_agent.git_providers import get_git_provider
from pr_agent.tools.pr_code_suggestions import PRCodeSuggestions
from pr_agent.tools.pr_description import PRDescription
from pr_agent.tools.pr_information_from_user import PRInformationFromUser
from pr_agent.tools.pr_similar_issue import PRSimilarIssue
from pr_agent.tools.pr_questions import PRQuestions
from pr_agent.tools.pr_reviewer import PRReviewer
from pr_agent.tools.pr_update_changelog import PRUpdateChangelog
@ -31,7 +30,6 @@ command2class = {
"update_changelog": PRUpdateChangelog,
"config": PRConfig,
"settings": PRConfig,
"similar_issue": PRSimilarIssue,
}
commands = list(command2class.keys())

View File

@ -1,5 +1,4 @@
MAX_TOKENS = {
'text-embedding-ada-002': 8000,
'gpt-3.5-turbo': 4000,
'gpt-3.5-turbo-0613': 4000,
'gpt-3.5-turbo-0301': 4000,
@ -12,5 +11,4 @@ MAX_TOKENS = {
'claude-2': 100000,
'command-nightly': 4096,
'replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1': 4096,
'meta-llama/Llama-2-7b-chat-hf': 4096
}

View File

@ -1,12 +1,13 @@
import logging
import os
import litellm
import openai
from litellm import acompletion
from openai.error import APIError, RateLimitError, Timeout, TryAgain
from retry import retry
from pr_agent.config_loader import get_settings
OPENAI_RETRIES = 5
@ -25,11 +26,7 @@ class AiHandler:
try:
openai.api_key = get_settings().openai.key
litellm.openai_key = get_settings().openai.key
if get_settings().get("litellm.use_client"):
litellm_token = get_settings().get("litellm.LITELLM_TOKEN")
assert litellm_token, "LITELLM_TOKEN is required"
os.environ["LITELLM_TOKEN"] = litellm_token
litellm.use_client = True
litellm.debugger = get_settings().config.litellm_debugger
self.azure = False
if get_settings().get("OPENAI.ORG", None):
litellm.organization = get_settings().openai.org
@ -51,8 +48,6 @@ class AiHandler:
litellm.replicate_key = get_settings().replicate.key
if get_settings().get("HUGGINGFACE.KEY", None):
litellm.huggingface_key = get_settings().huggingface.key
if get_settings().get("HUGGINGFACE.API_BASE", None):
litellm.api_base = get_settings().huggingface.api_base
except AttributeError as e:
raise ValueError("OpenAI key is required") from e

View File

@ -40,16 +40,12 @@ def extend_patch(original_file_str, patch_str, num_lines) -> str:
extended_patch_lines.extend(
original_lines[start1 + size1 - 1:start1 + size1 - 1 + num_lines])
res = list(match.groups())
for i in range(len(res)):
if res[i] is None:
res[i] = 0
try:
start1, size1, start2, size2 = map(int, res[:4])
start1, size1, start2, size2 = map(int, match.groups()[:4])
except: # '@@ -0,0 +1 @@' case
start1, size1, size2 = map(int, res[:3])
start1, size1, size2 = map(int, match.groups()[:3])
start2 = 0
section_header = res[4]
section_header = match.groups()[4]
extended_start1 = max(1, start1 - num_lines)
extended_size1 = size1 + (start1 - extended_start1) + num_lines
extended_start2 = max(1, start2 - num_lines)
@ -211,15 +207,10 @@ __old hunk__
old_content_lines = []
if match:
prev_header_line = header_line
res = list(match.groups())
for i in range(len(res)):
if res[i] is None:
res[i] = 0
try:
start1, size1, start2, size2 = map(int, res[:4])
start1, size1, start2, size2 = map(int, match.groups()[:4])
except: # '@@ -0,0 +1 @@' case
start1, size1, size2 = map(int, res[:3])
start1, size1, size2 = map(int, match.groups()[:3])
start2 = 0
elif line.startswith('+'):

View File

@ -42,11 +42,6 @@ def sort_files_by_main_languages(languages: Dict, files: list):
files_sorted = []
rest_files = {}
# if no languages detected, put all files in the "Other" category
if not languages:
files_sorted = [({"language": "Other", "files": list(files_filtered)})]
return files_sorted
main_extensions_flat = []
for ext in main_extensions:
main_extensions_flat.extend(ext)

View File

@ -1,19 +1,17 @@
from __future__ import annotations
import difflib
import logging
import re
import traceback
from typing import Any, Callable, List, Tuple
from typing import Callable, List, Tuple
from github import RateLimitExceededException
from pr_agent.algo import MAX_TOKENS
from pr_agent.algo.git_patch_processing import convert_to_hunks_with_lines_numbers, extend_patch, handle_patch_deletions
from pr_agent.algo.language_handler import sort_files_by_main_languages
from pr_agent.algo.token_handler import TokenHandler, get_token_encoder
from pr_agent.algo.token_handler import TokenHandler
from pr_agent.config_loader import get_settings
from pr_agent.git_providers.git_provider import FilePatchInfo, GitProvider
from pr_agent.git_providers.git_provider import GitProvider
DELETED_FILES_ = "Deleted files:\n"
@ -247,99 +245,6 @@ def _get_all_deployments(all_models: List[str]) -> List[str]:
return all_deployments
def find_line_number_of_relevant_line_in_file(diff_files: List[FilePatchInfo],
relevant_file: str,
relevant_line_in_file: str) -> Tuple[int, int]:
"""
Find the line number and absolute position of a relevant line in a file.
Args:
diff_files (List[FilePatchInfo]): A list of FilePatchInfo objects representing the patches of files.
relevant_file (str): The name of the file where the relevant line is located.
relevant_line_in_file (str): The content of the relevant line.
Returns:
Tuple[int, int]: A tuple containing the line number and absolute position of the relevant line in the file.
"""
position = -1
absolute_position = -1
re_hunk_header = re.compile(
r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@[ ]?(.*)")
for file in diff_files:
if file.filename.strip() == relevant_file:
patch = file.patch
patch_lines = patch.splitlines()
# try to find the line in the patch using difflib, with some margin of error
matches_difflib: list[str | Any] = difflib.get_close_matches(relevant_line_in_file,
patch_lines, n=3, cutoff=0.93)
if len(matches_difflib) == 1 and matches_difflib[0].startswith('+'):
relevant_line_in_file = matches_difflib[0]
delta = 0
start1, size1, start2, size2 = 0, 0, 0, 0
for i, line in enumerate(patch_lines):
if line.startswith('@@'):
delta = 0
match = re_hunk_header.match(line)
start1, size1, start2, size2 = map(int, match.groups()[:4])
elif not line.startswith('-'):
delta += 1
if relevant_line_in_file in line and line[0] != '-':
position = i
absolute_position = start2 + delta - 1
break
if position == -1 and relevant_line_in_file[0] == '+':
no_plus_line = relevant_line_in_file[1:].lstrip()
for i, line in enumerate(patch_lines):
if line.startswith('@@'):
delta = 0
match = re_hunk_header.match(line)
start1, size1, start2, size2 = map(int, match.groups()[:4])
elif not line.startswith('-'):
delta += 1
if no_plus_line in line and line[0] != '-':
# The model might add a '+' to the beginning of the relevant_line_in_file even if originally
# it's a context line
position = i
absolute_position = start2 + delta - 1
break
return position, absolute_position
def clip_tokens(text: str, max_tokens: int) -> str:
"""
Clip the number of tokens in a string to a maximum number of tokens.
Args:
text (str): The string to clip.
max_tokens (int): The maximum number of tokens allowed in the string.
Returns:
str: The clipped string.
"""
if not text:
return text
try:
encoder = get_token_encoder()
num_input_tokens = len(encoder.encode(text))
if num_input_tokens <= max_tokens:
return text
num_chars = len(text)
chars_per_token = num_chars / num_input_tokens
num_output_chars = int(chars_per_token * max_tokens)
clipped_text = text[:num_output_chars]
return clipped_text
except Exception as e:
logging.warning(f"Failed to clip tokens: {e}")
return text
def get_pr_multi_diffs(git_provider: GitProvider,
token_handler: TokenHandler,
model: str,

View File

@ -21,7 +21,7 @@ class TokenHandler:
method.
"""
def __init__(self, pr=None, vars: dict = {}, system="", user=""):
def __init__(self, vars: dict, system, user):
"""
Initializes the TokenHandler object.
@ -32,10 +32,9 @@ class TokenHandler:
- user: The user string.
"""
self.encoder = get_token_encoder()
if pr is not None:
self.prompt_tokens = self._get_system_user_tokens(pr, self.encoder, vars, system, user)
self.prompt_tokens = self._get_system_user_tokens(self.encoder, vars, system, user)
def _get_system_user_tokens(self, pr, encoder, vars: dict, system, user):
def _get_system_user_tokens(self, encoder, vars: dict, system, user):
"""
Calculates the number of tokens in the system and user strings.

View File

@ -5,14 +5,24 @@ import json
import logging
import re
import textwrap
from dataclasses import dataclass
from datetime import datetime
from typing import Any, List
from enum import Enum
from typing import Any, List, Tuple, Optional
import yaml
from starlette_context import context
from pr_agent.algo.token_handler import get_token_encoder
from pr_agent.config_loader import get_settings, global_settings
class EDIT_TYPE(Enum):
ADDED = 1
DELETED = 2
MODIFIED = 3
RENAMED = 4
def get_setting(key: str) -> Any:
try:
key = key.upper()
@ -20,7 +30,7 @@ def get_setting(key: str) -> Any:
except Exception:
return global_settings.get(key, None)
def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
def convert_to_markdown(output_data: dict) -> str:
"""
Convert a dictionary of data into markdown format.
Args:
@ -42,7 +52,6 @@ def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
"General suggestions": "💡",
"Insights from user's answers": "📝",
"Code feedback": "🤖",
"Estimated effort to review [1-5]": "⏱️",
}
for key, value in output_data.items():
@ -50,14 +59,11 @@ def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
continue
if isinstance(value, dict):
markdown_text += f"## {key}\n\n"
markdown_text += convert_to_markdown(value, gfm_supported)
markdown_text += convert_to_markdown(value)
elif isinstance(value, list):
emoji = emojis.get(key, "")
if key.lower() == 'code feedback':
if gfm_supported:
markdown_text += f"\n\n- **<details><summary> { emoji } Code feedback:**</summary>\n\n"
else:
markdown_text += f"\n\n- **{emoji} Code feedback:**\n\n"
markdown_text += f"\n\n- **<details><summary> { emoji } Code feedback:**</summary>\n\n"
else:
markdown_text += f"- {emoji} **{key}:**\n\n"
for item in value:
@ -66,10 +72,7 @@ def convert_to_markdown(output_data: dict, gfm_supported: bool=True) -> str:
elif item:
markdown_text += f" - {item}\n"
if key.lower() == 'code feedback':
if gfm_supported:
markdown_text += "</details>\n\n"
else:
markdown_text += "\n\n"
markdown_text += "</details>\n\n"
elif value != 'n/a':
emoji = emojis.get(key, "")
markdown_text += f"- {emoji} **{key}:** {value}\n"
@ -175,7 +178,7 @@ def fix_json_escape_char(json_message=None):
Raises:
None
"""
"""
try:
result = json.loads(json_message)
except Exception as e:
@ -202,7 +205,7 @@ def convert_str_to_datetime(date_str):
Example:
>>> convert_str_to_datetime('Mon, 01 Jan 2022 12:00:00 UTC')
datetime.datetime(2022, 1, 1, 12, 0, 0)
"""
"""
datetime_format = '%a, %d %b %Y %H:%M:%S %Z'
return datetime.strptime(date_str, datetime_format)
@ -301,3 +304,108 @@ def try_fix_yaml(review_text: str) -> dict:
except:
pass
return data
def clip_tokens(text: str, max_tokens: int) -> str:
"""
Clip the number of tokens in a string to a maximum number of tokens.
Args:
text (str): The string to clip.
max_tokens (int): The maximum number of tokens allowed in the string.
Returns:
str: The clipped string.
"""
if not text:
return text
try:
encoder = get_token_encoder()
num_input_tokens = len(encoder.encode(text))
if num_input_tokens <= max_tokens:
return text
num_chars = len(text)
chars_per_token = num_chars / num_input_tokens
num_output_chars = int(chars_per_token * max_tokens)
clipped_text = text[:num_output_chars]
return clipped_text
except Exception as e:
logging.warning(f"Failed to clip tokens: {e}")
return text
@dataclass
class FilePatchInfo:
base_file: str
head_file: str
patch: str
filename: str
tokens: int = -1
edit_type: EDIT_TYPE = EDIT_TYPE.MODIFIED
old_filename: str = None
language: Optional[str] = None
def find_line_number_of_relevant_line_in_file(diff_files: List[FilePatchInfo],
relevant_file: str,
relevant_line_in_file: str) -> Tuple[int, int]:
"""
Find the line number and absolute position of a relevant line in a file.
Args:
diff_files (List[FilePatchInfo]): A list of FilePatchInfo objects representing the patches of files.
relevant_file (str): The name of the file where the relevant line is located.
relevant_line_in_file (str): The content of the relevant line.
Returns:
Tuple[int, int]: A tuple containing the line number and absolute position of the relevant line in the file.
"""
position = -1
absolute_position = -1
re_hunk_header = re.compile(
r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@[ ]?(.*)")
for file in diff_files:
if file.filename.strip() == relevant_file:
patch = file.patch
patch_lines = patch.splitlines()
# try to find the line in the patch using difflib, with some margin of error
matches_difflib: list[str | Any] = difflib.get_close_matches(relevant_line_in_file,
patch_lines, n=3, cutoff=0.93)
if len(matches_difflib) == 1 and matches_difflib[0].startswith('+'):
relevant_line_in_file = matches_difflib[0]
delta = 0
start1, size1, start2, size2 = 0, 0, 0, 0
for i, line in enumerate(patch_lines):
if line.startswith('@@'):
delta = 0
match = re_hunk_header.match(line)
start1, size1, start2, size2 = map(int, match.groups()[:4])
elif not line.startswith('-'):
delta += 1
if relevant_line_in_file in line and line[0] != '-':
position = i
absolute_position = start2 + delta - 1
break
if position == -1 and relevant_line_in_file[0] == '+':
no_plus_line = relevant_line_in_file[1:].lstrip()
for i, line in enumerate(patch_lines):
if line.startswith('@@'):
delta = 0
match = re_hunk_header.match(line)
start1, size1, start2, size2 = map(int, match.groups()[:4])
elif not line.startswith('-'):
delta += 1
if no_plus_line in line and line[0] != '-':
# The model might add a '+' to the beginning of the relevant_line_in_file even if originally
# it's a context line
position = i
absolute_position = start2 + delta - 1
break
return position, absolute_position

View File

@ -17,7 +17,6 @@ For example:
- cli.py --pr_url=... improve
- cli.py --pr_url=... ask "write me a poem about this PR"
- cli.py --pr_url=... reflect
- cli.py --issue_url=... similar_issue
Supported commands:
-review / review_pr - Add a review that includes a summary of the PR and specific suggestions for improvement.
@ -38,22 +37,14 @@ Configuration:
To edit any configuration parameter from 'configuration.toml', just add -config_path=<value>.
For example: 'python cli.py --pr_url=... review --pr_reviewer.extra_instructions="focus on the file: ..."'
""")
parser.add_argument('--pr_url', type=str, help='The URL of the PR to review', default=None)
parser.add_argument('--issue_url', type=str, help='The URL of the Issue to review', default=None)
parser.add_argument('--pr_url', type=str, help='The URL of the PR to review', required=True)
parser.add_argument('command', type=str, help='The', choices=commands, default='review')
parser.add_argument('rest', nargs=argparse.REMAINDER, default=[])
args = parser.parse_args(inargs)
if not args.pr_url and not args.issue_url:
parser.print_help()
return
logging.basicConfig(level=os.environ.get("LOGLEVEL", "INFO"))
command = args.command.lower()
get_settings().set("CONFIG.CLI_MODE", True)
if args.issue_url:
result = asyncio.run(PRAgent().handle_request(args.issue_url, command + " " + " ".join(args.rest)))
else:
result = asyncio.run(PRAgent().handle_request(args.pr_url, command + " " + " ".join(args.rest)))
result = asyncio.run(PRAgent().handle_request(args.pr_url, command + " " + " ".join(args.rest)))
if not result:
parser.print_help()

View File

@ -5,8 +5,6 @@ from pr_agent.git_providers.github_provider import GithubProvider
from pr_agent.git_providers.gitlab_provider import GitLabProvider
from pr_agent.git_providers.local_git_provider import LocalGitProvider
from pr_agent.git_providers.azuredevops_provider import AzureDevopsProvider
from pr_agent.git_providers.gerrit_provider import GerritProvider
_GIT_PROVIDERS = {
'github': GithubProvider,
@ -14,8 +12,7 @@ _GIT_PROVIDERS = {
'bitbucket': BitbucketProvider,
'azure': AzureDevopsProvider,
'codecommit': CodeCommitProvider,
'local' : LocalGitProvider,
'gerrit': GerritProvider,
'local' : LocalGitProvider
}
def get_git_provider():

View File

@ -13,11 +13,9 @@ try:
except ImportError:
AZURE_DEVOPS_AVAILABLE = False
from ..algo.pr_processing import clip_tokens
from ..config_loader import get_settings
from ..algo.utils import load_large_diff
from ..algo.utils import load_large_diff, FilePatchInfo, EDIT_TYPE, clip_tokens
from ..algo.language_handler import is_valid_file
from .git_provider import EDIT_TYPE, FilePatchInfo
class AzureDevopsProvider:
@ -38,8 +36,7 @@ class AzureDevopsProvider:
self.set_pr(pr_url)
def is_supported(self, capability: str) -> bool:
if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels',
'remove_initial_comment', 'gfm_markdown']:
if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels', 'remove_initial_comment']:
return False
return True

View File

@ -7,9 +7,9 @@ import requests
from atlassian.bitbucket import Cloud
from starlette_context import context
from ..algo.pr_processing import clip_tokens, find_line_number_of_relevant_line_in_file
from ..config_loader import get_settings
from .git_provider import FilePatchInfo, GitProvider
from .git_provider import GitProvider
from ..algo.utils import FilePatchInfo
class BitbucketProvider(GitProvider):
@ -36,8 +36,9 @@ class BitbucketProvider(GitProvider):
self.incremental = incremental
if pr_url:
self.set_pr(pr_url)
self.bitbucket_comment_api_url = self.pr._BitbucketBase__data["links"]["comments"]["href"]
self.bitbucket_pull_request_api_url = self.pr._BitbucketBase__data["links"]['self']['href']
self.bitbucket_comment_api_url = self.pr._BitbucketBase__data["links"][
"comments"
]["href"]
def get_repo_settings(self):
try:
@ -101,7 +102,12 @@ class BitbucketProvider(GitProvider):
return False
def is_supported(self, capability: str) -> bool:
if capability in ['get_issue_comments', 'publish_inline_comments', 'get_labels', 'gfm_markdown']:
if capability in [
"get_issue_comments",
"create_inline_comment",
"publish_inline_comments",
"get_labels",
]:
return False
return True
@ -146,30 +152,17 @@ class BitbucketProvider(GitProvider):
except Exception as e:
logging.exception(f"Failed to remove temp comments, error: {e}")
# funtion to create_inline_comment
def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
position, absolute_position = find_line_number_of_relevant_line_in_file(self.get_diff_files(), relevant_file.strip('`'), relevant_line_in_file)
if position == -1:
if get_settings().config.verbosity_level >= 2:
logging.info(f"Could not find position for {relevant_file} {relevant_line_in_file}")
subject_type = "FILE"
else:
subject_type = "LINE"
path = relevant_file.strip()
return dict(body=body, path=path, position=absolute_position) if subject_type == "LINE" else {}
def publish_inline_comment(self, comment: str, from_line: int, file: str):
payload = json.dumps( {
"content": {
"raw": comment,
},
"inline": {
"to": from_line,
"path": file
},
})
def publish_inline_comment(
self, comment: str, from_line: int, to_line: int, file: str
):
payload = json.dumps(
{
"content": {
"raw": comment,
},
"inline": {"to": from_line, "path": file},
}
)
response = requests.request(
"POST", self.bitbucket_comment_api_url, data=payload, headers=self.headers
)
@ -177,7 +170,9 @@ class BitbucketProvider(GitProvider):
def publish_inline_comments(self, comments: list[dict]):
for comment in comments:
self.publish_inline_comment(comment['body'], comment['start_line'], comment['path'])
self.publish_inline_comment(
comment["body"], comment["start_line"], comment["line"], comment["path"]
)
def get_title(self):
return self.pr.title
@ -244,22 +239,16 @@ class BitbucketProvider(GitProvider):
def get_commit_messages(self):
return "" # not implemented yet
# bitbucket does not support labels
def publish_description(self, pr_title: str, description: str):
payload = json.dumps({
"description": description,
"title": pr_title
})
response = requests.request("PUT", self.bitbucket_pull_request_api_url, headers=self.headers, data=payload)
return response
# bitbucket does not support labels
def publish_labels(self, pr_types: list):
def publish_description(self, pr_title: str, pr_body: str):
pass
# bitbucket does not support labels
def create_inline_comment(
self, body: str, relevant_file: str, relevant_line_in_file: str
):
pass
def publish_labels(self, labels):
pass
def get_labels(self):
pass

View File

@ -1,6 +1,9 @@
import boto3
import botocore
try: # Allow this module to be imported without requiring boto3
import boto3
import botocore
except ModuleNotFoundError:
boto3 = None
botocore = None
class CodeCommitDifferencesResponse:
"""
@ -54,22 +57,17 @@ class CodeCommitClient:
def __init__(self):
self.boto_client = None
def is_supported(self, capability: str) -> bool:
if capability in ["gfm_markdown"]:
return False
return True
def _connect_boto_client(self):
try:
self.boto_client = boto3.client("codecommit")
except Exception as e:
raise ValueError(f"Failed to connect to AWS CodeCommit: {e}") from e
raise ValueError(f"Failed to connect to AWS CodeCommit: {e}")
def get_differences(self, repo_name: int, destination_commit: str, source_commit: str):
"""
Get the differences between two commits in CodeCommit.
Args:
Parameters:
- repo_name: Name of the repository
- destination_commit: Commit hash you want to merge into (the "before" hash) (usually on the main or master branch)
- source_commit: Commit hash of the code you are adding (the "after" branch)
@ -78,8 +76,8 @@ class CodeCommitClient:
- List of CodeCommitDifferencesResponse objects
Boto3 Documentation:
- aws codecommit get-differences
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_differences.html
aws codecommit get-differences
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_differences.html
"""
if self.boto_client is None:
self._connect_boto_client()
@ -95,11 +93,7 @@ class CodeCommitClient:
):
differences.extend(page.get("differences", []))
except botocore.exceptions.ClientError as e:
if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
raise ValueError(f"CodeCommit cannot retrieve differences: Repository does not exist: {repo_name}") from e
raise ValueError(f"CodeCommit cannot retrieve differences for {source_commit}..{destination_commit}") from e
except Exception as e:
raise ValueError(f"CodeCommit cannot retrieve differences for {source_commit}..{destination_commit}") from e
raise ValueError(f"Failed to retrieve differences from CodeCommit PR #{self.pr_num}") from e
output = []
for json in differences:
@ -110,7 +104,7 @@ class CodeCommitClient:
"""
Retrieve a file from CodeCommit.
Args:
Parameters:
- repo_name: Name of the repository
- file_path: Path to the file you are retrieving
- sha_hash: Commit hash of the file you are retrieving
@ -119,8 +113,8 @@ class CodeCommitClient:
- File contents
Boto3 Documentation:
- aws codecommit get_file
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_file.html
aws codecommit get_file
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_file.html
"""
if not file_path:
return ""
@ -131,8 +125,6 @@ class CodeCommitClient:
try:
response = self.boto_client.get_file(repositoryName=repo_name, commitSpecifier=sha_hash, filePath=file_path)
except botocore.exceptions.ClientError as e:
if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
raise ValueError(f"CodeCommit cannot retrieve PR: Repository does not exist: {repo_name}") from e
# if the file does not exist, but is flagged as optional, then return an empty string
if optional and e.response["Error"]["Code"] == 'FileDoesNotExistException':
return ""
@ -144,20 +136,19 @@ class CodeCommitClient:
return response.get("fileContent", "")
def get_pr(self, repo_name: str, pr_number: int):
def get_pr(self, pr_number: int):
"""
Get a information about a CodeCommit PR.
Args:
- repo_name: Name of the repository
Parameters:
- pr_number: The PR number you are requesting
Returns:
- CodeCommitPullRequestResponse object
Boto3 Documentation:
- aws codecommit get_pull_request
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_pull_request.html
aws codecommit get_pull_request
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/get_pull_request.html
"""
if self.boto_client is None:
self._connect_boto_client()
@ -167,8 +158,6 @@ class CodeCommitClient:
except botocore.exceptions.ClientError as e:
if e.response["Error"]["Code"] == 'PullRequestDoesNotExistException':
raise ValueError(f"CodeCommit cannot retrieve PR: PR number does not exist: {pr_number}") from e
if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
raise ValueError(f"CodeCommit cannot retrieve PR: Repository does not exist: {repo_name}") from e
raise ValueError(f"CodeCommit cannot retrieve PR: {pr_number}: boto client error") from e
except Exception as e:
raise ValueError(f"CodeCommit cannot retrieve PR: {pr_number}") from e
@ -178,95 +167,35 @@ class CodeCommitClient:
return CodeCommitPullRequestResponse(response.get("pullRequest", {}))
def publish_description(self, pr_number: int, pr_title: str, pr_body: str):
"""
Set the title and description on a pull request
Args:
- pr_number: the AWS CodeCommit pull request number
- pr_title: title of the pull request
- pr_body: body of the pull request
Returns:
- None
Boto3 Documentation:
- aws codecommit update_pull_request_title
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/update_pull_request_title.html
- aws codecommit update_pull_request_description
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/update_pull_request_description.html
"""
if self.boto_client is None:
self._connect_boto_client()
try:
self.boto_client.update_pull_request_title(pullRequestId=str(pr_number), title=pr_title)
self.boto_client.update_pull_request_description(pullRequestId=str(pr_number), description=pr_body)
except botocore.exceptions.ClientError as e:
if e.response["Error"]["Code"] == 'PullRequestDoesNotExistException':
raise ValueError(f"PR number does not exist: {pr_number}") from e
if e.response["Error"]["Code"] == 'InvalidTitleException':
raise ValueError(f"Invalid title for PR number: {pr_number}") from e
if e.response["Error"]["Code"] == 'InvalidDescriptionException':
raise ValueError(f"Invalid description for PR number: {pr_number}") from e
if e.response["Error"]["Code"] == 'PullRequestAlreadyClosedException':
raise ValueError(f"PR is already closed: PR number: {pr_number}") from e
raise ValueError(f"Boto3 client error calling publish_description") from e
except Exception as e:
raise ValueError(f"Error calling publish_description") from e
def publish_comment(self, repo_name: str, pr_number: int, destination_commit: str, source_commit: str, comment: str, annotation_file: str = None, annotation_line: int = None):
def publish_comment(self, repo_name: str, pr_number: int, destination_commit: str, source_commit: str, comment: str):
"""
Publish a comment to a pull request
Args:
Parameters:
- repo_name: name of the repository
- pr_number: number of the pull request
- destination_commit: The commit hash you want to merge into (the "before" hash) (usually on the main or master branch)
- source_commit: The commit hash of the code you are adding (the "after" branch)
- comment: The comment you want to publish
- annotation_file: The file you want to annotate (optional)
- annotation_line: The line number you want to annotate (optional)
Comment annotations for CodeCommit are different than GitHub.
CodeCommit only designates the starting line number for the comment.
It does not support the ending line number to highlight a range of lines.
- pr_comment: comment
Returns:
- None
Boto3 Documentation:
- aws codecommit post_comment_for_pull_request
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_pull_request.html
aws codecommit post_comment_for_pull_request
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_pull_request.html
"""
if self.boto_client is None:
self._connect_boto_client()
try:
# If the comment has code annotations,
# then set the file path and line number in the location dictionary
if annotation_file and annotation_line:
self.boto_client.post_comment_for_pull_request(
pullRequestId=str(pr_number),
repositoryName=repo_name,
beforeCommitId=destination_commit,
afterCommitId=source_commit,
content=comment,
location={
"filePath": annotation_file,
"filePosition": annotation_line,
"relativeFileVersion": "AFTER",
},
)
else:
# The comment does not have code annotations
self.boto_client.post_comment_for_pull_request(
pullRequestId=str(pr_number),
repositoryName=repo_name,
beforeCommitId=destination_commit,
afterCommitId=source_commit,
content=comment,
)
self.boto_client.post_comment_for_pull_request(
pullRequestId=str(pr_number),
repositoryName=repo_name,
beforeCommitId=destination_commit,
afterCommitId=source_commit,
content=comment,
)
except botocore.exceptions.ClientError as e:
if e.response["Error"]["Code"] == 'RepositoryDoesNotExistException':
raise ValueError(f"Repository does not exist: {repo_name}") from e

View File

@ -1,17 +1,15 @@
import logging
import os
import re
from collections import Counter
from typing import List, Optional, Tuple
from urllib.parse import urlparse
from ..algo.language_handler import is_valid_file, language_extension_map
from ..algo.pr_processing import clip_tokens
from ..algo.utils import load_large_diff
from ..config_loader import get_settings
from .git_provider import EDIT_TYPE, FilePatchInfo, GitProvider, IncrementalPR
from pr_agent.git_providers.codecommit_client import CodeCommitClient
from ..algo.language_handler import is_valid_file, language_extension_map
from ..algo.utils import EDIT_TYPE, FilePatchInfo, load_large_diff
from .git_provider import GitProvider
class PullRequestCCMimic:
"""
@ -74,7 +72,6 @@ class CodeCommitProvider(GitProvider):
"create_inline_comment",
"publish_inline_comments",
"get_labels",
"gfm_markdown"
]:
return False
return True
@ -155,63 +152,26 @@ class CodeCommitProvider(GitProvider):
return self.diff_files
def publish_description(self, pr_title: str, pr_body: str):
try:
self.codecommit_client.publish_description(
pr_number=self.pr_num,
pr_title=pr_title,
pr_body=CodeCommitProvider._add_additional_newlines(pr_body),
)
except Exception as e:
raise ValueError(f"CodeCommit Cannot publish description for PR: {self.pr_num}") from e
return "" # not implemented yet
def publish_comment(self, pr_comment: str, is_temporary: bool = False):
if is_temporary:
logging.info(pr_comment)
return
pr_comment = CodeCommitProvider._remove_markdown_html(pr_comment)
pr_comment = CodeCommitProvider._add_additional_newlines(pr_comment)
try:
self.codecommit_client.publish_comment(
repo_name=self.repo_name,
pr_number=self.pr_num,
pr_number=str(self.pr_num),
destination_commit=self.pr.destination_commit,
source_commit=self.pr.source_commit,
comment=pr_comment,
)
except Exception as e:
raise ValueError(f"CodeCommit Cannot publish comment for PR: {self.pr_num}") from e
raise ValueError(f"CodeCommit Cannot post comment for PR: {self.pr_num}") from e
def publish_code_suggestions(self, code_suggestions: list) -> bool:
counter = 1
for suggestion in code_suggestions:
# Verify that each suggestion has the required keys
if not all(key in suggestion for key in ["body", "relevant_file", "relevant_lines_start"]):
logging.warning(f"Skipping code suggestion #{counter}: Each suggestion must have 'body', 'relevant_file', 'relevant_lines_start' keys")
continue
# Publish the code suggestion to CodeCommit
try:
logging.debug(f"Code Suggestion #{counter} in file: {suggestion['relevant_file']}: {suggestion['relevant_lines_start']}")
self.codecommit_client.publish_comment(
repo_name=self.repo_name,
pr_number=self.pr_num,
destination_commit=self.pr.destination_commit,
source_commit=self.pr.source_commit,
comment=suggestion["body"],
annotation_file=suggestion["relevant_file"],
annotation_line=suggestion["relevant_lines_start"],
)
except Exception as e:
raise ValueError(f"CodeCommit Cannot publish code suggestions for PR: {self.pr_num}") from e
counter += 1
# The calling function passes in a list of code suggestions, and this function publishes each suggestion one at a time.
# If we were to return False here, the calling function will attempt to publish the same list of code suggestions again, one at a time.
# Since this function publishes the suggestions one at a time anyway, we always return True here to avoid the retry.
return True
return [""] # not implemented yet
def publish_labels(self, labels):
return [""] # not implemented yet
@ -223,7 +183,6 @@ class CodeCommitProvider(GitProvider):
return "" # not implemented yet
def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit/client/post_comment_for_compared_commit.html
raise NotImplementedError("CodeCommit provider does not support publishing inline comments yet")
def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
@ -233,26 +192,14 @@ class CodeCommitProvider(GitProvider):
raise NotImplementedError("CodeCommit provider does not support publishing inline comments yet")
def get_title(self):
return self.pr.title
return self.pr.get("title", "")
def get_pr_id(self):
"""
Returns the PR ID in the format: "repo_name/pr_number".
Note: This is an internal identifier for PR-Agent,
and is not the same as the CodeCommit PR identifier.
"""
try:
pr_id = f"{self.repo_name}/{self.pr_num}"
return pr_id
except:
return ""
def get_languages(self):
"""
Returns a dictionary of languages, containing the percentage of each language used in the PR.
Returns:
- dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR.
dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR.
"""
commit_files = self.get_files()
filenames = [ item.filename for item in commit_files ]
@ -296,29 +243,18 @@ class CodeCommitProvider(GitProvider):
return self.codecommit_client.get_file(self.repo_name, settings_filename, self.pr.source_commit, optional=True)
def add_eyes_reaction(self, issue_comment_id: int) -> Optional[int]:
logging.info("CodeCommit provider does not support eyes reaction yet")
return True
def remove_reaction(self, issue_comment_id: int, reaction_id: int) -> bool:
logging.info("CodeCommit provider does not support removing reactions yet")
return True
@staticmethod
def _parse_pr_url(pr_url: str) -> Tuple[str, int]:
"""
Parse the CodeCommit PR URL and return the repository name and PR number.
Args:
- pr_url: the full AWS CodeCommit pull request URL
Returns:
- Tuple[str, int]: A tuple containing the repository name and PR number.
"""
# Example PR URL:
# https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/__MY_REPO__/pull-requests/123456"
parsed_url = urlparse(pr_url)
if not CodeCommitProvider._is_valid_codecommit_hostname(parsed_url.netloc):
if "us-east-1.console.aws.amazon.com" not in parsed_url.netloc:
raise ValueError(f"The provided URL is not a valid CodeCommit URL: {pr_url}")
path_parts = parsed_url.path.strip("/").split("/")
@ -341,33 +277,17 @@ class CodeCommitProvider(GitProvider):
return repo_name, pr_number
@staticmethod
def _is_valid_codecommit_hostname(hostname: str) -> bool:
"""
Check if the provided hostname is a valid AWS CodeCommit hostname.
This is not an exhaustive check of AWS region names,
but instead uses a regex to check for matching AWS region patterns.
Args:
- hostname: the hostname to check
Returns:
- bool: True if the hostname is valid, False otherwise.
"""
return re.match(r"^[a-z]{2}-(gov-)?[a-z]+-\d\.console\.aws\.amazon\.com$", hostname) is not None
def _get_pr(self):
response = self.codecommit_client.get_pr(self.repo_name, self.pr_num)
response = self.codecommit_client.get_pr(self.pr_num)
if len(response.targets) == 0:
raise ValueError(f"No files found in CodeCommit PR: {self.pr_num}")
# TODO: implement support for multiple targets in one CodeCommit PR
# for now, we are only using the first target in the PR
# TODO: implement support for multiple commits in one CodeCommit PR
# for now, we are only using the first commit in the PR
if len(response.targets) > 1:
logging.warning(
"Multiple targets in one PR is not supported for CodeCommit yet. Continuing, using the first target only..."
"Multiple commits in one PR is not supported for CodeCommit yet. Continuing, using the first commit only..."
)
# Return our object that mimics PullRequest class from the PyGithub library
@ -385,52 +305,13 @@ class CodeCommitProvider(GitProvider):
return "" # not implemented yet
@staticmethod
def _add_additional_newlines(body: str) -> str:
"""
Replace single newlines in a PR body with double newlines.
CodeCommit Markdown does not seem to render as well as GitHub Markdown,
so we add additional newlines to the PR body to make it more readable in CodeCommit.
Args:
- body: the PR body
Returns:
- str: the PR body with the double newlines added
"""
return re.sub(r'(?<!\n)\n(?!\n)', '\n\n', body)
@staticmethod
def _remove_markdown_html(comment: str) -> str:
"""
Remove the HTML tags from a PR comment.
CodeCommit Markdown does not seem to render as well as GitHub Markdown,
so we remove the HTML tags from the PR comment to make it more readable in CodeCommit.
Args:
- comment: the PR comment
Returns:
- str: the PR comment with the HTML tags removed
"""
comment = comment.replace("<details>", "")
comment = comment.replace("</details>", "")
comment = comment.replace("<summary>", "")
comment = comment.replace("</summary>", "")
return comment
@staticmethod
def _get_edit_type(codecommit_change_type: str):
def _get_edit_type(codecommit_change_type):
"""
Convert the CodeCommit change type string to the EDIT_TYPE enum.
The CodeCommit change type string is returned from the get_differences SDK method.
Args:
- codecommit_change_type: the CodeCommit change type string
Returns:
- An EDIT_TYPE enum representing the modified, added, deleted, or renamed file in the PR diff.
An EDIT_TYPE enum representing the modified, added, deleted, or renamed file in the PR diff.
"""
t = codecommit_change_type.upper()
edit_type = None
@ -451,12 +332,6 @@ class CodeCommitProvider(GitProvider):
The returned extensions will include the dot "." prefix,
to accommodate for the dots in the existing language_extension_map settings.
Filenames with no extension will return an empty string for the extension.
Args:
- filenames: a list of filenames
Returns:
- list: A list of file extensions, including the dot "." prefix.
"""
extensions = []
for filename in filenames:
@ -473,12 +348,6 @@ class CodeCommitProvider(GitProvider):
Return a dictionary containing the programming language name (as the key),
and the percentage that language is used (as the value),
given a list of file extensions.
Args:
- extensions: a list of file extensions
Returns:
- dict: A dictionary where each key is a language name and the corresponding value is the percentage of that language in the PR.
"""
total_files = len(extensions)
if total_files == 0:

View File

@ -1,403 +0,0 @@
import json
import logging
import os
import pathlib
import shutil
import subprocess
import uuid
from collections import Counter, namedtuple
from pathlib import Path
from tempfile import mkdtemp, NamedTemporaryFile
import requests
import urllib3.util
from git import Repo
from pr_agent.config_loader import get_settings
from pr_agent.git_providers.git_provider import GitProvider, FilePatchInfo, \
EDIT_TYPE
from pr_agent.git_providers.local_git_provider import PullRequestMimic
logger = logging.getLogger(__name__)
def _call(*command, **kwargs) -> (int, str, str):
res = subprocess.run(
command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
check=True,
**kwargs,
)
return res.stdout.decode()
def clone(url, directory):
logger.info("Cloning %s to %s", url, directory)
stdout = _call('git', 'clone', "--depth", "1", url, directory)
logger.info(stdout)
def fetch(url, refspec, cwd):
logger.info("Fetching %s %s", url, refspec)
stdout = _call(
'git', 'fetch', '--depth', '2', url, refspec,
cwd=cwd
)
logger.info(stdout)
def checkout(cwd):
logger.info("Checking out")
stdout = _call('git', 'checkout', "FETCH_HEAD", cwd=cwd)
logger.info(stdout)
def show(*args, cwd=None):
logger.info("Show")
return _call('git', 'show', *args, cwd=cwd)
def diff(*args, cwd=None):
logger.info("Diff")
patch = _call('git', 'diff', *args, cwd=cwd)
if not patch:
logger.warning("No changes found")
return
return patch
def reset_local_changes(cwd):
logger.info("Reset local changes")
_call('git', 'checkout', "--force", cwd=cwd)
def add_comment(url: urllib3.util.Url, refspec, message):
*_, patchset, changenum = refspec.rsplit("/")
message = "'" + message.replace("'", "'\"'\"'") + "'"
return _call(
"ssh",
"-p", str(url.port),
f"{url.auth}@{url.host}",
"gerrit", "review",
"--message", message,
# "--code-review", score,
f"{patchset},{changenum}",
)
def list_comments(url: urllib3.util.Url, refspec):
*_, patchset, _ = refspec.rsplit("/")
stdout = _call(
"ssh",
"-p", str(url.port),
f"{url.auth}@{url.host}",
"gerrit", "query",
"--comments",
"--current-patch-set", patchset,
"--format", "JSON",
)
change_set, *_ = stdout.splitlines()
return json.loads(change_set)["currentPatchSet"]["comments"]
def prepare_repo(url: urllib3.util.Url, project, refspec):
repo_url = (f"{url.scheme}://{url.auth}@{url.host}:{url.port}/{project}")
directory = pathlib.Path(mkdtemp())
clone(repo_url, directory),
fetch(repo_url, refspec, cwd=directory)
checkout(cwd=directory)
return directory
def adopt_to_gerrit_message(message):
lines = message.splitlines()
buf = []
for line in lines:
# remove markdown formatting
line = (line.replace("*", "")
.replace("``", "`")
.replace("<details>", "")
.replace("</details>", "")
.replace("<summary>", "")
.replace("</summary>", ""))
line = line.strip()
if line.startswith('#'):
buf.append("\n" +
line.replace('#', '').removesuffix(":").strip() +
":")
continue
elif line.startswith('-'):
buf.append(line.removeprefix('-').strip())
continue
else:
buf.append(line)
return "\n".join(buf).strip()
def add_suggestion(src_filename, context: str, start, end: int):
with (
NamedTemporaryFile("w", delete=False) as tmp,
open(src_filename, "r") as src
):
lines = src.readlines()
tmp.writelines(lines[:start - 1])
if context:
tmp.write(context)
tmp.writelines(lines[end:])
shutil.copy(tmp.name, src_filename)
os.remove(tmp.name)
def upload_patch(patch, path):
patch_server_endpoint = get_settings().get(
'gerrit.patch_server_endpoint')
patch_server_token = get_settings().get(
'gerrit.patch_server_token')
response = requests.post(
patch_server_endpoint,
json={
"content": patch,
"path": path,
},
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {patch_server_token}",
}
)
response.raise_for_status()
patch_server_endpoint = patch_server_endpoint.rstrip("/")
return patch_server_endpoint + "/" + path
class GerritProvider(GitProvider):
def __init__(self, key: str, incremental=False):
self.project, self.refspec = key.split(':')
assert self.project, "Project name is required"
assert self.refspec, "Refspec is required"
base_url = get_settings().get('gerrit.url')
assert base_url, "Gerrit URL is required"
user = get_settings().get('gerrit.user')
assert user, "Gerrit user is required"
parsed = urllib3.util.parse_url(base_url)
self.parsed_url = urllib3.util.parse_url(
f"{parsed.scheme}://{user}@{parsed.host}:{parsed.port}"
)
self.repo_path = prepare_repo(
self.parsed_url, self.project, self.refspec
)
self.repo = Repo(self.repo_path)
assert self.repo
self.pr = PullRequestMimic(self.get_pr_title(), self.get_diff_files())
def get_pr_title(self):
"""
Substitutes the branch-name as the PR-mimic title.
"""
return self.repo.branches[0].name
def get_issue_comments(self):
comments = list_comments(self.parsed_url, self.refspec)
Comments = namedtuple('Comments', ['reversed'])
Comment = namedtuple('Comment', ['body'])
return Comments([Comment(c['message']) for c in reversed(comments)])
def get_labels(self):
raise NotImplementedError(
'Getting labels is not implemented for the gerrit provider')
def add_eyes_reaction(self, issue_comment_id: int):
raise NotImplementedError(
'Adding reactions is not implemented for the gerrit provider')
def remove_reaction(self, issue_comment_id: int, reaction_id: int):
raise NotImplementedError(
'Removing reactions is not implemented for the gerrit provider')
def get_commit_messages(self):
return [self.repo.head.commit.message]
def get_repo_settings(self):
try:
with open(self.repo_path / ".pr_agent.toml", 'rb') as f:
contents = f.read()
return contents
except OSError:
return b""
def get_diff_files(self) -> list[FilePatchInfo]:
diffs = self.repo.head.commit.diff(
self.repo.head.commit.parents[0], # previous commit
create_patch=True,
R=True
)
diff_files = []
for diff_item in diffs:
if diff_item.a_blob is not None:
original_file_content_str = (
diff_item.a_blob.data_stream.read().decode('utf-8')
)
else:
original_file_content_str = "" # empty file
if diff_item.b_blob is not None:
new_file_content_str = diff_item.b_blob.data_stream.read(). \
decode('utf-8')
else:
new_file_content_str = "" # empty file
edit_type = EDIT_TYPE.MODIFIED
if diff_item.new_file:
edit_type = EDIT_TYPE.ADDED
elif diff_item.deleted_file:
edit_type = EDIT_TYPE.DELETED
elif diff_item.renamed_file:
edit_type = EDIT_TYPE.RENAMED
diff_files.append(
FilePatchInfo(
original_file_content_str,
new_file_content_str,
diff_item.diff.decode('utf-8'),
diff_item.b_path,
edit_type=edit_type,
old_filename=None
if diff_item.a_path == diff_item.b_path
else diff_item.a_path
)
)
self.diff_files = diff_files
return diff_files
def get_files(self):
diff_index = self.repo.head.commit.diff(
self.repo.head.commit.parents[0], # previous commit
R=True
)
# Get the list of changed files
diff_files = [item.a_path for item in diff_index]
return diff_files
def get_languages(self):
"""
Calculate percentage of languages in repository. Used for hunk
prioritisation.
"""
# Get all files in repository
filepaths = [Path(item.path) for item in
self.repo.tree().traverse() if item.type == 'blob']
# Identify language by file extension and count
lang_count = Counter(
ext.lstrip('.') for filepath in filepaths for ext in
[filepath.suffix.lower()])
# Convert counts to percentages
total_files = len(filepaths)
lang_percentage = {lang: count / total_files * 100 for lang, count
in lang_count.items()}
return lang_percentage
def get_pr_description_full(self):
return self.repo.head.commit.message
def get_user_id(self):
return self.repo.head.commit.author.email
def is_supported(self, capability: str) -> bool:
if capability in [
# 'get_issue_comments',
'create_inline_comment',
'publish_inline_comments',
'get_labels',
'gfm_markdown'
]:
return False
return True
def split_suggestion(self, msg) -> tuple[str, str]:
is_code_context = False
description = []
context = []
for line in msg.splitlines():
if line.startswith('```suggestion'):
is_code_context = True
continue
if line.startswith('```'):
is_code_context = False
continue
if is_code_context:
context.append(line)
else:
description.append(
line.replace('*', '')
)
return (
'\n'.join(description),
'\n'.join(context) + '\n' if context else ''
)
def publish_code_suggestions(self, code_suggestions: list):
msg = []
for suggestion in code_suggestions:
description, code = self.split_suggestion(suggestion['body'])
add_suggestion(
pathlib.Path(self.repo_path) / suggestion["relevant_file"],
code,
suggestion["relevant_lines_start"],
suggestion["relevant_lines_end"],
)
patch = diff(cwd=self.repo_path)
patch_id = uuid.uuid4().hex[0:4]
path = "/".join(["codium-ai", self.refspec, patch_id])
full_path = upload_patch(patch, path)
reset_local_changes(self.repo_path)
msg.append(f'* {description}\n{full_path}')
if msg:
add_comment(self.parsed_url, self.refspec, "\n".join(msg))
return True
def publish_comment(self, pr_comment: str, is_temporary: bool = False):
if not is_temporary:
msg = adopt_to_gerrit_message(pr_comment)
add_comment(self.parsed_url, self.refspec, msg)
def publish_description(self, pr_title: str, pr_body: str):
msg = adopt_to_gerrit_message(pr_body)
add_comment(self.parsed_url, self.refspec, pr_title + '\n' + msg)
def publish_inline_comments(self, comments: list[dict]):
raise NotImplementedError(
'Publishing inline comments is not implemented for the gerrit '
'provider')
def publish_inline_comment(self, body: str, relevant_file: str,
relevant_line_in_file: str):
raise NotImplementedError(
'Publishing inline comments is not implemented for the gerrit '
'provider')
def create_inline_comment(self, body: str, relevant_file: str,
relevant_line_in_file: str):
raise NotImplementedError(
'Creating inline comments is not implemented for the gerrit '
'provider')
def publish_labels(self, labels):
# Not applicable to the local git provider,
# but required by the interface
pass
def remove_initial_comment(self):
# remove repo, cloned in previous steps
# shutil.rmtree(self.repo_path)
pass
def get_pr_branch(self):
return self.repo.head

View File

@ -1,28 +1,10 @@
import logging
from abc import ABC, abstractmethod
from dataclasses import dataclass
# enum EDIT_TYPE (ADDED, DELETED, MODIFIED, RENAMED)
from enum import Enum
from typing import Optional
class EDIT_TYPE(Enum):
ADDED = 1
DELETED = 2
MODIFIED = 3
RENAMED = 4
@dataclass
class FilePatchInfo:
base_file: str
head_file: str
patch: str
filename: str
tokens: int = -1
edit_type: EDIT_TYPE = EDIT_TYPE.MODIFIED
old_filename: str = None
from pr_agent.algo.utils import FilePatchInfo
class GitProvider(ABC):
@ -86,11 +68,11 @@ class GitProvider(ABC):
def get_pr_description_full(self) -> str:
pass
def get_pr_description(self, *, full: bool = True) -> str:
def get_pr_description(self) -> str:
from pr_agent.config_loader import get_settings
from pr_agent.algo.pr_processing import clip_tokens
from pr_agent.algo.utils import clip_tokens
max_tokens = get_settings().get("CONFIG.MAX_DESCRIPTION_TOKENS", None)
description = self.get_pr_description_full() if full else self.get_user_description()
description = self.get_pr_description_full()
if max_tokens:
return clip_tokens(description, max_tokens)
return description
@ -127,18 +109,11 @@ class GitProvider(ABC):
def get_commit_messages(self):
pass
def get_pr_id(self):
return ""
def get_main_pr_language(languages, files) -> str:
"""
Get the main language of the commit. Return an empty string if cannot determine.
"""
main_language_str = ""
if not languages:
logging.info("No languages detected")
return main_language_str
try:
top_language = max(languages, key=languages.get).lower()
@ -168,6 +143,7 @@ def get_main_pr_language(languages, files) -> str:
most_common_extension == 'scala' and top_language == 'scala' or \
most_common_extension == 'kt' and top_language == 'kotlin' or \
most_common_extension == 'pl' and top_language == 'perl' or \
most_common_extension == 'swift' and top_language == 'swift' or \
most_common_extension == top_language:
main_language_str = top_language

View File

@ -9,10 +9,9 @@ from github import AppAuthentication, Auth, Github, GithubException, Reaction
from retry import retry
from starlette_context import context
from .git_provider import FilePatchInfo, GitProvider, IncrementalPR
from .git_provider import GitProvider, IncrementalPR
from ..algo.language_handler import is_valid_file
from ..algo.utils import load_large_diff
from ..algo.pr_processing import find_line_number_of_relevant_line_in_file, clip_tokens
from ..algo.utils import load_large_diff, clip_tokens, find_line_number_of_relevant_line_in_file, FilePatchInfo
from ..config_loader import get_settings
from ..servers.utils import RateLimitExceeded
@ -32,7 +31,7 @@ class GithubProvider(GitProvider):
self.diff_files = None
self.git_files = None
self.incremental = incremental
if pr_url and 'pull' in pr_url:
if pr_url:
self.set_pr(pr_url)
self.last_commit_id = list(self.pr.get_commits())[-1]
@ -239,10 +238,9 @@ class GithubProvider(GitProvider):
def get_user_id(self):
if not self.github_user_id:
try:
self.github_user_id = self.github_client.get_user().raw_data['login']
self.github_user_id = self.github_client.get_user().login
except Exception as e:
self.github_user_id = ""
# logging.exception(f"Failed to get user id, error: {e}")
logging.exception(f"Failed to get user id, error: {e}")
return self.github_user_id
def get_notifications(self, since: datetime):
@ -310,35 +308,6 @@ class GithubProvider(GitProvider):
return repo_name, pr_number
@staticmethod
def _parse_issue_url(issue_url: str) -> Tuple[str, int]:
parsed_url = urlparse(issue_url)
if 'github.com' not in parsed_url.netloc:
raise ValueError("The provided URL is not a valid GitHub URL")
path_parts = parsed_url.path.strip('/').split('/')
if 'api.github.com' in parsed_url.netloc:
if len(path_parts) < 5 or path_parts[3] != 'issues':
raise ValueError("The provided URL does not appear to be a GitHub ISSUE URL")
repo_name = '/'.join(path_parts[1:3])
try:
issue_number = int(path_parts[4])
except ValueError as e:
raise ValueError("Unable to convert issue number to integer") from e
return repo_name, issue_number
if len(path_parts) < 4 or path_parts[2] != 'issues':
raise ValueError("The provided URL does not appear to be a GitHub PR issue")
repo_name = '/'.join(path_parts[:2])
try:
issue_number = int(path_parts[3])
except ValueError as e:
raise ValueError("Unable to convert issue number to integer") from e
return repo_name, issue_number
def _get_github_client(self):
deployment_type = get_settings().get("GITHUB.DEPLOYMENT_TYPE", "user")
@ -447,10 +416,3 @@ class GithubProvider(GitProvider):
logging.info(f"Failed adding line link, error: {e}")
return ""
def get_pr_id(self):
try:
pr_id = f"{self.repo}/{self.pr_num}"
return pr_id
except:
return ""

View File

@ -7,10 +7,9 @@ import gitlab
from gitlab import GitlabGetError
from ..algo.language_handler import is_valid_file
from ..algo.pr_processing import clip_tokens
from ..algo.utils import load_large_diff
from ..algo.utils import load_large_diff, clip_tokens, EDIT_TYPE, FilePatchInfo
from ..config_loader import get_settings
from .git_provider import EDIT_TYPE, FilePatchInfo, GitProvider
from .git_provider import GitProvider
logger = logging.getLogger()
@ -43,7 +42,7 @@ class GitLabProvider(GitProvider):
self.incremental = incremental
def is_supported(self, capability: str) -> bool:
if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'gfm_markdown']:
if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments']:
return False
return True
@ -378,11 +377,4 @@ class GitLabProvider(GitProvider):
commit_messages_str = ""
if max_tokens:
commit_messages_str = clip_tokens(commit_messages_str, max_tokens)
return commit_messages_str
def get_pr_id(self):
try:
pr_id = self.mr.web_url
return pr_id
except:
return ""
return commit_messages_str

View File

@ -0,0 +1,79 @@
import itertools
from collections import Counter
from typing import List, Optional
from pr_agent.algo.utils import FilePatchInfo
from pr_agent.git_providers.git_provider import GitProvider
class InMemoryProvider(GitProvider):
def __init__(self, head_branch: str, target_branch: str, files: List[FilePatchInfo]):
self.head_branch = head_branch
self.target_branch = target_branch
self.files = files
def is_supported(self, capability: str) -> bool:
pass
def get_files(self) -> list[FilePatchInfo]:
return self.files
def get_diff_files(self) -> list[FilePatchInfo]:
return self.get_files()
def publish_description(self, pr_title: str, pr_body: str):
pass
def publish_comment(self, pr_comment: str, is_temporary: bool = False):
pass
def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
pass
def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
pass
def publish_inline_comments(self, comments: list[dict]):
pass
def publish_code_suggestions(self, code_suggestions: list) -> bool:
pass
def publish_labels(self, labels):
pass
def get_labels(self):
pass
def remove_initial_comment(self):
pass
def get_languages(self):
language_count = Counter(file.language for file in self.files)
return dict(language_count)
def get_pr_branch(self):
pass
def get_user_id(self):
pass
def get_pr_description_full(self) -> str:
pass
def get_issue_comments(self):
pass
def get_repo_settings(self):
pass
def add_eyes_reaction(self, issue_comment_id: int) -> Optional[int]:
pass
def remove_reaction(self, issue_comment_id: int, reaction_id: int) -> bool:
pass
def get_commit_messages(self):
pass

View File

@ -6,7 +6,8 @@ from typing import List
from git import Repo
from pr_agent.config_loader import _find_repository_root, get_settings
from pr_agent.git_providers.git_provider import EDIT_TYPE, FilePatchInfo, GitProvider
from pr_agent.git_providers.git_provider import GitProvider
from pr_agent.algo.utils import EDIT_TYPE, FilePatchInfo
class PullRequestMimic:
@ -56,8 +57,7 @@ class LocalGitProvider(GitProvider):
raise KeyError(f'Branch: {self.target_branch_name} does not exist')
def is_supported(self, capability: str) -> bool:
if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels',
'gfm_markdown']:
if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments', 'get_labels']:
return False
return True

View File

@ -1,34 +0,0 @@
import os
from pr_agent.agent.pr_agent import PRAgent
from pr_agent.config_loader import get_settings
from pr_agent.tools.pr_reviewer import PRReviewer
import asyncio
async def run_action():
try:
pull_request_id = os.environ.get("BITBUCKET_PR_ID", '')
slug = os.environ.get("BITBUCKET_REPO_SLUG", '')
workspace = os.environ.get("BITBUCKET_WORKSPACE", '')
bearer_token = os.environ.get('BITBUCKET_BEARER_TOKEN', None)
OPENAI_KEY = os.environ.get('OPENAI_API_KEY') or os.environ.get('OPENAI.KEY')
OPENAI_ORG = os.environ.get('OPENAI_ORG') or os.environ.get('OPENAI.ORG')
# Check if required environment variables are set
if not bearer_token:
print("BITBUCKET_BEARER_TOKEN not set")
return
if not OPENAI_KEY:
print("OPENAI_KEY not set")
return
# Set the environment variables in the settings
get_settings().set("BITBUCKET.BEARER_TOKEN", bearer_token)
get_settings().set("OPENAI.KEY", OPENAI_KEY)
if OPENAI_ORG:
get_settings().set("OPENAI.ORG", OPENAI_ORG)
if pull_request_id and slug and workspace:
pr_url = f"https://bitbucket.org/{workspace}/{slug}/pull-requests/{pull_request_id}"
await PRReviewer(pr_url).run()
except Exception as e:
print(f"An error occurred: {e}")
if __name__ == "__main__":
asyncio.run(run_action())

View File

@ -1,78 +0,0 @@
import copy
import logging
import sys
from enum import Enum
from json import JSONDecodeError
import uvicorn
from fastapi import APIRouter, FastAPI, HTTPException
from pydantic import BaseModel
from starlette.middleware import Middleware
from starlette_context import context
from starlette_context.middleware import RawContextMiddleware
from pr_agent.agent.pr_agent import PRAgent
from pr_agent.config_loader import global_settings, get_settings
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
router = APIRouter()
class Action(str, Enum):
review = "review"
describe = "describe"
ask = "ask"
improve = "improve"
reflect = "reflect"
answer = "answer"
class Item(BaseModel):
refspec: str
project: str
msg: str
@router.post("/api/v1/gerrit/{action}")
async def handle_gerrit_request(action: Action, item: Item):
logging.debug("Received a Gerrit request")
context["settings"] = copy.deepcopy(global_settings)
if action == Action.ask:
if not item.msg:
return HTTPException(
status_code=400,
detail="msg is required for ask command"
)
await PRAgent().handle_request(
f"{item.project}:{item.refspec}",
f"/{item.msg.strip()}"
)
async def get_body(request):
try:
body = await request.json()
except JSONDecodeError as e:
logging.error("Error parsing request body", e)
return {}
return body
@router.get("/")
async def root():
return {"status": "ok"}
def start():
# to prevent adding help messages with the output
get_settings().set("CONFIG.CLI_MODE", True)
middleware = [Middleware(RawContextMiddleware)]
app = FastAPI(middleware=middleware)
app.include_router(router)
uvicorn.run(app, host="0.0.0.0", port=3000)
if __name__ == '__main__':
start()

View File

@ -5,8 +5,6 @@ import os
from pr_agent.agent.pr_agent import PRAgent
from pr_agent.config_loader import get_settings
from pr_agent.git_providers import get_git_provider
from pr_agent.tools.pr_code_suggestions import PRCodeSuggestions
from pr_agent.tools.pr_description import PRDescription
from pr_agent.tools.pr_reviewer import PRReviewer
@ -14,8 +12,8 @@ async def run_action():
# Get environment variables
GITHUB_EVENT_NAME = os.environ.get('GITHUB_EVENT_NAME')
GITHUB_EVENT_PATH = os.environ.get('GITHUB_EVENT_PATH')
OPENAI_KEY = os.environ.get('OPENAI_KEY') or os.environ.get('OPENAI.KEY')
OPENAI_ORG = os.environ.get('OPENAI_ORG') or os.environ.get('OPENAI.ORG')
OPENAI_KEY = os.environ.get('OPENAI_KEY')
OPENAI_ORG = os.environ.get('OPENAI_ORG')
GITHUB_TOKEN = os.environ.get('GITHUB_TOKEN')
get_settings().set("CONFIG.PUBLISH_OUTPUT_PROGRESS", False)
@ -55,15 +53,7 @@ async def run_action():
if action in ["opened", "reopened"]:
pr_url = event_payload.get("pull_request", {}).get("url")
if pr_url:
auto_review = os.environ.get('github_action.auto_review', None)
if auto_review is None or (isinstance(auto_review, str) and auto_review.lower() == 'true'):
await PRReviewer(pr_url).run()
auto_describe = os.environ.get('github_action.auto_describe', None)
if isinstance(auto_describe, str) and auto_describe.lower() == 'true':
await PRDescription(pr_url).run()
auto_improve = os.environ.get('github_action.auto_improve', None)
if isinstance(auto_improve, str) and auto_improve.lower() == 'true':
await PRCodeSuggestions(pr_url).run()
await PRReviewer(pr_url).run()
# Handle issue comment event
elif GITHUB_EVENT_NAME == "issue_comment":
@ -71,21 +61,12 @@ async def run_action():
if action in ["created", "edited"]:
comment_body = event_payload.get("comment", {}).get("body")
if comment_body:
is_pr = False
# check if issue is pull request
if event_payload.get("issue", {}).get("pull_request"):
url = event_payload.get("issue", {}).get("pull_request", {}).get("url")
is_pr = True
else:
url = event_payload.get("issue", {}).get("url")
if url:
pr_url = event_payload.get("issue", {}).get("pull_request", {}).get("url")
if pr_url:
body = comment_body.strip().lower()
comment_id = event_payload.get("comment", {}).get("id")
provider = get_git_provider()(pr_url=url)
if is_pr:
await PRAgent().handle_request(url, body, notify=lambda: provider.add_eyes_reaction(comment_id))
else:
await PRAgent().handle_request(url, body)
provider = get_git_provider()(pr_url=pr_url)
await PRAgent().handle_request(pr_url, body, notify=lambda: provider.add_eyes_reaction(comment_id))
if __name__ == '__main__':

View File

@ -12,7 +12,6 @@ from starlette_context import context
from starlette_context.middleware import RawContextMiddleware
from pr_agent.agent.pr_agent import PRAgent
from pr_agent.algo.utils import update_settings_from_args
from pr_agent.config_loader import get_settings, global_settings
from pr_agent.git_providers import get_git_provider
from pr_agent.servers.utils import verify_signature
@ -98,7 +97,6 @@ async def handle_request(body: Dict[str, Any], event: str):
api_url = body["comment"]["pull_request_url"]
else:
return {}
logging.info(body)
logging.info(f"Handling comment because of event={event} and action={action}")
comment_id = body.get("comment", {}).get("id")
provider = get_git_provider()(pr_url=api_url)
@ -125,14 +123,8 @@ async def handle_request(body: Dict[str, Any], event: str):
return {}
logging.info(f"Performing review because of event={event} and action={action}")
for command in get_settings().github_app.pr_commands:
split_command = command.split(" ")
command = split_command[0]
args = split_command[1:]
other_args = update_settings_from_args(args)
new_command = ' '.join([command] + other_args)
logging.info(body)
logging.info(f"Performing command: {new_command}")
await agent.handle_request(api_url, new_command)
logging.info(f"Performing command: {command}")
await agent.handle_request(api_url, command)
logging.info("event or action does not require handling")
return {}

View File

@ -1,51 +1,21 @@
import copy
import json
import logging
import sys
import uvicorn
from fastapi import APIRouter, FastAPI, Request, status
from fastapi.encoders import jsonable_encoder
from fastapi.responses import JSONResponse
from starlette.background import BackgroundTasks
from starlette.middleware import Middleware
from starlette_context import context
from starlette_context.middleware import RawContextMiddleware
from pr_agent.agent.pr_agent import PRAgent
from pr_agent.config_loader import get_settings, global_settings
from pr_agent.secret_providers import get_secret_provider
from pr_agent.config_loader import get_settings
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
app = FastAPI()
router = APIRouter()
secret_provider = get_secret_provider() if get_settings().get("CONFIG.SECRET_PROVIDER") else None
@router.post("/webhook")
async def gitlab_webhook(background_tasks: BackgroundTasks, request: Request):
if request.headers.get("X-Gitlab-Token") and secret_provider:
request_token = request.headers.get("X-Gitlab-Token")
secret = secret_provider.get_secret(request_token)
try:
secret_dict = json.loads(secret)
gitlab_token = secret_dict["gitlab_token"]
context["settings"] = copy.deepcopy(global_settings)
context["settings"].gitlab.personal_access_token = gitlab_token
except Exception as e:
logging.error(f"Failed to validate secret {request_token}: {e}")
return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
elif get_settings().get("GITLAB.SHARED_SECRET"):
secret = get_settings().get("GITLAB.SHARED_SECRET")
if not request.headers.get("X-Gitlab-Token") == secret:
return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
else:
return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
gitlab_token = get_settings().get("GITLAB.PERSONAL_ACCESS_TOKEN", None)
if not gitlab_token:
return JSONResponse(status_code=status.HTTP_401_UNAUTHORIZED, content=jsonable_encoder({"message": "unauthorized"}))
data = await request.json()
logging.info(json.dumps(data))
if data.get('object_kind') == 'merge_request' and data['object_attributes'].get('action') in ['open', 'reopen']:
logging.info(f"A merge request has been opened: {data['object_attributes'].get('title')}")
url = data['object_attributes'].get('url')
@ -58,18 +28,16 @@ async def gitlab_webhook(background_tasks: BackgroundTasks, request: Request):
background_tasks.add_task(PRAgent().handle_request, url, body)
return JSONResponse(status_code=status.HTTP_200_OK, content=jsonable_encoder({"message": "success"}))
@router.get("/")
async def root():
return {"status": "ok"}
def start():
gitlab_url = get_settings().get("GITLAB.URL", None)
if not gitlab_url:
raise ValueError("GITLAB.URL is not set")
gitlab_token = get_settings().get("GITLAB.PERSONAL_ACCESS_TOKEN", None)
if not gitlab_token:
raise ValueError("GITLAB.PERSONAL_ACCESS_TOKEN is not set")
get_settings().config.git_provider = "gitlab"
middleware = [Middleware(RawContextMiddleware)]
app = FastAPI(middleware=middleware)
app = FastAPI()
app.include_router(router)
uvicorn.run(app, host="0.0.0.0", port=3000)

View File

@ -16,10 +16,6 @@ key = "" # Acquire through https://platform.openai.com
#deployment_id = "" # The deployment name you chose when you deployed the engine
#fallback_deployments = [] # For each fallback model specified in configuration.toml in the [config] section, specify the appropriate deployment_id
[pinecone]
api_key = "..."
environment = "gcp-starter"
[anthropic]
key = "" # Optional, uncomment if you want to use Anthropic. Acquire through https://www.anthropic.com/
@ -28,14 +24,6 @@ key = "" # Optional, uncomment if you want to use Cohere. Acquire through https:
[replicate]
key = "" # Optional, uncomment if you want to use Replicate. Acquire through https://replicate.com/
[huggingface]
key = "" # Optional, uncomment if you want to use Huggingface Inference API. Acquire through https://huggingface.co/docs/api-inference/quicktour
api_base = "" # the base url for your huggingface inference endpoint
[ollama]
api_base = "" # the base url for your huggingface inference endpoint
[github]
# ---- Set the following only for deployment type == "user"
user_token = "" # A GitHub personal access token with 'repo' scope.
@ -55,12 +43,5 @@ webhook_secret = "<WEBHOOK SECRET>" # Optional, may be commented out.
personal_access_token = ""
[bitbucket]
# For Bitbucket personal/repository bearer token
# Bitbucket personal bearer token
bearer_token = ""
# For Bitbucket app
app_key = ""
base_url = ""
[litellm]
LITELLM_TOKEN = "" # see https://docs.litellm.ai/docs/debugging/hosted_debugging for details and instructions on how to get a token

View File

@ -10,15 +10,14 @@ use_repo_settings_file=true
ai_timeout=180
max_description_tokens = 500
max_commits_tokens = 500
litellm_debugger=false
secret_provider="google_cloud_storage"
cli_mode=false
[pr_reviewer] # /review #
require_focused_review=false
require_score_review=false
require_tests_review=true
require_security_review=true
require_estimate_effort_to_review=true
num_code_suggestions=4
inline_code_comments = false
ask_and_reflect=false
@ -26,14 +25,10 @@ automatic_review=true
extra_instructions = ""
[pr_description] # /describe #
publish_labels=true
publish_description_as_comment=false
add_original_user_description=false
keep_original_user_title=false
extra_instructions = ""
# markers
use_description_markers=false
include_generated_by_header=true
[pr_questions] # /ask #
@ -58,11 +53,6 @@ extra_instructions = ""
deployment_type = "user"
ratelimit_retries = 5
[github_action]
# auto_review = true # set as env var in .github/workflows/pr-agent.yaml
# auto_describe = true # set as env var in .github/workflows/pr-agent.yaml
# auto_improve = true # set as env var in .github/workflows/pr-agent.yaml
[github_app]
# these toggles allows running the github app from custom deployments
bot_user = "github-actions[bot]"
@ -94,27 +84,4 @@ polling_interval_seconds = 30
[local]
# LocalGitProvider settings - uncomment to use paths other than default
# description_path= "path/to/description.md"
# review_path= "path/to/review.md"
[gerrit]
# endpoint to the gerrit service
# url = "ssh://gerrit.example.com:29418"
# user for gerrit authentication
# user = "ai-reviewer"
# patch server where patches will be saved
# patch_server_endpoint = "http://127.0.0.1:5000/patch"
# token to authenticate in the patch server
# patch_server_token = ""
[litellm]
#use_client = false
[pr_similar_issue]
skip_comments = false
force_update_dataset = false
max_issues_to_scan = 500
[pinecone]
# fill and place in .secrets.toml
#api_key = ...
# environment = "gcp-starter"
# review_path= "path/to/review.md"

View File

@ -34,7 +34,7 @@ Specific instructions:
- Provide up to {{ num_code_suggestions }} code suggestions.
- Prioritize suggestions that address major problems, issues and bugs in the code.
As a second priority, suggestions should focus on best practices, code readability, maintainability, enhancments, performance, and other aspects.
Don't suggest to add docstring, type hints, or comments.
Don't suggest to add docstring or type hints.
Try to provide diverse and insightful suggestions.
- Suggestions should refer only to code from the '__new hunk__' sections, and focus on new lines of code (lines starting with '+').
Avoid making suggestions that have already been implemented in the PR code. For example, if you want to add logs, or change a variable to const, or anything else, make sure it isn't already in the '__new hunk__' code.
@ -68,17 +68,12 @@ Code suggestions:
type: string
description: |-
a code snippet showing the relevant code lines from a '__new hunk__' section.
It must be contiguous, correctly formatted and indented, and without line numbers.
relevant lines start:
type: integer
It must be continuous, correctly formatted and indented, and without line numbers.
relevant lines:
type: string
description: |-
The relevant line number from a '__new hunk__' section where the suggestion starts (inclusive).
Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above.
relevant lines end:
type: integer
description: |-
The relevant line number from a '__new hunk__' section where the suggestion ends (inclusive).
Should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above.
the relevant lines from a '__new hunk__' section, in the format of 'start_line-end_line'.
For example: '10-15'. They should be derived from the hunk line numbers, and correspond to the 'existing code' snippet above.
improved code:
type: string
description: |-
@ -95,8 +90,7 @@ Code suggestions:
Add a docstring to func1()
existing code: |-
def func1():
relevant lines start: 12
relevant lines end: 12
relevant lines: '12-12'
improved code: |-
...
```

View File

@ -22,13 +22,13 @@ code line that already existed in the file....
...
'
The review should focus on new code added in the PR (lines starting with '+'), and not on code that already existed in the file (lines starting with '-', or without prefix).
Thre review should focus on new code added in the PR (lines starting with '+'), and not on code that already existed in the file (lines starting with '-', or without prefix).
{%- if num_code_suggestions > 0 %}
- Provide up to {{ num_code_suggestions }} code suggestions.
- Focus on important suggestions like fixing code problems, issues and bugs. As a second priority, provide suggestions for meaningful code improvements, like performance, vulnerability, modularity, and best practices.
- Avoid making suggestions that have already been implemented in the PR code. For example, if you want to add logs, or change a variable to const, or anything else, make sure it isn't already in the PR code.
- Don't suggest to add docstring, type hints, or comments.
- Don't suggest to add docstring or type hints.
- Suggestions should focus on improving the new code added in the PR (lines starting with '+')
{%- endif %}
@ -85,14 +85,6 @@ PR Analysis:
code diff changes are too scattered, then the PR is not focused. Explain
your answer shortly.
{%- endif %}
{%- if require_estimate_effort_to_review %}
Estimated effort to review [1-5]:
type: string
description: >-
Estimate, on a scale of 1-5 (inclusive), the time and effort required to review this PR by an experienced and knowledgeable developer. 1 means short and easy review , 5 means long and hard review.
Take into account the size, complexity, quality, and the needed changes of the PR code diff.
Explain your answer shortly (1-2 sentences).
{%- endif %}
PR Feedback:
General suggestions:
type: string

View File

@ -22,10 +22,7 @@ class PRCodeSuggestions:
)
# extended mode
try:
self.is_extended = any(["extended" in arg for arg in args])
except:
self.is_extended = False
self.is_extended = any(["extended" in arg for arg in args])
if self.is_extended:
num_code_suggestions = get_settings().pr_code_suggestions.num_code_suggestions_per_chunk
else:
@ -45,39 +42,31 @@ class PRCodeSuggestions:
"extra_instructions": get_settings().pr_code_suggestions.extra_instructions,
"commit_messages_str": self.git_provider.get_commit_messages(),
}
self.token_handler = TokenHandler(self.git_provider.pr,
self.vars,
get_settings().pr_code_suggestions_prompt.system,
self.token_handler = TokenHandler(self.vars, get_settings().pr_code_suggestions_prompt.system,
get_settings().pr_code_suggestions_prompt.user)
async def run(self):
try:
logging.info('Generating code suggestions for PR...')
if get_settings().config.publish_output:
self.git_provider.publish_comment("Preparing review...", is_temporary=True)
logging.info('Generating code suggestions for PR...')
if get_settings().config.publish_output:
self.git_provider.publish_comment("Preparing review...", is_temporary=True)
logging.info('Preparing PR review...')
if not self.is_extended:
await retry_with_fallback_models(self._prepare_prediction)
data = self._prepare_pr_code_suggestions()
else:
data = await retry_with_fallback_models(self._prepare_prediction_extended)
if (not data) or (not 'Code suggestions' in data):
logging.info('No code suggestions found for PR.')
return
logging.info('Preparing PR review...')
if not self.is_extended:
await retry_with_fallback_models(self._prepare_prediction)
data = self._prepare_pr_code_suggestions()
else:
data = await retry_with_fallback_models(self._prepare_prediction_extended)
if (not self.is_extended and get_settings().pr_code_suggestions.rank_suggestions) or \
(self.is_extended and get_settings().pr_code_suggestions.rank_extended_suggestions):
logging.info('Ranking Suggestions...')
data['Code suggestions'] = await self.rank_suggestions(data['Code suggestions'])
if (not self.is_extended and get_settings().pr_code_suggestions.rank_suggestions) or \
(self.is_extended and get_settings().pr_code_suggestions.rank_extended_suggestions):
logging.info('Ranking Suggestions...')
data['Code suggestions'] = await self.rank_suggestions(data['Code suggestions'])
if get_settings().config.publish_output:
logging.info('Pushing PR review...')
self.git_provider.remove_initial_comment()
logging.info('Pushing inline code suggestions...')
self.push_inline_code_suggestions(data)
except Exception as e:
logging.error(f"Failed to generate code suggestions for PR, error: {e}")
if get_settings().config.publish_output:
logging.info('Pushing PR review...')
self.git_provider.remove_initial_comment()
logging.info('Pushing inline code suggestions...')
self.push_inline_code_suggestions(data)
async def _prepare_prediction(self, model: str):
logging.info('Getting PR diff...')
@ -122,8 +111,11 @@ class PRCodeSuggestions:
if get_settings().config.verbosity_level >= 2:
logging.info(f"suggestion: {d}")
relevant_file = d['relevant file'].strip()
relevant_lines_start = int(d['relevant lines start']) # absolute position
relevant_lines_end = int(d['relevant lines end'])
relevant_lines_str = d['relevant lines'].strip()
if ',' in relevant_lines_str: # handling 'relevant lines': '181, 190' or '178-184, 188-194'
relevant_lines_str = relevant_lines_str.split(',')[0]
relevant_lines_start = int(relevant_lines_str.split('-')[0]) # absolute position
relevant_lines_end = int(relevant_lines_str.split('-')[-1])
content = d['suggestion content']
new_code_snippet = d['improved code']

View File

@ -1,6 +1,5 @@
import copy
import json
import re
import logging
from typing import List, Tuple
@ -29,7 +28,6 @@ class PRDescription:
self.main_pr_language = get_main_pr_language(
self.git_provider.get_languages(), self.git_provider.get_files()
)
self.pr_id = self.git_provider.get_pr_id()
# Initialize the AI handler
self.ai_handler = AiHandler()
@ -38,7 +36,7 @@ class PRDescription:
self.vars = {
"title": self.git_provider.pr.title,
"branch": self.git_provider.get_pr_branch(),
"description": self.git_provider.get_pr_description(full=False),
"description": self.git_provider.get_pr_description(),
"language": self.main_pr_language,
"diff": "", # empty diff for initial calculation
"extra_instructions": get_settings().pr_description.extra_instructions,
@ -48,12 +46,8 @@ class PRDescription:
self.user_description = self.git_provider.get_user_description()
# Initialize the token handler
self.token_handler = TokenHandler(
self.git_provider.pr,
self.vars,
get_settings().pr_description_prompt.system,
get_settings().pr_description_prompt.user,
)
self.token_handler = TokenHandler(self.vars, get_settings().pr_description_prompt.system,
get_settings().pr_description_prompt.user)
# Initialize patches_diff and prediction attributes
self.patches_diff = None
@ -63,44 +57,27 @@ class PRDescription:
"""
Generates a PR description using an AI model and publishes it to the PR.
"""
try:
logging.info(f"Generating a PR description {self.pr_id}")
if get_settings().config.publish_output:
self.git_provider.publish_comment("Preparing PR description...", is_temporary=True)
await retry_with_fallback_models(self._prepare_prediction)
logging.info(f"Preparing answer {self.pr_id}")
if self.prediction:
self._prepare_data()
logging.info('Generating a PR description...')
if get_settings().config.publish_output:
self.git_provider.publish_comment("Preparing pr description...", is_temporary=True)
await retry_with_fallback_models(self._prepare_prediction)
logging.info('Preparing answer...')
pr_title, pr_body, pr_types, markdown_text = self._prepare_pr_answer()
if get_settings().config.publish_output:
logging.info('Pushing answer...')
if get_settings().pr_description.publish_description_as_comment:
self.git_provider.publish_comment(markdown_text)
else:
return None
pr_labels = []
if get_settings().pr_description.publish_labels:
pr_labels = self._prepare_labels()
if get_settings().pr_description.use_description_markers:
pr_title, pr_body = self._prepare_pr_answer_with_markers()
else:
pr_title, pr_body, = self._prepare_pr_answer()
full_markdown_description = f"## Title\n\n{pr_title}\n\n___\n{pr_body}"
if get_settings().config.publish_output:
logging.info(f"Pushing answer {self.pr_id}")
if get_settings().pr_description.publish_description_as_comment:
self.git_provider.publish_comment(full_markdown_description)
else:
self.git_provider.publish_description(pr_title, pr_body)
if get_settings().pr_description.publish_labels and self.git_provider.is_supported("get_labels"):
current_labels = self.git_provider.get_labels()
if current_labels is None:
current_labels = []
self.git_provider.publish_labels(pr_labels + current_labels)
self.git_provider.remove_initial_comment()
except Exception as e:
logging.error(f"Error generating PR description {self.pr_id}: {e}")
self.git_provider.publish_description(pr_title, pr_body)
if self.git_provider.is_supported("get_labels"):
current_labels = self.git_provider.get_labels()
if current_labels is None:
current_labels = []
self.git_provider.publish_labels(pr_types + current_labels)
self.git_provider.remove_initial_comment()
return ""
@ -118,12 +95,9 @@ class PRDescription:
Any exceptions raised by the 'get_pr_diff' and '_get_prediction' functions.
"""
if get_settings().pr_description.use_description_markers and 'pr_agent:' not in self.user_description:
return None
logging.info(f"Getting PR diff {self.pr_id}")
logging.info('Getting PR diff...')
self.patches_diff = get_pr_diff(self.git_provider, self.token_handler, model)
logging.info(f"Getting AI prediction {self.pr_id}")
logging.info('Getting AI prediction...')
self.prediction = await self._get_prediction(model)
async def _get_prediction(self, model: str) -> str:
@ -156,71 +130,34 @@ class PRDescription:
return response
def _prepare_data(self):
# Load the AI prediction data into a dictionary
self.data = load_yaml(self.prediction.strip())
if get_settings().pr_description.add_original_user_description and self.user_description:
self.data["User Description"] = self.user_description
def _prepare_labels(self) -> List[str]:
pr_types = []
# If the 'PR Type' key is present in the dictionary, split its value by comma and assign it to 'pr_types'
if 'PR Type' in self.data:
if type(self.data['PR Type']) == list:
pr_types = self.data['PR Type']
elif type(self.data['PR Type']) == str:
pr_types = self.data['PR Type'].split(',')
return pr_types
def _prepare_pr_answer_with_markers(self) -> Tuple[str, str]:
logging.info(f"Using description marker replacements {self.pr_id}")
title = self.vars["title"]
body = self.user_description
if get_settings().pr_description.include_generated_by_header:
ai_header = f"### 🤖 Generated by PR Agent at {self.git_provider.last_commit_id.sha}\n\n"
else:
ai_header = ""
ai_summary = self.data.get('PR Description')
if ai_summary and not re.search(r'<!--\s*pr_agent:summary\s*-->', body):
summary = f"{ai_header}{ai_summary}"
body = body.replace('pr_agent:summary', summary)
if not re.search(r'<!--\s*pr_agent:walkthrough\s*-->', body):
ai_walkthrough = self.data.get('PR Main Files Walkthrough')
if ai_walkthrough:
walkthrough = str(ai_header)
for file in ai_walkthrough:
filename = file['filename'].replace("'", "`")
description = file['changes in file'].replace("'", "`")
walkthrough += f'- `{filename}`: {description}\n'
body = body.replace('pr_agent:walkthrough', walkthrough)
return title, body
def _prepare_pr_answer(self) -> Tuple[str, str]:
def _prepare_pr_answer(self) -> Tuple[str, str, List[str], str]:
"""
Prepare the PR description based on the AI prediction data.
Returns:
- title: a string containing the PR title.
- pr_body: a string containing the PR description body in a markdown format.
- pr_body: a string containing the PR body in a markdown format.
- pr_types: a list of strings containing the PR types.
- markdown_text: a string containing the AI prediction data in a markdown format. used for publishing a comment
"""
# Load the AI prediction data into a dictionary
data = load_yaml(self.prediction.strip())
# Iterate over the dictionary items and append the key and value to 'markdown_text' in a markdown format
markdown_text = ""
for key, value in self.data.items():
markdown_text += f"## {key}\n\n"
markdown_text += f"{value}\n\n"
if get_settings().pr_description.add_original_user_description and self.user_description:
data["User Description"] = self.user_description
# Initialization
pr_types = []
# If the 'PR Type' key is present in the dictionary, split its value by comma and assign it to 'pr_types'
if 'PR Type' in data:
if type(data['PR Type']) == list:
pr_types = data['PR Type']
elif type(data['PR Type']) == str:
pr_types = data['PR Type'].split(',')
# Remove the 'PR Title' key from the dictionary
ai_title = self.data.pop('PR Title', self.vars["title"])
ai_title = data.pop('PR Title')
if get_settings().pr_description.keep_original_user_title:
# Assign the original PR title to the 'title' variable
title = self.vars["title"]
@ -231,27 +168,25 @@ class PRDescription:
# Iterate over the remaining dictionary items and append the key and value to 'pr_body' in a markdown format,
# except for the items containing the word 'walkthrough'
pr_body = ""
for idx, (key, value) in enumerate(self.data.items()):
for idx, (key, value) in enumerate(data.items()):
pr_body += f"## {key}:\n"
if 'walkthrough' in key.lower():
# for filename, description in value.items():
if self.git_provider.is_supported("gfm_markdown"):
pr_body += "<details> <summary>files:</summary>\n\n"
for file in value:
filename = file['filename'].replace("'", "`")
description = file['changes in file']
pr_body += f'`{filename}`: {description}\n'
if self.git_provider.is_supported("gfm_markdown"):
pr_body +="</details>\n"
else:
# if the value is a list, join its items by comma
if type(value) == list:
value = ', '.join(v for v in value)
pr_body += f"{value}\n"
if idx < len(self.data) - 1:
if idx < len(data) - 1:
pr_body += "\n___\n"
markdown_text = f"## Title\n\n{title}\n\n___\n{pr_body}"
if get_settings().config.verbosity_level >= 2:
logging.info(f"title:\n{title}\n{pr_body}")
return title, pr_body
return title, pr_body, pr_types, markdown_text

View File

@ -26,9 +26,7 @@ class PRInformationFromUser:
"diff": "", # empty diff for initial calculation
"commit_messages_str": self.git_provider.get_commit_messages(),
}
self.token_handler = TokenHandler(self.git_provider.pr,
self.vars,
get_settings().pr_information_from_user_prompt.system,
self.token_handler = TokenHandler(self.vars, get_settings().pr_information_from_user_prompt.system,
get_settings().pr_information_from_user_prompt.user)
self.patches_diff = None
self.prediction = None

View File

@ -29,9 +29,7 @@ class PRQuestions:
"questions": self.question_str,
"commit_messages_str": self.git_provider.get_commit_messages(),
}
self.token_handler = TokenHandler(self.git_provider.pr,
self.vars,
get_settings().pr_questions_prompt.system,
self.token_handler = TokenHandler(self.vars, get_settings().pr_questions_prompt.system,
get_settings().pr_questions_prompt.user)
self.patches_diff = None
self.prediction = None

View File

@ -9,8 +9,7 @@ from jinja2 import Environment, StrictUndefined
from yaml import SafeLoader
from pr_agent.algo.ai_handler import AiHandler
from pr_agent.algo.pr_processing import get_pr_diff, retry_with_fallback_models, \
find_line_number_of_relevant_line_in_file, clip_tokens
from pr_agent.algo.pr_processing import get_pr_diff, retry_with_fallback_models
from pr_agent.algo.token_handler import TokenHandler
from pr_agent.algo.utils import convert_to_markdown, try_fix_json, try_fix_yaml, load_yaml
from pr_agent.config_loader import get_settings
@ -59,7 +58,6 @@ class PRReviewer:
"require_tests": get_settings().pr_reviewer.require_tests_review,
"require_security": get_settings().pr_reviewer.require_security_review,
"require_focused": get_settings().pr_reviewer.require_focused_review,
"require_estimate_effort_to_review": get_settings().pr_reviewer.require_estimate_effort_to_review,
'num_code_suggestions': get_settings().pr_reviewer.num_code_suggestions,
'question_str': question_str,
'answer_str': answer_str,
@ -67,12 +65,8 @@ class PRReviewer:
"commit_messages_str": self.git_provider.get_commit_messages(),
}
self.token_handler = TokenHandler(
self.git_provider.pr,
self.vars,
get_settings().pr_review_prompt.system,
get_settings().pr_review_prompt.user
)
self.token_handler = TokenHandler(self.vars, get_settings().pr_review_prompt.system,
get_settings().pr_review_prompt.user)
def parse_args(self, args: List[str]) -> None:
"""
@ -95,32 +89,28 @@ class PRReviewer:
"""
Review the pull request and generate feedback.
"""
if self.is_auto and not get_settings().pr_reviewer.automatic_review:
logging.info(f'Automatic review is disabled {self.pr_url}')
return None
try:
if self.is_auto and not get_settings().pr_reviewer.automatic_review:
logging.info(f'Automatic review is disabled {self.pr_url}')
return None
logging.info(f'Reviewing PR: {self.pr_url} ...')
logging.info(f'Reviewing PR: {self.pr_url} ...')
if get_settings().config.publish_output:
self.git_provider.publish_comment("Preparing review...", is_temporary=True)
await retry_with_fallback_models(self._prepare_prediction)
logging.info('Preparing PR review...')
pr_comment = self._prepare_pr_review()
if get_settings().config.publish_output:
logging.info('Pushing PR review...')
self.git_provider.publish_comment(pr_comment)
self.git_provider.remove_initial_comment()
if get_settings().pr_reviewer.inline_code_comments:
logging.info('Pushing inline code comments...')
self._publish_inline_code_comments()
except Exception as e:
logging.error(f"Failed to review PR: {e}")
if get_settings().config.publish_output:
self.git_provider.publish_comment("Preparing review...", is_temporary=True)
await retry_with_fallback_models(self._prepare_prediction)
logging.info('Preparing PR review...')
pr_comment = self._prepare_pr_review()
if get_settings().config.publish_output:
logging.info('Pushing PR review...')
self.git_provider.publish_comment(pr_comment)
self.git_provider.remove_initial_comment()
if get_settings().pr_reviewer.inline_code_comments:
logging.info('Pushing inline code comments...')
self._publish_inline_code_comments()
async def _prepare_prediction(self, model: str) -> None:
"""
@ -219,11 +209,11 @@ class PRReviewer:
"⏮️ Review for commits since previous PR-Agent review": f"Starting from commit {last_commit_url}"}})
data.move_to_end('Incremental PR Review', last=False)
markdown_text = convert_to_markdown(data, self.git_provider.is_supported("gfm_markdown"))
markdown_text = convert_to_markdown(data)
user = self.git_provider.get_user_id()
# Add help text if not in CLI mode
if not get_settings().get("CONFIG.CLI_MODE", False):
# Add help text if not in CLI§ mode
if not get_settings().get("CONFIG.CLI§_MODE", False):
markdown_text += "\n### How to use\n"
if user and '[bot]' not in user:
markdown_text += bot_help_text(user)
@ -271,7 +261,7 @@ class PRReviewer:
self.git_provider.publish_inline_comment(content, relevant_file, relevant_line_in_file)
if comments:
self.git_provider.publish_inline_comments(comments)
self.git_provider.publish_inline_comments(comments)
def _get_user_answers(self) -> Tuple[str, str]:
"""

View File

@ -1,276 +0,0 @@
import copy
import json
import logging
from enum import Enum
from typing import List, Tuple
import pinecone
import openai
import pandas as pd
from pydantic import BaseModel, Field
from pr_agent.algo import MAX_TOKENS
from pr_agent.algo.token_handler import TokenHandler
from pr_agent.config_loader import get_settings
from pr_agent.git_providers import get_git_provider
from pinecone_datasets import Dataset, DatasetMetadata
MODEL = "text-embedding-ada-002"
class PRSimilarIssue:
def __init__(self, issue_url: str, args: list = None):
if get_settings().config.git_provider != "github":
raise Exception("Only github is supported for similar issue tool")
self.cli_mode = get_settings().CONFIG.CLI_MODE
self.max_issues_to_scan = get_settings().pr_similar_issue.max_issues_to_scan
self.issue_url = issue_url
self.git_provider = get_git_provider()()
repo_name, issue_number = self.git_provider._parse_issue_url(issue_url.split('=')[-1])
self.git_provider.repo = repo_name
self.git_provider.repo_obj = self.git_provider.github_client.get_repo(repo_name)
self.token_handler = TokenHandler()
repo_obj = self.git_provider.repo_obj
repo_name_for_index = self.repo_name_for_index = repo_obj.full_name.lower().replace('/', '-').replace('_/', '-')
index_name = self.index_name = "codium-ai-pr-agent-issues"
# assuming pinecone api key and environment are set in secrets file
try:
api_key = get_settings().pinecone.api_key
environment = get_settings().pinecone.environment
except Exception:
if not self.cli_mode:
repo_name, original_issue_number = self.git_provider._parse_issue_url(self.issue_url.split('=')[-1])
issue_main = self.git_provider.repo_obj.get_issue(original_issue_number)
issue_main.create_comment("Please set pinecone api key and environment in secrets file")
raise Exception("Please set pinecone api key and environment in secrets file")
# check if index exists, and if repo is already indexed
run_from_scratch = False
upsert = True
pinecone.init(api_key=api_key, environment=environment)
if not index_name in pinecone.list_indexes():
run_from_scratch = True
upsert = False
else:
if get_settings().pr_similar_issue.force_update_dataset:
upsert = True
else:
pinecone_index = pinecone.Index(index_name=index_name)
res = pinecone_index.fetch([f"example_issue_{repo_name_for_index}"]).to_dict()
if res["vectors"]:
upsert = False
if run_from_scratch or upsert: # index the entire repo
logging.info('Indexing the entire repo...')
logging.info('Getting issues...')
issues = list(repo_obj.get_issues(state='all'))
logging.info('Done')
self._update_index_with_issues(issues, repo_name_for_index, upsert=upsert)
else: # update index if needed
pinecone_index = pinecone.Index(index_name=index_name)
issues_to_update = []
issues_paginated_list = repo_obj.get_issues(state='all')
counter = 1
for issue in issues_paginated_list:
if issue.pull_request:
continue
issue_str, comments, number = self._process_issue(issue)
issue_key = f"issue_{number}"
id = issue_key + "." + "issue"
res = pinecone_index.fetch([id]).to_dict()
is_new_issue = True
for vector in res["vectors"].values():
if vector['metadata']['repo'] == repo_name_for_index:
is_new_issue = False
break
if is_new_issue:
counter += 1
issues_to_update.append(issue)
else:
break
if issues_to_update:
logging.info(f'Updating index with {counter} new issues...')
self._update_index_with_issues(issues_to_update, repo_name_for_index, upsert=True)
else:
logging.info('No new issues to update')
async def run(self):
logging.info('Getting issue...')
repo_name, original_issue_number = self.git_provider._parse_issue_url(self.issue_url.split('=')[-1])
issue_main = self.git_provider.repo_obj.get_issue(original_issue_number)
issue_str, comments, number = self._process_issue(issue_main)
openai.api_key = get_settings().openai.key
logging.info('Done')
logging.info('Querying...')
res = openai.Embedding.create(input=[issue_str], engine=MODEL)
embeds = [record['embedding'] for record in res['data']]
pinecone_index = pinecone.Index(index_name=self.index_name)
res = pinecone_index.query(embeds[0],
top_k=5,
filter={"repo": self.repo_name_for_index},
include_metadata=True).to_dict()
relevant_issues_number_list = []
relevant_comment_number_list = []
score_list = []
for r in res['matches']:
issue_number = int(r["id"].split('.')[0].split('_')[-1])
if original_issue_number == issue_number:
continue
if issue_number not in relevant_issues_number_list:
relevant_issues_number_list.append(issue_number)
if 'comment' in r["id"]:
relevant_comment_number_list.append(int(r["id"].split('.')[1].split('_')[-1]))
else:
relevant_comment_number_list.append(-1)
score_list.append(str("{:.2f}".format(r['score'])))
logging.info('Done')
logging.info('Publishing response...')
similar_issues_str = "### Similar Issues\n___\n\n"
for i, issue_number_similar in enumerate(relevant_issues_number_list):
issue = self.git_provider.repo_obj.get_issue(issue_number_similar)
title = issue.title
url = issue.html_url
if relevant_comment_number_list[i] != -1:
url = list(issue.get_comments())[relevant_comment_number_list[i]].html_url
similar_issues_str += f"{i + 1}. **[{title}]({url})** (score={score_list[i]})\n\n"
if get_settings().config.publish_output:
response = issue_main.create_comment(similar_issues_str)
logging.info(similar_issues_str)
logging.info('Done')
def _process_issue(self, issue):
header = issue.title
body = issue.body
number = issue.number
if get_settings().pr_similar_issue.skip_comments:
comments = []
else:
comments = list(issue.get_comments())
issue_str = f"Issue Header: \"{header}\"\n\nIssue Body:\n{body}"
return issue_str, comments, number
def _update_index_with_issues(self, issues_list, repo_name_for_index, upsert=False):
logging.info('Processing issues...')
corpus = Corpus()
example_issue_record = Record(
id=f"example_issue_{repo_name_for_index}",
text="example_issue",
metadata=Metadata(repo=repo_name_for_index)
)
corpus.append(example_issue_record)
counter = 0
for issue in issues_list:
if issue.pull_request:
continue
counter += 1
if counter % 100 == 0:
logging.info(f"Scanned {counter} issues")
if counter >= self.max_issues_to_scan:
logging.info(f"Scanned {self.max_issues_to_scan} issues, stopping")
break
issue_str, comments, number = self._process_issue(issue)
issue_key = f"issue_{number}"
username = issue.user.login
created_at = str(issue.created_at)
if len(issue_str) < 8000 or \
self.token_handler.count_tokens(issue_str) < MAX_TOKENS[MODEL]: # fast reject first
issue_record = Record(
id=issue_key + "." + "issue",
text=issue_str,
metadata=Metadata(repo=repo_name_for_index,
username=username,
created_at=created_at,
level=IssueLevel.ISSUE)
)
corpus.append(issue_record)
if comments:
for j, comment in enumerate(comments):
comment_body = comment.body
num_words_comment = len(comment_body.split())
if num_words_comment < 10 or not isinstance(comment_body, str):
continue
if len(comment_body) < 8000 or \
self.token_handler.count_tokens(comment_body) < MAX_TOKENS[MODEL]:
comment_record = Record(
id=issue_key + ".comment_" + str(j + 1),
text=comment_body,
metadata=Metadata(repo=repo_name_for_index,
username=username, # use issue username for all comments
created_at=created_at,
level=IssueLevel.COMMENT)
)
corpus.append(comment_record)
df = pd.DataFrame(corpus.dict()["documents"])
logging.info('Done')
logging.info('Embedding...')
openai.api_key = get_settings().openai.key
list_to_encode = list(df["text"].values)
try:
res = openai.Embedding.create(input=list_to_encode, engine=MODEL)
embeds = [record['embedding'] for record in res['data']]
except:
embeds = []
logging.error('Failed to embed entire list, embedding one by one...')
for i, text in enumerate(list_to_encode):
try:
res = openai.Embedding.create(input=[text], engine=MODEL)
embeds.append(res['data'][0]['embedding'])
except:
embeds.append([0] * 1536)
df["values"] = embeds
meta = DatasetMetadata.empty()
meta.dense_model.dimension = len(embeds[0])
ds = Dataset.from_pandas(df, meta)
logging.info('Done')
api_key = get_settings().pinecone.api_key
environment = get_settings().pinecone.environment
if not upsert:
logging.info('Creating index from scratch...')
ds.to_pinecone_index(self.index_name, api_key=api_key, environment=environment)
else:
logging.info('Upserting index...')
namespace = ""
batch_size: int = 100
concurrency: int = 10
pinecone.init(api_key=api_key, environment=environment)
ds._upsert_to_index(self.index_name, namespace, batch_size, concurrency)
logging.info('Done')
class IssueLevel(str, Enum):
ISSUE = "issue"
COMMENT = "comment"
class Metadata(BaseModel):
repo: str
username: str = Field(default="@codium")
created_at: str = Field(default="01-01-1970 00:00:00.00000")
level: IssueLevel = Field(default=IssueLevel.ISSUE)
class Config:
use_enum_values = True
class Record(BaseModel):
id: str
text: str
metadata: Metadata
class Corpus(BaseModel):
documents: List[Record] = Field(default=[])
def append(self, r: Record):
self.documents.append(r)

View File

@ -40,13 +40,11 @@ class PRUpdateChangelog:
"extra_instructions": get_settings().pr_update_changelog.extra_instructions,
"commit_messages_str": self.git_provider.get_commit_messages(),
}
self.token_handler = TokenHandler(self.git_provider.pr,
self.vars,
get_settings().pr_update_changelog_prompt.system,
self.token_handler = TokenHandler(self.vars, get_settings().pr_update_changelog_prompt.system,
get_settings().pr_update_changelog_prompt.user)
async def run(self):
# assert type(self.git_provider) == GithubProvider, "Currently only Github is supported"
assert type(self.git_provider) == GithubProvider, "Currently only Github is supported"
logging.info('Updating the changelog...')
if get_settings().config.publish_output:

View File

@ -35,12 +35,12 @@ dependencies = {file = ["requirements.txt"]}
"Homepage" = "https://github.com/Codium-ai/pr-agent"
[tool.setuptools]
include-package-data = false
include-package-data = true
license-files = ["LICENSE"]
[tool.setuptools.packages.find]
where = ["."]
include = ["pr_agent"]
include = ["pr_agent", "pr_agent.*"]
[project.scripts]
pr-agent = "pr_agent.cli:run"

View File

@ -1,23 +1,19 @@
dynaconf==3.1.12
fastapi==0.99.0
PyGithub==1.59.*
retry==0.9.2
openai==0.27.8
Jinja2==3.1.2
tiktoken==0.4.0
uvicorn==0.22.0
python-gitlab==3.15.0
pytest==7.4.0
aiohttp==3.8.4
atlassian-python-api==3.39.0
GitPython==3.1.32
PyYAML==6.0
starlette-context==0.3.6
litellm~=0.1.574
boto3==1.28.25
google-cloud-storage==2.10.0
ujson==5.8.0
azure-devops==7.1.0b3
msrest==0.7.1
pinecone-client
pinecone-datasets @ git+https://github.com/mrT23/pinecone-datasets.git@main
dynaconf~=3.1.12
fastapi~=0.103.0
PyGithub~=1.59.0
retry~=0.9.2
openai~=0.27.8
Jinja2~=3.1.2
tiktoken~=0.4.0
uvicorn~=0.22.0
python-gitlab~=3.15.0
pytest~=7.4.0
aiohttp~=3.8.4
atlassian-python-api~=3.39.0
GitPython~=3.1.32
PyYAML~=6.0
starlette-context~=0.3.6
litellm~=0.1.445
boto3~=1.28.25
google-cloud-storage~=2.10.0
ujson~=5.8.0

View File

@ -110,7 +110,7 @@ class TestCodeCommitProvider:
# Mock the response from the AWS client for get_pull_request method
api.boto_client.get_pull_request.return_value = {
"pullRequest": {
"pullRequestId": "321",
"pullRequestId": "3",
"title": "My PR",
"description": "My PR description",
"pullRequestTargets": [
@ -125,7 +125,7 @@ class TestCodeCommitProvider:
}
}
pr = api.get_pr("my_test_repo", 321)
pr = api.get_pr(321)
assert pr.title == "My PR"
assert pr.description == "My PR description"

View File

@ -1,9 +1,7 @@
import pytest
from unittest.mock import patch
from pr_agent.git_providers.codecommit_provider import CodeCommitFile
from pr_agent.git_providers.codecommit_provider import CodeCommitProvider
from pr_agent.git_providers.codecommit_provider import PullRequestCCMimic
from pr_agent.git_providers.git_provider import EDIT_TYPE
from pr_agent.algo.utils import EDIT_TYPE
class TestCodeCommitFile:
@ -27,64 +25,12 @@ class TestCodeCommitFile:
class TestCodeCommitProvider:
def test_get_title(self):
# Test that the get_title() function returns the PR title
with patch.object(CodeCommitProvider, "__init__", lambda x, y: None):
provider = CodeCommitProvider(None)
provider.pr = PullRequestCCMimic("My Test PR Title", [])
assert provider.get_title() == "My Test PR Title"
def test_get_pr_id(self):
# Test that the get_pr_id() function returns the correct ID
with patch.object(CodeCommitProvider, "__init__", lambda x, y: None):
provider = CodeCommitProvider(None)
provider.repo_name = "my_test_repo"
provider.pr_num = 321
assert provider.get_pr_id() == "my_test_repo/321"
def test_parse_pr_url(self):
# Test that the _parse_pr_url() function can extract the repo name and PR number from a CodeCommit URL
url = "https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/my_test_repo/pull-requests/321"
repo_name, pr_number = CodeCommitProvider._parse_pr_url(url)
assert repo_name == "my_test_repo"
assert pr_number == 321
def test_is_valid_codecommit_hostname(self):
# Test the various AWS regions
assert CodeCommitProvider._is_valid_codecommit_hostname("af-south-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-east-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-northeast-3.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-south-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-south-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-3.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ap-southeast-4.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("ca-central-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-central-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-central-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-north-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-south-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-south-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("eu-west-3.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("il-central-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("me-central-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("me-south-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("sa-east-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-east-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-east-2.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-gov-east-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-gov-west-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-west-1.console.aws.amazon.com")
assert CodeCommitProvider._is_valid_codecommit_hostname("us-west-2.console.aws.amazon.com")
# Test non-AWS regions
assert not CodeCommitProvider._is_valid_codecommit_hostname("no-such-region.console.aws.amazon.com")
assert not CodeCommitProvider._is_valid_codecommit_hostname("console.aws.amazon.com")
# Test that an error is raised when an invalid CodeCommit URL is provided to the set_pr() method of the CodeCommitProvider class.
# Generated by CodiumAI
def test_invalid_codecommit_url(self):
@ -160,7 +106,6 @@ class TestCodeCommitProvider:
assert percentages == {}
def test_get_edit_type(self):
# Test that the _get_edit_type() function can convert a CodeCommit letter to an EDIT_TYPE enum
assert CodeCommitProvider._get_edit_type("A") == EDIT_TYPE.ADDED
assert CodeCommitProvider._get_edit_type("D") == EDIT_TYPE.DELETED
assert CodeCommitProvider._get_edit_type("M") == EDIT_TYPE.MODIFIED
@ -172,18 +117,3 @@ class TestCodeCommitProvider:
assert CodeCommitProvider._get_edit_type("r") == EDIT_TYPE.RENAMED
assert CodeCommitProvider._get_edit_type("X") is None
def test_add_additional_newlines(self):
# a short string to test adding double newlines
input = "abc\ndef\n\n___\nghi\njkl\nmno\n\npqr\n"
expect = "abc\n\ndef\n\n___\n\nghi\n\njkl\n\nmno\n\npqr\n\n"
assert CodeCommitProvider._add_additional_newlines(input) == expect
# a test example from a real PR
input = "## PR Type:\nEnhancement\n\n___\n## PR Description:\nThis PR introduces a new feature to the script, allowing users to filter servers by name.\n\n___\n## PR Main Files Walkthrough:\n`foo`: The foo script has been updated to include a new command line option `-f` or `--filter`.\n`bar`: The bar script has been updated to list stopped servers.\n"
expect = "## PR Type:\n\nEnhancement\n\n___\n\n## PR Description:\n\nThis PR introduces a new feature to the script, allowing users to filter servers by name.\n\n___\n\n## PR Main Files Walkthrough:\n\n`foo`: The foo script has been updated to include a new command line option `-f` or `--filter`.\n\n`bar`: The bar script has been updated to list stopped servers.\n\n"
assert CodeCommitProvider._add_additional_newlines(input) == expect
def test_remove_markdown_html(self):
input = "## PR Feedback\n<details><summary>Code feedback:</summary>\nfile foo\n</summary>\n"
expect = "## PR Feedback\nCode feedback:\nfile foo\n\n"
assert CodeCommitProvider._remove_markdown_html(input) == expect

View File

@ -1,8 +1,6 @@
# Generated by CodiumAI
from pr_agent.git_providers.git_provider import FilePatchInfo
from pr_agent.algo.pr_processing import find_line_number_of_relevant_line_in_file
from pr_agent.algo.utils import FilePatchInfo, find_line_number_of_relevant_line_in_file
import pytest

View File

@ -61,7 +61,7 @@ class TestSortFilesByMainLanguages:
type('', (object,), {'filename': 'file1.py'})(),
type('', (object,), {'filename': 'file2.java'})()
]
expected_output = [{'language': 'Other', 'files': files}]
expected_output = [{'language': 'Other', 'files': []}]
assert sort_files_by_main_languages(languages, files) == expected_output
# Tests that function handles empty files list