Merge commit 'f1ab6ec88f4dc3e2abb90244de5a1f41d0492743' into docs/pr_compression_doc

# Conflicts:
#	README.md

.dockerignore

@@ -1 +1,2 @@
-venv/
+venv/
+pr_agent/settings/.secrets.toml

README.md

@@ -1,52 +1,60 @@
 <div align="center">
 
+<img src="./pics/Icon-7.png" alt="pr-agent_icon" width="100"/>
+
 # pr-agent
+
 [![GitHub license](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://github.com/Codium-ai/pr-agent/blob/main/LICENSE)
 [![Discord](https://badgen.net/badge/icon/discord?icon=discord&label&color=purple)](https://discord.com/channels/1057273017547378788/1126104260430528613)
 
-CodiumAI `PR-Agent` is an open-source tool that helps developers review PRs faster and more efficiently. 
+CodiumAI `pr-agent` is an open-source tool is powered by GPT-4 aming to help developers review PRs faster and more efficiently. It automatically analyzes the PR, and provides feedback and suggestions, and can answer questions.
-It automatically analyzes the PR, and provides feedback and suggestions, and can answer questions. 
+
-It is powered by GPT-4, and is based on the [CodiumAI](https://github.com/Codium-ai/) platform.
 </div>
 
+- [Quickstart](#Quickstart)
+- [Usage and Tools](#usage-and-tools)
+- [Configuration](#Configuration)
+- [How it works](#how-it-works)
+- [Roadmap](#roadmap)
+- [Similar projects](#similar-projects)
 
+## Live demo
 
-* [Quickstart](#Quickstart)
+Experience GPT-4 powered PR review on your public Github repository with our hosted pr-agent. To try it, mention @CodiumAI-Agent in a PR comment! The agent will generate the review in response ([see details in the Usage section](#usage-and-tools)).
-* [Configuration](#Configuration)
-* [Usage and Tools](#usage-and-tools)
-* [How it works](#how-it-works)
-* [Roadmap](#roadmap)
-* [Similar projects](#similar-projects)
 
-
+![Review generation process](./pics/pr-agent-review-process1.gif)
-## Quickstart
-
-To get started with PR-Agent quickly, you first need to acquire two tokens:
-1. An OpenAI key from [here](https://platform.openai.com/), with access to GPT-4.
-2. A GitHub personal access token (classic) with the repo scope.
-
-There are several ways to use PR-Agent. Let's start with the simplest one:
 
 ---
 
-### Method 1: Use Docker image (no installation required)
+## Quickstart
 
-To request a review for a PR, or ask a question about a PR, you can run the appropriate
+To get started with pr-agent quickly, you first need to acquire two tokens:
-Python scripts from the scripts folder. Here's how:
+
+1. An OpenAI key from [here](https://platform.openai.com/), with access to GPT-4.
+2. A GitHub personal access token (classic) with the repo scope.
+
+There are several ways to use pr-agent. Let's start with the simplest one:
+
+---
+
+#### Method 1: Use Docker image (no installation required)
+
+To request a review for a PR, or ask a question about a PR, you can run directly from the Docker image. Here's how:
 
 1. To request a review for a PR, run the following command:
+
 ```
 docker run --rm -it -e OPENAI.KEY=<your key> -e GITHUB.USER_TOKEN=<your token> codiumai/pr-agent --pr_url <pr url>
 ```
 
----
-
 2. To ask a question about a PR, run the following command:
+
 ```
 docker run --rm -it -e OPENAI.KEY=<your key> -e GITHUB.USER_TOKEN=<your token> codiumai/pr-agent --pr_url <pr url> --question "<your question>"
 ```
 
 Possible questions you can ask include:
+
 - What is the main theme of this PR?
 - Is the PR ready for merge?
 - What are the main changes in this PR?
@@ -55,25 +63,29 @@ Possible questions you can ask include:
 
 ---
 
-### Method 2: Run from source
+#### Method 2: Run from source
 
 1. Clone this repository:
+
 ```
 git clone https://github.com/Codium-ai/pr-agent.git
 ```
 
 2. Install the requirements in your favorite virtual environment:
+
 ```
 pip install -r requirements.txt
 ```
 
 3. Copy the secrets template file and fill in your OpenAI key and your GitHub user token:
+
 ```
 cp pr_agent/settings/.secrets_template.toml pr_agent/settings/.secrets
 # Edit .secrets file
 ```
 
 4. Run the appropriate Python scripts from the scripts folder:
+
 ```
 python pr_agent/cli.py --pr_url <pr url>
 python pr_agent/cli.py --pr_url <pr url> --question "<your question>"
@@ -81,19 +93,21 @@ python pr_agent/cli.py --pr_url <pr url> --question "<your question>"
 
 ---
 
-### Method 3: Method 3: Run as a polling server; request reviews by tagging your Github user on a PR
+#### Method 3: Method 3: Run as a polling server; request reviews by tagging your Github user on a PR
 
 Follow steps 1-3 of method 2.
 Run the following command to start the server:
+
 ```
 python pr_agent/servers/github_polling.py
 ```
 
 ---
 
-### Method 4: Run as a Github App, allowing you to automate the review process on your private or public repositories.
+#### Method 4: Run as a Github App, allowing you to automate the review process on your private or public repositories.
 
 1. Create a GitHub App from the [Github Developer Portal](https://docs.github.com/en/developers/apps/creating-a-github-app).
+
    - Set the following permissions:
      - Pull requests: Read & write
      - Issue comment: Read & write
@@ -103,15 +117,18 @@ python pr_agent/servers/github_polling.py
      - Pull request
 
 2. Generate a random secret for your app, and save it for later. For example, you can use:
+
 ```
 WEBHOOK_SECRET=$(python -c "import secrets; print(secrets.token_hex(10))")
 ```
 
 3. Acquire the following pieces of information from your app's settings page:
+
    - App private key (click "Generate a private key", and save the file)
    - App ID
 
 4. Clone this repository:
+
 ```
 git clone https://github.com/Codium-ai/pr-agent.git
 ```
@@ -122,21 +139,24 @@ git clone https://github.com/Codium-ai/pr-agent.git
    - Copy your app's private key to the private_key field.
    - Copy your app's ID to the app_id field.
    - Copy your app's webhook secret to the webhook_secret field.
+
 ```
 cp pr_agent/settings/.secrets_template.toml pr_agent/settings/.secrets
 # Edit .secrets file
 ```
 
 6. Build a Docker image for the app and optionally push it to a Docker repository. We'll use Dockerhub as an example:
+
 ```
 docker build . -t codiumai/pr-agent:github_app --target github_app -f docker/Dockerfile
 docker push codiumai/pr-agent:github_app  # Push to your Docker repository
 ```
 
-7. Host the app using a server, serverless function, or container environment. Alternatively, for development and 
+7. Host the app using a server, serverless function, or container environment. Alternatively, for development and
-   debugging, you may use tools like smee.io to forward webhooks to your local machine. 
+   debugging, you may use tools like smee.io to forward webhooks to your local machine.
 
 8. Go back to your app's settings, set the following:
+
    - Webhook URL: The URL of your app's server, or the URL of the smee.io channel.
    - Webhook secret: The secret you generated earlier.
 
@@ -145,11 +165,14 @@ docker push codiumai/pr-agent:github_app  # Push to your Docker repository
 ---
 
 ## Usage and Tools
-CodiumAI PR-Agent provides two types of interactions ("tools"): `"PR Reviewer"` and `"PR Q&A"`.
+
+CodiumAI pr-agent provides two types of interactions ("tools"): `"PR Reviewer"` and `"PR Q&A"`.
+
 - The "PR Reviewer" tool automatically analyzes PRs, and provides different types of feedbacks.
 - The "PR Q&A" tool answers free-text questions about the PR.
 
 ### PR Reviewer
+
 Here is a quick overview of the different sub-tools of PR Reviewer:
 
 - PR Analysis
@@ -163,10 +186,10 @@ Here is a quick overview of the different sub-tools of PR Reviewer:
   - General PR suggestions
   - Code suggestions
 
-
 This is how a typical output of the PR Reviewer looks like:
 
 ---
+
 #### PR Analysis
 
 - 🎯 **Main theme:** Adding language extension handler and token handler
@@ -182,26 +205,24 @@ This is how a typical output of the PR Reviewer looks like:
 
 - 🤖 **Code suggestions:**
 
-
   - **relevant file:** pr_agent/algo/language_handler.py
 
     **suggestion content:** Consider using a set instead of a list for 'bad_extensions' as checking membership in a set is faster than in a list. [medium]
 
-
   - **relevant file:** pr_agent/algo/language_handler.py
-   
+
     **suggestion content:** In the 'filter_bad_extensions' function, you are splitting the filename on '.' and taking the last element to get the extension. This might not work as expected if the filename contains multiple '.' characters. Consider using 'os.path.splitext' to get the file extension more reliably. [important]
 
 ---
 
-
 ### PR Q&A
+
 This tool answers free-text questions about the PR. This is how a typical output of the PR Q&A looks like:
 
----
 **Question**: summarize for me the PR in 4 bullet points
 
-**Answer**: 
+**Answer**:
+
 - The PR introduces a new feature to sort files by their main languages. It uses a mapping of programming languages to their file extensions to achieve this.
 - It also introduces a filter to exclude files with certain extensions, deemed as 'bad extensions', from the sorting process.
 - The PR modifies the `get_pr_diff` function in `pr_processing.py` to use the new sorting function. It also refactors the code to move the PR pruning logic into a separate function.
@@ -210,49 +231,56 @@ This tool answers free-text questions about the PR. This is how a typical output
 ---
 
 ## Configuration
-The different tools and sub-tools used by CodiumAI PR-Agent are easily configurable via the configuration file: `/settings/configuration.toml`.
+
+The different tools and sub-tools used by CodiumAI pr-agent are easily configurable via the configuration file: `/settings/configuration.toml`.
+
 #### Enabling/disabling sub-tools:
-You can enable/disable the different PR Reviewer sub-sections  with the following flags:
+
+You can enable/disable the different PR Reviewer sub-sections with the following flags:
+
 ```
 require_minimal_and_focused_review=true
 require_tests_review=true
 require_security_review=true
 ```
+
 #### Code Suggestions configuration:
+
 There are also configuration options to control different aspects of the `code suggestions` feature.
 The number of suggestions provided can be controlled by adjusting the following parameter:
+
 ```
 num_code_suggestions=4
 ```
+
 You can also enable more verbose and informative mode of code suggestions:
+
 ```
 extended_code_suggestions=false
-``` 
+```
+
 This is a comparison of the regular and extended code suggestions modes:
 
----
+- **relevant file:** sql.py
-Example for regular suggestion:
+- **suggestion content:** Remove hardcoded sensitive information like username and password. Use environment variables or a secure method to store these values. [important]
-
-  - **relevant file:** sql.py
-  - **suggestion content:** Remove hardcoded sensitive information like username and password. Use environment variables or a secure method to store these values. [important]
----
 
 Example for extended suggestion:
 
-  - **relevant file:** sql.py
+- **relevant file:** sql.py
-  - **suggestion content:** Remove hardcoded sensitive information (username and password) [important]
+- **suggestion content:** Remove hardcoded sensitive information (username and password) [important]
-  - **why:** Hardcoding sensitive information is a security risk. It's better to use environment variables or a secure way to store these values.
+- **why:** Hardcoding sensitive information is a security risk. It's better to use environment variables or a secure way to store these values.
-  - **code example:**
+- **code example:**
-    - **before code:**
+  - **before code:**
-        ```
+    ```
-        user = "root",
+    user = "root",
-        password = "Mysql@123",
+    password = "Mysql@123",
-        ```
+    ```
-    - **after code:**
+  - **after code:**
-        ```
+    ```
-        user = os.getenv('DB_USER'),
+    user = os.getenv('DB_USER'),
-        password = os.getenv('DB_PASSWORD'),
+    password = os.getenv('DB_PASSWORD'),
-        ```
+    ```
+
 ---
 
 ## How it works
@@ -262,6 +290,7 @@ Check out the [PR Compression strategy](./PR_COMPRESSION.md) page for more detai
 
 
 ## Roadmap
+
 - [ ] Support open-source models, as a replacement for openai models. Note that a minimal requirement for each open-source model is to have 8k+ context, and good support for generating json as an output
 - [ ] Support other Git providers, such as Gitlab and Bitbucket.
 - [ ] Develop additional logics for handling large PRs, and compressing git patches
@@ -276,7 +305,9 @@ Check out the [PR Compression strategy](./PR_COMPRESSION.md) page for more detai
   - [ ] ...
 
 ## Similar Projects
+
 - [CodiumAI - Meaningful tests for busy devs](https://github.com/Codium-ai/codiumai-vscode-release)
 - [Aider - GPT powered coding in your terminal](https://github.com/paul-gauthier/aider)
 - [GPT-Engineer](https://github.com/AntonOsika/gpt-engineer)
 - [CodeReview BOT](https://github.com/anc95/ChatGPT-CodeReview)
+- [AI-Maintainer](https://github.com/merwanehamadi/AI-Maintainer)

pr_agent/cli.py

@@ -15,11 +15,11 @@ def run():
     logging.basicConfig(level=os.environ.get("LOGLEVEL", "INFO"))
     if args.question:
         print(f"Question: {args.question} about PR {args.pr_url}")
-        reviewer = PRQuestions(args.pr_url, args.question, None)
+        reviewer = PRQuestions(args.pr_url, args.question, installation_id=None)
         asyncio.run(reviewer.answer())
     else:
         print(f"Reviewing PR: {args.pr_url}")
-        reviewer = PRReviewer(args.pr_url, None)
+        reviewer = PRReviewer(args.pr_url, installation_id=None, cli_mode=True)
         asyncio.run(reviewer.review())
 
 

pr_agent/config_loader.py

@@ -9,6 +9,7 @@ settings = Dynaconf(
          "settings/.secrets.toml",
          "settings/configuration.toml",
          "settings/pr_reviewer_prompts.toml",
-         "settings/pr_questions_prompts.toml"
+         "settings/pr_questions_prompts.toml",
+         "settings_prod/.secrets.toml"
         ]]
 )

pr_agent/servers/github_polling.py

@@ -20,6 +20,7 @@ def now() -> str:
 
 
 async def polling_loop():
+    handled_ids = set()
     since = [now()]
     last_modified = [None]
     git_provider = get_git_provider()()
@@ -54,6 +55,9 @@ async def polling_loop():
                         since[0] = None
                     notifications = await response.json()
                     for notification in notifications:
+                        if 'id' in notification and notification['id'] in handled_ids:
+                            continue
+                        handled_ids.add(notification['id'])
                         if 'reason' in notification and notification['reason'] == 'mention':
                             if 'subject' in notification and notification['subject']['type'] == 'PullRequest':
                                 pr_url = notification['subject']['url']

pr_agent/tools/pr_reviewer.py

@@ -14,7 +14,7 @@ from pr_agent.git_providers import get_git_provider
 
 
 class PRReviewer:
-    def __init__(self, pr_url: str, installation_id: Optional[int] = None):
+    def __init__(self, pr_url: str, installation_id: Optional[int] = None, cli_mode=False):
 
         self.git_provider = get_git_provider()(pr_url, installation_id)
         self.main_language = self.git_provider.get_main_pr_language()
@@ -22,6 +22,7 @@ class PRReviewer:
         self.ai_handler = AiHandler()
         self.patches_diff = None
         self.prediction = None
+        self.cli_mode = cli_mode
         self.vars = {
             "title": self.git_provider.pr.title,
             "branch": self.git_provider.get_pr_branch(),
@@ -91,16 +92,19 @@ class PRReviewer:
 
         markdown_text = convert_to_markdown(data)
         user = self.git_provider.get_user_id()
-        markdown_text += "\n### How to use\n"
+
-        if user and '[bot]' not in user:
+        if not self.cli_mode:
-            markdown_text += f"> Tag me in a comment '@{user}' to ask for a new review after you update the PR.\n"
+            markdown_text += "\n### How to use\n"
-            markdown_text += "> You can also tag me and ask any question, " \
+            if user and '[bot]' not in user:
-                             f"for example '@{user} is the PR ready for merge?'"
+                markdown_text += "\n### How to use\n"
-        else:
+                markdown_text += f"> Tag me in a comment '@{user}' to ask for a new review after you update the PR.\n"
-            markdown_text += "> Add a comment that says 'review' to ask for a new review " \
+                markdown_text += "> You can also tag me and ask any question, " \
-                             "after you update the PR.\n"
+                                 f"for example '@{user} is the PR ready for merge?'"
-            markdown_text += "> You can also add a comment that says 'answer QUESTION', " \
+            else:
-                             "for example 'answer is the PR ready for merge?'"
+                markdown_text += "> Add a comment that says 'review' to ask for a new review " \
+                                 "after you update the PR.\n"
+                markdown_text += "> You can also add a comment that says 'answer QUESTION', " \
+                                 "for example 'answer is the PR ready for merge?'"
 
         if settings.config.verbosity_level >= 2:
             logging.info(f"Markdown response:\n{markdown_text}")