Lint

Enhancement of PR Agent with User Interaction

.dockerignore

@@ -1,2 +1,3 @@
 venv/
-pr_agent/settings/.secrets.toml
+pr_agent/settings/.secrets.toml
+pics/

PR_COMPRESSION.md

@@ -39,4 +39,4 @@ We use [tiktoken](https://github.com/openai/tiktoken) to tokenize the patches af
 4. If we haven't reached the max token length, add the `deleted files` to the prompt until the prompt reaches the max token length (hard stop), skip the rest of the patches.
 
 ### Example
-![](./pics/git_patch_logic.png)
+![](https://codium.ai/images/git_patch_logic.png)

README.md

@@ -27,25 +27,25 @@ CodiumAI `PR-Agent` is an open-source tool aiming to help developers review PRs
 <h4>Describe:</h4>
 <div align="center">
 <p float="center">
-<img src="./pics/describe.gif" width="800">
+<img src="https://codium.ai/images/describe.gif" width="800">
 </p>
 </div>
 <h4>Review:</h4>
 <div align="center">
 <p float="center">
-<img src="./pics/review.gif" width="800">
+<img src="https://codium.ai/images/review.gif" width="800">
 </p>
 </div>
 <h4>Ask:</h4>
 <div align="center">
 <p float="center">
-<img src="./pics/ask.gif" width="800">
+<img src="https://codium.ai/images/ask.gif" width="800">
 </p>
 </div>
 <h4>Improve:</h4>
 <div align="center">
 <p float="center">
-<img src="./pics/improve.gif" width="800">
+<img src="https://codium.ai/images/improve.gif" width="800">
 </p>
 </div>
 <div align="left">
@@ -64,38 +64,40 @@ CodiumAI `PR-Agent` is an open-source tool aiming to help developers review PRs
 
 Experience GPT-4 powered PR review on your public GitHub repository with our hosted PR-Agent. To try it, just mention `@CodiumAI-Agent` and add the desired command in any PR comment! The agent will generate a response based on your command.
 
-![Review generation process](./pics/demo.gif)
+![Review generation process](https://codium.ai/images/demo.gif)
 
 To set up your own PR-Agent, see the [Quickstart](#Quickstart) section
 
 ---
 ## Overview
 `PR-Agent` offers extensive pull request functionalities across various git providers:
-|       |                                             | Github | Gitlab | Bitbucket |
-|-------|---------------------------------------------|--------|--------|-----------|
-| TOOLS | Review                                      | ✓      | ✓      | ✓         |
-|       | ⮑ Inline review                             | ✓     | ✓      |           |
-|       | Ask                                         | ✓      | ✓      |           |
-|       | Auto-Description                            | ✓      |        |           |
-|       | Improve Code                                | ✓      | ✓      |           |
+|       |                                             | GitHub | Gitlab | Bitbucket |
+|-------|---------------------------------------------|:------:|:------:|:---------:|
+| TOOLS | Review                                      |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |
+|       | ⮑ Inline review                             |   :white_check_mark:    |   :white_check_mark:    |           |
+|       | Ask                                         |   :white_check_mark:    |   :white_check_mark:    |           |
+|       | Auto-Description                            |   :white_check_mark:    |        |           |
+|       | Improve Code                                |   :white_check_mark:    |   :white_check_mark:    |           |
+|       | Reflect and Review                          |   :white_check_mark:    |                         |           |
 |       |                                             |        |        |           |
-| USAGE | CLI                                         | ✓      | ✓      | ✓         |
-|       | Tagging bot                                 | ✓      | ✓      |           |
-|       | Actions                                     | ✓      |        |           |
+| USAGE | CLI                                         |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |
+|       | Tagging bot                                 |   :white_check_mark:    |   :white_check_mark:    |           |
+|       | Actions                                     |   :white_check_mark:    |        |           |
 |       |                                             |        |        |           |
-| CORE  | PR compression                              | ✓      | ✓      | ✓         |
-|       | Repo language prioritization                | ✓      | ✓      | ✓         |
-|       | Adaptive and token-aware<br />file patch fitting | ✓      | ✓      | ✓         |
+| CORE  | PR compression                              |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |
+|       | Repo language prioritization                |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |
+|       | Adaptive and token-aware<br />file patch fitting |   :white_check_mark:    |   :white_check_mark:    |   :white_check_mark:       |
 
 Examples for invoking the different tools via the [CLI](#quickstart):
 - **Review**:       python cli.py --pr-url=<pr_url>  review
 - **Describe**:     python cli.py --pr-url=<pr_url>  describe
 - **Improve**:      python cli.py --pr-url=<pr_url>  improve
 - **Ask**:          python cli.py --pr-url=<pr_url>  ask "Write me a poem about this PR"
+- **Reflect**:      python cli.py --pr-url=<pr_url>  reflect
 
 "<pr_url>" is the url of the relevant PR (for example: https://github.com/Codium-ai/pr-agent/pull/50).
 
-In the [configuration](./CONFIGURATION.md) file you can select your git provider (Github, Gitlab, Bitbucket), and further configure the different tools.
+In the [configuration](./CONFIGURATION.md) file you can select your git provider (GitHub, Gitlab, Bitbucket), and further configure the different tools.
 
 ## Quickstart
 
@@ -111,25 +113,26 @@ There are several ways to use PR-Agent. Let's start with the simplest one:
 Here are several ways to install and run PR-Agent:
 
 - [Method 1: Use Docker image (no installation required)](INSTALL.md#method-1-use-docker-image-no-installation-required)
-- [Method 2: Run as a Github Action](INSTALL.md#method-2-run-as-a-github-action)
+- [Method 2: Run as a GitHub Action](INSTALL.md#method-2-run-as-a-github-action)
 - [Method 3: Run from source](INSTALL.md#method-3-run-from-source)
 - [Method 4: Run as a polling server](INSTALL.md#method-4-run-as-a-polling-server)
-  - Request reviews by tagging your Github user on a PR
-- [Method 5: Run as a Github App](INSTALL.md#method-5-run-as-a-github-app)
+  - Request reviews by tagging your GitHub user on a PR
+- [Method 5: Run as a GitHub App](INSTALL.md#method-5-run-as-a-github-app)
   - Allowing you to automate the review process on your private or public repositories
 
 ## Usage and Tools
 
-**PR-Agent** provides four types of interactions ("tools"): `"PR Reviewer"`, `"PR Q&A"`, `"PR Description"` and `"PR Code Sueggestions"`.
+**PR-Agent** provides five types of interactions ("tools"): `"PR Reviewer"`, `"PR Q&A"`, `"PR Description"`, `"PR Code Sueggestions"` and `"PR Reflect and Review"`.
 
-- The "PR Reviewer" tool automatically analyzes PRs, and provides different types of feedback.
-- The "PR Ask" tool answers free-text questions about the PR.
+- The "PR Reviewer" tool automatically analyzes PRs, and provides various types of feedback.
+- The "PR Q&A" tool answers free-text questions about the PR.
 - The "PR Description" tool automatically sets the PR Title and body.
 - The "PR Code Suggestion" tool provide inline code suggestions for the PR that can be applied and committed.
+- The "PR Reflect and Review" tool first initiates a dialog with the user and asks them to reflect on the PR, and then provides a review.
 
 ## How it works
 
-![PR-Agent Tools](./pics/pr_agent_overview.png)
+![PR-Agent Tools](https://codium.ai/images/pr_agent_overview.png)
 
 Check out the [PR Compression strategy](./PR_COMPRESSION.md) page for more details on how we convert a code diff to a manageable LLM prompt
 
@@ -138,11 +141,11 @@ Check out the [PR Compression strategy](./PR_COMPRESSION.md) page for more detai
 - [ ] Support open-source models, as a replacement for openai models. (Note - a minimal requirement for each open-source model is to have 8k+ context, and good support for generating json as an output)
 - [x] Support other Git providers, such as Gitlab and Bitbucket.
 - [ ] Develop additional logics for handling large PRs, and compressing git patches
-- [ ] Dedicated tools and sub-tools for specific programming languages (Python, Javascript, Java, C++, etc)
 - [ ] Add additional context to the prompt. For example, repo (or relevant files) summarization, with tools such a [ctags](https://github.com/universal-ctags/ctags)
 - [ ] Adding more tools. Possible directions:
   - [x] PR description
   - [x] Inline code suggestions
+  - [x] Reflect and review
   - [ ] Enforcing CONTRIBUTING.md guidelines
   - [ ] Performance (are there any performance issues)
   - [ ] Documentation (is the PR properly documented)

pr_agent/agent/pr_agent.py

@@ -2,8 +2,10 @@ import re
 
 from pr_agent.tools.pr_code_suggestions import PRCodeSuggestions
 from pr_agent.tools.pr_description import PRDescription
+from pr_agent.tools.pr_information_from_user import PRInformationFromUser
 from pr_agent.tools.pr_questions import PRQuestions
 from pr_agent.tools.pr_reviewer import PRReviewer
+from pr_agent.config_loader import settings
 
 
 class PRAgent:
@@ -11,8 +13,13 @@ class PRAgent:
         pass
 
     async def handle_request(self, pr_url, request) -> bool:
-        if any(cmd in request for cmd in ["/review", "/review_pr"]):
-            await PRReviewer(pr_url).review()
+        if any(cmd in request for cmd in ["/answer"]):
+            await PRReviewer(pr_url, is_answer=True).review()
+        elif any(cmd in request for cmd in ["/review", "/review_pr", "/reflect_and_review"]):
+            if settings.pr_reviewer.ask_and_reflect or "/reflect_and_review" in request:
+                await PRInformationFromUser(pr_url).generate_questions()
+            else:
+                await PRReviewer(pr_url).review()
         elif any(cmd in request for cmd in ["/describe", "/describe_pr"]):
             await PRDescription(pr_url).describe()
         elif any(cmd in request for cmd in ["/improve", "/improve_code"]):

pr_agent/cli.py

@@ -14,23 +14,26 @@ def run():
     parser = argparse.ArgumentParser(description='AI based pull request analyzer', usage="""\
 Usage: cli.py --pr-url <URL on supported git hosting service> <command> [<args>].
 For example:
-- cli.py --pr-url xxx review
-- cli.py --pr-url xxx describe
-- cli.py --pr-url xxx improve
-- cli.py --pr-url xxx ask "write me a poem about this PR"
+- cli.py --pr-url=... review
+- cli.py --pr-url=... describe
+- cli.py --pr-url=... improve
+- cli.py --pr-url=... ask "write me a poem about this PR"
+- cli.py --pr-url=... reflect
 
 Supported commands:
 review / review_pr - Add a review that includes a summary of the PR and specific suggestions for improvement.
 ask / ask_question [question] - Ask a question about the PR.
 describe / describe_pr - Modify the PR title and description based on the PR's contents.
 improve / improve_code - Suggest improvements to the code in the PR as pull request comments ready to commit.
+reflect - Ask the PR author questions about the PR.
 """)
     parser.add_argument('--pr_url', type=str, help='The URL of the PR to review', required=True)
     parser.add_argument('command', type=str, help='The', choices=['review', 'review_pr',
                                                                   'ask', 'ask_question',
                                                                   'describe', 'describe_pr',
                                                                   'improve', 'improve_code',
-                                                                  'user_questions'], default='review')
+                                                                  'reflect', 'review_after_reflect'],
+                        default='review')
     parser.add_argument('rest', nargs=argparse.REMAINDER, default=[])
     args = parser.parse_args()
     logging.basicConfig(level=os.environ.get("LOGLEVEL", "INFO"))
@@ -56,10 +59,14 @@ improve / improve_code - Suggest improvements to the code in the PR as pull requ
         print(f"Reviewing PR: {args.pr_url}")
         reviewer = PRReviewer(args.pr_url, cli_mode=True)
         asyncio.run(reviewer.review())
-    elif command in ['user_questions']:
+    elif command in ['reflect']:
         print(f"Asking the PR author questions: {args.pr_url}")
         reviewer = PRInformationFromUser(args.pr_url)
         asyncio.run(reviewer.generate_questions())
+    elif command in ['review_after_reflect']:
+        print(f"Processing author's answers and sending review: {args.pr_url}")
+        reviewer = PRReviewer(args.pr_url, cli_mode=True, is_answer=True)
+        asyncio.run(reviewer.review())
     else:
         print(f"Unknown command: {command}")
         parser.print_help()

pr_agent/git_providers/bitbucket_provider.py

@@ -25,6 +25,11 @@ class BitbucketProvider:
         if pr_url:
             self.set_pr(pr_url)
 
+    def is_supported(self, capability: str) -> bool:
+        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments']:
+            return False
+        return True
+
     def set_pr(self, pr_url: str):
         self.workspace_slug, self.repo_slug, self.pr_num = self._parse_pr_url(pr_url)
         self.pr = self._get_pr()
@@ -58,6 +63,12 @@ class BitbucketProvider:
     def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
         pass
 
+    def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
+        raise NotImplementedError("Bitbucket provider does not support creating inline comments yet")
+
+    def publish_inline_comments(self, comments: list[dict]):
+        raise NotImplementedError("Bitbucket provider does not support publishing inline comments yet")
+
     def get_title(self):
         return self.pr.title
 
@@ -74,6 +85,9 @@ class BitbucketProvider:
     def get_user_id(self):
         return 0
 
+    def get_issue_comments(self):
+        raise NotImplementedError("Bitbucket provider does not support issue comments yet")
+
     @staticmethod
     def _parse_pr_url(pr_url: str) -> Tuple[str, int]:
         parsed_url = urlparse(pr_url)

pr_agent/git_providers/git_provider.py

@@ -21,6 +21,10 @@ class FilePatchInfo:
 
 
 class GitProvider(ABC):
+    @abstractmethod
+    def is_supported(self, capability: str) -> bool:
+        pass
+
     @abstractmethod
     def get_diff_files(self) -> list[FilePatchInfo]:
         pass
@@ -37,6 +41,14 @@ class GitProvider(ABC):
     def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
         pass
 
+    @abstractmethod
+    def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
+        pass
+
+    @abstractmethod
+    def publish_inline_comments(self, comments: list[dict]):
+        pass
+
     @abstractmethod
     def publish_code_suggestion(self, body: str, relevant_file: str,
                                 relevant_lines_start: int, relevant_lines_end: int):
@@ -62,6 +74,10 @@ class GitProvider(ABC):
     def get_pr_description(self):
         pass
 
+    @abstractmethod
+    def get_issue_comments(self):
+        pass
+
 
 def get_main_pr_language(languages, files) -> str:
     """

pr_agent/git_providers/github_provider.py

@@ -3,7 +3,7 @@ from datetime import datetime
 from typing import Optional, Tuple
 from urllib.parse import urlparse
 
-from github import AppAuthentication, Github
+from github import AppAuthentication, Github, Auth
 
 from pr_agent.config_loader import settings
 
@@ -23,6 +23,9 @@ class GithubProvider(GitProvider):
             self.set_pr(pr_url)
             self.last_commit_id = list(self.pr.get_commits())[-1]
 
+    def is_supported(self, capability: str) -> bool:
+        return True
+
     def set_pr(self, pr_url: str):
         self.repo, self.pr_num = self._parse_pr_url(pr_url)
         self.pr = self._get_pr()
@@ -54,6 +57,9 @@ class GithubProvider(GitProvider):
         self.pr.comments_list.append(response)
 
     def publish_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
+        self.publish_inline_comments([self.create_inline_comment(body, relevant_file, relevant_line_in_file)])
+
+    def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
         self.diff_files = self.diff_files if self.diff_files else self.get_diff_files()
         position = -1
         for file in self.diff_files:
@@ -72,9 +78,16 @@ class GithubProvider(GitProvider):
         if position == -1:
             if settings.config.verbosity_level >= 2:
                 logging.info(f"Could not find position for {relevant_file} {relevant_line_in_file}")
+            subject_type = "FILE"
         else:
-            path = relevant_file.strip()
-            self.pr.create_review_comment(body=body, commit_id=self.last_commit_id, path=path, position=position)
+            subject_type = "LINE"
+        path = relevant_file.strip()
+        # placeholder for future API support (already supported in single inline comment)
+        # return dict(body=body, path=path, position=position, subject_type=subject_type)
+        return dict(body=body, path=path, position=position) if subject_type == "LINE" else {}
+
+    def publish_inline_comments(self, comments: list[dict]):
+        self.pr.create_review(commit=self.last_commit_id, comments=comments)
 
     def publish_code_suggestion(self, body: str,
                                 relevant_file: str,
@@ -161,6 +174,9 @@ class GithubProvider(GitProvider):
         notifications = self.github_client.get_user().get_notifications(since=since)
         return notifications
 
+    def get_issue_comments(self):
+        return self.pr.get_issue_comments()
+
     @staticmethod
     def _parse_pr_url(pr_url: str) -> Tuple[str, int]:
         parsed_url = urlparse(pr_url)
@@ -212,7 +228,7 @@ class GithubProvider(GitProvider):
                 raise ValueError(
                     "GitHub token is required when using user deployment. See: "
                     "https://github.com/Codium-ai/pr-agent#method-2-run-from-source") from e
-            return Github(token)
+            return Github(auth=Auth.Token(token))
 
     def _get_repo(self):
         return self.github_client.get_repo(self.repo)

pr_agent/git_providers/gitlab_provider.py

@@ -4,6 +4,7 @@ from typing import Optional, Tuple
 from urllib.parse import urlparse
 
 import gitlab
+from gitlab import GitlabGetError
 
 from pr_agent.config_loader import settings
 
@@ -31,6 +32,11 @@ class GitLabProvider(GitProvider):
         self.RE_HUNK_HEADER = re.compile(
             r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@[ ]?(.*)")
 
+    def is_supported(self, capability: str) -> bool:
+        if capability in ['get_issue_comments', 'create_inline_comment', 'publish_inline_comments']:
+            return False
+        return True
+
     @property
     def pr(self):
         '''The GitLab terminology is merge request (MR) instead of pull request (PR)'''
@@ -42,7 +48,11 @@ class GitLabProvider(GitProvider):
         self.last_diff = self.mr.diffs.list()[-1]
 
     def _get_pr_file_content(self, file_path: str, branch: str) -> str:
-        return self.gl.projects.get(self.id_project).files.get(file_path, branch).decode()
+        try:
+            return self.gl.projects.get(self.id_project).files.get(file_path, branch).decode()
+        except GitlabGetError:
+            # In case of file creation the method returns GitlabGetError (404 file not found). In this case we return an empty string for the diff.
+            return ''
 
     def get_diff_files(self) -> list[FilePatchInfo]:
         diffs = self.mr.changes()['changes']
@@ -58,8 +68,10 @@ class GitLabProvider(GitProvider):
             elif diff['renamed_file']:
                 edit_type = EDIT_TYPE.RENAMED
             try:
-                original_file_content_str = bytes.decode(original_file_content_str, 'utf-8')
-                new_file_content_str = bytes.decode(new_file_content_str, 'utf-8')
+                if isinstance(original_file_content_str, bytes):
+                    original_file_content_str = bytes.decode(original_file_content_str, 'utf-8')
+                if isinstance(new_file_content_str, bytes):
+                    new_file_content_str = bytes.decode(new_file_content_str, 'utf-8')
             except UnicodeDecodeError:
                 logging.warning(
                     f"Cannot decode file {diff['old_path']} or {diff['new_path']} in merge request {self.id_mr}")
@@ -89,6 +101,12 @@ class GitLabProvider(GitProvider):
         self.send_inline_comment(body, edit_type, found, relevant_file, relevant_line_in_file, source_line_no,
                                  target_file, target_line_no)
 
+    def create_inline_comment(self, body: str, relevant_file: str, relevant_line_in_file: str):
+        raise NotImplementedError("Gitlab provider does not support creating inline comments yet")
+
+    def create_inline_comment(self, comments: list[dict]):
+        raise NotImplementedError("Gitlab provider does not support publishing inline comments yet")
+
     def send_inline_comment(self, body, edit_type, found, relevant_file, relevant_line_in_file, source_line_no,
                             target_file, target_line_no):
         if not found:
@@ -123,26 +141,11 @@ class GitLabProvider(GitProvider):
         range = relevant_lines_end - relevant_lines_start + 1
         body = body.replace('```suggestion', f'```suggestion:-0+{range}')
 
-        d = self.last_diff
-        #
-        # pos_obj = {'position_type': 'text',
-        #            'new_path': target_file.filename,
-        #            'old_path': target_file.old_filename if target_file.old_filename else target_file.filename,
-        #            'base_sha': d.base_commit_sha, 'start_sha': d.start_commit_sha, 'head_sha': d.head_commit_sha}
         lines = target_file.head_file.splitlines()
         relevant_line_in_file = lines[relevant_lines_start - 1]
         edit_type, found, source_line_no, target_file, target_line_no = self.find_in_file(target_file, relevant_line_in_file)
         self.send_inline_comment(body, edit_type, found, relevant_file, relevant_line_in_file, source_line_no,
                                  target_file, target_line_no)
-        # if lines[relevant_lines_start][0] == '-':
-        #     pos_obj['old_line'] = relevant_lines_start
-        # elif lines[relevant_lines_start][0] == '+':
-        #     pos_obj['new_line'] = relevant_lines_start
-        # else:
-        #     pos_obj['new_line'] = relevant_lines_start
-        #     pos_obj['old_line'] = relevant_lines_start
-        # self.mr.discussions.create({'body': body,
-        #                             'position': pos_obj})
 
     def search_line(self, relevant_file, relevant_line_in_file):
         target_file = None
@@ -218,6 +221,9 @@ class GitLabProvider(GitProvider):
     def get_pr_description(self):
         return self.mr.description
 
+    def get_issue_comments(self):
+        raise NotImplementedError("GitLab provider does not support issue comments yet")
+
     def _parse_merge_request_url(self, merge_request_url: str) -> Tuple[int, int]:
         parsed_url = urlparse(merge_request_url)
 

pr_agent/servers/github_action_runner.py

@@ -3,9 +3,11 @@ import json
 import os
 import re
 
+from pr_agent.agent.pr_agent import PRAgent
 from pr_agent.config_loader import settings
 from pr_agent.tools.pr_code_suggestions import PRCodeSuggestions
 from pr_agent.tools.pr_description import PRDescription
+from pr_agent.tools.pr_information_from_user import PRInformationFromUser
 from pr_agent.tools.pr_questions import PRQuestions
 from pr_agent.tools.pr_reviewer import PRReviewer
 
@@ -53,20 +55,7 @@ async def run_action():
                 pr_url = event_payload.get("issue", {}).get("pull_request", {}).get("url", None)
                 if pr_url:
                     body = comment_body.strip().lower()
-                    if any(cmd in body for cmd in ["/review", "/review_pr"]):
-                        await PRReviewer(pr_url).review()
-                    elif any(cmd in body for cmd in ["/describe", "/describe_pr"]):
-                        await PRDescription(pr_url).describe()
-                    elif any(cmd in body for cmd in ["/improve", "/improve_code"]):
-                        await PRCodeSuggestions(pr_url).suggest()
-                    elif any(cmd in body for cmd in ["/ask", "/ask_question"]):
-                        pattern = r'(/ask|/ask_question)\s*(.*)'
-                        matches = re.findall(pattern, comment_body, re.IGNORECASE)
-                        if matches:
-                            question = matches[0][1]
-                            await PRQuestions(pr_url, question).answer()
-                    else:
-                        print(f"Unknown command: {body}")
+                    await PRAgent().handle_request(pr_url, body)
 
 
 if __name__ == '__main__':

pr_agent/settings/configuration.toml

@@ -1,8 +1,8 @@
 [config]
 model="gpt-4-0613"
 git_provider="github"
-publish_review=true
-verbosity_level=2 # 0,1,2
+publish_output=true
+verbosity_level=0 # 0,1,2
 
 [pr_reviewer]
 require_focused_review=true
@@ -10,6 +10,10 @@ require_tests_review=true
 require_security_review=true
 num_code_suggestions=3
 inline_code_comments = true
+ask_and_reflect=false
+
+[pr_description]
+publish_description_as_comment=false
 
 [pr_questions]
 

pr_agent/settings/pr_information_from_user_prompts.toml

@@ -1,16 +1,17 @@
 [pr_information_from_user_prompt]
 system="""You are CodiumAI-PR-Reviewer, a language model designed to review git pull requests.
-Given the PR Info and the PR Git Diff, generate 4 questions about the PR for the PR author.
+Given the PR Info and the PR Git Diff, generate 3 short questions about the PR code for the PR author.
 The goal of the questions is to help the language model understand the PR better, so the questions should be insightful, informative, non-trivial, and relevant to the PR.
-Prefer yes\\no or multiple choice questions. If you have to ask open-ended questions, make sure they are not too difficult, and can be answered in a sentence or two.
+You should prefer asking yes\\no questions, or multiple choice questions. Also add at least one open-ended question, but make sure they are not too difficult, and can be answered in a sentence or two.
 
 
 Example output:
 '
 Questions to better understand the PR:
-1. ...
-2. ...
+1) ...
+2) ...
 ...
+'
 """
 
 user="""PR Info:

pr_agent/settings/pr_reviewer_prompts.toml

@@ -2,8 +2,11 @@
 system="""You are CodiumAI-PR-Reviewer, a language model designed to review git pull requests.
 Your task is to provide constructive and concise feedback for the PR, and also provide meaningfull code suggestions to improve the new PR code (the '+' lines).
 - Provide up to {{ num_code_suggestions }} code suggestions.
+{%- if num_code_suggestions > 0 %}
 - Try to focus on important suggestions like fixing code problems, issues and bugs. As a second priority, provide suggestions for meaningfull code improvements, like performance, vulnerability, modularity, and best practices.
+- Suggestions should focus on improving the new added code lines.
 - Make sure not to provide suggestions repeating modifications already implemented in the new PR code (the '+' lines).
+{%- endif %}
 
 You must use the following JSON schema to format your answer:
 ```json
@@ -23,6 +26,12 @@ You must use the following JSON schema to format your answer:
       "description": "yes\\no question: does this PR have relevant tests ?"
     },
 {%- endif %}
+{%- if question_str %}
+    "Insights from user's answer": {
+      "type": "string",
+      "description": "shortly summarize the insights you gained from the user's answers to the questions"
+    },
+{%- endif %}
 {%- if require_focused %}
     "Focused PR": {
       "type": "string",
@@ -35,6 +44,7 @@ You must use the following JSON schema to format your answer:
       "type": "string",
       "description": "General suggestions and feedback for the contributors and maintainers of this PR. May include important suggestions for the overall structure, primary purpose, best practices, critical bugs, and other aspects of the PR. Explain your suggestions."
     },
+{%- if num_code_suggestions > 0 %}
     "Code suggestions": {
       "type": "array",
       "maxItems": {{ num_code_suggestions }},
@@ -54,6 +64,7 @@ You must use the following JSON schema to format your answer:
         }
       }
     },
+{%- endif %}
 {%- if require_security %}
     "Security concerns": {
       "type": "string",
@@ -82,6 +93,7 @@ Example output:
     "PR Feedback":
     {
         "General PR suggestions": "..., `xxx`...",
+{%- if num_code_suggestions > 0 %}
         "Code suggestions": [
             {
                 "relevant file": "directory/xxx.py",
@@ -90,6 +102,7 @@ Example output:
             },
             ...
         ]
+{%- endif %}
 {%- if require_security %},
        "Security concerns": "No, because ..."
 {%- endif %}
@@ -108,6 +121,16 @@ Description: '{{description}}'
 Main language: {{language}}
 {%- endif %}
 
+{%- if question_str %}
+######
+Here are questions to better understand the PR. Use the answers to provide better feedback.
+
+{{question_str|trim}}
+
+User answers:
+{{answer_str|trim}}
+######
+{%- endif %}
 
 The PR Git Diff:
 ```

pr_agent/tools/pr_code_suggestions.py

@@ -42,7 +42,7 @@ class PRCodeSuggestions:
         assert type(self.git_provider) != BitbucketProvider, "Bitbucket is not supported for now"
 
         logging.info('Generating code suggestions for PR...')
-        if settings.config.publish_review:
+        if settings.config.publish_output:
             self.git_provider.publish_comment("Preparing review...", is_temporary=True)
         logging.info('Getting PR diff...')
 
@@ -56,7 +56,7 @@ class PRCodeSuggestions:
         self.prediction = await self._get_prediction()
         logging.info('Preparing PR review...')
         data = self._prepare_pr_code_suggestions()
-        if settings.config.publish_review:
+        if settings.config.publish_output:
             logging.info('Pushing PR review...')
             self.git_provider.remove_initial_comment()
             logging.info('Pushing inline code comments...')

pr_agent/tools/pr_description.py

@@ -36,17 +36,20 @@ class PRDescription:
 
     async def describe(self):
         logging.info('Generating a PR description...')
-        if settings.config.publish_review:
+        if settings.config.publish_output:
             self.git_provider.publish_comment("Preparing pr description...", is_temporary=True)
         logging.info('Getting PR diff...')
         self.patches_diff = get_pr_diff(self.git_provider, self.token_handler)
         logging.info('Getting AI prediction...')
         self.prediction = await self._get_prediction()
         logging.info('Preparing answer...')
-        pr_title, pr_body = self._prepare_pr_answer()
-        if settings.config.publish_review:
+        pr_title, pr_body, markdown_text = self._prepare_pr_answer()
+        if settings.config.publish_output:
             logging.info('Pushing answer...')
-            self.git_provider.publish_description(pr_title, pr_body)
+            if settings.pr_description.publish_description_as_comment:
+                self.git_provider.publish_comment(markdown_text)
+            else:
+                self.git_provider.publish_description(pr_title, pr_body)
             self.git_provider.remove_initial_comment()
         return ""
 
@@ -66,10 +69,11 @@ class PRDescription:
 
     def _prepare_pr_answer(self):
         data = json.loads(self.prediction)
+        markdown_text = ""
+        for key, value in data.items():
+            markdown_text += f"## {key}\n\n"
+            markdown_text += f"{value}\n\n"
         pr_body = ""
-        # for key, value in data.items():
-        #     markdown_text += f"## {key}\n\n"
-        #     markdown_text += f"{value}\n\n"
         title = data['PR Title']
         del data['PR Title']
         for key, value in data.items():
@@ -80,4 +84,4 @@ class PRDescription:
                 pr_body += f"**{value}**\n\n___\n"
         if settings.config.verbosity_level >= 2:
             logging.info(f"title:\n{title}\n{pr_body}")
-        return title, pr_body
+        return title, pr_body, markdown_text

pr_agent/tools/pr_information_from_user.py

@@ -21,7 +21,7 @@ class PRInformationFromUser:
         self.vars = {
             "title": self.git_provider.pr.title,
             "branch": self.git_provider.get_pr_branch(),
-            "description": self.git_provider.get_description(),
+            "description": self.git_provider.get_pr_description(),
             "language": self.main_pr_language,
             "diff": "",  # empty diff for initial calculation
         }
@@ -34,15 +34,15 @@ class PRInformationFromUser:
 
     async def generate_questions(self):
         logging.info('Generating question to the user...')
-        if settings.config.publish_review:
-            self.git_provider.publish_comment("Preparing answer...", is_temporary=True)
+        if settings.config.publish_output:
+            self.git_provider.publish_comment("Preparing questions...", is_temporary=True)
         logging.info('Getting PR diff...')
         self.patches_diff = get_pr_diff(self.git_provider, self.token_handler)
         logging.info('Getting AI prediction...')
         self.prediction = await self._get_prediction()
         logging.info('Preparing questions...')
         pr_comment = self._prepare_pr_answer()
-        if settings.config.publish_review:
+        if settings.config.publish_output:
             logging.info('Pushing questions...')
             self.git_provider.publish_comment(pr_comment)
             self.git_provider.remove_initial_comment()
@@ -66,6 +66,6 @@ class PRInformationFromUser:
         model_output = self.prediction.strip()
         if settings.config.verbosity_level >= 2:
             logging.info(f"answer_str:\n{model_output}")
-        answer_str = f"{model_output}\n\n Please respond to the question above in the following format:\n\n" + \
-                     f"/answer <question_id> <answer>\n\n" + f"Example:\n'\n/answer\n1. Yes, because ...\n2. No, because ...\n'"
+        answer_str = f"{model_output}\n\n Please respond to the questions above in the following format:\n\n" +\
+                     f"\n>/answer\n>1) ...\n>2) ...\n>...\n"
         return answer_str

pr_agent/tools/pr_questions.py

@@ -36,7 +36,7 @@ class PRQuestions:
 
     async def answer(self):
         logging.info('Answering a PR question...')
-        if settings.config.publish_review:
+        if settings.config.publish_output:
             self.git_provider.publish_comment("Preparing answer...", is_temporary=True)
         logging.info('Getting PR diff...')
         self.patches_diff = get_pr_diff(self.git_provider, self.token_handler)
@@ -44,7 +44,7 @@ class PRQuestions:
         self.prediction = await self._get_prediction()
         logging.info('Preparing answer...')
         pr_comment = self._prepare_pr_answer()
-        if settings.config.publish_review:
+        if settings.config.publish_output:
             logging.info('Pushing answer...')
             self.git_provider.publish_comment(pr_comment)
             self.git_provider.remove_initial_comment()

pr_agent/tools/pr_reviewer.py

@@ -11,16 +11,20 @@ from pr_agent.algo.utils import convert_to_markdown, try_fix_json
 from pr_agent.config_loader import settings
 from pr_agent.git_providers import get_git_provider
 from pr_agent.git_providers.git_provider import get_main_pr_language
-from pr_agent.servers.help import bot_help_text, actions_help_text
+from pr_agent.servers.help import actions_help_text, bot_help_text
 
 
 class PRReviewer:
-    def __init__(self, pr_url: str, cli_mode=False):
+    def __init__(self, pr_url: str, cli_mode=False, is_answer: bool = False):
 
         self.git_provider = get_git_provider()(pr_url)
         self.main_language = get_main_pr_language(
             self.git_provider.get_languages(), self.git_provider.get_files()
         )
+        self.is_answer = is_answer
+        if self.is_answer and not self.git_provider.is_supported("get_issue_comments"):
+            raise Exception(f"Answer mode is not supported for {settings.config.git_provider} for now")
+        answer_str = question_str = self._get_user_answers()
         self.ai_handler = AiHandler()
         self.patches_diff = None
         self.prediction = None
@@ -35,6 +39,9 @@ class PRReviewer:
             "require_security": settings.pr_reviewer.require_security_review,
             "require_focused": settings.pr_reviewer.require_focused_review,
             'num_code_suggestions': settings.pr_reviewer.num_code_suggestions,
+            #
+            'question_str': question_str,
+            'answer_str': answer_str,
         }
         self.token_handler = TokenHandler(self.git_provider.pr,
                                           self.vars,
@@ -43,15 +50,15 @@ class PRReviewer:
 
     async def review(self):
         logging.info('Reviewing PR...')
-        if settings.config.publish_review:
-                self.git_provider.publish_comment("Preparing review...", is_temporary=True)
+        if settings.config.publish_output:
+            self.git_provider.publish_comment("Preparing review...", is_temporary=True)
         logging.info('Getting PR diff...')
         self.patches_diff = get_pr_diff(self.git_provider, self.token_handler)
         logging.info('Getting AI prediction...')
         self.prediction = await self._get_prediction()
         logging.info('Preparing PR review...')
         pr_comment = self._prepare_pr_review()
-        if settings.config.publish_review:
+        if settings.config.publish_output:
             logging.info('Pushing PR review...')
             self.git_provider.publish_comment(pr_comment)
             self.git_provider.remove_initial_comment()
@@ -89,8 +96,16 @@ class PRReviewer:
                 del data['PR Feedback']['Security concerns']
                 data['PR Analysis']['Security concerns'] = val
 
-        if settings.config.git_provider == 'github' and settings.pr_reviewer.inline_code_comments:
-            del data['PR Feedback']['Code suggestions']
+        if settings.config.git_provider == 'github' and \
+                settings.pr_reviewer.inline_code_comments and \
+                'Code suggestions' in data['PR Feedback']:
+            # keeping only code suggestions that can't be submitted as inline comments
+            data['PR Feedback']['Code suggestions'] = [
+                d for d in data['PR Feedback']['Code suggestions']
+                if any(key not in d for key in ('relevant file', 'relevant line in file', 'suggestion content'))
+            ]
+            if not data['PR Feedback']['Code suggestions']:
+                del data['PR Feedback']['Code suggestions']
 
         markdown_text = convert_to_markdown(data)
         user = self.git_provider.get_user_id()
@@ -107,15 +122,43 @@ class PRReviewer:
         return markdown_text
 
     def _publish_inline_code_comments(self):
+        if settings.pr_reviewer.num_code_suggestions == 0:
+            return
+
         review = self.prediction.strip()
         try:
             data = json.loads(review)
         except json.decoder.JSONDecodeError:
             data = try_fix_json(review)
 
+        comments = []
         for d in data['PR Feedback']['Code suggestions']:
-            relevant_file = d['relevant file'].strip()
-            relevant_line_in_file = d['relevant line in file'].strip()
-            content = d['suggestion content']
+            relevant_file = d.get('relevant file', '').strip()
+            relevant_line_in_file = d.get('relevant line in file', '').strip()
+            content = d.get('suggestion content', '')
+            if not relevant_file or not relevant_line_in_file or not content:
+                logging.info("Skipping inline comment with missing file/line/content")
+                continue
 
-            self.git_provider.publish_inline_comment(content, relevant_file, relevant_line_in_file)
+            if self.git_provider.is_supported("create_inline_comment"):
+                comment = self.git_provider.create_inline_comment(content, relevant_file, relevant_line_in_file)
+                if comment:
+                    comments.append(comment)
+            else:
+                self.git_provider.publish_inline_comment(content, relevant_file, relevant_line_in_file)
+
+        if comments:
+            self.git_provider.publish_inline_comments(comments)
+
+    def _get_user_answers(self):
+        answer_str = question_str = ""
+        if self.is_answer:
+            discussion_messages = self.git_provider.get_issue_comments()
+            for message in discussion_messages.reversed:
+                if "Questions to better understand the PR:" in message.body:
+                    question_str = message.body
+                elif '/answer' in message.body:
+                    answer_str = message.body
+                if answer_str and question_str:
+                    break
+        return question_str, answer_str

requirements.txt

@@ -1,6 +1,6 @@
 dynaconf==3.1.12
 fastapi==0.99.0
-PyGithub==1.58.2
+PyGithub==1.59.*
 retry==0.9.2
 openai==0.27.8
 Jinja2==3.1.2