xiaxinhang/pr-agent
1
0
Fork 0
mirror of https://github.com/qodo-ai/pr-agent.git synced 2025-07-09 07:10:37 +08:00
Code Issues Packages Projects Releases Wiki Activity

Qodo Merge rename

Browse Source
This commit is contained in:
mrT23
2024-09-29 17:15:49 +03:00
parent b4c20d683c
commit c25aaad176
37 changed files with 244 additions and 244 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/docs/finetuning_benchmark/index.md
View File

@ -1,10 +1,10 @@
# PR-Agent Code Fine-tuning Benchmark
# Qodo Merge Code Fine-tuning Benchmark
On coding tasks, the gap between open-source models and top closed-source models such as GPT4 is significant.
<br>
In practice, open-source models are unsuitable for most real-world code tasks, and require further fine-tuning to produce acceptable results.
_PR-Agent fine-tuning benchmark_ aims to benchmark open-source models on their ability to be fine-tuned for a coding task.
_Qodo Merge fine-tuning benchmark_ aims to benchmark open-source models on their ability to be fine-tuned for a coding task.
Specifically, we chose to fine-tune open-source models on the task of analyzing a pull request, and providing useful feedback and code suggestions.
Here are the results:
@ -53,8 +53,8 @@ Here are the results:
### Training dataset
Our training dataset comprises 25,000 pull requests, aggregated from permissive license repos. For each pull request, we generated responses for the three main tools of PR-Agent:
[Describe](https://pr-agent-docs.codium.ai/tools/describe/), [Review](https://pr-agent-docs.codium.ai/tools/improve/) and [Improve](https://pr-agent-docs.codium.ai/tools/improve/).
Our training dataset comprises 25,000 pull requests, aggregated from permissive license repos. For each pull request, we generated responses for the three main tools of Qodo Merge:
[Describe](https://qodo-merge-docs.qodo.ai/tools/describe/), [Review](https://qodo-merge-docs.qodo.ai/tools/improve/) and [Improve](https://qodo-merge-docs.qodo.ai/tools/improve/).
On the raw data collected, we employed various automatic and manual cleaning techniques to ensure the outputs were of the highest quality, and suitable for instruct-tuning.