Copilot pull request summaries is an AI-powered feature that allows you to create a summary of the changes that were made in a pull request, which files they impact, and what a reviewer should focus on when they conduct their review.
When a user requests a summary, Copilot scans through the pull request and provides an overview of the changes made in prose, as well as a bulleted list of changes with the files that they impact.
Copilot pull request summaries uses a simple-prompt flow leveraging the Copilot API, with no additional trained models. This utilizes the generic large language model.
The current process uses GPT 4 to initiate the auto-complete process and generate the pull request summary.
When a user requests a summary, a workflow is triggered. The workflow uses the code diffs to build a prompt call, which requests Copilot to generate a summary of the pull request. The summary request initiates a pipeline process which includes raw diffs from summarizable files in a prompt and requests Copilot to generate an overall summary for the pull request.
You can initiate this feature when creating a pull request, by editing the pull request description after creation, or in a comment in the pull request thread. Upon initiation, Copilot will generate a two part summary:
- A paragraph, written in prose, giving an overview of the changes in the pull request.
- A bulleted list of the key changes, linked to the respective lines of code where those changes occur.
Larger pull requests can take a couple minutes for Copilot to generate. You can share your feedback on a summary directly from the UI after a summary is generated to help us continue to improve the feature.
The goal of Copilot pull request summaries is to help optimize an author's ability to quickly provide context when they request a human review that requires sharing context of the changes that were made. It may help increase developer productivity by reducing the time taken to open a pull request.
For many users, it could provide more helpful context for the changes that were made within a pull request than would normally be available.
The feature is intended to supplement rather than replace a human's work to add context, and we encourage you to continue adding useful context and let Copilot do the busy work of parsing the code and linking to specific files. It remains your responsibility to review and assess the accuracy of information in a pull request that you create.
This feature is currently in beta. If you encounter any issues or limitations with Copilot pull request summaries, we recommend that you provide feedback through the link that appears the UI after a summary is generated. You can provide feedback through the text link which takes you to our survey.
Currently, our team is aware that there are limitations to this feature. Many of them are expected in leveraging our Copilot API; however, there are a few that are specific to Copilot pull request summaries which pertain to limited scope, longer processing times, and inaccurate responses. We also note that users should expect terms used in their PR to appear in the AI-generated summary. This feature has been subject to RAI Red Teaming and we will continue to monitor the efficacy and safety of the feature over time. For more information, see Microsoft AI Red Team building future of safer AI on the Microsoft security blog.
Because of capacity, we know that larger pull requests that reference 30 or more files will require more time to be processed thoroughly. We don't have an exact threshold currently, but have observed the first 30 files being accounted for and then any additional files being omitted from the summarization. We are working to address this current scope limitation.
In general, we expect a summary to be returned in 40 seconds or less after a user initiates the action. However, we have heard that this can take up to a minute, and in some cases a couple of minutes. We are working to decrease processing time and we know that users may not want to wait for this to finish before moving on to other parts of the pull request.
The more inputs and context provided to Copilot, the better the outputs will be. However, since the feature is quite new, it will take time to reach exact precision with the summaries that are generated. In the meantime, there may be cases where a user's generated summary is less accurate and requires the user to make modifications before saving and publishing their pull request with this description. Reviewing is a requirement, and careful review of the output is highly recommended by our team during the beta.
Because a summary is an outline of the changes that were made in a pull request, if harmful or offensive terms are within the content of the pull request, there is potential for the summary to also include those terms.