AI-powered coding tools are becoming more commonplace within developer workflows. GitHub Copilot is a popular AI coding assistant that can be integrated directly into IDEs or as a standalone chat interface. This tool helps you write code faster and with less effort by auto-completing code in real time, generating blocks of code from natural language prompts, and answering your questions to help you get over coding hurdles and roadblocks. Your organization will need to monitor Copilot usage so you can make informed decisions when adjusting your AI spend based on how Copilot impacts your developer workflows.
Datadog’s GitHub Copilot integration enables you to visualize Copilot metrics so you can track Copilot usage across your engineering organization. In this post, we’ll discuss how to use Datadog to gain insights into your organization’s Copilot adoption to help manage your assigned licenses as well as identify where Copilot creates the greatest impact.
Manage Copilot by tracking license distribution and user engagement
Depending on your use case, your organization is likely subscribing to either GitHub Copilot’s Business plan or the Enterprise plan—both of which charge based on seats (i.e., licenses) per billing cycle. Using the preconfigured Copilot dashboard in Datadog, you gain visibility into the number of active seats or users that have engaged with any Copilot feature each billing cycle, as well as the number of inactive seats that you’re currently paying for. These metrics enable you to adjust your cloud spend on Copilot licenses after first reminding inactive users that Copilot is an available tool for them and providing them steps to get started.
Our dashboard also helps you track how developers engage with Copilot. You can visualize the total number of suggestions against accepted suggestions and the average number of lines your developers use per acceptance across facets such as programming language and IDE. This can help you identify usage trends—for example, if Copilot usage and acceptance is high among developers that use Visual Studio Code. Copilot enables developers to ask questions directly within their IDE—using the dashboard, you can gain insights into how your developers take advantage of this feature and how often it’s used to insert or copy code.
Identify teams and coding languages that have high compatibility with AI assistance
After adopting Copilot as a development tool, you may begin to notice that its impact varies from team to team and project to project. Identifying where Copilot creates the greatest impact can help you understand what other projects Copilot can assist with, as well as teams where you may be able to introduce Copilot as a long-lasting development tool, starting from the onboarding period.
Copilot is able to suggest certain patterns of code that align particularly well with your organization’s coding practices. For instance, if you notice that your site reliability engineering team has had the most number of accepted Copilot events, it may indicate that Copilot is adept at assisting your SREs with writing IaC code or generating scripts to automate deployment workflows. Similarly, by using the language breakdown section of the GitHub Copilot dashboard, you can gain granular visibility into how Copilot accelerates development across different languages. In the example shown below, the dashboard shows that the vast majority of suggested events by developers have been in Python. Since this has proven to be effective, you might try expanding Copilot usage to other teams responsible for Python services for use cases such as writing boilerplate code, automating scripting tasks, or generating first drafts for the documentation of new features.
It’s important to bear in mind that the lump sum of accepted-suggestion events does not necessarily equal large improvements to developer velocity. For instance, in the example above, the average lines per suggestion for Go was 36 lines of code—however, developers are accepting on average only three lines of code. This suggests that an overwhelming majority of the code Copilot is suggesting for Go is being rejected, while developers are accepting only a handful of suggested code. Taking this context into account, you may want to re-evaluate how effective Copilot really is among developers that use it for Go services.
Get started with Datadog
Datadog’s Copilot integration can help you monitor usage metrics that we’ve discussed in this post, as well as other key metrics. You can learn more about our Copilot integration in our documentation. If you’re interested in reading more about new AI features Datadog supports, check out these blog posts.
If you don’t already have a Datadog account, sign up for a free 14-day trial today.
