We employed the Copilot AI and inquired about user experiences; the results were promising. Nine out of ten members of Crasman's Creative Nerds wish to continue using it. The increase in productivity and improved quality also serve our clients.

7th March 2023 was the day when GitHub Copilot for Business AI became part of every developer's toolkit at Crasman. Copilot is an AI-based virtual “programming companion” that makes suggestions and assists developers in writing program code faster.

The Copilot, developed by OpenAI, integrates into the developer's code editor and is a generative pre-trained OpenAI Codex language model. Let us clarify from the outset that GitHub does not store data processed by Copilot for Business, nor is it used for AI training.

About a month after the Copilot trial, we conducted an anonymous survey among our developers to gather experiences with the AI and its impact on developers' daily work.

In general, the results of the AI survey reinforce the notion that AI is a fundamental change in developers' daily lives, and there is no turning back. Of the developers who responded to the survey, 68% use Copilot daily.

The amount of interesting and high-quality work has increased

• 75% of respondents feel more productive

• 69% of respondents are able to focus more on interesting work

• 41% of respondents believe the quality of their programming work has improved

• Overall, Copilot has reduced frustration and increased job satisfaction

And perhaps the most essential question of all - do you want to continue using Copilot?

What problems or concerns arose during the Copilot trial?

The AI based on the language model is not infallible, and sometimes the code suggestions made by Copilot do not meet the needs.

We also noticed that Copilot works better with some programming languages than others. For example, Typescript flows much more smoothly than PHP, where code suggestions often contained errors or were non-functional.

A developer must always critically evaluate Copilot’s code suggestions. In this respect, nothing has changed from before, as the developer is responsible for the code produced, even if it is aided by Copilot.

Copilot is currently only available for a limited number of code editors, such as Visual Studio Code. If you are accustomed to using a specific code editor, it is possible Copilot may not be available for it.

Copyright issues related to program code have raised concerns in the international programming community, as Copilot has been trained with source code available from public sources.

Developers may find it difficult to be certain whether a code snippet suggested by Copilot is license-free or if there is a risk it may also include licensable parts.

• GitHub has responded to the concern by increasing Copilot’s transparency. Copilot will indicate how many times the suggested code snippet appears in GitHub's code repositories, helping the developer assess the snippet's validity.

• Copilot's generated code suggestions can be restricted so that it does not present suggestions that are in the same form found in multiple public GitHub code repositories and are thereby most likely under an open source license.

• Short code snippets are not automatically protected by copyright; they need to be sufficiently independent and original to receive copyright protection (known as the threshold of originality).

Copilot accelerated work by up to half

Based on these results, it is clear that the majority of developers have welcomed the Copilot AI with open arms, and it has increased job satisfaction. One developer mentioned in open survey feedback that they would be on the barricades if Copilot use didn't continue after the pilot phase.

Copyright matters require precision, but the same issues arise when code examples are picked from programming discussion forums to support one's work.

GitHub has also researched the impact of Copilot on productivity and developer satisfaction. In the study, productivity growth was measured by giving the same programming task to developers divided into two groups.

Developers in the Copilot-using group completed the task on average 55% faster than those in the control group. The results of GitHub's study were well aligned with the feedback gathered from Crasman's developer community. 

The growth in productivity and improved quality are factors that also serve our clients. Let us not forget the added value that arises from a satisfied and smiling creative nerd.

Crasman Insight monitors the advent and impact of AI on digital commerce. Explore previously published insights.

Crasman's CTO Mikko Tikkanen is a challenger of technologies, tools, and processes and a continuous developer of business operations.