AI Assistants for Software Engineers: A Deep Dive into GitHub Copilot
There's been a lot of buzz around AI, but many questions remain unanswered. - When will general AGI be developed? - Will it replace as many jobs as initially predicted? - Is it all just a gimmick?
This newsletter focuses on software engineering, so we'll concentrate on that aspect. Specifically, we'll discuss how AI assistants, or 'Copilots,’ can potentially aid software engineers. Can they replace them? Can they enhance productivity, or are they just a time-waster?
We'll explore various available tools and conclude with a summary for managers to decide which tool to use. Bear in mind that this field evolves rapidly, so the information provided may become outdated in a few months. However, it should offer a solid understanding of the distinct differences between the tools.
GitHub Copilot
Let's begin with the most significant example - GitHub Copilot. GitHub was acquired by Microsoft in 2018, and as we all know, Microsoft is a primary investor in OpenAI. This gives GitHub first-class access to OpenAI's models, a significant advantage and a head start. GitHub was among the first to announce a copilot for software engineers, showcasing GitHub Copilot X at the Web Summit in Rio 2023.
You might be wondering about Copilot and Copilot X. Copilot was launched in October 2021, with small models running behind the scenes. It initially supported autocomplete and some minor functions, but its functionality gradually expanded.
With the hype surrounding GPT 3 and 4, a demo of Copilot X was shown to tease its potential future capabilities. However, the release deadlines were unclear, and it was uncertain how much of the product was ready for market. Thus, Copilot X was primarily a marketing campaign, but its purpose was well-served. Most people associate Code AI with GitHub Copilot and overlook other Code AI vendors like Code Llama, Cody, and Cursor. It's impressive marketing, as I often read comments where people associate certain Copilot features with GitHub exclusively, which is usually not the case.
Despite this, GitHub does have a significant advantage through its cooperation with Microsoft and OpenAI. Copilot incorporated some features of Copilot X for the general public. Currently, there is no separate Copilot X, just Copilot. They also launched several models, such as Copilot Individual for indie developers and freelancers, at $10 per month, and Copilot Business, which has enhanced security and privacy features at $19 per month. This month, we should see the introduction of Copilot Enterprise at $39 per month per user. The pricing has changed significantly, making it quite expensive for medium to large organizations.
So, is it worth the cost?
As this blog is designed for Engineering Managers and decision-makers in organizations, the following section will delve into this question in more detail.
Where Does It Excel?
Setting up GitHub Copilot is simple, especially if you're already a GitHub user. Choosing between the different plans is also straightforward.
Its autocomplete feature, which it was initially designed for, is among the best on the market. However, it's important to remember that it's not magic. Its context is limited, and for autocomplete, no coding AI uses something as costly as GPT 3 or 4. Most AI vendors opt for cost-effective models to generate autocompletes for two simple reasons:
Autocompletes need to be generated quickly to keep up with your fast typing.
A lot of autocompletes are created when you write a lot of code.
They began with OpenAI's Codex and announced the introduction of a more intelligent model by the end of 2023.
Within the Visual Studio Code IDE, the plugin is user-friendly, and its chat functionality matches that of its competitors. I sometimes find it faster and equipped with a smarter context window.
What are the issues?
Finding technical details about Copilot is challenging unless you're an insider at GitHub or a significant enterprise customer who is privy to shared information. Which model is it using for what use case? When will specific features become available? GitHub often showcases impressive features but takes considerable time to make them accessible to the general public. This approach works for them but leaves customers uncertain about what they're getting and when.
Understanding the currently available features and upcoming developments on the GitHub Copilot page (https://github.com/features/copilot) can be time-consuming due to frequent updates and phased rollouts. These staggered releases are likely due to the significant computational power required for such features. As AI applications are cost-intensive, GitHub appears to be rolling out Copilot features cautiously to maintain a positive user experience. It would be helpful if they provided more insight into their overall strategy and plans.
Dependency on OpenAI is also problematic, as seen in the OpenAI Board meltdown in 2023. Sam Altman was fired as CEO but reinstated due to pressure from investors and Microsoft. During this period, the future was uncertain as employees considered leaving OpenAI. Now, all assets belong to Microsoft, which means trusting one large monopoly with your data (Office 365), your code (GitHub), and your AI (Copilot).
If you manage a larger company, you may realize that your code is either not hosted on GitHub or only partially. This issue is a concern, and it's unclear how extensively GitHub addresses it. Therefore, if you're not entirely on GitHub, you might need to consider other providers.
Personal Experience
I tested GitHub Copilot in four scenarios to evaluate its usefulness in everyday work.
Writing code, tests, and using autocomplete
Understanding a part of a codebase or snippet, for example, to fix a bug.
Creating documentation, commit messages, and pull requests
Assisting as a mentor to create a new project or discuss architecture
Autocomplete (code and tests)
While GitHub Copilot's autocomplete feature is generally effective, it can sometimes suggest incorrect information. It performs no checks so that it might repeat an erroneous input. The context of autocomplete is relatively limited.
Help me fix a bug
GitHub Copilot assists with bug fixes, as tested on an open-source project. However, understanding its intentions and writing the proper prompts requires developer skills. It's a tool that can enhance a good developer's abilities, enabling them to navigate unknown code segments quickly.
Create documentation, commit messages, or pull requests
Most code AIs excel in these areas, and GitHub Copilot is no exception. Its effectiveness depends on the commands from the AI vendor and the prompts used in the chat window. In the future, a software engineer who doesn't utilize AI for crafting commit messages might be deemed less effective. However, always double-check the AI's output as it's unreliable.
Create a new project
GitHub Copilot faces challenges when creating a new project due to the inherent limitations of current Language Models (LLMs). Most LLMs have a "knowledge cut,” meaning they're unaware of updates or changes in frameworks like React or Angular after a certain point (around 2021-2023). This issue requires a skilled engineer to spot and fix. For simple projects, GitHub Copilot can be a helpful tool. For more complex tasks, it can serve as a supplemental resource alongside platforms like StackOverflow, Google, and Reddit, but frequent double-checking is necessary. The marketing for Copilot X set high expectations, which can lead to disappointment and adverse reactions when users encounter these limitations.
Takeaways
Code Language Models (LLMs) are powerful tools, but they need to be used correctly. Understanding their limitations and the technical reasons behind them is crucial.
For managers, these tools can be beneficial for your organization, though finding Key Performance Indicators (KPIs) for them may be challenging. If you already have productivity metrics in place (both quantitative and qualitative), you might notice areas of improvement. Currently, these tools are more helpful for experienced software engineers who can quickly identify their limitations and use them to their advantage. Junior engineers, while they may be intrigued by the capabilities of AI, could be hindered by a lack of knowledge.
If your entire software engineering department uses GitHub, GitHub Copilot is a logical choice. It's easy to implement and doesn't require the introduction of another tool. For those seeking platform-independent Code LLMs, look out for my upcoming reviews.