Analysis of Possible Applications of AI in Code Refactoring

1. Introduction

Artificial intelligence (AI) is playing an increasingly important role in software development. AI-supported systems, so-called coding agents, have developed from conceptual models into practical tools that aim to increase efficiency in development and maintenance tasks. At the same time, the modernisation of legacy systems is a central challenge. These systems are often characterised by outdated and possibly no longer maintained dependencies, accumulated technical debt or a technology stack that is no longer up to date. The aim of this work is to systematically investigate the potential of modern AI coding assistants for the semi-automated modernisation of such legacy systems. The existing application pdf-as 4 serves as the specific object of investigation.

For the evaluation, a comparative analysis of four selected AI agents in their respective development environments (IDEs) is carried out: Junie and Sweep AI in the IntelliJ IDE, GitHub Copilot in Visual Studio Code, and the integrated assistant of the Kiro development environment. We look at their mode of operation, technical foundations and suitability for practical use in modernising existing applications.

2. Background

The use of AI coding agents to support software development tasks is based on the integration of Large Language Models (LLMs) into development environments. These are AI models that have been trained on vast amounts of (source) texts and, through their processing of context, are able not only to generate code but also to understand, refactor and correct it. Modern tools such as GitHub Copilot, Junie, Sweep AI or Kiro use different LLMs to provide various functions, ranging from autocompletion and coding support to autonomous changes.

2.1. LLM-based assistance: architecture and principles

LLMs such as GPT-4.1 or Claude Sonnet work purely sequentially on input prompts. Depending on their architecture and model size, they have different capabilities for information processing, task planning and self-reflection. The following components are central to coding agents:

• Context window: LLMs can only consider a limited amount of code at once. Claude Sonnet 3.7 and 4.0 offer up to 200k tokens of context, GPT-4.1 in the Copilot version up to 128k tokens. A larger context size allows the model to analyse larger projects or entire files at once.

• Tool-use capability: Agent systems such as Junie and Kiro can explicitly call tools, for example to start test runs or make code changes. Terminal applications such as find or git can also be called.

• Planning and self-direction: While classic LLMs such as GPT-3 deliver simple responses to inputs, modern coding agents such as Kiro or Junie build on “multistep reasoning”: they plan changes, create task lists and carry them out one after the other.

2.2. Types of coding assistants

Coding assistants can be roughly divided into three classes:

• Autocompleters: Classic tools such as Copilot (in inline mode) or Tabnine suggest individual lines or blocks of code based on prompts or preceding code. Because of their lower complexity, autocompleters can often also be run locally on the user’s machine, as is the case, for example, in JetBrains development environments [1].

• Conversational assistants: ChatGPT, Copilot Chat or the AI Chat in JetBrains IDEs are dialogue-based and support developers with more complex questions, although usually without direct execution. Some tools, such as JetBrains’ AI Chat, nevertheless help users to carry out the changes with the help of functions of the development environment, or can also change simple code snippets on their own.

• Agent systems: Junie, Sweep AI, Kiro or GitHub Copilot in agent mode carry out multi-step operations independently. They analyse code, create change plans, implement them, validate them through tests and can work with external tools such as the version control system in use.

Effectiveness depends heavily on the integration into the development environment. Sweep and Junie are deeply integrated into IntelliJ, while Kiro is based on a fork of VS Code. Copilot works natively in Visual Studio Code, with agent mode having been introduced by GitHub only recently [2]. It allows autonomous execution of change requests, from analysis through implementation to test validation and pull request creation.

2.3. Agents investigated

AI coding agents are mostly based on large language models (LLMs) that have been trained on huge amounts of data. They generate or modify code by responding to natural-language commands and using the project context. Four different agents are used in this report; an overview of their differences can be found in Table 1. JetBrains’ Junie, for instance, is an autonomous agent that can plan and execute multi-step tasks [3]. Junie can, for example, make large changes to the code base, run tests or terminal commands and report on progress. Technically, Junie sends code and project information to LLM providers; by default it uses Anthropic’s Claude Sonnet 3.7 (optionally Claude 4.0). In addition, JetBrains offers an interactive AI assistant (chat/autocompletion) that supports further LLM providers such as OpenAI’s o3 model or Google’s Gemini. When using Junie or AI Chat, a varying number of tokens is consumed depending on the complexity of the request, the model used and the size of the code base. Depending on the subscription, a different number of tokens is available to the user. Sweep AI works similarly as a plugin for IntelliJ-based IDEs: it offers an integrated agent that searches the code base, suggests changes and automatically runs builds/tests [4]. Sweep AI uses JetBrains’ static analysis tools to check the suggestions. It supports various models, such as Claude Sonnet 4, OpenAI o3 or Gemini 2.5 Pro. By default, the model is selected automatically. Sweep can also be self-hosted in the Enterprise version, but which models are available in this mode is not publicly stated [5].

GitHub Copilot is a coding assistant integrated into VS Code (and other IDEs) that supports a wide range of models. From May 2025, Copilot uses OpenAI’s GPT-4.1 as its default base model [6]. Copilot offers inline completion as well as a chat mode and an agent mode. It provides context-related code suggestions and example implementations, or answers questions based on the current code context and the request. Copilot supports further LLMs, but not all of them can be used in agent mode [7]. The Pro version of GitHub Copilot supports more recent models and offers a larger number of interactions with the agent [8]. GitHub Copilot is also available as a plugin for JetBrains development environments, but according to the plugin’s reviews the implementation is (still) not on the same level as the integration into Visual Studio Code [9]. Kiro is a new development environment from AWS designed for AI-supported development (based on VS Code OSS). It is powered by Anthropic’s Claude Sonnet 4.0 (with 3.7 as fallback) [10]. Kiro places particular emphasis on structured, specification-driven development: based on requirements, it automatically creates specifications, design documents and a task list [11]. In addition to the built-in chat and code functions, Kiro also supports so-called agent hooks and “Project Steering Files” to enable context and automation.

Table 1 Overview of the coding agents investigated.

Overall, these tools are similar in that they use LLMs and are deeply integrated into the development environment in use. Differences exist in the degree of autonomy and workflow integration: Junie creates a list of tasks to be carried out from the prompt, Sweep and Copilot start executing the task directly, and Kiro first starts by creating requirements and specifications before beginning the implementation.

3. Evaluation

In the experiment, an attempt was made to bring an existing Java/Tomcat application (pdf-as 4 [12]) up to date: we instruct the agents to update the dependencies in use (such as PDFBox, ZXing) to the current versions, to raise the Java and Tomcat versions, and to update the build environment to the current Gradle version. Each assistant worked on these tasks in its own setup: Junie and Sweep were used as plugins in IntelliJ IDEA, Copilot worked in Visual Studio Code, and Kiro ran as a standalone program that is essentially based on Visual Studio Code.

Approach: The agents were each given tasks in natural language, e.g. “This is an older project that we want to upgrade to use the latest version of dependencies. We also need to upgrade the Java version it builds with and other dependencies such as Tomcat. Make sure that the projects builds and tests run successfully”. Copilot and Sweep were able to search the code independently and apply changes. For each request, Junie creates a plan, which is then worked through successively. Kiro was started in specification mode (“Specs”). In this mode, the agent first creates a requirements document and a design document and generates from them an implementation plan with individual tasks. Each of these tasks is then worked through in separate chats with the agent.

As a general observation, all agents were able to carry out smaller, isolated dependency updates successfully. For larger changes that span several dependencies, such as upgrading the Java version or fundamental API changes in libraries, however, clear differences emerged in the capabilities and approaches of the assistants. The following subsections show the strengths and problems of each agent individually.

3.1. Sweep AI

Sweep AI started the task by updating a few dependencies, but did not consistently choose the latest version. For example, Apache CXF was updated to version 4.0.5 instead of version 4.1.2. The dependencies were thus presumably updated to the version known to the underlying model from its training data, but no online search for the latest version was made. Only after updating a few dependencies did the agent recognise the complexity of the task; it then created a Markdown file with a modernisation guide and ended its work with a request to carry out the remaining steps manually. This behaviour contradicts the purpose of using an autonomous agent.

Figure 1 After a certain period of time, Sweep AI ends the current task even if it has not yet been completed in full.

After an explicit instruction to continue the work with the help of the modernisation guide it had created, the agent continued the update. After changing a few files, however, Sweep AI repeatedly aborted the upgrade with the message that the task was too large. This happened, for example, during the migration from javax to jakarta, because the number of classes to be changed was too large, as shown in Figure 1. The agent ends the execution of the task with a summary of the changes made and still to be made. In this summary, the agent also gives concrete values for estimating the changes still outstanding. These are not always accurate, however: in Figure 2, Sweep AI states that imports still have to be adjusted in more than 600 files, although the original project contains only 316 files with such an import.

After continuing the task several times, the agent completed the update of Java and some further dependencies. After the supposed completion of the task, however, the project could no longer be compiled. Since the agent has no direct access to the terminal and thus to the debug output, the error messages that occurred during compilation always had to be released manually to the agent so that it could fix the underlying errors. After several problems had been resolved, however, the build error shown in Figure 3 could not be fixed by the agent even after repeated attempts, which illustrates the tedious interaction in which error messages have to be forwarded manually to the agent. The update of the build system’s Java version and of the javax packages was therefore not successful.

3.2. JetBrains Junie

JetBrains Junie was tested in IntelliJ. From the user’s initial prompt, it creates a plan with individual steps, as can be seen in Figure 4. The agent then works through this plan step by step. To update the dependencies, Junie uses the gradle-versions-plugin, which determines the latest library versions that are actually available instead of adopting outdated version numbers from the training data. For larger tasks, the agent also creates Markdown files with strategies, but these lack directly executable actions in the user interface, as Kiro has, for example. For the project, Junie chose a very conservative upgrade strategy in which all dependencies were first updated for Java 8 and only afterwards the upgrade to Java 11 (the next available LTS version) was tackled. This path led to problems, however: the lengthy process with many intermediate steps used up the entire token quota of the Pro subscription in only three to four days and led to inefficient updates in which changes were often temporarily reverted again because they were, for example, only compatible with a newer Java version.

In part, it is noticeable that many of the coding agents tested are still at an early stage. In Figure 5, for example, in addition to small errors in the user interface, one can also see a somewhat cryptic error message. It indicates that the character limit of the model used has been reached. As a result, the chat cannot be continued, which means that a new one has to be started. This in turn leads to the context of the old chat being lost and having to be explained to the agent again by the user. Junie does not yet have an automatic carry-over of the essential points of the chat (as of June 2025).

Junie failed (like the other agents) to upgrade Apache PDFBox to version 3.x. This version introduced major changes to the library’s programming interface. Even providing the link to the migration guide [13] did not help to complete the update successfully. After updating several classes, Junie aborted the upgrade and left behind a non-compiling code base with the option of rolling back the changes. In addition, it created a document with further steps necessary to complete the migration.

Figure 6 Example of a signature placeholder that was no longer reliably recognised after the update of ZXing.

A regression that occurred in the update of the QR code library ZXing was another problem. After updating from version 3.5.0 to 3.5.3, not all QR codes in the images extracted from PDF documents were recognised any more. This functionality is used in pdf-as to identify signature placeholders which, as shown in Figure 6, contain a QR code.

To recognise the placeholders, pdf-as first extracts all images from the PDF document using PDFBox. These images are then analysed with ZXing for QR codes they contain. A specific test case that verifies the correct recognition of the first of several signature placeholders in a PDF document failed after the update. The document used for this test contains three such placeholders. With the new ZXing version, only one of the three placeholders was recognised, which was not the first one in the document, causing the test to fail.

Instead of examining the dependency update as the cause of the failing test, Junie wrongly tried to change the expectation of the test case itself, as can be seen in Figure 7. Even after the explicit instruction that the test code is correct and must not be changed, Junie continued to try to change the test instead of finding the cause of the error. Only after the instruction to downgrade ZXing could the problem be solved. The agent was likewise unable to find the reason for the regression. The regression was introduced with version 3.5.1. A manual inspection of the commits of this version showed that a specific change, which was originally intended to improve performance when processing large images, causes the faulty recognition of the signature placeholders. This behaviour could be verified in the debugger and traced back to the corresponding commit [14].

Figure 7 After updating the QR code library, not all QR codes are recognised any more, which causes a test to fail. Even after an explicit request to leave the test untouched, Junie tries to fix the problem by changing the expectation in the test case.

3.3. GitHub Copilot (VSCode plugin)

The agent mode of GitHub Copilot in Visual Studio Code is a relatively new feature and is visibly still struggling with teething problems. During the test, errors in the user interface occurred repeatedly, such as a diff view that could not be displayed (see Figure 8), or opening the editor failing with an unexpected error. In addition, cryptic error messages such as “Failed to render content: e.isSettled is not a function” frequently appeared in the chat. During longer-running operations, Copilot repeatedly asks whether the action should be continued or cancelled. Nevertheless, timeouts (status 408) occurred quite often during the execution of tasks, as shown in Figure 9. Moreover, evaluating the agent with the free version repeatedly led to the rate limit being reached.

When updating the dependencies, the agent showed weaknesses in correctly identifying libraries. For example, the agent confused dependencies with similar names and tried, for instance, to update bcprov-jdk15on to version 1.78.1, a version that exists only for the successor of that dependency, bcprov-jdk18on. Similar to Junie, Copilot also uses a rather conservative update strategy by first applying smaller dependency updates and updating the project to Java 11 first. Like the other agents, Copilot also does not always select the latest version of the dependencies.

GitHub Copilot also failed at the two central modernisation tasks. The upgrade of the PDFBox library to version 3.x could not be carried out because of the extensive changes to the dependency’s programming interface, which led to a code base that could not run. Nor was Copilot able to diagnose the regression in the ZXing library correctly. Like the other agents, it misinterpreted the resulting test failure and tried to change the expectations of the test case instead of the faulty dependency, as shown in Figure 10. In addition, the agent suspects that the test file or the program code has changed. That neither is the case could have been found out by the agent through the version control system. Even after the hint that the test has not changed and that the cause is probably to be found in the dependency updates, the agent does not find the actual cause. The agent now suspects that the minor update of PDFBox might be responsible (see Figure 11).

3.4. AWS Kiro

The AI assistant in Kiro showed the best performance overall. This is probably due in part to the use of Anthropic’s latest model, Claude Sonnet 4. Another advantage lies in the more systematic approach in “Spec” mode. In this mode, Kiro first creates requirements, design and implementation plans before any changes are made to the code. For the update project, this approach resulted in a comprehensive implementation plan with 16 concrete tasks, each with several subtasks, which can be started individually directly from the implementation plan created. With the help of this plan, Kiro was able to carry out many of the updates on its own, including the upgrade of the Java version and of the build system. Despite this success, however, several weaknesses also emerged. Kiro repeatedly had problems finding current versions of libraries and instead suggested old or sometimes non-existent versions. While the agent searched Maven Central for current versions for some dependencies by using the command-line tool curl, versions were generated by the model for other dependencies. The OWASP dependency-check-gradle plugin, which is used in the project to detect dependencies with known security vulnerabilities, serves as an illustration here. The current version of the plugin at the time of the update is 12.1.3 [15]. In the course of the update by the agent, however, the plugin was only updated to version 11.1.0. After the hint that this version is not the latest, the agent tried to update to version 11.2.1. This version does not exist at all, however. This reveals a further problem of the agent or of the LLM used. When the user objects, the agent always immediately agrees with the user by saying “You’re absolutely right”, instead of questioning the statement where appropriate or asking follow-up questions when things are unclear.

The agent also had problems handling numbers: besides incorrect version numbers, the documents it created often contained incorrect dates; for example, a report generated in July 2025 was dated January 2025, as shown in Figure 12. In addition, especially after unexpected errors or deviations, the agent ignored tasks previously defined in the plan. On the positive side, however, Kiro’s handling of long chat histories deserves mention. As can be seen in Figure 13, the agent proactively offers to summarise the session and continue it in a new chat in order to get around the context limit. In addition, this limit is reached less often because of the split into individual subtasks carried out during planning, since each subtask is started in its own chat.

Figure 13 When the context limit is reached, Kiro offers to summarise the current history automatically and to start a new chat.

Like the other agents tested, Kiro failed at two central hurdles of modernisation. First, it was not able to update the Apache PDFBox library to version 3.x. The far-reaching and only roughly documented changes to the library’s programming interface led the agent to abort the upgrade and leave behind a code base that cannot be compiled, as shown in Figure 14. Despite the abort, the agent states that the task was implemented successfully and that the “PDF Processing Functionality” was tested. The unfinished code was also checked into the version control system.

Furthermore, the agent was likewise unable to find the cause of the failing test in the search for signature placeholders. Like the other agents, Kiro interpreted the test as faulty. As shown in Figure 15, the agent argued that the placeholder name checked in the test is only an implementation detail of PDFBox, although this library had not been updated at all because the update to version 3.x had failed. Even the direct hint could not get the agent to identify the regression in the ZXing library as the source of the error. Instead, the agent suspects that the change could be due to libraries such as slf4j, commons-io, ognl or Logback. Since the change is deterministic, the agent continues to assume that the test only checks an implementation detail and should therefore be changed.

After the agent was informed that the test file contains three QR signature placeholders, it at first seems to get onto the right track. The agent establishes that three images are indeed contained in the test document and that the QR code recognition may no longer work correctly because of the dependency updates, as can be seen in Figure 16. Kiro then creates its own class to recognise the QR codes in the extracted images. Unfortunately, the agent makes the mistake of building this class on the same version of ZXing that contains the regression. As a result, Kiro again concludes that there is only one signature placeholder in the document and that the test is therefore the problem, as shown in Figure 17. Only after the explicit instruction that the ZXing update is the problem does Kiro test with different versions of ZXing. The agent concludes that the test still works with version 3.4.1 and therefore carried out the downgrade to this version, see Figure 18. This version is, however, even older than the version 3.5.0 used before.

To check how many dependencies were successfully updated by the agent, the Gradle Versions Plugin [16] was used. According to this tool, 31 of the dependencies are up to date, while newer versions are available for 28 dependencies. Three of these outdated dependencies are part of the PDFBox update that was not successful, while two dependencies are part of ZXing. The remaining 23 dependencies were in some cases updated to a version that is not quite current (e.g. Apache commons-io to version 2.19.0 instead of version 2.20.0), while other dependencies were not updated.

A central element of coding agents is so-called prompt engineering. An agent’s success depends on how clearly and with how much context the task is formulated. Prompts that are too broad (“Update everything”) often lead to incomplete results or hallucinations. Structured task descriptions such as the following are more successful:

“Update the dependency ‘org.example:example‘ to version 3.x and adapt the code so that no deprecated APIs are being used. Search for the newest version on Maven Central or similar. Compile the code and run all the tests after updating. Finally, commit the resulting changes.”

From the prompt, Kiro creates requirements, design and implementation plans in Markdown format. Junie, in turn, generates a plan based on the prompt, on the basis of which the changes are then carried out. Sweep and Copilot start making the changes to the code directly after the prompt is entered.

Despite some successes in implementation, coding agents are not yet free of errors:

• Hallucinations: LLMs invent version numbers, classes, methods or libraries that do not exist, or confuse libraries with similar (package) names.

• Forgotten tasks: In complex changes, agents overlook intermediate steps, e.g. running the tests or checking in the changes. Especially in the case of unforeseen problems, the agents deviate from previously defined rules or subtasks and do not find their way back after the problem has been resolved.

• Loss of context: In large projects, the code often exceeds the context window, which means that a new chat has to be started and existing context may be lost.

• Early stage of development: Many of the agents tested still have teething problems, such as error messages when opening editors or UI problems.

3.5. Other areas of application

Beyond their use in updating existing applications, the coding agents were also tested in other projects. They proved particularly suitable for running terminal applications. With coding agents, these can be operated via natural language. As a result, rarely used commands do not have to be looked up in the manual or on the internet. For example, a method was removed when a dependency of a project was updated. Since this is not mentioned in the dependency’s changelog, the agent Junie was tasked in Figure 19 with finding the respective change with the help of Git. The agent found the change on its own and successfully recognised that the method had been split into two separate method calls. In addition, the agent was tasked with implementing the change in the project’s code. After several iterations, the agent was able to make the change.

4. Conclusion

AI-based programming agents offer high potential for supporting the modernisation of existing systems. They make it possible to automate time-consuming, repetitive tasks such as updating dependencies, refactorings and build adjustments. Nevertheless, they are not a replacement for experienced developers. Hallucinations, loss of context and incomplete implementations make human oversight indispensable. Kiro proved promising thanks to its comprehensive planning approach, while Junie worked well in the JetBrains environment. However, all agents still failed at larger updates such as the PDFBox 3.x upgrade. Likewise, extensive test coverage of the existing application is recommended in order to be able to detect regressions in updates. Even the analysis of such regressions still sometimes requires manual work.

References

[1] E. Ryabukha, „Full Line Code Completion in JetBrains IDEs: All You Need to Know,“ JetBrains s.r.o., 04 04 2024. [Online]. Available: https://blog.jetbrains.com/blog/2024/04/04/full-line-code-completion-injetbrains-ides-all-you-need-to-know/. [Zugriff am 28 07 2025].

[2] T. Dohmke, „Vibe coding with GitHub Copilot: Agent mode and MCP support rolling out to all VS Code users,“ Github, Inc., 04 04 2025. [Online]. Available: https://github.blog/news-insights/productnews/github-copilot-agent-mode-activated/. [Zugriff am 23 07 2025].

[3] JetBrains s.r.o., „Getting started with Junie,“ 21 07 2025. [Online]. Available: https://www.jetbrains.com/help/junie/get-started-with-junie.html.

[4] Sweep AI Inc., „Sweep: AI Plugin for JetBrains IDEs,“ [Online]. Available: https://docs.sweep.dev/.

[5] Sweep AI Inc., „Fully Air-Gapped Deployments,“ 2025. [Online]. Available: https://docs.sweep.dev/enterprise. [Zugriff am 24 07 2025].

[6] GitHub Inc., „OpenAI GPT-4.1 is now generally available in GitHub Copilot as the new default model,“ 08 05 2025. [Online]. Available: https://github.blog/changelog/2025-05-08-openai-gpt-4-1-is-nowgenerally-available-in-github-copilot-as-the-new-default-model/.

[7] GitHub, Inc., „Supported AI models in Copilot,“ 2025. [Online]. Available: https://docs.github.com/en/copilot/reference/ai-models/supported-models#supported-ai-models-incopilot. [Zugriff am 28 07 2025].

[8] Github, Inc., „Try Copilot Pro for 30 days free,“ 2025. [Online]. Available: https://github.com/githubcopilot/pro. [Zugriff am 28 07 2025].

[9] „GitHub Copilot - Ratings & Reviews,“ 2025. [Online]. Available: https://plugins.jetbrains.com/plugin/17718-github-copilot/reviews. [Zugriff am 24 07 2025].

[10] N. Swaminathan und D. Singh, „Introducing Kiro,“ 14 07 2025. [Online]. Available: https://kiro.dev/blog/introducing-kiro/.

[11] R. Yanchuleff, „From Chat to Specs: A Deep Dive into AI-Assisted Development with Kiro,“ 15 07 2025. [Online]. Available: https://kiro.dev/blog/from-chat-to-specs-deep-dive/.

[12] A-SIT, „PDF-AS,“ [Online]. Available: https://technology.a-sit.at/software/pdf-as/. [Zugriff am 22 07 2025].

[13] The Apache Software Foundation, „PDFBox 3.0 Migration Guide,“ [Online]. Available: https://pdfbox.apache.org/3.0/migration.html. [Zugriff am 28 07 2025].

[14] S. Owen, „Filter unconfirmed QR code finder patterns before testing triangles to avoid long-running analysis. Closes #1567,“ 01 11 2022. [Online]. Available: https://github.com/zxing/zxing/commit/110ef9e8f2733c051dd00e95bddc1bff6ab94ca5. [Zugriff am 25 07 2025].

[15] Gradle, Inc., „org.owasp.dependencycheck,“ 2025. [Online]. Available: https://plugins.gradle.org/plugin/org.owasp.dependencycheck. [Zugriff am 28 07 2025].

[16] B. Manes, „Gradle Versions Plugin,“ [Online]. Available: https://github.com/ben-manes/gradle-versionsplugin. [Zugriff am 28 07 2025].

[17] Github, „Comparing zxing-3.5.0 … zxing-3.5.1,“ 2022. [Online]. Available: https://github.com/zxing/zxing/compare/zxing-3.5.0...zxing-3.5.1. [Zugriff am 25 07 2025].