Microsoft has introduced a new tool for testing artificial intelligence called debug-gym, designed for debugging software. Despite significant progress in automating code writing, artificial intelligence is still unable to replace programmers, especially in the area of debugging.
This is reported by Finway
A study conducted by Microsoft Research shows that even with access to professional tools, AI models perform worse than humans on tasks. The debug-gym environment allows AI to use tools that were previously unavailable, such as breakpoints, code navigation, variable reading, and test writing. This brings them closer to the methods used by developers. However, experts note that the models only demonstrate 48.4% success in solving tasks.
“The fixes proposed by the debugging-capable coding agent, and then approved by a programmer, will be based on the context of the relevant codebase, program execution, and documentation, rather than relying solely on guesses based on previously reviewed training data,” Microsoft states.
Researchers believe that the main issue remains the lack of training data with step-by-step debugging scenarios, as well as the models’ inability to effectively utilize debugging tools. The next step may involve developing a supplementary model that gathers the necessary information and passes it to the main agent.
The authors of the study emphasize that the primary goal of such systems is to assist humans, not to replace them. Models often create vulnerabilities and unstable solutions, indicating that full automation of the development process is still premature. Experts note that despite progress, the role of humans in complex tasks of analysis, interpretation, and debugging remains critically important.
It is worth mentioning that recent information has emerged indicating that Shopify plans to hire only those employees who cannot be replaced by artificial intelligence.