7 min read Legacy Modernization

Modernizing legacy code with agents: translating the code is the easy part

Modernizing legacy code with agents: translating the code is the easy part

IBM announced on 9 July 2026 multi-agent capabilities and specialised modernization workflows for its agentic development platform, with dedicated packages for IBM Z, IBM i and Java migration. The mainframe offering reasons across COBOL, PL/I, assembler, JCL, CICS, IMS and Db2, and includes business-rule extraction intended to preserve institutional logic left undocumented after decades in production. The release cites a customer whose modernization effort, originally projected at nine months with fourteen engineers, was reportedly completed in three days.

A figure like that deserves the caution owed to any customer testimonial published by a vendor, but the order of magnitude is not implausible, and that is precisely what should worry people. Translating code is no longer the bottleneck. Yet the slowness of translation had until now served as an involuntary control: rewriting by hand forced somebody to understand what the code did. That safeguard has just disappeared, and the hard work, proving the new system behaves like the old one, has not accelerated at all. It has arguably got worse, since it now has to absorb a volume of transformed code bearing no relation to what it handled before.

The 29 June article on the SDLC in the agentic era described restructuring each phase of the development cycle. Legacy modernization is its extreme case, because it is the only situation where the specification exists nowhere except in the running system. What follows measures the real scale of the risk, shows why the most intuitive countermeasure does not work, and proposes four gates to pass in order.

Two apparently equivalent code blocks, with a single line quietly diverging on the right.
The code compiles, the tests pass, performance improves, and the behaviour has quietly changed.

What a study that bothered to check actually measures

Work published on arXiv in May 2026 by three researchers frames the question in the simplest possible terms: when a model modernizes code, can it be trusted to recognise that its own output has silently changed observable behaviour? The protocol comprises 1,980 real modernization calls across eleven production models from seven distinct families, applied to a balanced corpus of sixty Python 2 snippets, under three different instruction phrasings. Every output is evaluated by a type-strict behavioural oracle, and the producing model is then asked whether its own code preserved behaviour.

Drift rates fall into three clearly separated tiers. On control snippets, which require no real modernization, the drift rate is 7%, giving the background noise: even when there is nothing to do, models break something roughly one time in fourteen. On syntactic traps, whose modern equivalent is unambiguous, drift rises to 12.7%. On semantic-preservation traps, it reaches 39.7%. The dominant class is numeric semantics at 57% drift, far ahead of lazy evaluation at 21%.

The canonical example is instructive because it is trivial. In Python 2, dividing two integers returns an integer, 5 divided by 2 giving 2. In Python 3, the same expression returns 2.5. Migrated code that keeps the operator therefore produces a different result, with no error, no exception, and a perfectly plausible value. The authors stress a methodological point that belongs in any test strategy: comparing results with permissive equality rather than strict typing drops the measured drift rate from 39.7% to roughly 26.7%, because 2 and 2.0 count as equal. Thirteen points of drift disappear in the comparison, not in the code. A downstream consumer using that value as a list index, serialising it, or checking its type will see the difference.

Two further results deserve the attention of anyone building a tool selection strategy. First, drift is structural rather than random: cross-model correlation on which snippets are hard averages 0.52, and a core of snippets defeats at least eight models out of eleven under every instruction. Second, and this is the most awkward finding for a purchasing committee, the drift rate tracks neither capability nor price. It spans 5.6% to 46.7% across models, on a panel whose prices vary by a factor of two hundred, and the cheapest model in the set posts the best result. The authors draw the obvious conclusion: a selection rule based on vendor or model size does not protect against this class of drift, an explicit behavioural oracle is required.

Self-review is not a safety net

The most natural countermeasure, and by far the cheapest, is to ask the model to check its own work. That is exactly what the study tests, on the 262 cases where the oracle detected confirmed semantic drift.

The aggregate result might look acceptable: self-review catches around two thirds of drift. The remaining third is the problem, because it is not a mere coverage gap but an explicit endorsement. Of 262 cases, 83 are silently approved by the model that produced them, or 31.7%. On the most frequent class, numeric semantics, 75 drifts out of 207 pass review, or 36%.

Per-model behaviour is strongly bimodal, and that is what makes the countermeasure unusable as it stands. Five models miss almost none of their own drift. One widely deployed model misses all of it. More importantly, this self-blindness rate is not predicted by the drift rate: the model that drifts most, at 65%, catches 100% of its own errors, while a model drifting moderately, at 41.7%, endorses 100% of them. Self-review reliability is therefore a model property orthogonal to correctness, meaning no performance ranking tells you who can be trusted with verification.

The qualitative detail the authors report is the one to keep in mind when arbitrating. In several cases, the model opens its self-assessment by correctly stating the difference between Python 2 and Python 3 on the division operator, then concludes that behaviour is preserved. It describes the exact mechanism that broke its own output, and declares the output intact.

One last result is worth flagging because it contradicts common practice. The three instruction phrasings tested do not produce the same drift rate, and the most carefully crafted one performs worst: the instruction framed as a production contract, naming downstream callers, reaches around 32% drift against 19% for the direct instruction. The authors suggest the model then produces more elaborate rewrites, judged safer, and semantically different. Emphasising the stakes in the prompt does not reduce the risk, it relocates it.

Four successive gates in a modernization programme, only one of which is accelerated by tools.
Four gates, in order. The third is the one tools accelerate; the other three remain yours.

Four gates, and only one that tools pass for you

The consequence is an order of execution, and it inverts the one most programmes follow.

First gate, map. Establish the real perimeter: entry points, dependencies, datasets, batch jobs, and above all the business rules that live only in the code. This is the one place where agent-assisted extraction, as today's specialised packages offer it, delivers net value at low risk, since it produces documentation subject to human review rather than executable code.

Second gate, characterise through tests. Before the first line is modified, build a suite that pins current behaviour, correct or not. This is where agents serve best, and it is massively underused: generating tests from the observed inputs and outputs of a running system is a task models do well, at low risk, and it produces exactly the oracle the study above proves necessary. The rule that follows from the data is simple: your oracle must compare types as well as values, or it will mask the most frequent class of drift.

Third gate, transform in batches. This is the only one of the four that tools genuinely accelerate, and it should be exploited, provided batches are sized against verification capacity rather than generation capacity. A batch whose potential drift cannot be detected before production is too large, whatever time it took to produce.

Fourth gate, compare in a parallel run. Run the old and the new system side by side on real traffic, using only the old system's outputs, and compare. It is the only method that uses the existing system as a live oracle rather than as a memory, and it is what the study implies when it notes that behavioural equivalence is a more demanding criterion than passing tests.

The pattern recurring in programmes that succeed is exactly this: they spend most of their agentic budget on gates two and four, not gate three. Those that fail invested the bulk in generation, obtained an entire rewritten system very quickly, and discovered they had no way of knowing whether it was right.

What to put in motion this week

Check how your existing tests compare, before anything else. If your assertions compare values without comparing types, you already have an oracle that lets the most common class of drift through, and that flaw takes hours to fix.

Take a representative legacy module and have a characterization test suite generated from its observed behaviour, before any modification. Measure the coverage obtained and the time spent: that number should size the rest of the programme.

Forbid self-validation by the producing model, in writing. Whether verification runs through an executable oracle, a second model from another family, or human review, the rule is that it cannot come from whoever wrote the code.

Build a trap corpus specific to your own estate, on the study's model: twenty or so known cases where migration changes behaviour plausibly. Run it at every tool or model change. It will cost you a day and serve you for years.

Finally, stand up a parallel run on a narrow scope before industrialising. The cost of computing twice on real traffic is trivial next to a business divergence discovered three quarters later in an accounting reconciliation.

Conclusion

The arrival of agents able to rewrite an entire estate in days is good news, badly framed. It removes the constraint that has limited these programmes for twenty years, and leaves intact the one that makes them fail. The question to put to a vendor is no longer how fast it translates, which everyone now answers the same way, but how it proves equivalence, and the only acceptable answer points to something outside the model. A system that compiles, passes its tests, runs faster and computes something else is a more expensive failure than one that will not start.

Sources: As of July 2026

Godwin AVODAGBE