Jellyfish AI development study: The real sting has yet to land

Developers are adopting and eschewing AI coding tools depending on their trust in the new automation mantra, the level of freedom afforded by their incumbent projects, and whether there’s an “R” in the month.

Aiming to analyze the impact of currently used AI tools, software engineering intelligence, and AI impact analysis, the company Jellyfish says its recent AI Engineering Trends study is a comprehensive quantitative analysis of AI transformation in software engineering.

Based on data analytics drawn from more than 700 companies, 200,000 engineers, and 20 million pull requests (when developers, and now agents, request to merge a proposed code change into a shared codebase for team review), the study aims to benchmark the true impact of AI tools on software development teams.

Why should developers care?

Software application development professionals may be able to assess, from this analysis, the types of engineering practices currently proving most practical and productive and (eventually, if not always immediately) apply them to their own workflows and project goals. They may also gauge how safe the industry at large considers these tools to be.

More than half of the companies in Jellyfish’s study say they use AI coding tools consistently, and 64% generate “a majority of their code” with AI assistance. The bottom line here suggests that over the three months analyzed, development teams in the top quartile of AI adoption have seen a doubling of pull request throughput, compared to low adopters.

The sting in the tail

Of particular note, says Jellyfish, is that fully autonomous code agent activity (where pull requests are generated entirely by AI agents) remains low overall but is growing exponentially, and that is where the sting will be felt. Nicholas Arcolano, Ph.D., head of research at Jellyfish, thinks this growth tells a story; quite simply, he says, the most aggressive adopters of AI code tools are riding a big wave.

Fully autonomous code agent activity remains low overall, but is growing exponentially, and that is where the sting will be felt.

“AI coding tools are now the default option for engineering teams, and the productivity gains are real. Enterprises can use our metrics as an objective baseline to benchmark their organization’s AI adoption and impact. The data shows a clear link between deep AI tool integration and measurable improvements in delivery throughput and engineering outcomes. The most aggressive adopters are pulling away from the pack,” says Arcolano.

The company’s AI Engineering Trends portal allows anyone to assess their maturity against industry standards. Jellyfish customers get a slightly larger buffet plate and can use Jellyfish AI Impact capabilities to automatically parse and visualize their organization’s unique data and software tools. The output is delivered in custom dashboards that map the exact correlation between a team’s AI adoption and its engineering productivity.

AI coding tools are now the default option for engineering teams, the productivity gains are real… and the most aggressive adopters are pulling away from the pack.

“The data is clear: not adopting AI coding tools is now a competitive disadvantage. The conversation has moved past adoption – now it’s about scaling AI in ways that add up to something meaningful for your business. IDE-based tools like Copilot and Cursor were a natural entry point because they met developers exactly where they already work, but the future is autonomous agents. That’s a much bigger lift, because agents don’t just speed up coding, they fundamentally change how the software application development lifecycle works as a whole,” Arcolano, tells The New Stack.

Tentacular tribulations

This is where Arcolano points to what he says is a clear divergence. His team has identified a chasm: it says the 90th percentile of companies have increased AI code tool adoption by approximately seven times in the past year, while the bottom quartile is essentially at zero.

This is why (tentacle-based nematocyst discharge puns notwithstanding), the tail end of the software engineering community is getting stung.

Yagub Rahimov, CEO of Polygraf AI, agrees with the trends tabled here. He says that the teams genuinely pulling ahead aren’t just the ones who adopted AI fastest; they’re the ones who figured out that we can’t remove the human from review and testing, just because the machine writes the code. Therein, he says, lies another sting.

“The Jellyfish numbers are real – we’re seeing the same thing. With 15 engineers using Cursor and Claude Code in their stack, we’ve seen pull requests flying out faster than before. But here’s what the throughput metric doesn’t capture: code review is now taking longer and getting more complicated. More code coming in means more surface area to check, and AI-generated code has this specific quality where it looks right until it isn’t,” Rahimov tells The New Stack.

Frictionless productivity? Yeah, kind of

Talking about how his team is now operating, Rahimov tells us that what’s actually happened on his company’s teams is a shift, i.e., less time writing code, but more time verifying. He says it’s a net win, but it’s not the “frictionless productivity story the headline numbers suggest,” so he advises cautionary balance. He explains that his team is dogfooding its own software toolset right now, running a secure LLM with code-detection guardrails across the entire engineering team.

The productivity gains are real, he says, but so is the exposure if developers don’t pay attention to what’s flowing through their AI tooling.

Bridging an even wider consensus on the realities at play here is Kat Cosgrove, head of developer advocacy at container image vulnerabilities specialist Minimus. Drawing her opinion from real-world enterprise software use cases, Cosgrove currently serves as a member of the Kubernetes Steering Committee and more.

“The rising popularity of AI developer tooling is an increasingly large burden for open source maintainers. Code the submitter didn’t bother to run that doesn’t even compile, pull request descriptions that don’t even vaguely match the content of the commits, sweeping documentation changes that touches dozens of generated files, submitters who don’t understand the content of their code and can’t participate in a review – all of these and more are now regular weekly issues for reviewers who are already overburdened. The increase in output enabled by AI has not included an increase in capacity for maintainers to deal with these low-quality submissions.,” Cosgrove tells The New Stack, with palpable lament.

A tighter rope on AI pull requests

Jellyfish also recently partnered with Augment Code, a software company that works across the integrated development environment, command line interface, and code review space to bring AI telemetry to joint customers. Augment’s Context Engine is built to maintain a live understanding of a team’s complete software stack, encompassing code, dependencies, architecture, and history. The work here is intended to enable organizations to see exactly how Augment Code is used across engineering teams and measure its impact.

In addition to its core developer analysis work, Jellyfish has worked with OpenAI to publish a study that explored the real-world impact of AI on software development, including coding assistant and code review agent adoption by tool, new data on the growth of AI-generated code, and the impact of AI on pull request throughput and cycle time.

Jellyfish says its benchmark site is regularly updated with new data and analytics on this subject.

The post Jellyfish AI development study: The real sting has yet to land appeared first on The New Stack.