AI ROI: Why the Agent Isn't the Answer

Most AI agents underperform in production not because the model is wrong, but because the data around it is incomplete and the results are unmeasured. Two talks on the two constraints that actually limit agent ROI: what your agent can see, and whether you can tell if it's working.
Agenda
18:00 Venue opens
18:30 Talk 1: The ROI Isn't in the Agent  by Rob Willoughby
19:00 Talk 2: Data Blindspots: The Hidden Killer of AI ROI by Tom Scott, CEO, Streambased
19:30 Networking
20:30 THE END

The ROI Isn't in the Agent
Many agent ROI numbers are fiction. They come out of the spreadsheet every engineering leader already knows: hours saved per developer, multiplied by headcount, multiplied by a hopeful factor. The trouble is that this treats the agent as the thing you're buying, and treats the saving as something you bank once and move on from.
This talk argues that the return that actually lasts sits one layer up, in the environment around the agent: the context it works from, the evaluations, the verification that decides whether its output is any good. Savings from a faster tool get spent and don't recur. Money put into the harness keeps paying back, because each piece you build makes the next one cheaper and the data you throw off along the way accumulates.
It also looks at why none of that is bankable until you can measure it, and why getting from vibes to metrics is where an honest ROI story has to start.
Rob Willoughby, Member of Technical Staff, AI Research at Tessl
Rob works on evaluation research at Tessl, designing and running large-scale assessments of how coding agents behave in real-world codebases. His work focuses on figuring out what "good" looks like when an AI agent works with your code from eval design and rubric systems to understanding where models (and infrastructure) break down at scale.

Data Blindspots: The Hidden Killer of AI ROI
Why do so many AI agent projects fail to deliver meaningful ROI?
The usual answers focus on models, prompts, and tooling. But in practice, the biggest limitation is often much simpler: agents can't see enough of the business.
Most organisations give agents access to a handful of data sources and expect them to make decisions using a fragmented, incomplete view of reality. Every missing integration, delayed update, or inaccessible dataset becomes a blindspot. And every blindspot reduces accuracy, confidence, automation rates, and ultimately ROI.
In this talk, I'll explore why agent performance is fundamentally constrained by data visibility. We'll look at how blindspots emerge, why traditional integration approaches struggle to keep up with the growing demand for agent-ready data, and how modern event-driven architectures can make business context immediately available to AI systems.
We'll also examine the relationship between visibility and efficiency: why more data isn't always better, how context minimisation improves agent performance, and why the most successful agent deployments are often the ones that provide the clearest view of reality rather than the largest context windows.
Low ROI agents aren't stupid. They're blind.
Tom Scott, CEO, Streambased
Tom Scott runs Streambased, the company unifying Apache Kafka and Apache Iceberg for AI projects. Tom has more than 20 years of experience in distributed systems and data infrastructure working as a customer and vendor for some of the most exciting problems in event streaming and big data.

This event is brought to you as part of the AI Native Dev Community. Consider subscribing to the Mailing List, Podcast, and joining our Discord Community