Evaluating and Optimising AI in the Real World

Join the AI Native Dev community in London for a night of practical, technical insight into how modern AI systems actually perform in the real world. From NVIDIA’s work on scaling long-context LLM inference, to Hugging Face’s perspective on Skills and MCP, to Tessl’s lessons from running large-scale evals on AI coding agents, this meetup is all about what breaks, what works, and what developers need to know. Come for the talks, stay for the conversations, and meet others building the next generation of AI-native software.
Agenda
6pm:  Doors Open
Talks kick off6:30: Talk 1: Evaluating AI Skills in the Wild: What We Learned Running Evals at Scale by Rob Willoughby
7pm: Talk 2: Accelerating Long-Context Inference with Skip Softmax Attention by Dom Brown
7:30pm: Talk 3: Using Skills and MCP for Open Source ML by Shaun Smith
8pm: Networking
9pm: THE END

Evaluating AI Skills in the Wild: What We Learned Running Evals at Scale
How do you know if an AI coding agent is actually following your instructions? Or if that skill you wrote is having an impact? What can you learn from synthetic eval cases compared to running them in your project on your code? We'll share practical lessons from building and running large-scale evaluations of coding agents, covering eval design, the ways things break at scale, and what the results reveal about where today's models actually struggle.
Rob Willoughby, Member of Technical Staff, AI Research at Tessl
Rob works on evaluation research at Tessl, designing and running large-scale assessments of how coding agents behave in real-world codebases. His work focuses on figuring out what "good" looks like when an AI agent works with your code from eval design and rubric systems to understanding where models (and infrastructure) break down at scale.

Accelerating Long-Context Inference with Skip Softmax Attention
The growing demand for long-context inference capabilities in Large Language Models (LLMs) has intensified the computational and memory bottlenecks inherent to the standard attention mechanism. To address this challenge, we introduce Skip Softmax Attention, a drop-in sparse attention method that dynamically prunes the attention matrix without any pre-computation or proxy scores. Our method uses a fixed threshold and existing information from online softmax to identify negligible attention scores, skipping softmax computation, value block loading, and the subsequent matrix multiplication. This fits seamlessly into existing FlashAttention kernel designs with negligible latency overhead.
Dominic Brown, Senior AI DevTech Engineer at NVIDIA
Dom Brown is a senior AI developer technology engineer at NVIDIA. His work focuses on optimizing large language model inference. Dom studied computer science at the University of Warwick, UK. In his PhD thesis, he investigated the acceleration of particle-in-cell algorithms on CPU and GPU systems.

Upskilling Models, Agents and the ML Pipeline.

Recent advances in Language Models and Coding Agents have impacted the ML Stack itself, opening up new opportunities for automation and training.

At Hugging Face we've incorporated Skills and MCP in a number of areas - from automating model training, and having large models tutor smaller models for certain tasks. We'll share examples of MCP for model-to-model communication, practical tools for upskilling agents and improving our ML pipeline and share some of the tools lessons we've learned along the way.
Shaun Smith, Open source MCP Lead at HF
Shaun Smith leads Open Source MCP at Hugging Face, and is an MCP Steering Committee member serving as a Community Moderator and within the Transports Working Group.

This event is brought to you as part of the AI Native Dev Community. Consider subscribing to the Mailing List, Podcast, and joining our Discord Community