Text-to-SQL looks solved on academic benchmarks, and enterprise data shows it isn’t. Models that score above 90% on Spider 1.0 drop to about 21% on Spider 2.0, which uses real warehouse schemas that often have more than 1,000 columns. The benchmarks themselves are also unreliable. One audit found that 21 of the 25 empty-result queries in Spider’s test set have wrong gold labels, and that around 15% of test questions depend on an ambiguous “top-k” assumption. So a hand-written golden set won’t tell you whether your agent is right. This session covers how to generate evals automatically for a NL2SQL agent running against a large production data platform: → Generating question and SQL pairs from your schema and query logs → Dealing with questions that have more than one correct SQL answer → Picking match criteria (execution results, row order, DISTINCT) that don’t hide errors or invent them → Keeping the eval set current as schemas and metric definitions change