How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, Ramp
Build repeatable agent evals from realistic business data and task-specific success criteria, then use them to find regressions and improve the agent.
- Break the job into discrete tasks and define success criteria with practitioners before building an agent.
- Without enough user data, create synthetic businesses and replay the same task worlds to compare agent performance.
- Use evals to test model choices, remove unhelpful skills, test memory, and monitor recurring regressions.
The notes capture the evaluation workflow and main experiments; the talk is mostly explanation. · Best for: Teams building agents for specialized workflows with limited production data
Ryan Stevens describes how Ramp evaluated its bookkeeping agent, Stacks. The process centered on understanding bookkeeping tasks, creating synthetic business data, measuring task outcomes, and using recurring evals to improve and monitor the agent.
Define the work and success criteria0:12
- Start by breaking a job into tasks and defining what success means.
- Ramp grouped bookkeeping work into reconciliation, data entry, variance analysis, and schedules and accruals.
- For reconciliation, compare accounts payable records across systems, identify mismatches, and resolve them.
Build eval data before users are available3:20
- When user data is insufficient, use simulation and synthetic data.
- Create a realistic synthetic business with its ERP, customers, vendors, accounts payable, and other relevant records.
- Data quality is a bottleneck: evaluations need clean, usable data.
Run repeatable task backtests5:33
- Treat a task and its data as a reusable “world”; run the agent on it and evaluate the final output.
- Use roughly 10–20 criteria per task in Ramp’s benchmark.
- Measure correctness, cost, and speed; the initial focus was not on judging the agent’s trajectory.
Choose metrics and inspect failures8:05
- Accuracy gives partial credit when some criteria pass.
- Pass at K is all-or-nothing: it indicates whether a run eventually solved every criterion.
- For accounting, the team found reasoning failures more prominent than failures to gather the needed information.
Use evals to improve the agent10:24
- Compare models, then inspect failure types to find opportunities for improvement.
- Skill ablation tests whether adding or removing skills changes performance and can identify skills to delete or compress.
- For memory tests, write memory after one period and test whether using it improves performance on the task in a later period.
Monitor regressions16:50
- Ramp’s internal eval platform lets teams add and tag cases and schedule recurring runs.
- A background agent reviews failures and regressions, then sends a Slack message with a suspected cause.
- Stevens cautions against relying on CI gating for agent evals because results can be nondeterministic.
Step by step
- Break the workflow into discrete tasks and define task-specific success criteria with practitioners.
- If production data is insufficient, create realistic synthetic businesses and the records needed to complete the tasks.
- Save each task and dataset as a reusable world, run the agent against it, and score its final output.
- Track accuracy, pass-at-K, latency, and cost; inspect failures to identify model or harness improvements.
- Use ablation tests to assess skills and test memory by carrying it from one task period into a later one.
- Schedule recurring eval runs, review regressions, and investigate the reported failures.
Tools & resources mentioned
- Mercor — partnered with Ramp on building the synthetic-data benchmark.
- Excel — used in the reconciliation example to compare records from two systems.
- Ramp internal eval platform — hosts, tags, and schedules evaluation cases.
- Slack — receives notifications about suspected regressions and failures.
Notes are AI-generated from the video's transcript and may miss nuance.