Nubank uses simulated workflows to screen customer service agents
Understand this story
In a paper, Nubank proposes a "hypothesis-driven" simulation workflow: using synthetic customers and simulated tool outputs to screen customer service agents before deployment. Nubank says this process shortens the agent update cycle from 3 weeks to under 5 days. Across the 4 deployed versions of the Card Management agent, Nubank says simulation and production environment version-level binary evaluation scores are highly correlated; iterations guided by simulation increased tNPS by 36.69 points in online A/B tests. These correlations and the magnitude of the improvement are results reported by the paper's authors.
AI overview based on source reports · Updated 5 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIPaper: Nubank cuts AI agent update cycles from 3 weeks to under 5 days
The paper presents a hypothesis-driven simulation workflow using the Snowglobe simulator to screen candidate customer-experience agents before deployment, tested on Nubank's Card Delivery agent and its successor Card Management. Across 4 deployed versions, simulated and production version-level binary evaluator scores showed high correlation, and simulation-guided iteration raised transactional net promoter score by 36.69 points in a live A/B test. Screening open-weight configurations in over 16,000 simulated conversations led to a model that increased self-service rate by 8.82 percentage points in a subsequent live A/B test, with no statistically significant change in tNPS.
Activity for this story
- Current comparable activity
- 9
- Peak comparable activity
- 10Nov 6 09:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.