Contents
In brief
Chart QA benchmarks are often solved via shortcuts. Chartographer builds counterfactual charts—fixed questions, varied data/answers—to test genuine visual reasoning.
What they studied
Reverse-engineering charts to executable code, fidelity checks, seed-controlled variants, and VLM sensitivity metrics.
Key findings
- Strong single-chart scores hide counterfactual failures.
- Failures spike when answers need new visual reasoning paths.
- Framework extends existing chart-QA datasets.
What this means for developers
Add counterfactual eval pairs for VLMs; test dashboard AI when data changes under the same prompt; treat executable chart specs as regression assets.
Limitations
Chart-QA scope; exotic visuals may break code reconstruction.