We combine mathematical analysis, controlled experiments and measurements of actual AI systems.
01 / Research area
Reliable evaluation
What does a success score measure?
Match the score to the task.
Methods and context
Choosing a model or comparing an agent requires knowing what a result measures and how uncertain it is. Shared attempts, adaptive sampling, and a changed test protocol can all change that interpretation.
Define the quantity being measured and the right sampling unit.
Account for dependence, adaptive stopping, and uncertainty.
Test whether a comparison holds under the conditions of use.
Statistical inference
Agent evaluation
Sequential methods
Method schematicDots are attempts; each box is one complete run. Sampled or mutable state stays separate across fresh runs.
Use controlled comparisons to test an explanation.
Methods and context
A system can retrieve an outdated fact or change its prediction when the same input is represented differently. Understanding these effects helps identify what an intervention would need to address.
Vary inputs and context while holding the comparison conditions fixed.
Probe representations and intervene on model components where possible.
Check alternative explanations, side effects, and transfer to other settings.
Attention & interference
Input representations
Controlled interventions
Method schematicHold the input fixed, change one condition, and compare the outputs.
Limited memory, calibration data, and compute budgets shape how AI is used. A smaller model or a shorter output is valuable when the resulting system also preserves the behavior that matters for the task.
Study calibration and quantization under limited data and hardware budgets.
Measure cost per useful result, including serving overhead.
Compare deployed behavior with the reference and evaluate targeted improvements.
Model compression
Calibration
Deployment evaluation
Method schematicCompare quality, time and memory for the same task and deployment conditions.
Method schematicToken count and elapsed time are different measurements. Compare GPU time per solved problem under the intended workload and load.
Serving reasoning models
Similar accuracy can hide different serving costs.
GSM8K · Qwen3-4B · vLLM · RTX 4090
On GSM8K, tested Qwen3-4B configurations that met the study’s accuracy screen used different amounts of GPU time per solved problem. Token counts alone missed differences across kernels and serving forms.
Measure GPU time per solved problem on the workload and at the load you will serve.