Rendered at 16:29:26 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
syntacticsalt 17 hours ago [-]
As far as I could tell, benchstat seems to implicitly make a location-shift assumption in its interpretation of the Mann-Whitney U test, and I don't see why that assumption would hold.
witcher 14 hours ago [-]
Can you elaborate? And what effects it can have in practice? (:
tsenart 22 hours ago [-]
At last someone does this!
witcher 22 hours ago [-]
Heh thanks!
I wasn't the first one, Michael Pratt from the Go team did something similar 2y ago ((:
I just keep asking the agent to throw `hyperfine` and or /`criteron` at things and hoping it's enough.
Not a super clear sell here, to me, coming from this position of ignorance. I'd love to have better background, to compare, to know more. I'll try and see, I guess! This at least sounds not so bad:
> I built benchfmt.rs, a zero-dependency Rust benchmark runner with a thread-local heap allocation counter, pprof profiler integration, and the same CLI flags as go test -bench. For those who don’t trust a new benchmarking runtime (you shouldn’t from the start!), it also ships a Criterion and libtest converter that emits Go’s benchfmt format directly.
witcher 14 hours ago [-]
That's fair. One clear thing is that you completely miss allocations.
Having a richer standard way to make benchmarks (with allocs), review them, profile them would improve our agentic workflows IMO
I wasn't the first one, Michael Pratt from the Go team did something similar 2y ago ((:
https://github.com/prattmic/benchfmt-convert
Not a super clear sell here, to me, coming from this position of ignorance. I'd love to have better background, to compare, to know more. I'll try and see, I guess! This at least sounds not so bad:
> I built benchfmt.rs, a zero-dependency Rust benchmark runner with a thread-local heap allocation counter, pprof profiler integration, and the same CLI flags as go test -bench. For those who don’t trust a new benchmarking runtime (you shouldn’t from the start!), it also ships a Criterion and libtest converter that emits Go’s benchfmt format directly.
Having a richer standard way to make benchmarks (with allocs), review them, profile them would improve our agentic workflows IMO