criterion performance measurements

overview

want to understand this report?

sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vegito

5
4.95
5.05
5.10
5.15
5.2
5.25
5.30
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vegito time densities
mean
10
15
20
25
30
35
40
45
5×10³ iters
100
150
200
250
0 s
50 ms
regression
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vegito times
lower bound estimate upper bound
OLS regression 5.05 μs 5.06 μs 5.07 μs
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 5.05 μs 5.06 μs 5.08 μs
Standard deviation 27.6 ns 46.5 ns 87.6 ns

Outlying measurements have no (0.6%) effect on estimated standard deviation.

sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/gotenks

5
5.05
5.10
5.15
5.2
5.25
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/gotenks time densities
mean
10
15
20
25
30
35
40
45
5×10³ iters
100
150
200
250
0 s
50 ms
regression
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/gotenks times
lower bound estimate upper bound
OLS regression 5.06 μs 5.07 μs 5.09 μs
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 5.06 μs 5.08 μs 5.09 μs
Standard deviation 39.4 ns 48.8 ns 64.4 ns

Outlying measurements have slight (5.6%) effect on estimated standard deviation.

sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/conduit-combinators unqualified

1.05
1.1
1.15
1.2
1.25
1.3
1.35
1.4
1.45
1.5
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/conduit-combinators unqualified time densities
mean
100
150
200
50 iters
100
150
200
250
300
0 s
50 ms
regression
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/conduit-combinators unqualified times
lower bound estimate upper bound
OLS regression 1.16 ms 1.24 ms 1.32 ms
R² goodness-of-fit 0.973 0.981 0.995
Mean execution time 1.12 ms 1.14 ms 1.17 ms
Standard deviation 53.4 μs 85.3 μs 128 μs

Outlying measurements have severe (58.8%) effect on estimated standard deviation.

sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/conduit-combinators qualified

5.00
5.20
5.4
5.60
5.80
6.00
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/conduit-combinators qualified time densities
mean
10
15
20
25
30
35
40
45
5×10³ iters
100
150
200
250
0 s
50 ms
regression
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/conduit-combinators qualified times
lower bound estimate upper bound
OLS regression 5.06 μs 5.07 μs 5.08 μs
R² goodness-of-fit 0.999 1.000 1.000
Mean execution time 5.07 μs 5.10 μs 5.20 μs
Standard deviation 56.5 ns 162 ns 305 ns

Outlying measurements have moderate (39.6%) effect on estimated standard deviation.

sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/conduit

5.00
5.20
5.4
5.60
5.80
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/conduit time densities
mean
10
15
20
25
30
35
40
45
5×10³ iters
100
150
200
250
0 s
50 ms
regression
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/conduit times
lower bound estimate upper bound
OLS regression 5.05 μs 5.08 μs 5.12 μs
R² goodness-of-fit 0.998 0.999 1.000
Mean execution time 5.06 μs 5.08 μs 5.15 μs
Standard deviation 35.1 ns 120 ns 246 ns

Outlying measurements have moderate (26.9%) effect on estimated standard deviation.

sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vector boxed

5
5.05
5.10
5.15
5.2
5.25
5.3
5.35
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vector boxed time densities
mean
10
15
20
25
30
35
40
45
5×10³ iters
100
150
200
250
0 s
50 ms
regression
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vector boxed times
lower bound estimate upper bound
OLS regression 5.05 μs 5.06 μs 5.07 μs
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 5.06 μs 5.07 μs 5.09 μs
Standard deviation 31.9 ns 53.1 ns 86.3 ns

Outlying measurements have slight (6.9%) effect on estimated standard deviation.

sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vector unboxed

5.00
5.20
5.4
5.60
5.80
6.00
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vector unboxed time densities
mean
10
15
20
25
30
35
40
45
5×10³ iters
100
150
200
250
0 s
50 ms
regression
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vector unboxed times
lower bound estimate upper bound
OLS regression 5.07 μs 5.11 μs 5.16 μs
R² goodness-of-fit 0.998 0.999 1.000
Mean execution time 5.08 μs 5.10 μs 5.20 μs
Standard deviation 51.9 ns 141 ns 299 ns

Outlying measurements have moderate (32.9%) effect on estimated standard deviation.

sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vector unboxed foldM

5
5.05
5.10
5.15
5.2
5.25
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vector unboxed foldM time densities
mean
10
15
20
25
30
35
40
45
5×10³ iters
100
150
200
250
0 s
50 ms
regression
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/vector unboxed foldM times
lower bound estimate upper bound
OLS regression 5.05 μs 5.07 μs 5.09 μs
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 5.06 μs 5.07 μs 5.09 μs
Standard deviation 43.9 ns 55.8 ns 75.1 ns

Outlying measurements have slight (7.4%) effect on estimated standard deviation.

sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/direct implementation

5.9
6.00
6.10
6.20
6.30
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/direct implementation time densities
mean
10
15
20
25
30
35
5×10³ iters
100
150
200
250
0 s
50 ms
regression
sum $ map (+ 1) $ map (* 2) $ enumFromTo 1 9001/direct implementation times
lower bound estimate upper bound
OLS regression 5.98 μs 6.02 μs 6.08 μs
R² goodness-of-fit 0.999 1.000 1.000
Mean execution time 5.98 μs 6.00 μs 6.04 μs
Standard deviation 59.8 ns 82.2 ns 123 ns

Outlying measurements have moderate (10.9%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.