chickadee » nutils » bench-test

bench-test [BODY ...]syntax
bench-test [BODY ...] OUTLIERsyntax
bench-test ITERS [BODY ...]syntax
bench-test ITERS [BODY ...] OUTLIERsyntax
bench-test ITERS NSAMP [BODY ...]syntax
bench-test ITERS NSAMP [BODY ...] OUTLIERsyntax
bench-test ITERS NSAMP SHOW [BODY ...]syntax
bench-test ITERS NSAMP SHOW [BODY ...] OUTLIERsyntax
  • in - ITERS - number of iterations/sample. Default is 10000.
  • in - NSAMP - number of samples that will be run. Default is 10.
  • in - SHOW - a symbol, if supplied, specifies output options:
    • returnls/retls/ls -- prints summary and returns data list: sample mean, stddev, stderr, variance, number of iters/sample, faults, lowest-data-value, highest-data-value, list of sample values (sample total iteration time/num iters)
    • lsonly -- returns data list without other output
    • reps -- prints sample values and summary of results
    • repsls -- same as reps except also returns data list
    • (#f, empty) -- prints summary only. This is the default.
  • in - OUTLIER - if supplied, is a ratio of the given OUTLIER to the running average of measured values. A typical OUTLIER spec is >= 6.0. The purpose is reducing the number of unusably elevated datapoints. However the greater the number of datapoints screened out, the longer it takes to finish all sample runs. An OUTLIER value is not often useful or needed.
  • Summary output has a number of fields, most are self-explanatory.
    • Body: the expression being measured
    • Test: iterations/sample and number of samples
    • Mean: arithmetic mean, +/- stddev (nanoseconds/iteration)
    • Result: lowest-value//highest-value and median
    • Faults: number of "inverted" results, that is, where iteration start-time > ending-time. These values are excluded. Faults occur with very short interval times and reflect randomness of processing. Greater number of iterations/sample reduce random effects. Logically, as result nanosecs/iter increase fewer faults occur. When many faults are reported, the time to complete all samples lengthens disproportionately.
    • Outliers: described above.
    • (gen. *** sec): report generation time.
 (import nutils)
 ;; "empty body test" 
 ;; Shows minimum interval, in this case running in csi
 (bench-test 100000 100 [])
 ===================================================
 Body: 
 Test: 100000 iterations/sample * 100 samples.
 Mean:    19.01  +/-2.09  (nsec/iter)
 Result lo/hi:   15.13//27.52     Median: 18.57
 Faults: 0      Outliers: 0       (gen. 2.27 sec)

 (bench-test 100000 100 [(+ 1 2)])
 ===================================================
 Body: (+ 1 2)
 Test: 100000 iterations/sample * 100 samples.
 Mean:    50.47  +/-4.59  (nsec/iter)
 Result lo/hi:   43.68//66.91     Median: 49.41
 Faults: 0      Outliers: 0       (gen. 2.60 sec)

 (bench-test 100000 100 [(* 3 (sqrt (+ 233 101)))])
 ===================================================
 Body: (* 3 (sqrt (+ 233 101)))
 Test: 100000 iterations/sample * 100 samples.
 Mean:   331.49  +/-14.78  (nsec/iter)
 Result lo/hi:  312.78//425.71    Median: 328.01
 Faults: 0      Outliers: 0       (gen. 6.14 sec)
 
 (bench-test 100000 100 [(combine #\. "abc")])
 ===================================================
 Body: (combine #\. "abc")
 Test: 100000 iterations/sample * 100 samples.
 Mean:   120.28  +/-7.46  (nsec/iter)
 Result lo/hi:  112.47//154.95    Median: 119.47
 Faults: 0      Outliers: 0       (gen. 3.38 sec)