Skip to content

Benchmarks ​

These numbers come from docs/benchmarks.md in the repository (milestone D1, "dogfooding"): measure first, then remove reference-counting work in the backend without touching the programs. They compare the baseline code generation with the RcOpt pass. They were measured by the project, on a 4-core machine with clang 18 at -O1, best of 7 runs; they are not re-measured by this site's CI.

How to measure ​

  • Counters. minic build --stats prog.kt -o prog compiles runtime and program with -DMK_STATS. At exit the program prints four mk-stats: lines to stderr: allocations, frees, live objects at exit and peak, bytes (total, live, peak), and retain/release calls split into non-atomic, atomic (shared objects) and no-ops (null, static or stack objects). In a --stats build every retain and release is a runtime call so the counters see all of them.
  • Benchmark driver. compiler/bench/run.sh [--reps N] [--configs "base opt"] [--only name]. For every benchmark it builds both configurations (base = --no-rc-opt, opt = RC elision), checks that the output is identical, and takes the best time of N runs. Results go to compiler/build/bench/results.md.
  • Benchmarks (compiler/bench/micro/*.kt): arraylist, hashmap, stringbuilder, mapfilter (map/filter/fold chains), tree (binary-tree building), templates (string templates). Plus the self-hosted backend minic-backend turning a MIR file into C: selfhost-small and selfhost-large (the MIR of the backend itself, the largest input there is).

Results ​

base is the baseline (--no-rc-opt). opt is RcOpt plus inline retain/release fast paths in the generated C. Counts are logical operations from the --stats build.

benchmarkconfigtime (ms)allocsretainsreleases
arraylistbase4054 200 35767 417 45781 026 310
arraylistopt2654 200 35725 608 56037 017 688
hashmapbase1021 285 09411 218 44119 014 613
hashmapopt721 285 0945 511 36112 618 633
mapfilterbase2471 937 30836 133 67843 114 204
mapfilteropt1461 937 3088 853 11013 737 200
stringbuilderbase921 000 06212 400 02114 400 104
stringbuilderopt601 000 0624 200 0005 200 062
templatesbase1094 200 000600 0005 100 000
templatesopt1054 200 00004 200 000
treebase1002 172 8528 277 84618 128 592
treeopt612 172 85203 745 746
selfhost-smallbase62569 4185 294 3207 939 640
selfhost-smallopt49569 4181 781 7053 140 155
selfhost-largebase7495 290 60754 561 33283 114 527
selfhost-largeopt6035 290 60717 290 45830 511 031

Reading the table ​

  • Time drops from about 5 percent (templates, dominated by allocating the temporary strings) to 40 percent (tree, mapfilter, arraylist).
  • Retains drop by 60 to 100 percent, releases by 40 to 80 percent.
  • Allocations and peak memory are identical by construction. RcOpt moves and removes counting; it does not change what is allocated or when objects die.
  • The native self-hosted backend gets 20 percent faster on its large input (749 to 603 ms) even though it now also runs the RcOpt analysis on its input. That analysis costs about 120 ms of the 603; compiling with --no-rc-opt is 25 percent cheaper in the backend for a program that then runs slower. A cheaper analysis is a candidate for later work.
  • What remains counted is mostly owned results (ArrayList.get, HashMap.get, iterator next return +1 and the caller releases), boxing, and stores into collections. They would need borrowed return values and owned parameters for virtual slots, which are next steps.

Where the counting came from ​

Found with the counters and by reading the generated C:

  1. every read of a reference field was copied into a retained temporary;
  2. traversal variables (p = p.next) retained the next node and released the old one on every iteration;
  3. constructor arguments such as Tree(make(d - 1), make(d - 1)) created owned temporaries that the constructor retained and the caller released again;
  4. return x as T of an owned temporary (every generic getter) retained and released;
  5. releases of NULL were calls into the runtime;
  6. string temporaries of templates: one retain and release per piece, unchanged by D1.

The rules that remove them are in Reference counting and RcOpt.

Reproducing ​

sh
./mini --version
bash compiler/bench/run.sh --reps 7           # about 8 minutes; --only tree for one benchmark
cat compiler/build/bench/results.md
./minic build prog.kt -o prog --stats         # counters for any program; --no-rc-opt for the baseline

Timing noise on a shared 4-core machine is about 5 percent; the counters are exact.

Memory checker and annotations ​

None of these numbers needed @Owned annotations, free() calls or Strict mode. The conventions RcOpt infers (stored constructor parameters are owned, everything else borrowed, field reads are borrows while nothing overwrites a heap reference) are exactly what @Owned and @Borrowed state in Strict code, so the inference rules may become the default for unannotated parameters there.

Cross-language baseline ​

The same eight programs (binarytrees, nbody, fannkuch, spectralnorm, mandelbrot, wordcount, strbuild, json) in C, Rust, Go, Kotlin/JVM, Kotlin/Native and Mini, run by bench/run.sh. Host: Intel Xeon @ 2.30 GHz VM, 4 cores, Linux 6.18; clang 18.1.3, rustc 1.97.0, Go 1.24.7, OpenJDK 21, Kotlin 2.1.20. Best of 5; the output of every run is checked. Mini is built with minic build --cflag -O2; Mini-M is Managed mode with the C backend, Mini-M-ll the LLVM backend. The Arena and Strict variants (only binarytrees) are in docs/benchmarks.md. This run used the runtime before the new allocator (see below). It is a VM: differences below about 5 percent are noise.

Wall time (seconds)

benchmarkCRustGoJVMK/NMini-MMini-M-ll
binarytrees0.951.101.570.712.471.691.65
nbody0.990.751.581.201.211.801.98
fannkuch2.131.832.062.142.306.817.35
spectralnorm1.011.001.001.781.041.992.01
mandelbrot0.870.820.850.830.840.850.81
wordcount0.530.911.273.293.914.394.37
strbuild0.580.510.990.521.841.641.65
json0.820.500.610.760.791.431.79

Time relative to C (1.00 = C at -O2; higher is slower)

benchmarkCRustGoJVMK/NMini-MMini-M-ll
binarytrees1.001.161.650.752.611.781.75
nbody1.000.761.591.211.221.812.00
fannkuch1.000.860.961.001.083.193.44
spectralnorm1.000.990.991.761.031.971.99
mandelbrot1.000.950.970.960.970.980.92
wordcount1.001.712.386.157.318.218.17
strbuild1.000.881.710.903.192.842.86
json1.000.610.740.930.961.742.17

Peak RSS (MiB)

benchmarkCRustGoJVMK/NMini-MMini-M-ll
binarytrees343446501141130130
nbody22242322
fannkuch22242322
spectralnorm22242322
mandelbrot22240322
wordcount8886289414367328328
strbuild104104447397611205205
json10375168479364173173

Reading it honestly. Mini is on par with C on mandelbrot; about 1.7 to 2x C on binarytrees, nbody, spectralnorm and json; and far behind on fannkuch (3.2x), strbuild (2.8x) and wordcount (8x). Kotlin/Native is faster than Mini on most of them; Mini is ahead on binarytrees and strbuild. Start-up is 1.0 ms and the binaries are 80 to 210 KiB.

After the allocator. With the thread-local size-class allocator of PR #18 compiled into the same programs (Mini Managed, C backend, same host, best of 5):

benchmarkCMini, old runtimeMini, new runtime
binarytrees0.951.681.18
nbody0.991.761.74
fannkuch2.136.847.07
spectralnorm1.012.012.02
mandelbrot0.870.850.85
wordcount0.534.614.32
strbuild0.581.631.50
json0.821.941.43

The allocator helps where objects are allocated and freed in large numbers (binarytrees -30 percent, json -26 percent) and changes nothing elsewhere.

Quick wins. Inline array accessors (C and LLVM), borrowed array element reads in the reference-counting optimizer, and an snprintf-free integer to string conversion, Mini Managed, same host, best of 5, before and after:

benchmarkCMini-M beforeMini-M afterMini-M-ll beforeMini-M-ll after
binarytrees0.951.151.221.131.16
nbody0.991.721.031.881.25
fannkuch2.137.062.686.853.43
spectralnorm1.012.001.262.002.01
mandelbrot0.870.900.880.840.80
wordcount0.534.103.423.983.63
strbuild0.581.460.811.490.82
json0.821.361.371.251.33

The LLVM backend did not gain on spectralnorm (cause not found). The table below the profiles describes the state before these changes.

Where the time goes (profiles with valgrind --tool=callgrind on the current runtime; the full analysis is in docs/benchmarks.md):

  • fannkuch: 62 percent of the instructions are in the out-of-line array accessors mk_array_get_int and mk_array_set_int. With the accessors inlined (bounds check kept) it takes 2.65 s instead of 7.2 s.
  • nbody: every array element read retains and releases the element (500 M retains in 25 M steps). Inlining the accessor gives 1.20 s, and borrowing the element without reference counting 0.99 s, equal to C.
  • strbuild: 63 percent of the instructions are snprintf inside mk_string_from_int. A hand-written integer conversion gives 0.93 s instead of 1.48 s.
  • wordcount: the generic HashMap does two lookups per word (get and put), hashes the string every time, goes through virtual hashCode/equals, casts and out-of-line node accessors, and boxes every count. About 1,300 instructions per word.
  • json: out-of-line String.get per character (8.5 percent), is checks that walk the type hierarchy (6.3 percent), allocation and release.
  • binarytrees: allocation, zeroing and recursive release are the whole program; 137 M of 205 M releases are no-ops on null.

Common causes are that the runtime is a separate translation unit without LTO (so no accessor is inlined) and that array element reads are not borrowed by the reference-counting optimizer. Both are fixable in the backend; -flto alone already helps nbody by 24 percent and json by 12 percent.

To reproduce: ./mini --version && bench/run.sh --reps 5 --stats (results in bench/results/).

Released under the Apache License 2.0.