Comparing programming languages XII: The Native showdown

Introduction

Recently, I had a nice conversation on Twitter/X (Ah, the good old days when Twitter wasn't so terrible) about Scala Native, Kotlin, and so on.

The present post is inspired by that conversation.

The challenge

Over the last 5 years (and counting), I wrote several implementations of the Monkey Language interpreter. We'll test the performance of the implementations that compile natively. The traditional performance test for a Monkey Language implementation is to run a recursive Fibonacci function with 35 as a parameter.

let fibonacci = fn(x) {    	
    if (x < 2) {
    	return x;
    } else {
    	fibonacci(x - 1) + fibonacci(x - 2);
    }
};
fibonacci(35);

All tests are running on an MBP 14" M5 32 GB.

First fight: Kotlin Native vs Scala Native

In the blue corner, with a size of 949 Kb and consuming 14.87 Mb of memory... Kotlin Native 2.3.10.

In the red corner, with a size of 6.1 Mb and consuming 7.45 Mb of memory... Scala Native 0.5.10.

https://gist.github.com/MarioAriasC/1ed3d286f1dcdcd35b11b7a6caac34fe

9.46 faster for Kotlin, what a beat down!!.

This scandalous figure captured the attention of Lorenzo Gabriele, who pointed out that my code was using Scala boundaries, which are not fast in Scala Native. Lorenzo raised a PR to my Scala Native branch, changing the boundaries for native continuations.

Second fight: Kotlin Native vs Scala Native (with continuations) II

In the red corner, looking for a rematch, with a size of 6.5 Mb and consuming 7.31 Mb of memory... Scala Native with continuations

https://gist.github.com/MarioAriasC/4837dcb19243e7af8411fa0eba7f704b

Scala beats Kotlin by 1.23 times faster run.

Wow, very good. There is a thing that I don't like, and is that I'm using code that is specific for Scala Native (the continuations), and even when it compiles with the Scala JVM compiler it doesn't run. It is expected that in the next versions of Scala Native, we will not need to use the native continuations explicitly.

Third fight: Kotlin Native (with inline) vs Scala Native (with continuations) III

What is inline

It is a compiler option to enable inline code generation, which can increases performance in certain scenarios.

compilerOptions { 
  freeCompilerArgs.add("-Xbinary=preCodegenInlineThreshold=40") 
}

Coming back from defeat, in the blue corner, with a size of 996 Kb and consuming 14.96 Mb of Memory... Kotlin Native with inline

https://gist.github.com/MarioAriasC/8183c3bc1cb6ecab28b7ab33ad5a3cf1

Scala beats Kotlin again, but "only" with a 1.11 times faster run, which means that online gives us a ~ 12% increase. There are a lot more options to compile Kotlin Native, and maybe another combination of options can give us even better performance.

At this point, I must say that despite being slower (not by much), the Native DX is way better in Kotlin compared to Scala.

Fourth Fight: Kotlin Graal vs Scala Native (with continuations)

Looking to avenge the defeat of its brother, in the blue corner, with a size of 14 Mb and consuming 59.03 Mb of Memory(!!)... Kotlin Graal 25.0.2

https://gist.github.com/MarioAriasC/3c38183080b5ea7eaf8023016182aa4c

What a beatdown, Kotlin Graal is 3.22 times faster than Scala Native with continuations.

But hey, theoretically, it should be possible to compile Scala with Graal as well, no?

Fifth Fight: Kotlin Graal vs Scala Graal

Revenge is game that can be played by two. In the red corner, with a size of 15 Mb and consuming 59.20 Mb of Memory... Scala Graal 25.0.2

https://gist.github.com/MarioAriasC/a8c0016dc6d90f57fc67d18103730c37

Huh? Kotlin Graal is even faster compared to Scala Graal by 4.68 times!?!?

What is going on?

I'm not an expert on Graal, but I have a theory.

First, the facts. Graal's native-image takes the compiled bytecode generated by Kotlin and Scala and builds a native binary from it. The bytecode generated by Scala is big and complex, and the Scala standard library is 5 times the size of its Kotlin equivalent.

E.g. The jars generated by Kotlin for this project:

18k  annotations-13.0.jar 
1.8M kotlin-stdlib-2.3.10.jar 
162k monkey-common-jvm-25.jar 
261  monkey-jvm.jar

The jars generated by Scala for this project:

357k langur.langur-0.1.0-SNAPSHOT.jar 
9.2M org.scala-lang.scala-library-3.8.2.jar 
319  org.scala-lang.scala3-library_3-3.8.2.jar

My theory is that Graal's native-image cannot generate efficient native binaries from Scala bytecode. Is it maybe related to Scala boundaries?.

But in short, Scala Native is faster than Scala Graal, and the other way around, Kotlin Graal is faster than Kotlin Native.

Sixth Fight: Kotlin Graal vs Go

A new challenger, in the red corner, with a size of 2.8 Mb and consuming 27.75 Mb of Memory... Go 1.26.1!!

https://gist.github.com/MarioAriasC/d3d1312aa4d44045fd0718f80d00a3eb

Kotlin Graal defeats Go, 1.20 faster!!

Kind of surprised by this result, I thought that Kotlin Graal and Go would be closer.

Who can defeat Kotlin Graal?

Seventh Fight: Kotlin Graal vs Crystal

Our main event: in the red corner, with a size of 748 Kb and consuming 3.90 Mb of Memory... Crystal 1.19.0

https://gist.github.com/MarioAriasC/080064e5b099d0d21f5b987edcd12e38

And there you have it, Crystal is the fastest beating Kotlin Graal by 1.58 times.

Not only is Crystal the fastest, but it also has the smallest binary and the lowest memory consumption, what a bargain!.

Crystal is the most underhyped and overdelivered language of the last decade. Beautiful code, amazing compiler. It does need more tooling around it, but I hope that it gets the attention that it deserves, and then maybe the tooling gets better.

The tale of the tape

Binary	Best average run time(s)	Size (Mb)	Memory (Mb)
Scala Native	165.115	6.1	7.45
Scala Graal	23.878	15	59.20
Kotlin Native	17.746	0.949	14.87
Kotlin Native with inlines	16.397	0.996	14.96
Scala Native with Continuations	14.765	6.5	7.31
Go	6.003	2.8	27.75
Kotlin Graal	4.800	14	59.03
Crystal	3.231	0.748	3.9

FAQs

What about X/Y/Z language?

If I don't have it on this list, it's because I don't have an implementation for it. But I'm working on more implementations so I can extend this further.

Why don't you use an LLM to write other implementations?

I don't use LLMs.