Hi, all Common Lispers.

In the previous post, I introduced "Roswell script", the powerful scripting integration of Roswell. Not only does it allow to provide a command-line interface, but it makes it easy to install via Roswell. In this point of view, Roswell can be regarded as a distribution system for Common Lisp applications.

Today's topic is related — the speed of scripts.

Why my simple Roswell script is so slow?

Let's begin with the following simple Roswell script. It's a pretty simple one that just prints "Hello" and quits.

#!/bin/sh
#|-*- mode:lisp -*-|#
#|
exec ros -Q -- $0 "$@"
|#
(progn ;;init forms
  (ros:ensure-asdf))

(defpackage :ros.script.hello.3850273748
  (:use :cl))
(in-package :ros.script.hello.3850273748)

(defun main (&rest argv)
  (declare (ignorable argv))
  (write-line "Hello"))
;;; vim: set ft=lisp lisp:

$ ./hello.ros
Hello

Though people expect this simple script to end instantly, it takes longer.

$ time ./hello.ros
Hello

real    0m0.432s
user    0m0.315s
sys    0m0.117s

0.4 seconds to print "Hello". It feels like an early scripting language.

An equivalent script with sbcl --script is much faster for the record.

$ time ./hello.lisp
Hello

real    0m0.006s
user    0m0.006s
sys    0m0.000s

Fortunately, there're several solutions to this problem.

Hacks to speedup

I have to admit that the Roswell script can't be faster than sbcl --script since it does many things, but it's possible to make it closer.

Stop loading Quicklisp

The first bottleneck is "Quicklisp".

Quicklisp is a de fact standard and available anywhere today, so we may not realize the cost of loading it. But, it can't be ignored in scripting.

Fortunately, it's easy to disable Quicklisp in the Roswell script. Just replace -Q with +Q in the exec line.

 #!/bin/sh
 #|-*- mode:lisp -*-|#
 #|
-exec ros -Q -- $0 "$@"
+exec ros +Q -- $0 "$@"
 |#
 (progn ;;init forms
   (ros:ensure-asdf))

Let's see the difference.

# No Quicklisp version
$ time ./hello.ros
Hello

real    0m0.142s
user    0m0.119s
sys    0m0.020s

It's approximately 0.3 seconds faster. Conversely, it takes this long to load Quicklisp. This is not little time for a program that starts many times, like scripts.

Additionally, omit ros:ensure-asdf since ASDF is unnecessary in this script.

 exec ros +Q -- $0 "$@"
 |#
 (progn ;;init forms
-  (ros:ensure-asdf))
+  )

 (defpackage :ros.script.hello.3850273748
   (:use :cl))

$ time ./hello.ros
Hello

real    0m0.072s
user    0m0.052s
sys    0m0.020s

ASDF seems to require to load approximately 0.07 sec. Now, it's 6 times faster.

These changes are effective for small scripts which don't require Quicklisp or ASDF.

Even in the case of scripts that use Quicklisp and ASDF, this method can be applied partially by loading them conditionally.

For example, a script has several subcommands like run or help.

Let's suppose run requires Quicklisp to load the main application, and help doesn't.

If you use -Q option, Quicklisp will make help command slow though it doesn't require Quicklisp.

In this case, it is better to use the +Q option and load Quicklisp if necessary.

ros:quicklisp is a function to load Quicklisp manually, even when the ros started with +Q. By calling this function right before Quicklisp is needed, it's possible to make the other part faster.

Dump core with -m

What about in case of fairly complicated applications which must require Quicklisp to load external dependencies.

Building a binary is a prevailing solution.

I suppose it won't surprise you. It's a common technique even in no Roswell world. Also, I've mentioned ros build in the previous article, which makes a binary executable from a Roswell script.

However, we can't assume people always run ros build to speed up your application after installation.

Roswell takes care of it. Roswell has a feature to build the script dump implicitly to speed up its execution.

Add -m option to the exec ros line. Then, enable Quicklisp and ASDF to see how this feature is practical.

@@ -1,10 +1,10 @@
 #!/bin/sh
 #|-*- mode:lisp -*-|#
 #|
-exec ros +Q -- $0 "$@"
+exec ros -Q -m hello -- $0 "$@"
 |#
 (progn ;;init forms
-  )
+  (ros:ensure-asdf))

 (defpackage :ros.script.hello.3850273748
   (:use :cl))

And install the script.

$ ros install hello.ros
/home/fukamachi/.roswell/bin/hello

Let's try it. It'll take a little time for Roswell to dump a core named hello.core for the first time.

$ hello
Making core for Roswell...
building dump:/home/fukamachi/.roswell/impls/arm64/linux/sbcl-bin/2.2.0/dump/hello.core
WARNING: :SB-EVAL is no longer present in *FEATURES*
Hello

The second time, it's way faster.

$ time hello
Hello

real    0m0.032s
user    0m0.009s
sys    0m0.024s

It's approximately 13 times faster than the initial version. Of course, it includes the load time of Quicklisp and ASDF.

Remember that this requires the script to be installed at ~/.roswell/bin via ros install.

A living example is "lem", a text editor written in Common Lisp.

In the case of "lem", it requires lots of dependencies to run, and people expect a text editor to launch instantly. The dumping core works nicely for it.

# Installation
$ ros install lem-project/lem

# Takes a little time for the first time
$ lem
Making core for Roswell...
building dump:/home/fukamachi/.roswell/impls/arm64/linux/sbcl-bin/2.2.0/dump/lem-ncurses.core
WARNING: :SB-EVAL is no longer present in *FEATURES*

# === lem is opened in fullscreen ===
# Type C-x C-c to quit

It takes a little time to boot up the first time, but the second time is quicker. Also, it'll be dumped again when Roswell detects some file changes.

Note that the name of the core needs to be unique. If there is a conflict, Roswell will load a different core.

Actually, this behavior doesn't go with Qlot well. If there's an application installed in user local and another installed in project local, Roswell can't distinguish between their cores. So then, even if you think you have fixed the version of the library, it will be using a core with a different version loaded.

This is not a problem for independent software like lem, but you should be careful with applications that load other software while running. A bad example of this problem is "Lake".

Conclusion

In this article, I introduced a technique to speed up the startup of Roswell scripts.

• To startup faster
  • Add +Q to disable loading QuicklispF
  • Dump cores with -m option

Both have pros and cons.

The nice thing about the -m option is that the end-user doesn't need to be aware of it, which is a good part of Roswell as a distribution system for Common Lisp applications.