From Clojure to ClojureScript

Ever since I started working on the advenjure engine as a learning project for Clojure, I thought porting it to the browser would make a great excuse to get into ClojureScript as well. I finally took the time to do it a couple of weeks ago and now the engine is fully functional in both environments.

The process of going from zero to having some Clojure code running in the browser, source-maps included, was surprisingly easy. Making a fully functional Clojure project target the browser too was a bit more difficult, especially when dealing with JavaScript’s inherent asynchronous nature and setting up the ClojureScript compiler to bundle the project and its dependencies.

I document here the steps I took, useful reads, issues I found along the way and the sources where I got the solutions. For context, I’m using version 1.9.229 of ClojureScript.

Getting Started

Differences from Clojure was a good place to start getting an idea of what ClojureScript looks like coming from Clojure. After that, the Quick Start tutorial was all it took to get my code running in the browser, in Node.js and even in a ClojureScript REPL. With the boilerplate project from the tutorial I was able to start migrating bits of the advenjure codebase and running them in the browser console.

After testing most of the pure-logic parts of the project and being confortable that they could actually run in the browser, I had to deal with the files that were more or less dependant on Java interop. At this point I needed to review the Reader Conditionals documentation, here and here. The key takeaways where: Clojure specific logic goes in .clj files, ClojureScript in .cljs; shared logic goes in .cljc, using the conditional syntax in the parts that differ from one host to another:

(defn str->int [s]
  #?(:clj  (java.lang.Integer/parseInt s)
     :cljs (js/parseInt s)))

Because of how macros work in ClojureScript, those needed to go either in .clj or .cljc files, and required using the :require-macros option:

(ns example.dialogs
  #?(:clj (:require [advenjure.dialogs :refer [dialog]])
     :cljs (:require-macros [advenjure.dialogs :refer [dialog]])))

With Reader Conditionals, println and js/prompt I had a fairly functional version of the example game running in the browser. The next step was to include jQuery terminal in the web page and use it as the interface for the game.

JavaScript interop and asyncrhonous code

Interop syntax is pretty simple, this article and the CloureScript Cheatsheet covered all I needed: the js/ namespace to access JavaScript globals and built-ins, js-obj for object literals, aget and aset to access them, dot prefix to invoke methods.

Things got a bit more complicated as I started integrating the jQuery terminal: my library was more or less a REPL, designed around the idea of waiting for user input and then processing it, but the terminal, as most JavaScript libraries, relied on callbacks and asynchronous processing. When the user enters a command, a processing function is called, which is detached from the game loop that holds the current game state and knows how to handle that command.

After googling around, core.async seemed like the most suitable tool to emulate the synchronicity that my codebase required. I’ve read about it earlier in the Brave Clojure book; this article was also helpful to get code samples.

My solution was to create an input channel where the jQuery terminal would write the commands:

(ns advenjure.ui.input
  (:require-macros [cljs.core.async.macros :refer [go]])
  (:require [cljs.core.async :refer [<! >! chan]]))

(def input-chan (chan))

(defn process-command
  "Callback passed to jQuery terminal upon initialization"
  [command]
  (go (>! input-chan command)))

(defn get-input
  "Wait for input to be written in the input channel"
  [state]
  (go (<! input-chan)))

The main game loop that used to block waiting for user input now was a go-loop that “parked” until data came into the input channel:

(ns advenjure.game
  (:require [advenjure.ui.input :refer [get-input exit]]
            #?(:cljs [cljs.core.async :refer [<!]]))
  #?(:cljs (:require-macros [cljs.core.async.macros :refer [go-loop]])))

#?(:clj
    (defn run
      [game-state finished?]
      (loop [state game-state]
        (let [input (get-input state)
              new-state (process-input state input)]
          (if-not (finished? new-state)
            (recur new-state)
            (exit))))))

#?(:cljs
    (defn run
      [game-state finished?]
      (go-loop [state game-state]
        (let [input (<! (get-input state))
              new-state (process-input state input)]
          (if-not (finished? new-state)
            (recur new-state)
            (exit))))))

This works well although it requires some amount of duplication between the Clojure and ClojureScript versions of the code. Advenjure dialog trees introduce more sophisticated ways of reading and processing user input, which threatened to leak the core.async logic into other portions of the codebase, thus causing more duplication. I managed to keep that to an acceptable minimum without loosing functionality, but there’s definitely room for improvement, perhaps coming up with some macro that abstracts host-specific differences behind a common syntax.

Reading, evaluating and persisting Clojure code

Some of the features of advenjure, such as dialogs and post/pre conditions, required storing quoted Clojure code in the game state, for later evaluation. I found that some built-ins I used to implement them, like read-string and eval, are not directly available in ClojureScript, but a bit of googling revealed how to bring them back.

Based on this article I came up with the following function to replace the native eval, using the tools in the cljs.js namespace:

(ns advenjure.eval
  (:require [cljs.js]))

(defn eval [form] (cljs.js/eval
                    (cljs.js/empty-state)
                    form
                    {:eval cljs.js/js-eval
                     :source-map true
                     :context :expr}
                    :value))

As I learned later on, this snippet comes with one catch: when using Self-hosted ClojureScript (which is what cljs.js enables, evaluating ClojureScript code inside ClojureScript), you can’t use advanced compiler optimizations in your build.

While it’s not a built-in, read-string can be found in cljs.reader/read-string. In the Clojure version of my library, I was able to easily save and restore the game state to a file:

(defn save-game [game-state]
 (spit "saved.game" game-state))

(defn load-game []
 (read-string (slurp "saved.game")))

I intended to do the same in ClojureScript, using the browser localStorage. This didn’t work right away, though, because the ClojureScript reader doesn’t know how to read records back from the storage. This script gave me the solution:

(require '[cljs.reader :refer [read-string register-tag-parser!]]
         '[advenjure.items :refer [map->Item]]
         '[advenjure.rooms :refer [map->Room]])

(defn save-game [game-state]
 (aset js/localStorage "saved.game" (pr-str game-state)))

(register-tag-parser! "advenjure.items.Item" map->Item)
(register-tag-parser! "advenjure.rooms.Room" map->Room)

(defn load-game []
 (read-string (aget js/localStorage "saved.game")))

Leiningen cljsbuild plugin

So far I was doing all the work inside the hello-world project from the Quick Start tutorial. Now that most of the engine was working in ClojureScript I had to integrate it back into the Clojure project and fix anything I broke to make sure it targeted both platforms. I’m using Leiningen so I looked into the lein-cljsbuild plugin. Since advenjure is a library intended to be used as a dependency in other projects, it didn’t matter much what configuration I put in there; the example project, though, ended up with the following configuration in its project.clj:

:plugins [[lein-cljsbuild "1.1.4"]]
:cljsbuild
  {:builds
   {:main {:source-paths ["src"]
           :compiler {:output-to "main.js"
                      :main example.core
                      :optimizations :simple
                      :pretty-print false
                      :optimize-constants true
                      :static-fns true}}

    :dev {:source-paths ["src"]
          :compiler {:output-to "dev.js"
                     :main example.core
                     :optimizations :none
                     :source-map true
                     :pretty-print true}}}}

Then, running lein cljsbuild once would compile development and production versions of the game to be included in the HTML page. Note that, as mentioned, I couldn’t use :optimizations :advanced in the production build, because I was using the cljs.js namespace in my project.

Regular Expressions

Some of the features of advenjure relied on regular expressions. The Clojure related functions are backed by the host implementation of regexes, and JavaScript doesn’t support named capturing groups. To overcome this without changing the original code, I resorted to XRegExp, which fortunately respects the native JavaScript interfaces for regular expressions:

(def regexp #?(:clj re-pattern :cljs js/XRegExp))

(defn match-verb [verb-pattern input]
  (re-find (regexp verb-pattern) input))

Bundling foreign libs

Once everything worked as expected, I needed to figure out how to pack the library so it could be easily included in projects with minimum effort. Particularly, I needed a way to bundle the JavaScript dependencies (jQuery, jQuery terminal, etc.), so the users wouldn’t need to include them manually in their HTML. This topic can get a bit complex in ClojureScript, especially when dealing with advanced optimizations (which I learned along the way I wasn’t going to use). This and this are good references.

The CLJSJS project is an initiative that allows to easily require JavaScript libraries like regular Clojure dependencies. The problem is that the amount of supported libraries is limited, and contributing one of your own is not a trivial process (specifically, it seems to require Boot, and since I was already set up with Leiningen it didn’t look like an option at the moment).

I had to fallback to using the foreign-libs compiler option. For some reason, I couldn’t figure out how to make that work from the cljsbuild settings in my project.clj, so after reviewing this wiki entry I decided to include a deps.cljs file in the root of my source directory:

{:foreign-libs
 [{:file "jquery/jquery-3.1.1.js"
 :file-min "jquery/jquery-3.1.1.min.js"
 :provides ["jquery"]}
 {:file "jquery.terminal/jquery.terminal-0.11.10.js"
 :file-min "jquery.terminal/jquery.terminal-0.11.10.min.js"
 :requires ["jquery"]
 :provides ["jquery.terminal"]}
 {:file "jquery.terminal/jquery.mousewheel.js"
 :file-min "jquery.terminal/jquery.mousewheel.min.js"
 :requires ["jquery"]
 :provides ["jquery.mousewheel"]}
 {:file "xregexp/xregexp-all.js"
 :file-min "xregexp/xregexp-all.min.js"
 :provides ["xregexp"]}]
 :externs ["jquery/externs.js" "jquery.terminal/externs.js" "xregexp/externs.js"]}

Some notes about it:

  • I had to add the files and minified files to the resources folder of the library, to be used in the development and production builds respectively.
  • I needed to define a :provides name and require it in my codebase (no matter if the library exposes a global value that’s actually accesible through js/), in order for the compiler to include the library in the generated build.
  • The :requires is also important to establish dependencies between libraries; without it, the jQuery terminal code can be included before jQuery, which would cause a reference error when running in the browser.
  • The externs aren’t really necessary, since I wasn’t using advanced optimizations, but if I was I found this tool of great help in generating those files, especially for big libraries like jQuery. Smaller ones, like a jQuery plugin, I could create by hand; the CLJSJS packages can be a good reference in that case.