From Clojure to ClojureScript

Ever since I started working on the advenjure engine as a learning project for Clojure, I thought porting it to the browser would make a great excuse to get into ClojureScript as well. I finally took the time to do it a couple of weeks ago and now the engine is fully functional in both environments.

The process of going from zero to having some Clojure code running in the browser, source-maps included, was surprisingly easy. Making a fully functional Clojure project target the browser too was a bit more difficult, especially when dealing with JavaScript’s inherent asynchronous nature and setting up the ClojureScript compiler to bundle the project and its dependencies.

I document here the steps I took, useful reads, issues I found along the way and the sources where I got the solutions. For context, I’m using version 1.9.229 of ClojureScript.

Getting Started

Differences from Clojure was a good place to start getting an idea of what ClojureScript looks like coming from Clojure. After that, the Quick Start tutorial was all it took to get my code running in the browser, in Node.js and even in a ClojureScript REPL. With the boilerplate project from the tutorial I was able to start migrating bits of the advenjure codebase and running them in the browser console.

After testing most of the pure-logic parts of the project and being confortable that they could actually run in the browser, I had to deal with the files that were more or less dependant on Java interop. At this point I needed to review the Reader Conditionals documentation, here and here. The key takeaways where: Clojure specific logic goes in .clj files, ClojureScript in .cljs; shared logic goes in .cljc, using the conditional syntax in the parts that differ from one host to another:

(defn str->int [s]
  #?(:clj  (java.lang.Integer/parseInt s)
     :cljs (js/parseInt s)))

Because of how macros work in ClojureScript, those needed to go either in .clj or .cljc files, and required using the :require-macros option:

(ns example.dialogs
  #?(:clj (:require [advenjure.dialogs :refer [dialog]])
     :cljs (:require-macros [advenjure.dialogs :refer [dialog]])))

With Reader Conditionals, println and js/prompt I had a fairly functional version of the example game running in the browser. The next step was to include jQuery terminal in the web page and use it as the interface for the game.

JavaScript interop and asyncrhonous code

Interop syntax is pretty simple, this article and the CloureScript Cheatsheet covered all I needed: the js/ namespace to access JavaScript globals and built-ins, js-obj for object literals, aget and aset to access them, dot prefix to invoke methods.

Things got a bit more complicated as I started integrating the jQuery terminal: my library was more or less a REPL, designed around the idea of waiting for user input and then processing it, but the terminal, as most JavaScript libraries, relied on callbacks and asynchronous processing. When the user enters a command, a processing function is called, which is detached from the game loop that holds the current game state and knows how to handle that command.

After googling around, core.async seemed like the most suitable tool to emulate the synchronicity that my codebase required. I’ve read about it earlier in the Brave Clojure book; this article was also helpful to get code samples.

My solution was to create an input channel where the jQuery terminal would write the commands:

(ns advenjure.ui.input
  (:require-macros [cljs.core.async.macros :refer [go]])
  (:require [cljs.core.async :refer [<! >! chan]]))

(def input-chan (chan))

(defn process-command
  "Callback passed to jQuery terminal upon initialization"
  [command]
  (go (>! input-chan command)))

(defn get-input
  "Wait for input to be written in the input channel"
  [state]
  (go (<! input-chan)))

The main game loop that used to block waiting for user input now was a go-loop that “parked” until data came into the input channel:

(ns advenjure.game
  (:require [advenjure.ui.input :refer [get-input exit]]
            #?(:cljs [cljs.core.async :refer [<!]]))
  #?(:cljs (:require-macros [cljs.core.async.macros :refer [go-loop]])))

#?(:clj
    (defn run
      [game-state finished?]
      (loop [state game-state]
        (let [input (get-input state)
              new-state (process-input state input)]
          (if-not (finished? new-state)
            (recur new-state)
            (exit))))))

#?(:cljs
    (defn run
      [game-state finished?]
      (go-loop [state game-state]
        (let [input (<! (get-input state))
              new-state (process-input state input)]
          (if-not (finished? new-state)
            (recur new-state)
            (exit))))))

This works well although it requires some amount of duplication between the Clojure and ClojureScript versions of the code. Advenjure dialog trees introduce more sophisticated ways of reading and processing user input, which threatened to leak the core.async logic into other portions of the codebase, thus causing more duplication. I managed to keep that to an acceptable minimum without loosing functionality, but there’s definitely room for improvement, perhaps coming up with some macro that abstracts host-specific differences behind a common syntax.

Reading, evaluating and persisting Clojure code

Some of the features of advenjure, such as dialogs and post/pre conditions, required storing quoted Clojure code in the game state, for later evaluation. I found that some built-ins I used to implement them, like read-string and eval, are not directly available in ClojureScript, but a bit of googling revealed how to bring them back.

Based on this article I came up with the following function to replace the native eval, using the tools in the cljs.js namespace:

(ns advenjure.eval
  (:require [cljs.js]))

(defn eval [form] (cljs.js/eval
                    (cljs.js/empty-state)
                    form
                    {:eval cljs.js/js-eval
                     :source-map true
                     :context :expr}
                    :value))

As I learned later on, this snippet comes with one catch: when using Self-hosted ClojureScript (which is what cljs.js enables, evaluating ClojureScript code inside ClojureScript), you can’t use advanced compiler optimizations in your build.

While it’s not a built-in, read-string can be found in cljs.reader/read-string. In the Clojure version of my library, I was able to easily save and restore the game state to a file:

(defn save-game [game-state]
 (spit "saved.game" game-state))

(defn load-game []
 (read-string (slurp "saved.game")))

I intended to do the same in ClojureScript, using the browser localStorage. This didn’t work right away, though, because the ClojureScript reader doesn’t know how to read records back from the storage. This script gave me the solution:

(require '[cljs.reader :refer [read-string register-tag-parser!]]
         '[advenjure.items :refer [map->Item]]
         '[advenjure.rooms :refer [map->Room]])

(defn save-game [game-state]
 (aset js/localStorage "saved.game" (pr-str game-state)))

(register-tag-parser! "advenjure.items.Item" map->Item)
(register-tag-parser! "advenjure.rooms.Room" map->Room)

(defn load-game []
 (read-string (aget js/localStorage "saved.game")))

Leiningen cljsbuild plugin

So far I was doing all the work inside the hello-world project from the Quick Start tutorial. Now that most of the engine was working in ClojureScript I had to integrate it back into the Clojure project and fix anything I broke to make sure it targeted both platforms. I’m using Leiningen so I looked into the lein-cljsbuild plugin. Since advenjure is a library intended to be used as a dependency in other projects, it didn’t matter much what configuration I put in there; the example project, though, ended up with the following configuration in its project.clj:

:plugins [[lein-cljsbuild "1.1.4"]]
:cljsbuild
  {:builds
   {:main {:source-paths ["src"]
           :compiler {:output-to "main.js"
                      :main example.core
                      :optimizations :simple
                      :pretty-print false
                      :optimize-constants true
                      :static-fns true}}

    :dev {:source-paths ["src"]
          :compiler {:output-to "dev.js"
                     :main example.core
                     :optimizations :none
                     :source-map true
                     :pretty-print true}}}}

Then, running lein cljsbuild once would compile development and production versions of the game to be included in the HTML page. Note that, as mentioned, I couldn’t use :optimizations :advanced in the production build, because I was using the cljs.js namespace in my project.

Regular Expressions

Some of the features of advenjure relied on regular expressions. The Clojure related functions are backed by the host implementation of regexes, and JavaScript doesn’t support named capturing groups. To overcome this without changing the original code, I resorted to XRegExp, which fortunately respects the native JavaScript interfaces for regular expressions:

(def regexp #?(:clj re-pattern :cljs js/XRegExp))

(defn match-verb [verb-pattern input]
  (re-find (regexp verb-pattern) input))

Bundling foreign libs

Once everything worked as expected, I needed to figure out how to pack the library so it could be easily included in projects with minimum effort. Particularly, I needed a way to bundle the JavaScript dependencies (jQuery, jQuery terminal, etc.), so the users wouldn’t need to include them manually in their HTML. This topic can get a bit complex in ClojureScript, especially when dealing with advanced optimizations (which I learned along the way I wasn’t going to use). This and this are good references.

The CLJSJS project is an initiative that allows to easily require JavaScript libraries like regular Clojure dependencies. The problem is that the amount of supported libraries is limited, and contributing one of your own is not a trivial process (specifically, it seems to require Boot, and since I was already set up with Leiningen it didn’t look like an option at the moment).

I had to fallback to using the foreign-libs compiler option. For some reason, I couldn’t figure out how to make that work from the cljsbuild settings in my project.clj, so after reviewing this wiki entry I decided to include a deps.cljs file in the root of my source directory:

{:foreign-libs
 [{:file "jquery/jquery-3.1.1.js"
 :file-min "jquery/jquery-3.1.1.min.js"
 :provides ["jquery"]}
 {:file "jquery.terminal/jquery.terminal-0.11.10.js"
 :file-min "jquery.terminal/jquery.terminal-0.11.10.min.js"
 :requires ["jquery"]
 :provides ["jquery.terminal"]}
 {:file "jquery.terminal/jquery.mousewheel.js"
 :file-min "jquery.terminal/jquery.mousewheel.min.js"
 :requires ["jquery"]
 :provides ["jquery.mousewheel"]}
 {:file "xregexp/xregexp-all.js"
 :file-min "xregexp/xregexp-all.min.js"
 :provides ["xregexp"]}]
 :externs ["jquery/externs.js" "jquery.terminal/externs.js" "xregexp/externs.js"]}

Some notes about it:

  • I had to add the files and minified files to the resources folder of the library, to be used in the development and production builds respectively.
  • I needed to define a :provides name and require it in my codebase (no matter if the library exposes a global value that’s actually accesible through js/), in order for the compiler to include the library in the generated build.
  • The :requires is also important to establish dependencies between libraries; without it, the jQuery terminal code can be included before jQuery, which would cause a reference error when running in the browser.
  • The externs aren’t really necessary, since I wasn’t using advanced optimizations, but if I was I found this tool of great help in generating those files, especially for big libraries like jQuery. Smaller ones, like a jQuery plugin, I could create by hand; the CLJSJS packages can be a good reference in that case.
Advertisements

Clojure: the good, the bad and the ugly

Four years ago I wrote about my first (and last) experience with Common Lisp. I had high expectations and was disappointed; I ended up thinking maybe I should give Scheme or Clojure a try. It took a while, but I finally did it last month: learn Clojure. And it looks like a keeper.

Clojure has all the goodness of Lisp and functional programming, and it feels like a modern language: it addresses most of things that annoyed me about Common Lisp.

I’ve followed the great Clojure for the brave and true book by Daniel Higginbotham and then I’ve tackled a small project to train my skills. Here are my notes.

The good

  • A consistent syntax, operation names and polymorphic functions. No weird illegible names, no type specific versions of the same function (I’m looking at you Common Lisp).
  • Functional! Immutable! Expressive!
    • I don’t miss objects. Structuring programs in small functions; isolated, never changing data; those things just feel right. And I’m not even waving the old “shared state is bad for concurrency” flag; I don’t care —just now— about concurrency. This stuff makes programs simpler to reason about and test, and more fun to write.
    • There’s more: Clojure is so expressive and gives you enough options (I’m thinking loop, doseq, destructuring, etc.) that you don’t necessarily need to incur in “head/tail” recursive processing as much as I found in other functional languages, so the leap is not so rough.
  • Did I say I don’t care about concurrency? I don’t. Mostly. Not at the language level, anyway. Clojure has a lot of cool tools for concurrency (future, delay, promise, pmap, core.async). Too much options, maybe, but I don’t mind about that either. I can just RTFM whenever I do have the need to do things concurrently. And, yes, immutability and pure functions make it simpler.
  • A strong philosophy behind the language, that seems to drive its design. Python has this too and to me it’s its biggest selling point. Languages like C++ and increasingly JavaScript, on the other hand, feel like magic bags where features are added carelessly without consideration of the results. Java does have a strong philosophy: programmers are mostly idiots.
  • Runs on the JVM. Seriously? I personally couldn’t care less about that but JVM languages seem to attract a lot of attention. There are tons of Java devs for sure and some of them seem to have a symbiotic relationship with the JVM: it’s like they aren’t cheating on Java if they keep the deal inside the VM. Why would people learn Groovy instead of Ruby or Python, god only knows, but because of the “Runs in Java” part there’s a better chance of finding a Clojure Job than one using Haskell, Scheme or most other functional languages around. That alone is enough reason for me to stick with Clojure instead of keep trying Lisp dialects, even if some other one may fit my taste better.
  • Haven’t tried it yet, but the mere existence of ClojureScript is good news to me, specially considering how annoyed I am with the direction the JavaScript syntax and ecosystem is taking lately. This talk totally sold it to me.
  • Said it before and say it again: forget about parenthesis. People seem to worry a lot about them beforehand, but as with Python whitespace indentation, once you’ve used it for five minutes it just goes away. Specially if you use the darn awesome Parinfer.
  • Which brings me to: you don’t need Emacs for Lisp programming. Yes, I hear you, once I master Emacs I’ll be a more powerful programmer. But I’m trying to learn a weird language here, don’t make me also learn a weird, counter-intuitive editor at the same time. That would just increase the chances of me dropping the effort altogether. There are decent ports of Paredit for Sublime and Atom, which is good enough. But with Parinfer you just learn one command and forget about it, it just works.
  • REPL driven development. Because of pure functions it’s easy to write a piece of code and test it right away in the REPL. Together with unit tests it pretty much removes the need for debugging.
  • Leiningen looks good, it covers the small needs I had starting out and didn’t get in the way. Clojurians say it does a lot more than that, so great. Much better than the 17 tools you need to set up to have a Node.js project running these days.

The bad

  • Namespace syntax is complicated, there are too many operations and keywords to do it (require, refer, use, alias, import and ns —which can do all of the others with a slightly different notation). It’s flexible but boilerplatish, even when sticking to ns:
(ns advenjure.game
  (:require [advenjure.rooms :as rooms]
            [advenjure.utils :as utils]
            [advenjure.verb-map :refer [find-verb verb-map]]))
  • And while there’s no hard rule to keep a one to one relation between files and namespaces, there’s a strong convention to do it, so having to declare the package name in every file seems totally redundant (and Java-ish, let’s be honest).
  • contains? Works in a counter-intuitive way for vectors.

That’s all I got.

And the ugly

Ok, there wasn’t much bad stuff, but there is some maybe not so good or arguably not good things I can think of.

  • The built in operator set doesn’t follow the Unix and Python philosophy of small core and a lot of libraries that I like so much: the functions are way too many to easily remember, and they aren’t entirely orthogonal (some of them do the same thing in slightly different ways). Then again, the Clojure Cheatsheet, the REPL and doc are more than enough to cope with that.
  • Polymorphism is great: sequence and collection functions work as expected in all data structures. The downside is that to do so the results are always coerced to seqs, which may be unexpected, specially for hash maps. In practice, though, I found myself just chaining those functions and rarely caring about the resulting type.
  • Macros are powerful and awesome but the quoting syntax can get very tricky. I definitely need more experience to learn to reason about macro code, but the syntax will remain ugly. I guess that’s the cost you pay for being able to fiddle with how the language processes the code. In the end (much like Python metaclasses), macros are a great tool to keep in the box, but to use sparingly. So far every time I thought about implementing one I got away fine by using closures instead.
  • Java does sneak in quite a bit and that’s a turn off. (spoiler alert: I don’t like Java). OK, Java interop is simple and powerful, probably the most straightforward language interfacing I’ve seen (boy was SWIG a nightmare). That being said, Java code inside Clojure looks like, well, Java code inside Clojure: it reeks. This wouldn’t be so much of a problem if needed only to interact with some third party Java libraries, but in practice I’ve found that there’s basic stuff lacking in the Clojure standard library and it’s either add a dependency or use Java interop. I saw this while solving an exercise from the Brave Clojure book: it asked to list the first results of a google search. The request should be a one liner using the built in slurp function but, wait, you need to set the User-Agent header to request google, so you end up with:
(with-open [inputstream (-> (java.net.URL. url)
                            .openConnection
                            (doto (.setRequestProperty "User-Agent"
                                                       "Mozilla/5.0 ..."))
                            .getContent)]

The end

Even though it’s not my ideal language and it may be less ideal to me than Python was, it looks like I’ll start to look for excuses to use Clojure as much as possible and it’ll be a while before I jump to study another new language.

First impressions on (Common) Lisp

Because it’s good to learn a new Language, because lispers seem convinced that it’s the most powerful language ever, and because it’s supposed to be some sort of programming religious experience[1], I decided to give Lisp a try.

Since I had sincere hopes of end up using it on professional projects, after looking around I leaned towards Common Lisp, and particularly the book Practical Common Lisp by Peter Seibel. After the first 9 chapters, I’m somewhat disappointed. I’m guessing this is partly because of my high expectations and partly because most of lisp’s “cool” features are getting less and less uncommon as languages advance[2]. These are my first impressions.

It’s not readable. Strangely enough I don’t mind all the parentheses nor the prefix notation; I get that they’re part of what makes Lisp so powerful and actually got used to it pretty quickly. Still, the language is overall pretty cryptic, even for things that could with no effort be a lot clearer (t/nil instead of true/false?). It might not be for experienced Lisp programmers, but, well, if it’s readable only for the experienced, then it’s not readable.

Some examples from the first chapters of the book:

(format t "~{~{~a:~10t~a~%~}~%~}" *db*)

and:

(defmacro where (&rest clauses)
  `#'(lambda (cd) (and ,@(make-comparisons-list clauses))))

Suddenly Perl isn’t looking that bad.

The operators set is huge. That’s not good news for me. I like languages that have a small, orthogonal, easy-to-remember set of built-in operators, and good libraries built on top of them. This is the case of C and Python, and not of Lisp, which seems to have loads of built-in operators, between special operators, functions and macros. In the first chapters of the book, every bit of code introduces a couple of new ones. This not only makes it hard to keep track of them, but without seeing a thorough reference, I have no idea on what to expect to be available as a built-in and what I’m supposed to write myself.

From what I read, that small core is there somewhere, but at least this books makes it look like it’s too low level to get anything done with it, and instead uses the macros built on top of it.

…and it’s not orthogonal. There seems to be way too many operators for the same purposes but with slightly different behaviors. The most evident case of this is the equality: =, CHAR=, EQ, EQL, EQUAL, and EQUALP. Seriously?

…and some names are really, really bad. MACROEXPAND-1, LET and LET* (a common pattern for slightly different operators), Y-OR-N-P, etc.

It’s not intuitive. As a consequence of the previous points, it gets really hard to guess my way through a piece of code; instead I have to look up most operators. Anytime a new operator was introduced in the book, it worked different from what I expected.

Macros look promising. They certainly do; after catching just a glimpse of what can be done with them, macros look like a powerful tool that one can get quickly used to, and will miss in other languages. I particularly enjoy the way they’re used in the book as the driving technique to constantly refactor code, built bottom-up.

I’m aware that most of this issues would go away after some time using the language, but taking in account the downside of being not so widely used as other languages (meaning less documentation, less libraries, less users, etc.) so far it’s not looking like one I’d use for more than one or two projects. It might make sense for me to check out some other Lisp dialects, like Scheme or Clojure.

[1] Eric Raymond, How To Become A Hacker.
[2] Paul Graham, Revenge Of The Nerds.