Taylor ≈ Programmer

Using Cognitect's aws-api with AWS Transcribe

Sun, 24 Feb 2019 00:00:00 +0000

Using Cognitect’s aws-api library to work with AWS is nice and easy (and simple). This post shows how to use it with AWS Transcribe for speech-to-text.

aws-api is an idiomatic, data-oriented Clojure library for invoking AWS APIs.

Comparisons

Previously I’d used Amazonica for working with general AWS services, or more focused libraries like Faraday for working with DynamoDB. Amazonica takes a conceptually similar approach to aws-api in that its code is (mostly) generated by reflecting on AWS-published API specs. Faraday wraps Dynamo’s API in an idiomatic Clojure API, with some other niceties e.g. serializing Clojure data to/from Dynamo JSON.

Compared to these projects, aws-api’s approach has some key similarities and differences:

The majority of Amazonica and aws-api’s API surface is derived from AWS API definitions.
Amazonica and Faraday depend directly on AWS SDK libraries, but aws-api takes a “clean room” approach by calling AWS API endpoints directly — no AWS SDK dependencies.
Faraday provides a very Clojure-y wrapper for AWS Dynamo SDK. Amazonica provides generated (and bespoke) wrappers for AWS APIs that look/feel like idiomatic Clojure. aws-api takes a very data-centric approach, providing one function to invoke any AWS API operation.

I like that aws-api requires no transitive dependencies on AWS SDK. (No more transitive Jackson conflicts!) It has relatively few dependencies overall.

It’s extremely data-centric and doesn’t attempt to obscure the nature of data in AWS API requests/responses behind a Clojure-y façade. Considering it as a fairly mechanical translation of AWS API specs, it’s very much data-as-code.

The :TitleCase keywords were a little jarring at first but I came to appreciate only needing to reference the AWS API docs, as opposed to understanding both the AWS API and a bespoke Clojure wrapper. For example, I’d often need to reference Faraday’s source to see how its custom data shapes mapped to their Dynamo request/response counterparts, because they were often different for the sake of idiomacy (is this a word 🤷‍♂️).

Because it’s so data-centric, it’s trivial to enumerate all types of operations you can perform against each AWS service, along with their inputs and outputs. This makes it a very REPL-friendly tool.

API Interactions

First, we need to authenticate ourselves. The default provider covers most use cases:

(require '[cognitect.aws.credentials :as creds])
(def credentials
  (creds/default-credentials-provider))

If you’ve used AWS API you’re probably familiar with the concept of the default credentials provider chain, and maybe (or unfortunately) its order of precedence. Something I found very useful was the ability to create a custom credential provider chain (which I’d done with much less elegance before aws-api). Here I defined a single, chained provider that works locally and when deployed to EC2 or ECS:

(def credentials
  (creds/chain-credentials-provider
    [(creds/instance-profile-credentials-provider)
     (creds/container-credentials-provider)
     (creds/profile-credentials-provider "my-custom-profile")]))

The next thing we’ll need is a client to our AWS API, in this case Transcribe:

(require '[cognitect.aws.client.api :as aws])
(def transcribe
  (aws/client {:api :transcribe
               :region :us-east-1
               :credentials-provider credentials}))

Because aws-api is so data-centric, there’s only one function for interacting with the API: invoke. Everything else is data. If you want to see all operations for an API, you can use the ops function.

(aws/ops transcribe)
=>
{...
 :DeleteVocabulary {:name "DeleteVocabulary",
                    :documentation "<p>Deletes a vocabulary from Amazon Transcribe. </p>",
                    :request {:VocabularyName string},
                    :required [:VocabularyName],
                    :response nil}
 ...}

You can then pick/filter those results and get detailed documentation for each with the doc function. This tells us what the operation does, what inputs it requires, and what we can expect in the response. We can print docs for each of these operations in the REPL:

(aws/doc transcribe :StartTranscriptionJob)
-------------------------
StartTranscriptionJob

<p>Starts an asynchronous job to transcribe speech to text. </p>

-------------------------
Request

{:TranscriptionJobName string,
 :LanguageCode string,
 :MediaSampleRateHertz integer,
 :MediaFormat string,
 :Media {:MediaFileUri string},
 :OutputBucketName string,
 :Settings
 {:VocabularyName string,
  :ShowSpeakerLabels boolean,
  :MaxSpeakerLabels integer,
  :ChannelIdentification boolean}}

Required

[:TranscriptionJobName :LanguageCode :MediaFormat :Media]

-------------------------
Response

{:TranscriptionJob
 {:MediaFormat string,
  :CompletionTime timestamp,
  :MediaSampleRateHertz integer,
  :TranscriptionJobName string,
  :Settings
  {:VocabularyName string,
   :ShowSpeakerLabels boolean,
   :MaxSpeakerLabels integer,
   :ChannelIdentification boolean},
  :CreationTime timestamp,
  :Transcript {:TranscriptFileUri string},
  :TranscriptionJobStatus string,
  :FailureReason string,
  :LanguageCode string,
  :Media {:MediaFileUri string}}}

Invocation is easy, because the :op and :request values are easily interpretable from the output of the ops or doc functions:

(aws/invoke
  transcribe                        ;; the client
  {:op :StartTranscriptionJob       ;; the operation
   :request                         ;; the operation input
   {:TranscriptionJobName job-name
    :LanguageCode         "en-US"
    :MediaFormat          "wav"
    :Media                {:MediaFileUri s3-path}}})

Doing Things

I needed to transcribe some speech to text, and of course AWS has a service for that called Transcribe. It’s not very fast but it’s cheap and fairly easy to use. You can put your audio in a S3 bucket and request a transcription. Unfortunately you have to poll (or use SNS + Lambda?) for the resulting transcription, which will also go into a S3 bucket.

We can create a transcription job:

(defn start-transcribe [s3-path job-name]
  (aws/invoke
    transcribe
    {:op :StartTranscriptionJob
     :request
     {:TranscriptionJobName job-name
      :LanguageCode         "en-US"
      :MediaFormat          "wav"
      :Media                {:MediaFileUri s3-path}
      ;; NOTE output bucket is optional
      ;; AWS otherwise puts the transcript in a random bucket
      :OutputBucketName     "my-unique-bucket"}}))

(start-transcribe
 "https://s3.us-east-1.amazonaws.com/my-unique-bucket/sample.wav"
 "my-random-job-name")
=>
{:TranscriptionJob {:MediaFormat "wav",
                    :TranscriptionJobName "my-random-job-name",
                    :CreationTime #inst"2019-02-25T12:51:45.000-00:00",
                    :TranscriptionJobStatus "IN_PROGRESS",
                    :LanguageCode "en-US",
                    :Media {:MediaFileUri "https://s3.us-east-1.amazonaws.com/my-unique-bucket/sample.wav"}}}

But this only tells us the job is in progress, and we must wait for it to complete. I define another simple helper function to get the details of a job:

(defn get-transcribe-job [job-name]
  (aws/invoke transcribe
              {:op      :GetTranscriptionJob
               :request {:TranscriptionJobName job-name}}))

(get-transcribe-job "my-random-job-name")
=>
{:TranscriptionJob {:MediaFormat "wav",
                    :MediaSampleRateHertz 8000,
                    :TranscriptionJobName "my-random-job-name",
                    :Settings {:ChannelIdentification false},
                    :CreationTime #inst"2019-02-25T13:08:06.000-00:00",
                    :Transcript {},
                    :TranscriptionJobStatus "IN_PROGRESS",
                    :LanguageCode "en-US",
                    :Media {:MediaFileUri "https://s3.us-east-1.amazonaws.com/my-unique-bucket/sample.wav"}}}

We can see it’s still in progress, but when it’s finished it will contain a path to the transcript JSON in S3. I use a simple function to read the JSON from S3:

(defn read-file [bucket key]
  (-> (aws/invoke s3 {:op :GetObject, :request {:Bucket bucket :Key key}})
      (:Body)
      (slurp)))

And finally, a convenience function to poll for the results of a given job:

(defn await-transcription [job-name]
  (loop [job (get-transcribe-job job-name)]
    (case (get-in job [:TranscriptionJob :TranscriptionJobStatus])
      "IN_PROGRESS"
      (do (Thread/sleep 5000) ;; I have no shame
          (recur (get-transcribe-job job-name)))

      "COMPLETED"
      (-> (read-file "my-unique-bucket" (str job-name ".json"))
          (json/read-str) ;; clojure.data.json
          (get-in ["results" "transcripts" 0 "transcript"]))

      (throw (ex-info "Transcribe failed" job)))))

(await-transcription "my-random-job-name") ;; spins for ~1 minute
=> "Bravo, November Papa Romeo, Oscar."    ;; sample.wav contained an audio captcha

I was impressed with how easy it was to interact with an AWS API I’d never used before via aws-api. Everything I needed was right there in the REPL with a handful of functions: client, ops, and invoke. The immediacy and transparency of working with AWS via aws-api makes for a great Clojure experience.

GraalVM Polyglot with Clojure and JavaScript

Mon, 26 Nov 2018 00:00:00 +0000

GraalVM enables interesting new interop scenarios between its hosted languages. This post demonstrates some polyglot interop between Clojure as a host language and JavaScript as the hosted language.

You’ll need to run this Clojure code on a GraalVM SDK, and we’ll want to import some org.graalvm.polyglot types:

(ns polydact.core
  (:import (org.graalvm.polyglot Context Value)
           (org.graalvm.polyglot.proxy ProxyArray ProxyExecutable ProxyObject)))

Now instantiate a polyglot GraalVM Context for the JavaScript language, and a helper function for convenient evaluation:

(def context (.build (Context/newBuilder (into-array ["js"]))))
(defn eval-js [code]
  (.eval ^Context context "js" code))

The result of eval-js will be some GraalVM polyglot Value, which has an .as method for casting values to particular types. That’s all you need to start evaluating JavaScript from within Clojure:

(.as (eval-js "Number.MAX_VALUE") Object)
=> 1.7976931348623157E308
(type *1)
=> java.lang.Double

That’s a trivial scalar example, but how are more complex data structures translated between language contexts? This is where GraalVM’s polyglot API comes into play, and there are multiple possible behaviors. Consider a JavaScript array with an empty map in it:

(.as (eval-js "[{}]") Object)
=> {"0" {}} ;; an array becomes a map with indices as keys
(.as (eval-js "[{}]") java.util.List)
=> ({}) ;; a polyglot list with polyglot map inside

If we invoke the .as(Object.class) overload, Graal uses a set of rules (see .as docstring) to infer the desired result type, but it may not return what you want in some cases. If we invoke .as(List.class) we get a different (and perhaps more familiar) translation, but still not something we can easily work with in Clojure directly.

Idiomatic Coercion

We can write some Clojure to help marshal a polyglot Value to a sensible Clojure (or Java) value:

(defn value->clj
  "Returns a Clojure (or Java) value for given polyglot Value if possible,
   otherwise throws."
  [^Value v]
  (cond
    (.isNull v) nil
    (.isHostObject v) (.asHostObject v)
    (.isBoolean v) (.asBoolean v)
    (.isString v) (.asString v)
    (.isNumber v) (.as v Number)
    (.canExecute v) (reify-ifn v)
    (.hasArrayElements v) (into []
                                (for [i (range (.getArraySize v))]
                                  (value->clj (.getArrayElement v i))))
    (.hasMembers v) (into {}
                          (for [k (.getMemberKeys v)]
                            [k (value->clj (.getMember v k))]))
    :else (throw (Exception. "Unsupported value"))))

A couple things to note: 1) This is not a complete or well-tested implementation, and 2) it handles all my little toy examples for this article.

Sparing some details, the Value class is a container/boxed value for some polyglot value (or a pointer), and it has methods for inspecting what type of value it contains. Most of the cond clauses should be fairly self-explanatory. The most interesting ones have to do with .canExecute executable values, .hasArrayElements arrays, and .hasMembers associative structures.

Executables

The definition of reify-ifn is a macro that reifies IFn for the given Value. This is only a macro so that I can easily emit all the arities of invoke:

(defn- execute [^Value execable & args] ;; just a little sugar
  (.execute execable (object-array args)))

(defmacro ^:private reify-ifn
  "Convenience macro for reifying IFn for executable polyglot Values."
  [v]
  (let [invoke-arity
        (fn [n]
          (let [args (map #(symbol (str "arg" (inc %))) (range n))]
            (if (seq args)
              `(~'invoke [this# ~@args] (value->clj (execute ~v ~@args)))
              `(~'invoke [this#] (value->clj (execute ~v))))))]
    `(reify IFn
       ~@(map invoke-arity (range 22))
       (~'applyTo [this# args#] (value->clj (apply execute ~v args#))))))

A couple things to note: 1) This is not a complete or well-tested implementation, and 2) it handles all my little toy examples for this article.

Now we can treat executable polyglot values like typical Clojure functions!

Arrays

The .hasArrayElements case is pretty plain: convert it to a vector and recursively convert any contained values.

Associatives

The .hasMembers case is not so clear-cut. There are several different interpretations of what a member can be for a polyglot value. For example, for a Java object each member would be a field or class member. For a JSON object, each member would be a string key.

This example uses the simplest interpretation and treats everything that has members as a Clojure map. To paraphrase someone else: now it’s data, you’ve already won!

Polyglottin’

Equipped with just value->clj and a couple helpers, let’s play the ployglottery…

(def js->clj (comp value->clj eval-js)) ;; fn composition convenience

(js->clj "[{}]")
=> [{}]

(js->clj "false")
=> false

(js->clj "3 / 3.33")
=> 0.9009009009009009

(js->clj "123123123123123123123123123123123")
=> 1.2312312312312312E32

(js->clj "m = {foo: 1, bar: '2', baz: {0: false}};")
=> {"foo" 1, "bar" "2", "baz" {"0" false}}

(js->clj "1 + '1'")
=> "11" ;; a classic

(js->clj "!!\"false\" == !!\"true\"")
=> true ;; we've proven false!

(js->clj "['foo', 10, 2].sort()")
=> [10 2 "foo"] ;; JS sort so funny!

Nothing surprising there, just that we’re evaluating some JavaScript from a Clojure REPL and getting back Clojure values.

Note that we’re sharing a single Context for these evaluations, and contexts are stateful and mutable:

(eval-js "var foo = 0xFFFF")
(eval-js "console.log(foo);")
=> #object[org.graalvm.polyglot.Value 0x3f9d2028 "undefined"]
;65535

Functional Programming

What if you could write your functions in JavaScript, which I hear is getting better all the time, and call them from Clojure? Welcome to the next level!

(def doubler (js->clj "(n) => { return n * 2; }"))
(doubler 2)
=> 4 ;; checks out!

How about a beautiful, memoizing factorial function that returns a JSON object containing the factorial function and the backing memo array:

(def factorial
  (eval-js "
    var m = [];
    function factorial (n) {
      if (n == 0 || n == 1) return 1;
      if (m[n] > 0) return m[n];
      return m[n] = factorial(n - 1) * n;
    }
    x = {fn: factorial, memos: m};"))

((get (value->clj factorial) "fn") 12)
=> 479001600 ;; JS happens to be right here
(get (value->clj factorial) "memos")
=> [nil nil 2 6 24 120 720 5040 40320 362880 3628800 39916800 479001600]

((get (value->clj factorial) "fn") 24)
=> 6.204484017332394E23
(get (value->clj factorial) "memos")
;; the result of this may surprise you! (it now contains many more values)

We can use ProxyArray to pass collections to JavaScript functions so they can be treated as native JavaScript arrays.

(def js-aset
  (js->clj "(arr, idx, val) => { arr[idx] = val; return arr; }"))
(js-aset (ProxyArray/fromArray (object-array [1 2 3])) 1 nil)
=> [1 nil 3] ;; and we get a mutated vector back

JavaScript sports exotic, modern features like variadic functions, and we can treat them just like their Clojure counterparts. We can even see Clojure-specific values like keywords (boxed in polyglot values) passed through the foreign JavaScript function unharmed.

(def variadic-fn
  (js->clj "(x, y, ...z) => { return [x, y, z]; }"))
(apply variadic-fn :foo :bar (range 3))
=> [:foo :bar [0 1 2]]

Have you ever tried to sort a wildly heterogenous collection in Clojure?

(sort [{:b nil} \a 1 "a" "A" #{\a} :foo  -1 0 {:a nil} "bar"])
CompilerException java.lang.ClassCastException:
clojure.lang.PersistentArrayMap cannot be cast to java.lang.Character

I thought this was a dynamic language! When you just need to get the job done, JavaScript will oblige:

(def js-sort
  (js->clj "(...vs) => { return vs.sort(); }"))
(apply js-sort [{:b nil} \a 1 "a" "A" #{\a} :foo  -1 0 {:a nil} "bar"])
=> [-1 0 1 "A" #{\a} :foo {:a nil} {:b nil} "a" "a" "bar"]

It’s easy to miss in the above example, but notice again the keyword :foo and a set in the input collection we passed to the JavaScript .sort() function, and the sorted output contains said keyword and set sorted by some likely inscrutable JavaScript sorting logic. It “just works” despite the fact JavaScript has no concept of keywords.

Use the JSON serde functions that started it all:

(def ->json
  (js->clj "(x) => { return JSON.stringify(x); }"))
(->json [1 2 3])
=> "[1,2,3]"

;; Proxy hash map to be treated as a JSON object in this case
(->json (ProxyObject/fromMap {"foo" 1, "bar" nil}))
=> "{\"foo\":1,\"bar\":null}"

(def json->
  (js->clj "(x) => { return JSON.parse(x); }"))
;; take some round trips
(json-> (->json [1 2 3]))
=> [1 2 3]
(json-> (->json (ProxyObject/fromMap {"foo" 1})))
=> {"foo" 1}

;; access JSON object members naturally
(def json-object
  (js->clj "(m) => { return m.foo + m.foo; }"))
(json-object (ProxyObject/fromMap {"foo" 1}))
=> 2

Clojurvascriptception

We’ve made JavaScript functions double as Clojure functions, now let’s do the opposite. This function provides an instance of GraalVM’s ProxyExecutable that implements its sole .execute method for a given Clojure function (or really anything apply works on):

(defn proxy-fn
  "Returns a ProxyExecutable instance for given function, allowing it to be
   invoked from polyglot contexts."
  [f]
  (reify ProxyExecutable
    (execute [_this args]
      (apply f (map value->clj args)))))

That’s all we need to be able to pass executable code from Clojure into our polyglot context.

In this example we return a JavaScript function that closes over a nested map and takes another function to invoke with the map — maybe the term callback would be more apt in JavaScript parlance.

(def clj-lambda
  (js->clj "
  m = {foo: [1, 2, 3],
       bar: {
         baz: ['a', 'z']
       }};
  (fn) => { return fn(m); }
  "))

(clj-lambda
 (proxy-fn #(clojure.walk/prewalk
             (fn [v] (if (and (vector? v)
                              (not (map-entry? v)))
                       (vec (reverse v))
                       v))
             %)))
=> {"foo" [3 2 1], "bar" {"baz" ["z" "a"]}}

That’s Clojure using GraalVM to host and evaluate JavaScript, passing a Clojure function to a JavaScript lambda that invokes the Clojure function, which uses prewalk to reverse all the arrays in the nested JSON object, then getting back an idiomatic Clojure map value.

We can lean on JavaScript’s .reduce() and use it just like Clojure’s reduce with a lazy sequence:

(def js-reduce
  (let [reduce (js->clj "(f, coll) => { return coll.reduce(f); }")
        reduce-init (js->clj "(f, coll, init) => { return coll.reduce(f, init); }")]
    (fn
      ([f coll] (reduce f coll))
      ([f init coll] (reduce-init f coll init)))))
(js-reduce + (range 10))
=> 45
(js-reduce + -5.5 (range 10)
=> 39.5

We can pass in a Clojure reducing function, that internally uses the JavaScript doubler function defined above, and build up a Clojure map within JavaScript’s .reduce():

(js-reduce (fn [acc elem]
             (assoc acc (keyword (str elem)) (doubler elem)))
           {}
           (range 5))
=> {:0 0, :1 2, :2 4, :3 6, :4 8}

Some bad news about passing Clojure’s lazy sequences to JavaScript: they’ll be realized before being evaluated in JavaScript. Here we can see the Clojure prn statements all execute before the JavaScript logs the first value:

(def log-coll
  (js->clj "(coll) => { for (i in coll) console.log(coll[i]); }"))
(log-coll (repeatedly 3 #(do (prn 'sleeping)
                             (Thread/sleep 100)
                             (rand)))
sleeping
sleeping
sleeping
0.8793736393816931
0.9610193480723516
0.05981091977823816
=> nil

(log-coll (range)) ;; infinite seq will never complete

Loading External JavaScript

This example loads Underscore.js from GitHub and calls its .invert() function on a Clojure map:

(def invert
  (let [_invert
        (js->clj "
          load('https://raw.githubusercontent.com/jashkenas/underscore/master/underscore.js');
          (m) => { return _.invert(m); };")]
    (fn [m] (_invert (ProxyObject/fromMap m)))))

(invert {"foo" "bar", "1" "2"})
=> {"2" "1", "bar" "foo"}

This is similar to clojure.set/map-invert but in JavaScript all the keys must be strings.

Closing Remarks

I have no practical use for any of this at the moment, but I find this whole mixed-language runtime experiment very exciting!

While these examples only used JavaScript, the same things can be done with several other languages on GraalVM, like Ruby, Python, R, LLVM-based languages, etc. See this page for more information.

Here’s a gist with all the code from this experiment.

Clojure.spec Beginner's FAQ

Mon, 15 Oct 2018 00:00:00 +0000

Clojure.spec has been available (in alpha) for some time, and there are great talks and resources like the rationale and official guide. This post supplements those with some common questions and issues I’ve seen.

These are mostly things I’ve experienced myself or seen on Stack Overflow, Clojurians Slack, r/Clojure or Twitter. The community is extremely helpful so reach out if you have questions — it’s not unusual to get an answer straight from a Clojure maintainer. And while this post follows question & answer format, my answers are extremely non-authoritative. Please report any issues you find!

Usage

Q: How should I integrate clojure.spec into my project?

A: However you like!

Clojure.spec usage is totally à la carte. You can use only the features you want to whatever extent you want. This is great for consumers, but a lack of prescribed patterns doesn’t help newcomers wondering how to “best” leverage spec in their work.

Here are some things I’ve used spec for:

Validating inputs to service endpoints, and using spec metadata to return helpful details for invalid inputs.
Generating sample inputs to service endpoints for API documentation.
Writing specs for important functions, and instrumenting them during development/testing as a kind of run-time type checker.
Using specs to get corresponding test.check generators “for free”, using the generated data for other purposes.
Using a s/or spec as a kind of discriminated union to dispatch based on different types of inputs.

Organization

Q: Should I put specs in the same namespace as my code, or in a separate namespace?

A: It’s up to you.

You may find having specs alongside relevant code is helpful. You can just as easily put specs for data in separate namespaces, and specs for functions next to their defn counterparts — and you can put the spec before or after the defn.

One consideration is whether you’re using qualified keywords in s/keys map specs. I think a potential benefit of having data specs in the same namespace as your regular code is that it’s natural to inherit that namespace for namespaced map specs:

(in-ns 'customer)
(s/def ::id int?)
(s/def ::name string?)
(s/def ::contact (s/keys :req [::id ::name]))
(defn print [{::keys [id name]}] (prn id name))
(print #:customer{:id 1 :name "Jane" :some.other/id "FhI-1"})

Q: Can I document specs with docstrings?

A: No, but it’s being considered for a future release.

Instrumentation & Testing

instrument can be used to assert arguments to spec’d function invocations. See my post on function specs for more detailed examples. The 0-arity instrument will instrument every loaded, instrumentable function. There’s a cost associated with instrumentation, and you may want instrument to only affect particular functions:

(clojure.spec.test.alpha/instrument [`foo `bar])

And there’s an unstrument function for removing instrumentation.

Q: When and where should I call instrument?

A: It depends.

Your instrumentable functions must be loaded before you can instrument them. If you have per-profile entry points to your program e.g. -main, -dev-main, you might choose to call instrument in the dev entry point. If your project uses Leiningen, you could use :injections to call instrument on a per-profile basis:

:injections [(require 'lib.core) ;; all instrumented fns should be loaded here
             (require 'clojure.spec.test.alpha)
             (clojure.spec.test.alpha/instrument)]

Or you may only want to instrument functions during test runs, maybe as a clojure.test fixture or in the namespace itself.

Q: Should I use instrument in production builds?

A: Probably not.

From the guide:

Instrumentation is likely to be useful at both development time and during testing to discover errors in calling code. It is not recommended to use instrumentation in production due to the overhead involved with checking args specs.

Q: Why doesn’t instrument check my function return value?

A: It’s not meant to.

This is a common point of confusion. If you instrument a function spec’d with fdef or fspec, only its :args spec will be checked during each invocation. Why is that? I think one rationale might be that functional programs are generally a composition of functions where each output becomes an input to another function. If you have spec’d the :args and :ret of related functions (f (g x)), you’d be redundantly checking the same specs for the :ret of g and :args of f. For larger examples this could become very costly.

Q: So why do fdef/fspec accept :ret and :fn specs too?

A: They’re used to check the function.

checking a function involves generating many random inputs (from the :args spec) for the function, invoking the function with each input, and checking that the return value conforms to the :ret spec, and if :fn spec is defined that the relationship between the input and output values are satisfied. This is commonly called generative or property-based testing and spec relies on test.check for this.

You can use check as part of a test suite, for example with clojure.test:

(deftest foo-test
  (is (= 1 (-> (st/check `foo)
               (st/summarize-results)
               :check-passed))))

Sequences & Regex Specs

Q: What’s this odd nesting behavior with regex specs?

A: Nested regex specs compose to describe a single sequence.

Or as stated in the rationale:

These nest arbitrarily to form complex expressions.

I think this is much easier to understand through examples, which can be found in the official guide.

It’s easy to forget the s/& spec which is useful for adding additional predicates to regex specs, where using s/and would destroy the regex nesting behavior. Consider these two conforming examples where the only difference is s/& vs. s/and:

(s/conform
 (s/+ (s/alt :n number?
             :s (s/and (s/+ string?)
                       #(every? (complement empty?) %))))
 [1 ["x" "a"] 2 ["y"] ["z"]])

(s/conform
 (s/+ (s/alt :n number?
             :s (s/& (s/+ string?)
                     #(every? (complement empty?) %))))
 [1 "x" "a" 2 "y" "z"])

Q: How can I escape the regex spec nesting behavior?

A: Wrap the regex spec with s/spec.

This is also described in the official guide above, but here’s another example:

(s/conform
 (s/+ (s/alt :n number? :s (s/spec (s/* string?))))
 [1 2 3 ["x" "y" "z"]])
=> [[:n 1] [:n 2] [:n 3] [:s ["x" "y" "z"]]]

But this example might be better specified with s/coll-of or s/every:

(s/+ (s/alt :n number? :s (s/coll-of string?)))
(s/+ (s/alt :n number? :s (s/every string?)))

Map Specs

Q: How can I assign different specs to the same key in different maps?

A: That’s not supported (for qualified keys).

Spec encourages use of qualified map keys. If you control the shape of your map you should consider using qualified keys:

{:customer/name "Frieda" :company/name "Acme"}

Of course you may not have this luxury when working with external data sources where the same key may have different meanings at different paths:

{:name "Frieda" :company {:name "Acme"}}

In this case there’s a workaround if you’re using unqualified keys in your map specs. You can use qualified keys to name your specs, but create s/keys specs with unqualified keys e.g. :req-un and :opt-un:

(s/def :customer/name string?)
(s/def :company/name (s/nilable string?))
(s/def ::company (s/keys :req-un [:company/name]))
(s/def ::customer (s/keys :req-un [:customer/name ::company]))
(s/valid? ::customer {:name "Taylor" :company {:name nil}}) => true

Q: How can I make a strict/closed s/keys spec that disallows extra keys?

A: You should only do this when necessary.

In spec, map specs are meant to be open rather than closed. Adding data to a valid map should not invalidate the map.

There are cases where you might really want this, e.g. you may want to restrict what goes into your document storage or reject API requests containing extra data. While spec doesn’t provide this behavior by default, it does nothing to prevent it, and there are easy options at hand like this or this.

Q: Can I specify keys as conditionally required?

A: Yes, use and and or inside s/keys :req and :req-un specs.

or forms (and potentially nested ands) inside :req and :req-un values are interpreted to form logical disjunctions/conjuctions of required keys. (Note: a top-level and wouldn’t make much sense in the context of required keys.)

From s/keys docs:

The :req key vector supports ‘and’ and ‘or’ for key groups:

(s/keys :req [::x ::y (or ::secret (and ::user ::pwd))] :opt [::z])

To illustrate this also works for :req-un keys:

(s/valid?
 (s/keys :req-un [(or ::a ::b) (or ::c ::d)])
 {:b 0 :c 1})
=> true

Generation

Q: Why does generation fail with Couldn’t satisfy such-that predicate after 100 tries?

A: It might be the default generator for a s/and spec.

Generators are created from specs automagically, but for s/and specs the generator is based solely on the first spec inside the s/and. This spec conforms string palindromes:

(s/def ::palindrome
  (s/and string? #(= (clojure.string/reverse %) %)))

But you may get an error message when trying to exercise its generator (or possibly indirectly if the spec is involved in a check‘d function):

(gen/sample (s/gen ::palindrome))
ExceptionInfo Couldn't satisfy such-that predicate after 100 tries.  clojure.core/ex-info (core.clj:4739)

This is because the generator for ::palindrome is actually a generator for (s/gen string?), which will just generate random strings that are very unlikely to be palindromes, and after 100 random tries it gives up. In these cases you can provide your own generator with s/with-gen:

(s/def ::palindrome
  (s/with-gen
   (s/and string? #(= (clojure.string/reverse %) %))
   ;; use fmap to create palindromes of the generated strings
   #(gen/fmap (fn [s] (apply str s (rest (reverse s))))
              (s/gen string?))))
(gen/sample (s/gen ::palindrome))
=> ("" "" "e" "PA6AP" "OUdTdUO" "k" "N" "0" "1T353T1" "D4V4D")

Some spec functions also take an optional overrides map of custom generators without needing to associate them with the spec definitions directly.

Q: How can I generate data with parent/child relationships?

A: Use test.check functions like fmap, bind, or let macro.

As in the previous example you can use fmap to create a new generator that alters the results of a generator, so for recursive structures you can make any modifications there. For more complex scenarios — perhaps involving multiple generators — test.check provides a let macro that’s syntactically similar to clojure.core let. (Note that clojure.spec.gen.alpha only aliases some of test.check’s functionality; you’ll need to require test.check namespaces directly for its let macro.) The RHS of the bindings are generators and the LHS names are bound to the generated values, and it all expands to fmap and bind calls.

Consider an example of a recursive map spec:

(s/def ::id uuid?)
(s/def ::children (s/* ::my-map))
(s/def ::parent-id ::id)
(s/def ::my-map
  (s/keys :req-un [::id]
          :opt-un [::parent-id ::children]))
(gen/sample (s/gen ::my-map))

We need a custom generator to ensure the :parent-id for child maps is accurate, using fmap again:

(defn set-parent-ids [m]
  (clojure.walk/prewalk
   (fn [v]
     (if (map? v)
       (update v
         :children #(map (fn [c] (assoc c :parent-id (:id v))) %))
       v))
   m))
(gen/sample
  (gen/fmap set-parent-ids (s/gen ::my-map)))

Also see test.check’s recursive-gen for another approach.

Another example might be non-recursive relationships between values, like arguments to a function. Consider writing a function spec for clojure.core/subs:

(s/def ::subs-args (s/cat :str string? :i int?))
(s/fdef clojure.core/subs :args ::subs-args)
(s/exercise-fn `subs)
StringIndexOutOfBoundsException String index out of range: -1  java.lang.String.substring (String.java:1927)

This is because the generators for the strings and integer indices are independent and unrelated:

(gen/sample (s/gen (s/cat :str string? :i int?)))
=> (("" 0) ("" 0) ("" -1) ("C" 3) ("c9gD" 1) ("" 6) ("cfo2s7" -7) ("3fRj30" -2) ("W" 0) ("cEzS" -15))

If we want to ensure that the index argument refers to a position in the string input, we can use a custom generator:

(s/def ::subs-args
  (s/with-gen
   (s/cat :str string? :i int?)
   #(gen/let [str (s/gen string?)
              index (gen/large-integer* {:min 0 :max (count str)})]
      [str index])))
(st/summarize-results (st/check `subs))
=> {:total 1, :check-passed 1}

Q: Why is my recursive/sequential spec slow to generate or check?

A: The recursion may be too deep, or sequence spec generators may be unbounded.

Spec defines a dynamic binding *recursion-limit* that puts a “soft” limit on recursive generation depth. You may want to decrease this limit in cases where generated structures are unwieldy.

The other type of sizing/growth to be concerned about relates to sequences generated by specs like s/every, s/coll-of. The :gen-max option will limit the length of generated sequences:

(gen/sample (s/gen (s/coll-of string? :gen-max 2)))
=> (["" ""] [] ["1D" ""] ["I"] [] ["Ne4" "y6i"] ["93" "oe"] ["4wUue7"] [] [])

Specs as Data

The current version of spec doesn’t make it very easy to create or inspect specs programmatically. This is reportedly being worked on for a future release.

Q: How can I get the keys from a s/keys map spec?

A: Use s/form or s/describe on the spec.

s/form (and its abbreviated sibling s/describe) will return the original form of the spec:

(s/def ::my-map
  (s/keys :req-un [::id]
          :opt-un [::parent-id ::children]))
(->> (s/describe ::my-map) ;; get abbrev. form of original spec
     (rest)                ;; drop `keys` symbol
     (apply hash-map))     ;; put kwargs into map
=> {:req-un [:sandbox/id]
    :opt-un [:sandbox/parent-id :sandbox/children]}

Q: How can I get the :args, :ret, or :fn specs from a function spec?

A: Same as the previous answer, for any type of spec.

And if for some reason you also needed to recreate one of those specs, you could use eval:

(s/fdef foo :args (s/tuple number?) :ret number?)
(eval (->> (s/form `foo) ;; get non-abbrev. form
           (rest)
           (apply hash-map)
           :args))  ;; get :args spec
(gen/sample (s/gen *1))
=> ([2.0] [-1.5] [0] [-2] [-1.5] [-1.5] [14] [1] [##-Inf] [-5.875])

Off-label Usage

Q: How can I use spec to parse strings?

A: Spec is not intended for string parsing.

Specs can be used as parsers in that specs can describe a syntax, and conforming valid inputs produces tagged outputs that can be treated as a syntax tree, so… why not use that on strings of characters? You’ll have an easier time using a purpose-built parser for string inputs. Spec’s regex specs are also more limited than typical regular expression libraries.

Q: I can use s/conformer to transform data with specs. Is that a good idea?

A: Not really.

Cognitect’s Alex Miller says:

spec is not designed to provide arbitrary data transformation (Clojure’s core library can be used for that). It is possible to achieve this using s/conformer but it is not recommended to use that feature for arbitrary transformations like this (it is better suited to things like building custom specs).

There’s also a JIRA with discussion on the topic. The official advice is to do this separately from spec, and there’s a library to assist with that.

I think some simple coercion isn’t terrible in limited, internal use cases e.g. conforming date strings to date values at your API boundary.

Syntax Synonyms

Tue, 09 Oct 2018 12:00:00 +0000

I had an itch to implement some basic CSS selectors for Hickory, a Clojure HTML representation. Having done something similar in F# for XPath and HTML I thought the same approach would be a good starting point.

My first step was to build a parser for the CSS selector syntax I was interested in. After some entertaining trial and error in the REPL I had a Minimum Viable Parser using Instaparse.

Then I started thinking about how I could translate its syntax tree into an evaluator against a document tree. Hickory has a concept of selectors which very conveniently share similar semantics with CSS selectors.

What surprised me most was how easily I could leverage the similarities between the Instaparse syntax tree and Hickory’s selector combinators (in the syntax->selector map below) to turn the syntax tree into a functional evaluator. I’d expected much more of a struggle to get something working, but the bulk of the code ended up being declarative grammar and plain syntax transformation:

And using it is easy too:

(->> (slurp "https://clojure.org")
     (h/parse)
     (h/as-hickory)
     (s/select (parse-css-selector "a[href~=reference]")))
=>
[{:type :element,
  :attrs
  {:href "/reference/documentation", :class "w-nav-link clj-nav-link"},
  :tag :a,
  :content ["Reference‍"]}]

Nice

I thought this was a good example of how two well-factored Clojure libraries and a little combinatory glue can accomplish a relatively complex goal concisely.

See the repo and tests for more examples.

GraalVM Native Image HTTPS Support

Thu, 04 Oct 2018 12:00:00 +0000

GraalVM 1.0.0-RC7 adds HTTPS as a supported protocol, and this is a brief walkthrough for using it in a Clojure project with GraalVM Community Edition for macOS. See this example project for a demo.

How To

Enable HTTPS protocol support with native-image options: --enable-https or --enable-url-protocols=https
Configure path to libsunec (Sun Elliptic Curve crypto)

This shared object comes with the GraalVM distribution and can be found at $GRAALVM_HOME/jre/lib/libsunec.[so|dylib]. GraalVM uses System.loadLibrary to load it at run-time whenever it’s first used. The file must either be in the current working directory, or in a path specified in Java system property java.library.path.

I set the Java system property inside my application at run-time before first usage:
```
(System/setProperty "java.library.path"
                    (str (System/getenv "GRAALVM_HOME") "/jre/lib"))
```
See this and this for more information on HTTPS support in GraalVM and native images. If you’re distributing a native image, you’ll need to include libsunec. If it’s in the same directory as your image you don’t need to set java.library.path.

You’ll see a warning at run-time if this hasn’t been properly configured:
```
WARNING: The sunec native library could not be loaded.
```

Use more complete certificate store

GraalVM comes with a smaller set of CA certificates. For reasons they cannot yet distribute the Oracle JDK root certificates. You can workaround this by replacing GraalVM’s cacerts. I renamed the file and replaced it with a symbolic link to cacerts from the JRE that comes with macOS Mojave:

$ mv $GRAALVM_HOME/jre/lib/security/cacerts $GRAALVM_HOME/jre/lib/security/cacerts.bak
$ ln -s $(/usr/libexec/java_home)/jre/lib/security/cacerts $GRAALVM_HOME/jre/lib/security/cacerts

If you don’t do this, you might see such horrors as this when attempting HTTPS connections:

Exception in thread "main" javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
8<------------------------
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
8<------------------------
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Parsing with Clojure and Instaparse

Thu, 17 May 2018 12:00:00 +0000

Parsing is fun, and I’ve covered some simple Parsec-style parsers in previous posts, but this one’s about Clojure and Instaparse. Instead of building up parsers with combinators, Instaparse takes a grammar as a string and generates a parser. I think this makes for more concise parser definitions that are both easier to write and understand after the fact.

Bowling Scores

I’m not much of a bowler, but the game has an interesting scoring notation with some special rules. I don’t even know if this parser is totally correct but it should be very close. Generally the grammar definitions start top to bottom so things are declared in reverse dependency order. The top line is the general format—ten frames of bowling where each frame is F and 10TH has special rules. Reading downward you can see more terms are defined and combined.

(def bowling-score-parser
  (insta/parser
    "S = F F F F F F F F F 10TH
     F = OPEN | CLOSED
     OPEN = ROLL ROLL
     CLOSED = STRIKE | SPARE
     10TH = OPEN |
            SPARE (STRIKE | ROLL) |
            STRIKE (SPARE | ROLL ROLL) |
            STRIKE STRIKE (STRIKE | ROLL)
     STRIKE = 'X'
     SPARE = ROLL '/'
     ROLL = PINS | MISS
     PINS = #'[1-9]'
     MISS = '-'"))

And the parser outputs a tree of tagged values:

(bowling-score-parser "6271X9-8/XX3572-/X")
=>
[:S
 [:F [:OPEN [:ROLL [:PINS "6"]] [:ROLL [:PINS "2"]]]]
 [:F [:OPEN [:ROLL [:PINS "7"]] [:ROLL [:PINS "1"]]]]
 [:F [:CLOSED [:STRIKE "X"]]]
 [:F [:OPEN [:ROLL [:PINS "9"]] [:ROLL [:MISS "-"]]]]
 [:F [:CLOSED [:SPARE [:ROLL [:PINS "8"]] "/"]]]
 [:F [:CLOSED [:STRIKE "X"]]]
 [:F [:CLOSED [:STRIKE "X"]]]
 [:F [:OPEN [:ROLL [:PINS "3"]] [:ROLL [:PINS "5"]]]]
 [:F [:OPEN [:ROLL [:PINS "7"]] [:ROLL [:PINS "2"]]]]
 [:10TH [:SPARE [:ROLL [:MISS "-"]] "/"] [:STRIKE "X"]]]

;; some more scores to try
(bowling-score-parser "XXXXX6/XX7/XX5")
(bowling-score-parser "XXXXXXXXXXXX")
(bowling-score-parser "XXXXXXXXXX-/")
(bowling-score-parser "XXXXXXXXX--")
(bowling-score-parser "9-9-9-9-9-9-9-9-9-9-")
(bowling-score-parser "X7/9-X-88/-6XXX81")
(bowling-score-parser "X7/9-X-88/-6XX-/X")
(bowling-score-parser "X-/X5-8/9-X811-4/X")

Nested Infix Expressions

This is a nice example of the power of recursive parsers. Notice how the root S term is used several times in other terms. This allows for a simple grammar that can parse complex expressions like ((2*3)+1+2)/4.

(def expr-parser
  (insta/parser
    "<S> = VAL | EXPR | PAR
     PAR = <'('> S <')'>
     <EXPR> = S OP S
     VAL = #'[0-9]+'
     OP = '+' | '-' | '*' | '/'"))

The <angle brackets> around LHS terms indicate we don’t want those outputs tagged, and angle brackets on the RHS indicate the parsed value shouldn’t be returned at all. Omitting tags can make the output easier to work with, and in this case we don’t care about the actual parentheses characters.

(expr-parser "((2*3)+1+2)/4")
=> ([:PAR [:PAR [:VAL "2"] [:OP "*"] [:VAL "3"]]
          [:OP "+"] [:VAL "1"] [:OP "+"] [:VAL "2"]]
    [:OP "/"] [:VAL "4"])

Then with a simple prewalk we can interpret the output, ultimately transforming the string into its pure data structure form:

(walk/prewalk
  (fn [e]
    (if (vector? e)
      (let [[tag val :as p] e]
        (case tag
          :PAR (rest p)
          :VAL (Integer/parseInt val)
          :OP (symbol val)
          p))
      e))
  (expr-parser "((2*3)+1+2)/4"))
=> (((2 * 3) + 1 + 2) / 4)

Update: after reading this great article by Carin Meier I saw Instaparse provides a convenient transform function that can be passed a map from tags to value-interpreter functions:

(instaparse.transform/transform
  {:PAR list
   :VAL #(Integer/parseInt %)
   :OP symbol}
  (expr-parser "1+(2+3)*(3/3)"))
=> (1 + (2 + 3) * (3 / 3))

Happy parsing!

Logic programming in Clojure

Thu, 10 May 2018 12:00:00 +0000

This post covers some toy problems I’ve solved with Clojure’s core.logic.

Introduction

Logic and constraint-based programming interest me, which might explain why I enjoy SQL so much. I like the idea of describing a problem declaratively and having a computer give an exhaustive set of answers. For some problems, a “mechanical” solution might require much more code, and maybe much more complexity.

See this primer for basics, but I’ll say here the basic starting point for core.logic is usually the run* or run form, with binding(s) that you want to solve for:

(run* [q])

That won’t do anything by itself. We need to specify some goals/constraints on q:

(run* [q]
  (== [1 2 3] [1 q 3]))
=> (2)

This is telling us, for those two vectors to unify, the only possible value of q is 2.

Denomination Sums (or making change)

Given a set of denominations e.g. $1, $5, $10, how many ways are there to reach a given amount? This one was tricky!

First we define (another recursive) goal productsumo that will initially take a sequence of “fresh” logic variables, a set of denominations, and a sum that we want to reach. We first bind head and tail vars for both vars and dens, then we use arithmetic functions from the finite domain namespace of core.logic to constrain new fresh vars product and run-sum such that multiplying a denomination by some number and adding it to a running sum eventually reaches our desired sum (or not).

(defn productsumo [vars dens sum]
  (fresh [vhead vtail dhead dtail product run-sum]
    (conde
      [(emptyo vars) (== sum 0)]
      [(conso vhead vtail vars)
       (conso dhead dtail dens)
       (fd/* vhead dhead product)
       (fd/+ product run-sum sum)
       (productsumo vtail dtail run-sum)])))

That’s the only custom goal we need to solve the problem. Now we can wrap this in a function:

(defn change [amount denoms]
  (let [dens (sort > denoms)
        vars (repeatedly (count dens) lvar)]
    (run* [q]
      ;; we want a map from denominations to their quantities
      (== q (zipmap dens vars))
      ;; prune problem space...
      ;; every var/quantity must be 0 <= n <= amount
      (everyg #(fd/in % (fd/interval 0 amount)) vars)
      ;; the real work
      (productsumo vars dens amount))))

And now let’s find all the ways we can make 14 out of denominations 1, 2, 5, and 10:

(change 14 #{1 2 5 10})
=>
({10 0, 5 0, 2 0, 1 14}
 {10 1, 5 0, 2 0, 1 4}
 {10 0, 5 1, 2 0, 1 9}
 {10 0, 5 0, 2 1, 1 12}
 {10 1, 5 0, 2 1, 1 2}
 {10 1, 5 0, 2 2, 1 0}
 {10 0, 5 0, 2 2, 1 10}
 {10 0, 5 2, 2 0, 1 4}
 {10 0, 5 1, 2 1, 1 7}
 {10 0, 5 0, 2 3, 1 8}
 {10 0, 5 0, 2 4, 1 6}
 {10 0, 5 1, 2 2, 1 5}
 {10 0, 5 0, 2 5, 1 4}
 {10 0, 5 2, 2 1, 1 2}
 {10 0, 5 0, 2 6, 1 2}
 {10 0, 5 1, 2 3, 1 3}
 {10 0, 5 0, 2 7, 1 0}
 {10 0, 5 2, 2 2, 1 0}
 {10 0, 5 1, 2 4, 1 1})

🤯

Cryptarithmetic

I’d never seen this type of problem before this Stack Overflow question but it’s great for logic programming. See Wikipedia for a detailed description and examples:

Verbal arithmetic, also known as alphametics, cryptarithmetic, cryptarithm or word addition, is a type of mathematical game consisting of a mathematical equation among unknown numbers, whose digits are represented by letters. The goal is to identify the value of each letter.

The classic example, published in the July 1924 issue of Strand Magazine by Henry Dudeney is:

   SEND
+  MORE
-------
= MONEY

The solution to this puzzle is O = 0, M = 1, Y = 2, E = 5, N = 6, D = 7, R = 8, and S = 9.

I saw some similarities between this and the denominational sum problem above, and I was able to use the productsumo goal in solving this too. I also need another, simpler summation goal:

(defn sumo [vars sum]
  (fresh [vhead vtail run-sum]
    (conde
      [(== vars ()) (== sum 0)]
      [(conso vhead vtail vars)
       (fd/+ vhead run-sum sum)
       (sumo vtail run-sum)])))
;; and for getting sum of individual digits later
(defn magnitudes [n]
  (reverse (take n (iterate #(* 10 %) 1))))

The solver function creates logic variables based on the input words.

Get the set of all characters from all words
Zip those into a map with fresh logic variables, which will also be used to represent the answer
Get the logic variables for the first letter of each word; they must be constrained to non-zero numbers because the word-numbers can’t have leading zeroes
Create a logic variable per word to hold the sum of each word’s digits, reached with magnitudes and productsumo
Create groups of the character-level logic variables, representing each word

Then in run* the goals are tied together.

(defn cryptarithmetic [& words]
  (let [distinct-chars (distinct (apply concat words))
        char->lvar (zipmap distinct-chars (repeatedly (count distinct-chars) lvar))
        lvars (vals char->lvar)
        first-letter-lvars (distinct (map #(char->lvar (first %)) words))
        sum-lvars (repeatedly (count words) lvar)
        word-lvars (map #(map char->lvar %) words)]
    (run* [q]
      (everyg #(fd/in % (fd/interval 0 9)) lvars) ;; digits 0-9
      (everyg #(fd/!= % 0) first-letter-lvars)    ;; no leading zeroes
      (fd/distinct lvars)                         ;; only distinct digits
      (everyg (fn [[sum l]]                       ;; bind sums per word
                (productsumo l (magnitudes (count l)) sum))
              (map vector sum-lvars word-lvars))
      (fresh [s]
        (sumo (butlast sum-lvars) s) ;; sum all input word sums
        (fd/== s (last sum-lvars)))  ;; input word sums must equal last word sum
      (== q char->lvar))))

There are sample alphametics around the internet. It seems the most common example is “send more money”, which only has one possible solution. Most random phrases I’ve tried have either zero or many solutions — of course for the math to work out, the last word must be as long (i.e. have as many digits) as the longest word in the phrase.

(cryptarithmetic "send" "more" "money")
=> ({\s 9, \e 5, \n 6, \d 7, \m 1, \o 0, \r 8, \y 2})

The longest one I found is on Wikipedia, and takes about 30 seconds to run:

(def real-long-alphametic
  ["SO" "MANY" "MORE" "MEN" "SEEM" "TO"
   "SAY" "THAT" "THEY" "MAY" "SOON" "TRY"
   "TO" "STAY" "AT" "HOME" "SO" "AS" "TO"
   "SEE" "OR" "HEAR" "THE" "SAME" "ONE"
   "MAN" "TRY" "TO" "MEET" "THE" "TEAM"
   "ON" "THE" "MOON" "AS" "HE" "HAS"
   "AT" "THE" "OTHER" "TEN" "TESTS"])
(apply cryptarithmetic real-long-alphametic)
=> ({\A 7, \E 0, \H 5, \M 2, \N 6, \O 1, \R 8, \S 3, \T 9, \Y 4})

It’s useful to print these out:

(defn pprint-answer [char->digit words]
  (let [nums (map #(apply str (map char->digit %))
                  words)
        width (apply max (map count nums))
        width-format (str "%" width "s")
        pad #(format width-format %)]
    (println
     (clojure.string/join \newline
       (concat
        (map #(str "+ " (pad %)) (butlast nums))
        [(apply str (repeat (+ 2 width) \-))
         (str "= " (pad (last nums)))]))
     \newline)))

(defn pprint-cryptarithmetic [& words]
  (doseq [answer (apply cryptarithmetic words)]
    (pprint-answer answer words)))

(pprint-cryptarithmetic "what" "the" "hell")
+ 2368
+  831
------
= 3199 

+ 4526
+  651
------
= 5177 

+ 7826
+  685
------
= 8511 

+ 7852
+  281
------
= 8133

There’s only four possible answers for what the hell.

Update: I found a couple instances of this problem are also in the core.logic benchmark suite itself.

Non-consecutive Ordering

This is a permutation problem with a filtering step, but solving it with core.logic is fun. Given a sequence, we want to rearrange it such that no two values are repeated consecutively.

My approach was to define a goal that recursively constrained each pair in (a sliding window over) the input sequence to be unequal. Core.logic defines a firsto to set a goal for the first item in a sequence, but we need to define our own secondo that does the same but for the second item:

(defn secondo [l s]
  (fresh [x]
    (resto l x)
    (firsto x s)))

Using that we can write a nonconseco goal to recursively constrain the input:

(defn nonconseco [l]
  (conde
    ;; empty list is non-consecutive
    [(== l ())]
    ;; singleton list is non-consecutive
    [(fresh [x] (== l (list x)))]
    ;; the real work
    [(fresh [lhead lsecond ltail]
       (conso lhead ltail l)
       (secondo l lsecond)
       (!= lhead lsecond)
       (nonconseco ltail))]))

The third case in the conde defines three “fresh” logic variables for us to work with: lhead and ltail will be constrained to the first item of l and the rest of l respectively, by passing all three to conso. lsecond is constrained by our new secondo goal. Then we constrain lhead and lsecond to be unequal. Finally we recurse nonconseco with the the tail of l as its new input.

Those are the only two goals we need to define to make this work. Now we can just wrap it in a function and call it:

(defn non-consecutive [coll]
  (first
    (run 1 [q]
      (permuteo coll q)
      (nonconseco q))))

(non-consecutive [1 1 2 2 3 3 4 4 5 5 5])
=> (3 2 3 4 2 4 5 1 5 1 5)

This works fine with our custom nonconseco goal, but it’s not very fast for larger sequences, or when finding all non-consecutive permutations. We can define a predicate function to replace nonconseco which should be computationally cheaper than a recursive core.logic goal:

(defn non-consecutive? [coll]
  (every? (partial apply not=) (partition 2 1 coll)))

(run* [q]
  (permuteo [\a \b \b \a] q)
  (pred q non-consecutive?))
=> ((\b \a \b \a)
    (\a \b \a \b))

Here we found all possible answers by using run* instead of run. We used core.logic’s pred to apply our custom predicate function to q. We can’t apply it directly to q because q is a logic variable and we must get its actual value to apply a predicate. See the pred source for how it does that using project.

Non-relational Flatten

How about a slow, impractical version of clojure.core’s flatten?

(defn flatteno [l g]
  (condu
    [(emptyo l) (emptyo g)]
    [(fresh [h t]
       (conso h t l)
       (fresh [ft]
         (flatteno t ft)
         (condu
           [(fresh [fh]
              (flatteno h fh)
              (appendo fh ft g))]
           [(conso h ft g)])))]))

(defn logical-flatten [coll]
  (first (run 1 [q] (flatteno coll q))))

(logical-flatten '(((((1) 2)) 3) (4) 5 6 ((() 7)) 8 9))
=> (1 2 3 4 5 6 7 8 9)

I have a feeling flatteno can be simplified but I haven’t finished The Reasoned Schemer yet.

Building native Clojure images with GraalVM

Wed, 02 May 2018 12:00:00 +0000

GraalVM makes it possible to compile JVM-based programs ahead-of-time into a native executable, with significant improvements to startup time, resource usage, etc. This post will demonstrate a CLI tool written in Clojure and compiled to a native binary, and compare its performance to the JVM-based version.

Update 2018-05-25: I created a simple Leiningen plugin lein-native-image to simplify native-image usage with Leiningen projects; its README has more up-to-date info and additional example projects. I created a similar tool clj.native-image for use with deps.edn projects. This article was written beforehand and assumes neither is used.

This approach is similar to Jan Stępień’s article but runs in macOS rather than within a Linux container.

I’ll create a simple command line utility program that converts JSON to EDN, starting with a minimal Leiningen project.clj:

(defproject jadensmith "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.9.0"]
                 [org.clojure/data.json "0.2.6"]]
  :main ^:skip-aot jadensmith.core
  :profiles {:uberjar {:aot :all}})

And the implementation:

(ns jadensmith.core
  (:require [clojure.data.json :as json]
            [clojure.edn :as edn])
  (:gen-class))

(defn -main [& args]
  (prn (json/read *in* :key-fn keyword)))

It reads JSON from stdin and prints it to stdout in EDN format.

This article shows how to use GraalVM’s native-image on particular class files, but requires manually copying/unpacking some dependency JARs into the target directory beforehand. We can avoid that since native-image can also use JAR files, so let’s build an uberjar:

$ lein uberjar

Note: Building an uberjar is unnecessary when using lein native-image.

Obtaining GraalVM

Now you’ll need to obtain GraalVM. I downloaded the Community Edition for macOS and placed it in some odd directory on my hard drive, then set an environment variable similar to $JAVA_HOME:

export GRAALVM_HOME=/path/to/graalvm-ce-1.0.0-rc6/Contents/Home

On macOS, the contents of GraalVM’s Home directory should look similar to a JDK installation e.g. Home/bin/java.

Building a native image

We can now use native-image on the standalone uberjar:

$ cd target
$ $GRAALVM_HOME/bin/native-image -jar jadensmith-0.1.0-SNAPSHOT-standalone.jar
Build on Server(pid: 13184, port: 26681)*
   classlist:   2,240.42 ms
       (cap):   1,690.30 ms
       setup:   2,808.39 ms
  (typeflow):   4,677.28 ms
   (objects):   3,363.83 ms
  (features):      44.11 ms
    analysis:   8,229.62 ms
    universe:     326.01 ms
     (parse):   1,154.86 ms
    (inline):   1,200.41 ms
   (compile):   7,656.20 ms
     compile:  10,606.37 ms
       image:   1,926.50 ms
       write:   1,357.57 ms
     [total]:  27,586.92 ms

The resulting image:

$ la jadensmith-0.1.0-SNAPSHOT-standalone
-rwxr-xr-x  1 taylorwood  44583454   6.8M May  2 06:24 jadensmith-0.1.0-SNAPSHOT-standalone*

It’s slightly larger than the 4.2M uberjar, but the native image has no dependency on JRE/JDK and has some other interesting properties:

The resulting program does not run on the Java HotSpot VM, but uses necessary components like memory management, thread scheduling from a different implementation of a virtual machine, called Substrate VM. Substrate VM is written in Java and compiled into the native executable. The resulting program has faster startup time and lower runtime memory overhead compared to a Java VM.

Let’s time the JAR version by feeding it JSON:

$ /usr/bin/time -l sh -c 'cat test.json | java -jar jadensmith-0.1.0-SNAPSHOT-standalone.jar'
...
        1.02 real         1.79 user         0.11 sys
 117514240  maximum resident set size
8<-----------------------------------
     32345  page reclaims
8<-----------------------------------
         6  signals received
         4  voluntary context switches
      2860  involuntary context switches

Now let’s time the native image for comparison:

$ /usr/bin/time -l sh -c 'cat test.json | ./jadensmith-0.1.0-SNAPSHOT-standalone'
...
        0.02 real         0.01 user         0.00 sys
  10436608  maximum resident set size
8<-----------------------------------
      3449  page reclaims
8<-----------------------------------
         4  voluntary context switches
        48  involuntary context switches

We can see considerable differences in startup time, CPU usage, and memory usage. The JVM version uses ~11x more memory and takes ~10x longer. This seems promising for writing native CLI tools in Clojure.

Limitations

There are some limitations to GraalVM’s native image building:

To achieve such aggressive ahead-of-time optimizations, we run an aggressive static analysis that requires a closed-world assumption. We need to know all classes and all bytecodes that are reachable at run time. Therefore, it is not possible to load new classes that have not been available during ahead-of-time-compilation.

There are other limitations on reflection, documented here. Some of these issues can be worked around with hinting/configuration; see these projects for examples of such workarounds.

GraalVM’s native-image has a flag to treat these issues as exceptions at run-time, instead of its default behavior of failing to build. Hopefully some of these limitations will be lifted over time, and GraalVM can support more non-trivial Clojure apps out-of-the-box.

Excellent article by @janstepien "Native Clojure with GraalVM" creating a lightweight web server. We will incrementally lift restrictions for native image generation in upcoming releases. https://t.co/tsx0OBtssG pic.twitter.com/Td387PHRoa
— GraalVM (@graalvm) April 28, 2018

clojure.spec for functions by example

Sun, 15 Oct 2017 12:00:00 +0000

Clojure.spec for functions by example: their inputs, outputs, and the relationships between them. Updated 9/6/2018.

Simple Input & Output

We have a simple function that takes a single argument and returns a set:

(defn digits
  "Takes just an int and returns the set of its digit characters."
  [just-an-int]
  (into #{} (str just-an-int)))

Function specs are typically defined with s/fdef. Let’s provide a spec for the arguments:

(s/fdef digits :args (s/cat :just-an-int int?))

Note: with s/cat the function’s argument names and the keywords used to describe them in the spec don’t have to match — it’s the position that’s important.

Function argument lists can be spec’d as a sequence, which makes perfect sense as they are a sequence of arguments. I often prefer s/cat, treating the arguments as a tagged concatenation, although :args specs can also be described with s/tuple, s/alt, s/+, s/*, s/?, etc.

We can now instrument the function to assert calls against its :args spec:

(stest/instrument `digits)

This only covers the function’s inputs. We can also spec the return value:

(s/fdef digits
        :args (s/cat :just-an-int int?)
        :ret (s/coll-of char? :kind set? :min-count 1))

With a :ret spec, we can now check the function against our spec with generative tests:

(stest/check `digits)

Keyword Arguments

There’s a s/keys* macro that works for kwargs, similarly to how s/keys works for maps.

(defn big-fun
  [why when & {:keys [who which what]}]
  (format "%s %s %s %s %s" why when who which what))

(s/fdef big-fun
        :args (s/cat :why string?
                     :when #(instance? LocalDateTime %)
                     :kwargs (s/keys* :req-un [(or ::who
                                                   (and ::which ::what))])))

(stest/instrument `big-fun)
(big-fun "because" (LocalDateTime/now) :which 1 :what 4) ;; valid
(big-fun "said so" (LocalDateTime/now) :who "I")         ;; also valid
(big-fun "because" (LocalDateTime/now) :which 1)         ;; invalid

Here we defined a kwargs spec that requires either a :who key or a :which key and :what key, just like we could’ve done for a map.

Multi-arity & Variadic Functions

We can spec multi-arity functions using s/alt to define the alternate cases, and variadic functions using s/*. Here’s a function with two fixed arities and one variadic arity:

(defn such-arity
  ([] "nullary")
  ([one] "unary")
  ([one two & many] "one two many"))

(s/fdef such-arity
        :args (s/alt :nullary (s/cat)
                     :unary (s/cat :one any?)
                     :variadic (s/cat :one any?
                                      :two any?
                                      :many (s/* any?))))

We can spec the zero-arity with (s/cat); no arguments. We specify the variadic arguments with s/*. The “regex” specs compose to match a single sequence. This composition of s/cat and s/* produces a spec that wants a sequence of two arguments and then any number of additional arguments.

And here’s a variadic function with positional destructuring:

(defn slarp [path & [blurp? glurf?]]
  (str path blurp? glurf?))
(s/fdef slarp
  :args (s/cat :path string?
               :rest (s/? (s/cat :blurp? int?
                                 :glurf? boolean?))))
(s/exercise-fn `slarp)
=>
([("") ""]
 [("" -1 false) "-1false"]
 [("e7" 0 true) "e70true"]
 [("du9") "du9"]
 [("K1" -6 false) "K1-6false"]
 [("op0" -3 false) "op0-3false"]
 [("2y" 2 false) "2y2false"]
 [("qaGWVK") "qaGWVK"]
 [("9h1Rh") "9h1Rh"]
 [("") ""])

Higher-Order Functions

This is an only slightly different, more involved example than the one in the clojure.spec guide.

Here’s a spec for a function that takes any two arguments and returns one of them:

(s/def ::pick-fn
  (s/with-gen
    (s/fspec :args (s/cat :a any? :b any?)
             :ret any?
             :fn #(or (= (:ret %) (:a (:args %)))
                      (= (:ret %) (:b (:args %)))))
    #(gen/return (fn [a b] (rand-nth [a b])))))

This function spec is registered to a qualified keyword rather than a function symbol via s/fdef. This is defined with s/def and s/fspec because I want to use this function spec in more than one place, and specify a custom generator. Notice the additional :fn spec that asserts the returned value is one of the input values. This spec will be checked when we check the function with generative tests.

I use s/with-gen to specify a custom generator that returns a function that returns one of its arguments randomly. I could however just allow spec to create a function generator from my function spec; the generated functions would return random samples of its :ret spec, and any calls would have their arguments asserted against its :args spec. A spec-generated function would not be desirable in this contrived example because we want to ensure the function always returns something it was given.

How about this for some implementation of ::pick-fn:

(defn battle
  "Returns the victor."
  [contestant-1 contestant-2]
  (if (even? (System/nanoTime)) contestant-1 contestant-2))

Now I want to associate the ::pick-fn spec with battle function. We can achieve the same effect as (s/fdef battle ...) with s/def:

(s/def battle ::pick-fn)
(stest/instrument `battle)
(stest/check `battle)

Now let’s write a higher-order function to take a ::pick-fn-conformant function and a collection:

(defn winners
  "Uses f to pick a successor between each element of coll, returning sequence of successors."
  [f coll]
  (rest (reductions f coll)))
(winners battle (range 20))
;;=> (0 0 0 0 4 4 6 6 6 6 10 10 10 10 14 14 14 14 14 19)

And a spec for the function:

(s/fdef winners
        :args (s/cat :f ::pick-fn
                     :coll (s/coll-of any? :min-count 2))
        :ret coll?
        :fn (fn [{:keys [args ret]}]
              (and (= (count ret) (dec (count (:coll args))))
                   (set/subset? (set ret) (set (:coll args))))))
(stest/instrument `winners)
(stest/check `winners)

The winners function has some odd invariants. The f arg must conform to ::pick-fn and coll must have at least two elements. Additionally, the relationship between inputs and outputs for winners is expressed in the :fn spec:

It should always return a collection that has one less element than the input collection. This is because the first “pick” is for the first two input elements.
The returned elements should be a subset of the input elements.

Warning

Any function g you pass into a higher-order, instrumented function f may be invoked multiple times as spec tests its conformance! This might be a problem if g is expensive or has side effects.

(defn f [g] (g 1))
(s/fdef f :args (s/cat :g (s/fspec :args (s/tuple int?))))
(st/instrument `f)

Now if we call f with a side-effecting function (printing text) we can see the given function invoked many times with random inputs — a sort of miniature quick-check.

(f #(doto % prn inc))
-1
0
-2
...
59
-69770
-1165
1
=> 1

Possible Workarounds

Simplify the higher-order function’s spec:

(s/fdef f :args (s/cat :g ifn?))

Or disable the behavior using a dynamic binding in spec:

(binding [clojure.spec.alpha/*fspec-iterations* 0]
  (f #(doto % prn inc)))
1
=> 1

Run-time Checking

Instrumentation

You can call instrument with no arguments to instrument all loaded/instrumentable vars, and you may want to do this at dev/test time but not in production to avoid performance penalties. One convenient solution is to have a separate dev/test entrypoint in your program that calls instrument after the relevant namespaces are loaded. Another option with Leinengen is to use :injections in your project.clj under specific profile(s):

:injections [(require 'lib.core) ;; all instrumented fns should be loaded here
             (require 'clojure.spec.test.alpha)
             (clojure.spec.test.alpha/instrument)]

Spec’s instrument only checks function inputs (:args specs). Alternatively, the Orchestra library has a drop-in replacement that also checks :ret and :fn specs.

`:pre` and `:post` checks

This will throw an AssertionError if called with any non-nil, non-string value:

(defn stringer-bell
  "Prints a string and rings bell."
  [s]
  {:pre [(s/valid? (s/nilable string?) s)]}
  (println s "\007"))

Asserting

There’s also s/assert which can be used in function bodies:

(defn stringer-bell [s]
  (s/assert (s/nilable string?) s)
  (println s "\007"))

You need to enable spec’s off-by-default assertion checking for this to have an effect:

(s/check-asserts true)

s/assert will throw a more detailed exception than a :pre/:post condition, although be careful not to use s/assert there: it returns valid values and if the spec allows nil, the condition will fail when a valid nil is passed.

Infix DSL Validation with clojure.spec

Wed, 04 Oct 2017 12:00:00 +0000

Say you’re going to write some code to parse and validate logical boolean clauses as data. Maybe you’re writing a DSL to allow users to express some rules for a rules engine, e.g. “date is today and junk is true”. Wouldn’t it be nice to be able to validate these logical declarations, perhaps by defining a specification declaratively?!

Clojure.spec gives us the tools to define a specification, evaluate arbitrary input against it, and (by way of clojure/test.check) even generate random sample data accordant with the specification.

Simplifying Assumptions

Let’s assume a simple case where each logical expression is just a keyword, and they can be joined by boolean operators (also represented as keywords) in infix notation. We’ll assume only and and or operators are allowed.

[:x :and :y]

We can pretend :x and :y refer to some other logical expression for now.

Of course users want to group and nest these logical expressions.

[:x :and :y [:a :or :b [:all :or :nothing]]]

And of course we’ll assume we’re able to parse user input into this vector/keyword form.

Declaring Specs

When we consider this problem, there are only two unique ingredients to our expressions:

Expressions
Boolean operators

Clojure.spec defines a simple language of easily-composed primitives to allow very rich specifications over data. We can start by defining a spec per ingredient, and let’s start simple with the boolean operators.

(def op-keys #{:and :or})

Sets can be used as specs, so our spec for operators is simply a set of keywords.

Those operators need operands. Here we’ll define a spec called ::expression:

(s/def ::expression
  (s/and keyword?
         #(not (contains? op-keys %))))

Specs are resolved by name and they are named by namespaced keywords. The double-colon prefix is shorthand for whatever the current namespace is.

Our expression is just a keyword but it can’t be one of the operators, so we use spec’s and to combine two logical predicates.

We have our two ingredients but we need a recipe for how they can be combined.

(s/def ::group
  (s/cat :head ::expression
         :tail (s/*
                 (s/cat :op     op-keys
                        :clause (s/or :expr  ::expression
                                      :group ::group)))))

This ::group spec defines our overall expression as a concatenation of elements. It must always have some head element which conforms to our ::expression spec above. Maybe just one thing must be true, and that’s fine. 🤷‍♂️ After the head there can be any number * of additional expressions, which are also concatenations of an operator and… you guessed it, more expressions, which may contain more expressions, which may contain more expressions. Very expressive.

Our specification is recursive—a grouping of expressions is itself an expression. Luckily, defining a recursive spec is trivial! On the last line we simply refer to the ::group spec from within its own definition.

Trial & Error

Whenever I’m writing a spec, I’m always playing with it in the REPL as I go. Unsurprisingly I went through many (oft non-working) iterations before landing on the oh-so-elegant-and-plainly-correct spec above.

(s/conform ::group [:x])
=> {:head :x}

Clojure.spec’s conform function takes a spec (either a namespaced keyword that refers to one or an inline spec) and some data to be conformed. A very interesting aspect of clojure.spec is that the process of conforming input data to a spec can produce arbitrarily different output data. In this case, the cat and or functions will tag our output as you can see above; we get a map back telling us where our :head is.

(s/conform ::group [:x :y])
=> :clojure.spec.alpha/invalid

When the input data does not conform, we get back a special namespaced keyword. If we simply wanted a yes or no predicate we could use s/valid?.

Let’s put it to some slightly tougher tests:

(s/conform ::group [:x :and :y])
=> {:head :x, :tail [{:op :and, :clause [:expr :y]}]}
(s/conform ::group [:x :and :y :or]) ;; no good trailing operator
=> :clojure.spec.alpha/invalid
(s/conform ::group [:x :and :y :or :z])
=> {:head :x, :tail [{:op :and, :clause [:expr :y]} {:op :or, :clause [:expr :z]}]}
(s/conform ::group [:x :and :xx :and :yy :and [:y :or :z :or [:foo :and :bar]]])
=>
{:head :x,
 :tail [{:op :and, :clause [:expr :xx]}
        {:op :and, :clause [:expr :yy]}
        {:op :and,
         :clause [:group
                  {:head :y,
                   :tail [{:op :or, :clause [:expr :z]}
                          {:op :or, :clause [:group {:head :foo, :tail [{:op :and, :clause [:expr :bar]}]}]}]}]}]}

Generation

Surely after you validate these input clauses you’re going to want to do something with them? What if you could conjure up a universe of test case inputs for that something? (Spoiler: you can.) Using test.check we can randomly generate samples of our spec:

 (sgen/sample (s/gen ::group))
 => ;; abbreviated output
 (:o/h?)
 (:E :and :T :or (:X :or (:?* :or :f1/a :and (:+ :and :s))) :and (:z- :or (:r/D :and (:+M) :or :+W/-0)))
 (:s*/PC :or :-P :or (:N-/_ :and :W/_-) :and (:!))
 (:J__/e1 :and :K :and (:K) :and :X7m.W/Dt :or (:W))

Nice.

Bonus Round

What if you needed to emit some sort of string query for these expressions?

(defn clause-str [clause] ;; it's too early for doc strings
  (walk/postwalk
    (fn [elem]
      (if (coll? elem)
        (format "(%s)" (cs/join " " (map name elem)))
        elem))
    clause))

(clause-str [[:x :or :y] :and :x])
=> "((x or y) and x)"

Here we use clojure.walk’s postwalk to perform a depth-first traversal of our data strucure (a vector of keywords and/or vectors). The depth-first part is important because we want our transformed output to reflect the nested group structure of our data, which we accomplish here by simply parenthesizing the group strings. For the group elements we just take the name of each keyword joined by spaces.

Let’s exercise clause-str with those randomly generated samples.

(map clause-str (sgen/sample (s/gen ::group)))
=>
("(e)"
 "(u)"
 "(!8 or xQ and (b and ?! and (b)))"
 "(L1)"
 "(!M)"
 "(?0V)"
 "(+J or (v? or A and (+4 or (z+ or I or (d+))) and (uhd and ! or (i or (RS1)))) or *82 or SC and (B or _ or G+! or Lj or (_ or j and (O?* or +hK and (P) or Y8 or z and (p4+) and o-r) and (z8 or y*2 and n or (G-) or TFi and (FH)) or GSc or (ZF or (!-3)) and +?)))"
 "(OrT or (h-! or (c or (- and me and G) and (-6 and (G) and ?)) and M and (b or (+*+) and (c* and (n8-) or (v_)) and pKM or a7t or (l or o and (Fz) and (e7x) or (+9Y) and Cl or Mm) and (E or (-) or (_7G) and (M6q) or Uv and +m and (a*3))) or (!+ and ! or (+D? and (e) and (l) and (D)))) or (-K or (_* and p5 and (Wv and (b8+) or k! and W or R7? and l8V or oli) and p3p or (u or (w7P) or +Vi and n and x6 and (?) and (+_))) or n and !7v and rKQ) and t!3 or (c and (x2) or (_+ and (rE and (*2X) or (V) and C or (MW)) and (vx_ and (vZ3) and W5 or (+8?) or (G6)) or (X or d3A or J*9 and zS+) or (G_l or EFr) or (B-H and (T) or (!-C))) or Ne5) or Q and u)"
 "(-1 or o- or ?xE and -X8 or gY and w or (kD* and (hxj and (? or v or (N) and (_) or (z+0) and (+) and (rhc) or ?) or t+ and *) or T4 or d*))"
 "(?J and (A) or (c2 or c or - or (U or (kT and (q!) or (t) and (xF) and g! and -60 and d3H and (R6?)) and A5) and gW8))")

Or just call it with some bespoke inputs:

(s/conform ::group [[:x
                     :or :y
                     :or :z
                     :or [:q :and :z]]
                    :and :x
                    :and :y
                    :and :z
                    :and [:x
                          :or :y
                          :or :z
                          :or [:q :and :z]]])
=> "((x or y or z or (q and z)) and x and y and z and (x or y or z or (q and z)))"

Callers Behave

Now with a spec in hand, we can ensure clause-str is called with valid arguments using clojure.spec.test’s instrument.

(s/fdef clause-str                              ;; a spec for our function
        :args (s/cat :clause (s/spec ::group))) ;; it takes one argument and it better be a ::group
(stest/instrument `clause-str)                  ;; assert valid args on each call
(clause-str [:x :and :y])
=> "(x and y)"

An invalid call will now throw an exception with information:

(clause-str [:x :and])
CompilerException clojure.lang.ExceptionInfo: Call to #'playground.test/clause-str did not conform to spec:
In: [0] val: () fails spec: :playground.test/group at: [:args :clause :tail :clause] predicate: (or :expr :playground.test/expression :group :playground.test/group)...

Generative/Property-based Testing

For the sake of example, let’s add a (unfortunately meaningless/tautological) return value assertion to our function spec, then check the function’s properties:

(s/fdef clause-str
        :args (s/cat :clause (s/spec ::group))
        :ret  string?)
(stest/check `clause-str)

After a minute or so you’ll realize this isn’t going to finish any time soon. The issue is that test.check is generating very complex, deeply nested samples for our recursive spec. Luckily clojure.spec has a workaround via dynamic binding clojure.spec.alpha/*recursion-limit*.

(binding [s/*recursion-limit* 1]
  (stest/check `clause-str))

Putting a tight upper bound on generative recursion allows check to run in a few seconds.

Property-based testing is great, but you must accurately define the properties. In this case, our properties are derived from our spec, and our spec isn’t quite right. Calling our instrumented function with hand-generated inputs reveals such a problem:

(clause-str [[:x :or :y] :and :z])
CompilerException clojure.lang.ExceptionInfo: Call to #'playground.test/clause-str did not conform to spec:
In: [0 0] val: [:x :or :y] fails spec: :playground.test/expression at: [:args :clause :head] predicate: keyword?

Our ::group spec is a concatenation expecting a :head element that conforms to ::expression. In this case, our first/:head element is actually a sub-::group of expressions, which doesn’t conform to ::expression. We can fix it by revising our ::group spec to be recursive in the :head position too:

(s/def ::group
  (s/cat :head (s/or :g ::group :e ::expression)
         :tail (s/*
                 (s/cat :op     op-keys
                        :clause (s/or :expr  ::expression
                                      :group ::group)))))

Or we can get rid of that duplicated s/or with another forward-referencing spec:

(s/def ::subgroup (s/or :g ::group :e ::expression))
(s/def ::group
  (s/cat :head ::subgroup
         :tail (s/* (s/cat :op op-keys :clause ::subgroup))))

Try generating samples of ::group before and after this change to see the difference; the latter samples are much more complex.

Full Example

Many thanks to the awesome Clojure community! When I ran into issues, the maintainer of test.check chimed in instantly on Clojurians Slack with advice.

Taylor ≈ Programmer

Using Cognitect's aws-api with AWS Transcribe

Comparisons

API Interactions

Doing Things

GraalVM Polyglot with Clojure and JavaScript

Idiomatic Coercion

Executables

Arrays

Associatives

Polyglottin’

Functional Programming

Clojurvascriptception

Loading External JavaScript

Closing Remarks

Clojure.spec Beginner's FAQ

Usage

Organization

Instrumentation & Testing

Sequences & Regex Specs

Map Specs

Generation

Specs as Data

Off-label Usage

Syntax Synonyms

Nice

GraalVM Native Image HTTPS Support

How To

Parsing with Clojure and Instaparse

Bowling Scores

Nested Infix Expressions

Logic programming in Clojure

Introduction

Denomination Sums (or making change)

Cryptarithmetic

Non-consecutive Ordering

Non-relational Flatten

Building native Clojure images with GraalVM

Obtaining GraalVM

Building a native image

Limitations

clojure.spec for functions by example

Simple Input & Output

Keyword Arguments

Multi-arity & Variadic Functions

Higher-Order Functions

Warning

Possible Workarounds

Run-time Checking

Instrumentation

:pre and :post checks

Asserting

Infix DSL Validation with clojure.spec

Simplifying Assumptions

Declaring Specs

Trial & Error

Generation

Bonus Round

Callers Behave

Generative/Property-based Testing

Full Example

`:pre` and `:post` checks