The GUI is written entirely in Clojure with the Swing toolkit. Calculation state is stored in a series of atoms. Watches are used to update the output table automatically when an atom changes, an idea from Kotka. I used the excellent MiG Layout for general layout functionality, and generic Swing widgets (JTextFields and a JTable) for the input and output.

Clojure code

The Black-Scholes GUI is launched with themain function. The Swing GUI is actually created in the initialize-gui function, but since Swing widgets are not threadsafe, we must use the do-swing* helper function from Clojure Contrib to spin off the real GUI creation code into the Swing event thread.

(defn initialize-gui
  []
  (let [frame (JFrame. "Black-Scholes Option Modeler")]

    (doto frame
      (.setDefaultCloseOperation JFrame/DISPOSE_ON_CLOSE)
      (-> .getContentPane
          (.add (miglayout (JPanel.)

                           (input-panel
                            (fn [_]
                              (do-swing
                               (doto frame
                                 (.setVisible false)
                                 (.dispose)))))

                           (result-table)

                           )))

      (.setDefaultCloseOperation JFrame/DISPOSE_ON_CLOSE)
      (.pack)
      (.setVisible true))
     ))

(defn main
  []
  (do-swing* :now initialize-gui))

This is pretty straightforward. We make a (locally scoped) JFrame to be the main container at line 3. Within the context of the let, we add a JPane using MiG Layout to its content area (starting at line 8). The JPanel itself contains two subwidgets, which are those returned from the input-panel and result-table functions.

Note that we are passing an anonymous function to input-panel at line 11. The input panel contains the “Quit” button, and that button needs to know what to do when it’s pressed. We want it to close the GUI; that is, it should destroy the outer JFrame. To do that, it needs a reference to it. However, as a local variable in initialize-gui, the JFrame is not in scope elsewhere; we have to pass it in. The given function closes over the JFrame reference (line 13) and transports it to input-panel, where that function will be called as an event listener when the “Quit” button is pressed.

We could also have simply passed the bare JFrame rather than a function, and moved the frame-closing code into input-panel, but that would imply that the input panel must always be in a JFrame, which will not necessarily the case. Passing in an external anonymous function to be activated by the “Quit” button allows the input panel to be used generically in any sort of container, with the container deciding what to do when “Quit” is pressed.

The Input Panel

(defn- input-panel [quit-action]
    (let [
          ;; Fields
          spot (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                 (.setColumns 6)
                 (.setValue @*spot*))

          strike (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                   (.setColumns 6)
                   (.setValue @*strike*))

          days-till-expiry (doto (JFormattedTextField. (NumberFormat/getIntegerInstance))
                             (.setColumns 6)
                             (.setValue @*days-till-expiry*))

          riskfree (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                     (.setColumns 6)
                     (.setValue @*riskfree*))

          volatility (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                       (.setColumns 6)
                       (.setValue @*volatility*))

          ;; Buttons
          calculate (JButton. "Calculate")
          quit (JButton. "Quit")
          ]

      (add-action-listener
       calculate
       (fn [_]
         (update-all-calculations
          (Double/parseDouble (.getText spot))
          (/ (Integer/parseInt (.getText days-till-expiry)) *trading-days-per-year*)
          (Double/parseDouble (.getText strike))
          (Double/parseDouble (.getText riskfree))
          (Double/parseDouble (.getText volatility)))))

      (add-action-listener
       quit
       quit-action)

      (miglayout (JPanel.)
                 :layout  [:wrap 2 ]

                 (JLabel. "Spot") [:align "right"]
                 spot

                 (JLabel. "Strike") [:align "right"]
                 strike

                 (JLabel. "Risk-free rate") [:align "right"]
                 riskfree

                 (JLabel. "Volatility") [:align "right"]
                 volatility

                 (JLabel. "Days till expiry") [:align "right"]
                 days-till-expiry

                 calculate
                 quit)))

The input widgets are local variables in the input panel function, along with the two buttons. The widgets are initialized from the state atoms (which are then unused again; see discussion below.) The “Quit” button is given the event callback passed in as an argument (so it can manipulate the enclosing JFrame as per above.) The “Calculate” button is given an event callback that calls the update-all-calculations function with the values in the input widgets.

The lot are then stuffed into a JPanel with a MiG Layout and returned.

The Output Table

(defn- update-cell
  "Updates the given AbstractTableModel cell and fires a change event for that cell."
  [model data row col]
  (.setValueAt model data row col)
  (.fireTableCellUpdated model row col))

(defn- result-table []
  (let [
        column-names ["" "Call" "Put"]
        row-names ["Price" "Delta" "Theta" "Rho" "Gamma" "Vega"]
        table-model (proxy [AbstractTableModel] []
                      (getColumnCount [] (count column-names))
                      (getRowCount [] (count row-names))
                      (isCellEditable [] false)
                      (getColumnName [col] (nth column-names col))
                      (getValueAt [row col]
                                  (condp = col
                                      0 (nth row-names row) ;; Return the row name for column zero
                                      (condp = [row col]
                                          [0 1] @*call-price*
                                          [0 2] @*put-price*

                                          [1 1] @*call-delta*
                                          [1 2] @*put-delta*

                                          [2 1] @*call-theta*
                                          [2 2] @*put-theta*

                                          [3 1] @*call-rho*
                                          [3 2] @*put-rho*

                                          [4 1] @*gamma*
                                          [4 2] "=="

                                          [5 1] @*vega*
                                          [5 2] "=="

                                          nil))))

        table (doto (JTable. table-model)
                (.setGridColor java.awt.Color/DARK_GRAY))
        ]

    (add-watch *call-price* ::update-call-price (fn [_ _ _ newprice] (update-cell table-model newprice 0 1)))
    (add-watch *put-price* ::update-put-price (fn [_ _ _ newprice] (update-cell table-model newprice 0 2)))

    (add-watch *call-delta* ::update-call-delta (fn [_ _ _ newval] (update-cell table-model newval 1 1)))
    (add-watch *put-delta* ::update-put-delta (fn [_ _ _ newval] (update-cell table-model newval 1 2)))

    (add-watch *call-theta* ::update-call-theta (fn [_ _ _ newval] (update-cell table-model newval 2 1)))
    (add-watch *put-theta* ::update-put-theta (fn [_ _ _ newval] (update-cell table-model newval 2 2)))

    (add-watch *call-rho* ::update-call-rho (fn [_ _ _ newval] (update-cell table-model newval 3 1)))
    (add-watch *put-rho* ::update-put-rho (fn [_ _ _ newval] (update-cell table-model newval 3 2)))

    (add-watch *gamma* ::update-gamma (fn [_ _ _ newval] (update-cell table-model newval 4 1)))
    (add-watch *vega* ::update-vega (fn [_ _ _ newval] (update-cell table-model newval 5 1)))

    ;; This shrinks the table's preferred viewport down to its actual size.
    ;; (The default is to make a huge viewport, even though the table is small.)
    (.setPreferredScrollableViewportSize table (.getPreferredSize table))

    (JScrollPane. table)
    ))

The output table itself is a standard JTable. Its model is initialized from the output state atoms. The watches at lines 44-57 are the most interesting part; they allow arbitrary code to be called when the output state atoms change. This allows the Black-Scholes calculation to happen independently of the output widget. When one of those atoms is changed, it self-updates automatically via the watches.

The helper function at lines 1-5 updates the appropriate model cell and fires an event to tell the JTable to repaint it.

The scrollable viewport business at line 61 is to work around a questionable Java design decision where a JTable, by default, tells its container to make its viewport enormous, even though it have very little data. (That’s not an error, that’s subjunctive.)

Design considerations

GUI and business logic decoupling

Above all, the GUI has to be decoupled from the model itself. Internal changes to one should never affect the other, and the two should be independently testable. In such a simple application, this was easy enough. A more complex app that requires several information round trips would be trickier. We now have a good foundation for such an app. No business calculations are performed in the GUI code namespace. The sole point of connection is in the update-all-calculations function:

(defn- update-all-calculations [spot timeleft strike riskfree sigma]
  (swap! *call-price* (fn [_] (bs/call spot timeleft strike riskfree sigma)))
  (swap! *put-price* (fn [_] (bs/put spot timeleft strike riskfree sigma)))

  (swap! *call-delta* (fn [_] (bs/call-delta spot timeleft strike riskfree sigma)))
  (swap! *put-delta* (fn [_] (bs/put-delta spot timeleft strike riskfree sigma)))

  (swap! *call-theta* (fn [_] (bs/call-theta spot timeleft strike riskfree sigma)))
  (swap! *put-theta* (fn [_] (bs/put-theta spot timeleft strike riskfree sigma)))

  (swap! *call-rho* (fn [_] (bs/call-rho spot timeleft strike riskfree sigma)))
  (swap! *put-rho* (fn [_] (bs/put-rho spot timeleft strike riskfree sigma)))

  (swap! *vega* (fn [_] (bs/vega spot timeleft strike riskfree sigma)))
  (swap! *gamma* (fn [_] (bs/gamma spot timeleft strike riskfree sigma))))

Atoms to hold state

As we can see above, set of atoms holds state for the GUI, in keeping with the general Clojure principle of isolating state into transactional memory. (Actually, I cheated a bit. Our full state consists of the input values and the results of our Black-Scholes calculations. Although I created atoms for the input state values as well, I calculate the results directly from the widgets. It is a trivial exercise to alter the “Calculate” button callback to use the atoms. This would be more useful in the case where changes to a widget fire an event that triggers recalculation, which is not done in this demo as per below. I would use the atoms in a real application.)

I/O is inherently stateful, as are GUIs. There was therefore the temptation to use the Swing widget objects themselves as state-holders, and in fact it requires extra code to shuttle data back and forth between the widgets and the atoms. In such a simple app as this, using the GUI widgets as state-holders does not make much practical difference, but in a more complex app, a well designed set of state atoms and watches can automate much of the tedious GUI logic.

Using the widgets directly as state holders implies a tight coupling between the input widgets, the business logic, and the output widgets. Changing any one of them would invariably have required changes to the linking logic.

For example, suppose we were using widgets-as-state, and we changed the “days left till expiry” input to a slider rather than a text box, or suppose we wanted to read it from a file or from an API. We would have to update the code that reads it, calculates the model, and puts the results into the output table. Touching that code implies a non-zero likelihood of breaking it. Why even introduce the potential for breaking something that is unrelated to the task at hand, in this case the business logic and output code?

Again, this seems trivial with such a very small application that is only a few lines long, but the design principle is what’s important. Imagine applying it to a much larger application with hundreds of inputs spread across 4 windows, 3 databases, and 6 web feeds to see why this is a good idea.

Auto-updating results when input changes

I wanted the results to auto-update anytime an input is changed, without resorting to a “Calculate” button. Having the button trigger the calculation and output update allows the state of the input and the output to become decoupled, which can be confusing for the user. Worse, it could be unnoticed by a user who is doing other things at the same time.

Due to typical Java overengineering, there is no straightforward way to simply (add-action-listener) on the JTextField and get notifications when it changes. Instead, each Swing widget implements full-blown model-view separation, and so you have to get the document underlying the JTextField. Yes, the view components of your MVC application themselves contain both models and views. Each and every field contains an entire implementation of the document object and associated cruft, all for itself. If you want to monitor changes in the widget, you need to build out a DocumentListener to watch for such events.

In a real application, I’d bite the bullet and implement it, but it’s not worth the extra time for the purposes of this demo, so we are left with a “Calculate” button and manual updating.

Clojure-Options on Github

The entire package so far is now available on my GitHub page. Patches are welcome. Atomizing the input widgets and adding the appropriate event callbacks and watches to eliminate the “Calculate” button is particularly welcome.

;;
;; Black-Scholes option pricing model
;;
;; Copyright (C) 2010 Paul Legato. All rights reserved.
;; Licensed under the New BSD License. See the README file for details.
;;
;; Disclaimer: This code comes with NO WARRANTY, express or implied.
;; There may be any number of bugs. Use at your own risk.
;;
;; References:
;; - https://secure.wikimedia.org/wikipedia/en/wiki/Black%E2%80%93Scholes
;; - http://www.espenhaug.com/black_scholes.html

(ns clojure-options.black-scholes
  (:require incanter.distributions))

(defn d1
  [spot timeleft strike riskfree sigma]
  (/
   (+ (Math/log (/ spot strike))
      (* (+ riskfree (/ (* sigma sigma) 2))
         timeleft))
   (* sigma (Math/sqrt timeleft))))

(defn d2
  [spot timeleft strike riskfree sigma]
  (- (d1 spot timeleft strike riskfree sigma) (* sigma (Math/sqrt timeleft))))

(defn- n [val] (incanter.distributions/cdf (incanter.distributions/normal-distribution) val))

(defn call
  "Returns the theoretic value of a European call option on the given underlying based on the Black-Scholes model.

 * spot - spot price of the underlying
 * timeleft - time to expiration, in years
 * strike - option strike price
 * riskfree - annual continuous risk-free interest rate
 * sigma - volatility of the underlying

"
  [spot timeleft strike riskfree sigma]
  (- (* spot (n (d1 spot timeleft strike riskfree sigma)))
     (* strike
        (Math/exp (* (- riskfree) timeleft))
        (n (d2 spot timeleft strike riskfree sigma)))))

(defn put
  "Returns the theoretic value of a European put option on the given underlying based on the Black-Scholes model.

 * spot - spot price of the underlying
 * timeleft - time to expiration, in years
 * strike - option strike price
 * riskfree - annual continuous risk-free interest rate
 * sigma - volatility of the underlying

"
  [spot timeleft strike riskfree sigma]
  (- (* strike
        (Math/exp (* (- riskfree) timeleft))
        (n (- (d2 spot timeleft strike riskfree sigma))))
     (* spot (n (- (d1 spot timeleft strike riskfree sigma))))))

(defn call-delta
  [spot timeleft strike riskfree sigma]
  (n (d1 spot timeleft strike riskfree sigma)))

(defn put-delta
  [spot timeleft strike riskfree sigma]
  (- (n (d1 spot timeleft strike riskfree sigma)) 1))

(defn- nprime [val] (incanter.distributions/pdf (incanter.distributions/normal-distribution) val))

(defn gamma
  [spot timeleft strike riskfree sigma]
  (/ (nprime (d1 spot timeleft strike riskfree sigma))
     (* spot sigma (Math/sqrt timeleft))))

(defn vega
  [spot timeleft strike riskfree sigma]
  (* spot (nprime (d1 spot timeleft strike riskfree sigma)) (Math/sqrt timeleft)))

(defn call-theta
  [spot timeleft strike riskfree sigma]
  (-
   (- (/ (* spot (nprime (d1 spot timeleft strike riskfree sigma)) sigma)
         (* 2 (Math/sqrt timeleft))))
   (* riskfree strike (Math/exp (* (- riskfree) timeleft)) (n (d2 spot timeleft strike riskfree sigma)))))

(defn put-theta
  [spot timeleft strike riskfree sigma]
  (-
   (- (/ (* spot (nprime (d1 spot timeleft strike riskfree sigma)) sigma)
         (* 2 (Math/sqrt timeleft))))
   (* riskfree strike (Math/exp (* (- riskfree) timeleft)) (n (- (d2 spot timeleft strike riskfree sigma))))))

(defn call-rho
  [spot timeleft strike riskfree sigma]
  (* strike timeleft (Math/exp (* (- riskfree) timeleft)) (n (d2 spot timeleft strike riskfree sigma))))

(defn put-rho
  [spot timeleft strike riskfree sigma]
  (* (- strike) timeleft (Math/exp (* (- riskfree) timeleft)) (n (- (d2 spot timeleft strike riskfree sigma)))))

This code uses the new Incanter Distributions module, which hasn’t been released yet, so it requires the latest master version from Github.

Most of the values agree with those produced at BloBek’s calculator, with the exception of theta. I imagine this has to do with the units used for the “days left to expiration” field; my calculator uses fractions of a year, while BloBek doesn’t specify whether his days are to be trading days or calendar days. It could also be a bug in one of our implementations.

In Part 2, I’ll build a simple Swing GUI for the calculator.
Update: Part 2, with the Swing GUI, is now available. You can also get the entire project’s source code from GitHub.

> (log/spy [1 2 3 4 5])
[1 2 3 4 5] ;;; prints " [1 2 3 4 5] => [1 2 3 4 5]" to the log
> (log/spy (take 3 [1 2 3 4 5]))
(1 2 3) ;;; prints "(take 3 [1 2 3 4 5]) => clojure.lang.LazySeq@8592" to the log

The spy macro logs a string realization of the sequence per se, rather than its constituents. In the case of a concrete, fully realized sequence such as the literal vector in the first example, all values are known in advance, and the vector’s string realization in the log can helpfully be its actual contents. In the case of the lazy sequence returned by take, however, we get a less than helpful class name and some sort of instance identifier code rather than the actual contents of the sequence. A lazy sequence saves resources by merely promising to provide the values when asked. If they’re never asked for, they’re never computed. Since they haven’t been computed, they can’t be printed. A lazy sequence can also potentially be infinitely long, another good reason why they are not realized by default when converting the sequence to a string.

In our case, we don’t care about the resource usage, and we’re sure the sequence is finite. We want to force all values to be computed for manual inspection. (There are several ways of going about this.) The following modification of the spy macro converts the lazy sequence into a regular seq:

(defmacro lazy-spy
  "Evaluates expr and outputs the form and its result to the debug log.
   If the result is a lazy sequence, it is realized concretely in the log.
   Returns the result of expr."
  [expr]
  `(let [a# ~expr] (log/log :debug (str '~expr " => "
                                    (if (= clojure.lang.LazySeq (class a#))
                                      (seq a#)
                                       a#)))
        a#))

This gives us:

> (lazy-spy  (take 3 [1 2 3 4 5]))
(1 2 3) ;;; prints "(take 3 [1 2 3 4 5]) => (1 2 3)" to the log

Clojure posts you might also like:

• Log4J and Clojure Configuration
• Setting Clojure’s Log Level
• SQL WHERE clauses in Clojure from S-Expressions
• XML DTD Validation in Clojure: Turning It Off, Parsing Malformed XML

1. If using Leiningen, add

   :dependencies [
                    [log4j "1.2.15" :exclusions [javax.mail/mail
                                                 javax.jms/jms
                                                 com.sun.jdmk/jmxtools
                                                 com.sun.jmx/jmxri]]
   ]
   

   to your project.clj file and rerun lein deps to auto-install the Log4J JAR.

   If not using Leiningen, acquire a Log4J JAR and put it on your classpath.

2. Create a file called log4j.properties on your classpath and put the following into it:

   # Based on the example properties given at http://logging.apache.org/log4j/1.2/manual.html
   # Set root logger level to DEBUG and its only appender to A1.
   log4j.rootLogger=DEBUG, A1
   
   # A1 is set to be a ConsoleAppender.
   log4j.appender.A1=org.apache.log4j.ConsoleAppender
   
   # A1 uses PatternLayout.
   log4j.appender.A1.layout=org.apache.log4j.PatternLayout
   log4j.appender.A1.layout.ConversionPattern= %-5p %c - %m%n
   

That’s the short version. Read on for the gory details.

Logging philosophy

My logging requirements are simple at the moment. I want to write messages to the console, each tagged with a log level (“INFO”, “WARN”, “ERROR”, etc.) I want to be able to set a log level, and only see log messages of that severity or higher. That’s it. I take this to be the canonical base case of logging that covers 80% of needs. All other logging features (e.g. logging to a file, differentiating by class or namespace) are nice, but much less common, and ought to be handled via configurable deviations from this default case.

clojure.contrib.logging does not presently make it easy to do any of this common base case out of the box. The Lispy façade quickly falls apart; you’re dumped into the deep end of the Java overengineering mire without a life vest when you simply want to change the log level temporarily.

What’s wrong with clojure.contrib.logging’s defaults?

clojure.contrib.logging with no special configuration done prefaces each and every logged line on my system with:

[null] May 16, 2010 1:08:48 PM clojure.contrib.logging$eval__2077$impl_write_BANG___2088 invoke

I have no idea why. This makes it very hard to read the log when logging any nontrivial amount of information.

clojure.contrib.logging’s documentation contains a rather cryptic offhanded remark, with no further explanation: Note: your log configuration should display the name that was passed to the logging implementation, and not perform stack-inspection, otherwise you’ll see something like “fn__72$impl_write_BANG__39__auto____81″ in your logs.

My log configuration? Stack inspection? Why do I need to have a log configuration to just dump strings to the console? Why is the default case to print a verbose and useless header for each line? Isn’t this Javaesque mess — obscure 50 line XML configuration files just to get basic logging functionality working — exactly what we’re trying to avoid?

Log4J with clojure.contrib.logging configuration

Ah, well, there’s nothing for it, I suppose. Better just get a serious logging library and figure out how to configure it. The docs say that clojure.contrib.logging delegates to the first supported logging implementation that it finds. I am apparently using the builtin JDK logger by default, and Apache Commons’ logger is just another wrapper, so I added Log4J to Leiningen’s project.clj and it installed the jar for me.

Now, I get:

     [null] log4j:WARN No appenders could be found for logger (my-project-name).
     [null] log4j:WARN Please initialize the log4j system properly.

Followed by dead silence.

Hm.

I can only imagine that Log4j was never intended to be used without a custom configuration. Odd. Why people design software that comes without reasonable defaults pre-set to cover the common base case is beyond me. I fully understand that strange setups will require customized configuration, but “dumping strings to the console” does not seem like a bizarre edge case for a logging package. A default case of “send all log messages to the black hole unless configured to do otherwise” makes little sense.

So, how hard can it be to configure Log4J for this base case functionality? The “short introduction to log4j” is 19 pages long. Sigh. (The complete manual is proudly described as “over 200 pages.” I sometimes wonder if they overengineer Java libraries on purpose just to drive book sales.)

I do not want to know what a Nested Diagnostic Context is, or the precise semantic distinction between “Loggers,” “Appenders,” and “Layouts.” I don’t care about “Appender Additivity” and “Named Hierarchies,” and I certainly don’t want to write an XML configuration file. I Just want to log tagged strings to the console, and have them print without a bizarre autogenerated header line. That’s it. I fully appreciate that those things will be useful for some people — maybe even for me, when my project grows to become more complex. I have nothing against their existence; but for the base case, one should not need to know of it.

On page 9 of the “short” manual, after a boatload of irrelevant information, we have an answer: there is a class called BasicConfigurator with a static method that does reasonable initialization for logging to the console. We can just do (org.apache.log4j.BasicConfigurator/configure) and get that.

However, we are still getting log messages like:

     [null] 3702 [8633270@qtp-3149669-2] INFO my.namespace - log message

We have to learn something about Layouts now. The thread names generated by Clojure are apparently not very useful, so we’re going to turn them off. And while interesting when profiling, the number of milliseconds since program start is just going to be visual clutter 98% of the time, so I’m going to turn that off, too, until I actually need it.

The result is the log4j.properties file listed above. Just copy and paste it into your classpath. If you want more or different information in your logs, take a look at the PatternLayout API documentation and alter the ConversionPattern line according to your preferences.

The result:

     [null] INFO  my.namespace - log message

I’m still not sure where the [null] comes from, but overall, it’s very readable.

Clojure posts you might also like:

• Lazy-spy: Clojure logging/spy for lazy sequences
• Setting Clojure’s Log Level
• SQL WHERE clauses in Clojure from S-Expressions
• XML DTD Validation in Clojure: Turning It Off, Parsing Malformed XML

error: java.io.IOException: Server returned HTTP response code: 503 for URL: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd

It seems that the W3C blocked access to the DTDs two years ago, but Java still tries to load them by default anyway. The following allows clojure.xml to run without checking external DTDs:

(defn startparse-sax-non-validating [s ch]
  (.. (doto (. javax.xml.parsers.SAXParserFactory (newInstance))
       (.setValidating false)
       (.setFeature "http://apache.org/xml/features/nonvalidating/load-dtd-grammar" false)
       (.setFeature "http://apache.org/xml/features/nonvalidating/load-external-dtd" false)
       (.setFeature "http://xml.org/sax/features/validation" false)
       (.setFeature "http://xml.org/sax/features/external-general-entities" false)
       (.setFeature "http://xml.org/sax/features/external-parameter-entities" false))

       (newSAXParser) (parse s ch))))

Then you can simply (xml/parse "sourcefile.xml" startparse-sax-non-validating). This is not the ideal solution — ideally, we want to use a locally cached DTD — but it works well enough for one-off code. Read on for further information.

W3C’s malformed specification

The W3C webserver was being DDoSed by the massive number of XML parsers trying to download the DTDs constantly, so they just blocked everyone. This is a direct result of the poorly considered technical specification for DTD identification and URIs themselves, then exacerbated by the W3C’s institutional culture and philosophy.

Technical commentary

The root of the problem is, as they put it, that “these [DTD URIs] are not hyperlinks; these URIs are used for identification” (emphasis in original.) This is an abuse of the URL format. If it’s “for identification,” then it doesn’t need to specify a transport protocol! They are relying upon their particular interpretation of the old URI/URL debate, reducing the transport protocol element to a namespace identifier instead, and then assuming everyone will defer to their judgement and wisdom on the matter.

The W3C (and others who share their URL/URI interpretation) decided to conflate the semantic functions of locating and transporting a resource with the very distinct function of identifying a resource. “My name” is not the same thing as an instruction to “get in your car, drive to my street address, pick up someone with my name here, and then drive me back to your house,” even though there is indeed only one person with my name at this address. If you want to identify me, you would not give someone the latter instruction. Most likely, they didn’t think about this at all and just did it. Now everyone suffers for this poor design decision. They are being DDoSed and paying massive amounts of money to keep the servers going, and everyone’s XML parsers are broken because they’re now returing 503s for all the DTD URIs.

The W3C’s proposed solution is that everyone else should a) defer to their views on the URL/URI matter, and 2) rewrite their XML parsers to be massively more complex and build local caching DTD catalogs for every single one-off parse. In practice, most people will just turn DTD validation off, potentially breaking a lot of other things.

Institutional commentary on the W3C itself

The tone of naive incredulity in the W3C’s plea is most amazing of all. They seem genuinely baffled as to why anyone would ever choose to ignore their wisdom on the URL/URI matter, and even more confused as to how anyone could ignore any detail of a baroque and poorly communicated specification like those for XML and DTD cataloging. “We spent a long time writing that, and we have [the self-appointed] institutional authority to write specifications and make determinations like what a URI should be, so surely people would never just ignore us!”

For them, the only conceivable explanation for every detail of the specification not being implemented is sheer ignorance. They even gently chide the readers about their failure to use URIs properly and to build DTD caches into everything, much as a kindergarten teacher would a well-meaning but rather dumb child who continually ignores instructions about how to use his crayons. They think that they only have to patiently remind people enough times that there is indeed a right way to do it, that which is sanctioned by the appropriate institutional authorities (them, of course) — and finally everything will be fine once people get that message.

This is the sort of ingenuous attitude that can only develop in a pure research institute and among people who mostly have very high power distance index scores. The high PDI explains their naive arrogation of global authority and utterly genuine assumption that everyone ought to just do as they say, simply because they’re the (self-)designated authority. The research institute setting means that they’re shut off from the economic real world. Since money is never a big issue, the worship of artificially created fancy titles and institutional authority can become rampant, and it’s perfectly fine to spend extended amounts of time parsing a 2 page XML document “the right way” rather than 15 minutes hacking it and moving on to more pressing problems so your company doesn’t go bankrupt in the interim.

Moreover, they are still intransigently defending the original poor design choice to use URIs with embedded transport protocols as unique identifiers, because they (as an institution) cannot admit the possibility that this was a poor choice. That would create a great deal of cognitive dissonance vis-à-vis their conception of the nature of an institution and its role in dictating what is correct and what is not in the world; their view that institutional authority must be right, because, well, otherwise it wouldn’t be an institution. After all, everyone there does have more fancy titles than most people who are not there.

These attitudes are pervasive throughout most W3C work, and are the main reason that real-world implementations ignore much of their work.

Parsing malformed HTML

So, in any case, I have another problem besides Java trying to load the unavailable DTD. The input HTML is malformed. Not just a little; it’s utterly broken. It’s both externally-generated and unreplaceable, so fixing it at the source is not an option. I’m stuck with it; it’s the only way to get the data I need. (That’s real-world programming as opposed to academia right there.)

Disabling XML validation

Disabling all XML validation is the obvious solution to both problems, but there is no readily apparent way to do that clojure.xml. Calling setValidating(false) on the factory is insufficient. Thanks to Stack Overflow, I found that you must also use
setFeature() with several obscure “SAX2 Standard Feature Flags“, as above.

HTML Tidy to the rescue

This still leaves the problem of the malformed HTML. There does not seem to be any easy way to make Sax ignore errors. ((.setFeature "http://apache.org/xml/features/continue-after-fatal-error" true) has no apparent effect; it still doesn’t continue after a fatal parse error.)

In the end, I just ran the malformed input HTML through HTML Tidy before attempting to parse it. HTML Tidy coerces the malformed input into valid XML, and Sax is finally happy.

(I now get “Exception in thread "Swank REPL Thread" java.lang.RuntimeException: java.lang.IllegalMonitorStateException” from SLIME when I try to actually display the parsed XML in Emacs, but that’s another matter…)

The Psyleron system, unfortunately, costs hundreds of dollars and only runs on Windows, so I wrote some quick Clojure to do experimentation with randomness, using /dev/random as the source. (It appears that /dev/random on Mac OS X does not provide the same quality guarantees as it does on Linux, but it’s fine for a prototype and the implementation allows easy substitution of another randomness source later.)

Clojure random walk generator

(ns randomwalker.core
  (:use [incanter core charts]))

(defn random-walk
  "Returns a random walk of length steps. random-fn should return a random integer."
  [steps list random-fn]
  (if (= 1 steps)
    list
    (recur (dec steps)
           (conj list
                   (+
                    (if (odd? (random-fn)) 1 -1)
                    (or (last list) 0)))
           random-fn)))

(defn dev-random-walk
  "Returns a random walk of length steps based on random numbers read from /dev/random."
  [steps]
  (with-open [random (java.io.FileInputStream. "/dev/random")]
    (random-walk steps [0] (fn [] (.read random)))))

(defn view-walk
  "Visualize a random walk."
  [steps]
  (view (incanter.charts/xy-plot (range 0 (dec steps)) (dev-random-walk steps))))

(defn walk-ratio
  "Generates the given number of random walks and returns the ratio of (walks ending > 0) / (walks ending < 0)."
  [runs steps-per-run greater less]
  (if (zero? runs)
    (double (/ greater less))
    (let [final (last (dev-random-walk steps-per-run))
          ;foo (println final)
          new-greater (if (> final 0) (inc greater) greater)
          new-less (if (< final 0) (inc less) less)]
      (recur (dec runs) steps-per-run new-greater new-less))))

This code generates a simple random walk. The user can easily visualize the random walk with Incanter, and it looks suspiciously like a stock chart.

Screenshot of the Clojure Random Walker in action

Psyleron type functionality first vaguely appears in walk-ratio. This generates the requested number of random walks and returns the ratio of those that ended above zero to those that ended below. I imagine this will become the basis for some Psyleron Reflector style games and experiments.

This is probably not the most efficient way to implement this; any suggestions are welcome. Any ideas for a better source of randomness? Any insight into how the Psyleron games work? Their descriptions are a bit short on the details.

(defmacro sql-expand
  "Transforms nested s-expressions into SQL, for use in WHERE or HAVING clauses.

e.g.: (sql-expand (and (> foo 3)
                       (< bar 4)))

     -> '(foo > 3 AND bar < 4)'

 Nested clauses:

      (sql-expand (or (and (> foo 3) (< bar 4))
                      (> baz 6)))

      -> '((foo > 3 AND bar < 4) OR baz > 6)'

 Embedded arbitrary SQL:
         (sql-expand (and (> foo 3)
                          (< bar \"(SELECT max(foo) + 10 FROM bar)\")))
      -> '(foo > 3 AND bar < (SELECT max(foo) + 10 FROM bar))'

"
  [form]
  (let [head (first form)]
    (if (includes? '(and or) head)
      `(str "(" (sql-expand ~(second form)) " "
            ~(.toUpperCase (str head)) " "
            (sql-expand ~(last form)) ")")
      (str (second form) " " head " " (last form)))))

Generating SQL from Lisp has been approached in a few different ways:

CL-SQL version

CL-SQL uses reader macros to do something similar, but with square brackets and new syntax:

;; From the CL-SQL docs
(sql [select [foo] [bar] :from [baz]] 'having [= [foo id] [bar id]]
     'and [foo val] '< 5)
=> "SELECT FOO,BAR FROM BAZ HAVING (FOO.ID = BAR.ID) AND FOO.VAL < 5"

I like my version better as it works with regular S-expressions. No new syntactic constructs are necessary, so you can re-use the parsers already set up in your brain for quick and easy mental code scanning and construction.

Some people mentioned that you have to jump through a lot of hoops to make sure that CL-SQL’s reader macros are activated and deactivated correctly around your code. Apparently it’s the only major Common Lisp package that actually uses reader macros extensively. (Clojure doesn’t support user-provided reader macros anyway, at least not without messy unsupported hackery.)

Cynojure’s functional version

Cynojure has a similar approach to mine:

;; From Cynojure's blog
 :where (sql-and (sql-= 'role +role-programmer+)
                                 (sql-< 'age 30)))

Cynojure uses special sql-* function names instead of transforming an s-expression with a macro. Since and and or are already in use by Clojure itself, he had to use other names. Symbols naming SQL fields (role and age, in this case) must also be quoted. This adds a lot of unnecessary noise to the signal and, as he notes, “interrupts the flow” of the code when reading.

By virtue of not automatically evaluating its arguments, a macro-based version can re-use already reserved symbols like and and or, and the user doesn’t have to bother with messily quoting symbols. This results in much cleaner and more readable code. The user-supplied data for the macro version contain 100% signal and no noise.

Chouser’s recursive function within a macro version

Chouser has a different macro-based approach involving a recursive anonymous function:

;; By Chouser, from http://groups.google.com/group/clojure/msg/62437f3b96835b58
(defmacro where [e]
  (let [f (fn f [e]
            (if-not (list? e)
              [(str e)]
              (let [[p & r] e]
                (if (= p `unquote)
                  r
                  (apply concat (interpose [(str " " p " ")]
                                           (map f r)))))))]
    (list* `str "where " (f e))))

His macro allows variable interpolation with quasiquotes, which is really cool. For example, you can do:

;; Chouser's examples
user=> (where (and (> i (- 3 1)) (< i ~(+ 3 1))))
"where i > 3 - 1 and i < 4"
user=> (let [my-name "'chouser'"] (where (and (> id 0) (= name ~my-name))))
"where id > 0 and name = 'chouser'"

This version also avoids the inelegant repetition of the SQL template at lines 25-28 in my version, at the price of slightly more complexity.

The only problem is that it doesn’t write grouping parentheses in the output SQL string, which discards potentially important grouping information embedded in the s-expression tree and leaves evaluation grouping order to the SQL server:

(where (or (<= baz 123)
           (and (> foo 3) (> bar 6))))
-> "baz <= 123 or foo > 3 and bar > 6"

My version

By being aware of the and and or list heads as special case operators which join other clauses together, my version can wrap their output in parentheses, which preserves the evaluation priority implicit within the input S-expression in the final SQL output. It seems that there should be a way to do the same thing both generically and recursively (that is, without repetition of similar code and without conditional recursion), but I can’t make that work without making the macro significantly longer and more complex (read: fragile), so for the moment, I’m going to live with simple and slightly repetitious.

One problem lies in knowing when to add parentheses. Although bluntly wrapping the result of every SQL operator and outputting SQL like “((foo < 3) AND (bar > 4))” will execute fine on the SQL server, the extra parentheses are superfluous. Ideally, we want to use parentheses only when strictly necessary to preserve the semantic intent of the S-expression tree, not blindly wrap every single expression.

Some sort of awareness of clause-combinatory SQL operators such as AND and OR as special cases is therefore necessary to know when to add parentheses in the output, but I don’t like having to repeat what’s essentially the same almost identical output template. In all cases, we have 3 elements, and we’re just swapping the first two and then taking a string output.

The other issue is that the AND/OR case needs to recurse, but the ordinary case doesn’t; the ordinary case can be translated directly to a string. The logic would have to be adjusted to either differentiate between these cases and recurse conditionally, or else recurse unconditionally and then have a way to deal with the base case of translating non-trinary lists. That is, with unconditional recursion, a foo symbol will eventually be fed into the macro by itself and must return just the string "foo", which complicates the management of the SQL string template significantly by forcing you to have multiple templating procedures and then decide between them. (Well, we’re doing that anyway now, but that would be even worse.)

I messed around with conditionally interpolateing the template result into a '("(" ")") list, and with conditionally stringifying directly or else building a trinary list based on the structure of the input, but the result was ugly, far longer and far more complicated and confusing than just repeating the template. Any suggestions?

I also haven’t figured out the best way to add unquote support without making a huge mess. That would be a very cool feature. Any ideas on that are also welcome.

Someday, this macro could be a small part of a user-transparent Clojure object persistence library like Common Lisp’s Elephant. That would truly be a killer feature for the Clojure language.

Suppose that you’re working at a REPL and you have namespaces a, b, and c, where c :uses both a and b (meaning that it imports their namespace contents into its own namespace.) You write a function foo in a, but then later you decide to move it to b. Bam, you get a namespace collision exception the next time you reload c.

The symbol foo in c’s namespace is already bound to your code in memory. Simply deleting the source code from a’s source file is not enough.

Of course, you could be brutish and simply kill your Clojure REPL process and restart it, but who wants to go through all that? We need a way to unbind the obsolete symbol in a. The solution is simple: just

(ns-unmap 'a 'foo)

and the symbol is removed from package a’s namespace. (This is called undef or unintern in other Lisp dialects.)

Edit: Since we’ve :used a from c, there will also now be a foo symbol in the c namespace, and we must also (ns-unmap 'c 'foo). Thanks Meikel!

JDBC (at least, the PostgreSQL driver) can’t use Joda Time directly (without explicit type mapping). One way to convert a Joda LocalDate into something JDBC can use:

(defn to-sql-date [date]
  "Convert any Joda-readable date object (including a string) to a java.sql.Date"
  (java.sql.Date. (.. (LocalDate. date) toDateMidnight toInstant getMillis)))

It’s not pretty, but it works. You can follow a similar procedure for any of the Joda classes.

;;; This version works when (impl-get-log "") returns an org.apache.commons.logging.impl.Jdk14Logger
(use 'clojure.contrib.logging)
(defn set-log-level! [level]
  "Sets the root logger's level, and the level of all of its Handlers, to level.
   Level should be one of the constants defined in java.util.logging.Level."
  (let [logger (.getLogger (impl-get-log ""))]
    (.setLevel logger level)
    (doseq [handler (.getHandlers logger)]
      (. handler setLevel level))))

;;; This version works when (impl-get-log "") returns a java.util.logging.LogManager$RootLogger
(use 'clojure.contrib.logging)
(defn set-log-level! [level]
  "Sets the root logger's level, and the level of all of its Handlers, to level.
   Level should be one of the constants defined in java.util.logging.Level."
  (let [logger (impl-get-log "")]
    (.setLevel logger level)
    (doseq [handler (.getHandlers logger)]
      (. handler setLevel level))))

This log level setting function works with a standard out-of-the-box clojure.contrib.logger on my system — depending on what logging libraries it finds on your classpath, as per the docs, it may not work for you. In particular, you need to be using clojure.contrib.logger to wrap an Apache Commons Logging instance, which in turn wraps a java.util.logging instance. This is the way my system works without any configuration; YMMV. Hopefully, something like this will be assimilated into a universal wrapper in the next clojure.contrib.logging.

For the gory details of how this was constructed…

Changing log levels

I needed to log some data in my Clojure app. I looked through clojure-contrib and found clojure.contrib.logging. It provides a unified logging wrapper for Apache commons-logging, log4j, and java.util.logging.

The spy macro looks particularly useful for debugging intermediate functions without complicating their code with a bunch of logging cruft: just wrap them in a single tiny spy call, and you’re set.

Unfortunately, spy logs at the :debug level, and my logger by default seems to be set to :info level. No problem, I thought, I’ll just increase the log level to :debug… Hm… Turns out that there’s no simple way to do that.

clojure.contrib.logging presently includes nothing at all for setting the debug level, in fact, a curious omission of obvious and very basic functionality. So we dig.

Setting the log level must be done in the underlying logging library, since there’s no clojure.commons.logging wrapper. Looking at *impl-name*, it seems that I’m using org.apache.commons.logging.

Apache Commons Logging

OK, great. Let’s take a look at the Commons Logging docs. The Commons Logging User Guide sounds like a good place to start. Surely changing the log level is a very common task that will be prominently addressed there… Perhaps even in the Quick Start guide.

Hmm, nope, nothing about changing the log level, but there is a somewhat disturbing reference to “a simple commons-logging.properties file”, conjuring up long-ago memories of Java overengineering and bloat — an entire configuration file, just to temporarily change the log level?? Surely not… Also mildly disturbingly, there are references to “the underlying logging system”. Commons Logging seems to be, itself, just an abstraction around some other logging system. So I’m using two layers of logging wrappers now.

Well, let’s see what Clojure thinks we’re using:

> (log/impl-get-log "user")
#<Jdk14Logger org.apache.commons.logging.impl.Jdk14Logger@13e168>

Now we’re getting somewhere. Let’s look at Jdk14Logger’s JavaDocs: the getLogger() – “Return the native Logger instance we are using” method looks promising.

java.util.logging.Logger

> (.. (log/impl-get-log "user") getLogger)
#<Logger java.util.logging.Logger@1a32f73>

java.util.logging.Logger! Aha! Let’s see what we can do with that… “The log level … may also be dynamically changed by calls on the Logger.setLevel method.” Finally!

This being Java, setLevel() naturally requires an entire object to wrap the integer specifying the log level, with the integer in turn standing in for what we would call a symbol in Lisp (and being referenced symbolically through constants in the Level class.) Ugh. And confusingly, the log level constants in that class are named FINE, FINER, FINEST, INFO, OFF, SEVERE, and WARNING, which do not correspond to the :debug, :info, :warn, :error levels used by Clojure and Apache Commons Logging.

Let’s see what we’re at now:

user> (use 'clojure.contrib.logging)
nil
user> (warn "foo") ; -> WARNING: foo
nil
user> (info "foo") ; -> INFO: foo
nil
user> (debug "foo") ; nothing printed
nil
user> (.. (impl-get-log "user") getLogger getLevel)
nil
;;
;; Nil? WTF?
;;
user> (.. (impl-get-log "user") getLogger )
#<Logger java.util.logging.Logger@197952>
;;
;; Yep, we're using the right object...
;;
user> (.. (impl-get-log "user") getLogger getName)
"user"
;;
;; and we have the logger for the correct namespace...
;; it has a getParent() method - maybe it inherits?
;;
user> (.. (impl-get-log "user") getLogger getParent)
#<RootLogger java.util.logging.LogManager$RootLogger@1c2df08>
user> (.. (impl-get-log "user") getLogger getParent getLevel)
#<Level INFO>
;;
;; AHA. it does.
;;

So we’ve finally tracked down where the log level is set. According to the Javadocs, a logger with a null log level will inherit its parent’s log level, and changing the parent with setLevel() will also affect any children that are inheriting. The docs tell us that we can get the root logger by asking for the “” (empty string) name, so we can simply modify that:

(defn set-log-level! [level]
"Sets the root logger level."
  (.setLevel (.getLogger (log/impl-get-log "")) level))

But all is not well:

user> (.. (impl-get-log "") getLogger getLevel)
#<Level FINE>
user> (debug "foobar") ;; prints nothing
nil
user> (set-log-level! java.util.logging.Level/FINER)
nil
user> (debug "foobar") ;; prints nothing
nil
user> (set-log-level! java.util.logging.Level/FINEST)
nil
user> (debug "foobar") ;; prints nothing
nil
user> (set-log-level! java.util.logging.Level/ALL)
nil
user> (debug "foobar") ;; prints nothing
nil

Err… ok. Maybe that’s the wrong method? Maybe it’s not actually setting it?

user> (.. (impl-get-log "user") isDebugEnabled)
true
user> (.. (impl-get-log "") isDebugEnabled)
true
user> (log :debug "asdf") ;; prints nothing
nil
user> (debug "asdf") ;; prints nothing
nil
;;
;; Maybe printing directly to the Java object rather than going through contrib.logger will help?
;;
user> (. (impl-get-log "user") debug  "foo bar") ;; prints nothing
nil
user> (. (impl-get-log "") debug  "foo bar") ;; prints nothing
nil
user> (. (impl-get-log "user") warn  "foo bar") ;; prints "WARNING: foo bar"
nil

WTF.

Digging even further into the docs, we find:

On each logging call the Logger initially performs a cheap check of the request level (e.g. SEVERE or FINE) against the effective log level of the logger. If the request level is lower than the log level, the logging call returns immediately.

After passing this initial (cheap) test, the Logger will allocate a LogRecord to describe the logging message. It will then call a Filter (if present) to do a more detailed check on whether the record should be published. If that passes it will then publish the LogRecord to its output Handlers. By default, loggers also publish to their parent’s Handlers, recursively up the tree.

Great, more Filters of some kind (beyond the log level check filter), a LogRecord object to wrap a string, and an inherited tree of Handlers. All to write a simple level-bound log message to STDERR. (And this is the simple logging library. I shudder to think of log4j’s internals.)

user> (..  (impl-get-log "") getLogger getFilter)
nil
user> (..  (impl-get-log "user") getLogger getFilter)
nil
user> (..  (impl-get-log "") getLogger getHandlers)
#<Handler[] [Ljava.util.logging.Handler;@23f019>
user> (first (..  (impl-get-log "") getLogger getHandlers))
#<ConsoleHandler java.util.logging.ConsoleHandler@829d91>
user> (.. (first (..  (impl-get-log "") getLogger getHandlers)) getLevel)
#<Level INFO>
user>

So... in the "simple" logger package, we have two. seperate. levels. of filtering on the log level, which are entirely uncoordinated. And no mention in the documentation of what exactly these "Filters" and "Handlers" do, or that they can also discard log messages. Certainly no mention at the logger's setLevel() method docs that your setting might be overruled by a "Handler" in some other part of the object tree. (Now, why did I ever stop coding in Java, again...?)

The working function to set the log level

user> (defn set-log-level! [level]
  "Sets the root logger's level, and the level of all of its Handlers, to level."
  (let [logger (.getLogger (impl-get-log ""))]
    (.setLevel logger level)
    (doseq [handler (.getHandlers logger)]
      (. handler setLevel level))))
#'user/set-log-level!
user> (set-log-level! java.util.logging.Level/ALL)
nil
user> (debug "foo bar") ;; FINALLY prints "FINE: foo bar". Not quite what I wanted ("FINE"?), but close enough.
nil
user> (spy (str "abc" "def")) ;; -> " FINE: (str "abc" "def") => abcdef". cool.
"abcdef"

So there you have it. You can now adjust your clojure.contrib.logger log level at runtime — at least for clojure.contrib.logger wrapping Apache Commons Logging wrapping a java.util.logging logger. No doubt log4j requires its own unique set of black magic. I hope a future version of clojure.contrib.logger will include a universal loglevel-setting function — I’ll gladly donate this code to it.