iPhone Pros

• iPhones are all (almost) exactly the same. 4th generation iPhones do support an optional higher screen resolution than previous models, but they’re 100% binary backwards compatible. This is a trivial difference when put up against the dozens of different Android devices available with radically different hardware, and the hoops that you as a developer have to jump through to make sure your app runs correctly on all of them.
• Large, high-end user base that demonstrably spends money on apps. iPhone users have already shown their willingness to buy large numbers of apps; Android users, not so much. This seems to be slowly changing over time as the Android app store matures, but for now, iPhone is still the cash cow.
• Beautiful and well-designed UI and system architecture. iPhones have been designed as an integrated and coherent experience. Apple’s renowned mastery of intuitive human-computer interaction is evident at every step while using the device. There is an overall (and well documented) unifying strategic vision of how different subsystems should fit together seamlessly and present a unified interface to the user. Their development libraries have been designed to support this theme at every level. Android, by contrast, reads much more like KDE or Gnome; it was bolted together out of disparate pieces written by different groups working in isolation with minimal coordination. Perceived technical coolness from the perspective of a software engineer who is already intimately familiar with the system’s internals took precedent over usability. Android’s user interface is generally much uglier and much less intuitive.
• iPhones support low latency audio. Android uses ALSA internally, which does support low-latency audio, but the low-latency API is not exposed to developers, and Google does not seem to be interested in fixing this. Android latency is unacceptably high for realtime sound synthesis unless you hack it and use the ALSA API, which is officially forbidden and in any case could change significantly and without warning between Android operating system updates and between phones. iPhone, on the other hand, has an officially supported low latency audio API, and there are many successful realtime sound synthesizer apps in the app store.
• XCode. XCode is a really well done IDE. I normally don’t like IDEs. I normally use Emacs for everything. But, in all fairness, I decided to give XCode and Eclipse with the Android plugins a shot. Eclipse is nice, but a little rough around the edges. I ran into a few bizarre IDE error messages while developing my Android test projects that required extensive Googling and hours of experimentation to unravel. XCode, as one would expect from a flagship Apple product, is immaculately polished and completely integrated from top to bottom.

iPhone cons

• Objective C. While Objective C was really cool and innovative in 1988, today it seems archaic. Nobody uses it except Apple. It’s laden with tons of cruft. It still feels like it’s a bolt-on afterthought preprocessor for a C compiler. For example:
  • Selectors: you can’t refer to method names directly; you have to use the special @selector() compiler directive to look up their index number. This probably made great sense when CPUs were very slow and doing a lookup in an ASCII table was a significant performance hit over doing an integer offset jump, but on modern computers, this is just annoying. CPU time is cheaper than engineer time.

  • Manual reference-counted memory management is an obsolete and extremely fragile pain in the ass. Lisp had GC ~40 years ago. The vast set of problems introduced by manual memory management, such as buffer overflows and memory leaks, is just not worth the small gain in efficiency that you get from not periodically running a garbage collector. Again, CPU time is cheaper than engineer time. Android does GC on a phone just fine, so I don’t buy the argument that a phone can’t support that kind of overhead. Garbage collection was recently introduced for the Mac OS X desktop system, but is as of yet not available on the iPhone.

    (I’m aware that C and C++ are widely used and both still make you do manual memory management, too. Far from regarding this as a vehicle for a macho display of programming prowess, I find manual memory management to be more of an annoyance, something that the computer should be handling for me automatically, much like how figuring out the exact memory address of a function was automated long ago and is no longer something we as application-level programmers ever need to worry about much.)

  • The use of + and - to mark methods as class or instance methods. Principle of least surprise; why not just use, say, the English word class or static or something to that effect to designate class methods, and leave instance methods unmarked? Perhaps using 1 character symbols rather than entire English words as the token improved compile times on large, complex projects in the ’80s, but the time savings is infinitesimal on a modern computer, so I’ll vote for obviousness and clarity rather than crypticity.
  • The use of [] (square brackets) for message passing, and the requirement for named (but non-reorderable) arguments.

    Perhaps this was done out of a well-intentioned effort to make the programmer notice that these are messages and not function calls or C++ methods, and to make the code self-documenting. I find the introduction of new syntax rather unwarranted and distracting. I’d much prefer the familiar object.message(argument1, argument2) over the verbose and unwieldly [object message:argument1 parameterName:argument2]. You can’t even re-order the named arguments, so including the argument name is just noise in the code. I guess I’ve been spoiled by Clojure and Lisp, where there’s almost no syntax.

    If they really felt strongly about the need to pedantically differentiate messages from function calls or methods at the syntax level, they should at least have retained the general function/method syntactic template and chosen a different separator character, so as to enable programmers to re-use most of their neural syntax parsing circuitry rather than force us to introduce an entirely new syntactic template that performs only 1 task. For example, a more intuitive syntax that still emphasizes the fact that this is not a function or method call could be something like object#message(argument1, argument2) or object<-message(argument1, argument2).

• Closed platform. The source code to the vast Cocoa library is proprietary. You can develop with things Apple tells you you may develop with, and nothing else. They only recently relaxed the restriction on using any third party tools at all. The iPhone's CPU is widely reported to have built-in hardware level Java bytecode support, but that's off-limits.
• Vendor lock-in. This is a huge and obvious minus for the iPhone. Your ability to generate revenue is 100% dependent upon Apple's whims, upon remaining in Apple's good graces. Uncountable horror stories circulate online about developers who invested months and many thousands of dollars into a project only to have it rejected by Apple, often after a considerable delay and with little or no explanation.

Android pros

• Java (almost). Android apps are written in Java. (Well, technically not in Java due to licensing issues, but in a language that is 99.9% the same as Java.) If you already know Java, as I do, this is a big plus. You can easily re-use your code elsewhere, on any platform, with minor or no modifications. Objective C code is rather limited in what you can do with it: you can re-use it in your Apple desktop apps. (Yes, there is a free GNU Objective C compiler, but almost all real-world Objective C code is locked into Apple's proprietary Cocoa runtime system.)
• A JVM, almost. Android doesn't run a true JVM, unfortunately, as part of a licensing dispute between Google and Sun, but it's close. Android's Dalvik is a cleanroom reimplementation of part of Java, designed to avoid triggering what Google regards as onerous licensing fees for the use of a true JVM on a mobile handset. In any case, it's close enough that the door is open to potentially running Clojure and other JVM languages on it someday.
• No upfront developer fees. Develop for free. You can download the SDK and go nuts, upload your code to a real handset to test, send your apps to the app store for sale, whatever. There is no $99 developer tax.
• Larger ultimate user base. Android phones are now outselling iPhones, and this trend looks set to continue. The price you pay is coding for 200 slightly different handsets.
• Open source platform. Most of the Android system is freely available, precluding vendor lock-in, at least in theory.

Android cons

• Android emulator lacks hardware accelerated graphics. This is a big minus. I made a very simple bouncing-ball demo with Rokon that got about 6 or 7 fps in the emulator. Full games often take multiple seconds per frame, completely unusable. If you want to write games (or anything else where graphical performance is key), you absolutely need to have a hardware handset from the start. If you don't already own an Android phone anyway, this means you have to invest hundreds of dollars upfront, just to see if the platform is for you.
• Lack of interest from Google. Google has many, many pots cooking at once. While they do devote significant resources to Android's maintenance and expansion, it's nowhere near as important a product for them as the iPhone is for Apple, and you can tell. The iPhone is a core strategic product for Apple, whereas Android is a relatively minor satellite project for Google. Correspondingly, Apple's general support of the iPhone and its developers seems far better.
• Relatively few libraries. For the moment, there seem to be more third party libraries available for the iPhone, and they seem to be more mature and robust than what's out there for Android. This will change in time, as more and more app developers begin to use Android.

It was a close call. I did some exploratory coding on both platforms — there's no substitute for direct experience. Though I much prefer Java over Objective C as a language, the universe of extant Android libraries is much smaller and less mature. I may have been able to live with that in order to be able to code in Java and avoid Apple's vendor lock-in. In the end, the tipping point was practical: the Android emulator's lack of hardware accelerated graphics left me with an unusable framerate, so I'd have to spend hundreds of dollars on a handset to do even trivial demo development, whereas I already own an iPhone.

I'll be checking back in on Android periodically, as I expect its long-term position will improve substantially in time, but for now, I'm going with the iPhone.

The GUI is written entirely in Clojure with the Swing toolkit. Calculation state is stored in a series of atoms. Watches are used to update the output table automatically when an atom changes, an idea from Kotka. I used the excellent MiG Layout for general layout functionality, and generic Swing widgets (JTextFields and a JTable) for the input and output.

Clojure code

The Black-Scholes GUI is launched with themain function. The Swing GUI is actually created in the initialize-gui function, but since Swing widgets are not threadsafe, we must use the do-swing* helper function from Clojure Contrib to spin off the real GUI creation code into the Swing event thread.

(defn initialize-gui
  []
  (let [frame (JFrame. "Black-Scholes Option Modeler")]

    (doto frame
      (.setDefaultCloseOperation JFrame/DISPOSE_ON_CLOSE)
      (-> .getContentPane
          (.add (miglayout (JPanel.)

                           (input-panel
                            (fn [_]
                              (do-swing
                               (doto frame
                                 (.setVisible false)
                                 (.dispose)))))

                           (result-table)

                           )))

      (.setDefaultCloseOperation JFrame/DISPOSE_ON_CLOSE)
      (.pack)
      (.setVisible true))
     ))

(defn main
  []
  (do-swing* :now initialize-gui))

This is pretty straightforward. We make a (locally scoped) JFrame to be the main container at line 3. Within the context of the let, we add a JPane using MiG Layout to its content area (starting at line 8). The JPanel itself contains two subwidgets, which are those returned from the input-panel and result-table functions.

Note that we are passing an anonymous function to input-panel at line 11. The input panel contains the “Quit” button, and that button needs to know what to do when it’s pressed. We want it to close the GUI; that is, it should destroy the outer JFrame. To do that, it needs a reference to it. However, as a local variable in initialize-gui, the JFrame is not in scope elsewhere; we have to pass it in. The given function closes over the JFrame reference (line 13) and transports it to input-panel, where that function will be called as an event listener when the “Quit” button is pressed.

We could also have simply passed the bare JFrame rather than a function, and moved the frame-closing code into input-panel, but that would imply that the input panel must always be in a JFrame, which will not necessarily the case. Passing in an external anonymous function to be activated by the “Quit” button allows the input panel to be used generically in any sort of container, with the container deciding what to do when “Quit” is pressed.

The Input Panel

(defn- input-panel [quit-action]
    (let [
          ;; Fields
          spot (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                 (.setColumns 6)
                 (.setValue @*spot*))

          strike (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                   (.setColumns 6)
                   (.setValue @*strike*))

          days-till-expiry (doto (JFormattedTextField. (NumberFormat/getIntegerInstance))
                             (.setColumns 6)
                             (.setValue @*days-till-expiry*))

          riskfree (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                     (.setColumns 6)
                     (.setValue @*riskfree*))

          volatility (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                       (.setColumns 6)
                       (.setValue @*volatility*))

          ;; Buttons
          calculate (JButton. "Calculate")
          quit (JButton. "Quit")
          ]

      (add-action-listener
       calculate
       (fn [_]
         (update-all-calculations
          (Double/parseDouble (.getText spot))
          (/ (Integer/parseInt (.getText days-till-expiry)) *trading-days-per-year*)
          (Double/parseDouble (.getText strike))
          (Double/parseDouble (.getText riskfree))
          (Double/parseDouble (.getText volatility)))))

      (add-action-listener
       quit
       quit-action)

      (miglayout (JPanel.)
                 :layout  [:wrap 2 ]

                 (JLabel. "Spot") [:align "right"]
                 spot

                 (JLabel. "Strike") [:align "right"]
                 strike

                 (JLabel. "Risk-free rate") [:align "right"]
                 riskfree

                 (JLabel. "Volatility") [:align "right"]
                 volatility

                 (JLabel. "Days till expiry") [:align "right"]
                 days-till-expiry

                 calculate
                 quit)))

The input widgets are local variables in the input panel function, along with the two buttons. The widgets are initialized from the state atoms (which are then unused again; see discussion below.) The “Quit” button is given the event callback passed in as an argument (so it can manipulate the enclosing JFrame as per above.) The “Calculate” button is given an event callback that calls the update-all-calculations function with the values in the input widgets.

The lot are then stuffed into a JPanel with a MiG Layout and returned.

The Output Table

(defn- update-cell
  "Updates the given AbstractTableModel cell and fires a change event for that cell."
  [model data row col]
  (.setValueAt model data row col)
  (.fireTableCellUpdated model row col))

(defn- result-table []
  (let [
        column-names ["" "Call" "Put"]
        row-names ["Price" "Delta" "Theta" "Rho" "Gamma" "Vega"]
        table-model (proxy [AbstractTableModel] []
                      (getColumnCount [] (count column-names))
                      (getRowCount [] (count row-names))
                      (isCellEditable [] false)
                      (getColumnName [col] (nth column-names col))
                      (getValueAt [row col]
                                  (condp = col
                                      0 (nth row-names row) ;; Return the row name for column zero
                                      (condp = [row col]
                                          [0 1] @*call-price*
                                          [0 2] @*put-price*

                                          [1 1] @*call-delta*
                                          [1 2] @*put-delta*

                                          [2 1] @*call-theta*
                                          [2 2] @*put-theta*

                                          [3 1] @*call-rho*
                                          [3 2] @*put-rho*

                                          [4 1] @*gamma*
                                          [4 2] "=="

                                          [5 1] @*vega*
                                          [5 2] "=="

                                          nil))))

        table (doto (JTable. table-model)
                (.setGridColor java.awt.Color/DARK_GRAY))
        ]

    (add-watch *call-price* ::update-call-price (fn [_ _ _ newprice] (update-cell table-model newprice 0 1)))
    (add-watch *put-price* ::update-put-price (fn [_ _ _ newprice] (update-cell table-model newprice 0 2)))

    (add-watch *call-delta* ::update-call-delta (fn [_ _ _ newval] (update-cell table-model newval 1 1)))
    (add-watch *put-delta* ::update-put-delta (fn [_ _ _ newval] (update-cell table-model newval 1 2)))

    (add-watch *call-theta* ::update-call-theta (fn [_ _ _ newval] (update-cell table-model newval 2 1)))
    (add-watch *put-theta* ::update-put-theta (fn [_ _ _ newval] (update-cell table-model newval 2 2)))

    (add-watch *call-rho* ::update-call-rho (fn [_ _ _ newval] (update-cell table-model newval 3 1)))
    (add-watch *put-rho* ::update-put-rho (fn [_ _ _ newval] (update-cell table-model newval 3 2)))

    (add-watch *gamma* ::update-gamma (fn [_ _ _ newval] (update-cell table-model newval 4 1)))
    (add-watch *vega* ::update-vega (fn [_ _ _ newval] (update-cell table-model newval 5 1)))

    ;; This shrinks the table's preferred viewport down to its actual size.
    ;; (The default is to make a huge viewport, even though the table is small.)
    (.setPreferredScrollableViewportSize table (.getPreferredSize table))

    (JScrollPane. table)
    ))

The output table itself is a standard JTable. Its model is initialized from the output state atoms. The watches at lines 44-57 are the most interesting part; they allow arbitrary code to be called when the output state atoms change. This allows the Black-Scholes calculation to happen independently of the output widget. When one of those atoms is changed, it self-updates automatically via the watches.

The helper function at lines 1-5 updates the appropriate model cell and fires an event to tell the JTable to repaint it.

The scrollable viewport business at line 61 is to work around a questionable Java design decision where a JTable, by default, tells its container to make its viewport enormous, even though it have very little data. (That’s not an error, that’s subjunctive.)

Design considerations

GUI and business logic decoupling

Above all, the GUI has to be decoupled from the model itself. Internal changes to one should never affect the other, and the two should be independently testable. In such a simple application, this was easy enough. A more complex app that requires several information round trips would be trickier. We now have a good foundation for such an app. No business calculations are performed in the GUI code namespace. The sole point of connection is in the update-all-calculations function:

(defn- update-all-calculations [spot timeleft strike riskfree sigma]
  (swap! *call-price* (fn [_] (bs/call spot timeleft strike riskfree sigma)))
  (swap! *put-price* (fn [_] (bs/put spot timeleft strike riskfree sigma)))

  (swap! *call-delta* (fn [_] (bs/call-delta spot timeleft strike riskfree sigma)))
  (swap! *put-delta* (fn [_] (bs/put-delta spot timeleft strike riskfree sigma)))

  (swap! *call-theta* (fn [_] (bs/call-theta spot timeleft strike riskfree sigma)))
  (swap! *put-theta* (fn [_] (bs/put-theta spot timeleft strike riskfree sigma)))

  (swap! *call-rho* (fn [_] (bs/call-rho spot timeleft strike riskfree sigma)))
  (swap! *put-rho* (fn [_] (bs/put-rho spot timeleft strike riskfree sigma)))

  (swap! *vega* (fn [_] (bs/vega spot timeleft strike riskfree sigma)))
  (swap! *gamma* (fn [_] (bs/gamma spot timeleft strike riskfree sigma))))

Atoms to hold state

As we can see above, set of atoms holds state for the GUI, in keeping with the general Clojure principle of isolating state into transactional memory. (Actually, I cheated a bit. Our full state consists of the input values and the results of our Black-Scholes calculations. Although I created atoms for the input state values as well, I calculate the results directly from the widgets. It is a trivial exercise to alter the “Calculate” button callback to use the atoms. This would be more useful in the case where changes to a widget fire an event that triggers recalculation, which is not done in this demo as per below. I would use the atoms in a real application.)

I/O is inherently stateful, as are GUIs. There was therefore the temptation to use the Swing widget objects themselves as state-holders, and in fact it requires extra code to shuttle data back and forth between the widgets and the atoms. In such a simple app as this, using the GUI widgets as state-holders does not make much practical difference, but in a more complex app, a well designed set of state atoms and watches can automate much of the tedious GUI logic.

Using the widgets directly as state holders implies a tight coupling between the input widgets, the business logic, and the output widgets. Changing any one of them would invariably have required changes to the linking logic.

For example, suppose we were using widgets-as-state, and we changed the “days left till expiry” input to a slider rather than a text box, or suppose we wanted to read it from a file or from an API. We would have to update the code that reads it, calculates the model, and puts the results into the output table. Touching that code implies a non-zero likelihood of breaking it. Why even introduce the potential for breaking something that is unrelated to the task at hand, in this case the business logic and output code?

Again, this seems trivial with such a very small application that is only a few lines long, but the design principle is what’s important. Imagine applying it to a much larger application with hundreds of inputs spread across 4 windows, 3 databases, and 6 web feeds to see why this is a good idea.

Auto-updating results when input changes

I wanted the results to auto-update anytime an input is changed, without resorting to a “Calculate” button. Having the button trigger the calculation and output update allows the state of the input and the output to become decoupled, which can be confusing for the user. Worse, it could be unnoticed by a user who is doing other things at the same time.

Due to typical Java overengineering, there is no straightforward way to simply (add-action-listener) on the JTextField and get notifications when it changes. Instead, each Swing widget implements full-blown model-view separation, and so you have to get the document underlying the JTextField. Yes, the view components of your MVC application themselves contain both models and views. Each and every field contains an entire implementation of the document object and associated cruft, all for itself. If you want to monitor changes in the widget, you need to build out a DocumentListener to watch for such events.

In a real application, I’d bite the bullet and implement it, but it’s not worth the extra time for the purposes of this demo, so we are left with a “Calculate” button and manual updating.

Clojure-Options on Github

The entire package so far is now available on my GitHub page. Patches are welcome. Atomizing the input widgets and adding the appropriate event callbacks and watches to eliminate the “Calculate” button is particularly welcome.

1. If using Leiningen, add

   :dependencies [
                    [log4j "1.2.15" :exclusions [javax.mail/mail
                                                 javax.jms/jms
                                                 com.sun.jdmk/jmxtools
                                                 com.sun.jmx/jmxri]]
   ]
   

   to your project.clj file and rerun lein deps to auto-install the Log4J JAR.

   If not using Leiningen, acquire a Log4J JAR and put it on your classpath.

2. Create a file called log4j.properties on your classpath and put the following into it:

   # Based on the example properties given at http://logging.apache.org/log4j/1.2/manual.html
   # Set root logger level to DEBUG and its only appender to A1.
   log4j.rootLogger=DEBUG, A1
   
   # A1 is set to be a ConsoleAppender.
   log4j.appender.A1=org.apache.log4j.ConsoleAppender
   
   # A1 uses PatternLayout.
   log4j.appender.A1.layout=org.apache.log4j.PatternLayout
   log4j.appender.A1.layout.ConversionPattern= %-5p %c - %m%n
   

That’s the short version. Read on for the gory details.

Logging philosophy

My logging requirements are simple at the moment. I want to write messages to the console, each tagged with a log level (“INFO”, “WARN”, “ERROR”, etc.) I want to be able to set a log level, and only see log messages of that severity or higher. That’s it. I take this to be the canonical base case of logging that covers 80% of needs. All other logging features (e.g. logging to a file, differentiating by class or namespace) are nice, but much less common, and ought to be handled via configurable deviations from this default case.

clojure.contrib.logging does not presently make it easy to do any of this common base case out of the box. The Lispy façade quickly falls apart; you’re dumped into the deep end of the Java overengineering mire without a life vest when you simply want to change the log level temporarily.

What’s wrong with clojure.contrib.logging’s defaults?

clojure.contrib.logging with no special configuration done prefaces each and every logged line on my system with:

[null] May 16, 2010 1:08:48 PM clojure.contrib.logging$eval__2077$impl_write_BANG___2088 invoke

I have no idea why. This makes it very hard to read the log when logging any nontrivial amount of information.

clojure.contrib.logging’s documentation contains a rather cryptic offhanded remark, with no further explanation: Note: your log configuration should display the name that was passed to the logging implementation, and not perform stack-inspection, otherwise you’ll see something like “fn__72$impl_write_BANG__39__auto____81″ in your logs.

My log configuration? Stack inspection? Why do I need to have a log configuration to just dump strings to the console? Why is the default case to print a verbose and useless header for each line? Isn’t this Javaesque mess — obscure 50 line XML configuration files just to get basic logging functionality working — exactly what we’re trying to avoid?

Log4J with clojure.contrib.logging configuration

Ah, well, there’s nothing for it, I suppose. Better just get a serious logging library and figure out how to configure it. The docs say that clojure.contrib.logging delegates to the first supported logging implementation that it finds. I am apparently using the builtin JDK logger by default, and Apache Commons’ logger is just another wrapper, so I added Log4J to Leiningen’s project.clj and it installed the jar for me.

Now, I get:

     [null] log4j:WARN No appenders could be found for logger (my-project-name).
     [null] log4j:WARN Please initialize the log4j system properly.

Followed by dead silence.

Hm.

I can only imagine that Log4j was never intended to be used without a custom configuration. Odd. Why people design software that comes without reasonable defaults pre-set to cover the common base case is beyond me. I fully understand that strange setups will require customized configuration, but “dumping strings to the console” does not seem like a bizarre edge case for a logging package. A default case of “send all log messages to the black hole unless configured to do otherwise” makes little sense.

So, how hard can it be to configure Log4J for this base case functionality? The “short introduction to log4j” is 19 pages long. Sigh. (The complete manual is proudly described as “over 200 pages.” I sometimes wonder if they overengineer Java libraries on purpose just to drive book sales.)

I do not want to know what a Nested Diagnostic Context is, or the precise semantic distinction between “Loggers,” “Appenders,” and “Layouts.” I don’t care about “Appender Additivity” and “Named Hierarchies,” and I certainly don’t want to write an XML configuration file. I Just want to log tagged strings to the console, and have them print without a bizarre autogenerated header line. That’s it. I fully appreciate that those things will be useful for some people — maybe even for me, when my project grows to become more complex. I have nothing against their existence; but for the base case, one should not need to know of it.

On page 9 of the “short” manual, after a boatload of irrelevant information, we have an answer: there is a class called BasicConfigurator with a static method that does reasonable initialization for logging to the console. We can just do (org.apache.log4j.BasicConfigurator/configure) and get that.

However, we are still getting log messages like:

     [null] 3702 [8633270@qtp-3149669-2] INFO my.namespace - log message

We have to learn something about Layouts now. The thread names generated by Clojure are apparently not very useful, so we’re going to turn them off. And while interesting when profiling, the number of milliseconds since program start is just going to be visual clutter 98% of the time, so I’m going to turn that off, too, until I actually need it.

The result is the log4j.properties file listed above. Just copy and paste it into your classpath. If you want more or different information in your logs, take a look at the PatternLayout API documentation and alter the ConversionPattern line according to your preferences.

The result:

     [null] INFO  my.namespace - log message

I’m still not sure where the [null] comes from, but overall, it’s very readable.

Clojure posts you might also like:

• Lazy-spy: Clojure logging/spy for lazy sequences
• Setting Clojure’s Log Level
• SQL WHERE clauses in Clojure from S-Expressions
• XML DTD Validation in Clojure: Turning It Off, Parsing Malformed XML

error: java.io.IOException: Server returned HTTP response code: 503 for URL: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd

It seems that the W3C blocked access to the DTDs two years ago, but Java still tries to load them by default anyway. The following allows clojure.xml to run without checking external DTDs:

(defn startparse-sax-non-validating [s ch]
  (.. (doto (. javax.xml.parsers.SAXParserFactory (newInstance))
       (.setValidating false)
       (.setFeature "http://apache.org/xml/features/nonvalidating/load-dtd-grammar" false)
       (.setFeature "http://apache.org/xml/features/nonvalidating/load-external-dtd" false)
       (.setFeature "http://xml.org/sax/features/validation" false)
       (.setFeature "http://xml.org/sax/features/external-general-entities" false)
       (.setFeature "http://xml.org/sax/features/external-parameter-entities" false))

       (newSAXParser) (parse s ch))))

Then you can simply (xml/parse "sourcefile.xml" startparse-sax-non-validating). This is not the ideal solution — ideally, we want to use a locally cached DTD — but it works well enough for one-off code. Read on for further information.

W3C’s malformed specification

The W3C webserver was being DDoSed by the massive number of XML parsers trying to download the DTDs constantly, so they just blocked everyone. This is a direct result of the poorly considered technical specification for DTD identification and URIs themselves, then exacerbated by the W3C’s institutional culture and philosophy.

Technical commentary

The root of the problem is, as they put it, that “these [DTD URIs] are not hyperlinks; these URIs are used for identification” (emphasis in original.) This is an abuse of the URL format. If it’s “for identification,” then it doesn’t need to specify a transport protocol! They are relying upon their particular interpretation of the old URI/URL debate, reducing the transport protocol element to a namespace identifier instead, and then assuming everyone will defer to their judgement and wisdom on the matter.

The W3C (and others who share their URL/URI interpretation) decided to conflate the semantic functions of locating and transporting a resource with the very distinct function of identifying a resource. “My name” is not the same thing as an instruction to “get in your car, drive to my street address, pick up someone with my name here, and then drive me back to your house,” even though there is indeed only one person with my name at this address. If you want to identify me, you would not give someone the latter instruction. Most likely, they didn’t think about this at all and just did it. Now everyone suffers for this poor design decision. They are being DDoSed and paying massive amounts of money to keep the servers going, and everyone’s XML parsers are broken because they’re now returing 503s for all the DTD URIs.

The W3C’s proposed solution is that everyone else should a) defer to their views on the URL/URI matter, and 2) rewrite their XML parsers to be massively more complex and build local caching DTD catalogs for every single one-off parse. In practice, most people will just turn DTD validation off, potentially breaking a lot of other things.

Institutional commentary on the W3C itself

The tone of naive incredulity in the W3C’s plea is most amazing of all. They seem genuinely baffled as to why anyone would ever choose to ignore their wisdom on the URL/URI matter, and even more confused as to how anyone could ignore any detail of a baroque and poorly communicated specification like those for XML and DTD cataloging. “We spent a long time writing that, and we have [the self-appointed] institutional authority to write specifications and make determinations like what a URI should be, so surely people would never just ignore us!”

For them, the only conceivable explanation for every detail of the specification not being implemented is sheer ignorance. They even gently chide the readers about their failure to use URIs properly and to build DTD caches into everything, much as a kindergarten teacher would a well-meaning but rather dumb child who continually ignores instructions about how to use his crayons. They think that they only have to patiently remind people enough times that there is indeed a right way to do it, that which is sanctioned by the appropriate institutional authorities (them, of course) — and finally everything will be fine once people get that message.

This is the sort of ingenuous attitude that can only develop in a pure research institute and among people who mostly have very high power distance index scores. The high PDI explains their naive arrogation of global authority and utterly genuine assumption that everyone ought to just do as they say, simply because they’re the (self-)designated authority. The research institute setting means that they’re shut off from the economic real world. Since money is never a big issue, the worship of artificially created fancy titles and institutional authority can become rampant, and it’s perfectly fine to spend extended amounts of time parsing a 2 page XML document “the right way” rather than 15 minutes hacking it and moving on to more pressing problems so your company doesn’t go bankrupt in the interim.

Moreover, they are still intransigently defending the original poor design choice to use URIs with embedded transport protocols as unique identifiers, because they (as an institution) cannot admit the possibility that this was a poor choice. That would create a great deal of cognitive dissonance vis-à-vis their conception of the nature of an institution and its role in dictating what is correct and what is not in the world; their view that institutional authority must be right, because, well, otherwise it wouldn’t be an institution. After all, everyone there does have more fancy titles than most people who are not there.

These attitudes are pervasive throughout most W3C work, and are the main reason that real-world implementations ignore much of their work.

Parsing malformed HTML

So, in any case, I have another problem besides Java trying to load the unavailable DTD. The input HTML is malformed. Not just a little; it’s utterly broken. It’s both externally-generated and unreplaceable, so fixing it at the source is not an option. I’m stuck with it; it’s the only way to get the data I need. (That’s real-world programming as opposed to academia right there.)

Disabling XML validation

Disabling all XML validation is the obvious solution to both problems, but there is no readily apparent way to do that clojure.xml. Calling setValidating(false) on the factory is insufficient. Thanks to Stack Overflow, I found that you must also use
setFeature() with several obscure “SAX2 Standard Feature Flags“, as above.

HTML Tidy to the rescue

This still leaves the problem of the malformed HTML. There does not seem to be any easy way to make Sax ignore errors. ((.setFeature "http://apache.org/xml/features/continue-after-fatal-error" true) has no apparent effect; it still doesn’t continue after a fatal parse error.)

In the end, I just ran the malformed input HTML through HTML Tidy before attempting to parse it. HTML Tidy coerces the malformed input into valid XML, and Sax is finally happy.

(I now get “Exception in thread "Swank REPL Thread" java.lang.RuntimeException: java.lang.IllegalMonitorStateException” from SLIME when I try to actually display the parsed XML in Emacs, but that’s another matter…)

JDBC (at least, the PostgreSQL driver) can’t use Joda Time directly (without explicit type mapping). One way to convert a Joda LocalDate into something JDBC can use:

(defn to-sql-date [date]
  "Convert any Joda-readable date object (including a string) to a java.sql.Date"
  (java.sql.Date. (.. (LocalDate. date) toDateMidnight toInstant getMillis)))

It’s not pretty, but it works. You can follow a similar procedure for any of the Joda classes.

;;; This version works when (impl-get-log "") returns an org.apache.commons.logging.impl.Jdk14Logger
(use 'clojure.contrib.logging)
(defn set-log-level! [level]
  "Sets the root logger's level, and the level of all of its Handlers, to level.
   Level should be one of the constants defined in java.util.logging.Level."
  (let [logger (.getLogger (impl-get-log ""))]
    (.setLevel logger level)
    (doseq [handler (.getHandlers logger)]
      (. handler setLevel level))))

;;; This version works when (impl-get-log "") returns a java.util.logging.LogManager$RootLogger
(use 'clojure.contrib.logging)
(defn set-log-level! [level]
  "Sets the root logger's level, and the level of all of its Handlers, to level.
   Level should be one of the constants defined in java.util.logging.Level."
  (let [logger (impl-get-log "")]
    (.setLevel logger level)
    (doseq [handler (.getHandlers logger)]
      (. handler setLevel level))))

This log level setting function works with a standard out-of-the-box clojure.contrib.logger on my system — depending on what logging libraries it finds on your classpath, as per the docs, it may not work for you. In particular, you need to be using clojure.contrib.logger to wrap an Apache Commons Logging instance, which in turn wraps a java.util.logging instance. This is the way my system works without any configuration; YMMV. Hopefully, something like this will be assimilated into a universal wrapper in the next clojure.contrib.logging.

For the gory details of how this was constructed…

Changing log levels

I needed to log some data in my Clojure app. I looked through clojure-contrib and found clojure.contrib.logging. It provides a unified logging wrapper for Apache commons-logging, log4j, and java.util.logging.

The spy macro looks particularly useful for debugging intermediate functions without complicating their code with a bunch of logging cruft: just wrap them in a single tiny spy call, and you’re set.

Unfortunately, spy logs at the :debug level, and my logger by default seems to be set to :info level. No problem, I thought, I’ll just increase the log level to :debug… Hm… Turns out that there’s no simple way to do that.

clojure.contrib.logging presently includes nothing at all for setting the debug level, in fact, a curious omission of obvious and very basic functionality. So we dig.

Setting the log level must be done in the underlying logging library, since there’s no clojure.commons.logging wrapper. Looking at *impl-name*, it seems that I’m using org.apache.commons.logging.

Apache Commons Logging

OK, great. Let’s take a look at the Commons Logging docs. The Commons Logging User Guide sounds like a good place to start. Surely changing the log level is a very common task that will be prominently addressed there… Perhaps even in the Quick Start guide.

Hmm, nope, nothing about changing the log level, but there is a somewhat disturbing reference to “a simple commons-logging.properties file”, conjuring up long-ago memories of Java overengineering and bloat — an entire configuration file, just to temporarily change the log level?? Surely not… Also mildly disturbingly, there are references to “the underlying logging system”. Commons Logging seems to be, itself, just an abstraction around some other logging system. So I’m using two layers of logging wrappers now.

Well, let’s see what Clojure thinks we’re using:

> (log/impl-get-log "user")
#<Jdk14Logger org.apache.commons.logging.impl.Jdk14Logger@13e168>

Now we’re getting somewhere. Let’s look at Jdk14Logger’s JavaDocs: the getLogger() – “Return the native Logger instance we are using” method looks promising.

java.util.logging.Logger

> (.. (log/impl-get-log "user") getLogger)
#<Logger java.util.logging.Logger@1a32f73>

java.util.logging.Logger! Aha! Let’s see what we can do with that… “The log level … may also be dynamically changed by calls on the Logger.setLevel method.” Finally!

This being Java, setLevel() naturally requires an entire object to wrap the integer specifying the log level, with the integer in turn standing in for what we would call a symbol in Lisp (and being referenced symbolically through constants in the Level class.) Ugh. And confusingly, the log level constants in that class are named FINE, FINER, FINEST, INFO, OFF, SEVERE, and WARNING, which do not correspond to the :debug, :info, :warn, :error levels used by Clojure and Apache Commons Logging.

Let’s see what we’re at now:

user> (use 'clojure.contrib.logging)
nil
user> (warn "foo") ; -> WARNING: foo
nil
user> (info "foo") ; -> INFO: foo
nil
user> (debug "foo") ; nothing printed
nil
user> (.. (impl-get-log "user") getLogger getLevel)
nil
;;
;; Nil? WTF?
;;
user> (.. (impl-get-log "user") getLogger )
#<Logger java.util.logging.Logger@197952>
;;
;; Yep, we're using the right object...
;;
user> (.. (impl-get-log "user") getLogger getName)
"user"
;;
;; and we have the logger for the correct namespace...
;; it has a getParent() method - maybe it inherits?
;;
user> (.. (impl-get-log "user") getLogger getParent)
#<RootLogger java.util.logging.LogManager$RootLogger@1c2df08>
user> (.. (impl-get-log "user") getLogger getParent getLevel)
#<Level INFO>
;;
;; AHA. it does.
;;

So we’ve finally tracked down where the log level is set. According to the Javadocs, a logger with a null log level will inherit its parent’s log level, and changing the parent with setLevel() will also affect any children that are inheriting. The docs tell us that we can get the root logger by asking for the “” (empty string) name, so we can simply modify that:

(defn set-log-level! [level]
"Sets the root logger level."
  (.setLevel (.getLogger (log/impl-get-log "")) level))

But all is not well:

user> (.. (impl-get-log "") getLogger getLevel)
#<Level FINE>
user> (debug "foobar") ;; prints nothing
nil
user> (set-log-level! java.util.logging.Level/FINER)
nil
user> (debug "foobar") ;; prints nothing
nil
user> (set-log-level! java.util.logging.Level/FINEST)
nil
user> (debug "foobar") ;; prints nothing
nil
user> (set-log-level! java.util.logging.Level/ALL)
nil
user> (debug "foobar") ;; prints nothing
nil

Err… ok. Maybe that’s the wrong method? Maybe it’s not actually setting it?

user> (.. (impl-get-log "user") isDebugEnabled)
true
user> (.. (impl-get-log "") isDebugEnabled)
true
user> (log :debug "asdf") ;; prints nothing
nil
user> (debug "asdf") ;; prints nothing
nil
;;
;; Maybe printing directly to the Java object rather than going through contrib.logger will help?
;;
user> (. (impl-get-log "user") debug  "foo bar") ;; prints nothing
nil
user> (. (impl-get-log "") debug  "foo bar") ;; prints nothing
nil
user> (. (impl-get-log "user") warn  "foo bar") ;; prints "WARNING: foo bar"
nil

WTF.

Digging even further into the docs, we find:

On each logging call the Logger initially performs a cheap check of the request level (e.g. SEVERE or FINE) against the effective log level of the logger. If the request level is lower than the log level, the logging call returns immediately.

After passing this initial (cheap) test, the Logger will allocate a LogRecord to describe the logging message. It will then call a Filter (if present) to do a more detailed check on whether the record should be published. If that passes it will then publish the LogRecord to its output Handlers. By default, loggers also publish to their parent’s Handlers, recursively up the tree.

Great, more Filters of some kind (beyond the log level check filter), a LogRecord object to wrap a string, and an inherited tree of Handlers. All to write a simple level-bound log message to STDERR. (And this is the simple logging library. I shudder to think of log4j’s internals.)

user> (..  (impl-get-log "") getLogger getFilter)
nil
user> (..  (impl-get-log "user") getLogger getFilter)
nil
user> (..  (impl-get-log "") getLogger getHandlers)
#<Handler[] [Ljava.util.logging.Handler;@23f019>
user> (first (..  (impl-get-log "") getLogger getHandlers))
#<ConsoleHandler java.util.logging.ConsoleHandler@829d91>
user> (.. (first (..  (impl-get-log "") getLogger getHandlers)) getLevel)
#<Level INFO>
user>

So... in the "simple" logger package, we have two. seperate. levels. of filtering on the log level, which are entirely uncoordinated. And no mention in the documentation of what exactly these "Filters" and "Handlers" do, or that they can also discard log messages. Certainly no mention at the logger's setLevel() method docs that your setting might be overruled by a "Handler" in some other part of the object tree. (Now, why did I ever stop coding in Java, again...?)

The working function to set the log level

user> (defn set-log-level! [level]
  "Sets the root logger's level, and the level of all of its Handlers, to level."
  (let [logger (.getLogger (impl-get-log ""))]
    (.setLevel logger level)
    (doseq [handler (.getHandlers logger)]
      (. handler setLevel level))))
#'user/set-log-level!
user> (set-log-level! java.util.logging.Level/ALL)
nil
user> (debug "foo bar") ;; FINALLY prints "FINE: foo bar". Not quite what I wanted ("FINE"?), but close enough.
nil
user> (spy (str "abc" "def")) ;; -> " FINE: (str "abc" "def") => abcdef". cool.
"abcdef"

So there you have it. You can now adjust your clojure.contrib.logger log level at runtime — at least for clojure.contrib.logger wrapping Apache Commons Logging wrapping a java.util.logging logger. No doubt log4j requires its own unique set of black magic. I hope a future version of clojure.contrib.logger will include a universal loglevel-setting function — I’ll gladly donate this code to it.

Update: More recent Java bindings, which do not seem to be linked anywhere on the libcurl site, are available at http://www.gknw.net/viewvc/trunk/?root=curl-java. Still no multi support, though.

Project requirements and background

We want a fast and efficient way to download thousands of symbols’ worth of historic stock data from Yahoo Finance. Parallel HTTP queries and persistent connections are, therefore, required.

Technomancy’s clojure-http-client is beautifully written idiomatic Clojure, but unfortunately it does not yet support connection pooling. Needless to say, the built-in Java standard library HTTP client don’t seem to have it, either. I thought of libcurl, of course, but its Java bindings (and older bindings that the official site says to avoid) do not seem to be particularly complete or maintained. Googling turned up a HTTPClient last updated in 2001, a commercial package, and the free Apache HttpClient project, which seems to be reasonably active and to do almost everything that the Oakland Software commercial package does (the comparison on their website seems to have been written in 2003, when they had several features that Apache lacked, but which have since been implemented.) Oakland Software’s client has pipelining, which none of the others do, but that alone isn’t worth the price.

I went with Apache HttpClient. I figured that it would be easier to use than libcurl’s Java bindings (ha!) as it presented a higher-level user interface. (It turns out that whatever ease-of-use gain there was to be had there is more than compensated for by its confused documentation and general overengineered Java coding style.)

Apache HttpClient’s docs are a little out of date. I found an example of multithreaded parallel HTTP requests and the JavaDocs and went from there.

Clojure code

I began by translating the general outline of the Java example to Clojure, though it pained me to see such imperativeness in parentheses. I made a few updates: I changed it to use the Executor threading system rather than creating threads manually, and of course I used a Clojure function instead of a thread (since all Clojure functions implement Callable, which is really convenient.) I also used Joda-Time to replace the obscene GregorianCalendar/Date classes that come with Java.

The latest version of this code is available from GitHub.

;; clojure-yahoo-finance: Clojure interface to Yahoo! Finance historic stock data.
;;
;; Copyright (C) 2010 Paul Legato. All rights reserved.
;; Licensed under the BSD-new license: see the file LICENSE for details.
;;
;; This code is not endorsed by or connected with Yahoo in any way.
;;
(ns clojure-yahoo-finance.core
  (:import (org.apache.http
            HttpResponse
            HttpVersion
            client.HttpClient
            client.methods.HttpGet
            conn.scheme.PlainSocketFactory
            conn.scheme.Scheme
            conn.params.ConnManagerParams
            conn.params.ConnPerRouteBean
            conn.scheme.SchemeRegistry
            impl.client.DefaultHttpClient
            impl.conn.tsccm.ThreadSafeClientConnManager
            params.BasicHttpParams
            params.HttpConnectionParams
            params.HttpProtocolParams
            protocol.BasicHttpContext
            util.EntityUtils
            )
           (java.util.concurrent Executors)
           (org.joda.time LocalDate)
           ))

(defn- yahoo-request-thread
  "Internal function used as the thread to retrieve historic data for symbol from startDate to endDate via httpClient.

   The value will be either a map with the symbol name and the raw CSV data from Yahoo, or an HTTP status code (e.g. 404 means the symbol was not found.)

   Dates can be of any class that Joda-Time understands (including a 'YYYY-MM-DD' String.)
"
  [httpClient symbol startDate endDate]
   (let [
         start (LocalDate. startDate)
         end (LocalDate. endDate)

         httpGet (new HttpGet
                      (str "http://itable.finance.yahoo.com"
                           "/table.csv?s=" symbol "&g=d" +
                           "&a=" (- (.getMonthOfYear start) 1)
                           "&b=" (.getDayOfMonth start)
                           "&c=" (.getYear start)
                           "&d=" (- (.getMonthOfYear end) 1)
                           "&e=" (.getDayOfMonth end)
                           "&f=" (.getYear end)))

         ]
     (try
      (let [httpResponse  (. httpClient execute httpGet (new BasicHttpContext))
            entity (. httpResponse getEntity)
            data (if (= 200 (.. httpResponse getStatusLine getStatusCode))
                   (. EntityUtils toString entity)
                   404)]
        (.consumeContent entity) ;; required to release the connection back to the pool? Docs are unclear on whether EntityUtils does this itself.
        {symbol data}
        )
      (catch Exception e
        (.abort httpGet)
        (throw e)))))

(defn blocking-query
  "Returns historic data between startDate and endDate for the seq of symbols.

Options:
  :maxConnections - default 200.
  :threadPoolSize - default 300. Should be larger than maxConnections
                    for optimal efficiency. Set to 0 for an unbounded thread pool - be
                    advised that doing so may exhaust available memory if you're requesting a lot of symbols
                    relative to the amount of free memory and thread-creation overhead on your system.

The optimal values for these parameters are system-dependent.

"
  [startDate endDate symbols & options]
  (let [opts (when options (apply assoc {} options))

        params (new BasicHttpParams)
        schemeRegistry (new SchemeRegistry)

        results {}

        threadPoolSize (or (:threadPoolSize opts) 300)
        threadPool (if (= threadPoolSize 0)
                     (Executors/newCachedThreadPool)
                     (Executors/newFixedThreadPool threadPoolSize))
        ]

    ;; Wow, this is ugly. Now I remember why I stopped using Java.
    (. HttpProtocolParams setVersion params HttpVersion/HTTP_1_1)
    (. HttpProtocolParams setUserAgent params "Clojure-Yahoo-Finance")

    (. HttpConnectionParams setTcpNoDelay params true)

    ;; If these are turned too low, then you will get timeouts when running very large queries, because
    ;; all threads are started at once and each will block until it is allocated an HTTP connection from
    ;; the pool.
    (. HttpConnectionParams setConnectionTimeout params 100000)
    (. ConnManagerParams setTimeout params 100000)

    ;; These methods are marked deprecated in 4.1-alpha1, but their replacements don't work.
    (. ConnManagerParams setMaxConnectionsPerRoute params (new ConnPerRouteBean (or (:maxConnections opts) 200)))
    (. ConnManagerParams setMaxTotalConnections params (or (:maxConnections opts) 200))

    (. schemeRegistry register (new Scheme "http" (. PlainSocketFactory getSocketFactory) 80))

    (let [connectionManager (new ThreadSafeClientConnManager params schemeRegistry)
          httpClient (new DefaultHttpClient connectionManager params)
          ]

      ;; Broken in 4.1-alpha1 - need to use the deprecated method above instead
      ;;
      ;;(doto connectionManager
      ;; (.setMaxTotalConnections (or (:maxConnections opts) 200)))
      ;;        (.setDefaultMaxPerRoute (or (:maxConnections opts) 200))) 

      (try
       (let [tasks (map (fn [sym]
                          (fn [] (yahoo-request-thread httpClient sym startDate endDate)))
                        symbols
                        )
             futures (map (fn [x] (.get x)) (.invokeAll threadPool tasks))]
         (reduce conj futures))

         (finally
          (.. httpClient getConnectionManager shutdown)
          (.shutdown threadPool)))
       ))))

Notes

• The optimal values for :maxConnections and :threadPoolSize depend on how much free RAM is available to the Clojure process, how much memory overhead is involved in creating a new thread (JVM/OS dependent), and the latency characteristics of the network link between the user and Yahoo. The defaults should be reasonable for most modern desktop computers and broadband network environments, but those with access to particularly fast network pipes and with lots of RAM to spare may be able to get better performance by tweaking them. Note that you have to give java the -Xmx flag at startup to actually make more free RAM available to the JVM.
• Although using ConnManagerParams to set the maximum connections (lines 106-7) is marked deprecated in the JavaDocs for 4.1-alpha1, their replacements (lines 118-120) don’t actually do anything, so I had to use the deprecated methods instead. This took me a while to figure out, since of course this isn’t mentioned anywhere in the docs.
• The docs are inconsistent and vague about how an HTTP connection gets returned to the thread pool. Compounding the problem is that many of the docs seem to have been written for version 3.x, where it was apparently required to manually release connections. One doc mentions that 4.x automatically returns connections to the pool, another says you need to call either abort on the connection or consumeContent on an entity (included at line 60, just to be safe.)

No refs

My first version passed a ref to each thread and had it update it itself with its results That works fine, but I suspected that refs are not the most efficient way to go about this, as they incur transaction overhead in each thread. Profiling proved this theory out. Downloading one month of AAPL with the refs version above took about 0.60 seconds on average. I rewrote it to simply return the result from the thread rather than store it in a ref itself. The calling function then aggregates the results of all threads with reduce. This version gives us a slight performance boost, averaging around 0.45 seconds. (Subsequent additional optimizations boosted performance further, to the 0.24 seconds in the chart below.)

pmap / reduce version

A guy on IRC pointed me to pmap/reduce as a more Clojureish way of doing parallel calculation aggregation. This looks very interesting and is on the list for future projects.

Profiling

Tests were repeated 10 times, and the runtimes were averaged.

Clojure-yahoo-finance is quite a bit slower than the Ruby/Typhoeus version, but not so much slower as to render it unusable for most purposes. I suspect this is mainly because Typhoeus uses libcurl, written in C and heavily optimized over many years for speed, for all of its HTTP processing, whereas the Clojure version is using the pure-Java Apache HttpClient, replete with all the usual overengineered cruft one associates with Java, such as 5 million typed helper setter methods for generic “configuration options,” plus a helper class to actually set the 2 configuration options that exist, and Bean objects that do nothing but wrap an integer.

The next step is to figure out the Java bindings for libcurl and write a Clojure version with that. Stay tuned.

R-Java Bridge Motivation

I have some statistical functions written in R which would be annoying to re-implement in Java or Clojure, mainly because they depend on R libraries that are already written and for which there are no readily apparent Java/Clojure equivalents (the augmented Dickey-Fuller test, if you’re interested.) So, naturally, I decided to look into embedding R into Java, somehow.

Rincanter: R from Clojure

Rincanter came up, and it looks perfect: not only does it embed R into the JVM (via rJava‘s JRI component), it provides a nice Clojure wrapper and it integrates it with Incanter, a Clojure-based statistics and charting package that I’ve been wanting to try.

rJava and JRI: Low level bridge

rJava is, incidentally, mainly intended for calling Java code from R, which is the opposite of what I want to do. It absorbed the JRI project, which allows one to call R from Java, into its source tree a while ago, but so far it appears to be a direct graft onto the source tree with little or no integration. Perfect, since I don’t need to use the Java-from-R stuff anyway. So I downloaded rJava and built only the JRI subdirectory.

Going by Rincanter’s installation instructions, I need two things from JRIL: a generic JAR for the system-independent Java parts and a platform-specific JNI, a Java Native Interface to the actual R implementation on the system. The build produced two important files (hidden in the src/ subdirectory), JRI.jar and libjri.jnilib. (The name of the latter will vary depending on your build platform; it will be a .so file on Linux, a .dll file on Windows, etc.) I stuck them in my Clojure project’s lib/ subdirectory, restarted Leiningen, and went to the REPL to see whether everything worked.

Glancing through the examples that came with JRI, I decided to try a “(.new org.rosuda.JRI.Rengine)“. This failed spectacularly, and crashed the JVM with a complaint about being unable to locate the JNI library.

Loading JNI libraries with Leiningen

Further investigation showed that just placing the JNI library on the classpath doesn’t work. JNI libraries apparently have a separate classpath-like construct stored in the JVM system property java.library.path. Mine was set to ".:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java" by default.

I don’t want to install the JRI library system-wide. The default inclusion of "." was curious, but I also don’t want to start dumping files into my project’s root directory. What I need is a way to set this system property. Of course, I could be crude and just hammer it in in one of my Clojure source files, but, as it’s analogous to setting up the classpath, it seems rather like something that Leiningen should be doing as part of its config-file based JVM startup procedure.

Good news: Leiningen got the ability to set java.library.path last December. It’s still not documented in the README, but the code was merged into the official release. Searching through the codebase, I found that the defproject argument :native-path sets up JNI’s java.library.path. I made myself a jni/ subdirectory, added :native-path "jni" to my project.clj, and it worked.

It did clobber the default java.library.path property completely, replacing it with my “jni/” argument rather than appending it. That’s OK for my case, since I’m not using any systemwide JNI libraries, but it is something to be aware of in case you are.

Update: Rincanter apparently does its own management of java.library.path and JNI library loading. The docs ask for the .jnilib to be installed into Rincanter itself. It’s still good to know about the Leiningen way.