Project requirements and background

We want a fast and efficient way to download thousands of symbols’ worth of historic stock data from Yahoo Finance. Parallel HTTP queries and persistent connections are, therefore, required.

Technomancy’s clojure-http-client is beautifully written idiomatic Clojure, but unfortunately it does not yet support connection pooling. Needless to say, the built-in Java standard library HTTP client don’t seem to have it, either. I thought of libcurl, of course, but its Java bindings (and older bindings that the official site says to avoid) do not seem to be particularly complete or maintained. Googling turned up a HTTPClient last updated in 2001, a commercial package, and the free Apache HttpClient project, which seems to be reasonably active and to do almost everything that the Oakland Software commercial package does (the comparison on their website seems to have been written in 2003, when they had several features that Apache lacked, but which have since been implemented.) Oakland Software’s client has pipelining, which none of the others do, but that alone isn’t worth the price.

I went with Apache HttpClient. I figured that it would be easier to use than libcurl’s Java bindings (ha!) as it presented a higher-level user interface. (It turns out that whatever ease-of-use gain there was to be had there is more than compensated for by its confused documentation and general overengineered Java coding style.)

Apache HttpClient’s docs are a little out of date. I found an example of multithreaded parallel HTTP requests and the JavaDocs and went from there.

Clojure code

I began by translating the general outline of the Java example to Clojure, though it pained me to see such imperativeness in parentheses. I made a few updates: I changed it to use the Executor threading system rather than creating threads manually, and of course I used a Clojure function instead of a thread (since all Clojure functions implement Callable, which is really convenient.) I also used Joda-Time to replace the obscene GregorianCalendar/Date classes that come with Java.

The latest version of this code is available from GitHub.

;; clojure-yahoo-finance: Clojure interface to Yahoo! Finance historic stock data.
;;
;; Copyright (C) 2010 Paul Legato. All rights reserved.
;; Licensed under the BSD-new license: see the file LICENSE for details.
;;
;; This code is not endorsed by or connected with Yahoo in any way.
;;
(ns clojure-yahoo-finance.core
  (:import (org.apache.http
            HttpResponse
            HttpVersion
            client.HttpClient
            client.methods.HttpGet
            conn.scheme.PlainSocketFactory
            conn.scheme.Scheme
            conn.params.ConnManagerParams
            conn.params.ConnPerRouteBean
            conn.scheme.SchemeRegistry
            impl.client.DefaultHttpClient
            impl.conn.tsccm.ThreadSafeClientConnManager
            params.BasicHttpParams
            params.HttpConnectionParams
            params.HttpProtocolParams
            protocol.BasicHttpContext
            util.EntityUtils
            )
           (java.util.concurrent Executors)
           (org.joda.time LocalDate)
           ))

(defn- yahoo-request-thread
  "Internal function used as the thread to retrieve historic data for symbol from startDate to endDate via httpClient.

   The value will be either a map with the symbol name and the raw CSV data from Yahoo, or an HTTP status code (e.g. 404 means the symbol was not found.)

   Dates can be of any class that Joda-Time understands (including a 'YYYY-MM-DD' String.)
"
  [httpClient symbol startDate endDate]
   (let [
         start (LocalDate. startDate)
         end (LocalDate. endDate)

         httpGet (new HttpGet
                      (str "http://itable.finance.yahoo.com"
                           "/table.csv?s=" symbol "&g=d" +
                           "&a=" (- (.getMonthOfYear start) 1)
                           "&b=" (.getDayOfMonth start)
                           "&c=" (.getYear start)
                           "&d=" (- (.getMonthOfYear end) 1)
                           "&e=" (.getDayOfMonth end)
                           "&f=" (.getYear end)))

         ]
     (try
      (let [httpResponse  (. httpClient execute httpGet (new BasicHttpContext))
            entity (. httpResponse getEntity)
            data (if (= 200 (.. httpResponse getStatusLine getStatusCode))
                   (. EntityUtils toString entity)
                   404)]
        (.consumeContent entity) ;; required to release the connection back to the pool? Docs are unclear on whether EntityUtils does this itself.
        {symbol data}
        )
      (catch Exception e
        (.abort httpGet)
        (throw e)))))

(defn blocking-query
  "Returns historic data between startDate and endDate for the seq of symbols.

Options:
  :maxConnections - default 200.
  :threadPoolSize - default 300. Should be larger than maxConnections
                    for optimal efficiency. Set to 0 for an unbounded thread pool - be
                    advised that doing so may exhaust available memory if you're requesting a lot of symbols
                    relative to the amount of free memory and thread-creation overhead on your system.

The optimal values for these parameters are system-dependent.

"
  [startDate endDate symbols & options]
  (let [opts (when options (apply assoc {} options))

        params (new BasicHttpParams)
        schemeRegistry (new SchemeRegistry)

        results {}

        threadPoolSize (or (:threadPoolSize opts) 300)
        threadPool (if (= threadPoolSize 0)
                     (Executors/newCachedThreadPool)
                     (Executors/newFixedThreadPool threadPoolSize))
        ]

    ;; Wow, this is ugly. Now I remember why I stopped using Java.
    (. HttpProtocolParams setVersion params HttpVersion/HTTP_1_1)
    (. HttpProtocolParams setUserAgent params "Clojure-Yahoo-Finance")

    (. HttpConnectionParams setTcpNoDelay params true)

    ;; If these are turned too low, then you will get timeouts when running very large queries, because
    ;; all threads are started at once and each will block until it is allocated an HTTP connection from
    ;; the pool.
    (. HttpConnectionParams setConnectionTimeout params 100000)
    (. ConnManagerParams setTimeout params 100000)

    ;; These methods are marked deprecated in 4.1-alpha1, but their replacements don't work.
    (. ConnManagerParams setMaxConnectionsPerRoute params (new ConnPerRouteBean (or (:maxConnections opts) 200)))
    (. ConnManagerParams setMaxTotalConnections params (or (:maxConnections opts) 200))

    (. schemeRegistry register (new Scheme "http" (. PlainSocketFactory getSocketFactory) 80))

    (let [connectionManager (new ThreadSafeClientConnManager params schemeRegistry)
          httpClient (new DefaultHttpClient connectionManager params)
          ]

      ;; Broken in 4.1-alpha1 - need to use the deprecated method above instead
      ;;
      ;;(doto connectionManager
      ;; (.setMaxTotalConnections (or (:maxConnections opts) 200)))
      ;;        (.setDefaultMaxPerRoute (or (:maxConnections opts) 200))) 

      (try
       (let [tasks (map (fn [sym]
                          (fn [] (yahoo-request-thread httpClient sym startDate endDate)))
                        symbols
                        )
             futures (map (fn [x] (.get x)) (.invokeAll threadPool tasks))]
         (reduce conj futures))

         (finally
          (.. httpClient getConnectionManager shutdown)
          (.shutdown threadPool)))
       ))))

Notes

• The optimal values for :maxConnections and :threadPoolSize depend on how much free RAM is available to the Clojure process, how much memory overhead is involved in creating a new thread (JVM/OS dependent), and the latency characteristics of the network link between the user and Yahoo. The defaults should be reasonable for most modern desktop computers and broadband network environments, but those with access to particularly fast network pipes and with lots of RAM to spare may be able to get better performance by tweaking them. Note that you have to give java the -Xmx flag at startup to actually make more free RAM available to the JVM.
• Although using ConnManagerParams to set the maximum connections (lines 106-7) is marked deprecated in the JavaDocs for 4.1-alpha1, their replacements (lines 118-120) don’t actually do anything, so I had to use the deprecated methods instead. This took me a while to figure out, since of course this isn’t mentioned anywhere in the docs.
• The docs are inconsistent and vague about how an HTTP connection gets returned to the thread pool. Compounding the problem is that many of the docs seem to have been written for version 3.x, where it was apparently required to manually release connections. One doc mentions that 4.x automatically returns connections to the pool, another says you need to call either abort on the connection or consumeContent on an entity (included at line 60, just to be safe.)

No refs

My first version passed a ref to each thread and had it update it itself with its results That works fine, but I suspected that refs are not the most efficient way to go about this, as they incur transaction overhead in each thread. Profiling proved this theory out. Downloading one month of AAPL with the refs version above took about 0.60 seconds on average. I rewrote it to simply return the result from the thread rather than store it in a ref itself. The calling function then aggregates the results of all threads with reduce. This version gives us a slight performance boost, averaging around 0.45 seconds. (Subsequent additional optimizations boosted performance further, to the 0.24 seconds in the chart below.)

pmap / reduce version

A guy on IRC pointed me to pmap/reduce as a more Clojureish way of doing parallel calculation aggregation. This looks very interesting and is on the list for future projects.

Profiling

Tests were repeated 10 times, and the runtimes were averaged.

Clojure-yahoo-finance is quite a bit slower than the Ruby/Typhoeus version, but not so much slower as to render it unusable for most purposes. I suspect this is mainly because Typhoeus uses libcurl, written in C and heavily optimized over many years for speed, for all of its HTTP processing, whereas the Clojure version is using the pure-Java Apache HttpClient, replete with all the usual overengineered cruft one associates with Java, such as 5 million typed helper setter methods for generic “configuration options,” plus a helper class to actually set the 2 configuration options that exist, and Bean objects that do nothing but wrap an integer.

The next step is to figure out the Java bindings for libcurl and write a Clojure version with that. Stay tuned.

Paul Dix’s Typhoeus appeared while Googling today. Not only is it an awesomely fast wrapper for libcurl, it allows you to queue requests and then execute them concurrently.

Update: I’ve just released a Ruby gem based on this code. It’s on Gemcutter, and the source is on GitHub. You can install it by just doing a “sudo gem install yahoofinance-typhoeus“.

Why the YahooFinance gem is slow: Net::HTTP, serial connections

YahooFinance uses the Net::HTTP classes internally, which are extremely slow (the former helpfully summarizing his inspection of the code as “Ruby’s Net::HTTP implementation blows”.) There are several alternatives mentioned in those tests. I’ve settled on Typhoeus for now.

Typhoeus uses libcurl to do the heavy lifting. Libcurl is robust and well-tuned after years of real-world use. While I don’t know if it’s the fastest Ruby HTTP library available, it’s orders of magnitude better than Net::HTTP, and it’s Good Enough™ for now.

Net::HTTP is undoubtely far and away the largest single performance bottleneck in YahooFinance. There are other potential issues, but they are comparatively minor and could all easily be avoided during the conversion to Typhoeus anyway. For example…

Yahoo Finance (the actual web service, not the gem) apparently won’t return more than 200 days of historic data for some foreign stocks (according to the comments in the gem), so YahooFinance the gem includes chunking logic to retry and aggregate requests for more than that amount of data. This minorly impacts performance. I won’t ever be requesting more than 200 days in the production system (which downloads new data nightly) anyway, and I have very few foreign stocks. I suppose this will require revisiting and implementing chunking logic when I bootstrap a new server, but for this phase of testing, this is not an issue.

1. An entirely new TCP socket to Yahoo was opened and then torn down for each and every symbol, incurring that overhead every time. Libcurl uses persistent connections by default. (I’m not sure whether Typhoeus takes advantage of this functionality – it’s Fast Enough that I haven’t looked into it.) I imagine this is the major issue, since the amount of data transferred per request is not large enough to exceed even the 1k buffer in Ruby 1.8.6, much less the 16k one in 1.8.7.
2. The chunking logic discussed above.
3. YahooFinance (the gem) was then creating an entire new HistoricalQuote object for every symbol to pass back to my code. It apparently has an array-based option for better performance, but it was only slightly more trouble to refactor to Typhoeus than to that, so I took the opportunity to ditch Net::HTTP.
4. Not only does it use Net::HTTP, it uses Net::HTTP::Proxy. This is probably only a marginal slowdown, but on the other hand it sure wasn’t making it any faster.
5. YahooFinance.gem uses CSV.parse to parse the raw historic data sent back by Yahoo. Probably not all that big of a deal, but again, it was certainly not helping either. Why bother? I just split on \n, loop on that, and split on comma instead. I’m going to loop on it anyway to insert it into the database. Why loop twice?

Typhoeus code conversion

Typhoeus’ callback model required some minor modifications to my data loader design. My previous system was a rapid prototype designed for quick coding rather than performance (premature optimization being the root of all evil, as the sage said.) In short, I grabbed the list of stock symbols from the database and looped on them, doing a YahooFinance historic data request for each and stuffing the result back into the database. This worked fine for 20 or 30 or 100 symbols, but not so well with 10,000 symbols.

The overall logic with Typhoeus is similar: loop on the stock list, but instead of grabbing the data for each, queue a request with Typhoeus’ hydra dispatcher. Once all the requests have been queued, a blocking call to hydra.run begins the concurrent HTTP downloads.

Typhoeus does not seem to have a non-blocking way to run the HTTP queue. A further optimization would fork off a new thread to do the blockng run call in the background while still processing the database stock list in the main thread, then join the Hydra thread after all requests for all symbols have been queued. However, Ruby uses green threads – might this be an issue for running the C-based libcurl that powers Typhoeus under the hood? In any case, the code is now fast enough that I don’t have to worry about this optimization right now.

Yahoo Finance with Typhoeus Code

Here is a demo of the Yahoo Finance with Typhoeus code. I haven’t included app-specific parts for getting the stock symbols from my database or storing data from Yahoo Finance into the database. It should be easily adaptable to other applications. I’m thinking of writing a little wrapper around this and publishing it as a Ruby Gem.

Update: Gem now available at Gemcutter, and source on GitHub. You can install it with “gem install yahoofinance-typhoeus“.

#!/usr/bin/env ruby
#
# Load historic data for the given symbols for the last 14 calendar
# days. This will return varying amounts of data depending on how many
# trading days there were within the last 14 calendar days at the time
# you run this.
#
# Copyright (C) 2010 Paul Legato. License to use this code is granted
# under the terms of the GNU Public License version 2.
#

require 'rubygems'
require 'typhoeus'
require 'date'

Symbols = [ 'AAPL', 'GOOG', 'NonexistentSymbol', 'INTC', ]

Today = Date.today
StartDate = Today - 14

hydra = Typhoeus::Hydra.new(:max_concurrency => 20)
hydra.disable_memoization # save memory, since we won't ever use the memo

Symbols.each{|symbol|
  puts "Querying for #{ symbol }..."

  url = "http://itable.finance.yahoo.com" +
  "/table.csv?s=#{ symbol }&g=d" +
  "&a=#{ StartDate.month - 1 }&b=#{ StartDate.mday }&c=#{ StartDate.year }" +
  "&d=#{ Today.month - 1 }&e=#{ Today.mday }&f=#{ Today.year.to_s }"

  request = Typhoeus::Request.new(url, :method => :get)

  request.on_complete {|response|
    if response.code == 200
      if response.body[0..40] != "Date,Open,High,Low,Close,Volume,Adj Close"
        puts " * Error: Unknown response body from Yahoo - #{ response.body[0..40] } ..."
      else
        # good response. go.
        count = 0

        response.body.split("\n").each{|line|
          next if line[0..40] == "Date,Open,High,Low,Close,Volume,Adj Close" # skip header line

          # There's no point splitting it if we're just going to print it
          # back out as CSV, but this is a demo. You would typically be
          # stuffing this into a database here.
          data = line.split(",")
          puts("#{ symbol }," +
               "#{ data[0] }," + # date
               "#{ data[1] }," + # open
               "#{ data[2] }," + # high
               "#{ data[3] }," + # low
               "#{ data[4] }," + # close
               "#{ data[5] }," + # volume
               "#{ data[6] }") # adjusted close

          count += 1
        } # end each line of response

        puts "* #{ symbol } - loaded #{ count } days." if count > 0
      end
    elsif response.code == 404
      puts "#{ symbol } - 404 not found"
    else
      puts " * Error communicating with Yahoo: #{ url } - #{ response.inspect }"
    end

  } # end on_complete callback

  hydra.queue(request)

} # end each row from database

# All desired connections are now queued, so execute them.
puts "Starting hydra..."
hydra.run
puts "Hydra done."

YahooFinance-Typhoeus gem demo

With the gem, the demo is dead simple. First, gem install yahoofinance-typhoeus. Then just:

 require 'rubygems'
 require 'yahoofinance-typhoeus'

 yf = YahooFinance.new # or YahooFinance.new(max_number_of_concurrent_connections)

 yf.add_query("AAPL", "2008-12-01", "2008-12-15") {|response| puts response}
 yf.add_query("IBM", "2008-10-15", "2008-10-30") {|response| puts response}

 yf.run

For a quick, one-off query, it’s even simpler (although the quick_query method does not make use of Typhoeus’s parallelism):

require 'rubygems'
require 'yahoofinance-typhoeus'

 YahooFinance.quick_query("AAPL", "2009-01-01", "2009-02-01")

You can even do it from the command line and then pipe the output to awk or less or whatever. (Note that you can’t use -r to load rubygems due to a bug/feature in the Ruby interpreter).

ruby -rubygems -e "require 'yahoofinance-typhoeus' ; puts YahooFinance.quick_query('AAPL', '2009-01-01', '2009-02-01')" |less