HowTo: projectM visualization + mp3 = video for YouTube

projectM (or its Windows-only predecessor, Milkdrop) can be used to make interesting visualizations for your music. Want to put your music on YouTube? You need to supply a video to accompany it. A projectM visualization makes an interesting and relatively easy accompaniment — far better than the typical static “picture of a CD cover” that most bands use:

This HowTo tells you what software you need and how to do it on your Mac. Continue reading

Android or iPhone Development?

Android or iPhone? Which platform should I target first? iPhone, for now, though it’s close. Android cannot (yet) provide the same level of developer tools or paying user base, the Android emulator can’t handle graphics, and I already own an iPhone. I do expect that Android’s competitive position will improve significantly over the next few years, but it’s not there yet. Continue reading

Dell PowerEdge BIOS Upgrade on Linux

Dell only provides certain PowerEdge BIOS flash upgrades in the form of Windows-only self-extracting archives (despite the facts that they claim to officially support Linux and that the BIOS flash utility itself is a DOS program.) Ordinary unzip won’t open the self-extracting archive. You need to run it on a Windows system, which I don’t have. Here’s how to flash your PowerEdge BIOS without Windows.

Dell provides the biosdisk utility to make a bootable FreeDOS image with the BIOS upgrade executable on it, but it doesn’t work with the Windows-only PowerEdge 4400 BIOS upgrade, since that won’t run on FreeDOS. They claim that biosdisk isn’t intended for PowerEdge systems, since they all have Linux BIOS flashes available; this does not seem to actually be the case for the 4400.

The Windows self-extractor does run under WINE, but it demands an actual floppy drive device to write to, which my computer doesn’t have. WINE also doesn’t seem to support mounting an image file as a floppy device, for some reason. You’ll need to use something like VMWare or a computer with a physical floppy drive to get this to extract itself.

Once you have a floppy image, here’s how to flash without biosdisk or Windows available: Continue reading

Swing-Clojure GUI for the Black-Scholes Option Modeler

Now that we have implemented Black-Scholes in Clojure, let’s make a Swing GUI for it. The Swing GUI will have text boxes for all the necessary inputs, and calculate prices and Greeks when the button is pressed. It’s a simple and straightforward way to get started in Swing GUI programming in Clojure. Here’s what it looks like. Screenshot of the Swing GUI for the Black-Scholes option modeler, implemented in Clojure

The GUI is written entirely in Clojure with the Swing toolkit. Calculation state is stored in a series of atoms. Watches are used to update the output table automatically when an atom changes, an idea from Kotka. I used the excellent MiG Layout for general layout functionality, and generic Swing widgets (JTextFields and a JTable) for the input and output.
Continue reading

Kartik Athreya: Leave It to the “Experts”? Seriously??

Kartik Athreya’s article deriding the unwashed, mouth-breathing public for presumptuously daring to have an opinion about his area of expertise is making the rounds, predictably being slammed by those of us with the audacity to hold an opinion about some subject in which we lack a PhD. I list reasons why professional “expert” economists are useless or even dangerous, and why Athreya’s real problem is American culture.

  1. Economists’ general failure to have made many useful predictions in the past, as a profession. How many trained, qualified PhD economists warned us about the onset of the Great Recession in 2006? Right. About as many as predicted the Great Depression or the Panic of 1873 or the end of the gold standard or the ’70s oil shocks or the ’87 stock market crash or …

    I own Business Fluctuations, a 1952 textbook by UC Berkeley economics professor Robert Aaron Gordon, which has a quaint section entitled “IS THE BUSINESS CYCLE OBSOLETE?”

    How about bona fide, credentialed economist Irving Fisher‘s famous 1929 pronouncement that “Stock prices have reached what looks like a permanently high plateau”?

    It’s not just that one or a few fringe economists occasionally make wacky analyses of the economy that turn out to be totally false; the profession as a whole does it, and has been doing it repeatedly for as long as anyone can remember. These guys were not fringe nutjobs. These views were reasonably mainstream at the time they expressed them.

    Theories that do not make falsifiable, testable predictions are pseudoscience. A theory cannot explain every possible outcome, else there is no way to test it and it is not falsifiable.

    Embracing their theories as expert advice even after their repeated failure to demonstrate any accurate predictive ability is worse than useless. It’s dangerous. Remember that Alan Greenspan guy? What was his profession, again?

  2. The vast incidence of massively contradictory economic paradigms held by various “experts”. Keynes versus Friedman springs to mind at once, and let’s not even touch Marx or Soros. Efficient market hypothesis, anyone? Ask 10 economists today merely whether and why we will see deflation or inflation in the US, and you will get 10 contradictory, mutually exclusive answers.

    To extend Athreya’s cancer analogy, it’s as if one oncologist expert says cancer is caused by genes and requires gene therapy; another says it’s caused by fluoride in the water and needs wholesale tooth extraction; a third says it’s caused by ragweed pollen and just requires an antihistamine; a fourth says the patient’s bodily humours are merely imbalanced, but bloodletting with leeches is a sure cure in his vast and storied professional experience; and yet another says that cancer is caused by Mars having passed too close to Sagittarius on the patient’s birthday, so nothing can be done except perhaps a sacrifice to the gods. And still none of them can predict with any accuracy at all who will get cancer, much less tell you how to cure it with better-than-chance accuracy.

    Little respect would naturally develop for such a profession, and members of the public would feel well qualified to add their own opinions and contributions, since they couldn’t possibly do much worse than the self-designated “professionals”. And indeed, that was exactly how medicine worked in premodern times, until doctors began to put forth testable predictions that actually accurately predicted future conditions and developed rigorous, repeatable experiments to test them. Economics has not yet achieved any measure of remotely reliable prediction. Economics is today a protoscience at best.

    Athreya himself says, “For my part, seventeen years after my first PhD coursework, I still feel ill at ease with my grasp of many issues, and I am fairly confident that this is not just a question of limited intellect.” (This false humility is ostentatious and self-serving; it was written to say, “hey, this is so hard that even us experts don’t know what’s going on, so you amateurs should just all be quiet.”)

    Some amateurs, like Soros, have consistently done far, far better than the pros. The field is simply extremely immature and not capable of providing very good answers to anyone. “Experts” are not special in that regard. They’re barely more aware of what’s going on than anyone else, and often they develop emotional attachment to theories that they have invested professional capital in, resulting in tunnel vision. When they can consistently predict significant economic events before they happen, and consistently do so better than amateurs, then they can begin to merit the respect accorded to oncologists.

    As Nicholas Taleb says:

    The environment in financial economics is reminiscent of medieval medicine, which refused to incorporate the observations and experiences of the ple­beian barbers and surgeons. Medicine used to kill more patients than it saved – just as financial economics endangers the system by creating, not reducing, risk. But how did financial economics take on the appearance of a science? Not by experiments (perhaps the only true scientist who got the prize was Daniel Kahneman, who happens to be a psychologist, not an economist). It did so by drowning us in mathematics with abstract “theorems”. Prof Merton’s book Continuous Time Finance contains 339 mentions of the word “theorem” (or equivalent). An average physics book of the same length has 25 such mentions. Yet while economic models, it has been shown, work hardly better than random guesses or the intuition of cab drivers, physics can predict a wide range of phenomena with a tenth decimal precision.

  3. Economic policy requires concerted action by all, and its results have vast consequences for everyone. A choice of several potential cancer treatments is between the patient and his doctor. If oncologists disagree about which treatment is best, concluding a decision as to which to go with requires convincing at most a small handful of people: the patient and his family and close friends. The consequences of that choice will also rarely radiate much farther than that.

    Economic policy, on the other hand, requires the concerted efforts of substantially all of the population, via government agencies and their use (or lack thereof) of tax money (at least in a democracy.) Ordinary people don’t know or care about the abstruse details; they just want to know how it will affect them. Enter politicians and big businesses, who, needless to say, do not always have pure motives. Public opinion can be manipulated easily.

    Given that the choice of which policies to implement is a political choice that ultimately rests with the entire population; given that self-interested politicians, investment banks, and other capital institutions are actively seeking to manipulate that opinion for their own benefit; and given the gravity of policy choice ramifications for each and every member of society, Athreya’s position of “just leave it all to us experts and don’t insult us by trying to think about it” begins to seem positively Orwellian in tone.

  4. Perverse incentives reward bland economic thinking by the professionals; the “cover your ass” syndrome.

    Working professional economists are not rewarded for making accurate predictions about the future of the economy; they are rewarded for making safe predictions about it. (Safe for them, not for the economy.) You can see this in action every time new economic indicators come out. Bloomberg and the other financial news media run stories with “consensus estimates” which do not usually deviate very much from each other. When there’s a surprise figure, there is rarely a pro willing to go on record and call it in advance.

    Why? They have large career incentives to be safely mainstream and risk averse. The reward for correctly predicting an outlier economic data point is relatively small; maybe the financial press will perfunctorily laud you for half a day at most. The penalty for making an on-the-record inaccurate prediction of an outlier event is large in professional terms; other economists will suspect that you’re ignorant of the field’s basics. On the other hand, there is no penalty for making a safe prediction that is in line with the mainstream view. If the mainstream theory is right, all the better; naturally, everyone’s predictions will converge. If the mainstream theory is wrong and an outlier event occurs, no harm done; everyone else was wrong, too. How could anyone have known better?

    The same general process plays out in academia. A graduate student will not get a PhD, and a new professor will not get tenure, for making earth-shattering predictions or proposing radical new theories. She gets the PhD and tenure for making very minor improvements to some existing body of theory, fleshing out a few small details here and there, and flattering the preconceptions of a committee who have all invested decades of their lives into some particular paradigm of economics.

  5. Cultural factors related to perceptions of formal credentials. Many cultures in the world adore fancy titles and formal credentials. America is not one of them.

    “All men are created equal,” reads our founding document, the Declaration of Independence. It was written by upstart bourgeois traders and lawyers who were, in part, greatly annoyed that their potential for social advancement was limited in British society due to lack of aristocratic birth and an Oxford diploma. They came to America to live somewhere where that did not matter much.

    We are hardline egalitarians, proudly derisive of highfalutin institutional authority at every turn. The “experts” are a self-selected bunch who can often contribute something useful. Insofar as they can occasionally do so, great, we’ll keep them around. Just as often, they seek merely to guard their own status as elite members of an exclusive club, to defend their turf from outsiders. Not infrequently, they devolve into groupthink and dogmatism, eventually stifling progress if left unchecked. Mainstream society does not really care, because it pays little to no attention to them — though they are very often totally unaware of this and consider themselves important figures in society.

    I think this anti-authoritarian attitude is in no small part exactly what has driven American successes in so many fields of endeavor over the centuries. Sure, there’s nothing wrong with having a PhD here, and you can gain wonderful amounts of knowledge by doing so, which is great. But as far as anyone outside the narrow specialized field is concerned, that doctor’s theories are judged on their own merits, not on reputation or fancy titles. Moreover, we in no way whatsoever feel the need to refrain from commenting on a matter simply because we are not “experts.” (This does not mean that the laymen are necessarily or even often right, of course, which is another matter.) We certainly don’t refrain from criticizing some “experts” just because they are highly placed at a respected institution and bear its seal of approval.

    This is very much not the case in, say, Germany or China. Institutions there exist to imbue people with the right to have an opinion, and only after arduous study and prolonged ritualized vetting by the pre-existing high priests of the field. Everyone else simply shuts up and listen to them, because they know that they are not qualified to have an opinion on a complex matter without a PhD in hand.

    Immigrants to America, and sometimes their children, naturally carry a cultural legacy from their home country with them. Those who bear fancy degrees and come from countries with high regard for institutionalized authority often have a hard time adapting to the fact that Americans just don’t care about fancy degrees and titles very much. They feel stilted and insulted. Often they don’t even notice the cultural correlation, or if they do they dismissively condemn it as merely an attitude for the ignorant and the stupid, those who are simply too unenlightened to realize the vast power of institutional blessing for one’s opinions.

    I think this is the ultimate source of Athreya’s frustration. He’s annoyed that Americans are not exhibiting the proper deference to his institutionally sanctioned writ of authority to hold opinions about economics. His borderline-abusive derision of economics bloggers certainly supports this theory. How dare a “naif” layman hold a view that contradicts his, the great doctor, when he has spent so many years studying, and read so many books! He even goes to the trouble to be affectedly humble, disclaiming having a firm grasp of the subject himself so as to demonstrate just how tough it really is.

    I am unaware of his exact cultural provenance, but “Kartik” is the name of a month in Bengali. Indian subcontinent cultures exhibit an extreme case of this appeal-to-authority automatic reverence for institutional knowledge approval. He is trying to impose Bengali cultural norms (or wherever he’s actually from) upon Americans, a curious case of “reverse cultural imperialism”. Perhaps he grew up here, considers himself an American, and doesn’t even realize this cultural dimension himself.

None of this should be construed as a rejection of the formal academic process. Having a PhD is fine, and many PhDs do useful work. There are even many knowledgeable, useful professional PhD-bearing economists! The problem comes when the PhD is not taken as a marker of knowledge but rather as a cudgel with which to bludgeon those lesser beings who somehow have the audacity to fail to obey the expert’s opinion. This becomes a farce when done in a field like economics where there is no certain knowledge at all, a long track record of totally failing to predict major and devastating events clearly declared to be within the field’s purview, and indeed relatively few predictions that can be construed as “scientific” in any sense.

Lazy-spy: Clojure logging/spy for lazy sequences

Clojure.contrib’s logging/spy macro is really useful for quick investigation of bugs with unclear stack traces, and to check that intermediate values in a calculation are being returned and composed correctly. Lazy sequences don’t get logged correctly, however:

> (log/spy [1 2 3 4 5])
[1 2 3 4 5] ;;; prints " [1 2 3 4 5] => [1 2 3 4 5]" to the log
> (log/spy (take 3 [1 2 3 4 5]))
(1 2 3) ;;; prints "(take 3 [1 2 3 4 5]) => clojure.lang.LazySeq@8592" to the log

The spy macro logs a string realization of the sequence per se, rather than its constituents. In the case of a concrete, fully realized sequence such as the literal vector in the first example, all values are known in advance, and the vector’s string realization in the log can helpfully be its actual contents. In the case of the lazy sequence returned by take, however, we get a less than helpful class name and some sort of instance identifier code rather than the actual contents of the sequence. A lazy sequence saves resources by merely promising to provide the values when asked. If they’re never asked for, they’re never computed. Since they haven’t been computed, they can’t be printed. A lazy sequence can also potentially be infinitely long, another good reason why they are not realized by default when converting the sequence to a string.

In our case, we don’t care about the resource usage, and we’re sure the sequence is finite. We want to force all values to be computed for manual inspection. (There are several ways of going about this.) The following modification of the spy macro converts the lazy sequence into a regular seq:

(defmacro lazy-spy
  "Evaluates expr and outputs the form and its result to the debug log.
   If the result is a lazy sequence, it is realized concretely in the log.
   Returns the result of expr."
  [expr]
  `(let [a# ~expr] (log/log :debug (str '~expr " => "
                                    (if (= clojure.lang.LazySeq (class a#))
                                      (seq a#)
                                       a#))) 
        a#))

This gives us:

> (lazy-spy  (take 3 [1 2 3 4 5]))
(1 2 3) ;;; prints "(take 3 [1 2 3 4 5]) => (1 2 3)" to the log

Clojure posts you might also like:

Log4J and Clojure Configuration

Clojure, Log4J, and clojure.tools.logging can play nicely together. Here’s a reasonable default configuration.

Update: Consider using clj-logging-config to configure your logs in a more Clojurey way, rather than screwing around with the ugly Java-ish log4j.properties file described here.

  1. If using Leiningen, add

    :dependencies [
                     [log4j/log4j "1.2.16" :exclusions [javax.mail/mail
                                                  javax.jms/jms
                                                  com.sun.jdmk/jmxtools
                                                  com.sun.jmx/jmxri]]
                     [org.clojure/tools.logging "0.2.3"]
    ]
    

    to your project.clj file and rerun lein deps to auto-install the Log4J JAR.

    If not using Leiningen, acquire a Log4J JAR and put it on your classpath.

  2. Create a file called log4j.properties on your classpath and put the following into it:
    # Based on the example properties given at http://logging.apache.org/log4j/1.2/manual.html
    # Set root logger level to DEBUG and its only appender to A1.
    log4j.rootLogger=DEBUG, A1
    
    # A1 is set to be a ConsoleAppender.
    log4j.appender.A1=org.apache.log4j.ConsoleAppender
    
    # A1 uses PatternLayout.
    log4j.appender.A1.layout=org.apache.log4j.PatternLayout
    log4j.appender.A1.layout.ConversionPattern= %-5p %c - %m%n
    

That’s the short version. Read on for the gory details. Continue reading

Flash BIOS Without a Floppy

Flashing the BIOS without a floppy drive can be done with an Ultimate Boot CD (free!) and a USB memory stick. BIOS upgrades usually come in the form of an MS-DOS executable meant to be run from a bootable floppy, but modern computers don’t usually have floppy drives anymore, and who has a copy of DOS lying around anyway?

  1. Download a copy of the free Ultimate Boot CD and burn it to disk. FreeDOS (and a lot of other useful stuff) is preinstalled on the CD. As the name implies, FreeDOS is a free, open source, binary-compatible replacement for MS-DOS.
  2. Download the replacement BIOS from your motherboard’s manufacturer. Unpack it onto a USB flash drive.
  3. Plug your flash drive into the target computer and boot it with your Ultimate Boot CD. Select FreeDOS from the menu.
  4. FreeDOS asks a number of configuration questions while booting. You can usually leave all at their default except for the ASPIUSB driver. ASPIUSB is not enabled by default (at the time of this writing), but is necessary for the use of a USB flash drive under DOS, so enable it when prompted.
  5. If you miss the prompt for the USB driver, don’t worry. Type “menu” at a command prompt and enable USB devices from the popup menu.
  6. Change to your flash drive by typing e.g. R: and execute the BIOS flash utility as per the manufacturer’s instructions. (If you don’t know which drive letter to use, type “menu” and select the option to display active drive letters.)