Multiple Definitions

We had an interesting discussion at work yesterday debating whether Mathematica supports multimethods with its variety of pattern-matching. So I'm taking the opportunity to do a mini-survey of the multiple dispatch spectrum, starting with overloading. As far as Mathematica, it clearly gives you the power of selecting from multiple definitions based on runtime information; more on this in a minute.

Overloading allows you to write different definitions of a function or method, and the definition used when you call the function depends on the number and types of the arguments you pass them. That is, the overload resolution is done by the compiler at compile time with static type analysis. Whatever type your arguments are, and exactly those types, will determine which definition is chosen.

Multimethods let you write different definitions of a function or method, and the definition used when you call the function depends on the number and types of the arguments you pass to them, as examined at runtime. Now you can write one definition for a class, and specializations for its subclasses if so desired, and the definition will be chosen based on the actual type of the arguments at runtime.

You get into a fuzzy gray third area if the runtime values can also be used to select different definitions. This is where Mathematica lies, because its pattern matches can be used to differentiate between a list of at least one element and a list of at least two elements, or between the symbols Null and Infinity. What's useful for writing symbolic algorithms turns out to be useful for regular programmers.

It seems that ML's structural pattern matching is also in this fuzzy gray third area, and that helps me make an interesting connection. For my purposes, multiple dispatch is interesting because it's the way to do expression tree traversal. That is, it lets you write pretty printers and type checkers and the like without needing to code the dispatch yourself (if the node is actually a lambda abstraction, do this, but if it's a cons cell, do that). What I'm noticing now is that one way or another, multimethods and pattern matching are giving you the notational convenience that I enjoy in writing tree traversals, with still perhaps an edge to pattern matching on that score.

On "I can't believe I'm praising Tcl"

Today we look at a refreshing use case for a programming language, where economy of expression in calling functions from a read-eval-print loop is prized. Raganwald recently tagged "I can't believe I'm praising TCL", in which the embedded systems author helped us understand how TCL made for a great debugger command environment, and the "pop infix languages (C/Java/Python/Ruby/you name it)" don't.

In this case the author wants to define some glue function or functions, and then in the language's interactive interpreter, call his function over and over. He's not programming; he's commanding, so the function calls need to be short and sweet, so he doesn't mind typing them for hour after hour as he thinks about the real problem, a buggy piece of embedded hardware. The author wants a command shell, where he uses his command interface to an embedded device as a kind of gdb replacement. An example session to set breakpoints, inspect memory, etc., looks like this:

$ pmem 0 stat
IDLE
$ pmem 0 bkpt 0 0xbff
$ pmem 0 bkpt 1 0xa57
$ pmem 0 cmd run
$ pmem 0 stat
DEBUG
$ pmem 0 pc
0xbff
$ pmem 0 rstack
3 return addresses
addr 0: 0x0005
addr 1: 0x05a8
addr 2: 0x0766
$ pmem 0 cmd stp
$ pmem 0 pc
0xc00

The question then is whether a language you may be designing or using could support something close to this syntactic economy for calling functions.

The argument for Tcl over the pop infix language may perhaps be best summarized by this quote:

And then we have interactive shells. And in Python it’s doit("xx","yy"). And in Lisp it’s (doit "xx" "yy"), or (doit :xx :yy), or (doit xx yy) if you make it a macro. And in Ruby it’s doit :xx :yy, if you use symbols and omit parens. And that’s about as good as you can get without using your own parser as in doit "xx yy", which can suck in the (more rare) case when you do need to evaluate expressions before passing parameters, and doesn’t completely remove overhead. Also note how all these languages use (), which makes you press Shift, instead of [] which doesn’t. Ruby and Perl let you omit (), but it costs in readability. And [] is unanimously reserved for less important stuff than function calls.

Analysis

First we see the emphasis is not on defining functions, on programming, but on using, on the syntax for calling, functions. The author wants an internal DSL (Domain Specific Language).

Second, it should be noted that in discussing () that Scheme lets you use [] as well as (). There's good Scheme style, where [] is reserved for the let blocks, but if you open up Dr. Scheme or Chez Scheme, define some choice Turtle graphics functions, and start typing commands like [penup] [rt 45] [pendown] [fd 100] it will work fine.

One thing the author noted is that TCL's preference for strings over variables makes bkpt a string and $bkpt a variable, whereas in the pop infix languages, it's the variables that get lexical economy and strings that need delimiters. Because of this preference, calling a Tcl command lets you pass in what look like symbols, but you treat them as strings in the command definition. Hence, for the author's use case, a chief consideration seemed to be a way to write symbolic arguments, where the command in question may take a one-of-several subcommand or option name, without lexical decoration like string delimiters or single quotes or colons. I wonder if this was really a language design goal of Tcl, because it's hard to understand the motivation for the string-vs-variable syntax any other way. For all that, enumerated types or sum types are a known language feature that meet the author's criterion. In Standard ML you could define a datatype Subcommand = bkpt | status | memset or the like, and now undecorated references like bkpt can appear as arguments.

Note if you do define your functions as Scheme macros, to address the symbol/string problem, and if you modified Scheme to accept an S-expression forest on each line (i.e. no need to delimit a top-level input line with parens), you'd have the economic expression of Tcl. I think this is worth considering in some circles where Scheme may be more familiar.

Footnote

This could be a nice motify for the "language design of the day": extend the basic Scheme-like interpreter to support an extensible debugger command interface.

April Curmudgeon

Disclaimer: The April[0] Post you are about to read is not meant to be amusing. Please adjust your expectations now. Lighthearted, yes, but it will not make you laugh aloud. Really, it will not be amusing unless you somehow make it so by assigning a meaning it was never intended to have. By reading this post hopefully you will gain insights that allow you to be a funnier blogger. When Frank Bunker Gilbreth's children, whom he was teaching to touch type, asked if he could touch type, he only replied "They tell me Caruso's voice teacher can't sing a by-jingoed note." May it be so with this post.

---

It's challenging to write truly good April Fool's material about programming issues, but almost programmatically easy to write lame material. One reason things are funny is that (1) they violate expectations, and that (2) they do so in a way that doesn't personally impact you, as in Mel Brook's observation "Tragedy is when I cut my finger. Comedy is when you walk into an open sewer and die".

We're immersed in dark humor by the nature of our process-based profession. Programmers with customers (the most interesting kind of programmers) keep huge databases of failed expectations. These only get funny with distance: either in time or in organizational chart hops. Your coworkers' bugs are more likely to be funnier than your own, but only until your manager decides to reassign to you. Another team's bugs are funnier. Unless you rely on that feature. And so on.

The other kind of expectation, that program P (browser, language, Turing-capable text editor) expects input format X and generates output format Y, are also targets for humor: "Bwahaha, a web browser that accepts COBOL as an HTML scripting language" if it's well-known that this combination doesn't exist and doesn't seem to make sense. Humor playing on this type of expectation is fragile, because the expectations are broken on purpose all the time by the current of software development progress. It's only a matter of time when "Ruby VM in Javascript" moves from being funny to being accepted way of life. Maybe COBOL is funny today, but legacy systems have a way of creating needs that can be solved by software. If you want a joke candidate generator, write a program that generates T diagrams with all possible combinations of known programming languages at the different points. The funny ones will be funny because they violate the intended design of the language: Logo as an implementation language for a Fortress to Haskell compiler. (It's probably a good sign for a language if it's never funny at any point on the T.)

Take a look at the April Fools gags out there related to programming and see how many are analogous to "that idiot tried to plug a USB cable into a Firewire socket". The DailyWTF material is kind of a catalog of such expectations:

• You should write code without explicitly specifying every input case
  


• Code should have a purpose, unlike this if check:
  

  if( nextIndex != currIndex )
  {
    nextIndex = currIndex;
  }



• You should use a standard library function rather than writing your own buggy version


• You shouldn't transform data through several unnecessary layers (the digital camera/wooden table technique)


• Code should, ideally, do something productive, especially if you're paid to create it

The real humor, the best humor in programming topics and others, is the kind that also incorporates the self-recognition element, which I think is simple imitation or mimicry in its rawest form. This is the Jerry Seinfeld style, where you observe and describe a situation or behavior. There's something about hearing a description of a behavior, especially one we wouldn't readily admit to unprompted, that makes us laugh when we recognize we're guilty of it too.

---

Note: COBOL is one of those all-purpose humor words, kind of the "weasel" of programming humor. Source: Ask Mr. Language Person.

Lame subtlety spoiler: If you skipped the disclaimer, and thought this was supposed to be somehow funny just because it's April[0], check the title. Also, it's not realistic to expect to get a funny blog post in polynomial time from the T-diagram generator. Not that I tried.

Language Politics and Technology Decisions

When someone says "Java sux", or "Ruby rules", we need to always, always, always ask "for what?" The machinery of politics is alive and well in programming forums the world around. Everyone who has ever written a program or learned how to use a technology has a vested interest. They will continually be about the business of convincing others to use or learn it too. We need to understand that machinery in order to protect ourselves and have a more productive discussion. Better understanding of someone else's decisions as well as our own will also help us arrive at better decisions overall.

Decision Making

Politics is about decision making, and in technology we constantly make decisions. We make evaluations: Ruby or Java, XML or JSON, Flash or Ajax. We like to think these decisions are made on technical grounds, but I think if we look carefully we'll realize when we do the promoting we do so based on a mixture of reasons. (Do you make your decisions based on in-depth research, or whether you hate curly braces?)

Promotion of Interests

Politics has an important aspect when it comes to making decisions: it is always done in an environment of the promotion of interests. Not to be tedious, let me repeat those two parts:

1. promotion,
where the interesting thing is how are things promoted
and
2. interests,
where the interesting thing is identifying interests

Forms of Promotion

Promotion has different forms: direct or indirect, emotional or rational, personal or media-broadcast. We concentrate on rational discussion of technologies, and for good reason -- if something doesn't work, there's a price to pay, in starting over or cleaning up a mess and if we're not dead or fired, starting over. The rational basis for promoting a technology has to be there. But the emotional side is at work too, whether you like it or not. My wife maintains that if someone dresses sloppily that they won't be liked -- even if others don't understand why. Having an attractive website may or may not add to your presentation, but an ugly website almost certainly will detract from it. The take-home for this for technical people is that presentation and form play a part along with rational arguments. Iron your arguments, iron your shirt, iron your website. ("Iron your Python"?)

Identifying Interests

All the time we encounter people promoting this language or that tool. And often enough we ourselves are doing the promoting, even if not in Superbowl ads. As already noted by many others, even if you're not a salesman who stands to make a commission from convincing someone to adopt technique T, you stand to gain more benefit from your vested interest in T if another person uses it. (Note that "technology" derives from the same root as "technique"; design patterns are materially as much a technology as the Python 2.5.1 distribution). You can act as a consultant for newcomers to T, and you attract more people to maintain and improve work already using T. Cisco Systems actively promotes network administration training programs for high-school students, as part of making sure network usage grows as much as they'd like it to. Microsoft was visiting programming language groups, who are notoriously ill-funded, looking for people designing new languages offering funding to target the .NET platform. Some in industry like to talk about this as "growing the ecosystem." The point is that when you hear someone promoting a technique, they're promoting it from their interests, not necessarily yours. It's not always as obvious as when you get a call from a salesman.

Evaluating Proposals

The primary question to ask when you evaluate a technical proposal then is not unfamiliar: what's in it for me. Or rather you should be asking "what would this technique/language/framework do for me in the environment I have." The Hennessy and Patterson computer architecture books hammer home this maxim about evaluating a computer architecture:

The true benchmark is the wall clock time of the program or programs you will actually use most often

Paul Graham designed a language that Paul Graham liked. Maybe there was a different way he could have handled its announcement. But I venture that part of the over-hype problem with the Arc announcement last week is that when Paul Graham wrote about how great a language he was designing, people were hearing how great it would be for them, mentally filling in unsaid things with features they'd like to see, and not pausing to try to align what Paul's interests were with what their interests were. The thing is... though we may have an opinion, we may not always know what would help us.

Know Yourself

When someone tells you "you need better tools: try Lisp", ask "what about Lisp do you think would help me?" If they start listing reasons without first trying to understand who you are, may I say there's a problem.

This has me thinking about a conversation from the film Joe vs. the Volcano between Marshall the limo driver and Joe, who is preparing for a trip at the end of which he plans to jump into a live volcano. I've adapted it to acknowledge the need to know what we're trying to do, and at the same time how hard that can be. Instead of the limo driver, JFKBits plays the part of technical consultant's technical consultant, and instead of the volcano jumper preparing for his big trip with carte blanche Joe plays the part of a programmer for hire:

JFKBits: So what would you like to do?
JOE: Excuse me?
JFKBits: What would you like to do, sir?
Joe thinks for a moment.
JOE: I thought I might like to learn some programming language.
JFKBits: Okay. What would you like to program?
JOE: I don't know.
JFKBits: Alright.
JOE: What would you program?
JFKBits: For what? What do you need?
JOE: A web application.
JFKBits: What kind of web application? What does your client need?
JOE: I don't exactly know.
JFKBits pulls the car over and stops.
JOE: Why'd you stop?
JFKBits: I'm just hired to consult with you, mister. I'm not here to tell you who you are.
JOE: I didn't ask you to tell me who I am.
JFKBits: You were hinting around about programming languages. It happens that programming languages are very important to me, Mister..
JOE: Reader.
JFKBits: Reader. The programming language makes the man. I believe that. You say to me you wanna learn some programming, you wanna write a webapp, but you don't know what kind. You leave that hanging in the air, like I'm going to fill in the blank, that to me is like asking me who you are, and I don't know who you are, I don't wanna know. It's taken me my whole life to find out who I am and I'm tired now, you hear what I'm sayin'?

In the movie, the topic is clothes: "clothes make the man." As with clothes, there is a very personal element about the things we work with on a daily basis, and it's worthwhile understanding why you like what you like. I'd wager that for most people, past a certain age, their preferences and reasons are locked in and choice of tools is largely based on what's familiar.

Unlike Marshall, I think too many of us are far too eager to advocate our personal technical decisions to people with whose needs and environments we are not well acquainted.

Some Applications

When being consulted by others about what technical choice to make, we should always ask "what do you need it for? What are your interests?"

When being persuaded, we should always ask "what is this good for, and does that line up with my reasons?"

When persuading, we should exercise due diligence to identify why someone should care; in particular, what interests do we share such that what makes this decision appealing for me also makes it appealing to you.

Conclusion

When you see judgments ("Ruby rulz"; "the real WTF is they were using PHP"; "the tools for Python stink") in an online forum, realize they represent the end result of a decision-making process by a person who is not you and most likely has a very different set of interests. If you have an axe to grind, i.e. they're challenging your judgment, you may unnecessarily expend effort "debating", effort which may be more productively spent understanding the underlying interests and reasons for arriving at a conclusion.

If you can determine they really are your political opponent, that you are in direct competition for the same target audience, I suppose you could go at it. But I think we too often assume that different conclusions imply either competition or faulty reasoning processes. Just a little bit of effort in understanding "where are you coming from" may be more appropriate and ultimately more enlightening.

Pizza to Scala: The Ten Year Road of Language Design

James Gosling, yesterday:

There has been a lot of chatter about the closures proposal penned by Neal Gafter. And, in particular, whether or not I support it. I absolutely do.
--James Gosling, Thursday January 31, 2008

He goes on to add "Closures were left out of Java initially more because of time pressures than anything else." Java was introduced in 1995. Let's look a bit back into history to see what's been happening since then.

In 1998, my advisor advised me to take a look at some related work, something called Pizza, "a substantial companion to Java." Pizza was a joint project between Philip Wadler and Martin Odersky. They wanted to add closures, parametric polymorphism, and abstract datatypes to Java.

Then anonymous classes were introduced to Java, which went 80% of the way toward giving Java the utility of closures (4 out of 5 programmers thought it was good enough). Still a syntactic pain, but a net win considering the convenience of putting your code where it makes sense rather than a class definition elsewhere. Wadler and Odersky decided to focus their energies on generics, and moved on to Generic Java. I used the Generic Java compiler at the time simply because it was faster than Sun's javac for compiling ordinary Java code. The work on Generic Java was then actually adopted into the Java standard.

That brings us to 2008. Martin Odersky is now realizing the original goals of Pizza with Scala, and closures are being brought to Java.

I appreciate the efforts of Wadler and Odersky in bringing functional programming closer to the average programmer. In 1998 Wadler wrote an enjoyable piece "Why No One Uses Functional Languages", a kind of State of the Functional Programming Union, in which he lists 8 barriers to adoption of functional programming. At a casual glance, I count 5 out of 8 of these barriers broken down by the association with the Java platform. These include portability, libraries, and compatibility (e.g. foreign function interfaces and other ways to interface with legacy systems). A sixth reason, tools, is addressed or helped to a large degree by Eclipse, which while written in Java, largely owes its existence as far as spirit of design and the benefit of years of hard-toil experience to the Smalltalk community.

Joel Spolsky and they that hire have been bemoaning the dilution of fresh CS grads coming out of the Java mills. In light of the above technical developments, this is an ironic trend, considering the "training" barrier Wadler mentions:

To programmers practiced in C, C++, or Java, functional programs look odd. It takes a while to come to grips with writing f(x,y) as f x y. Curried food and curried functions are both acquired tastes.
--Philip Walder

My understanding of Spolsky's point is that "the kids aren't learning Scheme - they don't know the great ideas of computer science anymore, they just learn Java and design patterns." If it is the case, yes it does bother me. I don't know if that is the case; I don't hire people, and the people I've worked worth have been (mostly) great.

What I will say is this. The churn of progress has been steadily raising the level abstraction over time. We have needed, and will continue to need, people who create the new abstractions (languages and frameworks) and people who apply the new abstractions. Bright engineers out of the Java mill will eventually be challenged with problems their existing tools don't address (hey, this is tedious and error prone...) and if nothing else they'll reinvent the great ideas of the past, and relearn them the hard way. Learning things the hard way is effective (you tend to internalize lessons better when they involve words like "shots fired" and "you [and your slipped schedule] are gonna cost me a billion dollars!") and expensive. Managers should be noticing things like that. I hope some bright engineering managers, who sit on their local university's advisory board, notice these trends as they arise and say a few words on their next visit to the department.

---

Community service footnote: I sit on the advisory board for the CS department at my alma mater and provide feedback as an alum and industry representative. If a school you care about doesn't have such a board and you think they need to hear from you, start it. Even if you don't think you have influence, alums in the technical fields can always pretend they'll strike it rich someday and therefore should have some influence.

Things to Do With Your Programming Language Course

It's finals week across the country, and thousands of computer science students are finishing up a programming language course. At least a handful of these have been infected with the fascination of the subject, and I hope to meet them someday. From the enthusiast to the couldn't-care-less, here are my thoughts about where you can go from here with one programming language course under your belt.

Going Pro

Take more classes, participate in an open source language project, go to grad school, and get a job with a company that makes a compiler or interpreter (IBM, Microsoft, Wind River). Remember that there are lots of languages with a smaller reach that still need people to work on them. Embedded systems have lots of chips (purpose-built CPUs and microcontrollers) that need compilers and development environments.

Programming Tools Development

Similar to Going Pro. Not every parser is feeding a code generator. Join a team making an IDE, an IDE plugin, a code formatter, or a code checker (a la LINT). Eclipse puts this category within reach of anyone (so did EMACS). In order to develop at this level you probably want to take at least one more course or do a semester long project to have a fighting chance of getting enough experience.

In-House Tools Development

The need to make sweeping changes or do some kind of transformation on a code base comes up all the time on large projects. I should probably say the "opportunity" rather than the need, because some problems that could be solved by a transformational tool will be worked around. But here's where a little initiative and paying attention during those parsing lectures will pay off.

Better Programming and Debugging

Even if you don't directly use your programming languages course, even if you hated it, if you took the right kind of course you will have your mind trained in ways that make you a better programmer. It's not a substitute for reading good code and learning from experienced practitioners, but it is similar to learning how to disassemble, clean, and reassemble a car engine blindfolded. It's analytical training for a language, you see what parts compose the tokens and phrases of the language, and see how they're combined into your program. It helps dispel myths about how your program is likely to work under the hood. This of course is especially true if the language you implement is similar to the language you write in, but if you can't go too far wrong working with a language more "interesting" than you're likely to work in.

Even the time you spend studying language design is not wasted. At some point, programming on a large team involves discussions and development of a coding standard for the group. These discussions are in a sense exercises in language design, because a coding standard effectively limits the expressive power of a language into a subset of the language that everybody is comfortably working with.

Recommended Reading

"Essentials of Programming Language" by Friedman, Wand and Haynes. Build up your knowledge of interpreters and compilers, starting with a simple lambda calculus, then to Scheme, building up to the powerful and high-performance techniques of continuation-passing style and tail recursion. Sometimes used as a text in PL courses. Maybe you like my arch-geek friend Tony will use your newfound knowledge to spend your Christmas break writing a Scheme compiler for your HP scientific calculator.

"A Retargetable C Compiler" by Hanson and Fraser. A funny book, it uses literate programming to document and explain the implementation of lcc, a C compiler used to compile Quake III. From memory management (arenas with constant time deallocation) and string hashing, the authors show and explain their C code from the bottom up. The ANSI C style is a little odd, but for the modern student, C will perhaps seem odd anyway.

Dollars Per Gigabyte Per Month: On Backups and Reliability

Following "Half a Terabyte, Hold the Gravy," several JFKBits readers have been discussing the JungleDisk interface to Amazon S3 and I wanted to explore storage reliability, extending the discussion from dollars per gigabyte for raw ownership to dollars per gigabyte per month.

Reliability and Backups

First, a note about reliability in general, and CDs/DVDs in particular. Failure of any backup device or media can occur due to basically two things: aging and mishandling. The NIST publishes "Care and Handling of CDs and DVDs: A Guide For Librarians and Archivists" and it tells you in great detail exactly what goes wrong when you mishandle a disk. When we talk about reliability, we may easily remember numbers from quotes like "An accelerated aging study at NIST estimated the life expectancy of one type of DVD-R for authoring disc to be 30 years if stored at 25°C (77°F) and 50% relative humidity" (which is far under the industry consensus of 100-200 years). What we may not remember as easily is that reliability is directly affected by handling. Part of any backup plan needs to be hygiene practices, if you will, of taking care of the backup media and hardware.

Costing Storage Over Time

Amazon S3 is $0.15/GB/month, or $1.80/GB/year, and for someone looking to store a few hundred gigabytes forever, this is an important consideration. Surely the reliability of hard drives are better than one year, and why would we pay $1.80 for our gigabyte to Amazon when we can get away with copies on a couple of drives at 36 cents each or a couple of DVDs at 9 cents each?

The JungleDisk web site quotes a stat from a recent independent study on drive reliability at Carnegie-Mellon by Schroeder and Gibson stating that 15% of hard drives will fail in 5 years. If you ignore multiple drive failures, and counting on retiring a 5-year old drive, I figure that means a drive at price P will actually cost 1.15 P in replacement costs. Of course you'll probably want to be the replacement drives up front, and unless you're buying 100 drives or more, you're actually spending more on replacements that 15% extra, perhaps up to 5 P depending on how much redundance you want (we'll get to RAID later). At the minimum 2 P for us small-scalers who aren't putting 100 drives into active use simultaneously, my $182 LaCIE drive comes out to $364 over 5 years, or $0.013/GB/mo. That's still a healthy distance from $0.15/GB/mo. I could buy five more replacements, acknowledge that a "500GB drive" is only 460GB and still be at $0.04/GB/mo. What else do we need to consider?

Inexpensive Disks or Cheap Disks?

Marcus Smith is the owner of a company providing IT services including backup and recovery to small business, and had some things to say about hard drives as archival medium on the Association of Moving Picture Archivists mailing list:

I completely agree that hard drives are only one component of an overall backup strategy and that multiple technologies will need to be employed in order to provide truly safe nests for our precious ones and zeros. There are a couple of universities experimenting with the idea of ditching tape systems and instead building very large hard drive arrays in which drives are given a certain time to live and are replaced on a rotating schedule. If every drive actually lived up to an exact span this idea would probably be used more often. The sad truth is the hard drives can fail at any time and this unpredictability alone is enough to show the overwhelming undesirability to use them as an archival media.

The idea that cost-per-megabyte savings is an enabling feature needs more clarification. How does this translate exactly in to providing new methods of storage? One major problem with using a cost-per-megabyte approach in favor of large capacity drives is the intended market. Who is buying these drives? Who is using them for long-term storage?

When it comes down to it, I'd be willing to wager that far more people are using hard drives as their sole mechanism for storage because of their low cost, which therefore makes price a liability, not a benefit. As the price of drives drops, it only traps us tighter into relying on unreliable media.

To store the same amount of data on other mediums - tapes, really - we're already spending thousands of dollars. The low cost and high capacity of hard drives is exactly why ATA is used instead of SCSI, and it continues to have a draining effect on quality of any solution because of the lack of any other cheap storage media. As long as cheap drives push the market, that is what most people will use. Again, this points to the question of who the market really is. Are we talking about large archives who may or may not be able to spend money on developing proper, multiple technology storage solutions? Or are we talking about the "Prosumer" level where individuals migrate film or store native digital video on hard drives because there is simply no other reasonably attainable storage device? Since Fry's are offering cheap drives, and archival institutions probably aren't shopping there for serious storage solutions, it seems to me that we're really talking about individual consumers who need basic, fast, easy access storage. In short, folks who probably don't have tape drives, a multiple technology backup systems, or a well-planned rotating storage schedules.

Costing Data Failure

For JFKBits readers, I know our needs are all over the map: high res digital photos; Linux backups including both system configuration; personal creative works representing who knows how many hundreds of hours of labor. In most cases the scenario of losing our data affects mostly ourselves. I contacted Marcus for permission to quote him in this article, and he described his small business clients: lots of professionals, where other people depend on the data; losing information invokes the specter of a lawsuit. When we make our backup plans, we know the purpose is to prevent data disaster. But when we start costing it, sometimes the cost of the data disaster tends to be left out:

Determining the value of the data you want insured may not be easy. A single database or secret recipe may be the heart of your business, and estimating the worth of that intangible asset is a problem that can give CFOs nightmares
--Kevin Savetz in "Data Insurance", published in New Architect May 2002

RAID When Things Go Wrong

At this point, RAID comes forcefully to mind. Duplicate the data and add some automated error correction and detection. More caution, more consideration, urges Marcus:

RAID arrays can offer some benefit, but there are dangers here also, and they need to be addressed. RAID arrays do not technically have to be constructed with identical drives, unless one uses crappy hardware controllers. But for RAID to work efficiently, then identical geometry and capacity drives should be used to build the array. Most RAID systems are based on using identical drives. Here again money comes into play. RAID 5 is nice for the overall capacity, but using identical drives may be seen to have similar failure characteristics. Let's mentally build an array with five drives, and buy five more for replacement. It could very well be that within two years all ten drives may be exhausted and getting two year old drives can be very difficult to procure. Nearly impossible, actually. There are two obvious problems: (1) you're only as safe as the number of drives you buy, which is going to be many. Go ask Compaq for a 9.1gig hot-swap replacement for a Proliant server. For that matter, head over to Fry's and ask for an IBM 30-gig Deskstar drive. It will never happen. (2) If identical drives have similar failure characteristics, then it stands to reason that when one drive fails, the other drives in the array are not far behind. There are also two non-obvious problems: (1) If two drives fail before the operator knows there is a failure, all of the data on the array is lost. (2) Data recovery from missing RAID volumes is notoriously difficult and usually ends in failure of recovery.

Luckily there are other RAID options, like mirroring and striped mirrors. Again, be prepared to invest in drives. Lots of drives. These options use half the drive capacity of the total number of drives. I.e., for 500 gigs of space you need not five 100gig drives, but 10 100gig drives. You cannot do this with ATA drives because of the limitations of ATA controllers. I believe the largest possible number of drives on a single controller is the 3Ware 8-channel RAID adapter. Eight drives plus eight spares. Plus more if this is where the Company Policy places its long term future. Add a storage rack for this and a Very Expensive RAID adapter and suddenly the one-year warranties on the drives turn an expensive project into an expensive and unreliable project. The benefit over RAID 5 is that data recovery is much easier of this kind of array if it comes to it.

So yes, we need to think very, very differently about storage. RAID solutions are not as safe as their cracked up to be, and depend greatly on the vigilance of the systems administrator to keep things running smoothly.

Whew! The overriding message here I think is that RAID doesn't solve everything - you'll want to buy your replacement disks up front, as you probably can't get them in 5 years. You still need to keep watch over the health of the RAID array. (I had extra time to think about this on my way home tonight as I changed a flat tire.) And as the experience of one of our readers reveals, you still need a strategy to watch that your backups happen and that the backups themselves are OK. One JFKBits reader checked his backups after reading the recent article and discovered that his backup software actually scrozzled the data. Now how much would you pay to make sure this all gets done right? Ready to fork over that extra dime per gig per month? I thought you might be.

Conclusions

We all know we need backups. If we're thinking long term enough to realize that even the backup media may fail, we've started planning the cost over time. Making your plan depends on what you're doing; backups of a machine image or a database have different needs than backups of segmentable data such as are needed by pro photographers. Amazon S3's figure of $0.15/GB/mo is not unreasonable, but you still will want a plan to verify integrity for yourself. A $1000 RAID NAS box (maybe two) is reasonable, providing you buy the replacement drives up front and verify the backups. If your data is segmentable, DVD backups are quite reasonable, providing you have the time to burn plenty of copies, distribute them geographically, store them in a climate-controlled environment, and organize them well. Tape backups are not dead either, and we'll close by letting Marcus have the last word:

I personally am much more in favor of tape systems than hard drives. I just wish they were less expensive. My problem is the same as many others - I don't have $5000 to shell out for a tape drive to store 400gigs of materials. Unless, of course, someone would like to buy me a new LTO drive and a bunch of tapes.

Does anyone want to wax poetic about the beauty of Tar as backup software?

Half a Terabyte, Hold the Gravy

What do you do when you've got a device that creates very valuable 2MB files with the press of a button, an action that you may repeat up to 600 times an hour (maybe 20GB a week)? You end up buying another device, such as this puppy: the 500GB Lacie D2 HD Quadra 7200RPM 16MB external drive. I had never before heard of LaCIE, a French concern, but this drive seemed to fit the bill as a backup device. It has a reasonable dollars per gigabyte number, the reviews seemed encouraging, and I went ahead and paid the extra $60 to get the Firewire-capable model as this seems to be recommended over USB for sustained reads and writes typical of an external drive.

PriceGrabber got me thinking about the whole backup plan from the dollars per gigabyte angle for different types of storage devices, summarized here:

Tech	Sample Product	Price w/ S&H	Capacity	$/GB
DVD-R	Verbatim DVD-R 16x 100-pack	$42.27	470GB	$0.09/GB
external HD	500GB Lacie D2 HD Quadra 7200RPM 16MB	$182.60	500 GB	$0.36/GB
flash memory	PNY 4GB USB flash drive	$31.89	4 GB	$7.97/GB

DVD's are an economical choice as a backup medium, especially in my case where making modifications is not important. The 100 disk spindle is almost a direct comparison to the LaCIE drive in terms of capacity, since the LaCIE's actual capacity is more like 460GB. The external drive has the advantage that you don't need find the right disk when you need to retrieve your backup, so it is a good "live" backup solution. The external drive also gives you a little bit of portability, and the model I bought has four connection technologies (hence the name "Quadra") giving some flexibility in working with other devices.

The USB flash drive, on the other hand, is a terrible choice for long term storage, as far as I can tell. That salesman at the electronics store certainly had a lot of nerve trying to talk my Dad out of buying blank CDs in favor of buying a flash drive. The advantage of flash drives is portability, not economics. They weren't looking much cheaper than $8 a gigabyte on PriceGrabber for 1GB and up.

I've left out some options here which I didn't research as much. I've left out the venerable tape backup as well as internal hard drives and NAS. Internal hard drives in a RAID might be a good alternative to the single external drive because you get some automatic failure detection. And the NAS, or Network Attached Storage, goes one better in simplicity.

For this exercise, I've included in my personal backup plan ditching the idea of an "archive quality" medium such as tape in favor of any storage medium which has a long enough life to copy the data onto something newer.

Another idea is to build layers of redundance; one backup is not good enough, and maybe not even two. Somewhere in the back of my head in all this is my dear Grandma's judgment of disgust at the idea of computers, where you can erase everything at the press of a button. This information vulnerability is certainly an Achille's Heel of computing, and it's been an interesting exercise to price out and plan a moderately serious backup system. I'm certainly a novice to this, so if you have some experience to share, I'm all ears.

Adding value with programming languages and their tools

Languages of any kind are a two-edged sword. With the cooperative edge, you communicate. With the political edge, you control, using the utility of and dependence on communication to suit your interests. We who are involved in developing language processors and related tools can do well to keep these two edges in mind as we seek to add value with our work.

What is adding value

Clearly programming languages add value in terms of productivity. Even if we're fighting about the finer points of static versus dynamic typing, or Perl vs. Python vs. Pure LISP, almost anything is better than the truly low level of pointing and grunting the computer uses. For most purposes today a FORTRAN or C compiler is better than an assembler.

I've been dwelling on the proposition "programming languages as notation" (my wording) as summarized by Tablizer thus:

Most of the CS literature fits the pattern found in a typical book or chapter on Boolean algebra. Generally this is the order and nature of the presentation:

1. Givens - lay out base idioms or assumptions
2. Play around with those idioms and assumptions to create a math or notation
3. Show (somewhat) practical examples using the new notation or math
4. Introduce or reference related or derived topics

Step 2 is where the value is added, and it boils down down to a "math or notation". In my experience with programming languages, I'd reinterpret that as meaning

• math: give a mapping from one semantic domain to another (direct language translation)
• notation: programming language syntax
  

With that as a working definition, we can see analytically how the first FORTRAN compiler added value. It gave programmers a more familiar notation (syntax) in that they could write expressions ("FORmulas") with infix arithmetic operators and it provided the math (mapping) from that syntax to the lower level assembly or machine language.

We've been getting lots of new languages since FORTRAN, C and LISP. Justifications for new languages have included these ideas, "closing the semantic gap" between programmer and application. The hope is to achieve similar leaps of productivity as those initial leaps from the baseline of panel switches and card readers. It's the churn to gimme a better notation. The notation isn't the only thing of course, we want a decent mapping from the syntax to the semantic domain, to meet our performance requirements, and later, our software quality requirements: "let's eliminate whole categories of errors".

One curious result of the productivity leaps made by our newer languages is that they just make us want to write even bigger programs. We're not satisfied to write the old applications in an elegant, maintainable way using the new idioms and language features. We say "well, now we can do that project we only dreamed of," and proceed to applications with ever-increasing scope. (An entertaining take on this can be found in the Law of Leaky Abstractions).

With that said, I want to present a few ideas where language tools can be applied to the productivity challenge. These are opportunities for people with knowledge of compiler and interpreter internals but does not involve yet another new language. With acknowledgment of the irony in presenting these after just bemoaning application bloat, here are a few ideas for areas that could be improved in the practice of programming:

#1: Automatic refactoring support, like Danny Dig's refactoring engine which is included in Eclipse 3.2. This helps make progress in the old can't-refactor-because-it's-not-backwards-compatible problem by providing automatic tools to upgrade client code when shipping libraries with changes in APIs.

#2: Code visualization - tools to understand the static structure and dynamic flow of code . We can also improve in historical visualization, or how code has changed between revisions.

#3: Data visualization - Debuggers let you examine and even edit data structures at runtime, why not provide similar capabilities at code writing time? Why not let programs be specified by something like Emacs record-and-play macros where you "edit" the data structures? I think part of what makes programming difficult because it's hard to automate something without seeing what's being automated. While visual languages have not really taken off, I think it's an untapped potential, for some domains anyway.

#4: Automatic test generation - automatic creation of tests based on static inspection, to supplement hand-written tests.

#5: Machine-learning helpers for programming - instead of programming directly, use a tool that accepts examples of inputs and outputs and generates rules as starter code.

#6: Integration of coding environments with networked repositories - This means things like direct interfacing of language and tools help with team wikis to help share experience between programmers. Microsoft tools have started to do this, and there is room for growth. Instead of static coding style checkers definitions, what about augmenting them with networked information? Worse than Failure's Code Snippet of the Day could be formally encoded to flag catch ridiculous code patterns immediately.

#7: Easier ways to search code by pattern, not just textually, but by structure

These tools don't need to apply to new languages. As we found with AnnoDomini, a Y2K tool for fixing COBOL programs employing type theory and itself written using a functional language, tools can follow many years after their target language.

What adding value is not

We started by mentioning the two-edged sword of languages. The second side of languages, their use as a political tool, is not necessarily a way to add value, but it is to control. This phenomenon is studied in sociolinguistics, and you may have personal knowledge of it. What languages should be taught in schools? What languages can you get in trouble for speaking? What is the "official" language, or languages? What are the motivations for the proscribing or forbidding other people to use or not use a language?

While you think about that, let's talk about plugs and sockets. Let's say we make laptops. We will need to make a power supply, something that has two plugs. One is the AC side that plugs into the wall outlet using a standard connector. The other is the DC side that plugs into the laptop. For this DC connector, we have a choice of going standard, with an off-the-shelf standard size and shape, or going proprietary, designing a unique size and shape, possibly protected by patent. (For the record, I don't actually make laptop power supplies. So if they don't, for some reason, ever ever use standard connectors, I'd love to know, but I don't think it affects the analogy.) We can see the two edges of the language blade. As long as the DC plug and the laptops connector match, they cooperate, it works; the two side "communicate" power. The shape largely doesn't matter to the function of communication. But the choice of shape does affect what other laptops the power supply works with. By choosing a standard shape, we open the possibility of wider interconnectivity. By choosing proprietary, we restrict the possibilities. If someone needs a new power supply, now you have to come to us to buy it, or to a another vendor who licenses our connector patent.

The open vs. closed DC connector choice will sound like a familiar dichotomy from many technical fields, perhaps most familiar as the operating system question. I bring it up for the programming language audience as a reminder that any programming language represents a particular shape of interface, and we need to be aware of its potential for use as a political tool.

Back to human languages. We need to be aware of who is agitating for adoption of a language, and why they are doing it. Hear these motives with an ear for the programming language arena. "You may not use that language spoken by our enemies" emphasizes the enmity on a personal level. "You must now speak our language, vanquished foe" is part of a takeover plan. Language promotion can be sinister, as when members of an ethnic faction seek independence from a larger nation. They may appeal to ethnic identity as leverage to gain power, promoting the household ethnic language over the national one taught in schools. Language promotion need not be sinister. In an extremely fragmented situation it can achieve the communication productivity we spoke of before. Papua New Guinea is actively promoting Tok Pisin as a means of unifying its citizens who natively speak thousands of quite different languages, a situation created in part by its poor network connectivity. No, that's not it, I mean by the rugged terrain and isolated mountain villages. You may see language promotion taking place, and wonder what the language itself does for people. But we need to look for the bigger picture, which includes understanding who is behind the promotion and what they are trying to achieve. "You must use this language because you belong to this group" plays up uniqueness of the group, as with Finnish. Modern Finnish was developed in the 19th century as part of a growing nationalism to help distinguish it from neighboring Sweden and Russia. The emphasis on uniqueness, of preserving culture is interesting. Cultural forms involving language include expressions, wise sayings, jokes, stories, songs and poetry. These form a body of useful works. Preserving them is valuable for those steeped in the culture. You could say they have a vested interest in the culture.

Among other reasons, I think one reason languages are given away for free by businesses is that it helps spread adoption of the interface, creating a hook to create business in other forms. Once a customer has paid to develop working code in a language, it represents a vested interest in that language. We can think of the vast collection of COBOL applications as the prime example.

New languages have had it tough. In the 80's and 90's, without an absolutely killer feature, such as infix expressions when the alternative is per line arithmetic, new languages were being developed, but perhaps not rapidly. I'd venture that new languages are only good for new projects and new people (called "youth"). The rise of the web has brought with it a slew of opportunities for both those factors to coexist, giving rising to a growth in the number of available languages to use both browser side and server side. As the web matures, I see companies still trying to take advantage of the momentum to adopt new languages (*cough* Adobe Flex *cough*), and while it's exciting I'm starting to see the other side. Is it really a boost in productivity if we're retooling our businesses and schools to use a new language and environment every few years? Or are we moving sandpiles around, taking the same basic program (no pun intended) from one person's vested interests to another? Be aware of the other edge of the language blade. Adopting or creating a new language may not be adding value. Or if it is, you need to know, adding value for whom?

---

For more reading:
Cylindrical types of DC connectors at Wikipedia

Language politics at Wikipedia

Culture of Finland at Wikipedia

How TIOBE Says its Programming Community Index is calculated

Recently I mentioned and then featured the TIOBE Programming Community Index. This is one measure of the relative popularity of about 100 different programming languages.

As has been noted elsewhere, popularity is important to the user base of a particular language, because a popular language is an employable language. There are both considerations of vested interests ("I spent 2 years and $1400 in books, courses and conference fees learning C#") and personal preference ("I would love to write OCaml code all day long"). Unless you're a Paul Graham with your own business and the executive authority to decree All Will Be Done in LISP, you want to proselytize other developers to use your language, and monitoring popularity statistics is a way to track progress towards adoption of your pet language. (I should note that 'Siamese Fighting Fish' would make a great name for a programming language, especially if it was in a beta release.)

How TIOBE says the PCI is calculated

On the front page of the Index, TIOBE says

the ratings are based on the world-wide availability of skilled engineers, courses and third party vendors. The popular search engines Google, MSN, and Yahoo! are used to calculate the ratings.

However, there's a link to the "definition" of the PCI, and it says something a little different.

TIOBE rating comes from search engine result counts

If you follow the link in the sentence that says "the definition of the TIOBE index can be found here" (TIOBE loses points for web design as the definition page redirects itself if loaded outside a frame set), it explains how the various columns are calculated. The "rating" column is the one used to sort the table, and the one most people will quote when they say "Language X is number 12". It turns out this value is calculated from a cleaned-up page hit count for the search query

+"<language> programming"

Jobs, courses and vendors not mentioned in the definition

There doesn't seem to be a mention here in the definition page of those elements mentioned on the front page of "availability of skilled engineers, courses and third party vendors." The FoxPro community was kind of counting on that to be the case. I can't reconcile the discrepancy between the description on the index page and the definition given later. I've checked the TIOBE index via the WayBack Machine for 2004, the year of that FoxPro post, and the description and definition are unchanged in this respect.

"<language> programming" as a choice of search query

So now we all know what we need to do: stop writing about "programming in Ocaml", "OCaml hacking", and "writing a widget in OCaml", and starting writing about "making a widget through OCaml programming". I can see how this search query phrase might generate fewer false positives than using simply the language name (where "C", "D", "Natural", and "Logo" have an unfair edge), but is this really the best and only query to use? Fortunately TIOBE does a few more queries to fix up the results. They have a list of "groupings" and "exceptions", where searches for "J2EE programming" are counted towards "Java", and "3-D Programming" is excluded from results for "D programming". (There, I just artificially inflated some counts.) Still, I think there might be additional queries of different patterns that could be helpful.

Page hit count not well-understood

Technorati CEO David Sify wonders where the Google and Yahoo! page hit counts come from, if you can only view about 1000, and this thread is continued here in a discussion on blog.searchenginewatch.com. These numbers are also presented as an approximation, so I wonder how much these counts can be off and how that might affect the TIOBE rating.

Reproducing the PCI

Since TIOBE has given a recipe for the way the rating is calculated, it should be easy to reproduce it and compare the results to those published. The current recipe is a little unclear on the rating calculation, I think they're missing some parentheses, but the April 2006 description was a little more clear.

The algorithm seems to say the rating is an average of the ratings on each search engine, and the rating for a search engine is done by dividing the page hit count for the language in question by the sum of the page hit counts for the top 50 languages.

How do you know what the top 50 languages are until you calculate the ratings? Or do they start with the previous top 50? I also wonder how they handle languages not in the top 50.

So, if you have the inclination, you can try to recreate the TIOBE PCI and see what you get. I have great confidence I can do this quickly in my spare time with my favorite web scraping secret weapon, but for now, I leave you with this thought:
Logo (#27) beats out Haskell (#41), so all we have to do is add monads to Logo and we'll be cranking out MIT-bound schoolkids like there's no tomorrow.

Making the sale

We recently bought a car (see Web Security As it Really Is for the prequel). Good car, from a high-volume dealer. That means we talked with a salesman and arrived at a decision. Yesterday a lot of people made decisions, whether to stay at home or to go to the polls, and once at the polls, what to do with the little boxes or circles or touch-screen input areas.

Today I just am reflecting on how decisions get made, and the role of the salesman and his pitch in arriving at them. I am much more an engineer than a salesman, and sometimes I think I could work as an anti-salesman, somebody you bring along to a store or a car dealership so I can explain to you why you shouldn't buy something.

Malcolm Gladwell includes the Salesman as one of three important figures in his best-selling "The Tipping Point", talking about how messages get spread in an epidemic, explosive, exponential fashion. The role of the Salesman as I gathered was to be an emotional persuader, someone who incites people to act. The Salesman is an important role in society, but by nature he represents someone else's interests, not yours.

When we bought our car we researched long and hard. We read the illuminating "Confessions of a Car Salesman" on Edmunds.com. We were prepared to deal with the car salesman, and I think we did well. I put part of the credit for our success in all the hard work we did to research our purchase before we set foot on a dealership. What caught me off-guard was the business manager, who hit us up to buy an extended warranty after our hours-long ordeal inspecting, haggling over, and agreeing to buy the car that we had researched.

We didn't buy the extended warranty. But the pressure to buy was unexpected, and was palpable to an extent made possible, I think, by my lack of preparation for it. I knew the car in very definite terms, as a matter of fact we were familiar with the same make and model. The warranty was unknown to us, familiar only in vague terms an fuzzy feelings.

I'm afraid our voting decisions are more like my experience of trying to figure out whether to buy the warranty than buying the car.

A few years ago I attended a talk by someone who works as a political analyst. He presented some research from the 1994 Illinois governor's race between Jim Edgar and Dawn Clark Netsch. It featured a fascinating experiment where people are set in a room with a joystick that lets them indicate in real-time their approval of the candidate being interviewed on a TV screen in front of them. What we watched in the talk then was an averaged graph of people's approval over time superimposed over the same video clip they watched. As soon as Netsch mentioned something about not supporting mandatory tougher sentences on a certain type of criminal conviction, viewers' approval started to drop steadily. Netsch lost the election partly due to her stance on mandatory tougher sentences. What was interesting was that when you hear Netsch explain herself on the issue, it was not as clear a difference between her and Edgar. She objected to the mandatory part of the proposal. She reasoned that by making the sentence mandatory it takes away an element of flexibility from the judges actually present and hearing the particulars of a case. She wanted the judges to retain that flexibility. You can argue with me about whether this makes Netsch "soft on crime" as the voters apparently perceived, but this subtlety in the debate was lost on me when I was watching those commercials back in 1994.

Relying on TV commericials was not a good way for me to learn about the candidates.
The salesman, and I include here the architects of political messages, are not at heart interested in reasoned debate about their product. They're interested in getting someone to make a decision.

I heard an upper level software manager relate an imaginary conversation with a customer about a certain usability issue. As a developer, I wanted to help that customer with the usability issue. But the manager's position was summed up in his question "Come on, is this issue going to stop me from getting the sale?"

Decisions play a vital role in life. But we've got to realize that when it comes time to buy or vote, you better have done your research outside of the arena where salesman are pitching and people are trying to get you to make a decision in their favor. Because as my wife observed when told that the dealership doesn't fall in love with their cars, they want to move them: "yes, but we have to live with this car after the sale."

Of teachers and hoopsters

Let me say right off that I think teachers should be paid more.

But I'd also like to challenge an oft-spoken notion that we value sports players economically more than teachers.

What has this got to do with this technology-based blog, you ask? Well, we here at JFKBits value education, and we value sports. And we also wrote a program to arrive at our featured statistic. So, there you go. On with the show.

Have you ever thought about teacher's salaries on a very small scale? For example, have you ever thought what it would take to go in with some other parents and start your own school? How economically feasible would it be? Looking strictly at the salary cost of a school, ten families hiring one teacher for a school year might need to split a low $40,000 salary into a $4,000 per family share. A K-12 school with equal numbers of students per grade works out the same way, each grade level is essentially a one-room school where each child's family splits the teacher's salary costs, plus some fraction for administrators.

The basic idea here is the cost of supporting a teacher is proportional to class size. Or, how many customers can a teacher provide services to within a given time period.

This ratio is far different for sports players. Far, far more spectators derive entertainment per individual basketball player than children can be effectively taught by a normal human teacher.

What I wondered then was if we treat the relatively few sports players as members of our larger community, as we automatically do teachers, and divide the aggregate amount they get paid by the size of the community.

To answer that I consulted USA Today's page on NBA salaries for the 2005-06 season. I could have added the table up manually, but instead spent a few minutes using a prerelease version of my company's software to scrape the web page, get the numbers, and add them up. I was glad I did this when I later realized that I wanted to re-do the calculation for a different season.

My computer-enabled calculations tally the total NBA salaries for 2005-2006 to be in excess of $1.6 billion. Whoa, that is certainly a big number.

But so is 300 million, a number which made the news yesterday as the current estimated population of the USA. I figure that means the entire NBA payroll for last year could have come by having each US resident contribute about $5.50.

In other words, you might say that economically, we value basketball players in this country to the tune of $5.50 a person.

You think we value teachers a little more than that? I think so. Remember our earlier estimate for supporting your own private school was in the thousands of dollars per family.

Picking a number from near the top of Google's search results for "total teacher salaries", I get an estimate of $140 billion spent on K-12 salaries in 2002-2003. Using an estimate of 280 million people in the US in 2002, that's an average contribution of $500/person nationwide to teacher salaries.

I reran the basketball analysis for 2002-2003, to compare on even basis with these education numbers, and found that the NBA salaries are actually right about the same. The population was lower in 2002, so we get an average contribution figure of $5.92 per person towards NBA salaries.

By that comparison, teachers were about 84 times more important, economically, to us than basketball players in 2002-03. You could add in football, baseball and other sports to even out the comparison. But I think this is a nice way to think about the comparison.

One way of understanding the difference is the scale of service provider to customer. A sports player reaches a large audience of paying customers while one teacher can handle only 20 or 30 students, which is more or less related to salary. But this ignores the question how much sports do we consume as a nation compared with the amount of education required? I think these figures of total expenditures, or dollars per US citizen, help to combine the answer to that question with the ratio of service provider to customer. While not everyone buys basketball tickets or merchandise or greatly influences basketball advertising budgets, closer to everyone, on average, requires the services of a teacher.

My conclusion is that I find that in 2002-2003, we spent 84 times more money on teacher's salaries than NBA player's salaries, and I think that's a pretty powerful argument that we value education more than basketball. An individual teacher makes less than an individual basketball player, but we already knew that due to the nature of teaching and classroom sizes versus the nature of spectator sport and the size of arenas and TV audiences. Teachers should still be paid more, but I hope this will reassure somebody that schooling is still ahead of hoops.

Web Security as it Really Is

I've been reading "Web Security: A Matter of Trust", the Summer 1997 issue of O'Reilly's "World Wide Web Journal". Wanting to understand the magic behind https: a little more, I read "Cryptography and the Web" by Simson Garfinkel and Gene Spafford, and "Introducing SSL and Certificates Using SSLeay" by Frederick J. Hirsch.

My knowledge of SSL and web security could be told like this. Surf to the front page of a site you use securely, http://example.com. You fill out your username and correct password, click "Login" (or hit tab-space, if you're a mouse avoider like me), and voila, you're securely surfing the https://example.com space.

Meanwhile, my wife and I have been looking to buy a car, and I noticed something interesting about a popular car research website. I was logged in and happily doing my research, and happened to glance up at the URL. It started with "http:". With Ethereal, the free network sniffer, capturing packets, I re-accessed the front login page and examined the output. Sure enough, there was my email address, which the site uses for usernames, and my password, in very clear Courier New text. In crafting an email to their customer service, I came up with the following list of pros and cons for not using https: on this particular site:

1. Pro: An attacker who captures your credentials and logs in can't really do much that reflects badly on you. They can do car research, and I guess they can send nasty email to customer service. The site actually does use SSL when you originally order the service and use your credit card.


2. Pro: For such a popular site, maybe the server overhead of using https: on all the information queries is too burdensome


3. Con: Since the usernames are email addresses, they can be collected and sold


4. Con: Some people use the same password everywhere, so it still compromises a user's privacy and security to send it in the clear. Knowing one password may help gain entry elsewhere, especially when a username in other systems can be guessed from the email address used on the car site

This experience wasn't so bad. Plus, I like to be trusting, it wears me out to be suspicious of everyone and everything.

Then my heart skipped a beat when I noticed the same phenomenon happening with Google Mail. My wife had mentioned that she thought Gmail has an http: address after you log in, but I didn't think it possible (my faith in Google at work). But when I checked, she was right.

Ethereal was quickly called to the scene. I was horrified as the sniffer trace revealed email addresses of my quick contacts appear in clear text. But that ended up being all I found. The sniffer trace was loaded with SSL and TLS traffic. Google's extensive AJAX programming is apparently moving the sensitive pieces, such as credentials and mail content, via Javascript calls to https: addresses while keeping the main page framing the content as quickly-served static clear http.

A nice design, very economical, securing only what needs to be. Indeed, why waste Google server time encrypting who-knows how many copies of the same Gmail page with unique keys? Still, it seems to be moving in the wrong direction when a user needs a network sniffer, or an HTML/Javascript code inspection to find out what is secured and what's not. Remember, Google is not securing a list of email addresses found in my contacts list. I wonder if that's how my Gmail address finally got compromised to the spammers?

Kingdom of Nouns Response #1

My good friend recently mentioned the Kingdom of Nouns essay, hereafter referred to as KofN, which sparked a flurry of discussion back in March 2006.

Being both a Java programmer and a functional language programmer (I liked to tell people I wrote more lines in Standard ML during college than in any other language), I found the essay provocative and interesting. Rather than responding in kind to Steve Yegge's 3,164 word essay, I've decided to post little thoughts, this being the first.

My overall response to KofN is that Steve is right, Java is lacking in the way it works with functions, but his reasons are wrong, and the remedies he gives are not quite the remedies needed.

---

Nouns, Verbs, and Computation

Let's talk about where the nouns and verbs come from, first. After all this is programming we're talking about, and not linguistics. While linguistics, the study of human languages, often has compelling ideas that can be applied to computer languages, the terms noun and verb in KofN come from design analysis. When designing a new system you break down your problem domain by thinking of its nouns, and make them correspond to data, and then the verbs which become functions or code. Simple, nouns=data, verbs=code.

The basic statement from KofN I want to examine now is this one:

Nouns are things, and where would we be without things? But they're just things, that's all: the means to an end, or the ends themselves, or precious possessions, or names for the objects we observe around us. There's a building. Here's a rock. Any child can point out the nouns. It's the changes happening to those nouns that make them interesting.

I really struggle to find the point of this paragraph, the "topic sentence" as the writing teachers say. It has plenty of insinuations, like "any child can point out the nouns", which help build the emotional impact. But I think the point is that nouns are not interesting by themselves, and that they're the means to an end. Very well, if that's the point, then let me respond by invoking the simplest description of computation as

1. Receive input


2. Do some calculation on the input


3. Emit some output

This doesn't say anything about functional vs. procedural, about programming language, this is how computers work. At every clock cycle (or for analog computers at every instant in time) the computer is the embodiment of a function, turning inputs into outputs. If it's a desktop or server, maybe it's sitting in an idle loop waiting for input. What people care about, the whole reason the computer is there, is the output, or information.

But KofN states "nouns are a means to an end", but I think it's the other way around. The end of a computation is the output, and the means is the function. If the point in bringing this up is to promote verbs in relation to nouns, I think the case is overstated.

In light of the above model of computation, I think the programmer's job can be phrased as modeling real-world nouns as computer data structures, and implementing the functions, the verbs, that transform those data structures. Both are important, of course. But I don't think nouns are uninteresting to the end user, and I don't think the "end" of a computation is the code, except maybe for the computer itself, if it subscribes to the life-is-a-journey-not-a-destination philosophy.

The question at hand in all this is how a programming language should be designed to help programmers get things done. The point of KofN seems to be to call for the option of invoking functions as f(x,y) rather than forcing f to be "owned" by the noun x, as in x.f(y). There's certainly a strong precedent for invoking functions as f(x,y). The static methods in java.lang.Math bear testament to that.

With that thought I leave you til next time.

Alphonse Bertillon and 19th Century Databases

Earlier I described an idea that I called the Software Provenance Database. The essence of the idea is a database of information on how to tell what version of a piece of software you're using.

I didn't like the name though, and remembered reading about Alphonse Bertillon, a French police employee who in 1882 presented "anthropometry", a technique of identifying a person based on body measurements and other observations of unchanging features such as scars.

Hence I'd like to consider using Messr. Bertillon's name or story for this database.

Further, I wanted to add that this database certainly should not be limited to determining versions of an executable. The idea is to gather in one spot any information about how you can dynamically determine the state of your computer, and this can include configuration information (how much memory is installed?) and data schemas or file formats.

If you have a few moments, it would be interesting to read this account of how Bertillon's method works, and to think about a database search which involves no computational machinery. Notice his method for partitioning the records into equal portions, for example to make the search quicker. It is to me a good example of what I think Dijkstra meant when he said "Computer Science is no more about computers than astronomy is about telescopes."