Blog » Performance Economics
Posted on 1235217632|%A: %d %B, %Y|agohover
Another cooked-up performance "controversy" is bubbling on the haskell-cafe mailng list: is gcc a "better" compiler than ghc? The question hinges on the performance (speed) characteristics of a relatively small, simple chunk of code when run through the respective compilers. It seems gcc makes faster code in this isolated case. I can't follow the technical details, but I do know that this kind of "performance" comparison is pointless and probably harmful.
Such comparisons totally miss the point by misunderstanding the notion of performance. Performance is not about time and/or space efficiency, it's about economic efficiency, and the speed of compiled code is only one factor in the economic equation. Your compiler can produce the fastest code in the world, but it won't be of much use if it isn't correct. So any calculation of return on investment must factor in the "cost of correctness".
Assuming correct code and correct compilation, a compiler that produces faster code offers better Return On Investment. But every programmer knows you cannot make such assumptions, so an investment decision based solely on speed benchmarks is a poor decision based on incomplete information.
In any real-world project of even modest complexity this means a performance assessment of a toolset like C++ and gcc must account for all the effort involved in "proving" correctness. Since proof is not possible, this means not only the cost of testing, but also the cost of fixing bugs. Testing can never prove the absence of bugs, so all such code, even when published as production quality, is essentially experimental.
Hence functional languages have a major advantage in performance economics, since the cost of correctness drops considerably. We can never prove that a piece of software correctly addresses a customer's problems, or that the problem specification is accurate, but functional languages go a long way towards guaranteeing correctness of the code itself. And if you've proven the correctness of the code, there's not much point in making major investments in testing what needn't be tested. Some testing or validation will always be required - e.g. design of the user experience - but most expense involved in establishing code correctness is incurred up front in the effort it takes to prove correctness. Which in principle should be a lot less than the expense of testing and fixing code written in imperative languages.
Moore's law is a law of economics, not a law of nature. Engineering is essentially about economics. I would bet there is a similar law waiting to be discovered about the economics of programming, since functional languages (along with logic languages and some other idioms) promise to turn programming into something resembling real engineering. Haskell code has certain mathematical properties that the equivalent C/C++/Java/etc. code cannot hope to have, which means it can reliably be analyzed and transformed based on mathematically principles rather than heuristics. New methods of optimization are bound to arise that will be unavailable to imperative language compilers.
It's inevitable that some day functional language compilers will produce code that even the best hand-rolled assembler cannot match for speed and space efficiency. When that starts to happen and the economics of functional languages - or rather, languages with some kind of support for provable correctness - become more clear, it wouldn't be surprising to see chip designers revive the idea of machines designed for reduction optimization.
In any case, the point is that people who want to make the case for Haskell or other functional languages should not allow themselves to be suckered into discussions about performance isolated from other economic factors. Benchmark code should not use simple algorithms whose imperative implementations involve minimal side-effects and can thus be reasoned about informally with little effort. Instead they should involve a well-defined problem, as simple as possible, but one that requires the kind of non-trivial reasoning that complicates the life of an imperative programmer. Something that makes cost of correctness as much a part of the benchmark as space/time efficiency. And such benchmarks should always be called "performance economics benchmarks", never just "performance benchmarks".
Like this entry?
Leave a comment
I like your thesis.
The "ghc vs. gcc" discussion is silly, and flame wars derived from such a comparison perpetuate harmful patterns of thought. Maybe it is possible to compare two C compilers objectively and say that one is better than the other, but you'd need a lot of evidence. But comparing a C compiler and a Haskell compiler is like comparing Babe Ruth to Pele.
gcc when given input A produced code which ran in less time than ghc when given input B. But we have no theory telling us that A and B are the same program; simply that they print the same number when run. But I can write C (and Haskell) programs which are much faster and much slower than that benchmark, which print the same number.
This difference isn't purely academic, however. I bet that I can write a lambda calculus interpreter in Haskell which a C programmer would be hard-pressed to compete with. People want fast lambda calculus interpreters; such cannot be said for programs which print out some arbitrary fixed number.
I hate language shootouts, benchmarks between languages. Speed is not a property of a language, it is a property of a program. Unless you can show that two programs are the same (intensionally!), comparisons are meaningless. And even if you do show that they are the same, your comparison has correctly compared the speed of the compiler for one input.
I don't know how popular computer science has adopted such a nonscientific culture.
(Oh, also, whose bright idea was it to restrict the website field in your blog to 30 characters? :-)
Hi Luke
"But comparing a C compiler and a Haskell compiler is like comparing Babe Ruth to Pele." I was thinking more along the lines of a chain saw and an Intergalactic Telefunctor. ;)
"I don't know how popular computer science has adopted such a nonscientific culture." I'm willing to bet the marketing department had something to do with it. Just a hunch.
It would be useful to have a bona-fide economic analysis of the costs/benefits associated with the two styles of programming, but you'd have to have a combination economist/computer scientist to do it. For example, testing and bug fixing has a direct cost, but also an indirect cost of lost opportunity - time spent hunting down and fixing bugs could be spent improving the product. It would be interesting to see real (or at least principaled) numbers. I suppose eventually, once the real money starts flowing towards FPLs, we'll see something like that.
Sorry about the 30 character thing. I'll file a bug report.
Thanks,
-gregg
Post preview:
Close preview