Did immigrants (and Simpson's Paradox) cause the Great Stagnation?

Many people have commented on the fact that, after adjusting for chained CPI, median income has not risen significantly since the 1970's. Tyler Cowen points to this as evidence for his theory of the "Great Stagnation", which purports that the economy has grown more slowly during the latter parts of …

more ...

MapReduce explained in 41 words

Goal: count the number of books in the library.

Map: You count up shelf #1, I count up shelf #2.

(The more people we get, the faster this part goes. )

Reduce: We all get together and add up our individual counts.

more ...





Hadoop's MapWritable sometimes a performance hog

I've been using Hadoop a lot lately for a stealth mode project I'm working on. One of the big lessons I'm learning is that where medium to big data is concerned, data formats matter a lot. Where small filesizes are concerned, there is little harm in slinging around JSON objects …

more ...

Structural Shift in the Economy

It's becoming a more or less accepted economic theory that a large part component of our current recession is structural. This means that during the housing bubble we had lots of people in housing related jobs (construction worker, realtor ), but once the bubble burst those jobs ceased to exist, and …

more ...

Hadoop error - HTTP Response Code 503

I recently had a power failure, which resulting in my hadoop cluster shutting down. No matter, hadoop came back after a little while.

However, I ran into problems immediately after restarting it:

1
2
3
4
5
6
7
8
$ hadoop fsck /
Exception in thread "main" java.io.IOException: \
                    Server …
more ...

Inanity of overeating - don't ignore the bacon in the room

I recently saw an article by Gary Taubes get posted to Hacker News. He pushes the curious thesis that conventional wisdom on obesity is somehow incorrect since it fails to explain why people choose not to put down the chips.

He begins with some straightforward calculations:

When you're in your …

more ...