This weekend’s New York Times Magazine features a story about competitors in the Netflix Competition — an ongoing challenge to improve Netflix’s movie taste prediction system by 10% with a $1 million prize. After a couple years, some people are really close; the top team has improved on the system by 9.44%. Each step closer to 10% makes it harder to improve one’s algorithm.
Here’s where it gets interesting: the remaining hurdles are more or less identifiable. There are limited number of movies for which it is immensely difficult to predict someone’s reaction. ”Napoleon Dynamite” is the big one the article identifies. Apparently, “ND” alone accounts for approximately 15% of all the unexplained variation. That’s huge! Unbelieveable!
In describing the top 25 most difficult to predict movies, NYT staffer Clive Thompson writes, “I noticed they were all similar in some way to “Napoleon Dynamite” — culturally or politically polarizing and hard to classify, including “I Heart Huckabees,” “Lost in Translation,” “Fahrenheit 9/11,” “The Life Aquatic With Steve Zissou,” “Kill Bill: Volume 1” and “Sideways.””
That’s true, isn’t it? Those movies are culturally polarizing, but it’s more than that. They are culturally polarizing and, yet, mainstream. An even more polarizing movie like Von Trier’s “Dogville” is actually easy to predict because the type of people who see a movie like that tend to be of one mind. But who didn’t see “Lost in Translation”? What confuses me is that on a gut-level I know that those movies are culturally polarizing, but I can’t, for the life of me, explain why “The Life Aquatic” would be. Why is that?
No, I’m not talking about Allen Iverson. Here is the solution to our entire energy crisis.
A quick addendum to my earlier post …
A great speech at Cal-Tech’s commencement by NPR‘s Robert Krulwich on how to translate science for the public and the importance of doing so. Even though he speaks about natural science, of course the same should be said for social science.
I went to a sociology conference last week that incorporated many applied sociologists as well as educators at community colleges and other teaching institutions. In a session regarding public sociology, one speaker was absolutely thrilling. He began with the mundane observation that too many sociologists today are in highly specialized trenches and are only willing to speak inches away from our empirical evidence. Because of this, we’ve lost the tradition of bold public intellectuals (e.g., John Galbraith) who wrote “750 page Op-Ed pieces.” We’re too bogged down in jargon and small ideas, said the speaker. To explain only the fibula, rather than taking a stab at explaining the whole body of society is not only overly-cautious, it also keeps us from engaging with intelligent, interested non-scientists. To re-engage, we need to write Op-Ed pieces in newspapers, make public speeches, and write books sold in popular bookstores that make bold claims. We cannot be afraid to take risks. At the same time, we need to be honest and be careful thinkers even as we mount arguments (“people can sniff out dogma”). The speaker held out Contexts as a shining example of what sociology should be (“if it ain’t well-written, it ain’t gettin’ published”).
As someone who firmly believes that sociology has insight to offer our sociology, this argument rang all too true. The ensuing conversation pointed out that this won’t happen until the tenure and promotion structure reward these public activities. Moreover, several critics chastised this form of bourgeois public engagement; Op-ed pieces are minimal activism and not a form of direct service. Nonetheless, I believe that this is a course worth pursuing, alongside service-oriented public sociology.
I was catching up on back episodes of the great radio show On The Media (OTM to insiders) this weekend. In this special episode on investigative reporting, long time 60 Minutes producer, Lowell Bergman (Al Pacino’s character in The Insider) made the intuitatively reasonable sounding claim that as money for investigative journalism drys up, most Pulitzers are increasingly going to a small number of elite newspapers (New York Times, Washington Post, L.A. Times).
But is this empirically true? A quick perusal of recent Pulitzer winners suggests the following:
1. It’s not that elite newspapers win most of the prizes. It’s just that the Washington Post wins all of them.
2. Beyond the Post, most of the winners are newspapers with a circulation above 250,000, perhaps unsurprisingly.
3. Though the majority of the awards are won by big papers, small newspapers do win things with some regularity. And it’s not just the same few small papers over and over again.
I wonder if a longer term analysis would show consolidation of Pulitzer prizes by elite newspapers in recent times. Contrary to Bergman’s claim, I actually suspect that while the names shift over time, Pulitzer has always been dominated by a small number of elite newspapers. Perhaps the competition was just tighter in the past. Has anybody done research on this?