Brad Burnham has a great post (which I found via Fred Wilson’s blog) about what’s coming next in the inevitable progression of the computer industry. He argues that we’re in a world where we no longer care about hardware or software but rather we care about data (Think Facebook, Craigslist). After data, he suggests that governance (how to get data that delivers the most utility for the least overhead) and values are coming next. He makes some very compelling points and I highly recommend the post.
One area where I might put things differently is to suggest that at any given time, we care about different parts of the stack in different sectors. For example, the iPhone, computers in cars, connectivity on airplanes. There are a number of sectors within the industry where the data we’re trying to get to is known and the value is in the boxes and pipes. It rarely stays there though - I completely agree with Brad in this regard - value shifts up the stack quickly. As soon as the hardware and software become transparent, the data and it’s quality become the only things that matter.
Anyway, that’s neither here nor there - this is a post about governance. At Judy’s Book, managing the quality of user-generated content is something we’ve been talking about lately and Brad’s post really hit home.
The genius of Craigslist is in its governance system. It is its lightweight governance system that allows 21 people to administer 300 sites in 35 countries. I believe that the basis of competition in web services will shift from the data to the system that manages the acquisition, and use of that data. The governance system that yields the most utility for the largest number of users with the least overhead will ultimately manage the largest communities with the most valuable data.
I also came across an article on Reddit a few days ago which talked about the early days of HotOrNot.com and some of the scaling issues they had to deal with. One section in the article focused on how they created a system for editors to quickly determine whether a picture was obscene or not.
Scaling the Human Element
We had another problem with some users submitting pornography and other inappropriate photos. Initially, we decided to solve this by adding a link under each photo that said, “Click here if this picture is inappropriate.” If a photo received enough clicks, based on a formula we had derived, the picture was removed.
This worked pretty well, but not well enough. I sent the chairman of a large advertising network a link to our site with a note proclaiming that: “The odds of getting an inappropriate picture are extremely low.” Ten minutes later, I received his reply: “Unfortunately, the first picture I saw was that of a topless woman.”
He informed us that if we wanted companies to advertise on our site, we’d have to filter each picture as it came in. Jim built an interface for us to do so. However, we soon realized that we couldn’t spend all day screening pictures. The system’s human component wasn’t scalable. That’s when we arrived at the moderator idea.
We decided to build a system in which moderators could vote on whether to approve or reject a picture before it passed on to the main site. If a picture got enough votes either way, it was approved or rejected. By making the decision collective, no single moderator could approve or reject a picture independently.
To help detect any rogue moderators, the system tracks each moderator’s accuracy. A vote is counted as wrong when the moderator’s vote goes against the final outcome of the picture. For instance, if one person votes to approve, but all others vote to reject, the one person is wrong. Moderators whose accuracy ratings drop below our threshold are kicked out.
We decided to take the moderator system one step further by adding security levels. The higher a moderator’s security level, the more his or her votes counted. We also gave higher-ranking moderators special privileges, like an expert mode in which they could judge pictures much faster. We gave the highest ranking moderators the ability to reject or accept moderator applications, and the ability to kick out rogue moderators. Today, these top-level moderators essentially run the moderator section of the site and decide on the specific guidelines for what makes a photo inappropriate. More than 1000 moderators are currently active, and they form our strongest community.
It seems clear that some combination of technology and human input is needed to create a system that is reasonably scalable without having to invest a crippling amount of money in a pure technology system. One of the problems with simplistic algorithms (’x’ votes, ‘y’ comments, user rating over ‘z’) is that it’s incredibly hard to tune.
Another question to consider here is the notion of democracy versus editorial and how to create a balance within the two. A pure democracy is hard to achieve online - given that there will always be bad actors in the system and that the good actors will have varying levels of engagement (think about the 80-19-1 rule) - you will always need some measure of editorial control. This could be via a group of users who are engaged and are given tools and special interfaces to control content. Even Digg, sometimes the poster child for democratic content control puts a fair amount of editorial power in the hands of power users. Pure editorial seems great but has one major problem - it’s hard to scale.
The truth of the matter is that different strategies will work for different businesses. One thing is clear though - if you’re in the user generated content business, finding a governance system that will let you maintain the quality and tone of data you want in a scalable fashion will be a competitive advantage. Presenting new users with information they find valuable earlier in their interaction cycle makes them that much more likely to stick around, participate and contribute their own $0.02.
I realize I’m not directly debating Brad’s point here, but it got me thinking, which in my mind is the hallmark of a great post. There’s always something coming next and that’s something I love about this industry.