Starting Up by Rahul Pathak

Amazon is doing a great job of highlighting relevant content on their user review pages. This is something we strived for at Judy’s Book but never quite nailed.

When there is a profileration of content, people want to know what’s most relevant. This is a natural response to information overload. On a site where there are a lot of reviews, you want to know which reviews you should pay attention to. (At this point, the great Princess Bride quote: “You truly have a dizzying intellect,” is probably coming to mind but bear with me.)

At Judy’s Book, we typically listed reviews in reverse chronological order and displayed the TrustScore of the user. This was a useful proxy but it didn’t really capture whether other people found the review helpful. It wasn’t a true content quality score. Rather, it measured the credibility of the individual posting the review. This is useful and valuable, but doesn’t really address the review. I was trusted on Judy’s Book, but I know nothing about kid-friendly restaurants. There’s no way I could have written useful reviews in that domain.

The challenge with trying to get a quality assessment is that the percentage of users that will rate something is relatively low. As a result, you need a lot of traffic to make meaningful assessments.

Amazon nails this. Their product review page does a fantastic job of summarizing what’s relevant for a potential customer.

They don’t just show the average rating, they show the distribution. This isn’t a new feature. What I love is they highlight the positive and critical reviews that users have found useful. This is awesome. It let’s you see at a glance the most relevant information and gives a user comfort that they are getting the information they need to make a decision.

I love that Amazon displays the critical review as well. This has two big benefits. First, it feels authentic - no product is perfect; seeing the good and the bad lets you evaluate whether its failings are ones that will bother you. Second, even if a customer doesn’t buy this product, they will view Amazon as a place that makes it easy for them to make a good decision. They will either buy something else this session, or they’ll come back in the future.

This is a great example of taking a long view about customer satisfaction and hats off to Amazon. I’ve blogged about them before and remain impressed. Another area where they are doing a killer job is web services, but more on that later.

Related Posts:

• Jeff Bezos Interview in Harvard Business Review
• How Amazon Does A/B Testing

Chris has just started blogging and he has a great post up on his blog about the importance of taking smaller bites when starting a company.

…one big takeaway was a collective resolve to build our next business by “taking smaller bites”, working toward a vision by executing more thoroughly around discrete building blocks of the idea.

I completely agree with this perspective. Even if you’ve got a grand vision, you have to execute like crazy in one small area. Once you matter a lot to a group of people, it becomes easier to reach out to more. Trying to matter to everyone at once is a tough game.

Great post Chris and welcome to blogging. I’m looking forward to reading more.

One of the real-time lessons we learned at Judy’s Book is that while UGC is great, without vigilant moderation of that content, you’re going to get a lot of noise in the system, which is not great. When we were aggressively seeking reviews, we relied exclusively on the community to flag posts and then reviewed those posts for compliance with our terms of use. This approach is ok, but doesn’t go far enough. We should have looked at every review and only approved content that was of value for review readers.

When we transitioned to deals, we adopted the same - anything goes, we’ll deal with spam - approach but quickly found that deal posters were flooding the system with deals that from our perspective were questionable. The problem was compounded by the fact that with deals, there is a financial incentive to posters - the more of their links they publish, the more likely they are to get paid. Initially, we tried to combat the problem with algorithms that gave higher preference to known posters, and more aggressive deal removal tactics. This helped, but we constantly felt like we were behind the curve.

The solution was a simple re-framing of the problem. Rather than let anything onto the site and then remove bad deals, we decided to only allow good deals onto the site. The way we did this was by creating a moderation queue. Anybody could post anything, but before anything came onto the site, it would have to be approved by a member of the JB staff. This simple decision completely eliminated our spam problem. A simple, streamlined interface made it quick and efficient to review deals and we saved ourselves a great deal of frustration dealing with bad deals on the site’s primary pages.

If you’re thinking about incorporating UGC into your site, I’d strongly urge you to consider a moderation queue approach. While you do sacrifice content velocity and incur a certain amount of overhead due to the review process, you gain a great deal in terms of content quality and less policing of the site’s content pages. While I can envision a few scenarios where quantity should come before quality, they are rare. When in doubt, err on the side of what will generate the higher quality user experience.

Dave has a great post up about transitions and he talks about folks at Judy’s Book going above and beyond the call of duty to make sure that work gets done and things get fixed, even as they were moving on to other things. I’m in violent agreement with Dave on this one. People have been fantastic and I have nothing but great things to say about the team.

Andy has a post up on his blog about the decision to scale back operations at Judy’s Book.

Today was a tough day. For the second time in my life I had to tell a great team of people that the idea they’d worked so hard on was going away. After 3+ years, our management team and board of directors has decided to scale back our operations at Judy’s Book and seek a strategic acquiror.

As a CEO, I know this is the right thing to do for our investors. But as an entrepreneur it’s disappointing to stop chasing an idea just when it’s beginning to take root in the popular consciousness.

It’s been a crazy day but Andy’s done an amazing job helping the company deal with a difficult situation.

Last Friday, we had an important release at Judy’s Book which included several new features. Some visible to consumers, others more in the back end, but all of which should lead to higher quality deals on site.

The primary elements of this release were:

• Deal Approval Queue
• User Posted Local Deals (supported by Local Editors in select geographies)
• Map Improvements
• Fall Promotion Support 

You can see some of the results of this work in Seattle, New York, Atlanta and Chicago.

The entire team came together and cranked to make this happen in a very short period of time and I’m really excited by how things turned out.

For those of you who are interested in the gory details:

Deal Approval Queue

The intent here was to provide a way to support UGC but to enforce quality standards. User contributed deals are of no value if they are spammy, irrelevant, miscategorized etc. The approval queue allows us to solicit content from users but provides a mechanism for us to ensure that only high quality deals make it onto the site. All user posted deals go into the Deal Appproval Queue and remain there until they are explicitly approved. The system also stores the history of approved and rejected deals so we can use it to train spam detection systems.

Content quality is a critical part of the user experience (something Erin & Chris have pounded into me - thanks guys) and you need to be very focused on keeping it high.

User Posted Local Deals

To this point, we had supported adding online deals to the system but we didn’t have a public way for users to contribute local deals. One aspect of the local deal post process that creates complexity is the fact that you have to provide a way for users to add business listings without creating duplicates in your data. In addition, no one wants to type in a complete address, so you have to facilitate search. Oh and by the way, you need to prevent people from spamming your listing database. (Dave has a great post on some of the fun things that happen in local listing spam.)

The Deal Approval Queue was the strategy we came up with to control the flow of data into the system. Because we had a mechanism for screening content, we wanted to make it as easy as possible to participate. This led to us to move from a model where posting was for Judy’s Book members only to a model where all you needed was an email address. If at some point in the future, you sign up for membership, all your deals will be available in your profile.

In addition to supporting user generated content, we’ve been actively recruiting local editors in select geographies to ensure a high quality base of local content. Erin’s written about this on the Judy’s Book blog and I’d touched on the subject earlier in a more generic way in my post on aggregation and UGC.

Map Improvements

The map now displays stores with the most popular local deals in any given category. In addition, you can pan and drag the map and it will continually update in order to provide you with a list of the best local stores that it can.

Fall Promotion Support

We’re currently running a daily giveaway where users can win a $25 gift certificate to a local boutique of their choice and there’s a $1,000 grand prize at the end of the promotion. We needed to put in a significant amount of work to support the marketing efforts around this promotion. The goal was to provide very prominent placement without being intrusive or degrading the user experience for those who weren’t interested in the promotion.

I think the team did a great job coming up with an implementation that works. If you come to the site looking for the promotion, or click on a promotion call out, we use javascript to grey out the screen and present you with an overlay that provides a streamlined way to participate. In addition to signing up with your email address, if you post high quality local deals, you gain additional entries into the giveaway.

The user flow around the promotion is quick and easy when you step through it (as it should be). Features that are easy to use, however, are typically features that required a great deal of thought and effort. This implementation was no exception. In order for something to feel easy for a user, a lot of people put a lot of work in behind the scenes.

Measuring consumer behavior and watching consumers interact with your site are the best ways to find out what works and what doesn’t. There’s a great post on Signal vs. Noise about some of the things behind Amazon’s success. Two paragraphs that jumped out at me:

Use measurement and objective debate to separate the good from the bad. I’ve been to several presentations by ex-Amazoners and this is the aspect of Amazon that strikes me as uniquely different and interesting from other companies. Their deep seated ethic is to expose real customers to a choice and see which one works best and to make decisions based on those tests…

…This is done with techniques like A/B testing and Web Analytics. If you have a question about what you should do code it up, let people use it, and see which alternative gives you the results you want.

This data driven approach is especially valuable for a company in Amazon’s position where small changes in things like the click-to-sale rate could have a massive dollar impact. Even if you’re not in a position to do testing at this scale (or don’t have the traffic to make it meaningful), watching individual users is still incredibly valuable.

In an ideal world, you’d be able to do both - measure the aggregate and observe the individual.

P.S. If you find this mythical ‘ideal world’ tell me how to get there.

Related Posts:

• Designing and releasing a good product is hard
• How Amazon does A/B Testing
• How to Make Your Site Better
• User Engagement - Measuring the Right Things
• Iteration Speed vs. Iteration Quality (via Coding Horror)
• Usability isn’t as Hard as You Think

Sorry for the recent rash of local deals posts, I’m just excited about the direction we’re going in. I was just checking out the Seattle Deals page today and came across information on a sale at Ian, a clothing boutique in my neighborhood. I had no idea this was taking place. It’s really cool to benefit from the value proposition we’re trying to deliver for our users.

The best way to figure out what works and doesn’t work in your product is to watch a member of your target audience using it. Prior to our recent release we conducted a number of simple focus groups at Judy’s Book where we had people come into the office for individual 30 minute sessions with our site.

One point to note here: don’t ask users what they want. Observe them doing the actions they want to do. This will tell you far more. Also, while individual sessions may seem inefficient, they avoid group think, so you get an honest opinion from each person rather than one collective opinion.

The Process:

The sessions had the following format:

• Initial impressions with no guidance - we just put the site up on a screen and asked for their reactions.
  • What do you think?
  • Who do you think this site is for?
  • What catches your eye?
  • What’s the first thing you would click on?

• Targeted Questions
  • How would you find deals in your city?
  • What does that heading mean to you?
  • If you click on that link, where do you think it would take you?
  • What do you think of the store logos on the right of the page?

• Observing Simple Tasks
  • If you were looking for a digital camera deal, how would you go about doing it?
  • If you wanted to find deals from Amazon, how would you start?
  • If you were looking to see if a store offered free shipping, how would you go about it?

What we learned:

Putting even a handful of users through this exercise is incredibly instructive, and humbling. No matter how well you think your site works, watching someone new try to use it makes you cringe from time to time. When you’re closely involved with something it’s hard to see all its flaws but three 30 minute sessions with new users will bring them all out into sharp relief.

We learned a lot from these sessions but in this post, I’m going to focus on the challenge of displaying online and local deals on the same site. Prior to this release at Judy’s Book, our site showed users online only deals by default and then after clicking on a link labeled “View Local Results” you would be shown the local deals that were relevant. This seemed logical enough until we asked a user to see if there were any local deals she cared about. She didn’t have a clue how to proceed.

Once we showed her, we then asked her to find a deal from Amazon and she started trying to find it in the
Seattle store directory. After seeing a couple of users proceed in this way, it became clear that the distinction between online and local deals was meaningless to users. They just wanted to see the deals relevant to them.

From the user’s perspective, it became clear that they wanted to see the deals they could access in one place. Sitting in Seattle, I can buy from my local Target, but I can also buy from Amazon.com. The distinction between online and local isn’t relevant when I’m browsing deals - it only becomes relevant when I try and act on what I see.

What we did:

This insight led to a fundamental change to the site. Instead of creating silos of online and local content, we blended the two. As a result, users see all deals that are relevant to them, whether online or local. Using filters in the left nav, they can narrow the list if they want.

As a result, on our Seattle Deals page, a user now sees our best local and online deals blended together. This is a much more logical and natural experience. You don’t have to toggle between “Online” and “Local” to find Amazon.com versus your local Target. In hindsight, we probably should have thought of this earlier, but watching a real user wrestle with this issue in person really hammered the point home.

Going Forward:

It’s hard to find the time to invite people in and have them use your site but at the end of the day, it’s a very cost-effective way of improving things. Even though it can be hard at times, and it inevitably leads to changes which can also be problematic, simple usability testing like this is critical. The payoff in terms of feedback, for what is at the end of the day, a very small amount of time and money, is staggering.

We’ve been working hard to find creative ways to get hard to find local content into the hands of users - basically, doing work so users don’t have to. As most people in and around the local online sector know, it’s hard to get great local content. I think we’re starting to figure it out - check out our Seattle and Atlanta local deals pages.

The deals rock and they are relevant and unique - this data isn’t available elsewhere on the web. I’m really excited about the quality of the deals on these pages and I’m looking forward to them getting even better.PS: If you want to sign up for a weekly email with the top 10 local deals, just click on “Get a Deal Alert” in the left nav of the pages above.