Starting Up by Rahul Pathak

Measuring consumer behavior and watching consumers interact with your site are the best ways to find out what works and what doesn’t. There’s a great post on Signal vs. Noise about some of the things behind Amazon’s success. Two paragraphs that jumped out at me:

Use measurement and objective debate to separate the good from the bad. I’ve been to several presentations by ex-Amazoners and this is the aspect of Amazon that strikes me as uniquely different and interesting from other companies. Their deep seated ethic is to expose real customers to a choice and see which one works best and to make decisions based on those tests…

…This is done with techniques like A/B testing and Web Analytics. If you have a question about what you should do code it up, let people use it, and see which alternative gives you the results you want.

This data driven approach is especially valuable for a company in Amazon’s position where small changes in things like the click-to-sale rate could have a massive dollar impact. Even if you’re not in a position to do testing at this scale (or don’t have the traffic to make it meaningful), watching individual users is still incredibly valuable.

In an ideal world, you’d be able to do both - measure the aggregate and observe the individual.

P.S. If you find this mythical ‘ideal world’ tell me how to get there.

Related Posts:

• Designing and releasing a good product is hard
• How Amazon does A/B Testing
• How to Make Your Site Better
• User Engagement - Measuring the Right Things
• Iteration Speed vs. Iteration Quality (via Coding Horror)
• Usability isn’t as Hard as You Think

The decision to release a new feature or say that you support a new geography is ultimately driven by the answer to one question. “Can I deliver a good user experience if I do this?”

You have to be honest with yourself when you answer this question. If the answer is yes, then expose the feature to users or add the location. If the answer is no, wait. By waiting, you’ll ensure that people either have a good experience, or they have no experience at all. This is far better than some people having a good experience and others coming away saying “well, neat idea but they didn’t really have anything there I cared about.”

It may feel like you’re moving too slowly or not addressing a large enough segment, but exceeding expectations for a few is by far the better path to go down. Also, if you’re smart, you’ll find a way to get permission to email users who came but weren’t supported and invite them back when you’re ready.

Alex Iskold has a great post on Read/Write Web about Google’s Custom Search Engine and how it shifts the value in vertical search from infrastructure such as crawling and search relevance to UI and site selection.

I completely agree that Google CSE is a net positive for the vertical search space. However, it’s important to keep in mind that it only allows you to focus on finding web pages restricted to a specific slice. Having the crawling and search relevance components freely available is a huge boon to new entrants. That stuff is really hard. However, Google CSE doesn’t help you when you’re search for domain specific items that aren’t web pages e.g. products, flight information, resumes etc.

For example, take Couponlooker, which is focused on helping people find online coupon codes. If you search for “amazon coupon codes” on couponlooker, you’ll get back actual discount codes with expiry dates.

If you do the same search on a Google CSE, you’ll get back pages with coupons on them. A subtle difference to be sure, but it’s extra distance between the user and their goal.

Google’s Custom Search is a pretty incredible product, but you’re limited by the underlying infrastructure and search semantics. This is not a bad thing - the guts of the system are incredible and for web search haven’t been beaten yet. If you’re looking to do something different though, you’ll probably need another approach.

Seeing real time collaboration in action is super cool. Some friends and I were coordinating schedules via a spreadsheet and I logged in to Google Docs to update my piece. When I did so, I noticed two of my friends online and was able to watch them editing the document in real time. Each person’s input box was color-coded to their login name. Everyone who was online could also chat in a window off to the side. Super cool.

I’m a huge fan of Excel and but connectivity and collaboration could be game-changers for Google.

I noticed the ad below in Gmail today.

This may seem unremarkable until I tell you that my office is on Eastlake Avenue in Seattle. Wow.

From Google’s perspective, the best part of this is that because the ad in unobtrusive and relevant, it’s actually a positive experience for me.

The best way to figure out what works and doesn’t work in your product is to watch a member of your target audience using it. Prior to our recent release we conducted a number of simple focus groups at Judy’s Book where we had people come into the office for individual 30 minute sessions with our site.

One point to note here: don’t ask users what they want. Observe them doing the actions they want to do. This will tell you far more. Also, while individual sessions may seem inefficient, they avoid group think, so you get an honest opinion from each person rather than one collective opinion.

The Process:

The sessions had the following format:

• Initial impressions with no guidance - we just put the site up on a screen and asked for their reactions.
  • What do you think?
  • Who do you think this site is for?
  • What catches your eye?
  • What’s the first thing you would click on?

• Targeted Questions
  • How would you find deals in your city?
  • What does that heading mean to you?
  • If you click on that link, where do you think it would take you?
  • What do you think of the store logos on the right of the page?

• Observing Simple Tasks
  • If you were looking for a digital camera deal, how would you go about doing it?
  • If you wanted to find deals from Amazon, how would you start?
  • If you were looking to see if a store offered free shipping, how would you go about it?

What we learned:

Putting even a handful of users through this exercise is incredibly instructive, and humbling. No matter how well you think your site works, watching someone new try to use it makes you cringe from time to time. When you’re closely involved with something it’s hard to see all its flaws but three 30 minute sessions with new users will bring them all out into sharp relief.

We learned a lot from these sessions but in this post, I’m going to focus on the challenge of displaying online and local deals on the same site. Prior to this release at Judy’s Book, our site showed users online only deals by default and then after clicking on a link labeled “View Local Results” you would be shown the local deals that were relevant. This seemed logical enough until we asked a user to see if there were any local deals she cared about. She didn’t have a clue how to proceed.

Once we showed her, we then asked her to find a deal from Amazon and she started trying to find it in the
Seattle store directory. After seeing a couple of users proceed in this way, it became clear that the distinction between online and local deals was meaningless to users. They just wanted to see the deals relevant to them.

From the user’s perspective, it became clear that they wanted to see the deals they could access in one place. Sitting in Seattle, I can buy from my local Target, but I can also buy from Amazon.com. The distinction between online and local isn’t relevant when I’m browsing deals - it only becomes relevant when I try and act on what I see.

What we did:

This insight led to a fundamental change to the site. Instead of creating silos of online and local content, we blended the two. As a result, users see all deals that are relevant to them, whether online or local. Using filters in the left nav, they can narrow the list if they want.

As a result, on our Seattle Deals page, a user now sees our best local and online deals blended together. This is a much more logical and natural experience. You don’t have to toggle between “Online” and “Local” to find Amazon.com versus your local Target. In hindsight, we probably should have thought of this earlier, but watching a real user wrestle with this issue in person really hammered the point home.

Going Forward:

It’s hard to find the time to invite people in and have them use your site but at the end of the day, it’s a very cost-effective way of improving things. Even though it can be hard at times, and it inevitably leads to changes which can also be problematic, simple usability testing like this is critical. The payoff in terms of feedback, for what is at the end of the day, a very small amount of time and money, is staggering.

If you’re using the Better Gmail extension in Firefox, you can get a right-click message preview in Firefox.

The goal of enabling users to scan messages without unnecessary clicks is a noble one, but I don’t think it  should come at the cost of the expected right button behavior. I think a mouseover preview would have been better.

Seth’s post on follow through and caring about the last inch is a must-read if you haven’t read it already.

Obsessing about the last inch of follow through ensures that the important parts of what you do get just as much (if not more) commitment.

You can’t afford to stop caring about the little things. The equivalent argument when it comes to web sites is: “Well, so few people see that page, we can leave it looking crappy.”

This is incredibly dangerous thinking. The truth is that sometimes you have to cut corners to get something released, but you can’t accept leaving things in a shitty state and you better make sure you come back around and fix it. (By the way, the reverse, obsessing about the little things, is often how great products get built.)

If something isn’t worth doing right, don’t do it, or kill the feature. Otherwise, if it’s up and viewable to your users, then make sure it’s your best work.

Last week, we rolled out some small tweaks to Couponlooker that should make it even better for finding online coupon codes.

• Fewer sale announcements, more coupon codes:
  We now work harder to make sure that it’s easier to find actual codes that you can use at checkout rather than just links that you have to click on to obtain the discount. Our data suggested that the latter were often just indicators of sales and general savings rather than specific discounts that you could obtain when checking out.
  
  
• Merchant Normalization:
  A lot of coupon sites list merchants under different names e.g. Amazon, Amazon.com, Amazon com, Amazon Coupons, etc. This can lead to clutter in search results and is also detrimental to the user. From the user’s perspective, the 4 variations above should all be returned when they search for Amazon coupons. This is an ongoing process, but we’ve been working on this and results should start to improve.

Improving search results is a gradual process and the Tweak-Measure-Iterate cycle is in full effect.

With last night’s release, as Dave mentioned, there were a bunch of changes to Judy’s Book and I’ll be writing more about those next week as we shake out some of the outstanding issues. One feature that I’d like to mention in the near term is Deal Alerts. In the left nav on every page, you’ll see a box that allows you to sign up for a Deal Alert for the page that you’re viewing.

The signup process is streamlined and after you confirm your email address we’ll send you new deals that match the criteria you select. So, for example, you could sign up for “Seattle’s Best Shoe Deals” and receive that regularly via email. In keeping with the notion that you should control the communication you get, you can manage all your alerts from your profile. Membership is not required to sign up but if you do become a member, we bring all your preferences forward.

Like any new feature, perfecting this will take work, but I’m thrilled to have this released. In general, in spite of some issues we need to work out, last night’s release was a big one for us.