Categories
Product

Reviews on marketplaces

Uber drivers average a 4.8 rating.

Lyft drivers a 4.86.

Average Yelp restaurant reviews are 3.71

What’s the difference?

Is the average Uber driver delivery a better experience than the average restaurant on Yelp? I don’t think so. Is Uber deleting bad reviews? I don’t think so.

Part of this is selection bias. Uber and Lyft police their platforms in a way Yelp can’t. Bad drivers get kicked out. But I think that’s only part of the story, based on my own experience managing a system with many tens of millions of reviews.

My hypothesis: People rate versus expectations. Most experiences, Uber ride, meal, whatever, meet expectations. And for Americans, meeting expectations means you get five stars.

If you’re consistently not meeting expectations, you’re going to be kicked out of the platform or you’ll go out of business (or clean up your act, in which case your average rating should improve).

So, how does this explain the difference between Yelp and Uber? Uber knows whenever you take a ride and prompts you to review. The share of rides that generate a rating on Uber is probably very high. If most people leave reviews, and the typical experience is a five star experience, most reviews on the system will be five star.

It’s very different for Yelp. Unlike Uber or Lyft or OpenTable, Yelp typically doesn’t know when you’ve been to a restaurant. They can’t prompt you to review your typical five-star meal. Only a tiny fraction of dining events are reviewed on Yelp, and for many people it’s going to take something exceptional to get them to seek out Yelp and choosing to leave a comment.

An exceptionally good experience can’t get better than a 5-star rating. There’s a lot more room to move down. Hence lower average ratings.

Originally on Twitter.

Categories
Product

Goals

OpenTable just wrapped up Q2 goal planning. Every time we go through one of these, I refine my approach to goal setting, and wanted to codify some of my thoughts.

I’m a big believer in OKRs (aka “Objectives and Key Results”). They’re originally from High Output Management by Andy Grove, and were largely popularized in the tech sector by Google.

Writing a good OKR is hard

My standard rules:

Objective

  • The objective should be aspirational, and be something that can get people excited. For search, for example, one of my objectives is “make deploys, indexing, ranking, and autocomplete crazy fast and stable.”
  • For teams, I think many good objectives can stay the same quarter over quarter, serving as the core mission of the team.

Key results

  • Measurable (numbers, not “crazy fast”).
  • A spectrum — not boolean (e.g., “launch X” is just succeed/fail), because you want to be able to measure progress over the course of the quarter.
  • Aggressive. Google’s standard is to hit a 0.6 average in grading OKRs. People who are new to OKRs are often uncomfortable with this, but managers need to push hard on this.
  • Intelligible. Key results are a valuable communication tool to engineering/design teams and to counterparts throughout the rest of the company. People need to be able to understand what you’re measuring.
  • Time boxed. We do quarterly planning, so our goals are typically time boxed to the quarter. For some big new launches (that might take more than a quarter of work) I’m okay with doing goals a quarter in advance, but this should be rare.

Key Initiatives

  • Key initiatives are not part of classical OKRs
  • For product managers who are working with engineering teams on a day-to-day basis, I think it’s also important to highlight big projects, because these big projects are a good opportunity to align with other teams and highlight complicated/impressive projects the team aims to work on.
  • Initiatives are the PM or team’s hypothesis on what they’re going to do to meet the key results they’ve outlined. It’s really critical that key initiatives are hypothesis though — teams succeed or fail based on their key results, not on whether they deliver specific products.

OpenTable does OKRs by product development team, led by product managers. This means that my team has their own set of OKRs, but I think we’re missing out on OKRs that aren’t always tied to the business and on OKRs for other functions (like individual engineers).

Communication of the OKRs is just as important as writing them

We generally set aside a week at the end of every quarter for a series of death meetings to review every single team’s set of goals. It is striking how variable the approaches are for different teams.

I think the best goal meetings:

  • Has a POV, and is making the case for why the work of this team matters in a macro sense
  • Tell a story about what was accomplished, what will be accomplished, and how they’re related to the team’s core mission — and makes the delivering team the hero.
  • Explains some of the hardest problems the team has faced and is trying to solve in simple terms. I work with our data science team, and it’s particularly important for them — even a highly technical CTO won’t know what ensembles and mixture of experts are in the context of our business.

Finally, I tried something new this planning process which I’ll be sure to include in the future. A demo. In retrospect, this seems obvious — every good planning or mid-sprint check-in should have demos, so why shouldn’t an end-of-quarter? — but I think it’s easy to get too focused on the stats. Product companies should show how they’re building products.

Categories
Product

Communication

Product managers are in the business of communication, so concise and cogent writing is one of the most powerful tools available to us. It’s a hard muscle to develop, requiring consistent feedback and deliberate practice.

Business writing is a core part of the curriculum at the consulting firm McKinsey. Managers pore over every word written by new analysts, setting a high quality bar and demanding frequent revisions. Most companies don’t invest as much energy into training people how to write, so we’re left to figure it out on our own. It’s not easy.

One place to start is Barbara Minto’s Pyramid Principal. I was introduced to Barbara’s work by Harrison Metal and it was immediately familiar: She helped codify McKinsey’s writing style.

An easier place to start might be Julie Zhuo’s recent article on “Escaping Email Hell.” Julie offers specific changes to writing style which improve emails — and most of these tips could be applied to any kind of writing. Her best advice is to “know your end before you start.” Great writing should be a condensation of your thought — not every step in the winding journey you took.

Other books I’d recommend:

 

Categories
Product

Search, discovery and marketing

One of the clearest places to see this problem of ‘too much’ is Yelp. I’ve been fascinated by how many companies are effectively trying to unbundle Yelp, despite that fact that (unlike Craigslist) it’s a modern technology company that does most of the things one would expect it to. But where people unbundling Craigslist generally try to peel off a category and deliver a modern experience, the people going after restaurant listings are often doing so with constraint. That is, instead of giving you every single restaurant that’s within 2 miles and that lots of people liked, they give you 10 restaurants. … People are attacking crowdsourced universal scale with constraint, curation and personal preference.

Looking at these companies, it strikes me that actually, saying that ‘Yahoo’s directory didn’t scale’ misses the point. What we’re really seeing is a trade-off between two problems. You can have a list, solving discovery and recommendation, but once the domain gets big then your list is either unusably long or partial and incomplete (and perhaps uneconomic to maintain). Or you can have a searchable index of everything but you’re on your own working what’s good and finding things you didn’t know to search for.

Search, discovery and marketing from Benedict Evans

Curation also works best in places where taste matters — the “best” book, the “best” restaurant. An algorithm can’t provide social validation beyond “the most popular.” What’s interesting about Yelp is that their focus on user generated content  means it’d be hard for them to add sources of curation outside of their users — there’s no list of restaurants with Michelin stars, or James Beard winners, and there probably can’t be one without undermining the core conceit of the platform.

Categories
Product

Do mobile user education right

Redlaser 4.0 Mock

When we were designing RedLaser 4.0, one of our biggest questions was now to handle navigation. In particular, we weren’t sure where to put the button that launched our barcode scanner. It’s a critical function, but we were adding more features, so we needed to make sure it stood out.

We put it at the very center of our navigation bar, with a giant scan icon. We made the background red, so that it would really pop.

Sure, the app wouldn’t boot straight into the scan mode — we wanted users to see all the new features — but we were sure we’d done the next best thing.

The design failed, horribly, in testing. One user, a long-time RedLaser user, spent fifteen minutes trying to figure out how to launch scanning. She later told us she hadn’t even noticed the button.

This is a common story in mobile: Complex user interfaces are extremely hard, and a UX change can backfire in unexpected ways. If a button isn’t explicitly labeled and visible above the fold on the home page, it will get exponentially less usage.

This is a challenge: Phone screens are too small to show an explicit user interface for all but the simplest apps.

RedLaser isn’t alone in facing this issue. Swarm is a gorgeously designed app from Foursquare. It is the culmination of FourSquare’s efforts to re-envision and dramatically simplify the check in process. To do this, by default it continually broadcasts your current neighborhood to your connections. FourSquare realizes that while this behavior is critical to the app, it can also be really creepy: you don’t always want your friends to know when they’re nearby.

A very important part of the interface, then, is turning this tracking off. It’s easy, but also a new interaction element: You swipe left-to-right across the top bar and it changes your state:

FourSquare

While this interaction approach is easy and smooth, Swarm’s challenged because it isn’t a kind of interaction users expect — and there’s no explicit text to let you know it is even an option. Even savvy mobile users find it hidden:

https://twitter.com/danielbachhuber/status/474018824857989121

This leaves developers two choices: Build simpler apps, or teach users.

It looks like many of the largest companies in mobile are choosing the former. The triumph of Whatsapp is its focus and simplicity — and it is followed by focused apps from eBay, LinkedIn, Amazon, Google, and more. A single-purpose app can create a focused, simple, user-interface.

A single-purpose app, however, just creates different problems: User acquisition, and user retention. Big mobile companies with their own ecosystems can use apps to feed each other (just like clicking an eBay link in RedLaser will send users to the eBay app). The vast majority of mobile players don’t have this luxury, and user acquisition can be brutally hard. So when you’ve convinced a user to download your app, it needs to be a must-open, home-page-worth app — and this frequently will mean complexity.

Most app developers can’t afford to fragment their user base. So complex apps need to get user education right.

User education, however, is hard. The most common approach is to show an overlay on launch, calling out a few critical buttons. In Swarm, FourSquare insisted every user disable and then re-enable location tracking before permitting wider use of the app. Others require users to watch short videos or swipe through pages of text.

None of this works. Most users in a new app want to get up and running as quickly as they can. We’ve all been guilty of furiously button mashing through tutorials, trying to get to the meat of the app. So what should a designer or product lead do?

The most impressive user education system I’ve encountered is, surprisingly, for the app Secret. Secret is a social sharing app whitest people anonymously broadcast information.

Secret uses three complementary approaches to user education

Method 1: The traditional tutorial

Secret starts with a traditional user education flow When a user opens the app for the first time, they need to swipe through a series of screens describing the basic operation of the app. This happens before registration, which is important so that users get a sense of how the app works and aren’t driven to close or delete immediately. These screens are extremely simple, focusing on concepts rather than on specific functionality.

At RedLaser, we found people swipe through this kind of screen extremely quickly. The default assumption should be that the average user will not see this content.

HighlightsMethod 2: In-line education

Secret’s first innovation in user training is inline education. The key interface for the app is a user’s secret feed. When a user pauses to read a secret, it is blurred out and a single, focused, explanation of a user interface element appears. These messages are contextually relevant — they only appear when the user pauses to read a message that could benefit from further explanation.

The advice is effective and powerful because it is highly relevant to the user’s current situation. The messages either suggests a specific action the user can actually take on the secret they’re reading right then, or it provides context for the secret (if not from a friend) which makes the secrets more valuable.

Exiting these messages requires a click in a small area, and the exit button changes location (following the messages), which means that users can’t quickly skip through the messages.

Inline-NotesMethod 3: Spaced Retrieval Therapy

Secret’s final training method is spaced retrieval therapy. In SRT, you regularly re-expose people to information to lock it into their minds.

Secret uses this for key pieces of information: How to like secrets by swiping left to right (critical because this determines whether and how secrets are shared), and permissions requests.

This information is displayed inline, just like a normal secret, which means users are less likely to block the content out (as in banner blindness).

It isn’t clear when Secret decides to show this information. To most effectively train people about interfaces, they should display the information on a regular pattern with decreasing frequency (e.g., after 1 day, 3 days, 7 days, 14 days, 30 days, etc.). If a user positively responds to the content (for example, by taking the action without prompting), the frequency of reminders can become less frequently.


For other posts on mobile, read about the trend of increasing apps per company and iOS 8’s major pro-privacy changes.