Release early, release often / by Cory

netflix.gifI recently posted about the Netflix recommendation system and their reaching out to make it better. I also mentioned how I think that Netflix is one of the best examples on the internet of a site that is both beautifully designed yet very usable. User Interface Engineering took a deep look into what makes Netflix such a great site by talking with the Netflix design team.

Here are some insights to their "Release early, release often" mentality...

Benefits of Fast Iterations

  • Fail Fast A major benefit of fast iteration is you also fail fast. Failing fast means you invest less time in the things that don't work. If you quickly find out what works and what doesn't work, then you take action to turn it into something that does work.

    Ironically, teams that fail fast improve as fast, if not faster, than those who try to get it right the first time. The reason is simple: Teams trying to get it right the first time fail as often as everyone else does. However, when they fail, they fail really slowly and struggle to pinpoint problems because they've changed so much at once, making it harder to identify solutions

  • More Experimentation The faster you fail, the more experimentation you can do. You can try out ideas that might not have a lot of support, but could be potential winners. This allows for an innovative environment.

    Perhaps you've heard of Google's 20% time? They expect their engineers to work 20% of their time on a personal project -- an experiment they find personally interesting. This program has the effect of bootstrapping experimentation, so it will happen more often.

  • Learn Quickly We've all had the experience of sitting in meetings arguing whether something will work or not. Usually, both sides just don't have enough data to go on, and they end up going with their gut or with the loudest arguer (for better or worse). Fast iteration helps solve this problem by giving developers a platform on which they can test quickly, helping to collect data about any outstanding questions instead of resorting to opinionated arguments.
  • Provide Continuing Interest In addition to improving your design, fast iterations may have a psychological effect on users. Those users who use your app with any frequency will notice the changes, and if the good ones stick, they'll appreciate your ongoing efforts to improve.

    The best teams not only design the changes, but design the process for introducing the change. They experiment with methods to overcome the users' natural resistance to change, providing migration paths and clear benefits for each improvement.

  • Reduced Risk Quickly iterating helps reduce risk during design. If teams can make many small changes instead of a few larger ones, they mitigate risk because they know which changes have what effect. If a design team makes many changes at once, they have a harder time knowing which parts work and which parts don't. When you make only one or two changes at a time, you know immediately what effect it has. Reducing risk is a valuable outcome of moving to fast iteration.

Side Effects

These benefits don't come easy. There are significant changes design teams have to make to their core process to iterate quickly. It's not a switch a team can turn on or off.

  • Culture Change Most designers are accustomed to long release cycles. Fast iteration and fast evolution of design creates a different kind of design environment. Gone are the grand visions of the redesign, where teams spend months retooling vast areas of the site. Replacing it is the idea that the site is a living, breathing design that needs constant care and attention. The team at Netflix calls themselves "compulsive data wonks". They rarely dream very far in the future. Instead, they're concerned with what's happening right now.
  • Design Determinism When teams make the switch to fast iteration, it changes the site's testing methodology. Testing becomes ongoing. After a release, you test for a certain period of time to determine what to keep and what to throw away. Then you start the process over again immediately. And repeat.

    To some designers this sounds overly deterministic: Doesn't this take the fun out of design? If all the decisions are cut-and-dried, what does that say about creativity? What about longer-term effects? Is it possible that some features take longer to catch on than others, and that an early flop might not mean it's not a valuable feature? With fast iterations, if the feature doesn't work now, then it's not right for the site, no matter how creative it is.

  • You're Either With Us... Netflix's Chief Talent Officer, Patty McCord, told us their process of fast iteration causes uncomfortable situations for some designers. Once, a designer had spent time and energy working on a feature that testing showed didn't work. When it came time for the team to remove the feature from the site, the designer was distraught. He had become too emotionally invested in his design, and it got in the way of his job. He ended up parting ways with the team and moving on. Unfortunately, the process of fast iteration affected more than the product itself.