One of the most important decisions we make in product development is when to make a release. From a business perspective, it's better to release early and often, with a "minimum viable product".

It's also important to define explicit technical quality criteria, or we will waste a lot of resources and miss our target dates. We need to focus development and testing resources to provide the best possible balance between quality, time to market and implementation effort.

As part of this, we define an explicit set of target devices and put them in three classes, A, B and C. On a web project, our target devices are web browsers, and on a mobile project they are mobile devices.

Class A devices are continuously tested, e.g. on every ticket. If a feature doesn't work on a Class A device, we don't release.

Class B devices are supported in a degraded mode. They need to work, but may have less functionality. For example, they may be be too old to support advanced graphical effects, but all of the core functionality works. We test Class B devices as part of the release cycle, e.g. once every two weeks. We might allow a release if there is a minor problem on Class B devices, but block the release it if there is a serious problem.

Class C devices are explicitly out of the support window. They may be too old, unusual, or too new. If it works, we are happy, but we won't put in extra effort to support them.

Web example

On web projects, for Class A, we might support IE 10, latest Chrome and latest Firefox. For Class B, we might support Safari on Mac and IE 8. Safari only represents about 5% of total browsers, but is the default browser for Mac. Similarly, IE 8 is the last version of IE that runs on Windows XP, so if we have XP users, then we need to test on it. For Class C, we might say that we do not support IE 6 or Microsoft Edge browsers.

On some projects, mobile browsers are Class C, and other projects they are Class A. For example, we might support Mobile Safari on iPad even if we don't support Safari on Mac, e.g. the "lying in bed shopping for clothes" users sometimes represent 40% of sales for women's e-commerce sites. If we are making a native mobile app for a project, then full support for mobile web users would be excessive, we would detect them and display a page pointing them to the app store.

Mobile example

Apple users typically upgrade immediately to the latest iOS version that will run on their phone, so we normally only support the latest iOS version as Class A. Android users typically upgrade the OS when they get a new phone, every two years. So an OS release starts with about 5% of the market, holds there for about six months, then at about 18 months we see a very rapid shift (see the latest Android Play stats). So the latest Android OS version normally gets Class B support. The top 70% OS versions by usage get Class A support. Older OS versions get Class B support, and the bottom 5% gets Class C.

Android tablets are typically not a primary platform, so they are Class B or C, but if we were making a restaurant Point of Sale system, they would be Class A.

Apple hardware is very consistent, so we can realistically support all the modern iPhones. There is an incredible variety of Android hardware, however, with different screen sizes, screen resolutions and hardware capabilities (e.g. CPU, memory, graphics acceleration). We need to aggressively prioritize the hardware devices that represent the most popular and representative phones used by customers in the target market.

Developers normally use Google Nexus phones, as Google themselves does the Android OS port and releases regular OS updates. So that platform is automatically tested on every release. But it's common for something to work on the developer's phone and not work on an older, low end phone.

For startups, we normally support a flagship Samsung device and a few other popular top 10 phones. After that, we rely on partners and crash reporting software to identify problems.

We have produced apps with tens of millions of downloads. At those levels we effectively need to support every device, but we are dealing with all kinds of broken hardware and we need to prioritize testing efforts.

Quality criteria and risk management

We need to release and iterate in order to move forward with the business. Having explicit release criteria allows us to hit release dates effectively, getting to market and avoiding wasting resources and losing credibility with partners.

It is always possible to improve a product, but having regular releases is more important than having perfect releases. As we get to a final release, we need to objectively evaluate which defects should block the release and which we can fix later (or never).

We typically do that by classifying defects by severity vs impact.

In general we want Class A devices representing the majority of our users to have no significant defects (nothing High or Medium severity). A severe defect that affects 5% of users but has a workaround might be acceptable. A low severity bug that affects all users might be acceptable, e.g. a cosmetic or performance problem in certain situations.

We also need to use statistical techniques to identify how many unidentified defects may be in the product. For example, if we have done final release testing for one week and we have seen no High severity defects, one Medium severity defect and 10 Low severity defects, we might say that we can release if we have no Medium severity defects at the end of the next week and a maximum of 10 Low severity defects on Class A platforms and one Medium defect on Class B.

At a certain point we reach diminishing returns on testing and we need to present the product to the full set of devices and use cases to see the issues.

Some applications have particular high quality requirements, e.g. health care or financial. A "small" bug dealing with money has a much bigger impact than an ugly button. Problems with health care applications can literally cost lives or expose us to lawsuits from leaking protected health information.

In these cases we need more than quality assurance processes, we need processes to mitigate risk, detect problems and respond to issues. Monitoring becomes very important to identify issues in production and quickly deploy fixes. For embedded systems, we need to be able to upgrade devices remotely. It does us no good to fix a bug in our office and not be able fix devices deployed on customer sites.

Psychologically, it's difficult to put something out in the world that is not perfect, but we need to do it. If we fail to have explicit release criteria, we may delay the product launch until everything is perfect. The effect may be that we lose credibility with partners and investors. We make the release process so hard that we lose the ability to iterate and learn from our customers, and, ironically, produce a worse product.