How “early access” helps with feature development
In this post, we describe how we leverage “early access” Stadia Experiments to engage with our passionate community of Stadians and launch high-quality features as early as possible.
“Stadia Experiments” and why they are useful
When building a product like Stadia, there is a delicate balance between wanting to deliver new features for our community as fast and early as possible, and ensuring that everything we ship meets our high-quality standards. Furthermore, gathering and analyzing data is a critical part of our development process, helping us to identify and address bugs across a large and fragmented ecosystem of end-user environments. This “feature validation” phase is often the most time-consuming part of the product development process.
After launching Stadia in 2019, there were many new features on our product roadmap that we delivered in a short period of time. Validating performance of these features across all combinations of network environments, phone models, audio-video setups, controllers, and more would’ve been impossible for our team to do directly within the desired timeframe. This is where the idea for Stadia Experiments was born!
|Experiments section in the Stadia mobile app|
Experiments are new features that we launch to Stadia in “early access,” meaning we provide the features to users while they are still in development. Users need to proactively opt into using these Experiments, helping set the right expectations for their level of feature completeness.
This new mode of launching features solves two key challenges for us:
- It allows us to get new features into users’ hands as early as possible. By releasing the new features as Experiments, we can bring experiences to our users earlier than expected, and we avoid over-optimizing each feature ahead of it seeing real-world usage.
- We dramatically improve feature validation by providing a way for our community to help with the process of gathering feature performance data across many real-world environments. Our community provides incredibly valuable feedback through post-game quality surveys, bug reports, and other community channels, helping us identify what to improve before a full launch.
Once a feature has spent a sufficient amount of time in Experiments for us to gather enough data for validation, we “graduate” that feature to a full launch, increasing the feature’s visibility and exposure to more users within the product. We’ve relied on Experiments on multiple occasions to help gather real-world data and feedback ahead of making something widely available.
Scaling on Android via user feedback
The first feature we tried this with was expanded Android gameplay.
When Stadia launched in 2019, it supported gameplay on Pixel phones. This helped ensure a high bar of quality for gameplay, as we could directly test and verify that gameplay worked well on these devices. However, to expand beyond Pixel, we knew we needed a scalable plan to individually test and verify every Android phone.
We relied on Experiments to enable broad access to gameplay by letting users opt into playing Stadia on Android devices that we didn’t yet “officially support.” Users could play and provide feedback on how Stadia gameplay performed in a wide range of environments.
Thanks to this Experiment, we’ve continued to expand official gameplay support to more devices over time. Once we have enough data to demonstrate that a given set of devices meet our high quality bar, we graduate them to “officially supported” devices. And we’re able to do this even when we aren’t able to directly test and verify each device ourselves.
Building confidence in mobile data
We also used Experiments to help with the launch of gameplay on mobile data.
We knew that users wanted to play Stadia on mobile data networks, but had a significant amount of testing to complete. Given the variability of mobile connections, we needed an approach that would let us understand gameplay quality in a range of real-world environments, so that we could address any significant gaps ahead of fully rolling out mobile data support.
We launched support for gameplay over mobile data first as an Experiment. This meant that any Android user could opt in, provided they acknowledge a warning that the quality of the experience may vary. As with expanded gameplay on Android devices, this enabled us to gather feedback from real users across a wide variety of devices, networks, and network conditions.
We were able to use the data from this Experiment to tune and improve our modeling across these different conditions, in order to provide higher reliability while playing on a mobile data connection. We also used the experimental phase to make improvements to user messaging, especially around clarifying implications on data usage. After we had enough confidence in the overall experience of gameplay over mobile data, we graduated the feature to be fully supported.
Making new features available to users sooner is a great opportunity to delight superfans, while setting expectations appropriately if the features are still being built or refined. You can also avoid over-optimization ahead of exposure to real-world usage and use feedback from opt-in usage to inform what improvements to prioritize before fully launching the feature.
-- Alan Joyce, Product Manager and Eli Wald, Product Manager