<aside>
<img src="/icons/home_gray.svg" alt="/icons/home_gray.svg" width="40px" /> Menu
Latest News
Homa Belly SDK
N-Testing
Games
Ideas
Fast Market Validation
Homa Lab FAQ
Homa Academy
Contact Us
Map
</aside>
Table of Contents
What is N-testing and who can use it?
N-testing is a module on Homa Lab which enables game teams to A/B test new player behaviour by defining in-game parameters to change (configs), and the audience the changes apply to. Then, statistical tests are run to determine if the differences between the resulting groups are significant. Once a clear win or loss is found, the result is displayed alongside that config, which can be rolled out to the entire game audience to increase engagement or LTV.
N-testing essentials
- A/B tests are run only on new players - to ensure that all players are using the same game build and the spotted differences are relevant
- A proportion of players is selected for the test (e.g. 10% of players) and filters can be used to ensure the audience comparisons are relevant. For instance, for an LTV-based test it’s recommended to use one platform and one country (e.g. iOS, US) so that differences between groups are not caused by the groups containing users from different countries where LTV values vary significantly.
- It’s possible to configure the exact parameters changed in the game for each test - e.g. tutorial_on or tutorial_off
- The winners are selected by running statistical testing on the groups and it’s important to wait for a clear win/loss result before applying a config to avoid false positives/negatives
- Once a clear conclusion is reached, it’s possible to apply that configuration to all new players
How different teams use N-testing
How our team uses N-testing
Team members that frequently run A/B tests are: Publishing Managers, the Live Ops teams, internal game designers or producers. They use it to:
- Create A/B tests to compare results between multiple values of the same parameters - e.g. button_red or button_green. N-testing typically starts when a title has enough players, around the time of the launch, and from there continue throughout the game’s life.
- Estimate the number of users added to a test. When selecting the audience, Homa Lab will calculate and show how many players are expected to be added.
- Select the audience for the N-test by share (%) and key parameters (OS and geo). For instance, to get results quickly, a larger % of users could be targeted, or generic changes expected to increase playtime or retention across the board could target all geos.
- Deciding which config to roll out after the test - e.g. if winning config was rolled out to 10% of users for the test, you can then roll it out to 100% of new players without needing to push a new build. However, when there is a likely improvement rather than a certain improvement, the game team may choose to wait to get more conclusive results.
- Test the modification of parameters on a local device or from a specific IP to make sure they work well.
How our community of studios uses N-testing
Studios or game devs using Homa Lab can use N-testing to: