PeerPlay
In a recent post about getting more out of a tester swap, I gave developers a piece of advice that was half right. I suggested sending every tester the same short question, "What did you expect to happen after the first tap?", and asking it once, around the middle of the 14-day test.
On Indie Hackers, a reader pointed out the flaw. By the middle of the test, the first tap is a week old. And the middle of the test is when you can learn something completely different. The fix: ask two questions, at two different times. Here is how that works for Google Play's 12 testers, 14 days requirement.
Why one question at the midpoint misses
A first impression has a short shelf life. After a week of opening your app every day, a tester has learned where everything is. Ask them on day 7 what they expected after the first tap, and you get a reconstruction, not an impression. The confusion that nearly made them close the app on day 1 has been quietly forgotten.
At the same time, a single day-1 question would waste the most unusual thing about a closed test. For two weeks, the same group of people opens your app again and again. That is the only time before launch when you can ask anyone about coming back, not just about getting started.
Day 1: what did you expect?
Send the first question on the day a tester first opens the app, or in the same message where you share your opt-in link, so it is waiting for them. Keep it to one line: "What did you expect to happen after the first tap?"
Compare the answers with what your app actually does. If several testers describe something different, the problem is your first screen or your store listing, not your testers. This is the question that finds onboarding problems, and it only works while the app still feels new.
Don't explain the app before you ask. If your message says "it's a habit tracker with streaks", every tester will expect a habit tracker with streaks, and you learn nothing about what your first screen communicates on its own. Share the opt-in link, ask the question, and let the app speak first.
Halfway: what would bring you back?
The reader's version of the second question was about what made the tester come back. I would adjust it slightly for a closed test, because closed testers come back for a reason real users don't have: they promised to. On PeerPlay, testers open each app every day for about 30 seconds because that is the deal, not because they couldn't stay away.
So ask around day 7: "If nobody had asked you to test this, would you have opened it again this week? What would make you?" The first half tells you whether the app has a reason to exist in someone's week. The second half often names the missing feature, reminder or habit that would give it one.
This question catches a different kind of problem: repeat-use friction. A login that expires, a screen that loads slowly every single time, a feature that was exciting once and pointless the third time. None of that shows up on day 1.
Add context to every answer
The same reader had a second suggestion, which I now use for any feedback note: tag it with the tester's device, Android version and your app build. A slow first screen on one old phone and a confusing flow on every phone look exactly the same in a feedback list without that tag.
Some apps need more context. For a translation app, the reader would also note the language pair and whether the tester was somewhere quiet or noisy. Ask yourself what would change the answer for your app, and log that too.
What the two answers give you
When the 14 days are over and you apply for production, Google asks what you learned from testing and what you changed. Two rounds of identical questions make that easy to answer honestly. The day-1 answers show what you fixed in onboarding. The halfway answers show what you learned about why people would or wouldn't keep the app.
PeerPlay doesn't have a shared question set yet. Each app card has a "Send feedback" button with a "Did you find any bugs?" prompt. Until there is more, nothing stops you from sending the two questions yourself. It costs two short messages and turns 14 days of daily opens into something you can actually use.
FAQ
When should I ask closed testers for feedback?
Twice. Ask about first impressions on day 1, while the app still feels new, and ask about coming back around the middle of the 14 days.
What should I ask testers on day 1?
One short question, the same for everyone, such as "What did you expect to happen after the first tap?". Compare the answers with what your app actually does.
What should I ask testers halfway through the test?
Whether they would have opened the app again that week if nobody had asked them to, and what would make them. It finds repeat-use problems that day 1 can't show.
What context should I log with tester feedback?
At least the device, Android version and app build. Add anything else that changes the answer for your app, such as the language pair for a translation app.