Improvements to thumbnail AB testing

Feature Requests Website Features
1

Hello! I have a few suggestions to greatly improve the already awesome AB test feature.

I use this feature 2-3x times a day, and this would greatly improve my workflow. The main takeaway is that I have 30+ thumbnails on my computer, 30+ in the Roblox system, and when you are trying to manage these, it becomes a headache.

Quality-of-Life

  1. THE BIG ONE! When you upload a photo, sort it all the way at the top or at the bottom. Currently I think it is sorted by name? Which causes the sort to feel random. You will upload a thumb and then you have to go find it. This is a problem because we are often testing very very minor changes which are hard to see. Some new ones will be on page 1, some will be on page 3.
  2. Show the file name of the upload. This is so we can track versions better. We lose the name after uploading it and have to refer to thumbnails by their photo which is confusing.
  3. Allow deleting more than one thumbnail at a time. This is because you have to click the little 3 dots, then a popup appears, you have to click confirm/yes, then it goes away. Doing this on 30+ thumbnails takes a bit.
  4. If you have the page open for a long time and try to upload a thumbnail it says it fails. You have to refresh before uploading. I think this is a session/token issue. Not a big deal, but it’s confusing UX because the message says the upload failed.

Features

  1. Allow testing more than 5 thumbnails at a time. Whether it be in batches, or scheduled. The goal with this would be to automate testing better. We have a lot of thumbnails to test and its almost a full-time job refreshing, checking, and uploading.
  2. Historical stats. A new tab that shows all previous thumbnails and their results. I realize its super dependent on the day and such, but generally, our winners always win. It would be nice to have a historical list to look back on. Right now we screencap the results and log them into a Discord channel. This should persist through resets.
  3. Add a new chart/stat for Retention (per thumbnail). I’d like to see how each thumbnail retains players for day 1, day 7, etc. This would let us see if we are making misleading, or accurate, thumbnails for what we are offering in the game.
  4. Add a new column for Payer conversion rate (per thumbnail). This would let us see if the thumbnail is attracting low quality players or high quality players. We may be better off with less QPTR if the user is more valuable (in my eyes).

Thank you!

2

Some more ideas:

  1. Make the results show faster, for some reason it’s delayed by 8 hours
  2. When you do a new test, previously it reset the stats for all thumbs, this was nice and bad at the same time. Now it doesn’t reset them anymore, I think it should only reset the ones that are picked for the new test. The reason is: The qptr displayed uses the total impressions/plays it gets in it’s lifetime, so it takes longer to see how much the qptr is on the new test (As thumb performance may change over time), while the test itself doesn’t use it, it’s still useful to know the actual performance of the thumbnail. Now I have to reupload the thumbnail when doing a test
  3. More winning segment types, I’m not sure why but it’s only by age when I could imagine it would benefit from more splits, especially gender
1 Like