A8: A/B Testing and Finalize - due Monday, 11/28, 11:59pm (later due date because of Thanksgiving)

Brief

We are going to be using Google Analytics to run the online A/B test that you planned last week.

Step 1: Launch Online A/B Test

Set up your A/B test with Google Analytics; refer to Lab 7 for help. Since you will be statistically analyzing the results, recruit at least 20 users. Put another way, that means that each person on your team should recruit at least 7 people each: a couple of dorm-mates and a few family members, and you're basically there. (Thanksgiving break is a great time to get relatives at home to try your app!) You can also use social media like Facebook and Twitter to recruit online testers. You can even recruit other people in class. Make sure to launch your test several days before the deadline, though, since it will take time to collect and analyze your data. Here are some ideas from Optimizely for A/B testing.

Google Analytics takes at least 24 hours to report back data to you, so do not start this part last-minute! If you start this last-minute, then you will not be able to collect data in time to finish the assignment. Also, your Google Analytics setup code may have bugs, so you should start early and leave plenty of time to debug.

Step 2: Analyze A/B Test Data

Collect your results from your A/B test, which should be comparing the number of users who, for instance, either clicked through a navigational funnel or didn't, viewed a particular page or didn't, or pressed a button or didn't, in both your A and B conditions. Present your data and the results: show the inputs to your chi-squared test, your calculations, the resulting chi-squared value, and the p value (since you won't have that many users, we don't expect everyone to achieve p < 0.05 statistical significance, so don't worry if you don't). Tables of both your observed and expected values are the best way to visualize your data and calculations. This online tool is a helpful resource for calculating the chi-squared statistic. However, you must show your work for your calculations just like we showed in lecture. Watch this video to see how it's done. Remember that the chi-squared test compares numbers of users, not times or durations. If you find yourself using a different measure, e.g. time spent on a page, then find the appropriate statistical test to show your calculations and interpret its results. The TAs will not help with anything other than the chi-squared test, so do so at your own risk. Thus, we highly recommend you measure something that can be tested with a chi-squared test.

Next, discuss your findings in a very brief paragraph: can you draw solid conclusions or are additional experiments needed? What changes would you make based on these results? What other things would you test in the future?

Step 3: List of Revisions

What did you learn from these tests? Compile a list of revisions or improvements you would like to implement based off the results of the A/B tests. Additionally, include other revisions that your team would like to make that may not have been discovered through the user tests. (You may not have time to implement all of these by the end of the quarter, but we want you to at least think about possible revisions.)

Step 4: Finish Aesthetic Details

Focus on fixing the interaction of your app based off your A/B results. Also, now is finally the time to make it pretty! Pay attention to detail and ensure that the app is optimized for a mobile interface (if you're targeting mobile). By this deadline, your app should be completely done, both in interaction and aesthetic design.

Step 5: Brand Yourself

Create a finalized app name that encompasses the function and brands the app. Branding is an important part of showing uniqueness and attracting users!

Please pick a family-friendly and tasteful name, since this finalized app name will be used to introduce your team during final presentations.

Step 6: Create a 1.5-minute Demo Video

Create a 1.5-minute demo video showing your app in action with voice narration, which will be played during final presentations on the final Tuesday and Thursday of the quarter. Keep it short, while showing off your point of view, project concept, and all the main features of your application. Record the screen of your app itself in a web browser with screen recording software such as Quicktime on Mac or Camtasia (free limited-time trial download).

(We will be very strict about enforcing a 1.5-minute time limit for these videos, to be fair to all students. Think of this like a hard 'page limit' on written assignments; it is not fair to your classmates if you go over the allotted time, since we have only limited time in class to play the videos.)

Refer to the rubric to find out what to include in your video, since that is what we will be grading you on.

We will be playing your video in front of the entire class, so make something you are proud to show off!

Team Submission

For this assignment, ONE person will submit the assignment for their team, listing every team member's name and student ID number (PID) in the assignment submission.

Your write-up will contain the following in one single PDF:

  • The data that you collected. Organize this information so that we can get a sense of the raw data that you gathered. Explicitly state how many participants were in each condition. What feedback (quantitative or qualitative) did you receive? We recommend organizing this information into a table or spreadsheet, exported as PDF. (Online Test Results)
  • Summary of your findings. You should present any statistical analysis that you performed, and any insights or discoveries that your online A/B test helped to inform. (Analyze A/B Test Data)
  • List of the changes you would make (as a result of the online A/B tests). We understand that your test may have concluded only recently, and you would have had time to make only small changes. But we would like you to think about additional features or improvements that are motivated by your results. (List of Revisions)
  • Your final app name. This is the name that will be used in your final presentation. (Brand Yourself)
  • The URL of your final app. Do not forget to give us login information. Note: the URL must work at least until your assignment is graded. If it doesn't work, you'll receive no credit. Very important: the contents of this URL should not change in the upcoming week while your TA is grading this assignment, or else that is a violation of the academic honesty policy; test your new changes at a different URL. (Finish Aesthetic Details)
  • URL link to your 1.5-minute final presentation video (Presentation video URL)

Note: since we may grade your assignment up to a few days after submission, per the honor code, we expect that the prototype URL show the state of your prototype at the time of submission. You will very likely be updating your prototype after submission, but please do so at another URL.

Submit your single formatted PDF in Gradescope.

Evaluation criteria & Grading rubric

The rubric below contains criteria that are worth one point each and will be graded independently and in a binary fashion.

  1. Evidence that the A/B test has been performed on at least 20 users (i.e. screenshots of Google Analytics dashboards for your app).
  2. Statistical analysis is included, showing all mathematical work involved (think of it as showing your work on a math problem).
  3. Statistical analysis is correctly calculated, with the appropriate variable type used for the statistical test conducted.
  4. Conclusion drawn from A/B test is written, and is based off the results of the statistical analysis.
  5. List of revisions is based off the A/B test or other feedback that your teammates gathered.
  6. App has a clear interaction flow that makes usability understandable to new users.
  7. App interface is aesthetically polished.
  8. Functionality of prototype is complete. No more bugs found!
  9. Prototype is ready to be presented and doesn't need any more details or polishing.
  10. Accomplishing a primary goal in the app does not require the user to make unnecessary steps.
  11. Prototype follows an aesthetic design that is consistent across webpages.
  12. Prototype has a clean look; words/buttons are not cluttered and information is not presented in such large chunks that they can be overwhelming to a user.
  13. Prototype contains text that is easy to read (i.e. no white text on light background or vice versa) on all webpages.
  14. A family-friendly and tasteful final name for the app is submitted.
  15. Link to demo video is publicly viewable, and the video is 1.5-minutes long (no credit if it is longer than 1.5 minutes).
  16. Demo video communicates the app at a high level without seeming cluttered.
  17. Demo video sufficiently explains the point of view and project concept with relation to the studio brief.
  18. Demo video clearly articulates a user need that the app aims to address.
  19. Demo video shows a walkthrough through the important functionality of the actual app itself working in a web browser.
  20. Extra credit (up to 4 points) - In addition, your TA can give your team up to 4 extra credit points if your app is extraordinarily polished by this deadline, which provides an incentive to go above and beyond the basic requirements.