A9: Results

In this assignment, we use Google Analytics to run online A/B testing that you planned out in previous assignments. We statistically analyze our data. We interpret our results and use our interpretation to guide final changes to our prototypes. We reflect on the validity of our experiment. We settle on our prototype's branding, and finalize our aesthetics to reflect the essence of our design. We continue to update our dev plan, identifying new stretch goals that we might not have time for.

The following rubric items are independently assessed. In general, evaluation works such that if the student/team meets, for instance, 9 out of 10 rubric items, then they have earned a grade of 90%. The assignment will not be graded unless it is submitted on a single, well-formatted, and easily readable PDF.

This assignment requires one submission per: team

Evaluation Rubric

Students earn 1/2 point for each binary criterion met.

    Compile and Analyze
  1. Submit screenshots of your Google Analytics page as evidence the experiment was conducted with at least 10 users per test condition/variant.
  2. Write a few detailed sentences explaining your recruitment process, justifying a good faith effort in obtaining your target demographic
  3. Present your raw data in a 2 column table, one column for each condition
  4. Show the correct inputs to your chi-squared or other statistical test, your calculations, the correct resulting chi-squared value or test statistic, and resultant p value. Use 3 significant digits for all numbers. see another video for help
  5. Summarize all formal and informal, quantitative or qualitative, feedback you received from online testing

  6. Interpret and Implement
  7. Write a few detailed sentences interpreting the results of your analysis, speculating why the results are as they are
  8. Write a few detailed sentences describing the internal and external validity of the experiment
  9. Write a short paragraph delineating changes you have made, will make, or would like to make to your prototype, and mark them as such, and justify these changes based on your results (it is very rare for a test not to suggest changes)
  10. Write a few detailed sentences discussing what you learned from A/B testing and any other data generated from using Google Analytics
  11. Implement feasible changes you identified as a result of testing
  12. No usability bugs or console errors when reviewing the prototype. Prototype is polished as a result of online testing
  13. Accomplishing a goal in the app does not require unnecessary steps or processes

  14. Branding and Aesthetics
  15. Clearly identify your prototype's final brand name and point of view to be used during your final pitch
  16. Prototype's final brand name reflects its functionality and final point of view. You are allowed to continue to tweak your app.
  17. Prototype's visual design choices about color, typography, and layout reflect user needs and final point of view
  18. Prototype is visually compelling and mobile-optimized
  19. Prototype's visual design is consistent within and across pages
  20. Prototype has a clean look; words/buttons are not cluttered and information is not presented in such large chunks that they can be overwhelming to a user
  21. Prototype is easy to read (i.e. no white text on light background or vice versa) on all pages

  22. Development Plan
  23. Verify all tasks are actionable, prioritized, assigned an owner*, given a time estimate, and have actual time spent and completion date recorded. Upper left hand corner shows a Sum() of expected and actual hours with a similar-ish load for each team member; outliers should be justified.
  24. In your comments column, identify tasks that were newly added or removed/updated, and those derived from A/B testing.
  25. Add new stretch goals for your prototype and identify them in the comments column
  26. *every member of the team was assigned and completed at least one programming task

  27. Submit a clickable or easily typed link to your final prototype of the form “a9-projectname.herokuapp.com”. If your prototype is changed before grading is completed, you will receive no credit for the assignment
  28. Submit a clickable or easily typed link to your A design prototype of the form “a9-projectname.herokuapp.com/path/to/page_A” and B design prototype of the form “a9-projectname.herokuapp.com/path/to/page_B”
  29. Submit a clickable or easily typed link to your project repository on GitHub. Make sure the grader has access by the deadline
  30. Submit a clickable or easily typable link to -- or a readable, properly oriented, and complete snapshot of -- your dev plan. Make sure the grader has access
  31. Make ixd@ucsd.edu a collaborator for your Heroku app so that we can verify the commit that has been deployed

  32. Above and Beyond
  33. (Optional) This point is reserved for teams with prototypes that met 90% of the rubric items and went outside the box in their performance
Note: for this week's studio, also draft a 30 second elevator pitch, describing your point of view, user needs, and prototype

Student Examples

Here are some examples from prior years. Note assignments change from year to year, so use these examples as a reference, see where they succeed/breakdown, and make sure your final submissions adhere to the rubric for this year.

User Testing and A/B Prep: (1) (2) (3)

User Testing Plans: (1) (2)

Apps: (1)

App Demos (not required for this assignment): (1) (2)

Development plans: (1) (2) (3) (4)

Here is a cool video of the dynamic nature of implementation plans throughout the project.

Self Assessment

Self and team assessments must be completed before your TA grades your assignment.

Assess your group's work as a collective unit.
Assess your own work and each of your team member's work.