A3: Show your flow: Get feedback - due Thursday, 10/20, 11:59pm

Brief

In this assignment, you will conduct heuristic evaluations (HEs) of another team's paper prototypes. Your heuristic evaluations will follow the "How to Conduct a Heuristic Evaluation" and "Ten Usability Heuristics" readings by Jakob Nielsen.

You will be given time in class on Tuesday (10/18) and Thursday (10/20) to do Steps 1 and 2, but it is also recommended that you complete this over the prior weekend if possible, since it can be hard to coordinate with other groups on short notice. It is your responsibility to find teams on which to do heuristic evaluations.

[Friendly reminder: Read through this entire assignment carefully before doing your heuristic evaluation sessions so that you know exactly what you need to turn in. Otherwise you might not take the proper kinds of notes during your sessions and may have to re-do them!]

Step 1: Perform heuristic evaluation

Find one other team to meet with (you will be given time in class but that may not be enough time, so plan accordingly). Meet with that team and go through "using" their TWO paper prototypes in any order you prefer (i.e., simulating pressing buttons or fiddling with menu items), jotting down as many usability problems as you observe. Don't try to be "nice" by not reporting problems -- everything you write will help that team improve their project. You are helping them! Only one team member needs to be the evaluator for each prototype, but if your entire team wants to hang out and observe, that's fine as well.

Use Nielsen's ten heuristics as a guide and specify which heuristic(s) each problem is related to. If you come across problems that don't fall neatly into particular heuristics, mark that no heuristics apply. Be sure to also use Nielsen's Severity Ratings for Usability Problems to add a severity rating for every problem discussed in your heuristic evaluation.

Aside from marking down usability problems in each prototype, you must also make comparative feedback (i.e., comparing their two prototypes against one another) by noting if certain problems are found across both prototypes or just in one. If a problem exists in only one prototype, note which one by using the labels Prototype #1 and Prototype #2. You should highlight enough similarities and differences between the two prototypes so that the team receiving your feedback will understand the advantages and drawbacks of each design and know which one they should probably implement in code in the coming weeks.

As an evaluator, you should spend about 20-30 minutes reviewing both paper prototypes for a particular team.

NOTE: If the team you are evaluating has updated their prototypes based on prior feedback, you can evaluate their newer prototypes. However, you should still evaluate TWO different ones in your session so that you can make comparative feedback.

Step 2: Receive heuristic evaluation feedback

Your team will receive feedback in 1 or more Heuristic Evaluation (HE) sessions with students who are themselves doing Step 1 of this assignment. Just like Step 1, it is your responsibility to find enough evaluators (who must not be your own teammates).

At least 1 evaluation is required; more is optional but highly recommended. If you can find only 1 evaluator, that is fine, but if more evaluators want to test out your paper prototype and give you feedback since they are completing Step 1 of this assignment, please be accommodating and let them do so (within reason). We want everyone to be able to find other teams to evaluate their prototypes, so some teams may get evaluated more than once.

Nielsen recommends 3 to 5 evaluators to get good results for heuristic evaluation. For this assignment, we require only 1 evaluator, but if you get more, you are not only helping your classmates do Step 1 of this assignment, but you are also getting better feedback for improving your app. It's a win-win for everyone!

Before meeting each evaluator, your team should know the flow of your paper prototype and be able to quickly find/move the pieces of your prototype to simulate the experience of using a real app.

For each HE session, your team members should have specific roles:

  1. Facilitator: This team member will greet the evaluator, explain how the session works, and give a brief introduction to your two prototypes. Once the session starts, this person will observe and take notes/pictures.
  2. Computer: This team member will be the computer ("wizard-of-oz") that manipulates the prototype pieces (i.e. when the evaluator presses a button on the prototype, the computer will change the prototype pieces to show what the app should do next). See here for an example paper prototype evaluation session.
(Your third team member doesn't have to be present; they can go off and do heuristic evaluation on another team at the same time.)

Have a copy of Nielsen's heuristics ready for the evaluator to refer to. You may provide evaluators with this worksheet to fill out while evaluating. You may also want to create a Google Spreadsheet containing the heuristics (with one tab for each evaluator) so the evaluator can write their feedback directly to you. A Google Spreadsheet shared with your team and the evaluators is highly recommended because it makes it easier for the evaluator to do their Step 1 of this assignment and for you to improve your design.

For this step, your assignment submission should contain a substantive, non-trivial list of potential changes that your own team plans to implement based off evaluator feedback from your 1 or more heuristic evaluation sessions where other teams evaluated your prototypes. You can focus on one of your prototypes that you like the most, or combine features from both if it makes sense. Each listed change should be justified by explaining which piece of feedback you received caused you to propose this particular change.

Step 3: Reflection

Think about the general characteristics of the two prototypes you evaluated in Step 1, and suggest potential improvements to address major usability problems that you identified in that team's prototype (not your own prototype!). Decide which problems are most important, then brainstorm potential solutions for the team whose prototypes you evaluated.

Finally, distill all of this down to one paragraph where you address the major problems you all identified, as well as the potential solutions. Include a sentence or two reflecting on what kinds of things you found heuristic evaluation valuable for, and what kinds of things it's not very useful for, in your experience.

Note that technically only one team member needs to write up this step (preferably the person who did the evaluating), but feel free to work on it together as a team to improve writing quality. Remember, each team needs to evaluate only one other team (but you can do more if you feel like it!).

Step 4: Create a Video Prototype

As a team, create a 1-minute video prototype that demonstrates a scenario where your app might be used and the potential users that might use it. The video should illustrate the features that your team plans to implement and how those features will be used. (Choose one paper prototype to focus on in your video.) We will play each team's video prototypes during studio.

(We will be very strict about enforcing a 1-minute time limit for these videos, to be fair to all students. Think of this like a hard 'page limit' on written assignments; it is not fair to your classmates if you go over the allotted time.)

Student Examples

Example 1 - This student did an incredibly thorough heuristic evaluation. We especially like how this student incorporated comparative feedback. Keep in mind that the this assignment was slightly different last year.

Example 2 - This is a weak example of a heuristic evaluation, as it wasn’t very thorough. The general structure of this example is decent, however, where problems are listed for each prototype and the heuristics and severity scores are identified. This example would have benefitted more from comparison of the prototypes.

For some more inspiration: watch two students do heuristic evaluation

Team Submission

For this assignment, ONE person will submit the assignment for their team, listing every team member's name and student ID number (PID) in the assignment submission.

Your write-up will contain the following in one single PDF:

  • The names of the people in the team whose prototypes you evaluated in Step 1, their studio section, and a description of their project idea. (Prototype description)
  • A typed heuristic evaluation of another team's two paper prototypes. For each prototype, this comprises a bulleted list of usability issues you found, along with their type and severity according to Nielsen's classifications. (Perform heuristic evaluation on another team's paper prototypes)
  • A paragraph addressing the major problems identified with the other team's two prototypes and potential solutions, as well as your overall thoughts about heuristic evaluations. (Reflection)
  • A substantive, non-trivial list of potential changes that your own team plans to implement based off evaluator feedback from your 1 or more heuristic evaluation sessions where other teams evaluated your prototypes. You can focus on one of your prototypes that you like the most, or combine features from both if it makes sense. Each listed change should be justified by explaining which piece of feedback you received caused you to propose this particular change. (Receive heuristic evaluation feedback from others)
  • A link to the team's video prototype (e.g., on YouTube). (Video Prototype)

Submit your single formatted PDF in Gradescope.

Evaluation Criteria & Grading Rubric

The rubric below contains criteria that are worth one point each and will be graded independently and in a binary fashion.

  1. Your submission includes the names of the people on the team whose prototypes you evaluated, along with their studio section name and a description of their project idea.
  2. Your heuristic evaluation applies at least 3 of Nielsen's heuristics across the two prototypes, and the connections to each heuristic are easy to understand.
  3. Your heuristic evaluation applies at least 6 of Nielsen's heuristics across the two prototypes, and the connections to each heuristic are easy to understand.
  4. All violations of heuristics have an appropriate Nielsen severity rating that will help teams prioritize their changes.
  5. Your evaluation feedback draws a useful comparison between the two prototypes (i.e., comparative feedback).
  6. Your reflection paragraph addresses the major problems identified with the other team's two prototypes and potential solutions, as well as meaningful thoughts about heuristic evaluations.
  7. You provide a non-trivial list of proposed changes to your own team's prototype in response to heuristic evaluations.
  8. The list of proposed changes to your own team's prototype is clearly justified by the feedback that you received from heuristic evaluations.
  9. Your video prototype is viewable online and is 1-minute long.
  10. Video prototype clearly demonstrates the goal of the app.
  11. Video prototype clearly demonstrates different features that are meaningful to the overall goal.
  12. Video prototype shows who the potential users of your app may be.