Using the System Usability Scale (SUS) to evaluate Schedule Builder

Overview

In this report, I analyze the usability of Cal Poly’s Schedule Builder feature within the Student Center using John Brooke’s 10-question System Usability Scale. I recruited 7 participants, all current Cal Poly Students, to complete a set of tasks in an unmoderated usability test, and then had them rate the usability of Schedule Building using a 10-question Google Form. A System Usability score of 53.93 was calculated using the results of the survey, pointing to there being room for improvement.

System Usability Scale

As Peres et al. (2013) note, “For almost 20 years, practitioners and researchers have been using the System Usability Scale (SUS) as a “quick and dirty” method of evaluating the usability of different websites, software, and other human-machine systems”. The System Usability Scale, defined by John Brooke in 1986 is a set of 10 questions which are answered on a 1-5 scale, 1 being “strongly disagree” and 5 being “strongly agree”. The questions are as follows:

  1. I think that I would like to use this system frequently.

  2. I found the system unnecessarily complex.

  3. I thought the system was easy to use.

  4. I think that I would need the support of a technical person to be able to use this system.

  5. I found the various functions in this system were well integrated.

  6. I thought there was too much inconsistency in this system.

  7. I would imagine that most people would learn to use this system very quickly.

  8. I found the system very cumbersome/awkward to use.

  9. I felt very confident using the system.

  10. I needed to learn a lot of things before I could get going with this system.

Procedure

Here are the tasks I asked participants to complete before evaluating the usability of the Schedule Builder feature of student center:

  1. View the courses available that satisfy a C4 GE on the 2019-20 catalog, and pick one to add to the working course list

  2. Add CPE 101 to your working course list

  3. View the different schedules you could build

  4. Add one of the schedules to your favorites, and then remove it

  5. Directly compare two schedules using the "compare" functionality

 Survey Results

 Calculating SUS Score

As Nathan Thomas of Usability Geek notes, calculating the score from survey results is performed as follows:

  1. For each odd numbered question, subtract 1 from the score (score - 1)

  2. For each even numbered question, subtract it from 5 (5 - score)

  3. Add the adjusted question scores up

  4. Multiply by 2.5

  5. Repeat for all responses

  6. Calculate the average of all responses to get your Software Usability Score (SUS)

Thomas later notes that the average SUS score is 68, which can be thought of as a C. 80.3 or above can be considered an A and 51 or under can be considered an F. After performing these calculations, it turns out that my participants gave Schedule Builder a very low 53.93.

Discussion

For the most part, my participants answered within the 2-4 score range on each question, with only a handful of responses going to either extreme. Questionnaires such as the one I utilized have the definite benefit of being fast,easy, and cheap to administer, and have an easily quantifiable output that is easy to analyze. The main drawback of using the method I did is that because it is asking broad questions about the usability of the site, it does not point out specific areas or features that should be improved. 

With unmoderated usability tests like mine, I luckily do not need to worry about the Hawthorne effect, where participants’ behavior is modified due to my presence, or moderator bias. On the other hand, my participants were biased because they were all already at least vaguely familiar with the piece of software being evaluated. It is a very common opinion among Cal Poly students that all Cal Poly software is lackluster, which likely sways the opinion of my participants.

Conclusion

At the end of the day, it would seem that the Schedule Builder could use some work based on the results of my survey. Function integration and system learnability are two very broad areas that my participants saw that could be improved within Schedule Builder. In the future, testing users who are unfamiliar with Schedule Builder would be a more accurate way to assess the usability of the feature in Student Center.

References

Peres, S. C., Pham, T., & Phillips, R. (2013). Validation of the system usability scale (SUS). Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 57(1), 192–196. https://doi.org/10.1177/1541931213571043 

Thomas, N. (n.d.). How To Use The System Usability Scale (SUS) To Evaluate The Usability Of Your Website. UsabilityGeek.com. Retrieved December 9, 2021, from https://usabilitygeek.com/how-to-use-the-system-usability-scale-sus-to-evaluate-the-usability-of-your-website/.