This week, I conducted further usability tests on the same site (Seattle University’s College of Education website), with the same test script, the same test scenario, the same tasks, as last week, but this time on a mobile device — an iPhone 6s — with three new participants. I recorded the tests, again, using QuickTime Player on a MacBook Pro, by connecting the iPhone to the MacBook using a USB connection cord.
Mobile test participants
- #1: 36 year old male; heavy web user (8-10 hours a day); Seattle University employee (so familiar with the website generally) but never visited the College of Education website before.
- #2: 38 year old female; medium web user (3-4 hours a day, but generous estimate); not a smart phone user; full time graduate student at Seattle U (so familiar with the website generally) but never visited the College of Education website before.
- #3: 44 year old male; medium web user (~ 4 hours a day); not an iPhone user; research assistant at Seattle U (so familiar with the website generally) but never visited the College of Education website before.
The test scenario and the tasks
For the purposes of this usability test, I asked users to pretend they were all prospective graduate students looking for more information on the Master in Teaching (MIT) program at Seattle University.
Task 1: Find out the date of the next MIT information session.
Task 2: Find out the next deadline for applying to the MIT program.
Task 3: Find out what the admissions requirements are for the MIT program.
Task 4: Find out who to contact if you have additional questions about applying to the MIT program.
Task 1: Participant 1 navigated from the College of Education home page to the “Graduate Degrees” page (clicking on a black flag button link in the primary content section [rather than from the menu]) then scrolled down to find the right program, clicked the black flag button for the “MIT program” and then scrolled down to find the “Upcoming information sessions” he found toward the bottom of the page (though he noticed two things in this section he found concerning: 1) that the “upcoming” sessions had past dates still listed, and 2) that there was a typo “Novermber.” He completed the first task in about 30 seconds.
Task 2: The first thing Participant 1 did was click the “Apply Now” call to action button, but when he realized he had left the MIT program website and had gone to the Graduate Admissions site, he immediately hit the back button. Then he scrolled to the top of the Master in Teaching home page, and using the navigation menu, clicked on “Apply” and then immediately clicked on “Admissions requirements” and scrolling down found the application deadline in the data table right below the heading (but was irritated that the table did not fit nicely in the mobile screen). Again, it only took Participant 1 about 30 seconds to complete the task.
Task 3: As he was already on the Admissions requirements page, it took Participant 1 about 5 seconds to scroll down the page to find the list of admissions requirements.
Task 4: From the Admissions Requirements page, he scrolled back to the top and accessed the menu, first clicking on FAQs, and scanning the FAQ page content and not finding contact information, he went back to the MIT home page, and clicked on the “More information” call to action button which took him to the Graduate Admissions “Request information” form, which frustrated him. So he scrolled to the bottom of the page, and in the footer found contact information for Graduate Admissions and said that’s who he would contact. This took him about a minute to do, and while not actually finding contact information in the MIT program, contacting Graduate Admissions would also work, but be a more circuitous route.
Task 1: Participant 2 scrolled down to the bottom of the College of Education home page and found the “Information Sessions” information, completing task 1 in about 10 seconds (though she pointed out that the list of info sessions still included dates that had already passed).
Task 2: Participant 2 followed a similar route as Participant 1, scrolled up and clicked on the “Graduate Programs” button, and then the “MIT Program” button, all without accessing the menu. Scanned the MIT home page, taking a detour to the Graduate Admissions by clicking “Apply Now,” scanned the page and did not see any dates so she hit the back button, returned to the MIT page, and then tried the “Request Information” button which returned her to Graduate Admissions. This brought her to the web form but it did not load immediately on the phone, so initially it looked like a blank page which confused the participant and so she clicked back before the form was loaded.
When she returned to the MIT home page, she finally accessed the menu and selected the Apply page, and then the Admissions Requirements page. It took her about two and a half minutes to complete task 2.
Task 3: She completed task 3 in about 15 seconds by slowly scrolling down the “Admissions Requirements” page and finding the list there.
Task 4: Then participant 2 completed task 4 in about 20 seconds by returning to the “Apply” page and scrolling down to the bottom of the page stated she would contact the program at “firstname.lastname@example.org.”
Task 1: Participant 3 completed task 1 in about 10 seconds, quickly and easily scrolling down to the “Information session” section on the College of Education home page.
Task 2: Participant 3 seemed especially perplexed by Task 2 and navigated through several pages scanning for deadline information without finding what he was looking for. From the College of Education home page, he accessed the menu, clicked on “Graduate Degrees,” then “Master in Teaching.” Didn’t see any deadline information, so he clicked the call to action button “Apply now” and ended up in Graduate Admissions, didn’t see any deadlines on that page, then clicked the “Request information” call to action button and got a blank page for awhile, until the form loaded. Clicked back to the MIT program webpage, accessed the menu and selected “FAQ” which again did not present him with any information about application deadlines. He clicked the back button again and then went back to the menu and this time selected “Degree Options.” By this time he was very frustrated and said he was giving up on surfing for the information and was going to use the search bar. He was further frustrated trying to access the search bar because the main Seattle U navigation menu is so very long on a mobile device that he didn’t see the search bar at all when he first accessed the menu and had to scroll down to finally find it.
As soon as he found the search bar, and typed in “Master in Teaching deadline” the first search result was for the “Application requirements” page on the MIT website. It took him 7 and a half minutes to complete Task 2.
Task 3: After his frustration in the second task, Task three was much more straight forward and completed in 10 seconds by scrolling further down the “Application requirements” page.
Task 4: Task 4 was equally quick (about 10 seconds) for Participant 3, as he scrolled down to the footer and declared he would contact “email@example.com.”
Analysis and next steps
It is very interesting comparing and contrasting the results of these usability tests on a mobile device versus a laptop computer. It seems as though these sites were not designed “mobile first” and while it didn’t necessary hinder their ability to complete the tasks (with the exception of not accessing the menu quite as much, because the menu was hidden on the mobile site and had to be clicked on to access it, where as the users who did this usability test on the laptop had much easier/quicker access to the menu) the fact that things didn’t necessarily fit in the screen did seem to bother the participants to varying degrees.
Participant 1 pointed out several times where it was clear that the pages he was navigating to and through were not intended to be viewed on a mobile device and found it to be especially annoying while completing the tasks: such as the Jumbotron picture on the College of Education home page) didn’t fit on the mobile screen and neither do the embedded YouTube videos, nor the tables. Participant 1 also remarked that the font size for the headings seemed to be too large for the screen size and the font size for the regular text seemed to be too small, and that the low contrast of the yellow color of the headings on the white background was especially difficult on the mobile site.
Participant 2 spent a bit of time remarking on how strange it was that the information about the “Info sessions” was so far down on the page, and that she thought it was rather important and should be closer to the top (whereas on the laptop/wider screen, those content boxes appeared on the right-hand column of the screen and closer to the top – but with the narrower screen size, those content boxes were moved to the bottom of the page forcing mobile users to scroll a lot further down to find the information they were looking for. And Participant 1 and 3 both had a difficult time with clicking links with the iPhone, holding their finger down too long and inadvertently activating iPhone the zoom feature, and sometimes Participant 3 wasn’t quite sure what was a link/what was clickable on the mobile site and stated several times that he preferred to use the Seattle U website on a device with a larger screen.
I would recommend that Seattle U in general do some more mobile usability testing to determine if there are fixes they can do with the design template to make the conversion to the mobile size a little easier on their users.