Mobile Usability Test Summary
For this mobile usability test, I continued my analysis of the Central Co-op website: http://www.centralcoop.coop
The Central Co-op is an independent co-operative natural food grocer with locations in Seattle and Tacoma. A co-op is a business owned and operated by and for its users. Member-owners receive benefits such as voting privileges, access to special sales, an annual dividend based on individual expenditures, as well as regular communications about the workings of the business. To become a member-owner, an individual must purchase a share in the business.
A previous usability test of the desktop website was completed prior to this investigation of the mobile user experience. Result and recommendations of that test are published in this User Science Journal here: Usability Test of the Central Co-op website
As in the desktop site usability test, the mobile usability test examines the ease with which users are able to navigate the mobile site, in order to drive them to content supporting the following primary goals:
- Member-owner recruitment and education
- Locating and shopping in the physical store
- Promoting enrollment in new grocery delivery service
For this test, there was 1 test administrator (myself) and 4 test subjects.
The test script from the desktop site test was updated to reflect the use of the mobile site. Both were adapted from a free, downloadable script found in Rocket Surgery Made Easy© by Steve Krug.
The opening script was read aloud by the test administrator in order to explain the test procedure and provide opportunity for participants to ask questions prior to the start of the test.
At the end of the introductory script, subjects were asked to give permission for their test session to be recorded, all subjects agreed and signed a consent form prior to the start of the test.
A Logitech webcam with microphone attached to a small tripod was used to record test subject interaction with the mobile device. A white cloth was used under the test area to increase brightness and image quality.
The webcam was tethered to a laptop, which the test administrator used to view the test. The Chrome extension Screencastify was used to record both audio and web cam activity throughout the test from the laptop.
Subjects were asked by the test administrator to navigate to the website on the mobile device and give a general impression of the main page. This also allowed them to be comfortable with the test set up and web camera placement. Time was limited to 3 minutes.
After the initial view of the home page, the subjects were asked to complete 6 scripted tasks. All of the tasks were presented in written form and were also read aloud by the test administrator.
In the desktop site test summary, I made recommendations to simplify several of the more complex tasks, by breaking them into smaller parts to improve clarity for the test subject. For this mobile test, I revised accordingly, which increased the total number of tasks from 4 to 6.
Questions number 5 and 6 remain the same as in the previous test.
- Are you able to locate information about how to join the Central Co-op?
- Are you able to find the cost for membership?
- Are you able to find out what benefits are included with membership?
- Are you able to location information about what a co-operative business is?
- Are you able to locate the address and hours for the Co-op location in Seattle?
- The Co-op has started a new grocery delivery service. Are you able to find out whether your zip code is within their delivery area?
The total time for each session from start of introduction to the end of the recording was designed to last no more than 15 minutes.
The test administrator recruited volunteers from among personal friends and colleagues. Four test subjects volunteered to be participants in this test.
- Subject 1: Female, an employee of Seattle University, I-Phone 5
- Subject 2: Male, an employee of Seattle University, I-Phone 5
- Subject 3: Female, an employee of Seattle University, I-Phone 5
- Subject 4: Mail, an employee of the Federal Government, Samsung Galaxy 3
Only one of these participants, Subject 4, is a member-owners of the Central Co-op. None of the subjects had visited its website on desktop or mobile prior to the test.
Of the three participants who are not member-owners, all expressed having prior knowledge of this business as a brick and mortar store near Seattle University where they are employed.
All subjects used an I-Phone 5 (IP5), except for Subject 4 who used a Samsung Galaxy 3 (SG3) device.
The SG3 was a small version of the desktop site, with small hard to read items.
In order to complete the tasks for this test, it was necessary to turn the phone horizontally allowing a more readable presentation of the content.
The initial screen presentation of the Central Co-op’s homepage on the IP5 was substantially different than the SG3.
The IP5 delivered a more responsive layout in appearance, but hid many of the buttons and cards that normally appear on the desktop site.
This limited the test subject choices when navigating from the home page to complete tasks, but they were provided with a mobile friendly menu, which was not available to the SG3 user.
1. Are you able to locate information about how to join the Central Co-op?
4 of 4 subjects completed this task successfully.
All subjects were able to locate the correct page to join the Co-op, but there was not a clearly marked navigational path for them to follow from the home page. Most users were able to find this content in under 1 min, but Subject 1 searched for 1:35 minutes before finding the page.
Subject 1 Pathway:
After scanning entire home page for a prominent button, she expanded the top menu.
From here, she clicked on “Ownership” bringing her to a long page of content explaining the ownership structure of the Co-op. In the desktop version this page has clearly marked buttons on the right, but in IP5 these buttons are off the screen.
The subject does not notice a link at the bottom of this page to “Join.” The link is not underlined or styled much differently than the long page of text.
At this point, Subject 1 expands the menu again and clicks “About Us” opening another long page of content describing what type of co-operative business they run. Again, the subject does not see a link at the bottom of this page to “Join.” This link is nearly identical to the previous page, also not underlined or styled much differently than the long page of text.
Subject 1 returns to the home page to look again for a button or other recognizable path. From there the subject tries “Hours, Locations, Contact” and after scanning the page comments, “well, joining the Co-op is not super intuitive, I mean, one would think there would be a big button on the main page that says join.”
At this point in the task, the subject has spent approximately 01:08 searching for a way to join and asks the test administrator if she should continue looking. The administrator replies affirmatively and the subject navigates back to the home page to begin again.
From the home page, she clicks the “Weekly Owner Coupon” and is taken back to the “Ownership” page that she had visited near the start of this task. This time, the full page displays along with the large buttons that were off screen on her first visit. From here, she pinches out the view to be able to read the buttons and selects the “Join” button.
Total task time was 01:38.
Subject 2 Pathway:
Subject 2 expands the top menu to search for how to join. He selects “Ownership” and scans the page. After some reading, the subject notices the link at the bottom of the page (mentioned above in Subject 1 pathway) and selects it arriving at the correct page to join.
Total task time was 00:49.
Subject 3 Pathway:
After scanning the home page for a link to join, Subject 3 expands the top menu. She reads through the list of menu items and says, “I don’t know where I would click to join.”
She then scans the home page a second time and returns to the menu again and states, “I guess, I’ll try ownership?” When she arrives at the page, she scans it and finds the link at the bottom of the page to join. At this point she says, “I guess they assume that I would know that Co-ops are run on people power.”
Total task time was 00:41.
Subject 4 Pathway:
Subject 4 is the only subject not using an I-Phone, which means he had no mobile friendly menu at the top as did the other subjects. He chooses to use the search box to look for a way to join and quickly navigates to the “Ownership” page based on the results. At this point, he comments, “it brought me to ownership, which is a tab I can see, but I had no idea that ownership meant join.”
Since he was using an SG3 with the device held horizontally, the full width of the site is visible, including the large buttons on the right, which IP5 users weren’t always able to see. He selects the top “Join” button, but comments “the text is really small and buttons really skinny.”
Total task time was 00:25.
- Average task time among IP5 users was 00:63.
- Average task time among all users was 00:53.
Improve Visibility of Navigation Elements for Mobile. Even with a seemingly more mobile friendly look and feel, the I-Phone users took significantly more time than the Samsung user to complete this task. This is due, in part, to missing navigation elements, which are available on the desktop and SG3 displays, such as the large buttons on the home page and right side of most pages. How to join should not be a hidden element to any visitor to the site.
- Add buttons in text rather than off to the side on pages for “ownership”, “about us”, and all other pages where a non-member might visit for information.
- Replace “subtle” links at bottom of pages with more noticeable, clickable formatting, or buttons.
Make Menu Names More User Friendly. Three of the test subjects commented that they did not know that the word “ownership” would lead them to information about becoming a member of the Co-op. I would recommend clearer wording for navigation elements, such as
- Add “Join the Co-op” to the top navigation in the first position.
- Improve positioning of the current “Join the Co-op” button on the home page, which is underneath a news story about light rail. This button is omitted on IP5 and is too small to see on SG3.
Improve Search Function. Even though the SG3 user completed the task at a much faster pace than the other subjects, his strategy of entering “Join” in the search box still brought him to the wrong page initially. Improving the search function on the site would give those users who prefer to use the search function a seamless and nearly instantaneous result.
2. Are you able to find the cost for membership?
4 of 4 subjects completed this task successfully.
This task was easily accomplished by all subjects.The information can be found on the “Join” page where most subjects were positioned at the end of Task 1 and, even if they had navigated away from this page, most of them remembered seeing the information during the completion of the prior task.
Although this was one of the tasks most easily achieved for the subjects, there is still room for improvement. The textual content of the page is long and important details, like the cost of membership, could be better styled to stand out.
3. Are you able to find out what benefits are included with membership?
3 of 4 subjects completed this task successfully.
This task was by far the most difficult for the majority of test subjects.The three IP5 users scanned and then re-read the “Join” page anticipating that benefits of membership would be included on this page, or that there would be pathway to navigate to them in the content. Only the SG3 user hit on the link “return on your investment” right away.
Subject 1 read through the “Join” page for a minute before moving on to “Co-ops 101” spending nearly another minute reading content before giving up and commenting that she expected to find this information along with how to join.
Subject 2 visited 7 pages before returning to the “Join” page and noticing the “return on investment” link. After spending well over 2 minutes searching, he commented that he “didn’t feel guided” by the content.
Subject 3 spent over a minute reading and re-reading the “Join” page then navigated to “Co-ops 101” in search of the benefits, finally she decided return to the top menu, but this time on the desktop site (or “Full Site”) view and found the link to “Benefits” there in the top tabbed navigation.
Subject 4 had the fastest completion time (a mere 00:12), but commented that “return on investment is lingo” those new to the Co-op might not know, finally saying, “Why doesn’t it just say benefits?”
All in all, the subjects spent the bulk of their time divided between the “Join” and “Co-ops 101” pages.
*Subject 2 visited 7 pages total, time on those pages are not reflected on table.
Use Clear, Jargon-Free Language. Although the phrase “What is the return on my investment?” may be an accurate way to describe what a person can expect in return for becoming a member, it isn’t clear enough for a person new to the concept of a co-operative business. Alternatives might be:
- What are the benefits of becoming a member of the Co-op?
- What do I get when I become a member?
- What can I look forward to as a member of the Co-op?
- I’m ready to join, what happens now?
Even “benefits” is not an entirely jargon-free word choice. Including it in the task prompt did little to improve completion time, because the term is only used under the “Ownership” menu tab and not immediately obvious to a user seeking information on how to become a member of the Co-op.
Create Better Navigation for Mobile Users. One of the challenges for most of the test subjects, was lack of clear navigation to the “Benefits” page. Tab navigation that is visible on the desktop site, were hard to read and click for the SG3 user. IP5 users were only able to see the tabs while visiting the “Full Site”.
The SG3 user was never observed opening the tabs at all. Only one IP5 user was observed opening the ownership tab while searching for benefits.
Improve Clickability. Better styling of links in text, as in the case of the link for “return on investment”, should be more obvious, especially for mobile users. The content text is already rather small, having subtle styling of links makes it more difficult to find.
4. Are you able to locate information about what a co-operative business is?
4 of 4 subjects completed this task successfully.
The point of this task is to see if the subjects are able to locate general information about co-operative businesses that is located on the “Co-ops 101” page.
Three of the subjects completed this tasks easily.
- Subject 1 and 3 remembered visiting this page in error while searching for how to join, or member benefits and then navigated back to it to complete the task.
- Subject 4, using an SG3 with screen horizontal, was able to see the large button for “Co-ops 101” that is sometimes off screen for IP5 users.
Subject 2, however, had a more difficult time for these reasons:
- This subject didn’t remember landing on the “Co-ops 101” page in error, as Subjects 1 and 2 did, while completing earlier tasks.This is the only “advantage” Subject 2’s fellow IP5 users had in completing this task.
- For most of this task, Subject 2 wasn’t able to see the large button for “Co-ops 101” that Subject 4, the SG3 user, could see.
To complete this task, Subject 2 followed this path:
- “About Us” page
- “Ownership” page
- Expanded main menu
- Scrolled to the bottom of the home page
- Clicked the button to “Visit the Full Site”
- Scanned the “Full” version of the home page
- Scanned the footer links and finally found “Co-ops 101”
Throughout the majority of the steps, Subject 2 was reading through long pages of content, which added significantly to his total time of 01:52.
- Subject 1 total time 00:15
- Subject 2 total time 01:52
- Subject 3 total time 00:09
- Subject 4 total time 00:08
Use Clearer Button and Menu Titles. Subjects 1 and 3 completed this task very quickly, but only because they remembered visiting the page in error during previous tasks. Although memorability/learn-ability can be a positive testing result, in this case it suggests that the actual work of the task was shouldered by time spent completing previous tasks, which were difficult to complete and forced user to visit multiple pages.
Subject 4 had the benefit of more visible navigation elements and the quickest completion time, but commented “Why not say, What’s a Co-op? Instead of Co-ops 101?”
“What’s a Co-op?” is a clearer title than “Co-ops 101,” which, as demonstrated in this test, became a catch-all for subjects whenever they became bewildered during the tasks.
Improve Navigation for Mobile. Although Subject 2 didn’t remember visiting “Co-ops 101” in previous tasks, as soon as he saw it in the footer links, he was confident that he had completed the task. If he had had the same access to navigation elements as Subject 4, using an SG3, it is likely his overall would have been improved.
5. Are you able to locate the address and hours for the Co-op location in Seattle?
4 of 4 subjects completed this task successfully.
As in the desktop test, this task was successfully achieved in minimal time by the test subjects. Most commenting that they remembered seeing a link to “Hours, Location, Contact” either in the mobile menu for IP5 or in the header of the page. In fact, the IP5 test subjects performed better in the mobile test than the desktop test subjects, because their mobile view isolated the content at the top of the page with no diversions.
Even though this task was easily performed by the test subjects, improvement in the presentation of this content is recommended. There is a lot of white space available on this page and the text is rather small and some users were observed pinching out the page in order to increase the size of the text.
The SG3 user demonstrated the size he would prefer to see on a mobile page of this type below.
6. The Co-op has started a new grocery delivery service. Are you able to find out whether your zip code is within their delivery area?
3 of 4 subjects completed this task successfully.
Subjects performed this task with only moderate observable difficulty. Pathways to complete the task were split. Two subjects navigated to the Instacart page, which is part of a different domain, and 2 of them found the info page on the Co-op website.
Subject 1 and 3 remembered seeing the banner graphic for the delivery service during previous tasks where they visited the “Full SIte”. They easily navigated to the Instacart page from there.
Unlike subjects in the desktop test, neither subject were disturbed by being taken to a new domain. In fact, quite the opposite, as the Instacart page is very mobile friendly appeared consistent with their expectations of web app appearance and functionality.
Subject 2 used the IP5 mobile menu to find the informational page on the Co-op website.
He navigated to the page about the delivery service, which has a lot of textual content and began scanning for a way to check whether his zip code was in the delivery area.
The page was not very mobile friendly and required that he scroll back and forth to read the content. He also had to scroll down locate information about delivery areas. He didn’t see the link to check for zip codes at all. He scanned the list of selected neighborhoods and was satisfied that there was no point in looking at his exact zip code. He was the only subject who didn’t complete the task.
Subject 4, the SG3 user, employed a similar tactic as he had previously by using the search function to look for “delivery”. He commented that it was a little difficult to access the box, because it was small. His search returned 169 results on 17 different pages of the website.
Luckily, the first result took him to the correct page on the Co-op website. The page displayed more fully on SG3 with a rotated screen than it did for the IP5 user. There was less scrolling needed to find the pertinent information.
Subject 4 completed the task quickly and commented that the zip code popup was the most readable page he had seen throughout the entire test.
Improve Navigation Elements for Mobile. The IP5 users who had the best experience completing this tasks navigated from the graphic banner link only available from the “Full Site” as opposed to the mobile menu. This navigation element is memorable and should have a place in the initial mobile view for all users. Adding the same or similar graphic link at the top of the informational page that displays correctly would also attract more users to the Instacart page, which is more responsive than the page on the Co-op site.
Improve Responsiveness and Content. The delivery information page on the Co-op site has similar content issues as other pages we have discussed throughout this summary. There is a lot of content in small print that requires scrolling and pinching out for better viewing. Larger, clearer textual elements, like prominent headers instead of a large unclickable banner, that appear near the top of the page would assist users in locating information quickly and painlessly.
Mobile vs Desktop
The nature of a mobile device is to be “on the go.” Improving task times for mobile users is essential. In this test, users who went beyond a 00:40 second window in order to complete tasks became visibly irritated. Desktop testers didn’t seem to be as concerned with speed and also on average completed the entire test in less time.
Approximate Average Time for Entire Test*
- Desktop Users: 06:73
- Mobile Users: 07:66
*Time calculations do not account for time subjects spent commenting during the test.
I-Phone vs Samsung Galaxy
Although initially the I-Phone had a more mobile friendly appearing presentation, the Samsung user had the best completion times on nearly every task. In part, because he chose to use the search function, but also due to the fact that no elements of the website were hidden on his device.
In general, the look and feel of this site is pleasant for desktop users, but mobile users have a different experience. Both types of users would benefit from more accessible content.
Make Content More Scanable and Easier to Access. Although overall design of the site is pleasant visually, the textual content is presented in a small font and many pages could benefit from better use of headers and other graphic elements to break it up. One mobile subject commented that the pages where “cluttered” and they would like to see information broken up into smaller, logical chunks.
Make Site More Responsive. It is pretty clear that this site is meant for desktop and it performs best in that environment. When IP5 users receive a “friendly” version of the site, there is a lot of missing, or off screen content. The content not appearing doesn’t seem to be intentionally selected. Some of the missing content detracts from the users experience and ability to navigate the site, like buttons to direct them to join the Co-op.
It would be worthwhile to perform additional testing for both desktop and mobile users after implementing recommendations made throughout this summary. That said, if considering a substantial redevelopment of content based on these findings, consider performing additional testing across a wider breadth of both mobile devices and desktops. This would provide the practical information needed to create a better user experience overall.