usability test report€¦ · questionnaire. 1 hour usability test session was conducted for each...
TRANSCRIPT
BY
1
Usability Test Report FamilySearch Website
Contributors2
Team HASK Heather Reiss, [email protected]
Alex Marantz , [email protected]
Sarah Letham, [email protected]
Kavitha Krishnan, [email protected]
FamilySearch Team Tyson Stokes
Alan Blood
Executive Summary3
Project Goal – To identify strengths and weaknesses of FamilySearch website
Scope – Moderated usability sessions with experienced users and beginners with a prototype of the existing website design
Methodology – 12 participants (ages 30-70) were recruited by FamilySearch and HASK Designs. Participants were asked to fill pre-test questionnaire. 1 hour usability test session was conducted for each participant using the FamilySearch website. Participants filled SUS at the end of the session
What worked well – Participants like the overall website design, Site Map, Visual Design, Help, Getting Started materials help new users use website with limited training an support
Recommendations –Use more universal terminology, or define family history specific terms for beginners. Highlight new features of the site on the homepage. Make critical elements/actions on the site more visible
Agenda4
Introduction About FamilySearch About the Users
Methodology Usability Test Objectives Usability Test Method and Tools Scenario and Task Overview Demo of FamilySearch Prototype Definition of Severity Categories
Findings and Recommendations Overall Assessment SUS Analysis, Success & Failure by Task Summary of Positive Site Aspects Summary of Issues Prioritized by Severity Expert Review Comparison
Limitations and Future Research Appendix
FamilySearch5
FamilySearch is a non-profit organization that gathers, maintains, and shares genealogical records worldwide
FamilySearch.org is designed to help people learn more about their family history for free
FamilySearch has records of 3 billion names from all over the world. They rely on a large volunteer pool of over 4700 people to preserve and share the largest collection of genealogical and historic records in the world
Introduction Methodology Findings Future Steps
About FamilySearch Users6
Typically age 50 years or older
Almost equal number of beginner and experienced users
All participants have an interest in learning more about their family history
Users may not be computer proficient but are able to learn and use FamilySearch
Many participants belong to the Church of Jesus Christ of Latter-Day Saints (LDS Church)
Introduction Methodology Findings Future Steps
Recruited Participant Information7
12 participants, age range 30-70
8 beginner, 4 experienced
8 female, 4 male
8 members of the Church of Jesus Christ of Latter Day Saints Of the LDS members, 4 were beginners, 4 were experienced
3 participants were recruited from a convenience pool (friends and family of HASK Design)
7
Introduction Methodology Findings Future Steps
Usability Test Objectives8
To identify design inconsistencies and usability problems in the existing FamilySearch website design
To identify positive design elements in the existing website design
To determine the findability of critical elements within the FamilySearch site
To highlight the most critical inconsistencies and problems in the design
Introduction Methodology Findings Future Steps
Usability Test Method9
All but two usability sessions were conducted remotely due to the geographic location of the participants relative to the HASK team
9 out of 12 participants were provided by FamilySearch - mix of experienced and beginner users. The other 3 participants were beginner users recruited by the HASK team
Each session consisted of a moderator and a note taker. Test sessions were guided by pre-defined scenarios and tasks
The team used the client‟s WebEx account to run and record the sessions. Videos are already available for the client to view.
Introduction Methodology Findings Future Steps
Usability Test Method10
There were a total of 6 scenarios and 18 tasks. Due to time constraints, not all participants were able to complete all scenarios and tasks. (see appendix for full task list)
For each session, scenarios were randomized, as well as the tasks within each scenario
The analysis was primarily based on qualitative information gathered during the studies. Specifically, the team captured task success, and direct user feedback
Desired task success rate was at least 70%
Each user filled in the 10 question positively worded SUS survey after completion of the test
Introduction Methodology Findings Future Steps
Scenario & Task Overview11
There were a total of 6 Scenario Categories with one or more tasks under each category
Find Ancestors: 1 task focused on finding and attaching historical records to ancestor profiles
Build/Extend: 2 tasks focused on building and extending a family tree
Get Help: 4 tasks focused on finding help resources on the FamilySearch website
Volunteer: 6 tasks centered around different volunteer aspects
Account: Editing personal account information
Other: Tasks focused on the discoverability of providing feedback to FamilySearch, learning about new features and about the organization
Introduction Methodology Findings Future Steps
FamilySearch Prototype Demo12
http://invis.io/ACPZB2HB
Introduction Methodology Findings Future Steps
Definition of Severity Categories 13
High: Leads to task failure across multiple participants. Highest combination of frequency, business impact, and effect on performance
Medium: Leads to task failure and frustrates users
Low: Causes some hesitation or frustration but does not affect successful completion of the task
Scope Local – Exists on a single page or screen Global – Scope of the problem is larger than 1 page or screen
Introduction Methodology Findings Future Steps
Overall Assessment14
Overall positive experience Appealing visual design
Many positive features – Fan chart, Site Map and Help
Getting Started – adequate to help new users navigate website with limited training and support
A few issues pose potential roadblocks for users and need to be addressed Use more universal terminology, or define family history
specific terms for beginners
Make critical elements/actions on the site more visible
Introduction Methodology Findings Future Steps
SUS Findings
Utilized a 5 point scale SUS questionnaire.
All ten questions were worded positively.
Highest SUS score: 92.5
Lowest SUS score: 45.0
Based on research by Usability.gov, a score of 68 or above is considered an above average SUS score.
SUS Average: 71.0
15
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
41-50 51-60 61-70 71-80 81-90 91-100
Fr
eq
ue
nc
y
SUS SCORES
Frequency Distribution of SUS Scores
Introduction Methodology Findings Future Steps
SUS Findings
Highest rated question:
“I think I would like to use this system frequently.”
Lowest rated question:
“I could use the website without having to learn anything new.”
16
4.33
4
4
4
3.92
3.75
3.75
3.75
3.67
3.17
I think that I would like to use this system frequently.
I think that I could use the website without the support of a technical person.
I found the various functions in the website were well integrated.
I thought there was a lot of consistency in the website.
I felt very confident using the website.
I found the website to be simple.
I thought the website was easy to use.
I would imagine that most people would learn to use the website very quickly.
I found the website very intuitive.
I could use the website without having to learn anything new.
Mean SUS Scores by Question
Introduction Methodology Findings Future Steps
17
Success Rate by Task
7
9
5
8
6
3
5
4
2
2
6
6
7
6
8
4
1
1
3
1
4
1
6
3
3
2
6
5
5
1
1
1
5
Attach a Death Certificate
Edit Ancestors Birth Date
Locate Ancestor Profile Notifications
Locate Nearest Family History Center
Find Get Started Guide
Find info on State/Country Resources
Find Error Status Request
Learn more about Indexing
Start Indexing
Country Research Articles
Participate in Evaluations
Donate Records
Donate Money
Edit User Profile
Site Map
Find FamilySearch About Section
Give Feedback
Find "Whats New"
Number of Participants per Task (12 total participants tested)
Ta
sk C
ate
go
ry Success
Success w/ Assistance
Incomplete
Introduction Methodology Findings Future Steps
18
General Observations
Experienced vs. Beginner Performance19
Differences between experienced users and beginners:
Experienced users in most cases found all tasks to be simple
Experienced users had an overall understanding of the website functionality, family history terminology, and how to volunteer and access support when using the website
"I'm not sure a beginner would know to click on About to find out new features. That’s not intuitive to new users. In my opinion, this What's New should be on the front page"
Introduction Methodology Findings Future Steps
LDS vs. non-LDS Members20
Differences between LDS and non-LDS Members:
Members of the Church had little problem understanding FamilySearch terminology such as “Indexing”
“I would look at Indexing. Just because I know that word from church. I know people are involved in indexing projects.”
Members of the Church have easy access to trainers who help them learn how to use the website
Introduction Methodology Findings Future Steps
21
Positive Findings
Positive Findings22
Users had the best performance, most positive feedback, and highest success rates in the following areas:
Altering Information
Getting Started Guide
Contacting FamilySearch
Utilizing FamilySearch Resources
Site Map
Introduction Methodology Findings Future Steps
Altering Information23
Good performance overall on information editing tasks:
Change Mary Howell‟s Birth Year from 1867 to 1864 – 9/10 successful
Users were asked to edit the birth year of a relative on the relative‟s profile page
Change your mailing address in your FamilySearch contact information to 500 Main St. – 6/8 successful
This task tested the usability of editing of account information through the profile page
Users were able to find the correct path and procedure with few delays
Introduction Methodology Findings Future Steps
24
“Oh, there we go, that was easy.”
Change Mary Howell‟s Birth Year
Introduction Methodology Findings Future Steps
25
"I would immediately open settings, change address, and save changes."
Change Your Mailing Address
Introduction Methodology Findings Future Steps
1
2
3
Getting Started Guide26
Getting started guide for beginners are successfully found and utilized:
Where you would go on this website to learn how to begin compiling family history – 6/7 successful Users were asked to show the moderator where they would go to get started on
the FamilySearch website
Get started indexing – 5/6 successful After users were asked to identify the indexing portion of the website in a
previous task, they were asked to show where they would go to actually begin the process
Correct paths found quickly and easily by users New users can take multiple paths to get started, such as Getting Started
Guide or Get Started on Front Page
Introduction Methodology Findings Future Steps
27
“This is a great way for [my friend] to get started on [this site]”
Where to Begin Compiling Family History
Introduction Methodology Findings Future Steps
28
"For me, I would do the Test Drive first. If I was comfortable doing this, I would just find a project"
Indexing
Introduction Methodology Findings Future Steps
Contacting FamilySearch29
All Participants were successful on tasks involving contacting FamilySearch:
Give Feedback to FamilySearch– 8/8 successful
Users were asked to show how they would give feedback
Participants found multiple effective paths for contacting FamilySearch
Find the Location of the Nearest Family Center– 8/8 successful
In this task, users were asked to find a local family center to help with their research
Some participants experienced delays or difficulties finding task path, but all were successful
Introduction Methodology Findings Future Steps
30
On locating the feedback feature: “That was easy to find"
Give Feedback to FamilySearch
Introduction Methodology Findings Future Steps
31
“I would first go to Get Help and then Visit Us, and I would search for one near me. This is easy to find“
Locate the Nearest Family Center
Introduction Methodology Findings Future Steps
1
2
FamilySearch Resources32
Participants performed well when identifying resources related to FamilySearch organization:
Teach friend about FamilySearch organization background–6/7 successful
Subjects were asked to find the “About” section of FamilySearch, framed as showing the information to a friend
Donate Money to FamilySearch – 6/7 successful
Users were asked to donate money to the FamilySearch website
The participants experienced some difficulty and delay finding the link, so making the link more prominent may increase efficiency of the task
Introduction Methodology Findings Future Steps
33
"I would expect that 'About FamilySearch' would give a background information about the organization"
About FamilySearch Organization
Introduction Methodology Findings Future Steps
34
"I would go to „Give Back‟ from the home page, then to „Donate Money‟ "
Donate to FamilySearch
Introduction Methodology Findings Future Steps
1
2
Site Map35
Participants were able to find the Site Map and thought it was a good function:
Find Site Map–7/8 successful
Participants were asked to find the site map, a list of every page on the site
Subjects had a very positive reaction to Site Map if they had never used it before
Subjects said they thought it was a well-organized and useful tool
Site map not hard to find when user is looking for it, but some participants had been using the site for a long time and didn‟t know it was an option
Introduction Methodology Findings Future Steps
36
“This is cool, I like this list…it never would have occurred to me...buried at the bottom in such small print.”
Site Map
Introduction Methodology Findings Future Steps
Additional Comments and Feedback37
We received positive feedback about the following areas, but do not have task data about performance
Fan Chart A type of family tree with the user in the
center and previous generations extending outward, arranged in a “fan” shape
Users found the fan chart easy to use Multiple users navigated through the fan
chart for tasks “I love the fan chart. [My fan chart] is my
Facebook profile picture!”
Design aesthetics – clean and friendly design Overall comments positive about aesthetics "There‟s a lot of information, but it‟s laid out in a non-overwhelming way.
It‟s simple and easy to understand." We received mixed feedback on font size; some said it was too small, others
said the size is fine
Introduction Methodology Findings Future Steps
38
Areas for Improvement
39
Donate Family History Research and DocumentsExplanation:
-Participants did not associate donating family history records with “Volunteer”.
-Several participants attempted to upload documents to their own profile or family tree and assumed that information would be shared with others.
-Rated “High” since FamilySearch relies on record donations to help compile their collection of family history data, and 5 of 5 users failed the task
Recommendation:
Create a separate “Donate” feature, or change the “Volunteer” link to “Contribute to FamilySearch”
Severity: High Scope: Local Failure Frequency: 5 of 5 participants
“How would you like me to contribute this?”
Introduction Methodology Findings Future Steps
40
Find New Features of the SiteExplanation:
-Participants had difficulty locating new features on the site and stated that they would expect to see them on the homepage or receive an e-mail notification describing what was new.
-Participants most commonly navigated to the Site Map, Search, and Blog to find what was new.
- Rated “High” since new features will not be used if they are not easily found on the site, and success rate is below the desired 70%
Recommendation:
-Highlight new features on the homepage
Severity: High Scope: Local Failure Frequency: 5 of 9 participants
“I’m not positive where I would go from the homepage… I’m not sure.“
Introduction Methodology Findings Future Steps
41
Get Assistance with Country-Specific Family History Research
Explanation:
-Most users entered Scotland into the country search bar.
-Users could have been confused with the question overall: “You and many of your relatives are from Scotland, and you would like to add to the family history information you‟ve already collected. You know that FamilySearch compiles genealogy resources for several states and countries. Where would you go to find more information on these resources?”
-Rated “High” because all participants failed the task
Recommendation:
Rename link in help to “Regional Genealogy Research Assistance” Severity: High Scope: Local Failure Frequency: 6 of 6 participants
"I don’t know. Maybe in the immigration and naturalization section"
Introduction Methodology Findings Future Steps
Desired PathMost Common Path
1
2
3
42
“Watch” (Receive Notifications About) a Family Member or AncestorExplanation:
-Participants commonly confused “Watch” with the Discussion tab.
-Even experienced users familiar with the term “Watch” had difficulty finding the feature.
-Participants also expected to see updates about that user on the screen rather than via e-mail.
-Rated “Medium” since most users succeeded, but even experienced users familiar with the term had difficulty locating the feature
Recommendation: Make “Watch” feature more visible with larger font; show updates to “Watched” people on the profile screen Severity: Medium Scope: Local Failure Frequency: 3 of 8 participants
“One way to do it would be to start a discussion. I would click on discussion and do it that way. I think there’s also a way to have an email sent to me when there’s a change to Mary Howell.”
Introduction Methodology Findings Future Steps
43
Find Information About IndexingExplanation:
-The term “Indexing” was unfamiliar to non-LDS members.
-There was no explanation of the term on the homepage to guide participants to that section.
-LDS members were very familiar with the term and easily found the indexing page.
-Rated “Medium” since only 3 users failed, but there seemed to be a large difference between LDS and non-LDS members in recognizing the term “Indexing”, which impacted task performance
Recommendation: Provide a description of Indexing on the homepage. Severity: Medium Scope: Local Failure Frequency: 3 of 8 participants
"I would look at Indexing. Just because I know that word from church" .
Introduction Methodology Findings Future Steps
1
2
44
Submit a Country-Specific Family History ArticleExplanation:
-Both expert and beginner participants struggled to find the correct place to submit a wiki article.
-Participants demonstrated no consistent patterns of errors, but almost all participants failed to locate the wiki, search by country, and describe the submission process.
-Rated “Medium” since most users did not succeed, but the task is uncommon and local
Recommendation: Create a separate “Donate” feature, or change the “Volunteer” link to “Contribute to FamilySearch” Severity: Medium Scope: Local Failure Frequency: 6 of 8 participants
"Maybe under naturalization - that does not seem right"
Introduction Methodology Findings Future Steps
1
2
Expert Review vs. Study Findings45
Positive Findings Confirmed:
• Design: Participants were pleased with overall clean design aesthetic.
• Getting Started: The getting started guides were easy to find and a good introduction for beginner users.
Areas of Improvement Confirmed:
• Donate: Use of Volunteer section to encompass Donating records and financial contributions.
• Confusing Copy: Non-LDS members were confused by the term “Indexing”.
• Watch: Ancestor profile “Watch” notifications were hard to find. However, we did not assess in the expert review that this function may be confused with profile “Discussion” section.
Where the Findings Differed:
• Feedback: The participants had no issue locating the Feedback section.
• Research Articles: The expert reviews did not identify the issue of discovering where to contribute country-specific research articles.
• Confusing Copy: LDS members were familiar with the copy on the site, and were not confused by terms like, “Indexing”
Introduction Methodology Findings Future Steps
Limitations of the Study46
Technical Problems: We experienced technical difficulties with WebEx for three participants. As a result, we had to utilize three members of our convenience pool.
Number of Scenarios & Tasks: More than half of the participants were not able to get through all 6 scenarios and subsequent tasks. One potential factor that contributed to this was the participants‟ willingness to share stories about their own experiences with using FamilySearch.
Creation of Scenarios & Tasks: Task creation was originally written around testing the current site compared to new design. However, the tasks were still applicable to the study we conducted.
Prototype: Difficult to determine whether participants would experience all of the same problems on the live site vs. the prototype we tested on.
Removed Task: Multiple participants were confused by Scenario 4, Task 4. Users were asked to find on the website where they could volunteer to participate in evaluations. 2 out of 7 people were successful on this task. Most users went to the feedback link on the site and thought they had successfully completed the task. The team decided to remove these findings because the wording of the question was confusing to participants.
Introduction Methodology Findings Future Steps
Further Research to be Considered47
Action-Oriented Design: The initial plan was to test a prototype of the current site compared to a prototype of an action-oriented prototype. The action-oriented prototype is supposed to provide direct links to actions that users may perform frequently on the website, such as “Preserve” and “Find.” Going forward, FamilySearch can utilize the results from our study as a baseline to test the new prototype idea.
Task Structure Feedback: Experienced users had a lot of feedback about their experience with family history research. Two participants questioned the validity of the tasks they were asked to perform. One directly said, “I would never do this task” when asked to find the “About” section. The other user questioned why they would donate money when they already contribute financially to the LDS church. Consider soliciting experienced users‟ feedback to aid with task selection for future research studies.
Introduction Methodology Findings Future Steps
Questions48
Appendix49
• Participant Information
• Full List of Scenarios and Tasks
50
Recruited Participant Information50
AGE GENDER LOCATIONCHURCH
AFFILIATIONEXPERIENCE W/
FamilySearch
P1 40s Female Pennsylvania Catholic Beginner
P2 50s Female New YorkMormon/Latter-Day
Saint Experienced
P3 40s Female IdahoMormon/Latter-Day
Saint Experienced
P4 50s Male North CarolinaMormon/Latter-Day
Saint Beginner
P5 60s Female OklahomaMormon/Latter-Day
Saint Experienced
P6 30s Female CaliforniaMormon/Latter-Day
Saint Experienced
P7 40s Female IllinoisMormon/Latter-Day
Saint Beginner
P8 60s Male CaliforniaMormon/Latter-Day
Saint Beginner
P9 60s Female KansasMormon/Latter-Day
Saint Beginner
P10 60s Female Boston Catholic Beginner
P11 50s Male California Protestant Beginner
P12 60s Female California Protestant Beginner
Full List of Scenarios & Tasks51
SCENARIO 1:Your name is Terry Smith and you have had a FamilySearch account for the last three months. Assume you have started building your family tree on FamilySearch. You have already added four generations of your family history, but you are still missing information for some of your family members. Of those individuals whose life details are still missing, you are in the process of looking for more information.
Task 1: Some of the family history information you‟ve captured so far is based on your own knowledge and information you‟ve gotten from other living relatives. To ensure your family tree is accurate, you‟d like to attach official records to family members whenever possible. Show me where you would go to look for a death certificate for one of your relatives, Mary Howell, and attach the certificate to her profile.
Scenario 2:While you were attaching Mary Howell‟s death certificate to her profile, you noticed that the birth year you originally entered for her was incorrect.
Task 1: Change Mary Howell‟s birth year1864 to 1854. Task 2: You have other family members who might have additional information about Mary
Howell, and you would like to be notified when they add details to her profile. How would you do this?
Full List of Scenarios & Tasks52
Scenario 3: After completing four generations of your family tree, you‟ve run into some roadblocks and need some assistance. Task 1: You‟ve been unable to locate your great aunt‟s birth date, so you‟d like to find a local
family history center to get some help finding this information. Show me where you would go on this website to find the location of the nearest family history center.
Task 2: You and your friend were talking about the family tree you‟ve been building on FamilySearch, and your friend mentioned that he is interested in starting his own family tree. You‟d like to send him some resources on how to begin compiling his family history information. Show me where you would direct your friend on this website to learn how to begin compiling his family history.
Task 3: You and many of your relatives are from Scotland, and you would like to add to the family history information you‟ve already collected. You know that FamilySearch compiles genealogy resources for several states and countries. Where would you go to find more information on these resources?
Task 4: Last week you tried to upload a photo of your great grandmother, Mary Howell, to add to her profile, but you received an error saying the image was restricted. You submitted a request into FamilySearch to fix the issue so you can upload the photo. Where would you go to find out the status of the request?
Full List of Scenarios & Tasks53
Scenario 4: A friend of yours told you that there are several ways you could help make more resources available to people around the world through FamilySearch. Your friend explained that records such as birth certificates are not easily searchable since they are usually scanned in online. In order to make these records searchable through sites like FamilySearch, people can manually enter the information from the scanned documents to allow others to find those records for their family members. This is one area where sites like FamilySearch really need extra help. Task 1: Where would you go to find more information about manually converting scanned
documents into searchable records? Task 2: Now you actually want to get started manually converting scanned documents into
searchable records. What do you do? Task 3: A friend of yours from Germany recently wrote a piece on German history and submitted
it to FamilySearch. You grew up in Scotland and you‟d like to share your knowledge of the local history. Where would you go to submit your knowledge of Scotland?
Task 4: Let‟s say you had not been contacted by us to take part in this test today, but you still wanted to give FamilySearch your feedback and insight. What would you do?
Task 5: You have some family history research and documents for your own family tree information" in the form of GEDCOM files that you think would be really useful and valuable for FamilySearch. Show me where you would go to share these records.
Task 6: Now you feel like contributing money to FamilySearch, because you want to support their work. How would you contribute to FamilySearch?
Full List of Scenarios & Tasks54
Scenario 5: You recently moved into a new house, and would like to update your Mailing Address from your old address to your new address, 500 Main St. Draper, UT 84020.
Task 1: Change your mailing address in your FamilySearch contact information to 500 Main St.
Scenario 6: Your friend is very interested in researching their own family history, so they came over to have you show them what you‟ve done so far. You‟re starting by showing your friend around the FamilySearch website and you want to highlight what FamilySearch offers.
Task 1: Show me where you would go to view a list of every page on the website. (if successful: Is this what you expected to find?)
Task 2:You would like to teach your friend more information about the background of the FamilySearch organization. Walk me through that process.
Task 3: After browsing around the website with your friend, you have some comments to suggest to Family Search to improve the website‟s overall experience. Show me where you would go to provide an observation you‟ve made regarding FamilySearch.
Task 4: Show your friend where you would go to find the newest features available on the FamilySearch website.