We watched people use the web to find (or not) answers about coronavirus and saw some surprises.
By: Marilyn Salzman, Lori Eronimous, and Bill Skeet
At the start of the COVID-19 pandemic, we observed 30 people as they used the web to find answers to key coronavirus questions before they participated in a broader online survey (with 429 participants) about COVID-19 trusted news and information. These task research sessions yield insights into strategies for and challenges of navigating the online COVID-19 coronavirus news ecosystem. For previous and future articles on this research, visit our COVID-19 Research blog.
A few surprises
High success rates. People successfully answered three key questions: the number of confirmed cases? social distancing guidelines? and approved medicines?
Notable challenges. They struggled most with the most personal and health-critical task: What’s the process for getting tested in your area?
CDC site. Few people showed us the CDC site on their tour of top COVID-19 news sources, yet it was the top destination for answering questions about social distancing and approved medicines, as well as the “most trusted” source in the survey.
Politics and trust. Political perspectives influenced the sites people trusted and strategies they used to answer questions. The figure below captures quotes from several politically diverse participants. (We’ll take a closer look at this topic in a future article. Stay tuned to our COVID-19 Research blogs.)
Information overload already! The pandemic was just beginning and people were overwhelmed and frustrated by the online COVID-19 news ecosystem. Here is how two participants expressed this sentiment:
“It is a bit overwhelming. Maybe too much information. Jumbled information when all I wanted was some simple statistics. Sometimes over complicated.”
“You need to filter what's dangerous and what is lies. You don't want misinformation. Compare credited and uncredited sources and decide what is the truth.”
How people find answers
Favorite COVID-19 sources? When participants took us on a tour of their top news sources, Google, CNN, and The New York Times were the most popular showcases. That said, many people didn’t have a single “go-to” website and were happy to show us around. Some expressed gratitude to sources like the New York Times, who were providing free access to COVID-19 news. Others talked about the challenges of separating fact from fiction. Surprisingly, very few took us to the CDC site, which later served as a critical source for key tasks (social distancing and approved medicines) and ultimately was the “most trusted” site by these participants, as well as the larger survey sample.
COVID-19 Cases? COVID-19 World, US, and State confirmed cases were easily found via Google, Worldometer, and Johns Hopkins. However, in a few cases, our participants prefer local sources such as The Mercury News, WBRC TV news, and Connecticut’s state website.
Social Distancing? The CDC was the definitive source when it came to social distancing guidelines, with a vast majority of our participants (82%) seeking guidance there.
“When I try to find out things like this, I'm making sure to click on accredited sources like the CDC.”
How to get tested? This task proved challenging with 15% failing to find testing information. Participants also spent the most time on this task and rated it as the hardest. The good news is that this task appeared to get easier over time, with our last round of participants (on April 16) fully succeeding when participants in the early rounds (April 7 and 8) had not.
“It was hard to find the steps to take for getting tested.”
“You can't get tested unless you're dying. Last time I looked there were no real guidelines on how to get tested. None of us really know.”
“Looks like I'd have to go to Sacramento an hour and a half away. I don't see anything local, which is sort of disturbing.”
Approved medicine? At the time of the test, there were no approved COVID-19 medicines, but there were rumors about anti-malaria drugs (chloroquine and hydroxychloroquine) as treatment options. While all but one person successfully completed this task, people lamented the challenges of telling truth from fiction.
“I’m already up on this and what I’m gonna find here on the web here [on Google results] is a bunch of confusion... The stuff you get here is of no interest to me. What I can find on the first page of Google is pointless as far as I am concerned at this point.”
“I'm very interested, heard a lot of rumors. Then, I go to WHO to dispel myths.”
“The Texas Health and Human Services I wouldn't trust.”
A closer look at the task-based questions & answers
A shown in the graph below, the types of sources people used vary quite significantly by task. For example, while news media was the top “go-to” source for information, search and data aggregators were key for addressing questions about the number of confirmed cases, health organizations were critical for questions about social distancing and approved medications, and local government was the primary “go-to” source for testing. The national government really only came into play for the question about approved medicines, perhaps because it was a topic of discussion during the President’s press briefings at the time.
Below is a closer look at the top sources showcased during the initial task, the specific sources used to answer key questions, and the sources participants noted they liked. As mentioned earlier, while the CDC wasn’t top of mind as a source, it was one of the most important sites, as well as highly trusted. Unfortunately, as we highlight later, the site was not that easy to navigate. Another notable insight is that media sites seem to be fueling the COVID-19 conversation (and are top of mind), but they aren’t where people go to answer the questions we asked them.
The above word clouds illustrate the top sources participants showcased versus the sources they used to answer questions versus the sources participants cited as trusted in the follow-up survey. Word clouds were generated on wordclouds.com.
Top news sources?
Task: Using your browser window, give us a tour of your top online news sources. Spend up to 3 minutes and think out loud as you browse your news sites.
When displaying their top news sources, more than half of the participants referred to specific national and local news sources, for example, CNN, Fox News, and KESQ TV (54%), followed by sources from search results and aggregators (34%), and Google in particular. For this task, no participant referred to a government news source.
“I don’t have a go-to website, I just click on the articles that stand out in top stories.”
“With those sites [CNN, BBC, NPR] I get information that is not biased. I stay away from Fox News or MSNBC.”
“This is the only thing I do [search Google for COVID-19 and scan results], then I Make sure they're from a legit source like Boulder County, NPR, NYT."
Confirmed cases worldwide?
Task: Determine the latest number of confirmed cases: 1) Worldwide, 2) US, and 3) Your state.
For this question, most of the participants (68%) relied on search or data aggregators to find the confirmed cases, often spontaneously choosing a source at the top of search results rather than navigating to a specific news source.
Half of the participants who used a data aggregator map or table to answer this question drilled down within that same source from the worldwide level, to the US, and then to their state to answer subsequent questions about the number of confirmed cases. They used Google’s coronavirus disease news aggregator and map (31%), the Johns Hopkins Coronavirus Resource Center map (17%), Worldometer’s coronavirus data (21%), and MSN (3%).
Coronavirus cases in the US?
Task: Determine the latest number of confirmed cases: 1) Worldwide, 2) US, and 3) Your state.
A little more than half (55%) of the participants used search and news aggregator sites such as Google to find out the current number of confirmed cases in the US. About a quarter (28%) of the participants went to health and medical websites. The rest went to local news sources (17%). No participant used a government source.
“They [the Wall Street Journal] don’t make it easy. There’s extra material I don’t want or need. Kind of irritating. I’m confused about what to click.”
Cases in your state
Task: Determine the latest number of confirmed cases: 1) Worldwide, 2) the US, and 3) Your state.
Unlike strategies for previous tasks, participants used some local news and local government sources to find confirmed cases for their state (18%), such as Connecticut’s state website, and The Mercury News, although they continued to primarily use search and aggregators like Google (47%), health organizations or medical experts like the CDC (21%) and national news or syndicated media (14%) such as CNN.
“That's a lot [confirmed cases in NY], that's why I came back to Mexico for a while. I live in New York and don't know when I'm going back. This is stressful, I miss my friends.”
“I’m living in a top 6 state in the country [confirmed cases in MA]!
Social distancing guidelines?
Task: Find an answer on the web: What are the latest guidelines for social distancing?
To answer this question, the majority of participants (82%) used the same health organization or medical expert source--the CDC--and in many cases, the same webpage Social Distancing, Quarantine, and Isolation. The remaining participants referred to local government (7%), national news (7%), or local news (4%) sources.
"I make sure I look at a reputable site. CDC is obviously reputable."
“It's conflicting information. Nothing tells me exact dates, what's new, and what's not. What was true in February is not true today.”
As a side note, it has been interesting to watch the sites evolve during the testing window (April 6 through 16) and continuing to today. The figure below shows the evolution.
How to get tested?
Task: Find an answer on the web: What is the process to get tested for COVID-19 in your area?
When this research was conducted in early April, testing was not as available as it is now. This question was the most challenging for participants to answer and participants rated it more difficult to complete than the other tasks. Several participants indicated that they did not complete the task (13%) when asked “Could you complete this task? (Yes/No).”
About half of the participants used local government sources (54%), such as Onondaga County Coronavirus Information, Michigan’s state website, and the Washington State Department of Health, followed by health organizations and medical experts (27%) and local media (19%). The local government sources included state, county, and health and human services websites.
“I feel like the department of health would be the best thing to go to. I'm glad I clicked on this .pdf.”
“I’m spending more time than I should on this. It’s taking too long to find anything concrete--a doctor's office, library, or clinic.”
“I had to get really specific on my search to find this.”
Approved medicine or treatment?
Task: Find an answer on the web: Are there any approved medicines to treat or prevent COVID-19?
When this research was conducted in early April 2020, no medicine was approved for treating COVID-19. In May, after the test period, Remdesivir was approved by the FDA for emergency use. To answer this question, over half of the participants used the CDC COVID-19 site or FDA COVID-19 site. Overall, the top references were health organizations (60%), followed by the US government (20%), search and aggregator sites (17%), and national or international news (3%).
“I'm very interested, heard a lot of rumors. Then I go to WHO to dispel myths.”
“FDA and CDC are good sources for this type of thing.”
“Thank you for making me do this.”
“I really wish they would call it out at top--there is no treatment, please seek emergency treatment if you are sick. Wish there was a central place to find this basic information. It’s been quite frustrating at times.”
Task performance when finding answers
The figure below summarizes three dimensions of task performance: time on task, success rates, and ease of use ratings. Overall, participants were fairly successful and their ease of use ratings were strong (about 6 out of 7 for ease of use). The task that participants struggled with the most was “Finding how to get tested in your area”. Failure rates were the highest (13% for this task versus 0-3% for other tasks), time on task was the longest (2.75 mins), and ease of use ratings were the poorest at 5.1.
About this study
This task-based study is part of a broader study in which we combined survey and task-based research to understand how and where people get news and information about COVID-19, as well as the sources they trust. We conducted this research at the outset of the COVID-19 pandemic in early April 2020 (April 3 through April 16). The survey was a US-based survey with 429 participants, and a subset of 30 participants also took part in the task-based research described in this article.
Task-based research methodology
Thirty people took part in automated testing via TryMyUI, an online usability testing tool, on April 8, 9, and 16, 2020. From each session, we obtained videos of participants using a browser window, audio of their “think aloud” comments, numeric ratings, and typed answers to post-test questions. Participants also took part in a larger COVID-19 news and trust survey, which is discussed in Trusted News and Information in a Pandemic: A Survey of COVID-19 News Practices and Perceptions.
Participant criteria
We used TryMyUI’s pool of participants and set the target audience criteria to match the larger survey criteria as closely as possible. The only restriction was US Only for location and no restrictions for criteria such as gender, age, income, education, employment, family status, etc.
Participant demographics
With those criteria in mind, the 30-person task research subset was reasonably balanced by geographic location and gender, but slightly skewed Democratic (43% vs. April’s Gallup poll estimate of 31%). In addition, no person over 60 participated in task research, although they comprise 11.7% of the larger survey sample.
Test script
We introduced the test with a description of our goals:
We are a team of researchers interested in how and where you get news about COVID-19. TryMyUI and survey results will be used to inform organizations researching media and its effects. Your participation is voluntary and anonymous.
Participants started at www.google.com in a browser window and were asked to show which news sources they use to find information about COVID-19.
Using your browser window, give us a tour of your top online news sources. Spend up to 3 minutes and think out loud as you browse your news sites.
Next, participants completed four tasks specific to the coronavirus and rated how easy or difficult it was to complete:
Task: Determine the latest number of confirmed cases: 1) Worldwide, 2) US, and 3) Your state.
Task: Find an answer on the web: What are the latest guidelines for social distancing?
Task: Find an answer on the web: What is the process to get tested for COVID-19 in your area?
Task: Find an answer on the web: Are there any approved medicines to treat or prevent COVID-19?
Finally, participants completed a larger online SurveyMonkey survey and four post-test questions asking for feedback about the current state of the COVID-19 coronavirus news ecosystem (likes, dislikes, suggestions).
Stay tuned
Our analysis and research continue. You can find these posts on our COVID-19 Research blog or connect with our authors on LinkedIn or Facebook: Marilyn Salzman, Lori Eronimous, and Bill Skeet. We'll be sure to announce new publications there.