Voice assistant comprehension study in Nature Digital Medicine reveals platforms’ pain points
and calls for the need to improve accuracy of voice technology for health
TORONTO, ON/NEW YORK, NY – June 20, 2019 – New Klick Health research published today in Nature Digital Medicine found Google Assistant’s comprehension of the most commonly dispensed medication names in the U.S. was, on average, almost twice as accurate as Alexa and Siri’s comprehension. The study also concludes that while Google Assistant had the best performance rates of the artificial intelligence software platforms that respond to natural language and simulate human conversation, there is room for improvement.
According to the researchers behind “Do you understand the words that are comin outta my mouth? Voice assistant comprehension of medication names,” the study is a fundamental first step toward better understanding the technical capabilities of speech recognition for medication names – especially as Alexa, Google Assistant, and Siri are increasingly being adopted in home and healthcare environments as convenient, trusted tools for gathering information. Recent data from Pew Research Center estimates 46 percent of Americans use voice assistants for information searches and other commands.
The Klick study results also underscore why previously published research examining voice assistants’ ability to answer and/or provide advice to a variety of health-related questions have largely proven premature and why more attention is needed.
Lack of comprehension could lead to risk
“Before voice assistants like Alexa, Google Assistant, and Siri can understand emerging health-related scenarios and incorporate context into their responses, they first must be able to accurately recognize what people are saying, including commonly used medication names,” said Adam Palanica, PhD, Behavioral Scientist at Klick and co-author of the study.
“Digital voice assistants are becoming popular tools for gathering health information, so a lack of comprehension puts them at risk of providing poor, inconsistent, and potentially dangerous advice,” added Yan Fossat, Vice President of Klick Labs and fellow study co-author. “We were encouraged by some of the comprehension rates in our research, but more work needs to be done to ensure people’s health and safety.”
The study marks the first time that researchers have examined the syntactic comprehension performance of the brand and generic names of the top 50 most dispensed medications in the U.S.
From Acetaminophen to Zantac
Google Assistant had the best overall comprehension rates with 91.8 percent accuracy on brand-name medication names and 84.3 percent accuracy on generic names. In the same testing, Siri had 58.5 percent accuracy on brand-names and 51.2 percent on generics, while Alexa had 54.6 percent accuracy on brand names and 45.5 percent on generics.
For research participants with audible, foreign accents, both Siri and Alexa also came in with lower overall comprehension rates however, no significant differences were found between accent types when participants spoke to Google Assistant.
Interestingly, while all three platforms demonstrated strong understanding of over-the-counter pain relievers Aspirin (Google 100%, Alexa and Siri each 93%) and Tylenol (Google and Alexa each 98%; Siri 93%), Alexa showed significantly weaker comprehension rates for both Advil (Google 100%, Alexa 2%; Siri 59%) and the generic Ibuprofen (Google 100%, Alexa 4%; Siri 89%).
The Klick study was conducted by its Labs department, which is comprised of a diverse team of data scientists, biomedical engineers, and social scientists. The company conducts research and develops AI solutions as part of its work to help improve the well-being of patient populations using its proven commercial, scientific, medical, and technological expertise.
Earlier this year, Klick’s research on physicians’ perceptions of chatbots in healthcare was published in The Journal of Medical Internet Research. Among the key findings: While doctors think chatbots can be beneficial, they have concerns about these artificial conversational entities’ abilities to comprehend the emotional state of humans and provide expert medical knowledge and intelligence.
Klick Health is the world’s largest independent, full-service marketing and commercialization agency for life sciences. By relentlessly conspiring with the future, Klick helps its clients outsmart and outpace disease so people can live richer, longer lives. Klick has been named Agency of the Year six times over the last seven years by the industry’s leading publications and was recognized by the 2018 Cannes Lions Health Awards for its client campaigns. Follow Klick Health on Twitter at @KlickHealth.
About Klick Inc.
Klick operates with a fiercely independent spirit at the intersection of technology, creativity, and strategy. The Klick family is a diverse group of people who are driven and connected by a shared culture and commitment to the relentless pursuit of awesome at Klick Health, Katalyst, and Sensei Labs. Established in 1997, Klick has teams in Atlanta, Connecticut, Chicago, Los Angeles, New Jersey, New York, Philadelphia, San Diego, San Francisco, and Toronto. Klick has been consistently recognized as a leading Great Place to Work, Best Workplace for Women, Best Employer, Employee-Recommended Workplace, Fastest Growing Technology Company, and Best Managed Company. For more information on joining Klick, go to http://careers.klick.com.