In an initial study, Canary analyzed speech recordings with AI and identified early stage Alzheimer’s with 96 percent accuracy.
Dave Arnold retired in his 60s and began spending time volunteering in local schools. But then he started misplacing items, forgetting appointments and losing his sense of direction. Eventually he was diagnosed with early stage Alzheimer’s.
“Hearing the diagnosis made me very emotional and tearful,” he said. “I immediately thought of all my mom had experienced.” His mother suffered with the condition for years before passing away. Over the last year, Arnold has worked for the Alzheimer’s Association as one of its early stage advisors, sharing his insights to help others in the initial stages of the disease.
Arnold was diagnosed sooner than many others. It's important to find out early, when interventions can make the most difference. One promising avenue is looking at how people talk. Research has shown that Alzheimer’s affects a part of the brain that controls speech, resulting in small changes before people show other signs of the disease.
Now, Canary Speech, a company based in Utah, is using AI to examine elements like the pitch of a person’s voice and their pauses. In an initial study, Canary analyzed speech recordings with AI and identified early stage Alzheimer’s with 96 percent accuracy.
Developing the AI model
Canary Speech’s CEO, Henry O’Connell, met cofounder Jeff Adams about 40 years before they started the company. Back when they first crossed paths, they were both living in Bethesda, Maryland; O’Connell was a research fellow at the National Institutes of Health studying rare neurological diseases, while Adams was working to decode spy messages. Later on, Adams would specialize in building mathematical models to analyze speech and sound as a team leader in developing Amazon's Alexa.
It wasn't until 2015 that they decided to make use of the fit between their backgrounds. ““We established Canary Speech in 2017 to build a product that could be used in multiple languages in clinical environments,” O'Connell says.
The need is growing. About 55 million people worldwide currently live with Alzheimer’s, a number that is expected to double by 2050. Some scientists think the disease results from a buildup of plaque in the brain. It causes mild memory loss at first and, over time, this issue get worse while other symptoms, such as disorientation and hallucinations, can develop. Treatment to manage the disease is more effective in the earlier stages, but detection is difficult since mild symptoms are often attributed to the normal aging process.
O’Connell and Adams specialize in the complex ways that Alzheimer’s effects how people speak. Using AI, their mathematical model analyzes 15 million data points every minute, focusing on certain features of speech such as pitch, pauses and elongation of words. It also pays attention to how the vibrations of vocal cords change in different stages of the disease.
To create their model, the team used a type of machine learning called deep neural nets, which looks at multiple layers of data - in this case, the multiple features of a person’s speech patterns.
“Deep neural nets allow us to look at much, much larger data sets built out of millions of elements,” O’Connell explained. “Through machine learning and AI, we’ve identified features that are very sensitive to an Alzheimer’s patient versus [people without the disease] and also very sensitive to mild cognitive impairment, early stage and moderate Alzheimer's.” Based on their learnings, Canary is able to classify the disease stage very quickly, O’Connell said.
“When we’re listening to sublanguage elements, we’re really analyzing the direct result of changes in the brain in the physical body,” O’Connell said. “The brain controls your vocal cords: how fast they vibrate, the expansion of them, the contraction.” These factors, along with where people put their tongues when talking, function subconsciously and result in subtle changes in the sounds of speech.
Further testing is needed
In an initial trial, Canary analyzed speech recordings from phone calls to a large U.S. health insurer. They looked at the audio recordings of 651 policyholders who had early stage Alzheimer’s and 1018 who did not have the condition, aiming for a representative sample of age, gender and race. They used this data to create their first diagnostic model and found that it was 96 percent accurate in identifying Alzheimer’s.
Christian Herff, an assistant professor of neuroscience at Maastricht University in the Netherlands, praised this approach while adding that further testing is needed to assess its effectiveness.
“I think the general idea of identifying increased risk for cognitive impairment based on speech characteristics is very feasible, particularly when change in a user’s voice is monitored, for example, by recording speech every year,” Herff said. He noted that this can only be a first indication, not a full diagnosis. The accuracy still needs to be validated in studies that follows individuals over a period of time, he said.
Toby Walsh, a professor of artificial intelligence at the University of New South Wales, also thinks Canary’s tool has potential but highlights that Canary could diagnose some people who don’t really have the disease. “This is an interesting and promising application of AI,” he said, “but these tools need to be used carefully. Imagine the anxiety of being misdiagnosed with Alzheimer’s.”
As with many other AI tools, privacy and bias are additional issues to monitor closely, Walsh said.
Other languages
A related issue is that not everyone is fluent in English. Mahnaz Arvaneh, a senior lecturer in automatic control and systems engineering at the University of Sheffield, said this could be a blind spot.
“The system may not be very accurate for those who have English as their second language as their speaking patterns would be different, and any issue might be because of language deficiency rather than cognitive issues,” Arvaneh said.
The team is expanding to multiple languages starting with Japanese and Spanish. The elements of the model that make up the algorithm are very similar, but they need to be validated and retrained in a different language, which will require access to more data.
Recently, Canary analyzed the phone calls of 233 Japanese patients who had mild cognitive impairment and 704 healthy people. Using an English model they were able to identify the Japanese patients who had mild cognitive impairment with 78 percent accuracy. They also developed a model in Japanese that was 45 percent accurate, and they’re continuing to train it with more data.
The future
Canary is using their model to look at other diseases like Huntington’s and Parkinson’s. They’re also collaborating with pharmaceuticals to validate potential therapies for Alzheimer’s. By looking at speech patterns over time, Canary can get an indication of how well these drugs are working.
Dave Arnold and his wife dance at his nephew’s wedding in Rochester, New York, ten years ago, before his Alzheimer's diagnosis.
Dave Arnold
Ultimately, they want to integrate their tool into everyday life. “We want it to be used in a smartphone, or a teleconference call so that individuals could be examined in their home,” O’Connell said. “We could follow them over time and work with clinical teams and hospitals to improve the evaluation of patients and contribute towards an accurate diagnosis.”
Arnold, the patient with early stage Alzheimer’s, sees great promise. “The process of getting a diagnosis is already filled with so much anxiety,” he said. “Anything that can be done to make it easier and less stressful would be a good thing, as long as it’s proven accurate.”
Sonja Bauman, 39, and Mariela Florez, 83, taking one of their regular walks in Plantation, Fla. The pair met through an online platform called Papa that provides "family on demand." (Photo by Sonja Bauman)
In 2020, Papa connected Bauman, the 39-year-old Floridian, with another woman in her mid-70s who lives alone and has very limited mobility. Bauman began driving her to doctor’s appointments and helping her with chores around the house. “When I’m not there, she doesn’t leave her apartment,” said Bauman. The two have gone to the gym together, and they walk slowly through the neighborhood, chatting so it feels less like exercise.
Parker was driven to start Papa by the problem of social isolation among seniors, exacerbated by the pandemic, but he believes users of all ages can benefit. “Many of our Pals feel more comfortable opening up with older members than their same-aged friends,” he said.
Other platforms aim for similar, in-person connections. Generation Tech unites teens with seniors for technology training. And Mon Ami, which provides case management software for aging and disability service providers, has an app that connects isolated older people with college-age volunteers.
Making new connections through video
Several new sites match you with strangers for real-time video chatting on various topics, such as finding common ground on political issues. Other video platforms focus on intergenerational connections.
S. Jay Olshansky, a gerontology professor at the University of Illinois-Chicago, recalls the first time he saw Hyunseung Lee, an 11-year-old from Seoul, through his computer screen. The kid was shy, but Olshansky, 67, encouraged him to ask questions. “Turns out, he was thirsting for this kind of interaction.”
They’d connected through Eldera, a platform that pairs mentors age 60 and up with mentees, using an algorithm, for video conversations. “The time and wisdom of older adults is the most important natural resource we can give future generations,” said Dana Griffin, Eldera's CEO. “Connecting through a screen is the opposite of social media.”
In weekly meetings, Olshansky noticed Lee’s unique interest in math. “There’s something special in you,” Olshansky told him. “How do we bring it to the surface?” He suggested Lee write a book on his favorite subject, and the preteen ran with it, cranking out 70 pages in two weeks. Lee has published his love letter to theorems on Amazon.
Hyunseung Lee, age 11, of Korea, and U.S. college professor Jay Olshanksy, 67, discuss math, strategy and Hyunsung's budding career as a book author during their video chats through a platform called Eldera. (Photo by Dana Palmer/Eldera)
Lee’s parents told Olshansky that their son has become more assertive — a recurring theme, Griffin said. “Confidence is the number one thing parents tell us about.” Since Eldera’s inception last year, the number of mentors has grown exponentially. Even so, Griffin said the waitlist for mentors typically numbers 200 kids.
Another site, Big and Mini, hosts video interactions between seniors and young adults; about 10,000 active users have joined since 2019, said co-founder Aditi Merchant.
Users often bring the benefits of their video interactions to their real-world relationships. Olshansky views Lee as an older version of his grandkids. “Eldera teaches me how to interact with them.” Lee, high on confidence, began instructing his classmates in math. Griffin noted that a group of Eldera mentors in Memphis, who met initially on Eldera, now take walks together in-person to trade ideas for helping each other’s Eldera kids solve problems in their schools and communities.
“We’ve evolved into a community for older adults who want to give back to the world,” said Griffin. Other new tools for connection take the form of virtual reality apps.
Connecting in virtual reality
During pandemic isolation, record numbers of people bought devices for virtual and augmented reality. Such gadgets can convince you that you’re hanging out with friends, even if they’re in another hemisphere. Lifelike simulations from miles away could be especially useful for meaningful interactions between people of different generations, since they’re often geographically segregated.
VR’s benefits require further study, but users report less social isolation and depression, according to MIT research. The immersive, 3-D experience is more compelling than FaceTime or Zoom. “It’s like the difference between a phone call and video call,” said Rick Robinson, Vice President of AARP’s Innovation Labs.
“When VR is designed right, the medium disappears,” said Jeremy Bailenson of Stanford.
Dana Pierce, a 56-year-old government employee in Indiana, got Meta's VR headset in May, 2021, thinking she’d enjoy it more than a new laptop. After many virtual group tours of exotic destinations, she has no regrets. Her adventures occur on Alcove, an app made by Robinson’s Innovation Labs. He co-created it with VR-company Rendever and sought input from people over age 50 to tailor it to their interests. “I’m an introvert,” said Pierce. “I’ve been more socially active since getting my headset than I am in real life.”
Tagging along with her to places like Paris are avatars representing real people around the world. She’s gotten to know VR users in their 70s, 80s and 90s, as well as younger people and some her own age. One is a new friend she plays chess with in relaxing nature settings. Another is her oldest son. He lives 90 minutes away but, earlier this year, Pierce welcomed him and his girlfriend to her virtual house on Alcove. They chatted in the living room decorated with family photos uploaded by Pierce. Then they took out a boat to go VR fishing — because why not — until 2 a.m.
“When VR is designed right, the medium disappears,” said Jeremy Bailenson, a communications professor who directs Stanford’s Virtual Human Interaction Lab. He’s teaching a class of 175 students entirely in VR. After months of covid isolation, the first time the class met, “there was a big catharsis. It really feels like you’re in a big crowd.” Like-minded people meet in VR for events such as comedy shows and creative writing meetups, while the Swedish pop group ABBA has performed this year as digital versions of themselves (“ABBA-tars”) during a virtual concert tour.
Karen Fingerman, a psychologist and director of the Texas Aging and Longevity Center at the University of Texas-Austin, supports the idea of VR for social connection, though she added that some people need it more than others. Hospitals and assisted-living facilities are using products such as Penumbra’s REAL I-Series and MyndVR to bring VR excursions to isolated patients and seniors. “If you’re in a bed or facility, this gives you something to talk about,” said Gita Barry, Penumbra’s executive vice president.
Pierce uses it on most days. She may see another adult son, who lives with her, less often as a result. But VR helps her manage real-world stressors, more than escaping them. After a long workday, she visits her back porch on Alcove, which overlooks a pond. “It’s my little retreat,” she said. “VR improves my mood. It’s added a lot to my life.”
Some seniors are using more than one technology. Olshansky and Lee discuss strategy while playing Internet chess. And Olshansky recently began using VR. He sees his sister, who lives far away, in a virtual beach house. “It’d be a great way to interact with Hyunseung,” he said. “I should get him a headset.”
A version of this article first appeared in The Washington Post on December 3, 2021.
