In an initial study, Canary analyzed speech recordings with AI and identified early stage Alzheimer’s with 96 percent accuracy.
Dave Arnold retired in his 60s and began spending time volunteering in local schools. But then he started misplacing items, forgetting appointments and losing his sense of direction. Eventually he was diagnosed with early stage Alzheimer’s.
“Hearing the diagnosis made me very emotional and tearful,” he said. “I immediately thought of all my mom had experienced.” His mother suffered with the condition for years before passing away. Over the last year, Arnold has worked for the Alzheimer’s Association as one of its early stage advisors, sharing his insights to help others in the initial stages of the disease.
Arnold was diagnosed sooner than many others. It's important to find out early, when interventions can make the most difference. One promising avenue is looking at how people talk. Research has shown that Alzheimer’s affects a part of the brain that controls speech, resulting in small changes before people show other signs of the disease.
Now, Canary Speech, a company based in Utah, is using AI to examine elements like the pitch of a person’s voice and their pauses. In an initial study, Canary analyzed speech recordings with AI and identified early stage Alzheimer’s with 96 percent accuracy.
Developing the AI model
Canary Speech’s CEO, Henry O’Connell, met cofounder Jeff Adams about 40 years before they started the company. Back when they first crossed paths, they were both living in Bethesda, Maryland; O’Connell was a research fellow at the National Institutes of Health studying rare neurological diseases, while Adams was working to decode spy messages. Later on, Adams would specialize in building mathematical models to analyze speech and sound as a team leader in developing Amazon's Alexa.
It wasn't until 2015 that they decided to make use of the fit between their backgrounds. ““We established Canary Speech in 2017 to build a product that could be used in multiple languages in clinical environments,” O'Connell says.
The need is growing. About 55 million people worldwide currently live with Alzheimer’s, a number that is expected to double by 2050. Some scientists think the disease results from a buildup of plaque in the brain. It causes mild memory loss at first and, over time, this issue get worse while other symptoms, such as disorientation and hallucinations, can develop. Treatment to manage the disease is more effective in the earlier stages, but detection is difficult since mild symptoms are often attributed to the normal aging process.
O’Connell and Adams specialize in the complex ways that Alzheimer’s effects how people speak. Using AI, their mathematical model analyzes 15 million data points every minute, focusing on certain features of speech such as pitch, pauses and elongation of words. It also pays attention to how the vibrations of vocal cords change in different stages of the disease.
To create their model, the team used a type of machine learning called deep neural nets, which looks at multiple layers of data - in this case, the multiple features of a person’s speech patterns.
“Deep neural nets allow us to look at much, much larger data sets built out of millions of elements,” O’Connell explained. “Through machine learning and AI, we’ve identified features that are very sensitive to an Alzheimer’s patient versus [people without the disease] and also very sensitive to mild cognitive impairment, early stage and moderate Alzheimer's.” Based on their learnings, Canary is able to classify the disease stage very quickly, O’Connell said.
“When we’re listening to sublanguage elements, we’re really analyzing the direct result of changes in the brain in the physical body,” O’Connell said. “The brain controls your vocal cords: how fast they vibrate, the expansion of them, the contraction.” These factors, along with where people put their tongues when talking, function subconsciously and result in subtle changes in the sounds of speech.
Further testing is needed
In an initial trial, Canary analyzed speech recordings from phone calls to a large U.S. health insurer. They looked at the audio recordings of 651 policyholders who had early stage Alzheimer’s and 1018 who did not have the condition, aiming for a representative sample of age, gender and race. They used this data to create their first diagnostic model and found that it was 96 percent accurate in identifying Alzheimer’s.
Christian Herff, an assistant professor of neuroscience at Maastricht University in the Netherlands, praised this approach while adding that further testing is needed to assess its effectiveness.
“I think the general idea of identifying increased risk for cognitive impairment based on speech characteristics is very feasible, particularly when change in a user’s voice is monitored, for example, by recording speech every year,” Herff said. He noted that this can only be a first indication, not a full diagnosis. The accuracy still needs to be validated in studies that follows individuals over a period of time, he said.
Toby Walsh, a professor of artificial intelligence at the University of New South Wales, also thinks Canary’s tool has potential but highlights that Canary could diagnose some people who don’t really have the disease. “This is an interesting and promising application of AI,” he said, “but these tools need to be used carefully. Imagine the anxiety of being misdiagnosed with Alzheimer’s.”
As with many other AI tools, privacy and bias are additional issues to monitor closely, Walsh said.
Other languages
A related issue is that not everyone is fluent in English. Mahnaz Arvaneh, a senior lecturer in automatic control and systems engineering at the University of Sheffield, said this could be a blind spot.
“The system may not be very accurate for those who have English as their second language as their speaking patterns would be different, and any issue might be because of language deficiency rather than cognitive issues,” Arvaneh said.
The team is expanding to multiple languages starting with Japanese and Spanish. The elements of the model that make up the algorithm are very similar, but they need to be validated and retrained in a different language, which will require access to more data.
Recently, Canary analyzed the phone calls of 233 Japanese patients who had mild cognitive impairment and 704 healthy people. Using an English model they were able to identify the Japanese patients who had mild cognitive impairment with 78 percent accuracy. They also developed a model in Japanese that was 45 percent accurate, and they’re continuing to train it with more data.
The future
Canary is using their model to look at other diseases like Huntington’s and Parkinson’s. They’re also collaborating with pharmaceuticals to validate potential therapies for Alzheimer’s. By looking at speech patterns over time, Canary can get an indication of how well these drugs are working.
Dave Arnold and his wife dance at his nephew’s wedding in Rochester, New York, ten years ago, before his Alzheimer's diagnosis.
Dave Arnold
Ultimately, they want to integrate their tool into everyday life. “We want it to be used in a smartphone, or a teleconference call so that individuals could be examined in their home,” O’Connell said. “We could follow them over time and work with clinical teams and hospitals to improve the evaluation of patients and contribute towards an accurate diagnosis.”
Arnold, the patient with early stage Alzheimer’s, sees great promise. “The process of getting a diagnosis is already filled with so much anxiety,” he said. “Anything that can be done to make it easier and less stressful would be a good thing, as long as it’s proven accurate.”
Theo, an 18-month-old in rural Nebraska, walks with his father in their backyard. For many toddlers, the barriers to accessing COVID-19 vaccines are many, such as few locations giving vaccines to very young children.
Children are at risk
Data presented at the most recent FDA advisory panel on COVID-19 vaccines showed that in the last year infants under six months had the third highest rate of hospitalization. “From the beginning, the message has been that kids don’t get COVID, and then the message was, well kids get COVID, but it’s not serious,” says Elias Kass, a pediatrician in Seattle. “Then they waited so long on the initial vaccines that by the time kids could get vaccinated, the majority of them had been infected.”
A closer look at the data from the CDC also reveals that from January 2022 to January 2023 children aged 6 to 23 months were more likely to be hospitalized than all other vaccine eligible pediatric age groups.
“We sort of forced an entire generation of kids to be infected with a novel virus and just don't give a shit, like nobody cares about kids,” Kass says. In some cases, COVID has wreaked havoc with the immune systems of very young children at his practice, making them vulnerable to other illnesses, he said. “And now we have kids that have had COVID two or three times, and we don’t know what is going to happen to them.”
Jumping through hurdles
Children under five were the last group to have an emergency use authorization (EUA) granted for the COVID-19 vaccine, a year and a half after adult vaccine approval. In June 2022, 30,000 sites were initially available for children across the country. Six months later, when boosters became available, there were only 5,000.
Currently, only 3.8% of children under two have completed a primary series, according to the CDC. An even more abysmal 0.2% under two have gotten a booster.
Ariadne Labs, a health center affiliated with Harvard, is trying to understand why these gaps exist. In conjunction with Boston Children’s Hospital, they have created a vaccine equity planner that maps the locations of vaccine deserts based on factors such as social vulnerability indexes and transportation access.
“People are having to travel farther because the sites are just few and far between,” says Benjy Renton, a research assistant at Ariadne.
Michelle Baltes-Breitwisch, a pharmacist, and her two-year-old daughter, Charlee, live in Iowa. When the boosters first came out she expected her toddler could get it close to home, but her husband had to drive Charlee four hours roundtrip.
This experience hasn’t been uncommon, especially in rural parts of the U.S. If parents wanted vaccines for their young children shortly after approval, they faced the prospect of loading babies and toddlers, famous for their calm demeanor, into cars for lengthy rides. The situation continues today. Mrs. Smith*, a grant writer and non-profit advisor who lives in Idaho, is still unable to get her child the bivalent booster because a two-hour one-way drive in winter weather isn’t possible.
It can be more difficult for low wage earners to take time off, which poses challenges especially in a number of rural counties across the country, where weekend hours for getting the shots may be limited.
Protect Their Future (PTF), a grassroots organization focusing on advocacy for the health care of children, hears from parents several times a week who are having trouble finding vaccines. The vaccine rollout “has been a total mess,” says Tamara Lea Spira, co-founder of PTF “It’s been very hard for people to access vaccines for children, particularly those under three.”
Seventeen states have passed laws that give pharmacists authority to vaccinate as young as six months. Under federal law, the minimum age in other states is three. Even in the states that allow vaccination of toddlers, each pharmacy chain varies. Some require prescriptions.
It takes time to make phone calls to confirm availability and book appointments online. “So it means that the parents who are getting their children vaccinated are those who are even more motivated and with the time and the resources to understand whether and how their kids can get vaccinated,” says Tiffany Green, an associate professor in population health sciences at the University of Wisconsin at Madison.
Green adds, “And then we have the contraction of vaccine availability in terms of sites…who is most likely to be affected? It's the usual suspects, children of color, disabled children, low-income children.”
It can be more difficult for low wage earners to take time off, which poses challenges especially in a number of rural counties across the country, where weekend hours for getting the shots may be limited. In Bibb County, Ala., vaccinations take place only on Wednesdays from 1:45 to 3:00 pm.
“People who are focused on putting food on the table or stressed about having enough money to pay rent aren't going to prioritize getting vaccinated that day,” says Julia Raifman, assistant professor of health law, policy and management at Boston University. She created the COVID-19 U.S. State Policy Database, which tracks state health and economic policies related to the pandemic.
Most states in the U.S. lack paid sick leave policies, and the average paid sick days with private employers is about one week. Green says, “I think COVID should have been a wake-up call that this is necessary.”
Maskless waiting rooms
For her son, Holmes spent hours making phone calls but could uncover no clear answers. No one could estimate an arrival date for the booster. “It disappoints me greatly that the process for locating COVID-19 vaccinations for young children requires so much legwork in terms of time and resources,” she says.
In January, she found a pharmacy 30 minutes away that could vaccinate Theo. With her son being too young to mask, she waited in the car with him as long as possible to avoid a busy, maskless waiting room.
Kids under two, such as Theo, are advised not to wear masks, which make it too hard for them to breathe. With masking policies a rarity these days, waiting rooms for vaccines present another barrier to access. Even in healthcare settings, current CDC guidance only requires masking during high transmission or when treating COVID positive patients directly.
“This is a group that is really left behind,” says Raifman. “They cannot wear masks themselves. They really depend on others around them wearing masks. There's not even one train car they can go on if their parents need to take public transportation… and not risk COVID transmission.”
Yet another challenge is presented for those who don’t speak English or Spanish. According to Translators without Borders, 65 million people in America speak a language other than English. Most state departments of health have a COVID-19 web page that redirects to the federal vaccines.gov in English, with an option to translate to Spanish only.
The main avenue for accessing information on vaccines relies on an internet connection, but 22 percent of rural Americans lack broadband access. “People who lack digital access, or don’t speak English…or know how to navigate or work with computers are unable to use that service and then don’t have access to the vaccines because they just don’t know how to get to them,” Jirmanus, an affiliate of the FXB Center for Health and Human Rights at Harvard and a member of The People’s CDC explains. She sees this issue frequently when working with immigrant communities in Massachusetts. “You really have to meet people where they’re at, and that means physically where they’re at.”
Equitable solutions
Grassroots and advocacy organizations like PTF have been filling a lot of the holes left by spotty federal policy. “In many ways this collective care has been as important as our gains to access the vaccine itself,” says Spira, the PTF co-founder.
PTF facilitates peer-to-peer networks of parents that offer support to each other. At least one parent in the group has crowdsourced information on locations that are providing vaccines for the very young and created a spreadsheet displaying vaccine locations. “It is incredible to me still that this vacuum of information and support exists, and it took a totally grassroots and volunteer effort of parents and physicians to try and respond to this need.” says Spira.
Kass, who is also affiliated with PTF, has been vaccinating any child who comes to his independent practice, regardless of whether they’re one of his patients or have insurance. “I think putting everything on retail pharmacies is not appropriate. By the time the kids' vaccines were released, all of our mass vaccination sites had been taken down.” A big way to help parents and pediatricians would be to allow mixing and matching. Any child who has had the full Pfizer series has had to forgo a bivalent booster.
“I think getting those first two or three doses into kids should still be a priority, and I don’t want to lose sight of all that,” states Renton, the researcher at Ariadne Labs. Through the vaccine equity planner, he has been trying to see if there are places where mobile clinics can go to improve access. Renton continues to work with local and state planners to aid in vaccine planning. “I think any way we can make that process a lot easier…will go a long way into building vaccine confidence and getting people vaccinated,” Renton says.
Michelle Baltes-Breitwisch, a pharmacist, and her two-year-old daughter, Charlee, live in Iowa. Her husband had to drive four hours roundtrip to get the boosters for Charlee.
Michelle Baltes-Breitwisch
Other changes need to come from the CDC. Even though the CDC “has this historic reputation and a mission of valuing equity and promoting health,” Jirmanus says, “they’re really failing. The emphasis on personal responsibility is leaving a lot of people behind.” She believes another avenue for more equitable access is creating legislation for upgraded ventilation in indoor public spaces.
Given the gaps in state policies, federal leadership matters, Raifman says. With the FDA leaning toward a yearly COVID vaccine, an equity lens from the CDC will be even more critical. “We can have data driven approaches to using evidence based policies like mask policies, when and where they're most important,” she says. Raifman wants to see a sustainable system of vaccine delivery across the country complemented with a surge preparedness plan.
With the public health emergency ending and vaccines going to the private market sometime in 2023, it seems unlikely that vaccine access is going to improve. Now more than ever, ”We need to be able to extend to people the choice of not being infected with COVID,” Jirmanus says.
*Some names were changed for privacy reasons.