World’s First “Augmented Reality” Contact Lens Aims to Revolutionize Much More Than Medicine
Imagine a world without screens. Instead of endlessly staring at your computer or craning your neck down to scroll through social media feeds and emails, information simply appears in front of your eyes when you need it and disappears when you don't.
"The vision is super clear...I was reading the poem with my eyes closed."
No more rude interruptions during dinner, no more bumping into people on the street while trying to follow GPS directions — just the information you want, when you need it, projected directly onto your visual field.
While this screenless future sounds like science fiction, it may soon be a reality thanks to the new Silicon Valley startup Mojo Vision, creator of the world's first smart contact lens. With a 14,000 pixel-per-inch display with eye-tracking, image stabilization, and a custom wireless radio, the Mojo smart lens bills itself the "smallest and densest dynamic display ever made." Unlike current augmented reality wearables such as Google Glass or ThirdEye, which project images onto a glass screen, the Mojo smart lens can project images directly onto the retina.
A current prototype displayed at the Consumer Electronics Show earlier this year in Las Vegas includes a tiny screen positioned right above the most sensitive area of the pupil. "[The Mojo lens] is a contact lens that essentially has wireless power and data transmission for a small micro LED projector that is placed over the center of the eye," explains David Hobbs, Director of Product Management at Mojo Vision. "[It] displays critical heads-up information when you need it and fades into the background when you're ready to continue on with your day."
Eventually, Mojo Visions' technology could replace our beloved smart devices but the first generation of the Mojo smart lens will be used to help the 2.2 billion people globally who suffer from vision impairment.
"If you think of the eye as a camera [for the visually impaired], the sensors are not working properly," explains Dr. Ashley Tuan, Vice President of Medical Devices at Mojo Vision and fellow of the American Academy of Optometry. "For this population, our lens can process the image so the contrast can be enhanced, we can make the image larger, magnify it so that low-vision people can see it or we can make it smaller so they can check their environment." In January of this year, the FDA granted Breakthrough Device Designation to Mojo, allowing them to have early and frequent discussions with the FDA about technical, safety and efficacy topics before clinical trials can be done and certification granted.
For now, Dr. Tuan is one of the few people who has actually worn the Mojo lens. "I put the contact lens on my eye. It was very comfortable like any contact lenses I've worn before," she describes. "The vision is super clear and then when I put on the accessories, suddenly I see Yoda in front of me and I see my vital signs. And then I have my colleague that prepared a beautiful poem that I loved when I was young [and] I was reading the poem with my eyes closed."
At the moment, there are several electronic glasses on the market like Acesight and Nueyes Pro that provide similar solutions for those suffering from visual impairment, but they are large, cumbersome, and highly visible. Mojo lens would be a discreet, more comfortable alternative that offers users more freedom of movement and independence.
"In the case of augmented-reality contact lenses, there could be an opportunity to improve the lives of people with low vision," says Dr. Thomas Steinemann, spokesperson for the American Academy of Ophthalmology and professor of ophthalmology at MetroHealth Medical Center in Cleveland. "There are existing tools for people currently living with low vision—such as digital apps, magnifiers, etc.— but something wearable could provide more flexibility and significantly more aid in day-to-day tasks."
As one of the first examples of "invisible computing," the potential applications of Mojo lens in the medical field are endless.
According to Dr. Tuan, the visually impaired often suffer from depression due to their lack of mobility and 70 percent of them are underemployed. "We hope that they can use this device to gain their mobility so they can get that social aspect back in their lives and then, eventually, employment," she explains. "That is our first and most important goal."
But helping those with low visual capabilities is only Mojo lens' first possible medical application; augmented reality is already being used in medicine and is poised to revolutionize the field in the coming decades. For example, Accuvein, a device that uses lasers to provide real-time images of veins, is widely used by nurses and doctors to help with the insertion of needles for IVs and blood tests.
According to the National Center for Biotechnology Information, augmentation of reality has been used in surgery for many years with surgeons using devices such as Google Glass to overlay critical information about their patients into their visual field. Using software like the Holographic Navigation Platform by Scopsis, surgeons can see a mixed-reality overlay that can "show you complicated tumor boundaries, assist with implant placements and guide you along anatomical pathways," its developers say.
However, according to Dr. Tuan, augmented reality headsets have drawbacks in the surgical setting. "The advantage of [Mojo lens] is you don't need to worry about sweating or that the headset or glasses will slide down to your nose," she explains "Also, our lens is designed so that it will understand your intent, so when you don't want the image overlay it will disappear, it will not block your visual field, and when you need it, it will come back at the right time."
As one of the first examples of "invisible computing," the potential applications of Mojo lens in the medical field are endless. Possibilities include live translation of sign language for deaf people; helping those with autism to read emotions; and improving doctors' bedside manner by allowing them to fully engage with patients without relying on a computer.
"[By] monitoring those blood vessels we can [track] chronic disease progression: high blood pressure, diabetes, and Alzheimer's."
Furthermore, the lens could be used to monitor health issues. "We have image sensors in the lens right now that point to the world but we can have a camera pointing inside of your eye to your retina," says Dr. Tuan, "[By] monitoring those blood vessels we can [track] chronic disease progression: high blood pressure, diabetes, and Alzheimer's."
For the moment, the future medical applications of the Mojo lens are still theoretical, but the team is confident they can eventually become a reality after going through the proper regulatory review. The company is still in the process of design, prototype and testing of the lens, so they don't know exactly when it will be available for use, but they anticipate shipping the first available products in the next couple of years. Once it does go to market, it will be available by prescription only for those with visual impairments, but the team's goal is to bring it to broader consumer markets pending regulatory clearance.
"We see that right now there's a unique opportunity here for Mojo lens and invisible computing to help to shape what the next decade of technology development looks like," explains David Hobbs. "We can use [the Mojo lens] to better serve us as opposed to us serving technology better."
New tech for prison reform spreads to 11 states
A new non-profit called Recidiviz is using data technology to reduce the size of the U.S. criminal justice system. The bi-coastal company (SF and NYC) is currently working with 11 states to improve their systems and, so far, has helped remove nearly 69,000 people — ones left floundering in jail or on parole when they should have been released.
“The root cause is fragmentation,” says Clementine Jacoby, 31, a software engineer who worked at Google before co-founding Recidiviz in 2019. In the 1970s and 80s, the U.S. built a series of disconnected data systems, and this patchwork is still being used by criminal justice authorities today. It requires parole officers to manually calculate release dates, leading to errors in many cases. “[They] have done everything they need to do to earn their release, but they're still stuck in the system,” Jacoby says.
Recidiviz has built a platform that connects the different databases, with the goal of identifying people who are already qualified for release but remain behind bars or on supervision. “Think of Recidiviz like Google Maps,” says Jacoby, who worked on Maps when she was at the tech giant. Google Maps takes in data from different sources – satellite images, street maps, local business data — and organizes it into one easy view. “Recidiviz does something similar with criminal justice data,” Jacoby explains, “making it easy to identify people eligible to come home or to move to less intensive levels of supervision.”
People like Jacoby’s uncle. His experience with incarceration is what inspired her passion for criminal justice reform in the first place.
The problems are vast
The U.S. has the highest incarceration rate in the world — 2 million people according to the watchdog group, Prison Policy Initiative — at a cost of $182 billion a year. The numbers could be a lot lower if not for an array of problems including inaccurate sentencing calculations, flawed algorithms and parole violations laws.
Sentencing miscalculations
To determine eligibility for release, the current system requires corrections officers to check 21 different requirements spread across five different databases for each of the 90 to 100 people under their supervision. These manual calculations are time prohibitive, says Jacoby, and fall victim to human error.
In addition, Recidiviz found that policies aimed at helping to reduce the prison population don’t always work correctly. A key example is time off for good behavior laws that allow inmates to earn one day off for every 30 days of good behavior. Some states' data systems are built to calculate time off as one day per month of good behavior, rather than per day. Over the course of a decade-long sentence, Jacoby says these miscalculations can lead to a huge discrepancy in the calculated release data and the actual release date.
Algorithms
Commercial algorithm-based software systems for risk assessment continue to be widely used in the criminal justice system, even though a 2018 study published in Science Advances exposed their limitations. After the study went viral, it took three years for the Justice Department to issue a report on their own flawed algorithms used to reduce the federal prison population as part of the 2018 First Step Act. The program, it was determined, overestimated the risk of putting inmates of color into early-release programs.
Despite its name, Recidiviz does not build these types of algorithms for predicting recidivism, or whether someone will commit another crime after being released from prison. Rather, Jacoby says the company’s "descriptive analytics” approach is specifically intended to weed out incarceration inequalities and avoid algorithmic pitfalls.
Parole violation laws
Research shows that 350,000 people a year — about a quarter of the total prison population — are sent back not because they’ve committed another crime, but because they’ve broken a specific rule of their probation. “Things that wouldn't send you or I to prison, but would send someone on parole,” such as crossing county lines or being in the presence of alcohol when they shouldn’t be, are inflating the prison population, says Jacoby.
It’s personal for the co-founder and CEO
“I grew up with an uncle who went into the prison system,” Jacoby says. At 19, he was sentenced to ten years in prison for a non-violent crime. A few months after being released from jail, he was sent back for a non-violent parole violation.
“For my family, the fact that one in four prison admissions are driven not by a crime but by someone who's broken a rule on probation and parole was really profound because that happened to my uncle,” Jacoby says. The experience led her to begin studying criminal justice in high school, then college. She continued her dive into how the criminal justice system works as part of her Passion Project while at Google, a program that allows employees to spend 20 percent of their time on pro-bono work. Two colleagues whose family members had also been stuck in the system joined her.
As part of the project, Jacoby interviewed hundreds of people involved in the criminal justice system. “Those on the right, those on the left, agreed that bad data was slowing down reform,” she says. Their research brought them to North Dakota where they began to understand the root of the problem. The corrections department is making “huge, consequential decisions every day [without] … the data,” Jacoby says. In a new video by Recidiviz not yet released, Jacoby recounts her exchange with the state’s director of corrections who told her, “‘It’s not that we have the data and we just don’t know how to make it public; we don’t have the information you think we have.'"
A mock-up (with fake data) of the types of dashboards and insights that Recidiviz provides to state governments.
Recidiviz
As a software engineer, Jacoby says the comment made no sense to her — until she witnessed it first-hand. “We spent a lot of time driving around in cars with corrections directors and parole officers watching them use these incredibly taxing, frankly terrible, old data systems,” Jacoby says.
As they weeded through thousands of files — some computerized, some on paper — they unearthed the consequences of bad data: Hundreds of people in prison well past their release date and thousands more whose release from parole was delayed because of minor paperwork issues. They found individuals stuck in parole because they hadn’t checked one last item off their eligibility list — like simply failing to provide their parole officer with a paystub. And, even when parolees advocated for themselves, the archaic system made it difficult for their parole officers to confirm their eligibility, so they remained in the system. Jacoby and her team also unpacked specific policies that drive racial disparities — such as fines and fees.
The Solution
It’s more than a trivial technical challenge to bring the incomplete, fragmented data onto a 21st century data platform. It takes months for Recidiviz to sift through a state’s information systems to connect databases “with the goal of tracking a person all the way through their journey and find out what’s working for 18- to 25-year-old men, what’s working for new mothers,” explains Jacoby in the video.
TED Talk: How bad data traps people in the U.S. justice system
TED Fellow Clementine Jacoby's TED Talk went live on Jan. 13. It describes how we can fix bad data in the criminal justice system, "bringing thousands of people home, reducing costs and improving public safety along the way."
Clementine Jacoby • TED2022
Ojmarrh Mitchell, an associate professor in the School of Criminology and Criminal Justice at Arizona State University, who is not involved with the company, says what Recidiviz is doing is “remarkable.” His perspective goes beyond academic analysis. In his pre-academic years, Mitchell was a probation officer, working within the framework of the “well known, but invisible” information sharing issues that plague criminal justice departments. The flexibility of Recidiviz’s approach is what makes it especially innovative, he says. “They identify the specific gaps in each jurisdiction and tailor a solution for that jurisdiction.”
On the downside, the process used by Recidiviz is “a bit opaque,” Mitchell says, with few details available on how Recidiviz designs its tools and tracks outcomes. By sharing more information about how its actions lead to progress in a given jurisdiction, Recidiviz could help reformers in other places figure out which programs have the best potential to work well.
The eleven states in which Recidiviz is working include California, Colorado, Maine, Michigan, Missouri, Pennsylvania and Tennessee. And a pilot program launched last year in Idaho, if scaled nationally, with could reduce the number of people in the criminal justice system by a quarter of a million people, Jacoby says. As part of the pilot, rather than relying on manual calculations, Recidiviz is equipping leaders and the probation officers with actionable information with a few clicks of an app that Recidiviz built.
Mitchell is disappointed that there’s even the need for Recidiviz. “This is a problem that government agencies have a responsibility to address,” he says. “But they haven’t.” For one company to come along and fill such a large gap is “remarkable.”
How Leqembi became the biggest news in Alzheimer’s disease in 40 years, and what comes next
A few months ago, Betsy Groves traveled less than a mile from her home in Cambridge, Mass. to give a talk to a bunch of scientists. The scientists, who worked for the pharmaceutical companies Biogen and Eisai, wanted to know how she lived her life, how she thought about her future, and what it was like when a doctor’s appointment in 2021 gave her the worst possible news. Groves, 73, has Alzheimer’s disease. She caught it early, through a lumbar puncture that showed evidence of amyloid, an Alzheimer’s hallmark, in her cerebrospinal fluid. As a way of dealing with her diagnosis, she joined the Alzheimer’s Association’s National Early-Stage Advisory Board, which helped her shift into seeing her diagnosis as something she could use to help others.
After her talk, Groves stayed for lunch with the scientists, who were eager to put a face to their work. Biogen and Eisai were about to release the first drug to successfully combat Alzheimer’s in 40 years of experimental disaster. Their drug, which is known by the scientific name lecanemab and the marketing name Leqembi, was granted accelerated approval by the U.S. Food and Drug Administration last Friday, Jan. 6, after a study in 1,800 people showed that it reduced cognitive decline by 27 percent over 18 months.
It is no exaggeration to say that this result is a huge deal. The field of Alzheimer’s drug development has been absolutely littered with failures. Almost everything researchers have tried has tanked in clinical trials. “Most of the things that we've done have proven not to be effective, and it's not because we haven’t been taking a ton of shots at goal,” says Anton Porsteinsson, director of the University of Rochester Alzheimer's Disease Care, Research, and Education Program, who worked on the lecanemab trial. “I think it's fair to say you don't survive in this field unless you're an eternal optimist.”
As far back as 1984, a cure looked like it was within reach: Scientists discovered that the sticky plaques that develop in the brains of those who have Alzheimer’s are made up of a protein fragment called beta-amyloid. Buildup of beta-amyloid seemed to be sufficient to disrupt communication between, and eventually kill, memory cells. If that was true, then the cure should be straightforward: Stop the buildup of beta-amyloid; stop the Alzheimer’s disease.
It wasn’t so simple. Over the next 38 years, hundreds of drugs designed either to interfere with the production of abnormal amyloid or to clear it from the brain flamed out in trials. It got so bad that neuroscience drug divisions at major pharmaceutical companies (AstraZeneca, Pfizer, Bristol-Myers, GSK, Amgen) closed one by one, leaving the field to smaller, scrappier companies, like Cambridge-based Biogen and Tokyo-based Eisai. Some scientists began to dismiss the amyloid hypothesis altogether: If this protein fragment was so important to the disease, why didn’t ridding the brain of it do anything for patients? There was another abnormal protein that showed up in the brains of Alzheimer’s patients, called tau. Some researchers defected to the tau camp, or came to believe the proteins caused damage in combination.
The situation came to a head in 2021, when the FDA granted provisional approval to a drug called aducanumab, marketed as Aduhelm, against the advice of its own advisory council. The approval was based on proof that Aduhelm reduced beta-amyloid in the brain, even though one research trial showed it had no effect on people’s symptoms or daily life. Aduhelm could also cause serious side effects, like brain swelling and amyloid related imaging abnormalities (known as ARIA, these are basically micro-bleeds that appear on MRI scans). Without a clear benefit to memory loss that would make these risks worth it, Medicare refused to pay for Aduhelm among the general population. Two congressional committees launched an investigation into the drug’s approval, citing corporate greed, lapses in protocol, and an unjustifiably high price. (Aduhelm was also produced by the pharmaceutical company Biogen.)
To be clear, Leqembi is not the cure Alzheimer’s researchers hope for. While the drug is the first to show clear signs of a clinical benefit, the scientific establishment is split on how much of a difference Leqembi will make in the real world.
So far, Leqembi is like Aduhelm in that it has been given accelerated approval only for its ability to remove amyloid from the brain. Both are monoclonal antibodies that direct the immune system to attack and clear dysfunctional beta-amyloid. The difference is that, while that’s all Aduhelm was ever shown to do, Leqembi’s makers have already asked the FDA to give it full approval – a decision that would increase the likelihood that Medicare will cover it – based on data that show it also improves Alzheimer’s sufferer’s lives. Leqembi targets a different type of amyloid, a soluble version called “protofibrils,” and that appears to change the effect. “It can give individuals and their families three, six months longer to be participating in daily life and living independently,” says Claire Sexton, PhD, senior director of scientific programs & outreach for the Alzheimer's Association. “These types of changes matter for individuals and for their families.”
To be clear, Leqembi is not the cure Alzheimer’s researchers hope for. It does not halt or reverse the disease, and people do not get better. While the drug is the first to show clear signs of a clinical benefit, the scientific establishment is split on how much of a difference Leqembi will make in the real world. It has “a rather small effect,” wrote NIH Alzheimer’s researcher Madhav Thambisetty, MD, PhD, in an email to Leaps.org. “It is unclear how meaningful this difference will be to patients, and it is unlikely that this level of difference will be obvious to a patient (or their caregivers).” Another issue is cost: Leqembi will become available to patients later this month, but Eisai is setting the price at $26,500 per year, meaning that very few patients will be able to afford it unless Medicare chooses to reimburse them for it.
The same side effects that plagued Aduhelm are common in Leqembi treatment as well. In many patients, amyloid doesn’t just accumulate around neurons, it also forms deposits in the walls of blood vessels. Blood vessels that are shot through with amyloid are more brittle. If you infuse a drug that targets amyloid, brittle blood vessels in the brain can develop leakage that results in swelling or bleeds. Most of these come with no symptoms, and are only seen during testing, which is why they are called “imaging abnormalities.” But in situations where patients have multiple diseases or are prescribed incompatible drugs, they can be serious enough to cause death. The three deaths reported from Leqembi treatment (so far) are enough to make Thambisetty wonder “how well the drug may be tolerated in real world clinical practice where patients are likely to be sicker and have multiple other medical conditions in contrast to carefully selected patients in clinical trials.”
Porsteinsson believes that earlier detection of Alzheimer’s disease will be the next great advance in treatment, a more important step forward than Leqembi’s approval.
Still, there are reasons to be excited. A successful Alzheimer’s drug can pave the way for combination studies, in which patients try a known effective drug alongside newer, more experimental ones; or preventative studies, which take place years before symptoms occur. It also represents enormous strides in researchers’ understanding of the disease. For example, drug dosages have increased massively—in some cases quadrupling—from the early days of Alzheimer’s research. And patient selection for studies has changed drastically as well. Doctors now know that you’ve got to catch the disease early, through PET-scans or CSF tests for amyloid, if you want any chance of changing its course.
Porsteinsson believes that earlier detection of Alzheimer’s disease will be the next great advance in treatment, a more important step forward than Leqembi’s approval. His lab already uses blood tests for different types of amyloid, for different types of tau, and for measures of neuroinflammation, neural damage, and synaptic health, but commercially available versions from companies like C2N, Quest, and Fuji Rebio are likely to hit the market in the next couple of years. “[They are] going to transform the diagnosis of Alzheimer's disease,” Porsteinsson says. “If someone is experiencing memory problems, their physicians will be able to order a blood test that will tell us if this is the result of changes in your brain due to Alzheimer's disease. It will ultimately make it much easier to identify people at a very early stage of the disease, where they are most likely to benefit from treatment.”
Learn more about new blood tests to detect Alzheimer's
Early detection can help patients for more philosophical reasons as well. Betsy Groves credits finding her Alzheimer’s early with giving her the space to understand and process the changes that were happening to her before they got so bad that she couldn’t. She has been able to update her legal documents and, through her role on the Advisory Group, help the Alzheimer’s Association with developing its programs and support services for people in the early stages of the disease. She still drives, and because she and her husband love to travel, they are hoping to get out of grey, rainy Cambridge and off to Texas or Arizona this spring.
Because her Alzheimer’s disease involves amyloid deposits (a “substantial portion” do not, says Claire Sexton, which is an additional complication for research), and has not yet reached an advanced stage, Groves may be a good candidate to try Leqembi. She says she’d welcome the opportunity to take it. If she can get access, Groves hopes the drug will give her more days to be fully functioning with her husband, daughters, and three grandchildren. Mostly, she avoids thinking about what the latter stages of Alzheimer’s might be like, but she knows the time will come when it will be her reality. “So whatever lecanemab can do to extend my more productive ways of engaging with relationships in the world,” she says. “I'll take that in a minute.”