“Mindwriting” – Software Can Transform Thoughts About Handwriting Into Words and Sentences


Experts From Stanford developed software that turns ‘mental handwriting’ into on-screen words and sentences. They Call it “mindwriting.”

The combination of technology and mental effort has enabled a man with paralyzing limbs to interact by text at speeds rivaling those achieved by his fit peers texting on a smartphone.

Researchers from Stanford University have combined artificial intelligence software with a tool, called a brain-computer interface, embedded in the brain of a person with full-body paralysis. The Mindwriting software was able to decrypt data from the BCI to immediately transform the man’s thoughts about handwriting into text on a computer screen.

The man was able to communicate using this software more than twice as fast as he could using a prior approach developed by the Stanford scientists, who published those findings in 2017 in the journal eLife.

Mindwriting: Brain-to-text communication via handwriting

The latest discovery, published online in Nature on May 12, 2021, could drive more progress helping millions of people globally, who’ve lost their sense to use their upper arms or to speak due to strokes, or amyotrophic lateral sclerosis, also known as Lou Gehrig’s disorder, said Jaimie Henderson, MD, professor of neurosurgery.

“This software supported a person with paralysis to form sentences at speeds similar to those of fit adults of the same age group typing on a smartphone,” said Jene Blume, Henderson, and John — Robert and Ruth Halperin Professor. “The purpose is to restore the ability to interact by text.”

The candidate in the research punches out the text at a speed of about 18 words per minute. However, fit people of the same age group can produce about 23 words per minute on a smartphone.

The candidate, referred to as T5, lost almost all movement below the neck due to a spinal cord injury in 2007. Nine years later, Henderson implanted two brain-computer-interface chips on the left side of T5’s brain, each the size of baby aspirin. Each chip consists of 100 electrodes that fetch signals from neurons firing in the region of the motor cortex — a part of the brain’s outermost surface — that commands hand movement.

Those neural signals are transmitted by cables to a computer, where artificial intelligence algorithms decrypt the signals and surmise T5’s expected finger and hand motion. The algorithms were produced in Stanford’s Neural Prosthetics Translational Lab, co-managed by Vivian W. M. Lim and Hong Seh Professor of Engineering, Krishna Shenoy and Henderson, Ph.D., professor of electrical engineering.

Henderson and Shenoy, who have been cooperating on BCIs since 2005, are the senior co-authors of the latest research. The lead author of the study is Frank Willett, Ph.D., an analysis scientist in the lab and with the Howard Hughes Medical Institute.

“We’ve discovered that the brain keeps its ability to direct fine movements for almost a decade after the body has lost its ability to do those actions,” Willett said. “And we’ve discovered that complex actions including curved trajectories and varying speeds, like writing, can be performed more efficiently and more quickly by the artificial-intelligence algorithms we’re using than can more simplistic expected motions like moving a cursor in a straight path at a constant speed. Alphabetical letters are distinct from one another, so they’re simpler to tell apart.”

In the 2017 research, three candidates with limb paralysis, including T5 — all with BCIs implanted in the motor cortex — were asked to focus on using a hand and arm to move a cursor from one key to the next on a computer-screen keyboard display, then to concentrate on clicking on that key.

In that research, T5 set the all-time record of copying displayed sentences at approximately 40 characters per minute. Another research candidate was able to write freely, picking whatever words she wanted, at about 25 characters per minute.

If the 2017 research model was similar to typing, the model for the new Nature research is similar to handwriting. T5 focused on trying to write different alphabets on an imaginative legal pad with an imaginary pen, despite his frailty to move his hand or arm. He re-writes each alphabet 10 times, allowing the software to “learn” to identify the neural signals linked with his effort to write that specific letter.

In various long-hour gatherings that followed, T5 was presented with a collection of sentences and told to make a mental effort to “handwrite” each one. No uppercase alphabets were used. Examples of the sentences were “I intervened, unable to remain quiet,” and “in one minute the soldiers had arrived.” Over time, the algorithms enhanced their understanding to distinguish among the neural firing patterns typifying distinct characters. The algorithms’ analysis of whatever character T5 was trying to write appeared on the computer screen after a roughly half-second delay.

In further gatherings, T5 was told to write sentences the algorithms had never been exposed to. He was finally able to produce 90 characters or about 18 words per minute. Later, asked to give his replies to open-ended questions, which needed some intervals for thought, he produced 73.8 characters (about 15 words, on average) per minute, tripling the prior free-composition record set in the 2017 research.

T5’s sentence-copying error rate was around one mistake in every 18 or 19 attempted words. His free-composition error rate was nearly one in every 12 words. When the scientists employed an after-the-fact autocorrect function — similar to the ones included in our smartphone — to pick the correct words, those error rates were notably lower: just over 2% for freestyle and below 1% for copying.

These error frequencies are low in contrast to other BCIs, said Shenoy, who is also a Howard Hughes Medical Institute investigator.

“While writing can progress to 20 words per minute, we tend to speak almost 125 words per minute, and this is another interesting area that complements writing. If joined, these systems could collectively offer even more opportunities for patients to interact efficiently,” Shenoy said.

Journal Reference:

“High-performance brain-to-text communication via handwriting” by Francis R. Willett, Jaimie M. Henderson, Donald T. Avansino, Krishna V. Shenoy, and Leigh R. Hochberg, 12 May 2021, Nature.

DOI: 10.1038/s41586-021-03506-2