Tag Archives: Mind-controlled Exoskeleton

Brazil, the 2014 World Cup kickoff, and a mind-controlled exoskeleton (part four of five)

The Brain research, ethics, and nanotechnology (part one of five) May 19, 2014 post kicked off a series titled ‘Brains, prostheses, nanotechnology, and human enhancement’ which brings together a number of developments in the worlds of neuroscience, prosthetics, and, incidentally, nanotechnology in the field of interest called human enhancement. Parts one through four are an attempt to draw together a number of new developments, mostly in the US and in Europe. Due to my language skills which extend to English and, more tenuously, French, I can’t provide a more ‘global perspective’. Part five features a summary.

Brazil’s World Cup for soccer/football which opens on June 12, 2014 will be the first public viewing of someone with paraplegia demonstrating a mind-controlled exoskeleton (or a robotic suit as it’s sometimes called) by opening the 2014 games with the first kick-off.

I’ve been covering this story since 2011 and, even so, was late to the party as per this May 7, 2014 article by Alejandra Martins for BBC World news online,

The World Cup curtain-raiser will see the first public demonstration of a mind-controlled exoskeleton that will enable a person with paralysis to walk.

If all goes as planned, the robotic suit will spring to life in front of almost 70,000 spectators and a global audience of billions of people.

The exoskeleton was developed by an international team of scientists as part of the Walk Again Project and is the culmination of more than a decade of work for Dr Miguel Nicolelis, a Brazilian neuroscientist based at Duke University in North Carolina. [emphasis mine]

Since November [2013], Dr Nicolelis has been training eight patients at a lab in Sao Paulo, in the midst of huge media speculation that one of them will stand up from his or her wheelchair and deliver the first kick of this year’s World Cup.

“That was the original plan,” the Duke University researcher told the BBC. “But not even I could tell you the specifics of how the demonstration will take place. This is being discussed at the moment.”

Speaking in Portuguese from Sao Paulo, Miguel Nicolelis explained that all the patients are over 20 years of age, with the oldest about 35.

“We started the training in a virtual environment with a simulator. In the last few days, four patients have donned the exoskeleton to take their first steps and one of them has used mental control to kick a ball,” he explained.

The history of Nicolelis’ work is covered here in a series of a posts starting the with an Oct. 5, 2011 post (Advertising for the 21st Century: B-Reel, ‘storytelling’, and mind control; scroll down 2/3 of the way for a reference to Ed Yong’s article where I first learned of Nicolelis).

The work was explored in more depth in a March 16, 2012 posting (Monkeys, mind control, robots, prosthetics, and the 2014 World Cup (soccer/football) and then followed up a year later by two posts which link Nicoleliis’ work with the Brain Activity Map (now called, BRAIN [Brain Research through Advancing Innovative Neurotechnologies] initiative: a March 4, 2013 (Brain-to-brain communication, organic computers, and BAM [brain activity map], the connectome) and a March 8,  2013 post (Prosthetics and the human brain) directly linking exoskeleton work in Holland and the project at Duke with current brain research and the dawning of a new relationship to one’s prosthestics,

On the heels of research which suggests that humans tend to view their prostheses, including wheel chairs, as part of their bodies, researchers in Europe  have announced the development of a working exoskeleton powered by the wearer’s thoughts.

Getting back to Brazil and Nicolelis’ technology, Ian Sample offers an excellent description in an April 1, 2014 article for the Guardian (Note: Links have been removed),

The technology in question is a mind-controlled robotic exoskeleton. The complex and conspicuous robotic suit, built from lightweight alloys and powered by hydraulics, has a simple enough function. When a paraplegic person straps themselves in, the machine does the job that their leg muscles no longer can.

The exoskeleton is the culmination of years of work by an international team of scientists and engineers on the Walk Again project. The robotics work was coordinated by Gordon Cheng at the Technical University in Munich, and French researchers built the exoskeleton. Nicolelis’s team focused on ways to read people’s brain waves, and use those signals to control robotic limbs.

To operate the exoskeleton, the person is helped into the suit and given a cap to wear that is fitted with electrodes to pick up their brain waves. These signals are passed to a computer worn in a backpack, where they are decoded and used to move hydraulic drivers on the suit.

The exoskeleton is powered by a battery – also carried in the backpack – that allows for two hours of continuous use.

“The movements are very smooth,” Nicolelis told the Guardian. “They are human movements, not robotic movements.”

Nicolelis says that in trials so far, his patients seem have taken to the exoskeleton. “This thing was made for me,” one patient told him after being strapped into the suit.

The operator’s feet rest on plates which have sensors to detect when contact is made with the ground. With each footfall, a signal shoots up to a vibrating device sewn into the forearm of the wearer’s shirt. The device seems to fool the brain into thinking that the sensation came from their foot. In virtual reality simulations, patients felt that their legs were moving and touching something.

Sample’s article includes a good schematic of the ‘suit’ which I have not been able to find elsewhere (meaning the Guardian likely has a copyright for the schematic and is why you won’t see it here) and speculation about robotics and prosthetics in the future.

Nicolelis and his team have a Facebook page for the Walk Again Project where you can get some of the latest information with  both English and Portuguese language entries as they prepare for the June 12, 2014 kickoff.

One final thought, this kickoff project represents an unlikely confluence of events. After all, what are the odds

    • that a Brazil-born researcher (Nicolelis) would be working on a project to give paraplegics the ability to walk again? and
    • that Brazil would host the World Cup in 2014 (the first time since 1950)? and
    • that the timing would coincide so a public demonstration at one of the world’s largest athletic events (of a sport particularly loved in Brazil) could be planned?

It becomes even more extraordinary when one considers that Brazil had isolated itself somewhat in the 1980s with a policy of nationalism vis à vis the computer industry (from the Brazil Science and Technology webpage on the ITA website),

In the early 1980s, the policy of technological nationalism and self-sufficiency had narrowed to the computer sector, where protective legislation tried to shield the Brazilian mini- and microcomputer industries from foreign competition. Here again, the policy allowed for the growth of local industry and a few well-qualified firms, but the effect on the productive capabilities of the economy as a whole was negative; and the inability to follow the international market in price and quality forced the policy to be discontinued.

For those who may have forgotten, the growth of the computer industry (specifically personal computers) in the 1980s figured hugely in a country’s economic health and, in this case,with  a big negative impact in Brazil.

Returning to 2014, the kickoff in Brazil (if successful) symbolizes more than an international athletic competition or a technical/medical achievement, this kick-off symbolizes a technological future for Brazil and its place on the world stage (despite the protests and social unrest) .

Links to other posts in the Brains, prostheses, nanotechnology, and human enhancement five-part series

Part one: Brain research, ethics, and nanotechnology (May 19, 2014 post)

Part two: BRAIN and ethics in the US with some Canucks (not the hockey team) participating (May 19, 2014)

Part three: Gray Matters: Integrative Approaches for Neuroscience, Ethics, and Society issued May 2014 by US Presidential Bioethics Commission (May 20, 2014)

Part five: Brains, prostheses, nanotechnology, and human enhancement: summary (May 20, 2014)

ETA June 16, 2014: The kickoff seems to have been a disappointment (June 15, 2014 news item on phys.org) and for those who might be interested in some of the reasons for the World Cup unrest and protests in Brazil, John Oliver provides an excoriating overview of the organization which organizes the World Cup games while professing his great love of the games, http://www.youtube.com/watch?v=DlJEt2KU33I

Prosthetics and the human brain

On the heels of research which suggests that humans tend to view their prostheses, including wheel chairs, as part of their bodies, researchers in Europe  have announced the development of a working exoskeleton powered by the wearer’s thoughts.

First, there’s the ‘wheelchair’ research, from the Mar. 6, 2013 news item on ScienceDaily,

People with spinal cord injuries show strong association of wheelchairs as part of their body, not extension of immobile limbs.

The human brain can learn to treat relevant prosthetics as a substitute for a non-working body part, according to research published March 6 in the open access journal PLOS ONE by Mariella Pazzaglia and colleagues from Sapienza University and IRCCS Fondazione Santa Lucia of Rome in Italy, supported by the International Foundation for Research in Paraplegie.

The researchers found that wheelchair-bound study participants with spinal cord injuries perceived their body’s edges as being plastic and flexible to include the wheelchair, independent of time since their injury or experience with using a wheelchair. Patients with lower spinal cord injuries who retained upper body movement showed a stronger association of the wheelchair with their body than those who had spinal cord impairments in the entire body.

According to the authors, this suggests that rather than being thought of only as an extension of the immobile limbs, the wheelchairs had become tangible, functional substitutes for the affected body part. …

As I mentioned in a Jan. 30, 2013 posting,

There have been some recent legal challenges as to what constitutes one’s body (from The Economist article, You, robot? [you can find the article here: http://www.economist.com/node/21560986]),

If you are dependent on a robotic wheelchair for mobility, for example, does the wheelchair count as part of your body? Linda MacDonald Glenn, an American lawyer and bioethicist, thinks it does. Ms Glenn (who is not involved in the RoboLaw project) persuaded an initially sceptical insurance firm that a “mobility assistance device” damaged by airline staff was more than her client’s personal property, it was an extension of his physical body. The airline settled out of court.

According to the Mar. 6, 2013 news release on EurekAlert from the Public Library of Science (PLoS), the open access article by Pazzaglia and her colleagues can be found here (Note: I have added a link),

Pazzaglia M, Galli G, Scivoletto G, Molinari M (2013) A Functionally Relevant Tool for the Body following Spinal Cord Injury. PLOS ONE 8(3): e58312.doi:10.1371/journal.pone.0058312

At almost the same time as Pazzaglia’s work,  a “Mind-controlled Exoskeleton” is announced in a Mar. 7, 2013 news item on ScienceDaily,

Every year thousands of people in Europe are paralysed by a spinal cord injury. Many are young adults, facing the rest of their lives confined to a wheelchair. Although no medical cure currently exists, in the future they could be able to walk again thanks to a mind-controlled robotic exoskeleton being developed by EU-funded researchers.

The system, based on innovative ‘Brain-neural-computer interface’ (BNCI) technology — combined with a light-weight exoskeleton attached to users’ legs and a virtual reality environment for training — could also find applications in there habilitation of stroke victims and in assisting astronauts rebuild muscle mass after prolonged periods in space.

The Mar. 7, 2013 news release on CORDIS, which originated the news item, offers a description of the “Mindwalker” project,

‘Mindwalker was proposed as a very ambitious project intended to investigate promising approaches to exploit brain signals for the purpose of controlling advanced orthosis, and to design and implement a prototype system demonstrating the potential of related technologies,’ explains Michel Ilzkovitz, the project coordinator at Space Applications Services in Belgium.

The team’s approach relies on an advanced BNCI system that converts electroencephalography (EEG) signals from the brain, or electromyography (EMG) signals from shoulder muscles, into electronic commands to control the exoskeleton.

The Laboratory of Neurophysiology and Movement Biomechanics at the Université Libre de Bruxelles (ULB) focused on the exploitation of EEG and EMG signals treated by an artificial neural network, while the Foundation Santa Lucia in Italy developed techniques based on EMG signals modelled by the coupling of neural and biomechanical oscillators.

One approach for controlling the exoskeleton uses so-called ‘steady-state visually evoked potential’, a method that reads flickering visual stimuli produced at different frequencies to induce correlated EEG signals. Detection of these EEG signals is used to trigger commands such as ‘stand’, ‘walk’, ‘faster’ or ‘slower’.

A second approach is based on processing EMG signals generated by the user’s shoulders and exploits the natural arm-leg coordination in human walking: arm-swing patterns can be perceived in this way and converted into control signals commanding the exoskeleton’s legs.

A third approach, ‘ideation’, is also based on EEG-signal processing. It uses the identification and exploitation of EEG Theta cortical signals produced by the natural mental process associated with walking. The approach was investigated by the Mindwalker team but had to be dropped due to the difficulty, and time needed, in turning the results of early experiments into a fully exploitable system.

Regardless of which method is used, the BNCI signals have to be filtered and processed before they can be used to control the exoskeleton. To achieve this, the Mindwalker researchers fed the signals into a ‘Dynamic recurrent neural network'(DRNN), a processing technique capable of learning and exploiting the dynamic character of the BNCI signals.

‘This is appealing for kinematic control and allows a much more natural and fluid way of controlling an exoskeleton,’ Mr Ilzkovitz says.

The team adopted a similarly practical approach for collecting EEG signals from the user’s scalp. Most BNCI systems are either invasive, requiring electrodes to be placed directly into brain tissue, or require users to wear a ‘wet’ capon their head, necessitating lengthy fitting procedures and the use of special gels to reduce the electrical resistance at the interface between the skin and the electrodes. While such systems deliver signals of very good quality and signal-to-noise ratio, they are impractical for everyday use.

The Mindwalker team therefore turned to a ‘dry’ technology developed by Berlin-based eemagine Medical Imaging Solutions: a cap covered in electrodes that the user can fit themselves, and which uses innovative electronic components to amplify and optimise signals before sending them to the neural network.

‘The dry EEG cap can be placed by the subject on their head by themselves in less than a minute, just like a swimming cap,’ Mr Ilzkovitz says.

Before proceeding any further with details, here’s what the Mindwalker looks like,

© MINDWALKER (downladed from http://cordis.europa.eu/fetch?CALLER=OFFR_TM_EN&ACTION=D&RCN=10601)

© MINDWALKER (downloaded from http://cordis.europa.eu/fetch?CALLER=OFFR_TM_EN&ACTION=D&RCN=10601)

After finding a way to collect the EEG/EMG signals and interpret them, the researchers needed to create the exoskeleton (from the CORDIS news release),

The universities of Delft and Twente in the Netherlands proposed an innovative approach for the design of the exoskeleton and its control. The exoskeletonis designed to be sufficiently robust to bear the weight of a 100 kg adult and powerful enough to recover balance from external causes of instability such as the user’s own torso movements during walking or a gentle push from the back or side. Compared to other exoskeletons developed to date it is relatively light, weighing less than 30 kg without batteries, and, because a final version of the system should be self-powered, it is designed to minimise energy consumption.

The Mindwalker researchers achieved energy efficiency through the use of springs fitted inside the joints that are capable of absorbing and recovering some of the energy otherwise dissipated during walking, and through the development of an efficient strategy for controlling the exoskeleton.

Most exoskeletons are designed to be balanced when stationary or quasi-static and to move by little steps inside their ground stability perimeter, an approach known as ‘Zero moment point’, or ZMP. Although this approach is commonly used for controlling humanoid robots, when applied to exoskeletons, it makes them heavy and slow – and usually requires users to be assisted by a walking frame, sticks or some other support device when they move.

Alternatively, a more advanced and more natural control strategy can replicate the way humans actually walk, with a controlled loss of balance in the walking direction.

‘This approach is called “Limit-cycle walking” and has been implemented using model predictive control to predict the behaviour of the user and exoskeleton and for controlling the exoskeleton during the walk. This was the approach investigated in Mindwalker,’ Mr Ilzkovitz says.

To train users to control the exoskeleton, researchers from Space Applications Services developed a virtual-reality training platform, providing an immersive environment in which new users can safely become accustomed to using the system before testing it out in a clinical setting, and, the team hope, eventually using it in everyday life.

By the end of this year, tests with able-bodied trial users will be completed. The system will then be transferred to the Foundation Santa Lucia for conducting a clinical evaluation until May 2013 with five to 10volunteers suffering from spinal cord injuries. These trials will help identify shortcomings and any areas of performance improvement, the project coordinator says.

In the meantime, the project partners are continuing research on different components for a variety of potential applications. The project coordinator notes, for example, that elements of the system could be adapted for the rehabilitation of stroke victims or to develop easy-to-use exoskeletons for elderly people for mobility support.

Space Applications Services, meanwhile, is also exploring applications of the Mindwalker technology to train astronauts and help them rebuild muscle mass after spending long periods of time in zero-gravity environments.

There’s more about the European Commission’s Seventh Programme-funded Mindwalker project here.

Parallel with these developments in Europe, Miguel Nicolelis of Duke University has stated that he will have a working exoskeleton (Walk Again Project)  for the kickoff by a paraplegic individual for the opening of the World Cup (soccer/football) in Brazil in 2014. I mentioned Nicolelis and his work most recently in a Mar. 4, 2013 posting.

Taken together, this research which strongly suggests that people can perceive prostheses as being part of their bodies and exoskeletons that are powered by the wearer’s thoughts, we seem to be edging closer to a world where machines and humans become one.