Tag Archives: Brown University

Nanostrings that vibrate for a long, long, long time at ambient termperatures

It was the ‘ambient temperature’ that caught my attention, from a May 22, 204 news item on Nanowerk, Note: Links have been removed, Note: Much exciting work has to be conducted at very, very cold or very, very hot temperatures, so, ambient or room temperature is a big deal,

Researchers from TU Delft [Delft University of Technology] and Brown University have engineered string-like resonators capable of vibrating longer at ambient temperature than any previously known solid-state object — approaching what is currently only achievable near absolute zero temperatures. Their study, published in Nature Communications (“Centimeter-scale nanomechanical resonators with low dissipation”), pushes the edge of nanotechnology and machine learning to make some of the world’s most sensitive mechanical sensors.

Caption: Artist impression of new nanostrings that can vibrate for a very long time. These nanostrings vibrate more than 100.000 times per second. Because it’s difficult for energy to leak out, it also means environmental noise is hard to get in, making these some of the best sensors for room temperature environments. Credit: Richard Norte

A May 21, 2024 Delft University of Technology news release (also on EurekAlert but published May 22, 2024), which originated the news item, explains why the research is considered exciting,

A 100 year swing on a microchip

“Imagine a swing that, once pushed, keeps swinging for almost 100 years because it loses almost no energy through the ropes,” says associate professor Richard Norte. “Our nanostrings do something similar but rather than vibrating once per second like a swing, our strings vibrate 100,000 times per second. Because it’s difficult for energy to leak out, it also means environmental noise is hard to get in, making these some of the best sensors for room temperature environments.

This innovation is pivotal for studying macroscopic quantum phenomena at room temperature—environments where such phenomena were previously masked by noise. While the weird laws of quantum mechanics are usually only seen in single atoms, the nanostrings’ ability to isolate themselves from our everyday heat-based vibrational noise allows them to open a window into their own quantum signatures; strings made from billions of atoms. In everyday environments, this kind of capability would have interesting uses for quantum-based sensing.

Extraordinary match between simulation and experiment

“Our manufacturing process goes in a different direction with respect to what is possible in nanotechnology today,” said Dr. Andrea Cupertino, who spearheaded the experimental efforts. The strings are 3 centimetres long and 70 nanometres thick, but scaled up, this would be the equivalent of manufacturing guitar strings of glass that are suspended half a kilometre with almost no sag. “This kind of extreme structures are only feasible at nanoscales where the effects of gravity and weight enter differently. This allows for structures that would be unfeasible at our everyday scales but are particularly useful in miniature devices used to measure physical quantities such as pressure, temperature, acceleration and magnetic fields, which we call MEMS sensing,” explains Cupertino.

The nanostrings are crafted using advanced nanotechnology techniques developed at the TU Delft, pushing the boundaries of how thin and long suspended nanostructures can be made. A key of the collaboration is that these nanostructures can be made so perfectly on a microchip, that there is an extraordinary match between simulations and experiments – meaning that simulations can act as the data for machine learning algorithms, rather than costly experiments. “Our approach involved using machine learning algorithms to optimize the design without continuously fabricating prototypes,” noted lead author Dr. Dongil Shin, who developed these algorithms with Miguel Bessa. To further enhance efficiency of designing these large detailed structures, the machine learning algorithms smartly utilised insights from simpler, shorter string experiments to refine the designs of longer strings, making the development process both economical and effective.

According to Norte, the success of this project is a testament to the fruitful collaboration between experts in nanotechnology and machine learning, underscoring the interdisciplinary nature of cutting-edge scientific research.

Inertial navigation and next-generation microphones

The implications of these nanostrings extend beyond basic science. They offer promising new pathways for integrating highly sensitive sensors with standard microchip technology, leading to new approaches in vibration-based sensing. While these initial studies focus on strings, the concepts can be expanded to more complex designs to measure other important parameters like acceleration for inertial navigation or something looking more like a vibrating drumhead for next-generation microphones. This research demonstrates the vast array of possibilities when combining nanotechnology advances with machine learning to open new frontiers in technology.

Here’s a link to and a citation for the paper,

Centimeter-scale nanomechanical resonators with low dissipation by Andrea Cupertino, Dongil Shin, Leo Guo, Peter G. Steeneken, Miguel A. Bessa & Richard A. Norte. Nature Communications volume 15, Article number: 4255 (2024) DOI: https://doi.org/10.1038/s41467-024-48183-7 Published: 18 May 2024

This paper is open access.

I have highlighted work from this team previously in a September 15, 2022 posting, “One of world’s most precise microchip sensors thanks to nanotechnology, machine learning, extended cognition, and spiderwebs.”

Brain-inspired (neuromorphic) wireless system for gathering data from sensors the size of a grain of salt

This is what a sensor the size of a grain of salt looks like,

Caption: The sensor network is designed so the chips can be implanted into the body or integrated into wearable devices. Each submillimeter-sized silicon sensor mimics how neurons in the brain communicate through spikes of electrical activity. Credit: Nick Dentamaro/Brown University

A March 19, 2024 news item on Nanowerk announces this research from Brown University (Rhode Island, US), Note: A link has been removed,

Tiny chips may equal a big breakthrough for a team of scientists led by Brown University engineers.

Writing in Nature Electronics (“An asynchronous wireless network for capturing event-driven data from large populations of autonomous sensors”), the research team describes a novel approach for a wireless communication network that can efficiently transmit, receive and decode data from thousands of microelectronic chips that are each no larger than a grain of salt.

One of the potential applications is for brain (neural) implants,

Caption: Writing in Nature Electronics, the research team describes a novel approach for a wireless communication network that can efficiently transmit, receive and decode data from thousands of microelectronic chips that are each no larger than a grain of salt. Credit: Nick Dentamaro/Brown University

A March 19, 2024 Brown University news release (also on EurekAlert), which originated the news item, provides more detail about the research, Note: Links have been removed,

The sensor network is designed so the chips can be implanted into the body or integrated into wearable devices. Each submillimeter-sized silicon sensor mimics how neurons in the brain communicate through spikes of electrical activity. The sensors detect specific events as spikes and then transmit that data wirelessly in real time using radio waves, saving both energy and bandwidth.

“Our brain works in a very sparse way,” said Jihun Lee, a postdoctoral researcher at Brown and study lead author. “Neurons do not fire all the time. They compress data and fire sparsely so that they are very efficient. We are mimicking that structure here in our wireless telecommunication approach. The sensors would not be sending out data all the time — they’d just be sending relevant data as needed as short bursts of electrical spikes, and they would be able to do so independently of the other sensors and without coordinating with a central receiver. By doing this, we would manage to save a lot of energy and avoid flooding our central receiver hub with less meaningful data.”

This radiofrequency [sic] transmission scheme also makes the system scalable and tackles a common problem with current sensor communication networks: they all need to be perfectly synced to work well.

The researchers say the work marks a significant step forward in large-scale wireless sensor technology and may one day help shape how scientists collect and interpret information from these little silicon devices, especially since electronic sensors have become ubiquitous as a result of modern technology.

“We live in a world of sensors,” said Arto Nurmikko, a professor in Brown’s School of Engineering and the study’s senior author. “They are all over the place. They’re certainly in our automobiles, they are in so many places of work and increasingly getting into our homes. The most demanding environment for these sensors will always be inside the human body.”

That’s why the researchers believe the system can help lay the foundation for the next generation of implantable and wearable biomedical sensors. There is a growing need in medicine for microdevices that are efficient, unobtrusive and unnoticeable but that also operate as part of a large ensembles to map physiological activity across an entire area of interest.

“This is a milestone in terms of actually developing this type of spike-based wireless microsensor,” Lee said. “If we continue to use conventional methods, we cannot collect the high channel data these applications will require in these kinds of next-generation systems.”

The events the sensors identify and transmit can be specific occurrences such as changes in the environment they are monitoring, including temperature fluctuations or the presence of certain substances.

The sensors are able to use as little energy as they do because external transceivers supply wireless power to the sensors as they transmit their data — meaning they just need to be within range of the energy waves sent out by the transceiver to get a charge. This ability to operate without needing to be plugged into a power source or battery make them convenient and versatile for use in many different situations.

The team designed and simulated the complex electronics on a computer and has worked through several fabrication iterations to create the sensors. The work builds on previous research from Nurmikko’s lab at Brown that introduced a new kind of neural interface system called “neurograins.” This system used a coordinated network of tiny wireless sensors to record and stimulate brain activity.

“These chips are pretty sophisticated as miniature microelectronic devices, and it took us a while to get here,” said Nurmikko, who is also affiliated with Brown’s Carney Institute for Brain Science. “The amount of work and effort that is required in customizing the several different functions in manipulating the electronic nature of these sensors — that being basically squeezed to a fraction of a millimeter space of silicon — is not trivial.”

The researchers demonstrated the efficiency of their system as well as just how much it could potentially be scaled up. They tested the system using 78 sensors in the lab and found they were able to collect and send data with few errors, even when the sensors were transmitting at different times. Through simulations, they were able to show how to decode data collected from the brains of primates using about 8,000 hypothetically implanted sensors.

The researchers say next steps include optimizing the system for reduced power consumption and exploring broader applications beyond neurotechnology.

“The current work provides a methodology we can further build on,” Lee said.

Here’s a link to and a citation for the study,

An asynchronous wireless network for capturing event-driven data from large populations of autonomous sensors by Jihun Lee, Ah-Hyoung Lee, Vincent Leung, Farah Laiwalla, Miguel Angel Lopez-Gordo, Lawrence Larson & Arto Nurmikko. Nature Electronics volume 7, pages 313–324 (2024) DOI: https://doi.org/10.1038/s41928-024-01134-y Published: 19 March 2024 Issue Date: April 2024

This paper is behind a paywall.

Prior to this, 2021 seems to have been a banner year for Nurmikko’s lab. There’s this August 12, 2021 Brown University news release touting publication of a then new study in Nature Electronics and I have an April 2, 2021 post, “BrainGate demonstrates a high-bandwidth wireless brain-computer interface (BCI),” touting an earlier 2021 published study from the lab.

Neural (brain) implants and hype (long read)

There was a big splash a few weeks ago when it was announced that Neuralink’s (Elon Musk company) brain implant had been surgically inserted into its first human patient.

Getting approval

David Tuffley, senior lecturer in Applied Ethics & CyberSecurity at Griffith University (Australia), provides a good overview of the road Neuralink took to getting FDA (US Food and Drug Administration) approval for human clinical trials in his May 29, 2023 essay for The Conversation, Note: Links have been removed,

Since its founding in 2016, Elon Musk’s neurotechnology company Neuralink has had the ambitious mission to build a next-generation brain implant with at least 100 times more brain connections than devices currently approved by the US Food and Drug Administration (FDA).

The company has now reached a significant milestone, having received FDA approval to begin human trials. So what were the issues keeping the technology in the pre-clinical trial phase for as long as it was? And have these concerns been addressed?

Neuralink is making a Class III medical device known as a brain-computer interface (BCI). The device connects the brain to an external computer via a Bluetooth signal, enabling continuous communication back and forth.

The device itself is a coin-sized unit called a Link. It’s implanted within a small disk-shaped cutout in the skull using a precision surgical robot. The robot splices a thousand tiny threads from the Link to certain neurons in the brain. [emphasis mine] Each thread is about a quarter the diameter of a human hair.

The company says the device could enable precise control of prosthetic limbs, giving amputees natural motor skills. It could revolutionise treatment for conditions such as Parkinson’s disease, epilepsy and spinal cord injuries. It also shows some promise for potential treatment of obesity, autism, depression, schizophrenia and tinnitus.

Several other neurotechnology companies and researchers have already developed BCI technologies that have helped people with limited mobility regain movement and complete daily tasks.

In February 2021, Musk said Neuralink was working with the FDA to secure permission to start initial human trials later that year. But human trials didn’t commence in 2021.

Then, in March 2022, Neuralink made a further application to the FDA to establish its readiness to begin humans trials.

One year and three months later, on May 25 2023, Neuralink finally received FDA approval for its first human clinical trial. Given how hard Neuralink has pushed for permission to begin, we can assume it will begin very soon. [emphasis mine]

The approval has come less than six months after the US Office of the Inspector General launched an investigation into Neuralink over potential animal welfare violations. [emphasis mine]

In accessible language, Tuffley goes on to discuss the FDA’s specific technical issues with implants and how they were addressed in his May 29, 2023 essay.

More about how Neuralink’s implant works and some concerns

Canadian Broadcasting Corporation (CBC) journalist Andrew Chang offers an almost 13 minute video, “Neuralink brain chip’s first human patient. How does it work?” Chang is a little overenthused for my taste but he offers some good information about neural implants, along with informative graphics in his presentation.

So, Tuffley was right about Neuralink getting ready quickly for human clinical trials as you can guess from the title of Chang’s CBC video.

Jennifer Korn announced that recruitment had started in her September 20, 2023 article for CNN (Cable News Network), Note: Links have been removed,

Elon Musk’s controversial biotechnology startup Neuralink opened up recruitment for its first human clinical trial Tuesday, according to a company blog.

After receiving approval from an independent review board, Neuralink is set to begin offering brain implants to paralysis patients as part of the PRIME Study, the company said. PRIME, short for Precise Robotically Implanted Brain-Computer Interface, is being carried out to evaluate both the safety and functionality of the implant.

Trial patients will have a chip surgically placed in the part of the brain that controls the intention to move. The chip, installed by a robot, will then record and send brain signals to an app, with the initial goal being “to grant people the ability to control a computer cursor or keyboard using their thoughts alone,” the company wrote.

Those with quadriplegia [sometimes known as tetraplegia] due to cervical spinal cord injury or amyotrophic lateral sclerosis (ALS) may qualify for the six-year-long study – 18 months of at-home and clinic visits followed by follow-up visits over five years. Interested people can sign up in the patient registry on Neuralink’s website.

Musk has been working on Neuralink’s goal of using implants to connect the human brain to a computer for five years, but the company so far has only tested on animals. The company also faced scrutiny after a monkey died in project testing in 2022 as part of efforts to get the animal to play Pong, one of the first video games.

I mentioned three Reuters investigative journalists who were reporting on Neuralink’s animal abuse allegations (emphasized in Tuffley’s essay) in a July 7, 2023 posting, “Global dialogue on the ethics of neurotechnology on July 13, 2023 led by UNESCO.” Later that year, Neuralink was cleared by the US Department of Agriculture (see September 24,, 2023 article by Mahnoor Jehangir for BNN Breaking).

Plus, Neuralink was being investigated over more allegations according to a February 9, 2023 article by Rachel Levy for Reuters, this time regarding hazardous pathogens,

The U.S. Department of Transportation said on Thursday it is investigating Elon Musk’s brain-implant company Neuralink over the potentially illegal movement of hazardous pathogens.

A Department of Transportation spokesperson told Reuters about the probe after the Physicians Committee of Responsible Medicine (PCRM), an animal-welfare advocacy group,wrote to Secretary of Transportation Pete Buttigieg, opens new tab earlier on Thursday to alert it of records it obtained on the matter.

PCRM said it obtained emails and other documents that suggest unsafe packaging and movement of implants removed from the brains of monkeys. These implants may have carried infectious diseases in violation of federal law, PCRM said.

There’s an update about the hazardous materials in the next section. Spoiler alert, the company got fined.

Neuralink’s first human implant

A January 30, 2024 article (Associated Press with files from Reuters) on the Canadian Broadcasting Corporation’s (CBC) online news webspace heralded the latest about Neurlink’s human clinical trials,

The first human patient received an implant from Elon Musk’s computer-brain interface company Neuralink over the weekend, the billionaire says.

In a post Monday [January 29, 2024] on X, the platform formerly known as Twitter, Musk said that the patient received the implant the day prior and was “recovering well.” He added that “initial results show promising neuron spike detection.”

Spikes are activity by neurons, which the National Institutes of Health describe as cells that use electrical and chemical signals to send information around the brain and to the body.

The billionaire, who owns X and co-founded Neuralink, did not provide additional details about the patient.

When Neuralink announced in September [2023] that it would begin recruiting people, the company said it was searching for individuals with quadriplegia due to cervical spinal cord injury or amyotrophic lateral sclerosis, commonly known as ALS or Lou Gehrig’s disease.

Neuralink reposted Musk’s Monday [January 29, 2024] post on X, but did not publish any additional statements acknowledging the human implant. The company did not immediately respond to requests for comment from The Associated Press or Reuters on Tuesday [January 30, 2024].

In a separate Monday [January 29, 2024] post on X, Musk said that the first Neuralink product is called “Telepathy” — which, he said, will enable users to control their phones or computers “just by thinking.” He said initial users would be those who have lost use of their limbs.

The startup’s PRIME Study is a trial for its wireless brain-computer interface to evaluate the safety of the implant and surgical robot.

Now for the hazardous materials, January 30, 2024 article, Note: A link has been removed,

Earlier this month [January 2024], a Reuters investigation found that Neuralink was fined for violating U.S. Department of Transportation (DOT) rules regarding the movement of hazardous materials. During inspections of the company’s facilities in Texas and California in February 2023, DOT investigators found the company had failed to register itself as a transporter of hazardous material.

They also found improper packaging of hazardous waste, including the flammable liquid Xylene. Xylene can cause headaches, dizziness, confusion, loss of muscle co-ordination and even death, according to the U.S. Centers for Disease Control and Prevention.

The records do not say why Neuralink would need to transport hazardous materials or whether any harm resulted from the violations.

Skeptical thoughts about Elon Musk and Neuralink

Earlier this month (February 2024), the British Broadcasting Corporation (BBC) published an article by health reporters, Jim Reed and Joe McFadden, that highlights the history of brain implants, the possibilities, and notes some of Elon Musk’s more outrageous claims for Neuralink’s brain implants,

Elon Musk is no stranger to bold claims – from his plans to colonise Mars to his dreams of building transport links underneath our biggest cities. This week the world’s richest man said his Neuralink division had successfully implanted its first wireless brain chip into a human.

Is he right when he says this technology could – in the long term – save the human race itself?

Sticking electrodes into brain tissue is really nothing new.

In the 1960s and 70s electrical stimulation was used to trigger or suppress aggressive behaviour in cats. By the early 2000s monkeys were being trained to move a cursor around a computer screen using just their thoughts.

“It’s nothing novel, but implantable technology takes a long time to mature, and reach a stage where companies have all the pieces of the puzzle, and can really start to put them together,” says Anne Vanhoestenberghe, professor of active implantable medical devices, at King’s College London.

Neuralink is one of a growing number of companies and university departments attempting to refine and ultimately commercialise this technology. The focus, at least to start with, is on paralysis and the treatment of complex neurological conditions.

Reed and McFadden’s February 2024 BBC article describes a few of the other brain implant efforts, Note: Links have been removed,

One of its [Neuralink’s] main rivals, a start-up called Synchron backed by funding from investment firms controlled by Bill Gates and Jeff Bezos, has already implanted its stent-like device into 10 patients.

Back in December 2021, Philip O’Keefe, a 62-year old Australian who lives with a form of motor neurone disease, composed the first tweet using just his thoughts to control a cursor.

And researchers at Lausanne University in Switzerland have shown it is possible for a paralysed man to walk again by implanting multiple devices to bypass damage caused by a cycling accident.

In a research paper published this year, they demonstrated a signal could be beamed down from a device in his brain to a second device implanted at the base of his spine, which could then trigger his limbs to move.

Some people living with spinal injuries are sceptical about the sudden interest in this new kind of technology.

“These breakthroughs get announced time and time again and don’t seem to be getting any further along,” says Glyn Hayes, who was paralysed in a motorbike accident in 2017, and now runs public affairs for the Spinal Injuries Association.

If I could have anything back, it wouldn’t be the ability to walk. It would be putting more money into a way of removing nerve pain, for example, or ways to improve bowel, bladder and sexual function.” [emphasis mine]

Musk, however, is focused on something far more grand for Neuralink implants, from Reed and McFadden’s February 2024 BBC article, Note: A link has been removed,

But for Elon Musk, “solving” brain and spinal injuries is just the first step for Neuralink.

The longer-term goal is “human/AI symbiosis” [emphasis mine], something he describes as “species-level important”.

Musk himself has already talked about a future where his device could allow people to communicate with a phone or computer “faster than a speed typist or auctioneer”.

In the past, he has even said saving and replaying memories may be possible, although he recognised “this is sounding increasingly like a Black Mirror episode.”

One of the experts quoted in Reed and McFadden’s February 2024 BBC article asks a pointed question,

… “At the moment, I’m struggling to see an application that a consumer would benefit from, where they would take the risk of invasive surgery,” says Prof Vanhoestenberghe.

“You’ve got to ask yourself, would you risk brain surgery just to be able to order a pizza on your phone?”

Rae Hodge’s February 11, 2024 article about Elon Musk and his hyped up Neuralink implant for Salon is worth reading in its entirety but for those who don’t have the time or need a little persuading, here are a few excerpts, Note 1: This is a warning; Hodge provides more detail about the animal cruelty allegations; Note 2: Links have been removed,

Elon Musk’s controversial brain-computer interface (BCI) tech, Neuralink, has supposedly been implanted in its first recipient — and as much as I want to see progress for treatment of paralysis and neurodegenerative disease, I’m not celebrating. I bet the neuroscientists he reportedly drove out of the company aren’t either, especially not after seeing the gruesome torture of test monkeys and apparent cover-up that paved the way for this moment. 

All of which is an ethics horror show on its own. But the timing of Musk’s overhyped implant announcement gives it an additional insulting subtext. Football players are currently in a battle for their lives against concussion-based brain diseases that plague autopsy reports of former NFL players. And Musk’s boast of false hope came just two weeks before living players take the field in the biggest and most brutal game of the year. [2024 Super Bowl LVIII]

ESPN’s Kevin Seifert reports neuro-damage is up this year as “players suffered a total of 52 concussions from the start of training camp to the beginning of the regular season. The combined total of 213 preseason and regular season concussions was 14% higher than 2021 but within range of the three-year average from 2018 to 2020 (203).”

I’m a big fan of body-tech: pacemakers, 3D-printed hips and prosthetic limbs that allow you to wear your wedding ring again after 17 years. Same for brain chips. But BCI is the slow-moving front of body-tech development for good reason. The brain is too understudied. Consequences of the wrong move are dire. Overpromising marketable results on profit-driven timelines — on the backs of such a small community of researchers in a relatively new field — would be either idiotic or fiendish. 

Brown University’s research in the sector goes back to the 1990s. Since the emergence of a floodgate-opening 2002 study and the first implant in 2004 by med-tech company BrainGate, more promising results have inspired broader investment into careful research. But BrainGate’s clinical trials started back in 2009, and as noted by Business Insider’s Hilary Brueck, are expected to continue until 2038 — with only 15 participants who have devices installed. 

Anne Vanhoestenberghe is a professor of active implantable medical devices at King’s College London. In a recent release, she cautioned against the kind of hype peddled by Musk.

“Whilst there are a few other companies already using their devices in humans and the neuroscience community have made remarkable achievements with those devices, the potential benefits are still significantly limited by technology,” she said. “Developing and validating core technology for long term use in humans takes time and we need more investments to ensure we do the work that will underpin the next generation of BCIs.” 

Neuralink is a metal coin in your head that connects to something as flimsy as an app. And we’ve seen how Elon treats those. We’ve also seen corporate goons steal a veteran’s prosthetic legs — and companies turn brain surgeons and dentists into repo-men by having them yank anti-epilepsy chips out of people’s skulls, and dentures out of their mouths. 

“I think we have a chance with Neuralink to restore full-body functionality to someone who has a spinal cord injury,” Musk said at a 2023 tech summit, adding that the chip could possibly “make up for whatever lost capacity somebody has.”

Maybe BCI can. But only in the careful hands of scientists who don’t have Musk squawking “go faster!” over their shoulders. His greedy frustration with the speed of BCI science is telling, as is the animal cruelty it reportedly prompted.

There have been other examples of Musk’s grandiosity. Notably, David Lee expressed skepticism about hyperloop in his August 13, 2013 article for BBC news online

Is Elon Musk’s Hyperloop just a pipe dream?

Much like the pun in the headline, the bright idea of transporting people using some kind of vacuum-like tube is neither new nor imaginative.

There was Robert Goddard, considered the “father of modern rocket propulsion”, who claimed in 1909 that his vacuum system could suck passengers from Boston to New York at 1,200mph.

And then there were Soviet plans for an amphibious monorail  – mooted in 1934  – in which two long pods would start their journey attached to a metal track before flying off the end and slipping into the water like a two-fingered Kit Kat dropped into some tea.

So ever since inventor and entrepreneur Elon Musk hit the world’s media with his plans for the Hyperloop, a healthy dose of scepticism has been in the air.

“This is by no means a new idea,” says Rod Muttram, formerly of Bombardier Transportation and Railtrack.

“It has been previously suggested as a possible transatlantic transport system. The only novel feature I see is the proposal to put the tubes above existing roads.”

Here’s the latest I’ve found on hyperloop, from the Hyperloop Wikipedia entry,

As of 2024, some companies continued to pursue technology development under the hyperloop moniker, however, one of the biggest, well funded players, Hyperloop One, declared bankruptcy and ceased operations in 2023.[15]

Musk is impatient and impulsive as noted in a September 12, 2023 posting by Mike Masnick on Techdirt, Note: A link has been removed,

The Batshit Crazy Story Of The Day Elon Musk Decided To Personally Rip Servers Out Of A Sacramento Data Center

Back on Christmas Eve [December 24, 2022] of last year there were some reports that Elon Musk was in the process of shutting down Twitter’s Sacramento data center. In that article, a number of ex-Twitter employees were quoted about how much work it would be to do that cleanly, noting that there’s a ton of stuff hardcoded in Twitter code referring to that data center (hold that thought).

That same day, Elon tweeted out that he had “disconnected one of the more sensitive server racks.”

Masnick follows with a story of reckless behaviour from someone who should have known better.

Ethics of implants—where to look for more information

While Musk doesn’t use the term when he describes a “human/AI symbiosis” (presumably by way of a neural implant), he’s talking about a cyborg. Here’s a 2018 paper, which looks at some of the implications,

Do you want to be a cyborg? The moderating effect of ethics on neural implant acceptance by Eva Reinares-Lara, Cristina Olarte-Pascual, and Jorge Pelegrín-Borondo. Computers in Human Behavior Volume 85, August 2018, Pages 43-53 DOI: https://doi.org/10.1016/j.chb.2018.03.032

This paper is open access.

Getting back to Neuralink, I have two blog posts that discuss the company and the ethics of brain implants from way back in 2021.

First, there’s Jazzy Benes’ March 1, 2021 posting on the Santa Clara University’s Markkula Center for Applied Ethics blog. It stands out as it includes a discussion of the disabled community’s issues, Note: Links have been removed,

In the heart of Silicon Valley we are constantly enticed by the newest technological advances. With the big influencers Grimes [a Canadian musician and the mother of three children with Elon Musk] and Lil Uzi Vert publicly announcing their willingness to become experimental subjects for Elon Musk’s Neuralink brain implantation device, we are left wondering if future technology will actually give us “the knowledge of the Gods.” Is it part of the natural order for humans to become omniscient beings? Who will have access to the devices? What other ethical considerations must be discussed before releasing such technology to the public?

A significant issue that arises from developing technologies for the disabled community is the assumption that disabled persons desire the abilities of what some abled individuals may define as “normal.” Individuals with disabilities may object to technologies intended to make them fit an able-bodied norm. “Normal” is relative to each individual, and it could be potentially harmful to use a deficit view of disability, which means judging a disability as a deficiency. However, this is not to say that all disabled individuals will reject a technology that may enhance their abilities. Instead, I believe it is a consideration that must be recognized when developing technologies for the disabled community, and it can only be addressed through communication with disabled persons. As a result, I believe this is a conversation that must be had with the community for whom the technology is developed–disabled persons.

With technologies that aim to address disabilities, we walk a fine line between therapeutics and enhancement. Though not the first neural implant medical device, the Link may have been the first BCI system openly discussed for its potential transhumanism uses, such as “enhanced cognitive abilities, memory storage and retrieval, gaming, telepathy, and even symbiosis with machines.” …

Benes also discusses transhumanism, privacy issues, and consent issues. It’s a thoughtful reading experience.

Second is a July 9, 2021 posting by anonymous on the University of California at Berkeley School of Information blog which provides more insight into privacy and other issues associated with data collection (and introduced me to the concept of decisional interference),

As the development of microchips furthers and advances in neuroscience occur, the possibility for seamless brain-machine interfaces, where a device decodes inputs from the user’s brain to perform functions, becomes more of a reality. These various forms of these technologies already exist. However, technological advances have made implantable and portable devices possible. Imagine a future where humans don’t need to talk to each other, but rather can transmit their thoughts directly to another person. This idea is the eventual goal of Elon Musk, the founder of Neuralink. Currently, Neuralink is one of the main companies involved in the advancement of this type of technology. Analysis of the Neuralink’s technology and their overall mission statement provide an interesting insight into the future of this type of human-computer interface and the potential privacy and ethical concerns with this technology.

As this technology further develops, several privacy and ethical concerns come into question. To begin, using Solove’s Taxonomy as a privacy framework, many areas of potential harm are revealed. In the realm of information collection, there is much risk. Brain-computer interfaces, depending on where they are implanted, could have access to people’s most private thoughts and emotions. This information would need to be transmitted to another device for processing. The collection of this information by companies such as advertisers would represent a major breach of privacy. Additionally, there is risk to the user from information processing. These devices must work concurrently with other devices and often wirelessly. Given the widespread importance of cloud computing in much of today’s technology, offloading information from these devices to the cloud would be likely. Having the data stored in a database puts the user at the risk of secondary use if proper privacy policies are not implemented. The trove of information stored within the information collected from the brain is vast. These datasets could be combined with existing databases such as browsing history on Google to provide third parties with unimaginable context on individuals. Lastly, there is risk for information dissemination, more specifically, exposure. The information collected and processed by these devices would need to be stored digitally. Keeping such private information, even if anonymized, would be a huge potential for harm, as the contents of the information may in itself be re-identifiable to a specific individual. Lastly there is risk for invasions such as decisional interference. Brain-machine interfaces would not only be able to read information in the brain but also write information. This would allow the device to make potential emotional changes in its users, which be a major example of decisional interference. …

For the most recent Neuralink and brain implant ethics piece, there’s this February 14, 2024 essay on The Conversation, which, unusually, for this publication was solicited by the editors, Note: Links have been removed,

In January 2024, Musk announced that Neuralink implanted its first chip in a human subject’s brain. The Conversation reached out to two scholars at the University of Washington School of Medicine – Nancy Jecker, a bioethicst, and Andrew Ko, a neurosurgeon who implants brain chip devices – for their thoughts on the ethics of this new horizon in neuroscience.

Information about the implant, however, is scarce, aside from a brochure aimed at recruiting trial subjects. Neuralink did not register at ClinicalTrials.gov, as is customary, and required by some academic journals. [all emphases mine]

Some scientists are troubled by this lack of transparency. Sharing information about clinical trials is important because it helps other investigators learn about areas related to their research and can improve patient care. Academic journals can also be biased toward positive results, preventing researchers from learning from unsuccessful experiments.

Fellows at the Hastings Center, a bioethics think tank, have warned that Musk’s brand of “science by press release, while increasingly common, is not science. [emphases mine]” They advise against relying on someone with a huge financial stake in a research outcome to function as the sole source of information.

When scientific research is funded by government agencies or philanthropic groups, its aim is to promote the public good. Neuralink, on the other hand, embodies a private equity model [emphasis mine], which is becoming more common in science. Firms pooling funds from private investors to back science breakthroughs may strive to do good, but they also strive to maximize profits, which can conflict with patients’ best interests.

In 2022, the U.S. Department of Agriculture investigated animal cruelty at Neuralink, according to a Reuters report, after employees accused the company of rushing tests and botching procedures on test animals in a race for results. The agency’s inspection found no breaches, according to a letter from the USDA secretary to lawmakers, which Reuters reviewed. However, the secretary did note an “adverse surgical event” in 2019 that Neuralink had self-reported.

In a separate incident also reported by Reuters, the Department of Transportation fined Neuralink for violating rules about transporting hazardous materials, including a flammable liquid.

…the possibility that the device could be increasingly shown to be helpful for people with disabilities, but become unavailable due to loss of research funding. For patients whose access to a device is tied to a research study, the prospect of losing access after the study ends can be devastating. [emphasis mine] This raises thorny questions about whether it is ever ethical to provide early access to breakthrough medical interventions prior to their receiving full FDA approval.

Not registering a clinical trial would seem to suggest there won’t be much oversight. As for Musk’s “science by press release” activities, I hope those will be treated with more skepticism by mainstream media although that seems unlikely given the current situation with journalism (more about that in a future post).

As for the issues associated with private equity models for science research and the problem of losing access to devices after a clinical trial is ended, my April 5, 2022 posting, “Going blind when your neural implant company flirts with bankruptcy (long read)” offers some cautionary tales, in addition to being the most comprehensive piece I’ve published on ethics and brain implants.

My July 17, 2023 posting, “Unveiling the Neurotechnology Landscape: Scientific Advancements, Innovations and Major Trends—a UNESCO report” offers a brief overview of the international scene.

BrainGate demonstrates a high-bandwidth wireless brain-computer interface (BCI)

I wrote about some brain computer interface (BCI) work out of Stanford University (California, US), in a Sept. 17, 2020 posting (Turning brain-controlled wireless electronic prostheses into reality plus some ethical points), which may have contributed to what is now the first demonstration of a wireless brain-computer interface for people with tetraplegia (also known as quadriplegia).

From an April 1, 2021 news item on ScienceDaily,

In an important step toward a fully implantable intracortical brain-computer interface system, BrainGate researchers demonstrated human use of a wireless transmitter capable of delivering high-bandwidth neural signals.

Brain-computer interfaces (BCIs) are an emerging assistive technology, enabling people with paralysis to type on computer screens or manipulate robotic prostheses just by thinking about moving their own bodies. For years, investigational BCIs used in clinical trials have required cables to connect the sensing array in the brain to computers that decode the signals and use them to drive external devices.

Now, for the first time, BrainGate clinical trial participants with tetraplegia have demonstrated use of an intracortical wireless BCI with an external wireless transmitter. The system is capable of transmitting brain signals at single-neuron resolution and in full broadband fidelity without physically tethering the user to a decoding system. The traditional cables are replaced by a small transmitter about 2 inches in its largest dimension and weighing a little over 1.5 ounces. The unit sits on top of a user’s head and connects to an electrode array within the brain’s motor cortex using the same port used by wired systems.

For a study published in IEEE Transactions on Biomedical Engineering, two clinical trial participants with paralysis used the BrainGate system with a wireless transmitter to point, click and type on a standard tablet computer. The study showed that the wireless system transmitted signals with virtually the same fidelity as wired systems, and participants achieved similar point-and-click accuracy and typing speeds.

A March 31, 2021 Brown University news release (also on EurekAlert but published April 1, 2021), which originated the news item, provides more detail,

“We’ve demonstrated that this wireless system is functionally equivalent to the wired systems that have been the gold standard in BCI performance for years,” said John Simeral, an assistant professor of engineering (research) at Brown University, a member of the BrainGate research consortium and the study’s lead author. “The signals are recorded and transmitted with appropriately similar fidelity, which means we can use the same decoding algorithms we used with wired equipment. The only difference is that people no longer need to be physically tethered to our equipment, which opens up new possibilities in terms of how the system can be used.”

The researchers say the study represents an early but important step toward a major objective in BCI research: a fully implantable intracortical system that aids in restoring independence for people who have lost the ability to move. While wireless devices with lower bandwidth have been reported previously, this is the first device to transmit the full spectrum of signals recorded by an intracortical sensor. That high-broadband wireless signal enables clinical research and basic human neuroscience that is much more difficult to perform with wired BCIs.

The new study demonstrated some of those new possibilities. The trial participants — a 35-year-old man and a 63-year-old man, both paralyzed by spinal cord injuries — were able to use the system in their homes, as opposed to the lab setting where most BCI research takes place. Unencumbered by cables, the participants were able to use the BCI continuously for up to 24 hours, giving the researchers long-duration data including while participants slept.

“We want to understand how neural signals evolve over time,” said Leigh Hochberg, an engineering professor at Brown, a researcher at Brown’s Carney Institute for Brain Science and leader of the BrainGate clinical trial. “With this system, we’re able to look at brain activity, at home, over long periods in a way that was nearly impossible before. This will help us to design decoding algorithms that provide for the seamless, intuitive, reliable restoration of communication and mobility for people with paralysis.”

The device used in the study was first developed at Brown in the lab of Arto Nurmikko, a professor in Brown’s School of Engineering. Dubbed the Brown Wireless Device (BWD), it was designed to transmit high-fidelity signals while drawing minimal power. In the current study, two devices used together recorded neural signals at 48 megabits per second from 200 electrodes with a battery life of over 36 hours.

While the BWD has been used successfully for several years in basic neuroscience research, additional testing and regulatory permission were required prior to using the system in the BrainGate trial. Nurmikko says the step to human use marks a key moment in the development of BCI technology.

“I am privileged to be part of a team pushing the frontiers of brain-machine interfaces for human use,” Nurmikko said. “Importantly, the wireless technology described in our paper has helped us to gain crucial insight for the road ahead in pursuit of next generation of neurotechnologies, such as fully implanted high-density wireless electronic interfaces for the brain.”

The new study marks another significant advance by researchers with the BrainGate consortium, an interdisciplinary group of researchers from Brown, Stanford and Case Western Reserve universities, as well as the Providence Veterans Affairs Medical Center and Massachusetts General Hospital. In 2012, the team published landmark research in which clinical trial participants were able, for the first time, to operate multidimensional robotic prosthetics using a BCI. That work has been followed by a steady stream of refinements to the system, as well as new clinical breakthroughs that have enabled people to type on computers, use tablet apps and even move their own paralyzed limbs.

“The evolution of intracortical BCIs from requiring a wire cable to instead using a miniature wireless transmitter is a major step toward functional use of fully implanted, high-performance neural interfaces,” said study co-author Sharlene Flesher, who was a postdoctoral fellow at Stanford and is now a hardware engineer at Apple. “As the field heads toward reducing transmitted bandwidth while preserving the accuracy of assistive device control, this study may be one of few that captures the full breadth of cortical signals for extended periods of time, including during practical BCI use.”

The new wireless technology is already paying dividends in unexpected ways, the researchers say. Because participants are able to use the wireless device in their homes without a technician on hand to maintain the wired connection, the BrainGate team has been able to continue their work during the COVID-19 pandemic.

“In March 2020, it became clear that we would not be able to visit our research participants’ homes,” said Hochberg, who is also a critical care neurologist at Massachusetts General Hospital and director of the V.A. Rehabilitation Research and Development Center for Neurorestoration and Neurotechnology. “But by training caregivers how to establish the wireless connection, a trial participant was able to use the BCI without members of our team physically being there. So not only were we able to continue our research, this technology allowed us to continue with the full bandwidth and fidelity that we had before.”

Simeral noted that, “Multiple companies have wonderfully entered the BCI field, and some have already demonstrated human use of low-bandwidth wireless systems, including some that are fully implanted. In this report, we’re excited to have used a high-bandwidth wireless system that advances the scientific and clinical capabilities for future systems.”

Brown has a licensing agreement with Blackrock Microsystems to make the device available to neuroscience researchers around the world. The BrainGate team plans to continue to use the device in ongoing clinical trials.

Here’s a link to and a citation for the paper,

Home Use of a Percutaneous Wireless Intracortical Brain-Computer Interface by Individuals With Tetraplegia by John D Simeral, Thomas Hosman, Jad Saab, Sharlene N Flesher, Marco Vilela, Brian Franco, Jessica Kelemen, David M Brandman, John G Ciancibello, Paymon G Rezaii, Emad N. Eskandar, David M Rosler, Krishna V Shenoy, Jaimie M. Henderson, Arto V Nurmikko, Leigh R. Hochberg. IEEE Transactions on Biomedical Engineering, 2021; 1 DOI: 10.1109/TBME.2021.3069119 Date of Publication: 30 March 2021

This paper is open access.

If you don’t happen to be familiar with the IEEE, it’s the Institute of Electrical and Electronics Engineers. BrainGate can be found here, and Blackrock Microsystems can be found here.

The first story here to feature BrainGate was in a May 17, 2012 posting. (Unfortunately, the video featuring a participant picking up a cup of coffee is no longer embedded in the post.) There’s also an October 31, 2016 posting and an April 24, 2017 posting, both of which mention BrainGate. As for my Sept. 17, 2020 posting (Turning brain-controlled wireless electronic prostheses into reality plus some ethical points), you may want to look at those ethical points.

New boron nanostructure—carbon, watch out!

Carbon nanotubes, buckminsterfullerenes (also known as, buckyballs), and/or graphene are names for different carbon nanoscale structures and, as far as I’m aware,carbon is the only element that merits some distinct names at the nanoscale. By comparison, gold can be gold nanorods, gold nanostars, gold nanoparticles, and so on. In short, nanostructures made of gold (and most other elements) are always prefaced with the word ‘gold’ followed by a word with ‘nano’ in it.

Scientists naming a new boron nanoscale structure seem to have adopted both strategies for a hybrid name. Here’s more from a June 25, 2020 news item on phys.org,

The discovery of carbon nanostructures like two-dimensional graphene and soccer ball-shaped buckyballs helped to launch a nanotechnology revolution. In recent years, researchers from Brown University [located in Rhode Island, US] and elsewhere have shown that boron, carbon’s neighbor on the periodic table, can make interesting nanostructures too, including two-dimensional borophene and a buckyball-like hollow cage structure called borospherene.

Caption: The family of boron-based nanostructures has a new member: metallo-borospherenes, hollow cages made from 18 boron atoms and three atoms of lanthanide elements. Credit: Wang Lab / Brown University

A June 25, 2020 Brown University news release (also on EurekAlert), wbich originated the news item, describes these new structures in detail,

Now, researchers from Brown and Tsinghua University have added another boron nanostructure to the list. In a paper published in Nature Communications, they show that clusters of 18 boron atoms and three atoms of lanthanide elements form a bizarre cage-like structure unlike anything they’ve ever seen.

“This is just not a type of structure you expect to see in chemistry,” said Lai-Sheng Wang, a professor of chemistry at Brown and the study’s senior author. “When we wrote the paper we really struggled to describe it. It’s basically a spherical trihedron. Normally you can’t have a closed three-dimensional structure with only three sides, but since it’s spherical, it works.”

The researchers are hopeful that the nanostructure may shed light on the bulk structure and chemical bonding behavior of boron lanthanides, an important class of materials widely used in electronics and other applications. The nanostructure by itself may have interesting properties as well, the researchers say.

“Lanthanide elements are important magnetic materials, each with very different magnetic moments,” Wang said. “We think any of the lanthanides will make this structure, so they could have very interesting magnetic properties.”

Wang and his students created the lanthanide-boron clusters by focusing a powerful laser onto a solid target made of a mixture of boron and a lanthanide element. The clusters are formed upon cooling of the vaporized atoms. Then they used a technique called photoelectron spectroscopy to study the electronic properties of the clusters. The technique involves zapping clusters of atoms with another high-powered laser. Each zap knocks an electron out of the cluster. By measuring the kinetic energies of those freed electrons, researchers can create a spectrum of binding energies for the electrons that bond the cluster together.

“When we see a simple, beautiful spectrum, we know there’s a beautiful structure behind it,” Wang said.

To figure out what that structure looks like, Wang compared the photoelectron spectra with theoretical calculations done by Professor Jun Li and his students from Tsinghua. Once they find a theoretical structure with a binding spectrum that matches the experiment, they know they’ve found the right structure.

“This structure was something we never would have predicted,” Wang said. “That’s the value of combining theoretical calculation with experimental data.”

Wang and his colleagues have dubbed the new structures metallo-borospherenes, and they’re hopeful that further research will reveal their properties.

Here’s a link to and a citation for the paper,

Spherical trihedral metallo-borospherenes by Teng-Teng Chen, Wan-Lu Li, Wei-Jia Chen, Xiao-Hu Yu, Xin-Ran Dong, Jun Li & Lai-Sheng Wang. Nature Communications volume 11, Article number: 2766 (2020) DOI: https://doi.org/10.1038/s41467-020-16532-x Published: 02 June 2020

This paper is open access.

The physics of Jackson Pollock’s painting technique

I long ago stumbled across the fascination that Jackson Pollock’s art work exerts over physicists but this work from Brown University adds some colours to the picture (wordplay intended).

One: Number 31, 1950. Jackson Pollock (American, 1912–1956). 1950. Oil and enamel paint on canvas, 8′ 10″ x 17′ 5 5/8″ (269.5 x 530.8 cm) Courtesy: Museum of Modern Art (MOMA) [downloaded from: https://www.moma.org/learn/moma_learning/jackson-pollock-one-number-31-1950-1950/]

From an October 30, 2019 Brown University news release (also on EurekAlert),

The celebrated painter Jackson Pollock created his most iconic works not with a brush, but by pouring paint onto the canvas from above, weaving sinuous filaments of color into abstract masterpieces. A team of researchers analyzing the physics of Pollock’s technique has shown that the artist had a keen understanding of a classic phenomenon in fluid dynamics — whether he was aware of it or not.

In a paper published in the journal PLOS ONE, the researchers show that Pollock’s technique seems to intentionally avoid what’s known as coiling instability — the tendency of a viscous fluid to form curls and coils when poured on a surface.

“Like most painters, Jackson Pollock went through a long process of experimentation in order to perfect his technique,” said Roberto Zenit, a professor in Brown’s School of Engineering and senior author on the paper. “What we were trying to do with this research is figure out what conclusions Pollock reached in order to execute his paintings the way he wanted. Our main finding in this paper was that Pollock’s movements and the properties of his paints were such he avoided this coiling instability.”

Pollock’s technique typically involved pouring paint straight from a can or along a stick onto a canvas lying horizontally on the floor. It’s often referred to as the “drip technique,” but that’s a bit of a misnomer in the parlance of fluid mechanics, Zenit says. In fluid mechanics, “dripping” would be dispensing the fluid in a way that makes discrete droplets on the canvas. Pollock largely avoided droplets, in favor of unbroken filaments of paint stretching across the canvas.

In order to understand exactly how the technique worked, Zenit and colleagues from the Universidad Nacional Autonoma de Mexico analyzed extensive video of Pollock at work, taking careful measure of how fast he moved and how far from the canvas he poured his paints. Having gathered data on how Pollock worked, the researchers used an experimental setup to recreate his technique. Using the setup, the researchers could deposit paint using a syringe mounted at varying heights onto a canvas moving at varying speeds. The experiments helped to zero in on the most important aspects of what Pollock was doing.

“We can vary one thing at a time so we can decipher the key elements of the technique,” Zenit said. “For example, we could vary the height from which the paint is poured and keep the speed constant to see how that changes things.”

The researchers found that the combination of Pollock’s hand speed, the distance he maintained from the canvas and the viscosity of his paint seem to be aimed at avoiding coiling instability. Anyone who’s ever poured a viscous fluid — perhaps some honey on toast — has likely seen some coiling instability. When a small amount of a viscous fluid is poured, it tends to stack up like a coil of rope before oozing across the surface.

In the context of Pollock’s technique, the instability can result in paint filaments making pigtail-like curls when poured from the can. Some prior research had concluded that that the curved lines in Pollock’s paintings were a result of this instability, but this latest research shows the opposite.

“What we found is that he moved his hand at a sufficiently high speed and a sufficiently short height such that this coiling would not occur,” Zenit said.

Zenit says the findings could be useful in authenticating Pollock’s works. Too many tight curls might suggest that a drip-style painting is not a Pollock. The work could also inform other settings in which viscous fluids are stretched into filaments, such as the manufacture of fiber optics. But Zenit says his main interest in the work is that it’s simply a fascinating way to explore interesting questions in fluid mechanics.

“I consider myself to be a fluid mechanics messenger,” he said. “This is my excuse to talk science. It’s fascinating to see that painters are really fluid mechanicians, even though they may not know it.”

Here’s a link to and a citation for the paper,

Pollock avoided hydrodynamic instabilities to paint with his dripping technique by Bernardo Palacios, Alfonso Rosario, Monica M. Wilhelmus, Sandra Zetina, Roberto Zenit. PLOS ONE DOI: https://doi.org/10.1371/journal.pone.0223706 Published: October 30, 2019

This paper is open access.

I could not find any videos related to this research that I know how to embed but Palacios, Zetina, and Zenit have investigated Polock’s ‘physics’ before,

If you want to see Pollock dripping his painting in action, there’s a 10 min. 13 secs. film made in 1950 (Note: Links have been removed from text; link to 10 min. film is below),

In the summer of 1950, Hans Namuth approached Jackson Pollock and asked the abstract expressionist painter if he could photograph him in his studio, working with his “drip” technique of painting. When Namuth arrived, he found:

“A dripping wet canvas covered the entire floor. Blinding shafts of sunlight hit the wet canvas, making its surface hard to see. There was complete silence…. Pollock looked at the painting. Then unexpectedly, he picked up can and paintbrush and started to move around the canvas. It was as if he suddenly realized the painting was not finished. His movements, slow at first, gradually became faster and more dancelike as he flung black, white and rust-colored paint onto the canvas.”

The images from this shoot “helped transform Pollock from a talented, cranky loner into the first media-driven superstar of American contemporary art, the jeans-clad, chain-smoking poster boy of abstract expressionism,” one critic later wrote in The Washington Post.

You can find the film and accompanying Open Culture text intact with links here.

Smartphone as augmented reality system with software from Brown University

You need to see this,

Amazing, eh? The researchers are scheduled to present this work sometime this week at the ACM Symposium on User Interface Software and Technology (UIST) being held in New Orleans, US, from October 20-23, 2019.

Here’s more about ‘Portal-ble’ in an October 16, 2019 news item on ScienceDaily,

A new software system developed by Brown University [US] researchers turns cell phones into augmented reality portals, enabling users to place virtual building blocks, furniture and other objects into real-world backdrops, and use their hands to manipulate those objects as if they were really there.

The developers hope the new system, called Portal-ble, could be a tool for artists, designers, game developers and others to experiment with augmented reality (AR). The team will present the work later this month at the ACM Symposium on User Interface Software and Technology (UIST 2019) in New Orleans. The source code for Andriod is freely available for download on the researchers’ website, and iPhone code will follow soon.

“AR is going to be a great new mode of interaction,” said Jeff Huang, an assistant professor of computer science at Brown who developed the system with his students. “We wanted to make something that made AR portable so that people could use anywhere without any bulky headsets. We also wanted people to be able to interact with the virtual world in a natural way using their hands.”

An October 16, 2019 Brown University news release (also on EurekAlert), which originated the news item, provides more detail,

Huang said the idea for Portal-ble’s “hands-on” interaction grew out of some frustration with AR apps like Pokemon GO. AR apps use smartphones to place virtual objects (like Pokemon characters) into real-world scenes, but interacting with those objects requires users to swipe on the screen.

“Swiping just wasn’t a satisfying way of interacting,” Huang said. “In the real world, we interact with objects with our hands. We turn doorknobs, pick things up and throw things. So we thought manipulating virtual objects by hand would be much more powerful than swiping. That’s what’s different about Portal-ble.”

The platform makes use of a small infrared sensor mounted on the back of a phone. The sensor tracks the position of people’s hands in relation to virtual objects, enabling users to pick objects up, turn them, stack them or drop them. It also lets people use their hands to virtually “paint” onto real-world backdrops. As a demonstration, Huang and his students used the system to paint a virtual garden into a green space on Brown’s College Hill campus.

Huang says the main technical contribution of the work was developing the right accommodations and feedback tools to enable people to interact intuitively with virtual objects.

“It turns out that picking up a virtual object is really hard if you try to apply real-world physics,” Huang said. “People try to grab in the wrong place, or they put their fingers through the objects. So we had to observe how people tried to interact with these objects and then make our system able accommodate those tendencies.”

To do that, Huang enlisted students in a class he was teaching to come up with tasks they might want to do in the AR world — stacking a set of blocks, for example. The students then asked other people to try performing those tasks using Portal-ble, while recording what people were able to do and what they couldn’t. They could then adjust the system’s physics and user interface to make interactions more successful.

“It’s a little like what happens when people draw lines in Photoshop,” Huang said. “The lines people draw are never perfect, but the program can smooth them out and make them perfectly straight. Those were the kinds of accommodations we were trying to make with these virtual objects.”

The team also added sensory feedback — visual highlights on objects and phone vibrations — to make interactions easier. Huang said he was somewhat surprised that phone vibrations helped users to interact. Users feel the vibrations in the hand they’re using to hold the phone, not in the hand that’s actually grabbing for the virtual object. Still, Huang said, vibration feedback still helped users to more successfully interact with objects.

In follow-up studies, users reported that the accommodations and feedback used by the system made tasks significantly easier, less time-consuming and more satisfying.

Huang and his students plan to continue working with Portal-ble — expanding its object library, refining interactions and developing new activities. They also hope to streamline the system to make it run entirely on a phone. Currently the infrared sensor requires an infrared sensor and external compute stick for extra processing power.

Huang hopes people will download the freely available source code and try it for themselves. 
“We really just want to put this out there and see what people do with it,” he said. “The code is on our website for people to download, edit and build off of. It will be interesting to see what people do with it.

Co-authors on the research paper were Jing Qian, Jiaju Ma, Xiangyu Li, Benjamin Attal, Haoming Lai, James Tompkin and John Hughes. The work was supported by the National Science Foundation (IIS-1552663) and by a gift from Pixar.

You can find the conference paper here on jeffhuang.com,

Portal-ble: Intuitive Free-hand Manipulationin Unbounded Smartphone-based Augmented Reality by Jing Qian, Jiaju Ma, Xiangyu Li∗, Benjamin Attal, Haoming Lai,James Tompkin, John F. Hughes, Jeff Huang. Brown University, Providence RI, USA; Southeast University, Nanjing, China. Presented at ACM Symposium on User Interface Software and Technology (UIST) being held in New Orleans, US

This is the first time I’ve seen an augmented reality system that seems accessible, i.e., affordable. You can find out more on the Portal-ble ‘resource’ page where you’ll also find a link to the source code repository. The researchers, as noted in the news release, have an Android version available now with an iPhone version to be released in the future.