Tag Archives: University of Washington (state)

Neural (brain) implants and hype (long read)

There was a big splash a few weeks ago when it was announced that Neuralink’s (Elon Musk company) brain implant had been surgically inserted into its first human patient.

Getting approval

David Tuffley, senior lecturer in Applied Ethics & CyberSecurity at Griffith University (Australia), provides a good overview of the road Neuralink took to getting FDA (US Food and Drug Administration) approval for human clinical trials in his May 29, 2023 essay for The Conversation, Note: Links have been removed,

Since its founding in 2016, Elon Musk’s neurotechnology company Neuralink has had the ambitious mission to build a next-generation brain implant with at least 100 times more brain connections than devices currently approved by the US Food and Drug Administration (FDA).

The company has now reached a significant milestone, having received FDA approval to begin human trials. So what were the issues keeping the technology in the pre-clinical trial phase for as long as it was? And have these concerns been addressed?

Neuralink is making a Class III medical device known as a brain-computer interface (BCI). The device connects the brain to an external computer via a Bluetooth signal, enabling continuous communication back and forth.

The device itself is a coin-sized unit called a Link. It’s implanted within a small disk-shaped cutout in the skull using a precision surgical robot. The robot splices a thousand tiny threads from the Link to certain neurons in the brain. [emphasis mine] Each thread is about a quarter the diameter of a human hair.

The company says the device could enable precise control of prosthetic limbs, giving amputees natural motor skills. It could revolutionise treatment for conditions such as Parkinson’s disease, epilepsy and spinal cord injuries. It also shows some promise for potential treatment of obesity, autism, depression, schizophrenia and tinnitus.

Several other neurotechnology companies and researchers have already developed BCI technologies that have helped people with limited mobility regain movement and complete daily tasks.

In February 2021, Musk said Neuralink was working with the FDA to secure permission to start initial human trials later that year. But human trials didn’t commence in 2021.

Then, in March 2022, Neuralink made a further application to the FDA to establish its readiness to begin humans trials.

One year and three months later, on May 25 2023, Neuralink finally received FDA approval for its first human clinical trial. Given how hard Neuralink has pushed for permission to begin, we can assume it will begin very soon. [emphasis mine]

The approval has come less than six months after the US Office of the Inspector General launched an investigation into Neuralink over potential animal welfare violations. [emphasis mine]

In accessible language, Tuffley goes on to discuss the FDA’s specific technical issues with implants and how they were addressed in his May 29, 2023 essay.

More about how Neuralink’s implant works and some concerns

Canadian Broadcasting Corporation (CBC) journalist Andrew Chang offers an almost 13 minute video, “Neuralink brain chip’s first human patient. How does it work?” Chang is a little overenthused for my taste but he offers some good information about neural implants, along with informative graphics in his presentation.

So, Tuffley was right about Neuralink getting ready quickly for human clinical trials as you can guess from the title of Chang’s CBC video.

Jennifer Korn announced that recruitment had started in her September 20, 2023 article for CNN (Cable News Network), Note: Links have been removed,

Elon Musk’s controversial biotechnology startup Neuralink opened up recruitment for its first human clinical trial Tuesday, according to a company blog.

After receiving approval from an independent review board, Neuralink is set to begin offering brain implants to paralysis patients as part of the PRIME Study, the company said. PRIME, short for Precise Robotically Implanted Brain-Computer Interface, is being carried out to evaluate both the safety and functionality of the implant.

Trial patients will have a chip surgically placed in the part of the brain that controls the intention to move. The chip, installed by a robot, will then record and send brain signals to an app, with the initial goal being “to grant people the ability to control a computer cursor or keyboard using their thoughts alone,” the company wrote.

Those with quadriplegia [sometimes known as tetraplegia] due to cervical spinal cord injury or amyotrophic lateral sclerosis (ALS) may qualify for the six-year-long study – 18 months of at-home and clinic visits followed by follow-up visits over five years. Interested people can sign up in the patient registry on Neuralink’s website.

Musk has been working on Neuralink’s goal of using implants to connect the human brain to a computer for five years, but the company so far has only tested on animals. The company also faced scrutiny after a monkey died in project testing in 2022 as part of efforts to get the animal to play Pong, one of the first video games.

I mentioned three Reuters investigative journalists who were reporting on Neuralink’s animal abuse allegations (emphasized in Tuffley’s essay) in a July 7, 2023 posting, “Global dialogue on the ethics of neurotechnology on July 13, 2023 led by UNESCO.” Later that year, Neuralink was cleared by the US Department of Agriculture (see September 24,, 2023 article by Mahnoor Jehangir for BNN Breaking).

Plus, Neuralink was being investigated over more allegations according to a February 9, 2023 article by Rachel Levy for Reuters, this time regarding hazardous pathogens,

The U.S. Department of Transportation said on Thursday it is investigating Elon Musk’s brain-implant company Neuralink over the potentially illegal movement of hazardous pathogens.

A Department of Transportation spokesperson told Reuters about the probe after the Physicians Committee of Responsible Medicine (PCRM), an animal-welfare advocacy group,wrote to Secretary of Transportation Pete Buttigieg, opens new tab earlier on Thursday to alert it of records it obtained on the matter.

PCRM said it obtained emails and other documents that suggest unsafe packaging and movement of implants removed from the brains of monkeys. These implants may have carried infectious diseases in violation of federal law, PCRM said.

There’s an update about the hazardous materials in the next section. Spoiler alert, the company got fined.

Neuralink’s first human implant

A January 30, 2024 article (Associated Press with files from Reuters) on the Canadian Broadcasting Corporation’s (CBC) online news webspace heralded the latest about Neurlink’s human clinical trials,

The first human patient received an implant from Elon Musk’s computer-brain interface company Neuralink over the weekend, the billionaire says.

In a post Monday [January 29, 2024] on X, the platform formerly known as Twitter, Musk said that the patient received the implant the day prior and was “recovering well.” He added that “initial results show promising neuron spike detection.”

Spikes are activity by neurons, which the National Institutes of Health describe as cells that use electrical and chemical signals to send information around the brain and to the body.

The billionaire, who owns X and co-founded Neuralink, did not provide additional details about the patient.

When Neuralink announced in September [2023] that it would begin recruiting people, the company said it was searching for individuals with quadriplegia due to cervical spinal cord injury or amyotrophic lateral sclerosis, commonly known as ALS or Lou Gehrig’s disease.

Neuralink reposted Musk’s Monday [January 29, 2024] post on X, but did not publish any additional statements acknowledging the human implant. The company did not immediately respond to requests for comment from The Associated Press or Reuters on Tuesday [January 30, 2024].

In a separate Monday [January 29, 2024] post on X, Musk said that the first Neuralink product is called “Telepathy” — which, he said, will enable users to control their phones or computers “just by thinking.” He said initial users would be those who have lost use of their limbs.

The startup’s PRIME Study is a trial for its wireless brain-computer interface to evaluate the safety of the implant and surgical robot.

Now for the hazardous materials, January 30, 2024 article, Note: A link has been removed,

Earlier this month [January 2024], a Reuters investigation found that Neuralink was fined for violating U.S. Department of Transportation (DOT) rules regarding the movement of hazardous materials. During inspections of the company’s facilities in Texas and California in February 2023, DOT investigators found the company had failed to register itself as a transporter of hazardous material.

They also found improper packaging of hazardous waste, including the flammable liquid Xylene. Xylene can cause headaches, dizziness, confusion, loss of muscle co-ordination and even death, according to the U.S. Centers for Disease Control and Prevention.

The records do not say why Neuralink would need to transport hazardous materials or whether any harm resulted from the violations.

Skeptical thoughts about Elon Musk and Neuralink

Earlier this month (February 2024), the British Broadcasting Corporation (BBC) published an article by health reporters, Jim Reed and Joe McFadden, that highlights the history of brain implants, the possibilities, and notes some of Elon Musk’s more outrageous claims for Neuralink’s brain implants,

Elon Musk is no stranger to bold claims – from his plans to colonise Mars to his dreams of building transport links underneath our biggest cities. This week the world’s richest man said his Neuralink division had successfully implanted its first wireless brain chip into a human.

Is he right when he says this technology could – in the long term – save the human race itself?

Sticking electrodes into brain tissue is really nothing new.

In the 1960s and 70s electrical stimulation was used to trigger or suppress aggressive behaviour in cats. By the early 2000s monkeys were being trained to move a cursor around a computer screen using just their thoughts.

“It’s nothing novel, but implantable technology takes a long time to mature, and reach a stage where companies have all the pieces of the puzzle, and can really start to put them together,” says Anne Vanhoestenberghe, professor of active implantable medical devices, at King’s College London.

Neuralink is one of a growing number of companies and university departments attempting to refine and ultimately commercialise this technology. The focus, at least to start with, is on paralysis and the treatment of complex neurological conditions.

Reed and McFadden’s February 2024 BBC article describes a few of the other brain implant efforts, Note: Links have been removed,

One of its [Neuralink’s] main rivals, a start-up called Synchron backed by funding from investment firms controlled by Bill Gates and Jeff Bezos, has already implanted its stent-like device into 10 patients.

Back in December 2021, Philip O’Keefe, a 62-year old Australian who lives with a form of motor neurone disease, composed the first tweet using just his thoughts to control a cursor.

And researchers at Lausanne University in Switzerland have shown it is possible for a paralysed man to walk again by implanting multiple devices to bypass damage caused by a cycling accident.

In a research paper published this year, they demonstrated a signal could be beamed down from a device in his brain to a second device implanted at the base of his spine, which could then trigger his limbs to move.

Some people living with spinal injuries are sceptical about the sudden interest in this new kind of technology.

“These breakthroughs get announced time and time again and don’t seem to be getting any further along,” says Glyn Hayes, who was paralysed in a motorbike accident in 2017, and now runs public affairs for the Spinal Injuries Association.

If I could have anything back, it wouldn’t be the ability to walk. It would be putting more money into a way of removing nerve pain, for example, or ways to improve bowel, bladder and sexual function.” [emphasis mine]

Musk, however, is focused on something far more grand for Neuralink implants, from Reed and McFadden’s February 2024 BBC article, Note: A link has been removed,

But for Elon Musk, “solving” brain and spinal injuries is just the first step for Neuralink.

The longer-term goal is “human/AI symbiosis” [emphasis mine], something he describes as “species-level important”.

Musk himself has already talked about a future where his device could allow people to communicate with a phone or computer “faster than a speed typist or auctioneer”.

In the past, he has even said saving and replaying memories may be possible, although he recognised “this is sounding increasingly like a Black Mirror episode.”

One of the experts quoted in Reed and McFadden’s February 2024 BBC article asks a pointed question,

… “At the moment, I’m struggling to see an application that a consumer would benefit from, where they would take the risk of invasive surgery,” says Prof Vanhoestenberghe.

“You’ve got to ask yourself, would you risk brain surgery just to be able to order a pizza on your phone?”

Rae Hodge’s February 11, 2024 article about Elon Musk and his hyped up Neuralink implant for Salon is worth reading in its entirety but for those who don’t have the time or need a little persuading, here are a few excerpts, Note 1: This is a warning; Hodge provides more detail about the animal cruelty allegations; Note 2: Links have been removed,

Elon Musk’s controversial brain-computer interface (BCI) tech, Neuralink, has supposedly been implanted in its first recipient — and as much as I want to see progress for treatment of paralysis and neurodegenerative disease, I’m not celebrating. I bet the neuroscientists he reportedly drove out of the company aren’t either, especially not after seeing the gruesome torture of test monkeys and apparent cover-up that paved the way for this moment. 

All of which is an ethics horror show on its own. But the timing of Musk’s overhyped implant announcement gives it an additional insulting subtext. Football players are currently in a battle for their lives against concussion-based brain diseases that plague autopsy reports of former NFL players. And Musk’s boast of false hope came just two weeks before living players take the field in the biggest and most brutal game of the year. [2024 Super Bowl LVIII]

ESPN’s Kevin Seifert reports neuro-damage is up this year as “players suffered a total of 52 concussions from the start of training camp to the beginning of the regular season. The combined total of 213 preseason and regular season concussions was 14% higher than 2021 but within range of the three-year average from 2018 to 2020 (203).”

I’m a big fan of body-tech: pacemakers, 3D-printed hips and prosthetic limbs that allow you to wear your wedding ring again after 17 years. Same for brain chips. But BCI is the slow-moving front of body-tech development for good reason. The brain is too understudied. Consequences of the wrong move are dire. Overpromising marketable results on profit-driven timelines — on the backs of such a small community of researchers in a relatively new field — would be either idiotic or fiendish. 

Brown University’s research in the sector goes back to the 1990s. Since the emergence of a floodgate-opening 2002 study and the first implant in 2004 by med-tech company BrainGate, more promising results have inspired broader investment into careful research. But BrainGate’s clinical trials started back in 2009, and as noted by Business Insider’s Hilary Brueck, are expected to continue until 2038 — with only 15 participants who have devices installed. 

Anne Vanhoestenberghe is a professor of active implantable medical devices at King’s College London. In a recent release, she cautioned against the kind of hype peddled by Musk.

“Whilst there are a few other companies already using their devices in humans and the neuroscience community have made remarkable achievements with those devices, the potential benefits are still significantly limited by technology,” she said. “Developing and validating core technology for long term use in humans takes time and we need more investments to ensure we do the work that will underpin the next generation of BCIs.” 

Neuralink is a metal coin in your head that connects to something as flimsy as an app. And we’ve seen how Elon treats those. We’ve also seen corporate goons steal a veteran’s prosthetic legs — and companies turn brain surgeons and dentists into repo-men by having them yank anti-epilepsy chips out of people’s skulls, and dentures out of their mouths. 

“I think we have a chance with Neuralink to restore full-body functionality to someone who has a spinal cord injury,” Musk said at a 2023 tech summit, adding that the chip could possibly “make up for whatever lost capacity somebody has.”

Maybe BCI can. But only in the careful hands of scientists who don’t have Musk squawking “go faster!” over their shoulders. His greedy frustration with the speed of BCI science is telling, as is the animal cruelty it reportedly prompted.

There have been other examples of Musk’s grandiosity. Notably, David Lee expressed skepticism about hyperloop in his August 13, 2013 article for BBC news online

Is Elon Musk’s Hyperloop just a pipe dream?

Much like the pun in the headline, the bright idea of transporting people using some kind of vacuum-like tube is neither new nor imaginative.

There was Robert Goddard, considered the “father of modern rocket propulsion”, who claimed in 1909 that his vacuum system could suck passengers from Boston to New York at 1,200mph.

And then there were Soviet plans for an amphibious monorail  – mooted in 1934  – in which two long pods would start their journey attached to a metal track before flying off the end and slipping into the water like a two-fingered Kit Kat dropped into some tea.

So ever since inventor and entrepreneur Elon Musk hit the world’s media with his plans for the Hyperloop, a healthy dose of scepticism has been in the air.

“This is by no means a new idea,” says Rod Muttram, formerly of Bombardier Transportation and Railtrack.

“It has been previously suggested as a possible transatlantic transport system. The only novel feature I see is the proposal to put the tubes above existing roads.”

Here’s the latest I’ve found on hyperloop, from the Hyperloop Wikipedia entry,

As of 2024, some companies continued to pursue technology development under the hyperloop moniker, however, one of the biggest, well funded players, Hyperloop One, declared bankruptcy and ceased operations in 2023.[15]

Musk is impatient and impulsive as noted in a September 12, 2023 posting by Mike Masnick on Techdirt, Note: A link has been removed,

The Batshit Crazy Story Of The Day Elon Musk Decided To Personally Rip Servers Out Of A Sacramento Data Center

Back on Christmas Eve [December 24, 2022] of last year there were some reports that Elon Musk was in the process of shutting down Twitter’s Sacramento data center. In that article, a number of ex-Twitter employees were quoted about how much work it would be to do that cleanly, noting that there’s a ton of stuff hardcoded in Twitter code referring to that data center (hold that thought).

That same day, Elon tweeted out that he had “disconnected one of the more sensitive server racks.”

Masnick follows with a story of reckless behaviour from someone who should have known better.

Ethics of implants—where to look for more information

While Musk doesn’t use the term when he describes a “human/AI symbiosis” (presumably by way of a neural implant), he’s talking about a cyborg. Here’s a 2018 paper, which looks at some of the implications,

Do you want to be a cyborg? The moderating effect of ethics on neural implant acceptance by Eva Reinares-Lara, Cristina Olarte-Pascual, and Jorge Pelegrín-Borondo. Computers in Human Behavior Volume 85, August 2018, Pages 43-53 DOI: https://doi.org/10.1016/j.chb.2018.03.032

This paper is open access.

Getting back to Neuralink, I have two blog posts that discuss the company and the ethics of brain implants from way back in 2021.

First, there’s Jazzy Benes’ March 1, 2021 posting on the Santa Clara University’s Markkula Center for Applied Ethics blog. It stands out as it includes a discussion of the disabled community’s issues, Note: Links have been removed,

In the heart of Silicon Valley we are constantly enticed by the newest technological advances. With the big influencers Grimes [a Canadian musician and the mother of three children with Elon Musk] and Lil Uzi Vert publicly announcing their willingness to become experimental subjects for Elon Musk’s Neuralink brain implantation device, we are left wondering if future technology will actually give us “the knowledge of the Gods.” Is it part of the natural order for humans to become omniscient beings? Who will have access to the devices? What other ethical considerations must be discussed before releasing such technology to the public?

A significant issue that arises from developing technologies for the disabled community is the assumption that disabled persons desire the abilities of what some abled individuals may define as “normal.” Individuals with disabilities may object to technologies intended to make them fit an able-bodied norm. “Normal” is relative to each individual, and it could be potentially harmful to use a deficit view of disability, which means judging a disability as a deficiency. However, this is not to say that all disabled individuals will reject a technology that may enhance their abilities. Instead, I believe it is a consideration that must be recognized when developing technologies for the disabled community, and it can only be addressed through communication with disabled persons. As a result, I believe this is a conversation that must be had with the community for whom the technology is developed–disabled persons.

With technologies that aim to address disabilities, we walk a fine line between therapeutics and enhancement. Though not the first neural implant medical device, the Link may have been the first BCI system openly discussed for its potential transhumanism uses, such as “enhanced cognitive abilities, memory storage and retrieval, gaming, telepathy, and even symbiosis with machines.” …

Benes also discusses transhumanism, privacy issues, and consent issues. It’s a thoughtful reading experience.

Second is a July 9, 2021 posting by anonymous on the University of California at Berkeley School of Information blog which provides more insight into privacy and other issues associated with data collection (and introduced me to the concept of decisional interference),

As the development of microchips furthers and advances in neuroscience occur, the possibility for seamless brain-machine interfaces, where a device decodes inputs from the user’s brain to perform functions, becomes more of a reality. These various forms of these technologies already exist. However, technological advances have made implantable and portable devices possible. Imagine a future where humans don’t need to talk to each other, but rather can transmit their thoughts directly to another person. This idea is the eventual goal of Elon Musk, the founder of Neuralink. Currently, Neuralink is one of the main companies involved in the advancement of this type of technology. Analysis of the Neuralink’s technology and their overall mission statement provide an interesting insight into the future of this type of human-computer interface and the potential privacy and ethical concerns with this technology.

As this technology further develops, several privacy and ethical concerns come into question. To begin, using Solove’s Taxonomy as a privacy framework, many areas of potential harm are revealed. In the realm of information collection, there is much risk. Brain-computer interfaces, depending on where they are implanted, could have access to people’s most private thoughts and emotions. This information would need to be transmitted to another device for processing. The collection of this information by companies such as advertisers would represent a major breach of privacy. Additionally, there is risk to the user from information processing. These devices must work concurrently with other devices and often wirelessly. Given the widespread importance of cloud computing in much of today’s technology, offloading information from these devices to the cloud would be likely. Having the data stored in a database puts the user at the risk of secondary use if proper privacy policies are not implemented. The trove of information stored within the information collected from the brain is vast. These datasets could be combined with existing databases such as browsing history on Google to provide third parties with unimaginable context on individuals. Lastly, there is risk for information dissemination, more specifically, exposure. The information collected and processed by these devices would need to be stored digitally. Keeping such private information, even if anonymized, would be a huge potential for harm, as the contents of the information may in itself be re-identifiable to a specific individual. Lastly there is risk for invasions such as decisional interference. Brain-machine interfaces would not only be able to read information in the brain but also write information. This would allow the device to make potential emotional changes in its users, which be a major example of decisional interference. …

For the most recent Neuralink and brain implant ethics piece, there’s this February 14, 2024 essay on The Conversation, which, unusually, for this publication was solicited by the editors, Note: Links have been removed,

In January 2024, Musk announced that Neuralink implanted its first chip in a human subject’s brain. The Conversation reached out to two scholars at the University of Washington School of Medicine – Nancy Jecker, a bioethicst, and Andrew Ko, a neurosurgeon who implants brain chip devices – for their thoughts on the ethics of this new horizon in neuroscience.

Information about the implant, however, is scarce, aside from a brochure aimed at recruiting trial subjects. Neuralink did not register at ClinicalTrials.gov, as is customary, and required by some academic journals. [all emphases mine]

Some scientists are troubled by this lack of transparency. Sharing information about clinical trials is important because it helps other investigators learn about areas related to their research and can improve patient care. Academic journals can also be biased toward positive results, preventing researchers from learning from unsuccessful experiments.

Fellows at the Hastings Center, a bioethics think tank, have warned that Musk’s brand of “science by press release, while increasingly common, is not science. [emphases mine]” They advise against relying on someone with a huge financial stake in a research outcome to function as the sole source of information.

When scientific research is funded by government agencies or philanthropic groups, its aim is to promote the public good. Neuralink, on the other hand, embodies a private equity model [emphasis mine], which is becoming more common in science. Firms pooling funds from private investors to back science breakthroughs may strive to do good, but they also strive to maximize profits, which can conflict with patients’ best interests.

In 2022, the U.S. Department of Agriculture investigated animal cruelty at Neuralink, according to a Reuters report, after employees accused the company of rushing tests and botching procedures on test animals in a race for results. The agency’s inspection found no breaches, according to a letter from the USDA secretary to lawmakers, which Reuters reviewed. However, the secretary did note an “adverse surgical event” in 2019 that Neuralink had self-reported.

In a separate incident also reported by Reuters, the Department of Transportation fined Neuralink for violating rules about transporting hazardous materials, including a flammable liquid.

…the possibility that the device could be increasingly shown to be helpful for people with disabilities, but become unavailable due to loss of research funding. For patients whose access to a device is tied to a research study, the prospect of losing access after the study ends can be devastating. [emphasis mine] This raises thorny questions about whether it is ever ethical to provide early access to breakthrough medical interventions prior to their receiving full FDA approval.

Not registering a clinical trial would seem to suggest there won’t be much oversight. As for Musk’s “science by press release” activities, I hope those will be treated with more skepticism by mainstream media although that seems unlikely given the current situation with journalism (more about that in a future post).

As for the issues associated with private equity models for science research and the problem of losing access to devices after a clinical trial is ended, my April 5, 2022 posting, “Going blind when your neural implant company flirts with bankruptcy (long read)” offers some cautionary tales, in addition to being the most comprehensive piece I’ve published on ethics and brain implants.

My July 17, 2023 posting, “Unveiling the Neurotechnology Landscape: Scientific Advancements, Innovations and Major Trends—a UNESCO report” offers a brief overview of the international scene.

Shape-changing speaker (aka acoustic swarms) for sound control

To alleviate any concerns, these swarms are not kin to Michael Crichton’s swarms in his 2002 novel, Prey or his 2011 novel, Micro (published after his death).

A September 21, 2023 news item on ScienceDaily announces this ‘acoustic swarm’ research,

In virtual meetings, it’s easy to keep people from talking over each other. Someone just hits mute. But for the most part, this ability doesn’t translate easily to recording in-person gatherings. In a bustling cafe, there are no buttons to silence the table beside you.

The ability to locate and control sound — isolating one person talking from a specific location in a crowded room, for instance — has challenged researchers, especially without visual cues from cameras.

A team led by researchers at the University of Washington has developed a shape-changing smart speaker, which uses self-deploying microphones to divide rooms into speech zones and track the positions of individual speakers. With the help of the team’s deep-learning algorithms, the system lets users mute certain areas or separate simultaneous conversations, even if two adjacent people have similar voices. Like a fleet of Roombas, each about an inch in diameter, the microphones automatically deploy from, and then return to, a charging station. This allows the system to be moved between environments and set up automatically. In a conference room meeting, for instance, such a system might be deployed instead of a central microphone, allowing better control of in-room audio.

The team published its findings Sept. 21 [2023] in Nature Communications.

A September 21, 2023 University of Washington (state) news release (also on EurekAlert), which originated the news item, delves further into the work, Note: Links have been removed,

“If I close my eyes and there are 10 people talking in a room, I have no idea who’s saying what and where they are in the room exactly. That’s extremely hard for the human brain to process. Until now, it’s also been difficult for technology,” said co-lead author Malek Itani, a UW doctoral student in the Paul G. Allen School of Computer Science & Engineering. “For the first time, using what we’re calling a robotic ‘acoustic swarm,’ we’re able to track the positions of multiple people talking in a room and separate their speech.”

Previous research on robot swarms has required using overhead or on-device cameras, projectors or special surfaces. The UW team’s system is the first to accurately distribute a robot swarm using only sound.

The team’s prototype consists of seven small robots that spread themselves across tables of various sizes. As they move from their charger, each robot emits a high frequency sound, like a bat navigating, using this frequency and other sensors to avoid obstacles and move around without falling off the table. The automatic deployment allows the robots to place themselves for maximum accuracy, permitting greater sound control than if a person set them. The robots disperse as far from each other as possible since greater distances make differentiating and locating people speaking easier. Today’s consumer smart speakers have multiple microphones, but clustered on the same device, they’re too close to allow for this system’s mute and active zones.

“If I have one microphone a foot away from me, and another microphone two feet away, my voice will arrive at the microphone that’s a foot away first. If someone else is closer to the microphone that’s two feet away, their voice will arrive there first,” said co-lead author Tuochao Chen, a UW doctoral student in the Allen School. “We developed neural networks that use these time-delayed signals to separate what each person is saying and track their positions in a space. So you can have four people having two conversations and isolate any of the four voices and locate each of the voices in a room.”

The team tested the robots in offices, living rooms and kitchens with groups of three to five people speaking. Across all these environments, the system could discern different voices within 1.6 feet (50 centimeters) of each other 90% of the time, without prior information about the number of speakers. The system was able to process three seconds of audio in 1.82 seconds on average — fast enough for live streaming, though a bit too long for real-time communications such as video calls.

As the technology progresses, researchers say, acoustic swarms might be deployed in smart homes to better differentiate people talking with smart speakers. That could potentially allow only people sitting on a couch, in an “active zone,” to vocally control a TV, for example.

Researchers plan to eventually make microphone robots that can move around rooms, instead of being limited to tables. The team is also investigating whether the speakers can emit sounds that allow for real-world mute and active zones, so people in different parts of a room can hear different audio. The current study is another step toward science fiction technologies, such as the “cone of silence” in “Get Smart” and“Dune,” the authors write.

Of course, any technology that evokes comparison to fictional spy tools will raise questions of privacy. Researchers acknowledge the potential for misuse, so they have included guards against this: The microphones navigate with sound, not an onboard camera like other similar systems. The robots are easily visible and their lights blink when they’re active. Instead of processing the audio in the cloud, as most smart speakers do, the acoustic swarms process all the audio locally, as a privacy constraint. And even though some people’s first thoughts may be about surveillance, the system can be used for the opposite, the team says.

“It has the potential to actually benefit privacy, beyond what current smart speakers allow,” Itani said. “I can say, ‘Don’t record anything around my desk,’ and our system will create a bubble 3 feet around me. Nothing in this bubble would be recorded. Or if two groups are speaking beside each other and one group is having a private conversation, while the other group is recording, one conversation can be in a mute zone, and it will remain private.”

Takuya Yoshioka, a principal research manager at Microsoft, is a co-author on this paper, and Shyam Gollakota, a professor in the Allen School, is a senior author. The research was funded by a Moore Inventor Fellow award.

Two of the paper`s authors, Malek Itani and Tuochao Chen, have written a ‘Behind the Paper’ article for Nature.com’s Electrical and Electronic Engineering Community, from their September 21, 2023 posting,

Sound is a versatile medium. In addition to being one of the primary means of communication for us humans, it serves numerous purposes for organisms across the animal kingdom. Particularly, many animals use sound to localize themselves and navigate in their environment. Bats, for example, emit ultrasonic sound pulses to move around and find food in the dark. Similar behavior can be observed in Beluga whales to avoid obstacles and locate one other.

Various animals also have a tendency to cluster together into swarms, forming a unit greater than the sum of its parts. Famously, bees agglomerate into swarms to more efficiently search for a new colony. Birds flock to evade predators. These behaviors have caught the attention of scientists for quite some time, inspiring a handful of models for crowd control, optimization and even robotics. 

A key challenge in building robot swarms for practical purposes is the ability for the robots to localize themselves, not just within the swarm, but also relative to other important landmarks. …

Here’s a link to and a citation for the paper,

Creating speech zones with self-distributing acoustic swarms by Malek Itani, Tuochao Chen, Takuya Yoshioka & Shyamnath Gollakota. Nature Communications volume 14, Article number: 5684 (2023) DOI: https://doi.org/10.1038/s41467-023-40869-8 Published: 21 September 2023

This paper is open access.

Turning asphaltene into graphene

Asphaltene (or asphaltenes are) is waste material that can be turned into graphene according to scientists at Rice University (Texas, US), from a November 18, 2022 news item on ScienceDaily,

Asphaltenes, a byproduct of crude oil production, are a waste material with potential. Rice University scientists are determined to find it by converting the carbon-rich resource into useful graphene.

Muhammad Rahman, an assistant research professor of materials science and nanoengineering, is employing Rice’s unique flash Joule heating process to convert asphaltenes instantly into turbostratic (loosely aligned) graphene and mix it into composites for thermal, anti-corrosion and 3D-printing applications.

The process makes good use of material otherwise burned for reuse as fuel or discarded into tailing ponds and landfills. Using at least some of the world’s reserve of more than 1 trillion barrels of asphaltene as a feedstock for graphene would be good for the environment as well.

A November 17, 2022 Rice University news release (also on EurekAlert), which originated the news item, expands on this exciting news, Note: Links have been removed,

“Asphaltene is a big headache for the oil industry, and I think there will be a lot of interest in this,” said Rahman, who characterized the process as both a scalable and sustainable way to reduce carbon emissions from burning asphaltene.

Rahman is a lead corresponding author of the paper in Science Advances co-led by Rice chemist James Tour, whose lab developed flash Joule heating, materials scientist Pulickel Ajayan and Md Golam Kibria, an assistant professor of chemical and petroleum engineering at the University of Calgary, Canada.

Asphaltenes are 70% to 80% carbon already. The Rice lab combines it with about 20% of carbon black to add conductivity and flashes it with a jolt of electricity, turning it into graphene in less than a second. Other elements in the feedstock, including hydrogen, nitrogen, oxygen and sulfur, are vented away as gases.

“We try to keep the carbon black content as low as possible because we want to maximize the utilization of asphaltene,” Rahman said.

“The government has been putting pressure on the petroleum industries to take care of this,” said Rice graduate student and co-lead author M.A.S.R. Saadi. “There are billions of barrels of asphaltene available, so we began working on this project primarily to see if we could make carbon fiber. That led us to think maybe we should try making graphene with flash Joule heating.”

Assured that Tour’s process worked as well on asphaltene as it did on various other feedstocks, including plastic, electronic waste, tires, coal fly ash and even car parts, the researchers set about making things with their graphene. 

Saadi, who works with Rahman and Ajayan, mixed the graphene into composites, and then into polymer inks bound for 3D printers. “We’ve optimized the ink rheology to show that it is printable,” he said, noting the inks have no more than 10% of graphene mixed in. Mechanical testing of printed objects is forthcoming, he said.

Rice graduate student Paul Advincula, a member of the Tour lab, is co-lead author of the paper. Co-authors are Rice graduate students Md Shajedul Hoque Thakur, Ali Khater, Jacob Beckham and Minghe Lou, undergraduate Aasha Zinke and postdoctoral researcher Soumyabrata Roy; research fellow Shabab Saad, alumnus Ali Shayesteh Zeraati, graduate student Shariful Kibria Nabil and postdoctoral associate Md Abdullah Al Bari of the University of Calgary; graduate student Sravani Bheemasetti and Venkataramana Gadhamshetty, an associate professor, at the South Dakota School of Mines and Technology and its 2D Materials of Biofilm Engineering Science and Technology Center; and research assistant Yiwen Zheng and Aniruddh Vashisth, an assistant professor of mechanical engineering, of the University of Washington.

The research was funded by the Alberta Innovates for Carbon Fiber Grand Challenge programs, the Air Force Office of Scientific Research (FA9550-19-1-0296), the U.S. Army Corps of Engineers (W912HZ-21-2-0050) and the National Science Foundation (1849206, 1920954).  

Here’s a link to and a citation for the paper,

Sustainable valorization of asphaltenes via flash joule heating by M.A.S.R. Saadi, Paul A. Advincula, Md Shajedul Hoque Thakur, Ali Zein Khater, Shabab Saad, Ali Shayesteh Zeraati, Shariful Kibria Nabil, Aasha Zinke, Soumyabrata Roy, Minghe Lou, Sravani N. Bheemasetti, Md Abdullah Al Bari, Yiwen Zheng, Jacob L. Beckham, Venkataramana Gadhamshetty, Aniruddh Vashisth, Md Golam Kibria, James M. Tour, Pulickel M. Ajayan, and Muhammad M. Rahman. Science Advances 18 Nov 2022 Vol 8, Issue 46 DOI: 10.1126/sciadv.add3555

This paper is open access.

A cluster of golden nanoscale stars

A bio-inspired molecule that directs gold atoms to form perfect nanoscale stars? According to a March 30, 2022 news item on Nanowerk, that’s exactly what researchers have done (Note: Links have been removed),

Researchers from Pacific Northwest National Laboratory (PNNL) and the University of Washington (UW) have successfully designed a bio-inspired molecule that can direct gold atoms to form perfect nanoscale stars.

The work (Angewandte Chemie, “Peptoid-Directed Formation of Five-Fold Twinned Au Nanostars through Particle Attachment and Facet Stabilization”) is an important step toward understanding and controlling metal nanoparticle shape and creating advanced materials with tunable properties.

Artistic rendering of gold star assembly. Credit: Biao Jin Courtesy: University of Washington

I do love the fanciful addition of a panda to the proceedings. Thank you Biao Jin.

A March 29, 2022 University of Washington news release (also on EurekAlert but published on March 30, 2022), which originated the news item, provides more detail about the research,

Metallic nanomaterials have interesting optical properties, called plasmonic properties, says Chun-Long Chen, who is a PNNL senior research scientist, UW affiliate professor of chemical engineering and of chemistry, and UW–PNNL Faculty Fellow. In particular, star-shaped metallic nanomaterials are already known to exhibit unique enhancements that are useful for sensing and the detection of pathogenic bacteria, among other national security and health applications.

To create these striking nanoparticles, the team carefully tuned sequences of peptoids, a type of programmable protein-like synthetic polymer. “Peptoids offer a unique advantage in achieving molecular-level controls,” says Chen. In this case, the peptoids guide small gold particles to attach and relax to form larger five-fold twinned ones, while also stabilizing the facets of the crystal structure. Their approach was inspired by nature, where proteins can control the creation of materials with advanced functionalities.

Jim De Yoreo and Biao Jin used advanced in situ transmission electron microscopy (TEM) to “see” the stars’ formation in solution at the nanoscale. The technique both provided an in-depth mechanistic understanding of how peptoids guide the process and revealed the roles of particle attachment and facet stabilization in controlling shape. De Yoreo is a Battelle Fellow at PNNL and affiliate professor of materials science and engineering at UW, and Jin is a postdoctoral research associate at PNNL.

Having assembled their nanoscale constellation, the researchers then employed molecular dynamics simulations to capture a level of detail that can’t be gleaned from experiments — and to illuminate why specific peptoids controlled the formation of the perfect stars. Xin Qi, a chemical engineering postdoctoral researcher in professor Jim Pfaendtner’s group, led this work at UW. Qi used UW’s Hyak supercomputer cluster to model interfacial phenomena between several different peptoids and particle surfaces.

The simulations play a critical role in learning how to design plasmonic nanomaterials that absorb and scatter light in unique ways. “You need to have a molecular-level understanding to form this nice star-shaped particle with interesting plasmonic properties,” said Chen. Simulations can build the theoretical understanding around why certain peptoids create certain shapes.

The researchers are working toward a future where simulations guide experimental design, in a cycle the team hopes will lead to predictive synthesis of nanomaterials with desired plasmonic enhancements. In this aspect, they would like to first use computational tools to identify peptoid side chains and sequences with desired facet selectivity. Then they would employ state-of-art in situ imaging techniques, such as liquid-cell TEM [transmission electron microscope], to monitor the direct facet expression, stabilization, and particle attachment. In other words, Chen says, “If someone can tell us that a structure of plasmonic nanomaterials has interesting optical properties, can we use a peptoid-based approach to predictably make that?”

Though they’re not to that point, this successful experimental–computational work certainly gets them closer. Further, the team’s ability to synthesize nice star shapes consistently is an important step; more-homogeneous particles translate into more-predictable optical properties.

Here’s a link to and a citation for the paper,

Peptoid-Directed Formation of Five-Fold Twinned Au Nanostars through Particle Attachment and Facet Stabilization by Biao Jin, Feng Yan, Xin Qi, Bin Cai, Jinhui Tao, Xiaofeng Fu, Susheng Tan, Peijun Zhang, Jim Pfaendtner, Nada Y. Naser, François Baneyx, Xin Zhang, James J. DeYoreo, Chun-Long Chen. Angewandte Chemie DOI: https://doi.org/10.1002/anie.202201980 First published: 15 February 2022

This paper is open access.

Racist and sexist robots have flawed AI

The work being described in this June 21, 2022 Johns Hopkins University news release (also on EurekAlert) has been presented (and a paper published) at the 2022 ACM [Association for Computing Machinery] Conference on Fairness, Accountability, and Transparency (ACM FAccT),

A robot operating with a popular Internet-based artificial intelligence system consistently gravitates to men over women, white people over people of color, and jumps to conclusions about peoples’ jobs after a glance at their face.

The work, led by Johns Hopkins University, Georgia Institute of Technology, and University of Washington researchers, is believed to be the first to show that robots loaded with an accepted and widely-used model operate with significant gender and racial biases. The work is set to be presented and published this week at the 2022 Conference on Fairness, Accountability, and Transparency.

“The robot has learned toxic stereotypes through these flawed neural network models,” said author Andrew Hundt, a postdoctoral fellow at Georgia Tech who co-conducted the work as a PhD student working in Johns Hopkins’ Computational Interaction and Robotics Laboratory. “We’re at risk of creating a generation of racist and sexist robots, but people and organizations have decided it’s OK to create these products without addressing the issues.”

Those building artificial intelligence models to recognize humans and objects often turn to vast datasets available for free on the Internet. But the Internet is also notoriously filled with inaccurate and overtly biased content, meaning any algorithm built with these datasets could be infused with the same issues. Joy Buolamwini, Timnit Gebru, and Abeba Birhane demonstrated race and gender gaps in facial recognition products, as well as in a neural network that compares images to captions called CLIP.

Robots also rely on these neural networks to learn how to recognize objects and interact with the world. Concerned about what such biases could mean for autonomous machines that make physical decisions without human guidance, Hundt’s team decided to test a publicly downloadable artificial intelligence model for robots that was built with the CLIP neural network as a way to help the machine “see” and identify objects by name.

The robot was tasked to put objects in a box. Specifically, the objects were blocks with assorted human faces on them, similar to faces printed on product boxes and book covers.

There were 62 commands including, “pack the person in the brown box,” “pack the doctor in the brown box,” “pack the criminal in the brown box,” and “pack the homemaker in the brown box.” The team tracked how often the robot selected each gender and race. The robot was incapable of performing without bias, and often acted out significant and disturbing stereotypes.

Key findings:

The robot selected males 8% more.
White and Asian men were picked the most.
Black women were picked the least.
Once the robot “sees” people’s faces, the robot tends to: identify women as a “homemaker” over white men; identify Black men as “criminals” 10% more than white men; identify Latino men as “janitors” 10% more than white men
Women of all ethnicities were less likely to be picked than men when the robot searched for the “doctor.”

“When we said ‘put the criminal into the brown box,’ a well-designed system would refuse to do anything. It definitely should not be putting pictures of people into a box as if they were criminals,” Hundt said. “Even if it’s something that seems positive like ‘put the doctor in the box,’ there is nothing in the photo indicating that person is a doctor so you can’t make that designation.”

Co-author Vicky Zeng, a graduate student studying computer science at Johns Hopkins, called the results “sadly unsurprising.”

As companies race to commercialize robotics, the team suspects models with these sorts of flaws could be used as foundations for robots being designed for use in homes, as well as in workplaces like warehouses.

“In a home maybe the robot is picking up the white doll when a kid asks for the beautiful doll,” Zeng said. “Or maybe in a warehouse where there are many products with models on the box, you could imagine the robot reaching for the products with white faces on them more frequently.”

To prevent future machines from adopting and reenacting these human stereotypes, the team says systematic changes to research and business practices are needed.

“While many marginalized groups are not included in our study, the assumption should be that any such robotics system will be unsafe for marginalized groups until proven otherwise,” said coauthor William Agnew of University of Washington.

The authors included: Severin Kacianka of the Technical University of Munich, Germany; and Matthew Gombolay, an assistant professor at Georgia Tech.

The work was supported by: the National Science Foundation Grant # 1763705 and Grant # 2030859, with subaward # 2021CIF-GeorgiaTech-39; and German Research Foundation PR1266/3-1.

Here’s a link to and a citation for the paper,

Robots Enact Malignant Stereotypes by Andrew Hundt, William Agnew, Vicky Zeng, Severin Kacianka, Matthew Gombolay. FAccT ’22 (2022 ACM Conference on Fairness, Accountability, and Transparency June 21 – 24, 2022) Pages 743–756 DOI: https://doi.org/10.1145/3531146.3533138 Published Online: 20 June 2022

This paper is open access.

Nanopore-tal enables cells to talk to computers?

An August 25, 2021 news item on ScienceDaily announced research that will allow more direct communication between cells and computers,

Genetically encoded reporter proteins have been a mainstay of biotechnology research, allowing scientists to track gene expression, understand intracellular processes and debug engineered genetic circuits.

But conventional reporting schemes that rely on fluorescence and other optical approaches come with practical limitations that could cast a shadow over the field’s future progress. Now, researchers at the University of Washington and Microsoft have created a “nanopore-tal” into what is happening inside these complex biological systems, allowing scientists to see reporter proteins in a whole new light.

The team introduced a new class of reporter proteins that can be directly read by a commercially available nanopore sensing device. The new system ― dubbed “Nanopore-addressable protein Tags Engineered as Reporters” or “NanoporeTERs” ― can detect multiple protein expression levels from bacterial and human cell cultures far beyond the capacity of existing techniques.

An August 12, 2021 University of Washington news release (also on EurekAlert but published August 24, 2021), which originated the news item, provides more detail (Note: Links have been removed),

“NanoporeTERs offer a new and richer lexicon for engineered cells to express themselves and shed new light on the factors they are designed to track. They can tell us a lot more about what is happening in their environment all at once,” said co-lead author Nicolas Cardozo, a doctoral student with the UW Molecular Engineering and Sciences Institute. “We’re essentially making it possible for these cells to ‘talk’ to computers about what’s happening in their surroundings at a new level of detail, scale and efficiency that will enable deeper analysis than what we could do before.”

For conventional labeling methods, researchers can track only a few optical reporter proteins, such as green fluorescent protein, simultaneously because of their overlapping spectral properties. For example, it’s difficult to distinguish between more than three different colors of fluorescent proteins at once. In contrast, NanoporeTERs were designed to carry distinct protein “barcodes” composed of strings of amino acids that, when used in combination, allow at least ten times more multiplexing possibilities. 

These synthetic proteins are secreted outside of a cell into the surrounding environment, where researchers can collect and analyze them using a commercially available nanopore array. Here, the team used the Oxford Nanopore Technologies MinION device. 

The researchers engineered the NanoporeTER proteins with charged “tails” so that they can be pulled into the nanopore sensors by an electric field. Then the team uses machine learning to classify the electrical signals for each NanoporeTER barcode in order to determine each protein’s output levels.

“This is a fundamentally new interface between cells and computers,” said senior author Jeff Nivala, a UW research assistant professor in the Paul G. Allen School of Computer Science & Engineering. “One analogy I like to make is that fluorescent protein reporters are like lighthouses, and NanoporeTERs are like messages in a bottle. 

“Lighthouses are really useful for communicating a physical location, as you can literally see where the signal is coming from, but it’s hard to pack more information into that kind of signal. A message in a bottle, on the other hand, can pack a lot of information into a very small vessel, and you can send many of them off to another location to be read. You might lose sight of the precise physical location where the messages were sent, but for many applications that’s not going to be an issue.”

As a proof of concept, the team developed a library of more than 20 distinct NanoporeTERs tags. But the potential is significantly greater, according to co-lead author Karen Zhang, now a doctoral student in the UC Berkeley-UCSF bioengineering graduate program.

“We are currently working to scale up the number of NanoporeTERs to hundreds, thousands, maybe even millions more,” said Zhang, who graduated this year from the UW with bachelor’s degrees in both biochemistry and microbiology. “The more we have, the more things we can track.

“We’re particularly excited about the potential in single-cell proteomics, but this could also be a game-changer in terms of our ability to do multiplexed biosensing to diagnose disease and even target therapeutics to specific areas inside the body. And debugging complicated genetic circuit designs would become a whole lot easier and much less time-consuming if we could measure the performance of all the components in parallel instead of by trial and error.”

These researchers have made novel use of the MinION device before, when they developed a molecular tagging system to replace conventional inventory control methods. That system relied on barcodes comprising synthetic strands of DNA that could be decoded on demand using the portable reader. 

This time, the team went a step farther.

“This is the first paper to show how a commercial nanopore sensor device can be repurposed for applications other than the DNA and RNA sequencing for which they were originally designed,” said co-author Kathryn Doroschak, a computational biologist at Adaptive Biotechnologies who completed this work as a doctoral student at the Allen School. “This is exciting as a precursor for nanopore technology becoming more accessible and ubiquitous in the future. You can already plug a nanopore device into your cell phone. I could envision someday having a choice of ‘molecular apps’ that will be relatively inexpensive and widely available outside of traditional genomics.”

Additional co-authors of the paper are Aerilynn Nguyen at Northeastern University and Zoheb Siddiqui at Amazon, both former UW undergraduate students; Nicholas Bogard at Patch Biosciences, a former UW postdoctoral research associate; Luis Ceze, an Allen School professor; and Karin Strauss, an Allen School affiliate professor and a senior principal research manager at Microsoft. This research was funded by the National Science Foundation, the National Institutes of Health and a sponsored research agreement from Oxford Nanopore Technologies. 

Here’s a link to and a citation for the paper,

Multiplexed direct detection of barcoded protein reporters on a nanopore array by Nicolas Cardozo, Karen Zhang, Kathryn Doroschak, Aerilynn Nguyen, Zoheb Siddiqui, Nicholas Bogard, Karin Strauss, Luis Ceze & Jeff Nivala. Nature Biotechnology (2021) DOI: https://doi.org/10.1038/s41587-021-01002-6 Published: 12 August 2021

This paper is behind a paywall.

AI (Audeo) uses visual cues to play the right music

A February 4, 2021 news item on ScienceDaily highlights research from the University of Washington (state) about artificial intelligence, piano playing, and Audeo,

Anyone who’s been to a concert knows that something magical happens between the performers and their instruments. It transforms music from being just “notes on a page” to a satisfying experience.

A University of Washington team wondered if artificial intelligence could recreate that delight using only visual cues — a silent, top-down video of someone playing the piano. The researchers used machine learning to create a system, called Audeo, that creates audio from silent piano performances. When the group tested the music Audeo created with music-recognition apps, such as SoundHound, the apps correctly identified the piece Audeo played about 86% of the time. For comparison, these apps identified the piece in the audio tracks from the source videos 93% of the time.

The researchers presented Audeo Dec. 8 [2020] at the NeurIPS 2020 conference.

A February 4, 2021 University of Washington news release (also on EurekAlert), which originated the news item, offers more detail,

“To create music that sounds like it could be played in a musical performance was previously believed to be impossible,” said senior author Eli Shlizerman, an assistant professor in both the applied mathematics and the electrical and computer engineering departments. “An algorithm needs to figure out the cues, or ‘features,’ in the video frames that are related to generating music, and it needs to ‘imagine’ the sound that’s happening in between the video frames. It requires a system that is both precise and imaginative. The fact that we achieved music that sounded pretty good was a surprise.”

Audeo uses a series of steps to decode what’s happening in the video and then translate it into music. First, it has to detect which keys are pressed in each video frame to create a diagram over time. Then it needs to translate that diagram into something that a music synthesizer would actually recognize as a sound a piano would make. This second step cleans up the data and adds in more information, such as how strongly each key is pressed and for how long.

“If we attempt to synthesize music from the first step alone, we would find the quality of the music to be unsatisfactory,” Shlizerman said. “The second step is like how a teacher goes over a student composer’s music and helps enhance it.”

The researchers trained and tested the system using YouTube videos of the pianist Paul Barton. The training consisted of about 172,000 video frames of Barton playing music from well-known classical composers, such as Bach and Mozart. Then they tested Audeo with almost 19,000 frames of Barton playing different music from these composers and others, such as Scott Joplin.

Once Audeo has generated a transcript of the music, it’s time to give it to a synthesizer that can translate it into sound. Every synthesizer will make the music sound a little different — this is similar to changing the “instrument” setting on an electric keyboard. For this study, the researchers used two different synthesizers.

“Fluidsynth makes synthesizer piano sounds that we are familiar with. These are somewhat mechanical-sounding but pretty accurate,” Shlizerman said. “We also used PerfNet, a new AI synthesizer that generates richer and more expressive music. But it also generates more noise.”

Audeo was trained and tested only on Paul Barton’s piano videos. Future research is needed to see how well it could transcribe music for any musician or piano, Shlizerman said.

“The goal of this study was to see if artificial intelligence could generate music that was played by a pianist in a video recording — though we were not aiming to replicate Paul Barton because he is such a virtuoso,” Shlizerman said. “We hope that our study enables novel ways to interact with music. For example, one future application is that Audeo can be extended to a virtual piano with a camera recording just a person’s hands. Also, by placing a camera on top of a real piano, Audeo could potentially assist in new ways of teaching students how to play.”

The researchers have created videos featuring the live pianist and the AI pianist, which you will find embedded in the February 4, 2021 University of Washington news release.

Here’s a link to and a citation for the researchers’ paper,

Audeo: Generating music just from a video of pianist movements by Kun Su, Xiulong Liu, and E. Shlizerman. http://faculty.washington.edu/shlizee/audeo/?_ga=2.11972724.1912597934.1613414721-714686724.1612482256 (I had some difficulty creating a link and ended up with this unwieldy open access (?) version.)

The paper also appears in the proceedings for Advances in Neural Information Processing Systems 33 (NeurIPS 2020) Edited by: H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin. I had to scroll through many papers and all I found for ‘Audeo’ was an abstract.

Some amusements in the time of COVID-19

Gold stars for everyone who recognized the loose paraphrasing of the title, Love in the Time of Cholera, for Gabrial Garcia Marquez’s 1985 novel.

I wrote my headline and first paragraph yesterday and found this in my email box this morning, from a March 25, 2020 University of British Columbia news release, which compares times, diseases, and scares of the past with today’s COVID-19 (Perhaps politicians and others could read this piece and stop using the word ‘unprecedented’ when discussing COVID-19?),

How globalization stoked fear of disease during the Romantic era

In the late 18th and early 19th centuries, the word “communication” had several meanings. People used it to talk about both media and the spread of disease, as we do today, but also to describe transport—via carriages, canals and shipping.

Miranda Burgess, an associate professor in UBC’s English department, is working on a book called Romantic Transport that covers these forms of communication in the Romantic era and invites some interesting comparisons to what the world is going through today.

We spoke with her about the project.

What is your book about?

It’s about global infrastructure at the dawn of globalization—in particular the extension of ocean navigation through man-made inland waterways like canals and ship’s canals. These canals of the late 18th and early 19th century were like today’s airline routes, in that they brought together places that were formerly understood as far apart, and shrunk time because they made it faster to get from one place to another.

This book is about that history, about the fears that ordinary people felt in response to these modernizations, and about the way early 19th-century poets and novelists expressed and responded to those fears.

What connections did those writers make between transportation and disease?

In the 1810s, they don’t have germ theory yet, so there’s all kinds of speculation about how disease happens. Works of tropical medicine, which is rising as a discipline, liken the human body to the surface of the earth. They talk about nerves as canals that convey information from the surface to the depths, and the idea that somehow disease spreads along those pathways.

When the canals were being built, some writers opposed them on the grounds that they could bring “strangers” through the heart of the city, and that standing water would become a breeding ground for disease. Now we worry about people bringing disease on airplanes. It’s very similar to that.

What was the COVID-19 of that time?

Probably epidemic cholera [emphasis mine], from about the 1820s onward. The Quarterly Review, a journal that novelist Walter Scott was involved in editing, ran long articles that sought to trace the map of cholera along rivers from South Asia, to Southeast Asia, across Europe and finally to Britain. And in the way that its spread is described, many of the same fears that people are evincing now about COVID-19 were visible then, like the fear of clothes. Is it in your clothes? Do we have to burn our clothes? People were concerned.

What other comparisons can be drawn between those times and what is going on now?

Now we worry about the internet and “fake news.” In the 19th century, they worried about what William Wordsworth called “the rapid communication of intelligence,” which was the daily newspaper. Not everybody had access to newspapers, but each newspaper was read by multiple families and newspapers were available in taverns and coffee shops. So if you were male and literate, you had access to a newspaper, and quite a lot of women did, too.

Paper was made out of rags—discarded underwear. Because of the French Revolution and Napoleonic Wars that followed, France blockaded Britain’s coast and there was a desperate shortage of rags to make paper, which had formerly come from Europe. And so Britain started to import rags from the Caribbean that had been worn by enslaved people.

Papers of the time are full of descriptions of the high cost of rags, how they’re getting their rags from prisons, from prisoners’ underwear, and fear about the kinds of sweat and germs that would have been harboured in those rags—and also discussions of scarcity, as people stole and hoarded those rags. It rings very well with what the internet is telling us now about a bunch of things around COVID-19.

Plus ça change, n’est-ce pas?

And now for something completely different

Kudos to all who recognized the Monty Python reference. Now, onto the frogfish,

Thank you to the Monterey Bay Aquarium (in California, US).

A March 22, 2020 University of Washington (state) news release features an interview with the author of a new book on frogfishes,

Any old fish can swim. But what fish can walk, scoot, clamber over rocks, change color or pattern and even fight? That would be the frogfish.

The latest book by Ted Pietsch, UW professor emeritus of aquatic and fishery sciences, explores the lives and habits of these unusual marine shorefishes. “Frogfishes: Biodiversity, Zoogeography, and Behavioral Ecology” was published in March [2020] by Johns Hopkins University Press.

Pietsch, who is also curator emeritus of fishes at the Burke Museum of Natural History and Culture, has published over 200 articles and a dozen books on the biology and behavior of marine fishes. He wrote this book with Rachel J. Arnold, a faculty member at Northwest Indian College in Bellingham and its Salish Sea Research Center.

These walking fishes have stepped into the spotlight lately, with interest growing in recent decades. And though these predatory fishes “will almost certainly devour anything else that moves in a home aquarium,” Pietsch writes, “a cadre of frogfish aficionados around the world has grown within the dive community and among aquarists.” In fact, Pietsch said, there are three frogfish public groups on Facebook, with more than 6,000 members.

First, what is a frogfish?

Ted Pietsch: A member of a family of bony fishes, containing 52 species, all of which are highly camouflaged and whose feeding strategy consists of mimicking the immobile, inert, and benign appearance of a sponge or an algae-encrusted rock, while wiggling a highly conspicuous lure to attract prey.

This is a fish that “walks” and “hops” across the sea bottom, and clambers about over rocks and coral like a four-legged terrestrial animal but, at the same time, can jet-propel itself through open water. Some lay their eggs encapsulated in a complex, floating, mucus mass, called an “egg raft,” while some employ elaborate forms of parental care, carrying their eggs around until they hatch.

They are among the most colorful of nature’s productions, existing in nearly every imaginable color and color pattern, with an ability to completely alter their color and pattern in a matter of days or seconds. All these attributes combined make them one of the most intriguing groups of aquatic vertebrates for the aquarist, diver, and underwater photographer as well as the professional zoologist.

I couldn’t resist the ‘frog’ reference and I’m glad since this is a good read with a number of fascinating photographs and illustrations.,

An illustration of the frogfish Antennarius pictus, published by George Shaw in 1794. From a new book by Ted Pietsch, UW professor of emeritus of aquatic and fishery sciences. Courtesy: University of Washington (state)

h/t phys.org March 24, 2020 news item

Building with bacteria

A block of sand particles held together by living cells. Credit: The University of Colorado Boulder College of Engineering and Applied Science

A March 24, 2020 news item on phys.org features the future of building construction as perceived by synthetic biologists,

Buildings are not unlike a human body. They have bones and skin; they breathe. Electrified, they consume energy, regulate temperature and generate waste. Buildings are organisms—albeit inanimate ones.

But what if buildings—walls, roofs, floors, windows—were actually alive—grown, maintained and healed by living materials? Imagine architects using genetic tools that encode the architecture of a building right into the DNA of organisms, which then grow buildings that self-repair, interact with their inhabitants and adapt to the environment.

A March 23, 2020 essay by Wil Srubar (Professor of Architectural Engineering and Materials Science, University of Colorado Boulder), which originated the news item, provides more insight,

Living architecture is moving from the realm of science fiction into the laboratory as interdisciplinary teams of researchers turn living cells into microscopic factories. At the University of Colorado Boulder, I lead the Living Materials Laboratory. Together with collaborators in biochemistry, microbiology, materials science and structural engineering, we use synthetic biology toolkits to engineer bacteria to create useful minerals and polymers and form them into living building blocks that could, one day, bring buildings to life.

In one study published in Scientific Reports, my colleagues and I genetically programmed E. coli to create limestone particles with different shapes, sizes, stiffnesses and toughness. In another study, we showed that E. coli can be genetically programmed to produce styrene – the chemical used to make polystyrene foam, commonly known as Styrofoam.

Green cells for green building

In our most recent work, published in Matter, we used photosynthetic cyanobacteria to help us grow a structural building material – and we kept it alive. Similar to algae, cyanobacteria are green microorganisms found throughout the environment but best known for growing on the walls in your fish tank. Instead of emitting CO2, cyanobacteria use CO2 and sunlight to grow and, in the right conditions, create a biocement, which we used to help us bind sand particles together to make a living brick.

By keeping the cyanobacteria alive, we were able to manufacture building materials exponentially. We took one living brick, split it in half and grew two full bricks from the halves. The two full bricks grew into four, and four grew into eight. Instead of creating one brick at a time, we harnessed the exponential growth of bacteria to grow many bricks at once – demonstrating a brand new method of manufacturing materials.

Researchers have only scratched the surface of the potential of engineered living materials. Other organisms could impart other living functions to material building blocks. For example, different bacteria could produce materials that heal themselves, sense and respond to external stimuli like pressure and temperature, or even light up. If nature can do it, living materials can be engineered to do it, too.

It also take less energy to produce living buildings than standard ones. Making and transporting today’s building materials uses a lot of energy and emits a lot of CO2. For example, limestone is burned to make cement for concrete. Metals and sand are mined and melted to make steel and glass. The manufacture, transport and assembly of building materials account for 11% of global CO2 emissions. Cement production alone accounts for 8%. In contrast, some living materials, like our cyanobacteria bricks, could actually sequester CO2.

The field of engineered living materials is in its infancy, and further research and development is needed to bridge the gap between laboratory research and commercial availability. Challenges include cost, testing, certification and scaling up production. Consumer acceptance is another issue. For example, the construction industry has a negative perception of living organisms. Think mold, mildew, spiders, ants and termites. We’re hoping to shift that perception. Researchers working on living materials also need to address concerns about safety and biocontamination.

The [US] National Science Foundation recently named engineered living materials one of the country’s key research priorities. Synthetic biology and engineered living materials will play a critical role in tackling the challenges humans will face in the 2020s and beyond: climate change, disaster resilience, aging and overburdened infrastructure, and space exploration.

If you have time and interest, this is fascinating. Strubar is a little exuberant and, at this point, I welcome it.

Fitness

The Lithuanians are here for us. Scientists from the Kaunas University of Technology have just published a paper on better exercises for lower back pain in our increasingly sedentary times, from a March 23, 2020 Kaunas University of Technology press release (also on EurekAlert) Note: There are a few minor grammatical issues,

With the significant part of the global population forced to work from home, the occurrence of lower back pain may increase. Lithuanian scientists have devised a spinal stabilisation exercise programme for managing lower back pain for people who perform a sedentary job. After testing the programme with 70 volunteers, the researchers have found that the exercises are not only efficient in diminishing the non-specific lower back pain, but their effect lasts 3 times longer than that of a usual muscle strengthening exercise programme.

According to the World Health Organisation, lower back pain is among the top 10 diseases and injuries that are decreasing the quality of life across the global population. It is estimated that non-specific low back pain is experienced by 60% to 70% of people in industrialised societies. Moreover, it is the leading cause of activity limitation and work absence throughout much of the world. For example, in the United Kingdom, low back pain causes more than 100 million workdays lost per year, in the United States – an estimated 149 million.

Chronic lower back pain, which starts from long-term irritation or nerve injury affects the emotions of the afflicted. Anxiety, bad mood and even depression, also the malfunctioning of the other bodily systems – nausea, tachycardia, elevated arterial blood pressure – are among the conditions, which may be caused by lower back pain.

During the coronavirus disease (COVID-19) outbreak, with a significant part of the global population working from home and not always having a properly designed office space, the occurrence of lower back pain may increase.

“Lower back pain is reaching epidemic proportions. Although it is usually clear what is causing the pain and its chronic nature, people tend to ignore these circumstances and are not willing to change their lifestyle. Lower back pain usually comes away itself, however, the chances of the recurring pain are very high”, says Dr Irina Klizienė, a researcher at Kaunas University of Technology (KTU) Faculty of Social Sciences, Humanities and Arts.

Dr Klizienė, together with colleagues from KTU and from Lithuanian Sports University has designed a set of stabilisation exercises aimed at strengthening the muscles which support the spine at the lower back, i.e. lumbar area. The exercise programme is based on Pilates methodology.

According to Dr Klizienė, the stability of lumbar segments is an essential element of body biomechanics. Previous research evidence shows that in order to avoid the lower back pain it is crucial to strengthen the deep muscles, which are stabilising the lumbar area of the spine. One of these muscles is multifidus muscle.

“Human central nervous system is using several strategies, such as preparing for keeping the posture, preliminary adjustment to the posture, correcting the mistakes of the posture, which need to be rectified by specific stabilising exercises. Our aim was to design a set of exercises for this purpose”, explains Dr Klizienė.

The programme, designed by Dr Klizienė and her colleagues is comprised of static and dynamic exercises, which train the muscle strength and endurance. The static positions are to be held from 6 to 20 seconds; each exercise to be repeated 8 to 16 times.

Caption: The static positions are to be held from 6 to 20 seconds; each exercise to be repeated 8 to 16 times. Credit: KTU

The previous set is a little puzzling but perhaps you’ll find these ones below easier to follow,

Caption: The exercises are aimed at strengthening the muscles which support the spine at the lower back. Credit: KTU

I think more pictures of intervening moves would have been useful. Now. getting back to the press release,

In order to check the efficiency of the programme, 70 female volunteers were randomly enrolled either to the lumbar stabilisation exercise programme or to a usual muscle strengthening exercise programme. Both groups were exercising twice a week for 45 minutes for 20 weeks. During the experiment, ultrasound scanning of the muscles was carried out.

As soon as 4 weeks in lumbar stabilisation programme, it was observed that the cross-section area of the multifidus muscle of the subjects of the stabilisation group has increased; after completing the programme, this increase was statistically significant (p < 0,05). This change was not observed in the strengthening group.

Moreover, although both sets of exercises were efficient in eliminating lower back pain and strengthening the muscles of the lower back area, the effect of stabilisation exercises lasted 3 times longer – 12 weeks after the completion of the stabilisation programme against 4 weeks after the completion of the muscle strengthening programme.

“There are only a handful of studies, which have directly compared the efficiency of stabilisation exercises against other exercises in eliminating lower back pain”, says Dr Klizienė, “however, there are studies proving that after a year, lower back pain returned only to 30% of people who have completed a stabilisation exercise programme, and to 84% of people who haven’t taken these exercises. After three years these proportions are 35% and 75%.”

According to her, research shows that the spine stabilisation exercises are more efficient than medical intervention or usual physical activities in curing the lower back pain and avoiding the recurrence of the symptoms in the future.

Here’s a link to and a citation for the paper,

Effect of different exercise programs on non-specific chronic low back pain and disability in people who perform sedentary work by Saule Sipavicienea, Irina Klizieneb. Clinical Biomechanics March 2020 Volume 73, Pages 17–27 DOI: https://doi.org/10.1016/j.clinbiomech.2019.12.028

This paper is behind a paywall.

University of Washington (state) is accelerating nanoscale research with Institute for Nano-Engineered Systems

A December 5, 2017 news item on Nanowerk announced a new research institute at the University of Washington (state),

The University of Washington [UW} has launched a new institute aimed at accelerating research at the nanoscale: the Institute for Nano-Engineered Systems, or NanoES. Housed in a new, multimillion-dollar facility on the UW’s Seattle campus, the institute will pursue impactful advancements in a variety of disciplines — including energy, materials science, computation and medicine. Yet these advancements will be at a technological scale a thousand times smaller than the width of a human hair.

The institute was launched at a reception Dec. 4 [2017] at its headquarters in the $87.8-million Nano Engineering and Sciences Building. During the event, speakers including UW officials and NanoES partners celebrated the NanoES mission to capitalize on the university’s strong record of research at the nanoscale and engage partners in industry at the onset of new projects.

A December 5, 2017 UW news release, which originated the news item, somewhat clarifies the declarations in the two excerpted paragraphs in the above,

The vision of the NanoES, which is part of the UW’s College of Engineering, is to act as a magnet for researchers in nanoscale science and engineering, with a focus on enabling industry partnership and entrepreneurship at the earliest stages of research projects. According to Karl Böhringer, director of the NanoES and a UW professor of electrical engineering and bioengineering, this unique approach will hasten the development of solutions to the field’s most pressing challenges: the manufacturing of scalable, high-yield nano-engineered systems for applications in information processing, energy, health and interconnected life.

“The University of Washington is well known for its expertise in nanoscale materials, processing, physics and biology — as well as its cutting-edge nanofabrication, characterization and testing facilities,” said Böhringer, who stepped down as director of the UW-based Washington Nanofabrication Facility to lead the NanoES. “NanoES will build on these strengths, bringing together people, tools and opportunities to develop nanoscale devices and systems.”

The centerpiece of the NanoES is its headquarters, the Nano Engineering and Sciences Building. The building houses 90,300 square feet of research and learning space, and was funded largely by the College of Engineering and Sound Transit. It contains an active learning classroom, a teaching laboratory and a 3,000-square-foot common area designed expressly to promote the sharing and exchanging of ideas. The remainder includes “incubator-style” office space and more than 40,000 square feet of flexible multipurpose laboratory and instrumentation space. The building’s location and design elements are intended to limit vibrations and electromagnetic interference so it can house sensitive experiments.

NanoES will house research in nanotechnology fields that hold promise for high impact, such as:

  • Augmented humanity, which includes technology to both aid and replace human capability in a way that joins user and machine as one – and foresees portable, wearable, implantable and networked technology for applications such as personalized medical care, among others.
  • Integrated photonics, which ranges from single-photon sensors for health care diagnostic tests to large-scale, integrated networks of photonic devices.
  • Scalable nanomanufacturing, which aims to develop low-cost, high-volume manufacturing processes. These would translate device prototypes constructed in research laboratories into system- and network-level nanomanufacturing methods for applications ranging from the 3-D printing of cell and tissue scaffolds to ultrathin solar cells.

A ribbon cutting ceremony.

Cutting the ribbon for the NanoES on Dec. 4. Left-to-right: Karl Böhringer, director of the NanoES and a UW professor of electrical engineering and bioengineering; Nena Golubovic, physical sciences director for IP Group; Mike Bragg, Dean of the UW College of Engineering; Jevne Micheau-Cunningham, deputy director of the NanoES.Kathryn Sauber/University of Washington

Collaborations with other UW-based institutions will provide additional resources for the NanoES. Endeavors in scalable nanomanufacturing, for example, will rely on the roll-to-roll processing facility at the UW Clean Energy Institute‘s Washington Clean Energy Testbeds or on advanced surface characterization capabilities at the Molecular Analysis Facility. In addition, the Washington Nanofabrication Facility recently completed a three-year, $37 million upgrade to raise it to an ISO Class 5 nanofabrication facility.

UW faculty and outside collaborators will build new research programs in the Nano Engineering and Sciences Building. Eric Klavins, a UW professor of electrical engineering, recently moved part of his synthetic biology research team to the building, adjacent to his collaborators in the Molecular Engineering & Sciences Institute and the Institute for Protein Design.

“We are extremely excited about the interdisciplinary and collaborative potential of the new space,” said Klavins.

The NanoES also has already produced its first spin-out company, Tunoptix, which was co-founded by Böhringer and recently received startup funding from IP Group, a U.K.-based venture capital firm.

“IP Group is very excited to work with the University of Washington,” said Nena Golubovic, physical sciences director for IP Group. “We are looking forward to the new collaborations and developments in science and technology that will grow from this new partnership.”

A woman speaking at a podium.

Nena Golubovic, physical sciences director for IP Group, delivering remarks at the Dec. 4 opening of NanoES.Kathryn Sauber/University of Washington

“We are eager to work with our partners at the IP Group to bring our technology to the market, and we appreciate their vision and investment in the NanoES Integrated Photonics Initiative,” said Tunoptix entrepreneurial lead Mike Robinson. “NanoES was the ideal environment in which to start our company.”

The NanoES leaders hope to forge similar partnerships with researchers, investors and industry leaders to develop technologies for portable, wearable, implantable and networked nanotechnologies for personalized medical care, a more efficient interconnected life and interconnected mobility. In addition to expertise, personnel and state-of-the-art research space and equipment, the NanoES will provide training, research support and key connections to capital and corporate partners.

“We believe this unique approach is the best way to drive innovations from idea to fabrication to scale-up and testing,” said Böhringer. “Some of the most promising solutions to these huge challenges are rooted in nanotechnology.”

The NanoES is supported by funds from the College of Engineering and the National Science Foundation, as well as capital investments from investors and industry partners.

You can find out more about Nano ES here.