Tag Archives: University of Washington (UW)

Shape-changing speaker (aka acoustic swarms) for sound control

To alleviate any concerns, these swarms are not kin to Michael Crichton’s swarms in his 2002 novel, Prey or his 2011 novel, Micro (published after his death).

A September 21, 2023 news item on ScienceDaily announces this ‘acoustic swarm’ research,

In virtual meetings, it’s easy to keep people from talking over each other. Someone just hits mute. But for the most part, this ability doesn’t translate easily to recording in-person gatherings. In a bustling cafe, there are no buttons to silence the table beside you.

The ability to locate and control sound — isolating one person talking from a specific location in a crowded room, for instance — has challenged researchers, especially without visual cues from cameras.

A team led by researchers at the University of Washington has developed a shape-changing smart speaker, which uses self-deploying microphones to divide rooms into speech zones and track the positions of individual speakers. With the help of the team’s deep-learning algorithms, the system lets users mute certain areas or separate simultaneous conversations, even if two adjacent people have similar voices. Like a fleet of Roombas, each about an inch in diameter, the microphones automatically deploy from, and then return to, a charging station. This allows the system to be moved between environments and set up automatically. In a conference room meeting, for instance, such a system might be deployed instead of a central microphone, allowing better control of in-room audio.

The team published its findings Sept. 21 [2023] in Nature Communications.

A September 21, 2023 University of Washington (state) news release (also on EurekAlert), which originated the news item, delves further into the work, Note: Links have been removed,

“If I close my eyes and there are 10 people talking in a room, I have no idea who’s saying what and where they are in the room exactly. That’s extremely hard for the human brain to process. Until now, it’s also been difficult for technology,” said co-lead author Malek Itani, a UW doctoral student in the Paul G. Allen School of Computer Science & Engineering. “For the first time, using what we’re calling a robotic ‘acoustic swarm,’ we’re able to track the positions of multiple people talking in a room and separate their speech.”

Previous research on robot swarms has required using overhead or on-device cameras, projectors or special surfaces. The UW team’s system is the first to accurately distribute a robot swarm using only sound.

The team’s prototype consists of seven small robots that spread themselves across tables of various sizes. As they move from their charger, each robot emits a high frequency sound, like a bat navigating, using this frequency and other sensors to avoid obstacles and move around without falling off the table. The automatic deployment allows the robots to place themselves for maximum accuracy, permitting greater sound control than if a person set them. The robots disperse as far from each other as possible since greater distances make differentiating and locating people speaking easier. Today’s consumer smart speakers have multiple microphones, but clustered on the same device, they’re too close to allow for this system’s mute and active zones.

“If I have one microphone a foot away from me, and another microphone two feet away, my voice will arrive at the microphone that’s a foot away first. If someone else is closer to the microphone that’s two feet away, their voice will arrive there first,” said co-lead author Tuochao Chen, a UW doctoral student in the Allen School. “We developed neural networks that use these time-delayed signals to separate what each person is saying and track their positions in a space. So you can have four people having two conversations and isolate any of the four voices and locate each of the voices in a room.”

The team tested the robots in offices, living rooms and kitchens with groups of three to five people speaking. Across all these environments, the system could discern different voices within 1.6 feet (50 centimeters) of each other 90% of the time, without prior information about the number of speakers. The system was able to process three seconds of audio in 1.82 seconds on average — fast enough for live streaming, though a bit too long for real-time communications such as video calls.

As the technology progresses, researchers say, acoustic swarms might be deployed in smart homes to better differentiate people talking with smart speakers. That could potentially allow only people sitting on a couch, in an “active zone,” to vocally control a TV, for example.

Researchers plan to eventually make microphone robots that can move around rooms, instead of being limited to tables. The team is also investigating whether the speakers can emit sounds that allow for real-world mute and active zones, so people in different parts of a room can hear different audio. The current study is another step toward science fiction technologies, such as the “cone of silence” in “Get Smart” and“Dune,” the authors write.

Of course, any technology that evokes comparison to fictional spy tools will raise questions of privacy. Researchers acknowledge the potential for misuse, so they have included guards against this: The microphones navigate with sound, not an onboard camera like other similar systems. The robots are easily visible and their lights blink when they’re active. Instead of processing the audio in the cloud, as most smart speakers do, the acoustic swarms process all the audio locally, as a privacy constraint. And even though some people’s first thoughts may be about surveillance, the system can be used for the opposite, the team says.

“It has the potential to actually benefit privacy, beyond what current smart speakers allow,” Itani said. “I can say, ‘Don’t record anything around my desk,’ and our system will create a bubble 3 feet around me. Nothing in this bubble would be recorded. Or if two groups are speaking beside each other and one group is having a private conversation, while the other group is recording, one conversation can be in a mute zone, and it will remain private.”

Takuya Yoshioka, a principal research manager at Microsoft, is a co-author on this paper, and Shyam Gollakota, a professor in the Allen School, is a senior author. The research was funded by a Moore Inventor Fellow award.

Two of the paper`s authors, Malek Itani and Tuochao Chen, have written a ‘Behind the Paper’ article for Nature.com’s Electrical and Electronic Engineering Community, from their September 21, 2023 posting,

Sound is a versatile medium. In addition to being one of the primary means of communication for us humans, it serves numerous purposes for organisms across the animal kingdom. Particularly, many animals use sound to localize themselves and navigate in their environment. Bats, for example, emit ultrasonic sound pulses to move around and find food in the dark. Similar behavior can be observed in Beluga whales to avoid obstacles and locate one other.

Various animals also have a tendency to cluster together into swarms, forming a unit greater than the sum of its parts. Famously, bees agglomerate into swarms to more efficiently search for a new colony. Birds flock to evade predators. These behaviors have caught the attention of scientists for quite some time, inspiring a handful of models for crowd control, optimization and even robotics. 

A key challenge in building robot swarms for practical purposes is the ability for the robots to localize themselves, not just within the swarm, but also relative to other important landmarks. …

Here’s a link to and a citation for the paper,

Creating speech zones with self-distributing acoustic swarms by Malek Itani, Tuochao Chen, Takuya Yoshioka & Shyamnath Gollakota. Nature Communications volume 14, Article number: 5684 (2023) DOI: https://doi.org/10.1038/s41467-023-40869-8 Published: 21 September 2023

This paper is open access.

Filmmaking beetles wearing teeny, tiny wireless cameras

Researchers at the University of Washington have developed a tiny camera that can ride aboard an insect. Here a Pinacate beetle explores the UW campus with the camera on its back. Credit: Mark Stone/University of Washington

Scientists at Washington University have created a removable wireless camera backpack for beetles and for tiny robots resembling beetles. I’m embedding a video shot by a beetle later in this post with a citation and link for the paper, near the end of this post where you’ll also find links to my other posts on insects and technology.

As for the latest on insects and technology, there’s a July 15, 2020 news item on ScienceDaily,

In the movie “Ant-Man,” the title character can shrink in size and travel by soaring on the back of an insect. Now researchers at the University of Washington have developed a tiny wireless steerable camera that can also ride aboard an insect, giving everyone a chance to see an Ant-Man view of the world.

The camera, which streams video to a smartphone at 1 to 5 frames per second, sits on a mechanical arm that can pivot 60 degrees. This allows a viewer to capture a high-resolution, panoramic shot or track a moving object while expending a minimal amount of energy. To demonstrate the versatility of this system, which weighs about 250 milligrams — about one-tenth the weight of a playing card — the team mounted it on top of live beetles and insect-sized robots.

A July 15, 2020 University of Washington news release (also on EurekAlert), which originated the news item, provides more technical detail (although I still have a few questions) about the work,

“We have created a low-power, low-weight, wireless camera system that can capture a first-person view of what’s happening from an actual live insect or create vision for small robots,” said senior author Shyam Gollakota, a UW associate professor in the Paul G. Allen School of Computer Science & Engineering. “Vision is so important for communication and for navigation, but it’s extremely challenging to do it at such a small scale. As a result, prior to our work, wireless vision has not been possible for small robots or insects.”

Typical small cameras, such as those used in smartphones, use a lot of power to capture wide-angle, high-resolution photos, and that doesn’t work at the insect scale. While the cameras themselves are lightweight, the batteries they need to support them make the overall system too big and heavy for insects — or insect-sized robots — to lug around. So the team took a lesson from biology.

“Similar to cameras, vision in animals requires a lot of power,” said co-author Sawyer Fuller, a UW assistant professor of mechanical engineering. “It’s less of a big deal in larger creatures like humans, but flies are using 10 to 20% of their resting energy just to power their brains, most of which is devoted to visual processing. To help cut the cost, some flies have a small, high-resolution region of their compound eyes. They turn their heads to steer where they want to see with extra clarity, such as for chasing prey or a mate. This saves power over having high resolution over their entire visual field.”

To mimic an animal’s vision, the researchers used a tiny, ultra-low-power black-and-white camera that can sweep across a field of view with the help of a mechanical arm. The arm moves when the team applies a high voltage, which makes the material bend and move the camera to the desired position. Unless the team applies more power, the arm stays at that angle for about a minute before relaxing back to its original position. This is similar to how people can keep their head turned in one direction for only a short period of time before returning to a more neutral position.

“One advantage to being able to move the camera is that you can get a wide-angle view of what’s happening without consuming a huge amount of power,” said co-lead author Vikram Iyer, a UW doctoral student in electrical and computer engineering. “We can track a moving object without having to spend the energy to move a whole robot. These images are also at a higher resolution than if we used a wide-angle lens, which would create an image with the same number of pixels divided up over a much larger area.”

The camera and arm are controlled via Bluetooth from a smartphone from a distance up to 120 meters away, just a little longer than a football field.

The researchers attached their removable system to the backs of two different types of beetles — a death-feigning beetle and a Pinacate beetle. Similar beetles have been known to be able to carry loads heavier than half a gram, the researchers said.

“We made sure the beetles could still move properly when they were carrying our system,” said co-lead author Ali Najafi, a UW doctoral student in electrical and computer engineering. “They were able to navigate freely across gravel, up a slope and even climb trees.”

The beetles also lived for at least a year after the experiment ended. [emphasis mine]

“We added a small accelerometer to our system to be able to detect when the beetle moves. Then it only captures images during that time,” Iyer said. “If the camera is just continuously streaming without this accelerometer, we could record one to two hours before the battery died. With the accelerometer, we could record for six hours or more, depending on the beetle’s activity level.”

The researchers also used their camera system to design the world’s smallest terrestrial, power-autonomous robot with wireless vision. This insect-sized robot uses vibrations to move and consumes almost the same power as low-power Bluetooth radios need to operate.

The team found, however, that the vibrations shook the camera and produced distorted images. The researchers solved this issue by having the robot stop momentarily, take a picture and then resume its journey. With this strategy, the system was still able to move about 2 to 3 centimeters per second — faster than any other tiny robot that uses vibrations to move — and had a battery life of about 90 minutes.

While the team is excited about the potential for lightweight and low-power mobile cameras, the researchers acknowledge that this technology comes with a new set of privacy risks.

“As researchers we strongly believe that it’s really important to put things in the public domain so people are aware of the risks and so people can start coming up with solutions to address them,” Gollakota said.

Applications could range from biology to exploring novel environments, the researchers said. The team hopes that future versions of the camera will require even less power and be battery free, potentially solar-powered.

“This is the first time that we’ve had a first-person view from the back of a beetle while it’s walking around. There are so many questions you could explore, such as how does the beetle respond to different stimuli that it sees in the environment?” Iyer said. “But also, insects can traverse rocky environments, which is really challenging for robots to do at this scale. So this system can also help us out by letting us see or collect samples from hard-to-navigate spaces.”

###

Johannes James, a UW mechanical engineering doctoral student, is also a co-author on this paper. This research was funded by a Microsoft fellowship and the National Science Foundation.

I’m surprised there’s no funding from a military agency as the military and covert operation applications seem like an obvious pairing. In any event, here’s a link to and a citation for the paper,

Wireless steerable vision for live insects and insect-scale robots by Vikram Iyer, Ali Najafi, Johannes James, Sawyer Fuller, and Shyamnath Gollakota. Science Robotics 15 Jul 2020: Vol. 5, Issue 44, eabb0839 DOI: 10.1126/scirobotics.abb0839

This paper is behind a paywall.

Video and links

As promised, here’s the video the scientists have released,

These posts feature some fairly ruthless uses of the insects.

  1. The first mention of insects and technology here is in a July 27, 2009 posting titled: Nanotechnology enables robots and human enhancement: part 4. The mention is in the second to last paragraph of the post. Then,.
  2. A November 23, 2011 post titled: Cyborg insects and trust,
  3. A January 9, 2012 post titled: Controlling cyborg insects,
  4. A June 26, 2013 post titled: Steering cockroaches in the lab and in your backyard—cutting edge neuroscience, and, finally,
  5. An April 11, 2014 post titled: Computerized cockroaches as precursors to new healing techniques.

As for my questions (how do you put the backpacks on the beetles? is there a strap, is it glue, is it something else? how heavy is the backpack and camera? how old are the beetles you use for this experiment? where did you get the beetles from? do you have your own beetle farm where you breed them?), I’ll see if I can get some answers.

Democratizing science .. neuroscience that is

What is going on with the neuroscience folks? First it was Montreal Neuro opening up its science  as featured in my January 22, 2016 posting,

The Montreal Neurological Institute (MNI) in Québec, Canada, known informally and widely as Montreal Neuro, has ‘opened’ its science research to the world. David Bruggeman tells the story in a Jan. 21, 2016 posting on his Pasco Phronesis blog (Note: Links have been removed),

The Montreal Neurological Institute (MNI) at McGill University announced that it will be the first academic research institute to become what it calls ‘Open Science.’  As Science is reporting, the MNI will make available all research results and research data at the time of publication.  Additionally it will not seek patents on any of the discoveries made on research at the Institute.

Will this catch on?  I have no idea if this particular combination of open access research data and results with no patents will spread to other university research institutes.  But I do believe that those elements will continue to spread.  More universities and federal agencies are pursuing open access options for research they support.  Elon Musk has opted to not pursue patent litigation for any of Tesla Motors’ patents, and has not pursued patents for SpaceX technology (though it has pursued litigation over patents in rocket technology). …

Whether or not they were inspired by the MNI, the scientists at the University of Washington (UW [state]) have found their own unique way of opening up science. From a March 15, 2018 UW news blog posting (also on EurekAlert) by James Urton, Note: Links have been removed,

Over the past few years, scientists have faced a problem: They often cannot reproduce the results of experiments done by themselves or their peers.

This “replication crisis” plagues fields from medicine to physics, and likely has many causes. But one is undoubtedly the difficulty of sharing the vast amounts of data collected and analyses performed in so-called “big data” studies. The volume and complexity of the information also can make these scientific endeavors unwieldy when it comes time for researchers to share their data and findings with peers and the public.

Researchers at the University of Washington have developed a set of tools to make one critical area of big data research — that of our central nervous system — easier to share. In a paper published online March 5 [2018] in Nature Communications, the UW team describes an open-access browser they developed to display, analyze and share neurological data collected through a type of magnetic resonance imaging study known as diffusion-weighted MRI.

“There has been a lot of talk among researchers about the replication crisis,” said lead author Jason Yeatman. “But we wanted a tool — ready, widely available and easy to use — that would actually help fight the replication crisis.”

Yeatman — who is an assistant professor in the UW Department of Speech & Hearing Sciences and the Institute for Learning & Brain Sciences (I-LABS) — is describing AFQ-Browser. This web browser-based tool, freely available online, is a platform for uploading, visualizing, analyzing and sharing diffusion MRI data in a format that is publicly accessible, improving transparency and data-sharing methods for neurological studies. In addition, since it runs in the web browser, AFQ-Browser is portable — requiring no additional software package or equipment beyond a computer and an internet connection.

“One major barrier to data transparency in neuroscience is that so much data collection, storage and analysis occurs on local computers with special software packages,” said senior author Ariel Rokem, a senior data scientist in the UW eScience Institute. “But using AFQ-Browser, we eliminate those requirements and make uploading, sharing and analyzing diffusion-weighted MRI data a simple, straightforward process.”

Diffusion-weighted MRI measures the movement of fluid in the brain and spinal cord, revealing the structure and function of white-matter tracts. These are the connections of the central nervous system, tissue that are made up primarily of axons that transmit long-range signals between neural circuits. Diffusion MRI research on brain connectivity has fundamentally changed the way neuroscientists understand human brain function: The state, organization and layout of white matter tracts are at the core of cognitive functions such as memory, learning and other capabilities. Data collected using diffusion-weighted MRI can be used to diagnose complex neurological conditions such as multiple sclerosis (MS) and amyotrophic lateral sclerosis (ALS). Researchers also use diffusion-weighted MRI data to study the neurological underpinnings of conditions such as dyslexia and learning disabilities.

“This is a widely-used technique in neuroscience research, and it is particularly amenable to the benefits that can be gleaned from big data, so it became a logical starting point for developing browser-based, open-access tools for the field,” said Yeatman.

The AFQ-Browser — the AFQ stands for Automated Fiber-tract Quantification — can receive diffusion-weighted MRI data and perform tract analysis for each individual subject. The analyses occur via a remote server, again eliminating technical and financial barriers for researchers. The AFQ-Browser also contains interactive tools to display data for multiple subjects — allowing a researcher to easily visualize how white matter tracts might be similar or different among subjects, identify trends in the data and generate hypotheses for future experiments. Researchers also can insert additional code to analyze the data, as well as save, upload and share data instantly with fellow researchers.

“We wanted this tool to be as generalizable as possible, regardless of research goals,” said Rokem. “In addition, the format is easy for scientists from a variety of backgrounds to use and understand — so that neuroscientists, statisticians and other researchers can collaborate, view data and share methods toward greater reproducibility.”

The idea for the AFQ-Browser came out of a UW course on data visualization, and the researchers worked with several graduate students to develop and perfect the browser. They tested it on existing diffusion-weighted MRI datasets, including research subjects with ALS and MS. In the future, they hope that the AFQ-Browser can be improved to do automated analyses — and possibly even diagnoses — based on diffusion-weighted MRI data.

“AFQ-Browser is really just the start of what could be a number of tools for sharing neuroscience data and experiments,” said Yeatman. “Our goal here is greater reproducibility and transparency, and a more robust scientific process.”

Here are a couple of images the researchers have used to illustrate their work,

AFQ-Browser.Jason Yeatman/Ariel Rokem Courtesy: University of Washington

Depiction of the left hemisphere of the human brain. Colored regions are selected white matter regions that could be measured using diffusion-weighted MRI: Corticospinal tract (orange), arcuate fasciculus (blue) and cingulum (green).Jason Yeatman/Ariel Rokem

You can find an embedded version of the AFQ-Browser here: http://www.washington.edu/news/2018/03/15/democratizing-science-researchers-make-neuroscience-experiments-easier-to-share-reproduce/ (scroll down about 50 – 55% of the way).

As for the paper, here’s a link and a citation,

A browser-based tool for visualization and analysis of diffusion MRI data by Jason D. Yeatman, Adam Richie-Halford, Josh K. Smith, Anisha Keshavan, & Ariel Rokem. Nature Communicationsvolume 9, Article number: 940 (2018) doi:10.1038/s41467-018-03297-7 Published online: 05 March 2018

Fittingly, this paper is open access.