A May 16, 2022 news item on phys.org announces work on a new machine learning model that could be useful in the research into engineered nanoparticles for medical purposes (Note: Links have been removed),
With antibiotic-resistant infections on the rise and a continually morphing pandemic virus, it’s easy to see why researchers want to be able to design engineered nanoparticles that can shut down these infections.
A new machine learning model that predicts interactions between nanoparticles and proteins, developed at the University of Michigan, brings us a step closer to that reality.
“We have reimagined nanoparticles to be more than mere drug delivery vehicles. We consider them to be active drugs in and of themselves,” said J. Scott VanEpps, an assistant professor of emergency medicine and an author of the study in Nature Computational Science.
Discovering drugs is a slow and unpredictable process, which is why so many antibiotics are variations on a previous drug. Drug developers would like to design medicines that can attack bacteria and viruses in ways that they choose, taking advantage of the “lock-and-key” mechanisms that dominate interactions between biological molecules. But it was unclear how to transition from the abstract idea of using nanoparticles to disrupt infections to practical implementation of the concept.
“By applying mathematical methods to protein-protein interactions, we have streamlined the design of nanoparticles that mimic one of the proteins in these pairs,” said Nicholas Kotov, the Irving Langmuir Distinguished University Professor of Chemical Sciences and Engineering and corresponding author of the study.
“Nanoparticles are more stable than biomolecules and can lead to entirely new classes of antibacterial and antiviral agents.”
The new machine learning algorithm compares nanoparticles to proteins using three different ways to describe them. While the first was a conventional chemical description, the two that concerned structure turned out to be most important for making predictions about whether a nanoparticle would be a lock-and-key match with a specific protein.
Between them, these two structural descriptions captured the protein’s complex surface and how it might reconfigure itself to enable lock-and-key fits. This includes pockets that a nanoparticle could fit into, along with the size such a nanoparticle would need to be. The descriptions also included chirality, a clockwise or counterclockwise twist that is important for predicting how a protein and nanoparticle will lock in.
“There are many proteins outside and inside bacteria that we can target. We can use this model as a first screening to discover which nanoparticles will bind with which proteins,” said Emine Sumeyra Turali Emre, a postdoctoral researcher in chemical engineering and co-first author of the paper, along with Minjeong Cha, a PhD student in materials science and engineering.
Emre and Cha explained that researchers could follow up on matches identified by their algorithm with more detailed simulations and experiments. One such match could stop the spread of MRSA, a common antibiotic-resistant strain, using zinc oxide nanopyramids that block metabolic enzymes in the bacteria.
“Machine learning algorithms like ours will provide a design tool for nanoparticles that can be used in many biological processes. Inhibition of the virus that causes COVID-19 is one good example,” said Cha. “We can use this algorithm to efficiently design nanoparticles that have broad-spectrum antiviral activity against all variants.”
This breakthrough was enabled by the Blue Sky Initiative at the University of Michigan College of Engineering. It provided $1.5 million to support the interdisciplinary team carrying out the fundamental exploration of whether a machine learning approach could be effective when data on the biological activity of nanoparticles is so sparse.
“The core of the Blue Sky idea is exactly what this work covers: finding a way to represent proteins and nanoparticles in a unified approach to understand and design new classes of drugs that have multiple ways of working against bacteria,” said Angela Violi, an Arthur F. Thurnau Professor, a professor of mechanical engineering and leader of the nanobiotics Blue Sky project.
Emre led the building of a database of interactions between proteins that could help to predict nanoparticle and protein interaction. Cha then identified structural descriptors that would serve equally well for nanoparticles and proteins, working with collaborators at the University of Southern California, Los Angeles to develop a machine learning algorithm that combed through the database and used the patterns it found to predict how proteins and nanoparticles would interact with one another. Finally, the team compared these predictions for lock-and-key matches with the results from experiments and detailed simulations, finding that they closely matched.
Additional collaborators on the project include Ji-Young Kim, a postdoctoral researcher in chemical engineering at U-M, who helped calculate chirality in the proteins and nanoparticles. Paul Bogdan and Xiongye Xiao, a professor and PhD student, respectively, in electrical and computer engineering at USC [University of Southern California] contributed to the graph theory descriptors. Cha then worked with them to design and train the neural network, comparing different machine learning models. All authors helped analyze the data.
Here are links to and a citation for the research briefing and paper, respectively,
Who is an artist? What is an artist? Can everyone be an artist? These are the kinds of questions you can expect with the rise of artificially intelligent artists/collaborators. Of course, these same questions have been asked many times before the rise of AI (artificial intelligence) agents/programs in the field of visual art. Each time the questions are raised is an opportunity to examine our beliefs from a different perspective. And, not to be forgotten, there are questions about money.
First, the ‘art’,
Shanti Escalante-De Mattei’s September 1, 2022 article for ArtNews.com provides an overview of the latest AI art controversy (Note: A link has been removed),
The debate around AI art went viral once again when a man won first place at the Colorado State Fair’s art competition in the digital category with a work he made using text-to-image AI generator Midjourney.
Twitter user and digital artist Genel Jumalon tweeted out a screenshot from a Discord channel in which user Sincarnate, aka game designer Jason Allen, celebrated his win at the fair. Jumalon wrote, “Someone entered an art competition with an AI-generated piece and won the first prize. Yeah that’s pretty fucking shitty.”
The comments on the post range from despair and anger as artists, both digital and traditional, worry that their livelihoods might be at stake after years of believing that creative work would be safe from AI-driven automation. [emphasis mine]
Rachel Metz’s September 3, 2022 article for CNN provides more details about how the work was generated (Note: Links have been removed),
Jason M. Allen was almost too nervous to enter his first art competition. Now, his award-winning image is sparking controversy about whether art can be generated by a computer, and what, exactly, it means to be an artist.
In August , Allen, a game designer who lives in Pueblo West, Colorado, won first place in the emerging artist division’s “digital arts/digitally-manipulated photography” category at the Colorado State Fair Fine Arts Competition. His winning image, titled “Théâtre D’opéra Spatial” (French for “Space Opera Theater”), was made with Midjourney — an artificial intelligence system that can produce detailed images when fed written prompts. A $300 prize accompanied his win.
Allen’s winning image looks like a bright, surreal cross between a Renaissance and steampunk painting. It’s one of three such images he entered in the competition. In total, 11 people entered 18 pieces of art in the same category in the emerging artist division.
The definition for the category in which Allen competed states that digital art refers to works that use “digital technology as part of the creative or presentation process.” Allen stated that Midjourney was used to create his image when he entered the contest, he said.
The newness of these tools, how they’re used to produce images, and, in some cases, the gatekeeping for access to some of the most powerful ones has led to debates about whether they can truly make art or assist humans in making art.
This came into sharp focus for Allen not long after his win. Allen had posted excitedly about his win on Midjourney’s Discord server on August 25 , along with pictures of his three entries; it went viral on Twitter days later, with many artists angered by Allen’s win because of his use of AI to create the image, as a story by Vice’s Motherboard reported earlier this week.
“This sucks for the exact same reason we don’t let robots participate in the Olympics,” one Twitter user wrote.
“This is the literal definition of ‘pressed a few buttons to make a digital art piece’,” another Tweeted. “AI artwork is the ‘banana taped to the wall’ of the digital world now.”
Yet while Allen didn’t use a paintbrush to create his winning piece, there was plenty of work involved, he said.
“It’s not like you’re just smashing words together and winning competitions,” he said.
You can feed a phrase like “an oil painting of an angry strawberry” to Midjourney and receive several images from the AI system within seconds, but Allen’s process wasn’t that simple. To get the final three images he entered in the competition, he said, took more than 80 hours.
First, he said, he played around with phrasing that led Midjourney to generate images of women in frilly dresses and space helmets — he was trying to mash up Victorian-style costuming with space themes, he said. Over time, with many slight tweaks to his written prompt (such as to adjust lighting and color harmony), he created 900 iterations of what led to his final three images. He cleaned up those three images in Photoshop, such as by giving one of the female figures in his winning image a head with wavy, dark hair after Midjourney had rendered her headless. Then he ran the images through another software program called Gigapixel AI that can improve resolution and had the images printed on canvas at a local print shop.
Ars Technica has run a number of articles on the subject of Art and AI, Benj Edwards in an August 31, 2022 article seems to have been one of the first to comment on Jason Allen’s win (Note 1: Links have been removed; Note 2: Look at how Edwards identifies Jason Allen as an artist),
A synthetic media artist named Jason Allen entered AI-generated artwork into the Colorado State Fair fine arts competition and announced last week that he won first place in the Digital Arts/Digitally Manipulated Photography category, Vice reported Wednesday [August 31, 2022?] based on a viral tweet.
Allen’s victory prompted lively discussions on Twitter, Reddit, and the Midjourney Discord server about the nature of art and what it means to be an artist. Some commenters think human artistry is doomed thanks to AI and that all artists are destined to be replaced by machines. Others think art will evolve and adapt with new technologies that come along, citing synthesizers in music. It’s a hot debate that Wired covered in July .
It’s worth noting that the invention of the camera in the 1800s prompted similar criticism related to the medium of photography, since the camera seemingly did all the work compared to an artist that labored to craft an artwork by hand with a brush or pencil. Some feared that painters would forever become obsolete with the advent of color photography. In some applications, photography replaced more laborious illustration methods (such as engraving), but human fine art painters are still around today.
Benj Edwards in a September 12, 2022 article for Ars Technica examines how some art communities are responding (Note: Links have been removed),
Confronted with an overwhelming amount of artificial-intelligence-generated artwork flooding in, some online art communities have taken dramatic steps to ban or curb its presence on their sites, including Newgrounds, Inkblot Art, and Fur Affinity, according to Andy Baio of Waxy.org.
Baio, who has been following AI art ethics closely on his blog, first noticed the bans and reported about them on Friday [Sept. 9, 2022?]. …
The arrival of widely available image synthesis models such as Midjourney and Stable Diffusion has provoked an intense online battle between artists who view AI-assisted artwork as a form of theft (more on that below) and artists who enthusiastically embrace the new creative tools.
… a quickly evolving debate about how art communities (and art professionals) can adapt to software that can potentially produce unlimited works of beautiful art at a rate that no human working without the tools could match.
A few weeks ago, some artists began discovering their artwork in the Stable Diffusion data set, and they weren’t happy about it. Charlie Warzel wrote a detailed report about these reactions for The Atlantic last week [September 7, 2022]. With battle lines being drawn firmly in the sand and new AI creativity tools coming out steadily, this debate will likely continue for some time to come.
Filthy lucre becomes more prominent in the conversation
Lizzie O’Leary in a September 12, 2022 article for Fast Company presents a transcript of an interview (from the TBD podcast) she conducted with Drew Harwell, tech reporter covering A.I. for Washington Post) about the ‘Jason Allen’ win,
I’m struck by how quickly these art A.I.s are advancing. DALL-E was released in January of last year and there were some pretty basic images. And then, a year later, DALL-E 2 is using complex, faster methods. Midjourney, the one Jason Allen used, has a feature that allows you to upscale and downscale images. Where is this sudden supply and demand for A.I. art coming from?
You could look back to five years ago when they had these text-to-image generators and the output would be really crude. You could sort of see what the A.I. was trying to get at, but we’ve only really been able to cross that photorealistic uncanny valley in the last year or so. And I think the things that have contributed to that are, one, better data. You’re seeing people invest a lot of money and brainpower and resources into adding more stuff into bigger data sets. We have whole groups that are taking every image they can get on the internet. Billions, billions of images from Pinterest and Amazon and Facebook. You have bigger data sets, so the A.I. is learning more. You also have better computing power, and those are the two ingredients to any good piece of A.I. So now you have A.I. that is not only trained to understand the world a little bit better, but it can now really quickly spit out a very finely detailed generated image.
Is there any way to know, when you look at a piece of A.I. art, what images it referenced to create what it’s doing? Or is it just so vast that you can’t kind of unspool it backward?
When you’re doing an image that’s totally generated out of nowhere, it’s taking bits of information from billions of images. It’s creating it in a much more sophisticated way so that it’s really hard to unspool.
Art generated by A.I. isn’t just a gee-whiz phenomenon, something that wins prizes, or even a fascinating subject for debate—it has valuable commercial uses, too. Some that are a little frightening if you’re, say, a graphic designer.
You’re already starting to see some of these images illustrating news articles, being used as logos for companies, being used in the form of stock art for small businesses and websites. Anything where somebody would’ve gone and paid an illustrator or graphic designer or artist to make something, they can now go to this A.I. and create something in a few seconds that is maybe not perfect, maybe would be beaten by a human in a head-to-head, but is good enough. From a commercial perspective, that’s scary, because we have an industry of people whose whole job is to create images, now running up against A.I.
And the A.I., again, in the last five years, the A.I. has gotten better and better. It’s still not perfect. I don’t think it’ll ever be perfect, whatever that looks like. It processes information in a different, maybe more literal, way than a human. I think human artists will still sort of have the upper hand in being able to imagine things a little more outside of the box. And yet, if you’re just looking for three people in a classroom or a pretty simple logo, you’re going to go to A.I. and you’re going to take potentially a job away from a freelancer whom you would’ve given it to 10 years ago.
I can see a use case here in marketing, in advertising. The A.I. doesn’t need health insurance, it doesn’t need paid vacation days, and I really do wonder about this idea that the A.I. could replace the jobs of visual artists. Do you think that is a legitimate fear, or is that overwrought at this moment?
I think it is a legitimate fear. When something can mirror your skill set, not 100 percent of the way, but enough of the way that it could replace you, that’s an issue. Do these A.I. creators have any kind of moral responsibility to not create it because it could put people out of jobs? I think that’s a debate, but I don’t think they see it that way. They see it like they’re just creating the new generation of digital camera, the new generation of Photoshop. But I think it is worth worrying about because even compared with cameras and Photoshop, the A.I. is a little bit more of the full package and it is so accessible and so hard to match in terms. It’s really going to be up to human artists to find some way to differentiate themselves from the A.I.
This is making me wonder about the humans underneath the data sets that the A.I. is trained on. The criticism is, of course, that these businesses are making money off thousands of artists’ work without their consent or knowledge and it undermines their work. Some people looked at the Stable Diffusion and they didn’t have access to its whole data set, but they found that Thomas Kinkade, the landscape painter, was the most referenced artist in the data set. Is the A.I. just piggybacking? And if it’s not Thomas Kinkade, if it’s someone who’s alive, are they piggybacking on that person’s work without that person getting paid?
Here’s a bit more on the topic of money and art in a September 19, 2022 article by John Herrman for New York Magazine. First, he starts with the literary arts, Note: Links have been removed,
Artificial-intelligence experts are excited about the progress of the past few years. You can tell! They’ve been telling reporters things like “Everything’s in bloom,” “Billions of lives will be affected,” and “I know a person when I talk to it — it doesn’t matter whether they have a brain made of meat in their head.”
We don’t have to take their word for it, though. Recently, AI-powered tools have been making themselves known directly to the public, flooding our social feeds with bizarre and shocking and often very funny machine-generated content. OpenAI’s GPT-3 took simple text prompts — to write a news article about AI or to imagine a rose ceremony from The Bachelor in Middle English — and produced convincing results.
Deepfakes graduated from a looming threat to something an enterprising teenager can put together for a TikTok, and chatbots are occasionally sending their creators into crisis.
More widespread, and probably most evocative of a creative artificial intelligence, is the new crop of image-creation tools, including DALL-E, Imagen, Craiyon, and Midjourney, which all do versions of the same thing. You ask them to render something. Then, with models trained on vast sets of images gathered from around the web and elsewhere, they try — “Bart Simpson in the style of Soviet statuary”; “goldendoodle megafauna in the streets of Chelsea”; “a spaghetti dinner in hell”; “a logo for a carpet-cleaning company, blue and red, round”; “the meaning of life.”
This flood of machine-generated media has already altered the discourse around AI for the better, probably, though it couldn’t have been much worse. In contrast with the glib intra-VC debate about avoiding human enslavement by a future superintelligence, discussions about image-generation technology have been driven by users and artists and focus on labor, intellectual property, AI bias, and the ethics of artistic borrowing and reproduction [emphasis mine]. Early controversies have cut to the chase: Is the guy who entered generated art into a fine-art contest in Colorado (and won!) an asshole? Artists and designers who already feel underappreciated or exploited in their industries — from concept artists in gaming and film and TV to freelance logo designers — are understandably concerned about automation. Some art communities and marketplaces have banned AI-generated images entirely.
Requests are effectively thrown into “a giant swirling whirlpool” of “10,000 graphics cards,” Holz [David Holz, Midjourney founder] said, after which users gradually watch them take shape, gaining sharpness but also changing form as Midjourney refines its work.
This hints at an externality beyond the worlds of art and design. “Almost all the money goes to paying for those machines,” Holz said. New users are given a small number of free image generations before they’re cut off and asked to pay; each request initiates a massive computational task, which means using a lot of electricity.
High compute costs [emphasis mine] — which are largely energy costs — are why other services have been cautious about adding new users. …
Another Midjourney user, Gila von Meissner, is a graphic designer and children’s-book author-illustrator from “the boondocks in north Germany.” Her agent is currently shopping around a book that combines generated images with her own art and characters. Like Pluckebaum [Brian Pluckebaum who works in automotive-semiconductor marketing and designs board games], she brought up the balance of power with publishers. “Picture books pay peanuts,” she said. “Most illustrators struggle financially.” Why not make the work easier and faster? “It’s my character, my edits on the AI backgrounds, my voice, and my story.” A process that took months now takes a week, she said. “Does that make it less original?”
User MoeHong, a graphic designer and typographer for the state of California, has been using Midjourney to make what he called generic illustrations (“backgrounds, people at work, kids at school, etc.”) for government websites, pamphlets, and literature: “I get some of the benefits of using custom art — not that we have a budget for commissions! — without the paying-an-artist part.” He said he has mostly replaced stock art, but he’s not entirely comfortable with the situation. “I have a number of friends who are commercial illustrators, and I’ve been very careful not to show them what I’ve made,” he said. He’s convinced that tools like this could eventually put people in his trade out of work. “But I’m already in my 50s,” he said, “and I hope I’ll be gone by the time that happens.”
The last article I’m featuring here is a September 15, 2021 piece by Agnieszka Cichocka for DailyArt, which provides good, brief descriptions of algorithms, generative creative networks, machine learning, artificial neural networks, and more. She is an enthusiast (Note: Links have been removed),
I keep wondering if Leonardo da Vinci, who, in my opinion, was the most forward thinking artist of all time, would have ever imagined that art would one day be created by AI. He worked on numerous ideas and was constantly experimenting, and, although some were failures, he persistently tried new products, helping to move our world forward. Without such people, progress would not be possible.
As humans, we learn by acquiring knowledge through observations, senses, experiences, etc. This is similar to computers. Machine learning is a process in which a computer system learns how to perform a task better in two ways—either through exposure to environments that provide punishments and rewards (reinforcement learning) or by training with specific data sets (the system learns automatically and improves from previous experiences). Both methods help the systems improve their accuracy. Machines then use patterns and attempt to make an accurate analysis of things they have not seen before. To give an example, let’s say we feed the computer with thousands of photos of a dog. Consequently, it can learn what a dog looks like based on those. Later, even when faced with a picture it has never seen before, it can tell that the photo shows a dog.
If you want to see some creative machine learning experiments in art, check out ML x ART. This is a website with hundreds of artworks created using AI tools.
As the saying goes “a picture is worth a thousand words” and, now, It seems that pictures will be made from words or so suggests the example of Jason M. Allen feeding prompts to the AI system Midjourney.
I suspect (as others have suggested) that in the end, artists who use AI systems will be absorbed into the art world in much the same way as artists who use photography, or are considered performance artists and/or conceptual artists, and/or use video have been absorbed. There will be some displacements and discomfort as the questions I opened this posting with (Who is an artist? What is an artist? Can everyone be an artist?) are passionately discussed and considered. Underlying many of these questions is the issue of money.
The impact on people’s livelihoods is cheering or concerning depending on how the AI system is being used. Herrman’s September 19, 2022 article highlights two examples that focus on graphic designers. Gila von Meissner, the illustrator and designer, who uses her own art to illustrate her children’s books in a faster, more cost effective way with an AI system and MoeHong, a graphic designer for the state of California, who uses an AI system to make ‘customized generic art’ for which the state government doesn’t have to pay.
So far, the focus has been on Midjourney and other AI agents that have been created by developers for use by visual artists and writers. What happens when the visual artist or the writer is the developer? A September 12, 2022 article by Brandon Scott Roye for Cool Hunting approaches the question (Note: Links have been removed),
Mario Klingemann and Sasha Stiles on Semi-Autonomous AI Artists
An artist and engineer at the forefront of generating AI artwork, Mario Klingemann and first-generation Kalmyk-American poet, artist and researcher Sasha Stiles both approach AI from a more human, personal angle. Creators of semi-autonomous systems, both Klingemann and Stiles are the minds behind Botto and Technelegy, respectively. They are both artists in their own right, but their creations are too. Within web3, the identity of the “artist” who creates with visuals and the “writer” who creates with words is enjoying a foundational shift and expansion. Many have fashioned themselves a new title as “engineer.”
Based on their primary identities as an artist and poet, Klingemann and Stiles face the conundrum of becoming engineers who design the tools, rather than artists responsible for the final piece. They now have the ability to remove themselves from influencing inputs and outputs.
If you have time, I suggest reading Roye’s September 12, 2022 article as it provides some very interesting ideas although I don’t necessarily agree with them, e.g., “They now have the ability to remove themselves from influencing inputs and outputs.” Anyone who’s following the ethics discussion around AI knows that biases are built into the algorithms whether we like it or not. As for artists and writers calling themselves ‘engineers’, they may get a little resistance from the engineering community.
As users of open source software, Klingemann and Stiles should not have to worry too much about intellectual property. However, it seems copyright for the actual works and patents for the software could raise some interesting issues especially since money is involved.
Who gets the patent and/or the copyright? Assuming you and I are employing machine learning to train our AI agents separately, could there be an argument that if my version of the AI is different than yours and proves more popular with other content creators/ artists that I should own/share the patent to the software and rights to whatever the software produces?
Getting back to Herrman’s comment about high compute costs and energy, we seem to have an insatiable appetite for energy and that is not only a high cost financially but also environmentally.
Here’s more about Klingemann’s artist exhibition by Botto (from an October 6, 2022 announcement received via email),
Mario Klingemann is a pioneering figurehead in the field of AI art, working deep in the field of Machine Learning. Governed by a community of 5,000 people, Klingemann developed Botto around an idea of creating an autonomous entity that is able to be creative and co-creative. Inspired by Goethe’s artificial man in Faust, Botto is a genderless AI entity that is guided by an international community and art historical trends. Botto creates 350 art pieces per week that are presented to its community. Members of the community give feedback on these art fragments by voting, expressing their individual preferences on what is aesthetically pleasing to them. Then collectively the votes are used as feedback for Botto’s generative algorithm, dictating what direction Botto should take in its next series of art pieces.
The creative capacity of its algorithm is far beyond the capacities of an individual to combine and find relationships within all the information available to the AI. Botto faces similar issues as a human artist, and it is programmed to self-reflect and ask, “I’ve created this type of work before. What can I show them that’s different this week?”
Once a week, Botto auctions the art fragment with the most votes on SuperRare. All proceeds from the auction go back to the community. The AI artist auctioned its first three pieces, Asymmetrical Liberation, Scene Precede, and Trickery Contagion for more than $900,000 dollars, the most successful AI artist premiere. Today, Botto has produced upwards of 22 artworks and current sales have generated over $2 million in total [emphasis mine].
From March 2022 when Botto had made $1M to October 2022 where it’s made over $2M. It seems Botto is a very financially successful artist.
This exhibition (October 26 – 30, 2022) is being held in London, England at this location:
The Department Store, Brixton 248 Ferndale Road London SW9 8FR United Kingdom
I love science stories about the inspirational qualities of spiderwebs. A November 26, 2021 news item on phys.org describes how spiderwebs have inspired advances in sensors and, potentially, quantum computing,,
A team of researchers from TU Delft [Delft University of Technology; Netherlands] managed to design one of the world’s most precise microchip sensors. The device can function at room temperature—a ‘holy grail’ for quantum technologies and sensing. Combining nanotechnology and machine learning inspired by nature’s spiderwebs, they were able to make a nanomechanical sensor vibrate in extreme isolation from everyday noise. This breakthrough, published in the Advanced Materials Rising Stars Issue, has implications for the study of gravity and dark matter as well as the fields of quantum internet, navigation and sensing.
One of the biggest challenges for studying vibrating objects at the smallest scale, like those used in sensors or quantum hardware, is how to keep ambient thermal noise from interacting with their fragile states. Quantum hardware for example is usually kept at near absolute zero (−273.15°C) temperatures, with refrigerators costing half a million euros apiece. Researchers from TU Delft created a web-shaped microchip sensor which resonates extremely well in isolation from room temperature noise. Among other applications, their discovery will make building quantum devices much more affordable.
Hitchhiking on evolution Richard Norte and Miguel Bessa, who led the research, were looking for new ways to combine nanotechnology and machine learning. How did they come up with the idea to use spiderwebs as a model? Richard Norte: “I’ve been doing this work already for a decade when during lockdown, I noticed a lot of spiderwebs on my terrace. I realised spiderwebs are really good vibration detectors, in that they want to measure vibrations inside the web to find their prey, but not outside of it, like wind through a tree. So why not hitchhike on millions of years of evolution and use a spiderweb as an initial model for an ultra-sensitive device?”
Since the team did not know anything about spiderwebs’ complexities, they let machine learning guide the discovery process. Miguel Bessa: “We knew that the experiments and simulations were costly and time-consuming, so with my group we decided to use an algorithm called Bayesian optimization, to find a good design using few attempts.” Dongil Shin, co-first author in this work, then implemented the computer model and applied the machine learning algorithm to find the new device design.
Microchip sensor based on spiderwebs To the researcher’s surprise, the algorithm proposed a relatively simple spiderweb out of 150 different spiderweb designs, which consists of only six strings put together in a deceivingly simple way. Bessa: “Dongil’s computer simulations showed that this device could work at room temperature, in which atoms vibrate a lot, but still have an incredibly low amount of energy leaking in from the environment – a higher Quality factor in other words. With machine learning and optimization we managed to adapt Richard’s spider web concept towards this much better quality factor.”
Based on this new design, co-first author Andrea Cupertino built a microchip sensor with an ultra-thin, nanometre-thick film of ceramic material called Silicon Nitride. They tested the model by forcefully vibrating the microchip ‘web’ and measuring the time it takes for the vibrations to stop. The result was spectacular: a record-breaking isolated vibration at room temperature. Norte: “We found almost no energy loss outside of our microchip web: the vibrations move in a circle on the inside and don’t touch the outside. This is somewhat like giving someone a single push on a swing, and having them swing on for nearly a century without stopping.”
Implications for fundamental and applied sciences With their spiderweb-based sensor, the researchers’ show how this interdisciplinary strategy opens a path to new breakthroughs in science, by combining bio-inspired designs, machine learning and nanotechnology. This novel paradigm has interesting implications for quantum internet, sensing, microchip technologies and fundamental physics: exploring ultra-small forces for example, like gravity or dark matter which are notoriously difficult to measure. According to the researchers, the discovery would not have been possible without the university’s Cohesion grant, which led to this collaboration between nanotechnology and machine learning.
If spiderwebs can be sensors, can they also think?
it’s called ‘extended cognition’ or ‘extended mind thesis’ (Wikipedia entry) and the theory holds that the mind is not solely in the brain or even in the body. Predictably, the theory has both its supporters and critics as noted in Joshua Sokol’s article “The Thoughts of a Spiderweb” originally published on May 22, 2017 in Quanta Magazine (Note: Links have been removed),
Millions of years ago, a few spiders abandoned the kind of round webs that the word “spiderweb” calls to mind and started to focus on a new strategy. Before, they would wait for prey to become ensnared in their webs and then walk out to retrieve it. Then they began building horizontal nets to use as a fishing platform. Now their modern descendants, the cobweb spiders, dangle sticky threads below, wait until insects walk by and get snagged, and reel their unlucky victims in.
In 2008, the researcher Hilton Japyassú prompted 12 species of orb spiders collected from all over Brazil to go through this transition again. He waited until the spiders wove an ordinary web. Then he snipped its threads so that the silk drooped to where crickets wandered below. When a cricket got hooked, not all the orb spiders could fully pull it up, as a cobweb spider does. But some could, and all at least began to reel it in with their two front legs.
Their ability to recapitulate the ancient spiders’ innovation got Japyassú, a biologist at the Federal University of Bahia in Brazil, thinking. When the spider was confronted with a problem to solve that it might not have seen before, how did it figure out what to do? “Where is this information?” he said. “Where is it? Is it in her head, or does this information emerge during the interaction with the altered web?”
In February , Japyassú and Kevin Laland, an evolutionary biologist at the University of Saint Andrews, proposed a bold answer to the question. They argued in a review paper, published in the journal Animal Cognition, that a spider’s web is at least an adjustable part of its sensory apparatus, and at most an extension of the spider’s cognitive system.
This would make the web a model example of extended cognition, an idea first proposed by the philosophers Andy Clark and David Chalmers in 1998 to apply to human thought. In accounts of extended cognition, processes like checking a grocery list or rearranging Scrabble tiles in a tray are close enough to memory-retrieval or problem-solving tasks that happen entirely inside the brain that proponents argue they are actually part of a single, larger, “extended” mind.
Among philosophers of mind, that idea has racked up citations, including supporters and critics. And by its very design, Japyassú’s paper, which aims to export extended cognition as a testable idea to the field of animal behavior, is already stirring up antibodies among scientists. …
It seems there is no definitive answer to the question of whether there is an ‘extended mind’ but it’s an intriguing question made (in my opinion) even more so with the spiderweb-inspired sensors from TU Delft.
Does artificial intelligence have a place in such a fickle and quirky environment as the secondary art market? Can an algorithm learn to predict the value assigned to an artwork at auction?
These questions, among others, were analysed by a group of researchers including Roman Kräussl, professor at the Department of Finance at the University of Luxembourg and co-authors Mathieu Aubry (École des Ponts ParisTech), Gustavo Manso (Haas School of Business, University of California at Berkeley), and Christophe Spaenjers (HEC Paris). The resulting paper, Biased Auctioneers, has been accepted for publication in the top-ranked Journal of Finance.
Training a neural network to appraise art
In this study, which combines fields of finance and computer science, researchers used machine learning and artificial intelligence to create a neural network algorithm that mimics the work of human appraisers by generating price predictions for art at auction. This algorithm relies on data using both visual and non-visual characteristics of artwork. The authors of this study unleashed their algorithm on a vast set of art sales data capturing 1.2 million painting auctions from 2008 to 2014, training the neural network with both an image of the artwork, and information such as the artist, the medium and the auction house where the work was sold. Once trained to this dataset, the authors asked the neural network to predict the auction house pre-sale estimates, ‘buy-in’ price (the minimum price at which the work will be sold), as well as the final auction price for art sales in the year 2015. It became then possible to compare the algorithm’s estimate with the real-word data, and determine whether the relative level of the machine-generated price predictions predicts relative price outcomes.
The path towards a more efficient market?
Not too surprisingly, the human experts’ predications [sic] were more accurate than the algorithm, whose prediction, in turn, was more accurate than the standard linear hedonic model which researchers used to benchmark the study. Reasons for the discrepancy between human and machine include, as the authors argue, mainly access to a larger amount of information about the individual works of art including provenance, condition and historical context. Although interesting, the authors’ goal was not to pit human against machine on this specific task. On the contrary, the authors aimed at discovering the usefulness and potential applications of machine-based valuations. For example, using such an algorithm, it may be possible to determine whether an auctioneer’s pre-sale valuations are too pessimistic or too optimistic, effectively predicting the prediction errors of the auctioneers. Ultimately, this information could be used to correct for these kinds of man-made market inefficiencies.
Beyond the auction block
The implications of this methodology and the applied computational power, however, is not limited to the art world. Other markets trading in ‘real’ assets, which rely heavily on human appraisers, namely the real estate market, can benefit from the research. While AI is not likely to replace humans just yet, machine-learning technology as demonstrated by the researchers may become an important tool for investors and intermediaries, who wish to gain access to as much information, as quickly and as cheaply as possible.
Here’s a link to and a citation for the paper,
Biased Auctioneers by Mathieu Aubry, Roman Kräussl, Gustavo Manso, and Christophe Spaenjers. Journal of Finance, Forthcoming [print issue], Available at SSRN: https://ssrn.com/abstract=3347175 or http://dx.doi.org/10.2139/ssrn.3347175 Published online: January 6, 2022
This paper appears to be open access online and was last revised on January 13, 2022.
It seems machine learning is getting a tune-up. A November 29, 2021 news item on ScienceDaily describes research into improving machine learning from an international team of researchers,
Researchers have developed a new approach to machine learning that ‘learns how to learn’ and out-performs current machine learning methods for drug design, which in turn could accelerate the search for new disease treatments.
The method, called transformational machine learning (TML), was developed by a team from the UK, Sweden, India and Netherlands. It learns from multiple problems and improves performance while it learns.
TML could accelerate the identification and production of new drugs by improving the machine learning systems which are used to identify them. The results are reported in the Proceedings of the National Academy of Sciences.
Most types of machine learning (ML) use labelled examples, and these examples are almost always represented in the computer using intrinsic features, such as the colour or shape of an object. The computer then forms general rules that relate the features to the labels.
“It’s sort of like teaching a child to identify different animals: this is a rabbit, this is a donkey and so on,” said Professor Ross King from Cambridge’s Department of Chemical Engineering and Biotechnology, who led the research. “If you teach a machine learning algorithm what a rabbit looks like, it will be able to tell whether an animal is or isn’t a rabbit. This is the way that most machine learning works – it deals with problems one at a time.”
However, this is not the way that human learning works: instead of dealing with a single issue at a time, we get better at learning because we have learned things in the past.
“To develop TML, we applied this approach to machine learning, and developed a system that learns information from previous problems it has encountered in order to better learn new problems,” said King, who is also a Fellow at The Alan Turing Institute. “Where a typical ML system has to start from scratch when learning to identify a new type of animal – say a kitten – TML can use the similarity to existing animals: kittens are cute like rabbits, but don’t have long ears like rabbits and donkeys. This makes TML a much more powerful approach to machine learning.”
The researchers demonstrated the effectiveness of their idea on thousands of problems from across science and engineering. They say it shows particular promise in the area of drug discovery, where this approach speeds up the process by checking what other ML models say about a particular molecule. A typical ML approach will search for drug molecules of a particular shape, for example. TML instead uses the connection of the drugs to other drug discovery problems.
“I was surprised how well it works – better than anything else we know for drug design,” said King. “It’s better at choosing drugs than humans are – and without the best science, we won’t get the best results.”
It’s hard to believe that a brain-on-a-chip might need sleep but that seems to be the case as far as the US Dept. of Energy’s Los Alamos National Laboratory is concerned. Before pursuing that line of thought, here’s some work from the Massachusetts Institute of Technology (MIT) involving memristors and a brain-on-a-chip. From a June 8, 2020 news item on ScienceDaily,
MIT engineers have designed a “brain-on-a-chip,” smaller than a piece of confetti, that is made from tens of thousands of artificial brain synapses known as memristors — silicon-based components that mimic the information-transmitting synapses in the human brain.
The researchers borrowed from principles of metallurgy to fabricate each memristor from alloys of silver and copper, along with silicon. When they ran the chip through several visual tasks, the chip was able to “remember” stored images and reproduce them many times over, in versions that were crisper and cleaner compared with existing memristor designs made with unalloyed elements.
Their results, published today in the journal Nature Nanotechnology, demonstrate a promising new memristor design for neuromorphic devices — electronics that are based on a new type of circuit that processes information in a way that mimics the brain’s neural architecture. Such brain-inspired circuits could be built into small, portable devices, and would carry out complex computational tasks that only today’s supercomputers can handle.
This ‘metallurgical’ approach differs somewhat from the protein nanowire approach used by the University of Massachusetts at Amherst team mentioned in my June 15, 2020 posting. Scientists are pursuing multiple pathways and we may find that we arrive with not ‘a single artificial brain but with many types of artificial brains.
“So far, artificial synapse networks exist as software. We’re trying to build real neural network hardware for portable artificial intelligence systems,” says Jeehwan Kim, associate professor of mechanical engineering at MIT. “Imagine connecting a neuromorphic device to a camera on your car, and having it recognize lights and objects and make a decision immediately, without having to connect to the internet. We hope to use energy-efficient memristors to do those tasks on-site, in real-time.”
Memristors, or memory transistors [Note: Memristors are usually described as memory resistors; this is the first time I’ve seen ‘memory transistor’], are an essential element in neuromorphic computing. In a neuromorphic device, a memristor would serve as the transistor in a circuit, though its workings would more closely resemble a brain synapse — the junction between two neurons. The synapse receives signals from one neuron, in the form of ions, and sends a corresponding signal to the next neuron.
A transistor in a conventional circuit transmits information by switching between one of only two values, 0 and 1, and doing so only when the signal it receives, in the form of an electric current, is of a particular strength. In contrast, a memristor would work along a gradient, much like a synapse in the brain. The signal it produces would vary depending on the strength of the signal that it receives. This would enable a single memristor to have many values, and therefore carry out a far wider range of operations than binary transistors.
Like a brain synapse, a memristor would also be able to “remember” the value associated with a given current strength, and produce the exact same signal the next time it receives a similar current. This could ensure that the answer to a complex equation, or the visual classification of an object, is reliable — a feat that normally involves multiple transistors and capacitors.
Ultimately, scientists envision that memristors would require far less chip real estate than conventional transistors, enabling powerful, portable computing devices that do not rely on supercomputers, or even connections to the Internet.
Existing memristor designs, however, are limited in their performance. A single memristor is made of a positive and negative electrode, separated by a “switching medium,” or space between the electrodes. When a voltage is applied to one electrode, ions from that electrode flow through the medium, forming a “conduction channel” to the other electrode. The received ions make up the electrical signal that the memristor transmits through the circuit. The size of the ion channel (and the signal that the memristor ultimately produces) should be proportional to the strength of the stimulating voltage.
Kim says that existing memristor designs work pretty well in cases where voltage stimulates a large conduction channel, or a heavy flow of ions from one electrode to the other. But these designs are less reliable when memristors need to generate subtler signals, via thinner conduction channels.
The thinner a conduction channel, and the lighter the flow of ions from one electrode to the other, the harder it is for individual ions to stay together. Instead, they tend to wander from the group, disbanding within the medium. As a result, it’s difficult for the receiving electrode to reliably capture the same number of ions, and therefore transmit the same signal, when stimulated with a certain low range of current.
Borrowing from metallurgy
Kim and his colleagues found a way around this limitation by borrowing a technique from metallurgy, the science of melding metals into alloys and studying their combined properties.
“Traditionally, metallurgists try to add different atoms into a bulk matrix to strengthen materials, and we thought, why not tweak the atomic interactions in our memristor, and add some alloying element to control the movement of ions in our medium,” Kim says.
Engineers typically use silver as the material for a memristor’s positive electrode. Kim’s team looked through the literature to find an element that they could combine with silver to effectively hold silver ions together, while allowing them to flow quickly through to the other electrode.
The team landed on copper as the ideal alloying element, as it is able to bind both with silver, and with silicon.
“It acts as a sort of bridge, and stabilizes the silver-silicon interface,” Kim says.
To make memristors using their new alloy, the group first fabricated a negative electrode out of silicon, then made a positive electrode by depositing a slight amount of copper, followed by a layer of silver. They sandwiched the two electrodes around an amorphous silicon medium. In this way, they patterned a millimeter-square silicon chip with tens of thousands of memristors.
As a first test of the chip, they recreated a gray-scale image of the Captain America shield. They equated each pixel in the image to a corresponding memristor in the chip. They then modulated the conductance of each memristor that was relative in strength to the color in the corresponding pixel.
The chip produced the same crisp image of the shield, and was able to “remember” the image and reproduce it many times, compared with chips made of other materials.
The team also ran the chip through an image processing task, programming the memristors to alter an image, in this case of MIT’s Killian Court, in several specific ways, including sharpening and blurring the original image. Again, their design produced the reprogrammed images more reliably than existing memristor designs.
“We’re using artificial synapses to do real inference tests,” Kim says. “We would like to develop this technology further to have larger-scale arrays to do image recognition tasks. And some day, you might be able to carry around artificial brains to do these kinds of tasks, without connecting to supercomputers, the internet, or the cloud.”
Here’s a link to and a citation for the paper,
Alloying conducting channels for reliable neuromorphic computing by Hanwool Yeon, Peng Lin, Chanyeol Choi, Scott H. Tan, Yongmo Park, Doyoon Lee, Jaeyong Lee, Feng Xu, Bin Gao, Huaqiang Wu, He Qian, Yifan Nie, Seyoung Kim & Jeehwan Kim. Nature Nanotechnology (2020 DOI: https://doi.org/10.1038/s41565-020-0694-5 Published: 08 June 2020
This paper is behind a paywall.
Electric sheep and sleeping androids
I find it impossible to mention that androids might need sleep without reference to Philip K. Dick’s 1968 novel, “Do Androids Dream of Electric Sheep?”; its Wikipedia entry is here.
As it happens, I’m not the only one who felt the need to reference the novel, from a June 8, 2020 news item on ScienceDaily,
No one can say whether androids will dream of electric sheep, but they will almost certainly need periods of rest that offer benefits similar to those that sleep provides to living brains, according to new research from Los Alamos National Laboratory.
“We study spiking neural networks, which are systems that learn much as living brains do,” said Los Alamos National Laboratory computer scientist Yijing Watkins. “We were fascinated by the prospect of training a neuromorphic processor in a manner analogous to how humans and other biological systems learn from their environment during childhood development.”
Watkins and her research team found that the network simulations became unstable after continuous periods of unsupervised learning. When they exposed the networks to states that are analogous to the waves that living brains experience during sleep, stability was restored. “It was as though we were giving the neural networks the equivalent of a good night’s rest,” said Watkins.
The discovery came about as the research team worked to develop neural networks that closely approximate how humans and other biological systems learn to see. The group initially struggled with stabilizing simulated neural networks undergoing unsupervised dictionary training, which involves classifying objects without having prior examples to compare them to.
“The issue of how to keep learning systems from becoming unstable really only arises when attempting to utilize biologically realistic, spiking neuromorphic processors or when trying to understand biology itself,” said Los Alamos computer scientist and study coauthor Garrett Kenyon. “The vast majority of machine learning, deep learning, and AI researchers never encounter this issue because in the very artificial systems they study they have the luxury of performing global mathematical operations that have the effect of regulating the overall dynamical gain of the system.”
The researchers characterize the decision to expose the networks to an artificial analog of sleep as nearly a last ditch effort to stabilize them. They experimented with various types of noise, roughly comparable to the static you might encounter between stations while tuning a radio. The best results came when they used waves of so-called Gaussian noise, which includes a wide range of frequencies and amplitudes. They hypothesize that the noise mimics the input received by biological neurons during slow-wave sleep. The results suggest that slow-wave sleep may act, in part, to ensure that cortical neurons maintain their stability and do not hallucinate.
The groups’ next goal is to implement their algorithm on Intel’s Loihi neuromorphic chip. They hope allowing Loihi to sleep from time to time will enable it to stably process information from a silicon retina camera in real time. If the findings confirm the need for sleep in artificial brains, we can probably expect the same to be true of androids and other intelligent machines that may come about in the future.
Watkins will be presenting the research at the Women in Computer Vision Workshop on June 14  in Seattle.
The 2020 Women in Computer Vition Workshop (WICV) website is here. As is becoming standard practice for these times, the workshop was held in a virtual environment. Here’s a link to and a citation for the poster presentation paper,
I went down a rabbit hole while trying to figure out the difference between ‘organic’ memristors and standard memristors. I have put the results of my investigation at the end of this post. First, there’s the news.
An April 21, 2020 news item on ScienceDaily explains why researchers are so focused on memristors and brainlike computing,
The advent of artificial intelligence, machine learning and the internet of things is expected to change modern electronics and bring forth the fourth Industrial Revolution. The pressing question for many researchers is how to handle this technological revolution.
“It is important for us to understand that the computing platforms of today will not be able to sustain at-scale implementations of AI algorithms on massive datasets,” said Thirumalai Venkatesan, one of the authors of a paper published in Applied Physics Reviews, from AIP Publishing.
“Today’s computing is way too energy-intensive to handle big data. We need to rethink our approaches to computation on all levels: materials, devices and architecture that can enable ultralow energy computing.”
Brain-inspired electronics with organic memristors could offer a functionally promising and cost- effective platform, according to Venkatesan. Memristive devices are electronic devices with an inherent memory that are capable of both storing data and performing computation. Since memristors are functionally analogous to the operation of neurons, the computing units in the brain, they are optimal candidates for brain-inspired computing platforms.
Until now, oxides have been the leading candidate as the optimum material for memristors. Different material systems have been proposed but none have been successful so far.
“Over the last 20 years, there have been several attempts to come up with organic memristors, but none of those have shown any promise,” said Sreetosh Goswami, lead author on the paper. “The primary reason behind this failure is their lack of stability, reproducibility and ambiguity in mechanistic understanding. At a device level, we are now able to solve most of these problems,”
This new generation of organic memristors is developed based on metal azo complex devices, which are the brainchild of Sreebata Goswami, a professor at the Indian Association for the Cultivation of Science in Kolkata and another author on the paper.
“In thin films, the molecules are so robust and stable that these devices can eventually be the right choice for many wearable and implantable technologies or a body net, because these could be bendable and stretchable,” said Sreebata Goswami. A body net is a series of wireless sensors that stick to the skin and track health.
The next challenge will be to produce these organic memristors at scale, said Venkatesan.
“Now we are making individual devices in the laboratory. We need to make circuits for large-scale functional implementation of these devices.”
This undated article on Nanowerk provides a relatively complete and technical description of memristors in general (Note: A link has been removed),
A memristor (named as a portmanteau of memory and resistor) is a non-volatile electronic memory device that was first theorized by Leon Ong Chua in 1971 as the fourth fundamental two-terminal circuit element following the resistor, the capacitor, and the inductor (IEEE Transactions on Circuit Theory, “Memristor-The missing circuit element”).
Its special property is that its resistance can be programmed (resistor function) and subsequently remains stored (memory function). Unlike other memories that exist today in modern electronics, memristors are stable and remember their state even if the device loses power.
However, it was only almost 40 years later that the first practical device was fabricated. This was in 2008, when a group led by Stanley Williams at HP Research Labs realized that switching of the resistance between a conducting and less conducting state in metal-oxide thin-film devices was showing Leon Chua’s memristor behavior. …
The article on Nanowerk includes an embedded video presentation on memristors given by Stanley Williams (also known as R. Stanley Williams).
The memristor is composed of the transition metal ruthenium complexed with “azo-aromatic ligands.” [emphasis mine] The theoretical work enabling this material was performed at Yale, and the organic molecules were synthesized at the Indian Association for the Cultivation of Sciences. …
I highlighted ‘ligands’ because that appears to be the difference. However, there is more than one type of ligand on Wikipedia.
Ligand, an atom, ion, or functional group that donates one or more of its electrons through a coordinate covalent bond to one or more central atoms or ions
Ligand (biochemistry), a substance that binds to a protein
a ‘guest’ in host–guest chemistry
I did take a look at the paper and did not see any references to proteins or other biomolecules that I could recognize as such. I’m not sure why the researchers are describing their device as an ‘organic’ memristor but this may reflect a shortcoming in the definitions I have found or shortcomings in my reading of the paper rather than an error on their parts.
Hopefully, more research will be forthcoming and it will be possible to better understand the terminology.
So we’re still stuck in 20th century concepts about artificial intelligence (AI), eh? Sean Captain’s February 21, 2020 article (for Fast Company) about the new AI exhibit in San Francisco suggests that artists can help us revise our ideas (Note: Links have been removed),
Though we’re well into the age of machine learning, popular culture is stuck with a 20th century notion of artificial intelligence. While algorithms are shaping our lives in real ways—playing on our desires, insecurities, and suspicions in social media, for instance—Hollywood is still feeding us clichéd images of sexy, deadly robots in shows like Westworld and Star Trek Picard.
The old-school humanlike sentient robot “is an important trope that has defined the visual vocabulary around this human-machine relationship for a very long period of time,” says Claudia Schmuckli, curator of contemporary art and programming at the Fine Arts Museums of San Francisco. It’s also a naïve and outdated metaphor, one she is challenging with a new exhibition at San Francisco’s de Young Museum, called Uncanny Valley, that opens on February 22 .
The show’s name [Uncanny Valley: Being Human in the Age of AI] is a kind of double entendre referencing both the dated and emerging conceptions of AI. Coined in the 1970s, the term “uncanny valley” describes the rise and then sudden drop off of empathy we feel toward a machine as its resemblance to a human increases. Putting a set of cartoony eyes on a robot may make it endearing. But fitting it with anatomically accurate eyes, lips, and facial gestures gets creepy. As the gap between the synthetic and organic narrows, the inability to completely close that gap becomes all the more unsettling.
But the artists in this exhibit are also looking to another valley—Silicon Valley, and the uncanny nature of the real AI the region is building. “One of the positions of this exhibition is that it may be time to rethink the coordinates of the Uncanny Valley and propose a different visual vocabulary,” says Schmuckli.
… the resemblance to humans is only synthetic-skin deep. Bina48 can string together a long series of sentences in response to provocative questions from Dinkins, such as, “Do you know racism?” But the answers are sometimes barely intelligible, or at least lack the depth and nuance of a conversation with a real human. The robot’s jerky attempts at humanlike motion also stand in stark contrast to Dinkins’s calm bearing and fluid movement. Advanced as she is by today’s standards, Bina48 is tragically far from the sci-fi concept of artificial life. Her glaring shortcomings hammer home why the humanoid metaphor is not the right framework for understanding at least today’s level of artificial intelligence.
What are the invisible mechanisms of current forms of artificial intelligence (AI)? How is AI impacting our personal lives and socioeconomic spheres? How do we define intelligence? How do we envision the future of humanity?
SAN FRANCISCO (September 26, 2019) — As technological innovation continues to shape our identities and societies, the question of what it means to be, or remain human has become the subject of fervent debate. Taking advantage of the de Young museum’s proximity to Silicon Valley, Uncanny Valley: Being Human in the Age of AI arrives as the first major exhibition in the US to explore the relationship between humans and intelligent machines through an artistic lens. Organized by the Fine Arts Museums of San Francisco, with San Francisco as its sole venue, Uncanny Valley: Being Human in the Age of AI will be on view from February 22 to October 25, 2020.
“Technology is changing our world, with artificial intelligence both a new frontier of possibility but also a development fraught with anxiety,” says Thomas P. Campbell, Director and CEO of the Fine Arts Museums of San Francisco. “Uncanny Valley: Being Human in the Age of AI brings artistic exploration of this tension to the ground zero of emerging technology, raising challenging questions about the future interface of human and machine.”
The exhibition, which extends through the first floor of the de Young and into the museum’s sculpture garden, explores the current juncture through philosophical, political, and poetic questions and problems raised by AI. New and recent works by an intergenerational, international group of artists and activist collectives—including Zach Blas, Ian Cheng, Simon Denny, Stephanie Dinkins, Forensic Architecture, Lynn Hershman Leeson, Pierre Huyghe, Christopher Kulendran Thomas in collaboration with Annika Kuhlmann, Agnieszka Kurant, Lawrence Lek, Trevor Paglen, Hito Steyerl, Martine Syms, and the Zairja Collective—will be presented.
The Uncanny Valley
In 1970 Japanese engineer Masahiro Mori introduced the concept of the “uncanny valley” as a terrain of existential uncertainty that humans experience when confronted with autonomous machines that mimic their physical and mental properties. An enduring metaphor for the uneasy relationship between human beings and lifelike robots or thinking machines, the uncanny valley and its edges have captured the popular imagination ever since. Over time, the rapid growth and affordability of computers, cloud infrastructure, online search engines, and data sets have fueled developments in machine learning that fundamentally alter our modes of existence, giving rise to a newly expanded uncanny valley.
“As our lives are increasingly organized and shaped by algorithms that track, collect, evaluate, and monetize our data, the uncanny valley has grown to encompass the invisible mechanisms of behavioral engineering and automation,” says Claudia Schmuckli, Curator in Charge of Contemporary Art and Programming at the Fine Arts Museums of San Francisco. “By paying close attention to the imminent and nuanced realities of AI’s possibilities and pitfalls, the artists in the exhibition seek to thicken the discourse around AI. Although fables like HBO’s sci-fi drama Westworld, or Spike Jonze’s feature film Her still populate the collective imagination with dystopian visions of a mechanized future, the artists in this exhibition treat such fictions as relics of a humanist tradition that has little relevance today.”
Ian Cheng’s digitally simulated AI creature BOB (Bag of Beliefs) reflects on the interdependency of carbon and silicon forms of intelligence. An algorithmic Tamagotchi, it is capable of evolution, but its growth, behavior, and personality are molded by online interaction with visitors who assume collective responsibility for its wellbeing.
In A.A.I. (artificial artificial intelligence), an installation of multiple termite mounds of colored sand, gold, glitter and crystals, Agnieszka Kurant offers a vibrant critique of new AI economies, with their online crowdsourcing marketplace platforms employing invisible armies of human labor at sub-minimum wages.
Simon Denny ‘s Amazon worker cage patent drawing as virtual King Island Brown Thornbill cage (US 9,280,157 B2: “System and method for transporting personnel within an active workspace”, 2016) (2019) also examines the intersection of labor, resources, and automation. He presents 3-D prints and a cage-like sculpture based on an unrealized machine patent filed by Amazon to contain human workers. Inside the cage an augmented reality application triggers the appearance of a King Island Brown Thornbill — a bird on the verge of extinction; casting human labor as the proverbial canary in the mine. The humanitarian and ecological costs of today’s data economy also informs a group of works by the Zairja Collective that reflect on the extractive dynamics of algorithmic data mining.
Hito Steyerl addresses the political risks of introducing machine learning into the social sphere. Her installation The City of Broken Windows presents a collision between commercial applications of AI in urban planning along with communal and artistic acts of resistance against neighborhood tipping: one of its short films depicts a group of technicians purposefully smashing windows to teach an algorithm how to recognize the sound of breaking glass, and another follows a group of activists through a Camden, NJ neighborhood as they work to keep decay at bay by replacing broken windows in abandoned homes with paintings.
Addressing the perpetuation of societal biases and discrimination within AI, Trevor Paglen’s They Took the Faces from the Accused and the Dead…(SD18), presents a large gridded installation of more than three thousand mugshots from the archives of the American National Standards Institute. The institute’s collections of such images were used to train ealry facial-recognition technologies — without the consent of those pictured. Lynn Hershman Leeson’s new installation Shadow Stalker critiques the problematic reliance on algorithmic systems, such as the military forecasting tool Predpol now widely used for policing, that categorize individuals into preexisting and often false “embodied metrics.”
Stephanie Dinkins extends the inquiry into how value systems are built into AI and the construction of identity in Conversations with Bina48, examining the social robot’s (and by extension our society’s) coding of technology, race, gender and social equity. In the same territory, Martine Syms posits AI as a “shamespace” for misrepresentation. For Mythiccbeing she has created an avatar of herself that viewers can interact with through text messaging. But unlike service agents such as Siri and Alexa, who readily respond to questions and demands, Syms’s Teeny is a contrarious interlocutor, turning each interaction into an opportunity to voice personal observations and frustrations about racial inequality and social injustice.
Countering the abusive potential of machine learning, Forensic Architecture pioneers an application to the pursuit of social justice. Their proposition of a Model Zoo marks the beginnings of a new research tool for civil society built of military vehicles, missile fragments, and bomb clouds—evidence of human-rights violations by states and militaries around the world. Christopher Kulendran Thomas’s video Being Human, created in collaboration with Annika Kuhlmann, poses the philosophical question of what it means to be human when machines are able to synthesize human understanding ever more convincingly. Set in Sri Lanka, it employs AI-generated characters of singer Taylor Swift and artist Oscar Murillo to reflect on issues of individual authenticity, collective sovereignty, and the future of human rights.
Lawrence Lek’s sci-fi-inflected film Aidol, which explores the relationship between algorithmic automation and human creativity, projects this question into the future. It transports the viewer into the computer-generated “sinofuturist” world of the 2065 eSports Olympics: when the popular singer Diva enlists the super-intelligent Geomancer to help her stage her artistic comeback during the game’s halftime show, she unleashes an existential and philosophical battle that explodes the divide between humans and machines.
The Doors, a newly commissioned installation by Zach Blas, by contrast shines the spotlight back onto the present and on the culture and ethos of Silicon Valley — the ground zero for the development of AI. Inspired by the ubiquity of enclosed gardens on tech campuses, he has created an artificial garden framed by a six-channel video projected on glass panes that convey a sense of algorithmic psychedelia aiming to open new “doors of perception.” While luring visitors into AI’s promises, it also asks what might become possible when such glass doors begin to crack.
Unveiled in late spring Pierre Huyghe‘s Exomind (Deep Water), a sculpture of a crouched female nude with a live beehive as its head will be nestled within the museum’s garden. With its buzzing colony pollinating the surrounding flora, it offers a poignant metaphor for the modeling of neural networks on the biological brain and an understanding of intelligence as grounded in natural forms and processes.
Since 2018, Forensic Architecture has used machine learning / AI to aid in humanitarian work, using synthetic images—photorealistic digital renderings based around 3-D models—to train algorithmic classifiers to identify tear gas munitions and chemical bombs deployed against protesters worldwide, including in Hong Kong, Chile, the US, Venezuela, and Sudan.
Their project, Model Zoo, on view in Uncanny Valley represents a growing collection of munitions and weapons used in conflict today and the algorithmic models developed to identify them. It shows a collection of models being used to track and hold accountable human rights violators around the world. The piece joins work by 14 contemporary artists reflecting on the philosophical and political consequences of the application of AI into the social sphere.
We are deeply saddened that Weizman will not be allowed to travel to celebrate the opening of the exhibition. We stand with him and Forensic Architecture’s partner communities who continue to resist violent states and corporate practices, and who are increasingly exposed to the regime of “security algorithms.”
—Claudia Schmuckli, Curator-in-Charge, Contemporary Art & Programming, & Thomas P. Campbell, Director and CEO, Fine Arts Museums of San Francisco
There is a February 20, 2020 article (for Fast Company) by Eyal Weizman chronicling his experience with being denied entry by an algorithm. Do read it in its entirety (the Fast Company is itself an excerpt from Weizman’s essay) if you have the time, if not, here’s the description of how he tried to gain entry after being denied the first time,
The following day I went to the U.S. Embassy in London to apply for a visa. In my interview, the officer informed me that my authorization to travel had been revoked because the “algorithm” had identified a security threat. He said he did not know what had triggered the algorithm but suggested that it could be something I was involved in, people I am or was in contact with, places to which I had traveled (had I recently been in Syria, Iran, Iraq, Yemen, or Somalia or met their nationals?), hotels at which I stayed, or a certain pattern of relations among these things. I was asked to supply the Embassy with additional information, including 15 years of travel history, in particular where I had gone and who had paid for it. The officer said that Homeland Security’s investigators could assess my case more promptly if I supplied the names of anyone in my network whom I believed might have triggered the algorithm. I declined to provide this information.
I hope the exhibition is successful; it has certainly experienced a thought-provoking start.
Finally, I have often featured postings that discuss the ‘uncanny valley’. To find those postings, just use that phrase in the blog search engine. You might also went to search ‘Hiroshi Ishiguro’, a Japanese scientist and robotocist who specializes in humanoid robots.
Once you get past the technical language (there’s a lot of it), you’ll find that they make the link between biomimicry and memristors explicit. Admittedly I’m not an expert but if I understand the research correctly, the scientists are suggesting that the algorithms used in machine learning today cannot allow memristors to be properly integrated for use in true neuromorphic computing and this work from Russia and Greece points to a new paradigm. If you understand it differently, please do let me know in the comments.
Lobachevsky University scientists together with their colleagues from the National Research Center “Kurchatov Institute” (Moscow) and the National Research Center “Demokritos” (Athens) are working on the hardware implementation of a spiking neural network based on memristors.
The key elements of such a network, along with pulsed neurons, are artificial synaptic connections that can change the strength (weight) of connection between neurons during the learning (Microelectronic Engineering, “Yttria-stabilized zirconia cross-point memristive devices for neuromorphic applications”).
For this purpose, memristive devices based on metal-oxide-metal nanostructures developed at the UNN Physics and Technology Research Institute (PTRI) are suitable, but their use in specific spiking neural network architectures developed at the Kurchatov Institute requires demonstration of biologically plausible learning principles.
The biological mechanism of learning of neural systems is described by Hebb’s rule, according to which learning occurs as a result of an increase in the strength of connection (synaptic weight) between simultaneously active neurons, which indicates the presence of a causal relationship in their excitation. One of the clarifying forms of this fundamental rule is plasticity, which depends on the time of arrival of pulses (Spike-Timing Dependent Plasticity – STDP).
In accordance with STDP, synaptic weight increases if the postsynaptic neuron generates a pulse (spike) immediately after the presynaptic one, and vice versa, the synaptic weight decreases if the postsynaptic neuron generates a spike right before the presynaptic one. Moreover, the smaller the time difference Δt between the pre- and postsynaptic spikes, the more pronounced the weight change will be.
According to one of the researchers, Head of the UNN PTRI laboratory Alexei Mikhailov, in order to demonstrate the STDP principle, memristive nanostructures based on yttria-stabilized zirconia (YSZ) thin films were used. YSZ is a well-known solid-state electrolyte with high oxygen ion mobility.
“Due to a specified concentration of oxygen vacancies, which is determined by the controlled concentration of yttrium impurities, and the heterogeneous structure of the films obtained by magnetron sputtering, such memristive structures demonstrate controlled bipolar switching between different resistive states in a wide resistance range. The switching is associated with the formation and destruction of conductive channels along grain boundaries in the polycrystalline ZrO2 (Y) film,” notes Alexei Mikhailov.
An array of memristive devices for research was implemented in the form of a microchip mounted in a standard cermet casing, which facilitates the integration of the array into a neural network’s analog circuit. The full technological cycle for creating memristive microchips is currently implemented at the UNN PTRI. In the future, it is possible to scale the devices down to the minimum size of about 50 nm, as was established by Greek partners. Our studies of the dynamic plasticity of the memoristive devices, continues Alexey Mikhailov, have shown that the form of the conductance change depending on Δt is in good agreement with the STDP learning rules. It should be also noted that if the initial value of the memristor conductance is close to the maximum, it is easy to reduce the corresponding weight while it is difficult to enhance it, and in the case of a memristor with a minimum conductance in the initial state, it is difficult to reduce its weight, but it is easy to enhance it.
According to Vyacheslav Demin, director-coordinator in the area of nature-like technologies of the Kurchatov Institute, who is one of the ideologues of this work, the established pattern of change in the memristor conductance clearly demonstrates the possibility of hardware implementation of the so-called local learning rules. Such rules for changing the strength of synaptic connections depend only on the values of variables that are present locally at each time point (neuron activities and current weights).
“This essentially distinguishes such principle from the traditional learning algorithm, which is based on global rules for changing weights, using information on the error values at the current time point for each neuron of the output neural network layer (in a widely popular group of error back propagation methods). The traditional principle is not biosimilar, it requires “external” (expert) knowledge of the correct answers for each example presented to the network (that is, they do not have the property of self-learning). This principle is difficult to implement on the basis of memristors, since it requires controlled precise changes of memristor conductances, as opposed to local rules. Such precise control is not always possible due to the natural variability (a wide range of parameters) of memristors as analog elements,” says Vyacheslav Demin.
Local learning rules of the STDP type implemented in hardware on memristors provide the basis for autonomous (“unsupervised”) learning of a spiking neural network. In this case, the final state of the network does not depend on its initial state, but depends only on the learning conditions (a specific sequence of pulses). According to Vyacheslav Demin, this opens up prospects for the application of local learning rules based on memristors when solving artificial intelligence problems with the use of complex spiking neural network architectures.
Both of these bits have a music focus but they represent two entirely different science-based approaches to that form of art and one is solely about the music and the other is included as one of the art-making processes being investigated..
Large Interactive Virtual Environment Laboratory (LIVELab) at McMaster University
Laurel Trainor and Dan J. Bosnyak both of McMaster University (Ontario, Canada) have written an October 27, 2019 essay about the LiveLab and their work for The Conversation website (Note: Links have been removed),
The Large Interactive Virtual Environment Laboratory (LIVELab) at McMaster University is a research concert hall. It functions as both a high-tech laboratory and theatre, opening up tremendous opportunities for research and investigation.
As the only facility of its kind in the world, the LIVELab is a 106-seat concert hall equipped with dozens of microphones, speakers and sensors to measure brain responses, physiological responses such as heart rate, breathing rates, perspiration and movements in multiple musicians and audience members at the same time.
Engineers, psychologists and clinician-researchers from many disciplines work alongside musicians, media artists and industry to study performance, perception, neural processing and human interaction.
In the LIVELab, acoustics are digitally controlled so the experience can change instantly from extremely silent with almost no reverberation to a noisy restaurant to a subway platform or to the acoustics of Carnegie Hall.
Real-time physiological data such as heart rate can be synchronized with data from other systems such as motion capture, and monitored and recorded from both performers and audience members. The result is that the reams of data that can now be collected in a few hours in the LIVELab used to take weeks or months to collect in a traditional lab. And having measurements of multiple people simultaneously is pushing forward our understanding of real-time human interactions.
Consider the implications of how music might help people with Parkinson’s disease to walk more smoothly or children with dyslexia to read better.
[…] area of ongoing research is the effectiveness of hearing aids. By the age of 60, nearly 49 per cent of people will suffer from some hearing loss. People who wear hearing aids are often frustrated when listening to music because the hearing aids distort the sound and cannot deal with the dynamic range of the music.
The LIVELab is working with the Hamilton Philharmonic Orchestra to solve this problem. During a recent concert, researchers evaluated new ways of delivering sound directly to participants’ hearing aids to enhance sounds.
Researchers hope new technologies can not only increase live musical enjoyment but alleviate the social isolation caused by hearing loss.
Imagine the possibilities for understanding music and sound: How it might help to improve cognitive decline, manage social performance anxiety, help children with developmental disorders, aid in treatment of depression or keep the mind focused. Every time we conceive and design a study, we think of new possibilities.
The essay also includes an embedded 12 min. video about LIVELab and details about studies conducted on musicians and live audiences. Apparently, audiences experience live performance differently than recorded performances and musicians use body sway to create cohesive performances. You can find the McMaster Institute for Music & the Mind here and McMaster’s LIVELab here.
Metacreation Lab at Simon Fraser University (SFU)
I just recently discovered that there’s a Metacreation Lab at Simon Fraser University (Vancouver, Canada), which on its homepage has this ” Metacreation is the idea of endowing machines with creative behavior.” Here’s more from the homepage,
As the contemporary approach to generative art, Metacreation involves using tools and techniques from artificial intelligence, artificial life, and machine learning to develop software that partially or completely automates creative tasks. Through the collaboration between scientists, experts in artificial intelligence, cognitive sciences, designers and artists, the Metacreation Lab for Creative AI is at the forefront of the development of generative systems, be they embedded in interactive experiences or integrated into current creative software. Scientific research in the Metacreation Lab explores how various creative tasks can be automated and enriched. These tasks include music composition [emphasis mine], sound design, video editing, audio/visual effect generation, 3D animation, choreography, and video game design.
Besides scientific research, the team designs interactive and generative artworks that build upon the algorithms and research developed in the Lab. This work often challenges the social and cultural discourse on AI.
Much to my surprise I received the Metacreation Lab’s inaugural email newsletter (received via email on Friday, November 15, 2019),
We decided to start a mailing list for disseminating news, updates, and announcements regarding generative art, creative AI and New Media. In this newsletter:
ISEA 2020: The International Symposium on Electronic Art. ISEA return to Montreal, check the CFP bellow and contribute!
2015: A transcription of Sara Diamond’s keynote address “Action Agenda:
Vancouver’s Prescient Media Arts” is now available for download.
Art, the book: we are happy to announce the release of the first
comprehensive volume on Brain Art. Edited by Anton Nijholt, and
published by Springer.
Here are more details from the newsletter,
ISEA2020 – 26th International Symposium on Electronic Arts
Montreal, September 24, 2019 Montreal Digital Spring (Printemps numérique) is launching a call for participation as part of ISEA2020 / MTL connect to be held from May 19 to 24, 2020 in Montreal, Canada. Founded in 1990, ISEA is one of the world’s most prominent international arts and technology events, bringing together scholarly, artistic, and scientific domains in an interdisciplinary discussion and showcase of creative productions applying new technologies in art, interactivity, and electronic and digital media. For 2020, ISEA Montreal turns towards the theme of sentience.
ISEA2020 will be fully dedicated to examining the resurgence of sentience—feeling-sensing-making sense—in recent art and design, media studies, science and technology studies, philosophy, anthropology, history of science and the natural scientific realm—notably biology, neuroscience and computing. We ask: why sentience? Why and how does sentience matter? Why have artists and scholars become interested in sensing and feeling beyond, with and around our strictly human bodies and selves? Why has this notion been brought to the fore in an array of disciplines in the 21st century?
CALL FOR PARTICIPATION: WHY SENTIENCE? ISEA2020 invites artists, designers, scholars, researchers, innovators and creators to participate in the various activities deployed from May 19 to 24, 2020. To complete an application, please fill in the forms and follow the instructions.
You can apply for several categories. All profiles are welcome. Notifications of acceptance will be sent around January 13, 2020.
Important: please note that the Call for participation for MTL connect is not yet launched, but you can also apply to participate in the programming of the other Pavilions (4 other themes) when registrations are open (coming soon): mtlconnecte.ca/enTICKETS
Registration is now available to assist to ISEA2020 / MTL connect, from May 19 to 24, 2020. Book today your Full Pass and get the early-bird rate!
The first book that surveys how brain activity can be monitored and manipulated for artistic purposes, with contributions by interactive media artists, brain-computer interface researchers, and neuroscientists. View the Book Here
As per the Leonardo review from Cristina Albu:
“Another seminal contribution of the volume is the presentation of multiple taxonomies of “brain art,” which can help art critics develop better criteria for assessing this genre. Mirjana Prpa and Philippe Pasquier’s meticulous classification shows how diverse such works have become as artists consider a whole range of variables of neurofeedback.”Read the Review
Should this kind of information excite and motivate you do start metacreating, you can get in touch with the lab,
Our mailing address is: Metacreation Lab for Creative AI School of Interactive Arts & Technology Simon Fraser University 250-13450 102 Ave. Surrey, BC V3T 0A3 Web: http://metacreation.net/ Email: metacreation_admin (at) sfu (dot) ca