Tag Archives: Marin Soljačić

Automated science writing?

It seems that automated science writing is not ready—yet. Still, an April 18, 2019 news item on ScienceDaily suggests that progress is being made,

The work of a science writer, including this one, includes reading journal papers filled with specialized technical terminology, and figuring out how to explain their contents in language that readers without a scientific background can understand.

Now, a team of scientists at MIT [Massachusetts Institute of Technology] and elsewhere has developed a neural network, a form of artificial intelligence (AI), that can do much the same thing, at least to a limited extent: It can read scientific papers and render a plain-English summary in a sentence or two.

An April 17, 2019 MIT news release, which originated the news item, delves into the research and its implications,

Even in this limited form, such a neural network could be useful for helping editors, writers, and scientists [emphasis mine] scan a large number of papers to get a preliminary sense of what they’re about. But the approach the team developed could also find applications in a variety of other areas besides language processing, including machine translation and speech recognition.

The work is described in the journal Transactions of the Association for Computational Linguistics, in a paper by Rumen Dangovski and Li Jing, both MIT graduate students; Marin Soljačić, a professor of physics at MIT; Preslav Nakov, a principal scientist at the Qatar Computing Research Institute, HBKU; and Mićo Tatalović, a former Knight Science Journalism fellow at MIT and a former editor at New Scientist magazine.

From AI for physics to natural language

The work came about as a result of an unrelated project, which involved developing new artificial intelligence approaches based on neural networks, aimed at tackling certain thorny problems in physics. However, the researchers soon realized that the same approach could be used to address other difficult computational problems, including natural language processing, in ways that might outperform existing neural network systems.

“We have been doing various kinds of work in AI for a few years now,” Soljačić says. “We use AI to help with our research, basically to do physics better. And as we got to be  more familiar with AI, we would notice that every once in a while there is an opportunity to add to the field of AI because of something that we know from physics — a certain mathematical construct or a certain law in physics. We noticed that hey, if we use that, it could actually help with this or that particular AI algorithm.”

This approach could be useful in a variety of specific kinds of tasks, he says, but not all. “We can’t say this is useful for all of AI, but there are instances where we can use an insight from physics to improve on a given AI algorithm.”

Neural networks in general are an attempt to mimic the way humans learn certain new things: The computer examines many different examples and “learns” what the key underlying patterns are. Such systems are widely used for pattern recognition, such as learning to identify objects depicted in photos.

But neural networks in general have difficulty correlating information from a long string of data, such as is required in interpreting a research paper. Various tricks have been used to improve this capability, including techniques known as long short-term memory (LSTM) and gated recurrent units (GRU), but these still fall well short of what’s needed for real natural-language processing, the researchers say.

The team came up with an alternative system, which instead of being based on the multiplication of matrices, as most conventional neural networks are, is based on vectors rotating in a multidimensional space. The key concept is something they call a rotational unit of memory (RUM).

Essentially, the system represents each word in the text by a vector in multidimensional space — a line of a certain length pointing in a particular direction. Each subsequent word swings this vector in some direction, represented in a theoretical space that can ultimately have thousands of dimensions. At the end of the process, the final vector or set of vectors is translated back into its corresponding string of words.

“RUM helps neural networks to do two things very well,” Nakov says. “It helps them to remember better, and it enables them to recall information more accurately.”

After developing the RUM system to help with certain tough physics problems such as the behavior of light in complex engineered materials, “we realized one of the places where we thought this approach could be useful would be natural language processing,” says Soljačić,  recalling a conversation with Tatalović, who noted that such a tool would be useful for his work as an editor trying to decide which papers to write about. Tatalović was at the time exploring AI in science journalism as his Knight fellowship project.

“And so we tried a few natural language processing tasks on it,” Soljačić says. “One that we tried was summarizing articles, and that seems to be working quite well.”

The proof is in the reading

As an example, they fed the same research paper through a conventional LSTM-based neural network and through their RUM-based system. The resulting summaries were dramatically different.

The LSTM system yielded this highly repetitive and fairly technical summary: “Baylisascariasis,” kills mice, has endangered the allegheny woodrat and has caused disease like blindness or severe consequences. This infection, termed “baylisascariasis,” kills mice, has endangered the allegheny woodrat and has caused disease like blindness or severe consequences. This infection, termed “baylisascariasis,” kills mice, has endangered the allegheny woodrat.

Based on the same paper, the RUM system produced a much more readable summary, and one that did not include the needless repetition of phrases: Urban raccoons may infect people more than previously assumed. 7 percent of surveyed individuals tested positive for raccoon roundworm antibodies. Over 90 percent of raccoons in Santa Barbara play host to this parasite.

Already, the RUM-based system has been expanded so it can “read” through entire research papers, not just the abstracts, to produce a summary of their contents. The researchers have even tried using the system on their own research paper describing these findings — the paper that this news story is attempting to summarize.

Here is the new neural network’s summary: Researchers have developed a new representation process on the rotational unit of RUM, a recurrent memory that can be used to solve a broad spectrum of the neural revolution in natural language processing.

It may not be elegant prose, but it does at least hit the key points of information.

Çağlar Gülçehre, a research scientist at the British AI company Deepmind Technologies, who was not involved in this work, says this research tackles an important problem in neural networks, having to do with relating pieces of information that are widely separated in time or space. “This problem has been a very fundamental issue in AI due to the necessity to do reasoning over long time-delays in sequence-prediction tasks,” he says. “Although I do not think this paper completely solves this problem, it shows promising results on the long-term dependency tasks such as question-answering, text summarization, and associative recall.”

Gülçehre adds, “Since the experiments conducted and model proposed in this paper are released as open-source on Github, as a result many researchers will be interested in trying it on their own tasks. … To be more specific, potentially the approach proposed in this paper can have very high impact on the fields of natural language processing and reinforcement learning, where the long-term dependencies are very crucial.”

The research received support from the Army Research Office, the National Science Foundation, the MIT-SenseTime Alliance on Artificial Intelligence, and the Semiconductor Research Corporation. The team also had help from the Science Daily website, whose articles were used in training some of the AI models in this research.

As usual, this ‘automated writing system’ is framed as a ‘helper’ not an usurper of anyone’s job. However, its potential for changing the nature of the work is there. About five years ago I featured another ‘automated writing’ story in a July 16, 2014 posting titled: ‘Writing and AI or is a robot writing this blog?’ You may have been reading ‘automated’ news stories for years. At the time, the focus was on sports and business.

Getting back to 2019 and science writing, here’s a link to and a citation for the paper,

Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications by Rumen Dangovski, Li Jing, Preslav Nakov, Mićo Tatalović and Marin Soljačić. Transactions of the Association for Computational Linguistics Volume 07, 2019 pp.121-138 DOI: https://doi.org/10.1162/tacl_a_00258 Posted Online 2019

© 2019 Association for Computational Linguistics. Distributed under a CC-BY 4.0 license.

This paper is open access.

Getting back to incandescent light (recycling the military way)

MIT (Massachusetts Institute of Technology) issued two news releases about this research into reclaiming incandescent light or as they call it “recycling light.” First off, there’s the Jan. 11, 2016 MIT Institute of Soldier Nanotechnologies news release by Paola Rebusco on EurekAlert,

Humanity started recycling relatively early in its evolution: there are proofs that trash recycling was taking place as early as in the 500 BC. What about light recycling? Consider light bulbs: more than one hundred and thirty years ago Thomas Edison patented the first commercially viable incandescent light bulb, so that “none but the extravagant” would ever “burn tallow candles”, paving the way for more than a century of incandescent lighting. In fact, emergence of electric lighting was the main motivating factor for deployment of electricity into every home in the world. The incandescent bulb is an example of a high temperature thermal emitter. It is very useful, but only a small fraction of the emitted light (and therefore energy) is used: most of the light is emitted in the infrared, invisible to the human eye, and in this context wasted.

Now, in a study published in Nature Nanotechnology on January 11th 2016 (online), a team of MIT researchers describes another way to recycle light emitted at unwanted infrared wavelengths while optimizing the emission at useful visible wavelengths. …

“For a thermal emitter at moderate temperatures one usually nano-patterns its surface to alter the emission,” says Ilic [postdoc Ognjen Ilic], the lead author of the study. “At high temperatures” – a light bulb filament reaches 3000K! – “such nanostructures deteriorate and it is impossible to alter the emission spectrum by having a nanostructure directly on the surface of the emitter.” The team solved the problem by surrounding the hot object with special nanophotonic structures that spectrally filter the emitted light, meaning that they let the light reflect or pass through based on its color (i.e. its wavelength). Because the filters are not in direct physical contact with the emitter, temperatures can be very high.

To showcase this idea, the team picked one of the highest temperature thermal emitters available – an incandescent light bulb. The authors designed nanofilters to recycle the infrared light, while allowing the visible light to go through. “The key advance was to design a photonic structure that transmits visible light and reflects infrared light for a very wide range of angles,” explains Ilic. “Conventional photonic filters usually operate for a single incidence angle. The challenge for us was to extend the desired optical properties across all directions,” a feat the authors achieved using special numerical optimization techniques.

However, for this scheme to work, the authors had to redesign the incandescent filament from scratch. “In a regular light bulb, the filament is a long and curly piece of tungsten wire. Here, the filament is laser-machined out of a flat sheet of tungsten: it is completely planar,” says Bermel [professor Peter Bermel now at Purdue University]. A planar filament has a large area, and is therefore very efficient in re-absorbing the light that was reflected by the filter. In describing how the new device differs from previously suggested concepts, Soljačić [professor Marin Soljačić], the project lead, emphasizes that “it is the combination of the exceptional properties of the filter and the shape of the filament that enabled substantial recycling of unwanted radiated light.”

In the new-concept light bulb prototype built by the authors, the efficiency approaches some fluorescent and LED bulbs. Nonetheless, the theoretical model predicts plenty of room for improvement. “This experimental device is a proof-of-concept, at the low end of performance that could be ultimately achieved by this approach,” argues Celanovic [principal research scientist Ivan Celanovic]. There are other advantages of this approach: “An important feature is that our demonstrated device achieves near-ideal rendering of colors,” notes Ilic, referring to the requirement of light sources to faithfully reproduce surrounding colors. That is precisely the reason why incandescent lights remained dominant for so long: their warm light has remained preferable to drab fluorescent lighting for decades.

Some practical questions need to be addressed before this technology can be widely adopted. “We will work closely with our mechanical engineering colleagues at MIT to try to tackle the issues of thermal stability and long-lifetime,” says Soljačić. The authors are particularly excited about the potential for producing these devices cheaply. “The materials we need are abundant and inexpensive,” Joannopoulos [professor John Joannopoulos] notes, “and the filters themselves–consisting of stacks of commonly deposited materials–are amenable to large-scale deposition.”

Chen [professor Gang Chen] comments further: “The lighting potential of this technology is exciting, but the same approach could also be used to improve the performance of energy conversion schemes such as thermo-photovoltaics.” In a thermo-photovoltaic device, external heat causes the material to glow, emitting light that is converted into an electric current by an absorbing photovoltaic element.

The last point captures the main motivation behind the work. “Light radiated from a hot object can be quite useful, whether that object is an incandescent filament or the Sun,” Ilic says. At its core, this work is about recycling thermal light for a specific application; “a 3000-degree filament is one of the hottest and the most challenging sources to work with,” Ilic continues. “It’s also what makes it a crucial test of our approach.”

There are a few more details in the 2nd Jan. 11, 2016 MIT news release on EurekAlert,

Light recycling

The key is to create a two-stage process, the researchers report. The first stage involves a conventional heated metal filament, with all its attendant losses. But instead of allowing the waste heat to dissipate in the form of infrared radiation, secondary structures surrounding the filament capture this radiation and reflect it back to the filament to be re-absorbed and re-emitted as visible light. These structures, a form of photonic crystal, are made of Earth-abundant elements and can be made using conventional material-deposition technology.

That second step makes a dramatic difference in how efficiently the system converts light into electricity. The efficiency of conventional incandescent lights is between 2 and 3 percent, while that of fluorescents (including CFLs) is currently between 7 and 13 percent, and that of LEDs between 5 and 13 percent. In contrast, the new two-stage incandescents could reach efficiencies as high as 40 percent, the team says.

The first proof-of-concept units made by the team do not yet reach that level, achieving about 6.6 percent efficiency. But even that preliminary result matches the efficiency of some of today’s CFLs and LEDs, they point out. And it is already a threefold improvement over the efficiency of today’s incandescents.

The team refers to their approach as “light recycling,” says Ilic, since their material takes in the unwanted, useless wavelengths of energy and converts them into the visible light wavelengths that are desired. “It recycles the energy that would otherwise be wasted,” says Soljačić.

Bulbs and beyond

One key to their success was designing a photonic crystal that works for a very wide range of wavelengths and angles. The photonic crystal itself is made as a stack of thin layers, deposited on a substrate. “When you put together layers, with the right thicknesses and sequence,” Ilic explains, you can get very efficient tuning of how the material interacts with light. In their system, the desired visible wavelengths pass right through the material and on out of the bulb, but the infrared wavelengths get reflected as if from a mirror. They then travel back to the filament, adding more heat that then gets converted to more light. Since only the visible ever gets out, the heat just keeps bouncing back in toward the filament until it finally ends up as visible light.

I appreciate both MIT news release writers for “Thomas Edison patented the first commercially viable incandescent light bulb” (Rebusco) and the unidentified writer of the 2nd MIT news release for this, from the news release, “Incandescent bulbs, commercially developed by Thomas Edison (and still used by cartoonists as the symbol of inventive insight) … .” Edison did not invent the light bulb. BTW, the emphases are mine.

For interested parties, here’s a link to and a citation for the paper,

Tailoring high-temperature radiation and the resurrection of the incandescent source by Ognjen Ilic, Peter Bermel, Gang Chen, John D. Joannopoulos, Ivan Celanovic, & Marin Soljačić. Nature Nanotechnology  (2016) doi:10.1038/nnano.2015.309 Published online 11 January 2016

This paper is behind a paywall.