Tech News

This mathematical brain model may pave the way for more human-like AI

Last week, Google Research held an online workshop on the conceptual understanding of deep learning. The workshop, which featured presentations by award-winning computer scientists and neuroscientists, discussed how new findings in deep learning and neuroscience can help create better artificial intelligence systems.

While all the presentations and discussions were worth watching (and I might revisit them again in the coming weeks), one, in particular, stood out for me: A talk on word representations in the brain by Christos Papadimitriou, professor of computer science at the University of Columbia.

In his presentation, Papadimitriou, a recipient of the Gödel Prize and Knuth Prize, discussed how our growing understanding of information-processing mechanisms in the brain might help create algorithms that are more robust in understanding and engaging in conversations. Papadimitriou presented a simple and efficient model that explains how different areas of the brain inter-communicate to solve cognitive problems.

“What is happening now is perhaps one of the world’s greatest wonders,” Papadimitriou said, referring to how he was communicating with the audience. The brain translates structured knowledge into airwaves that are transferred across different mediums and reach the ears of the listener, where they are again processed and transformed into structured knowledge by the brain.

“There’s little doubt that all of this happens with spikes, neurons, and synapses. But how? This is a huge question,” Papadimitriou said. “I believe that we are going to have a much better idea of the details of how this happens over the next decade.”

Assemblies of neurons in the brain

The cognitive and neuroscience communities are trying to make sense of how neural activity in the brain translates to language, mathematics, logic, reasoning, planning, and other functions. If scientists succeed at formulating the workings of the brain in terms of mathematical models, then they will open a new door to creating artificial intelligence systems that can emulate the human mind.

A lot of studies focus on activities at the level of single neurons. Until a few decades ago, scientists thought that single neurons corresponded to single thoughts. The most popular example is the “grandmother cell” theory, which claims there’s a single neuron in the brain that spikes every time you see your grandmother. More recent discoveries have refuted this claim and have proven that large groups of neurons are associated with each concept, and there might be overlaps between neurons that link to different concepts.

These groups of brain cells are called “assemblies,” which Papadimitriou describes as “a highly connected, stable set of neurons which represent something: a word, an idea, an object, etc.”

Award-winning neuroscientist György Buzsáki describes assemblies as “the alphabet of the brain.”

A mathematical model of the brain

To better understand the role of assemblies, Papadimitriou proposes a mathematical model of the brain called “interacting recurrent nets.” Under this model, the brain is divided into a finite number of areas, each of which contains several million neurons. There is recursion within each area, which means the neurons interact with each other. And each of these areas has connections to several other areas. These inter-area connections can be excited or inhibited.

This model provides randomness, plasticity, and inhibition. Randomness means the neurons in each brain area are randomly connected. Also, different areas have random connections between them. Plasticity enables the connections between the neurons and areas to adjust through experience and training. And inhibition means that at any moment, a limited number of neurons are excited.

Papadimitriou describes this as a very simple mathematical model that is based on “the three main forces of life.”

Along with a group of scientists from different academic institutions, Papadimitriou detailed this model in a paper published last year in the peer-reviewed scientific journal Proceedings of the National Academy of Sciences. Assemblies were the key component of the model and enabled what the scientists called “assembly calculus,” a set of operations that can enable the processing, storing, and retrieval of information.

“The operations are not just pulled out of thin air. I believe these operations are real,” Papadimitriou said. “We can prove mathematically and validate by simulations that these operations correspond to true behaviors… these operations correspond to behaviors that have been observed [in the brain].”

Papadimitriou and his colleagues hypothesize that assemblies and assembly calculus are the correct model that explain cognitive functions of the brain such as reasoning, planning, and language.

“Much of cognition could fit that,” Papadimitriou said in his talk at the Google deep learning conference.

Natural language processing with assembly calculus

To test their model of the mind, Papadimitriou and his colleagues tried implementing a natural language processing system that uses assembly calculus to parse English sentences. In effect, they were trying to create an artificial intelligence system that simulates areas of the brain that house the assemblies that correspond to lexicon and language understanding.

“What happens is that if a sequence of words excites these assemblies in lex, this engine is going to produce a parse of the sentence,” Papadimitriou said.

The system works exclusively through simulated neuron spikes (as the brain does), and these spikes are caused by assembly calculus operations. The assemblies correspond to areas in the medial temporal lobe, Wernicke’s area, and Broca’s area, three parts of the brain that are highly engaged in language processing. The model receives a sequence of words and produces a syntax tree. And their experiments show that in terms of speed and frequency of neuron spikes, their model’s activity corresponds very closely to what happens in the brain.

The AI model is still very rudimentary and is missing many important parts of language, Papadimitriou acknowledges. The researchers are working on plans to fill the linguistic gaps that exist. But they believe that all these pieces can be added with assembly calculus, a hypothesis that will need to pass the test of time.

brain areas language processing

“Can this be the neural basis of language? Are we all born with such a thing in [the left hemisphere of our brain],” Papadimitriou asked. There are still many questions about how language works in the human mind and how it relates to other cognitive functions. But Papadimitriou believes that the assembly model brings us closer to understanding these functions and answering the remaining questions.

Language parsing is just one way to test the assembly calculus theory. Papadimitriou and his collaborators are working on other applications, including learning and planning in the way that children do at a very young age.

“The hypothesis is that the assembly calculus—or something like it—fills the bill for access logic,” Papadimitriou said. “In other words, it is a useful abstraction of the way our brain does computation.”

This article was originally published by Ben Dickson on TechTalks, a publication that examines trends in technology, how they affect the way we live and do business, and the problems they solve. But we also discuss the evil side of technology, the darker implications of new tech, and what we need to look out for. You can read the original article here.

Repost: Original Source and Author Link

Tech News

Google’s new AI can have eerily human-like conversations

Google has unveiled a creepily human-like language model called LaMDA that CEO Sundar Pichai says “can converse on any topic.”

During a demo at Google I/O 2021 on Wednesday, Pichai showcased how LaMDA can enable new ways of conversing with data, like chatting to Pluto about life in outer space or asking a paper airplane about its worst travel experiences.

Credit: Google

AI Weekly: Continual learning offers a path toward more humanlike AI

Join GamesBeat Summit 2021 this April 28-29. Register for a free or VIP pass today.

State-of-the-art AI systems are remarkably capable, but they suffer from a key limitation: statisticity. Algorithms are trained once on a dataset and rarely again, making them incapable of learning new information without retraining. This is as opposed to the human brain, which learns constantly, using knowledge gained over time and building on it as it encounters new information. While there’s been progress toward bridging the gap, solving the problem of “continual learning” remains a grand challenge in AI.

This challenge motivated a team of AI and neuroscience researchers to found ContinualAI, a nonprofit organization and open community of continual and lifelong learning enthusiasts. ContinualAI recently announced Avalanche, a library of tools compiled over the course of a year from over 40 contributors to make continual learning research easier and more reproducible. The group also hosts conference-style presentations, sponsors workshops and AI competitions, and maintains a repository of tutorial, code, and guides.

As Vincenzo Lomonaco, cofounding president and assistant professor at the University of Pisa, explains, ContinualAI is one of the largest organizations on a topic its members consider fundamental for the future of AI. “Even before the COVID-19 pandemic began, ContinualAI was funded with the idea of pushing the boundaries of science through distributed, open collaboration,” he told VentureBeat via email. “We provide a comprehensive platform to produce, discuss and share original research in AI. And we do this completely for free, for anyone.”

Even highly sophisticated deep learning algorithms can experience catastrophic learning or catastrophic interference, a phenomenon where deep networks fail to recall what they’ve learned from a training dataset. The result is that the networks have to be constantly reminded of the knowledge they’ve gained or risk becoming “stuck” with their most recent “memories.”

OpenAI research scientist Jeff Clune, who helped to cofound Uber AI Labs in 2017, has called catastrophic forgetting the “Achilles’ heel” of machine learning and believes that solving it is the fastest path to artificial general intelligence (AGI). Last February, Clune coauthored a paper detailing ANML, an algorithm that managed to learn 600 sequential tasks with minimal catastrophic forgetting by “meta-learning” solutions to problems instead of manually engineering solutions. Separately, Alphabet’s DeepMind has published research suggesting that catastrophic forgetting isn’t an insurmountable challenge for neural networks. And Facebook is advancing a number of techniques and benchmarks for continual learning, including a model that it claims is effective in preventing the forgetting of task-specific skills.

But while the past several years have seen a resurgence of research into the issue, catastrophic forgetting largely remains unsolved, according to Keiland Cooper, a cofounding member of ContinualAI and a neuroscience research associate at the University of California, Irvine. “The potential of continual learning exceeds catastrophic forgetting and begins to touch on more interesting questions of implementing other cognitive learning properties in AI,” Cooper told VentureBeat. “Transfer learning is one example, where when humans or animals learn something previously, sometimes this learning can be applied to a new context or aid learning in other domains … Even more alluring is that continual learning is an attempt to push AI from narrow, savant-like systems to broader, more general ones.”

Even if continual learning doesn’t yield the sort of AGI depicted science fiction, Cooper notes that there are immediate advantages to it across a range of domains. Cutting-edge models are being trained on increasingly larger datasets in search of better performance, but this training comes at a cost — whether waiting weeks for training to finish or the impact of the electricity usage on the environment.

“Say you run a certain AI organization that built a natural language model that was trained over weeks on 45 terabytes of data for a few million dollars,” Cooper explained. “If you want to teach that model something new, well, you’d very likely have to start from scratch or risk overwriting what it had already learned, unless you added continual learning additions to the model. Moreover, at some point, the cost to store that data will be exceedingly high for an organization, or even impossible. Beyond this, there are many cases where you can only see the data once and so retraining isn’t even an option.”

While the blueprint for a continual learning AI system remains elusive, ContinualAI aims to connect researchers and stakeholders interested in the area and support and provide a platform for projects and research. It’s grown to over 1,000 members in the three years since its founding.

“For me personally, while there has been a renewed interest in continual learning in AI research, the neuroscience of how humans and animals can accomplish these feats is still largely unknown,” Cooper said. “I’d love to see more of an interaction with AI researchers, cognitive scientists, and neuroscientists to communicate and build upon each of their fields ides towards a common goal of understanding one of the most vital aspects of learning and intelligence. I think an organization like ContinualAI is best positioned to do just that, which allows for the sharing of ideas without the boundaries of the academic or industry walls, siloed fields, or distant geolocation.”

Beyond the mission of dissemination information about continual learning, Lomonaco believes that ContinualAI has the potential to become a reference points for a more inclusive and collaborative way of doing research in AI. “Elite university and private company labs still work mostly behind close doors, [but] we truly believe in inclusion and diversity rather than selective elitiarity. We favor transparency and open-source rather than protective IP licenses. We make sure anyone has access to the learning resources she needs to achieve her potential.”

For AI coverage, send news tips to Kyle Wiggers — and be sure to subscribe to the AI Weekly newsletter and bookmark our AI channel, The Machine.

Thanks for reading,

Kyle Wiggers

AI Staff Writer


VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member

Repost: Original Source and Author Link