Inquire about purchasing the book | Table of Contents | Annotated Bibliography | Class on Nature of Mind
These are excerpts and elaborations from my book "The Nature of Consciousness"
|
Energy-based Models In 1982 the US physicist
John Hopfield("Neural Networks And Physical Systems With Emergent
Collective Computational Abilities) revived the field by proving the second
milestone theorem of neural networks. He developed a model inspired by the
"spin glass" material, which resembles a one-layer neural network in
which: weights are distributed in a symmetrical fashion; the learning rule is
Hebbian (the rule that the strength of a connection is proportional to how
frequently it is used, a rule originally proposed by the Canadian psychologist
DonaldHebb); neurons are binary; and each neuron is connected to every other
neuron. As they learn, Hopfield's nets develop configurations that are
dynamically stable (or "ultrastable"). Their dynamics is dominated by
a tendency towards a very high number of locally stable states, or
"attractors". Every memory is a local "minimum" for an
energy function similar to potential energy. Hopfield's argument, based on
Physics, proved that, despite Minsky's critique, neural networks are feasible. Hopfield's key intuition was
to note the similarity with statistical mechanics. Statistical mechanics
translates the laws of Thermodynamics into statistical properties of large sets
of particles. The fundamental tool of statistical mechanics (and soon of this
new generation of neural networks) is the Boltzmanndistribution (actually
discovered by Josiah-Willard Gibbs in 1901), a method to calculate the
probability that a physical system is in a specified state. Research on neural networks
picked up again. In 1982 Kunihiko Fukushima built the "Neocognitron",
based on a model of the visual system. Building on Hopfields
ideas, the British computer scientist Geoffrey Hinton and Terrence Sejnowsky
(Massively parallel architectures for A.I., 1983) developed an algorithm for
the "Boltzmannmachine" based on Hopfield's simulated annealing. In that machine, Hopfield's learning rule is
replaced with the rule of annealing in metallurgy (start off the system at very
high "temperature" and then gradually drop the temperature to zero),
which several mathematicians were proposing as a general-purpose optimization
rule. In this model, therefore, units
update their state based on a stochastic decision rule. The Boltzmann machine turned out to be even
more stable than Hopfield's, as it will always ends in a global minimum (the
lowest energy state). Probabilistic reasoning had
been introduced into Artificial Intelligence by the Israeli computer scientist
Judea Pearl with his Bayesian networks (1985).
In 1974 Paul Werbos proposed a "backpropagation" algorithm for neural networks.
In 1986 David Rumelhart and George Hinton rediscovered Werbos' backpropagation algorithm ("Learning Representations by Back-propagating Errors", 1986),
a "gradient-descent" algorithm that
quickly became the most popular learning rule.
The generalized "Delta
Rule" was basically an adaptation of the Widrow-Hoff error-correction rule
to the case of multi-layered networks, by moving backwards from the output
layer to the input layer. This was also the definitive answer to Minsky's critique, as it proved to be able to solve all of the unsolved
problems. Hinton and Rumelhart focused
on gradient-descent learning procedures. Each connection computes the
derivative, with respect to its strength, of a global measure of error in the
performance of the network, and then adjusts its strength in the direction that
decreases the error. In other words,
the network adjusts itself to counter the error it made. Tuning a network to perform a specific task
is a matter of stepwise approximation. The problem with these
methods was that they are cumbersome (if not plain impossible) when applied to
deeply-layered neural networks, precisely the ones needed to mimic what the
brain does. |
| 
| 
| 
|
