One possible route to low power computing is neuromorphic computing- computation modeled after the human brain.
Neural nets, like the ones powering LLMs are quite simple compared to the brain, but operate on the same principle as the brain. A node makes a calculation- in the case of AI, some linear function, then passes that signal to out to the connected nodes.
How does signal move around in the brain?
The human brain has about 86 billion neurons and 1000 times as many synapses making connections between neurons. This node and branch structure makes the brain an assemblage similar to the rhizome, though it lacks some properties of rhizomes. I will discuss that in a later post.
While imagining the future, gathering and processing sensory information, sending signals to our limbs, writing blog posts, and all other tasks, the brain only demands a measly 20W of power. For reference, this computer I’m using has an 850W power supply, though it leaves a lot of overhead to run at peak efficiency.
Neurons in the brain fire according to what is known as the “leaky integrate and fire” (LIF) model. Signal from upstream flows into a neuron as a series of pulses from each synapse. Charge accumulates on the neuron and slowly leaks out like water leaking from a hole in a bucket. If a threshold voltage is reached, the neuron fires and sends a signal downstream to each connected neuron. Finally, the neuron resets to baseline voltage. To further complicate this process, each synapse is weighted differently, and some synapses can delay the signal between neurons.
The brain uses little power, is highly stochastic, and has no internal clock. Brains also exhibit plasticity, making new connections between neurons. Synapse weights in the brain are analog, real numbers and are stored in the synapses themselves. [1]
Any computational device or technique that seeks to use any of these properties of the brain- low power, high stochastic algorithms, in-media storage of synaptic weights, plasticity in artificial neurons, or analog synaptic weights, is referred to as “neuromorphic computing”.
[1] Marković, Danijela, Alice Mizrahi, Damien Querlioz, and Julie Grollier. 2020. “Physics for Neuromorphic Computing.” Nature Reviews Physics 2 (9): 499–510. https://doi.org/10.1038/s42254-020-0208-2.
Stuff Cam Wrote 