Ten years of neuroscience at Google yields maps of human brain

The human brain uses no more power than a dim incandescent light bulb, yet it can accomplish feats still not possible with the largest artificial computing systems. To understand how requires a level of understanding more profound than knowing what part of the brain is responsible for what function. The field of connectomics aims to achieve this by precisely mapping how each cell is connected to others. Building detailed maps of how brains from many types of organisms are wired is transforming our understanding of how brains work. This could help researchers understand neurological disorders and also answer fundamental questions, such as how memories form.

This year marks the tenth anniversary of the formation of Google Research’s Connectomics research team. We have made connectomes possible through the development of machine learning (ML) algorithms and software tools that process and visualize data at unprecedented scale. Marking this anniversary, today we publish in Science, “A petavoxel fragment of human cerebral cortex reconstructed at nanoscale resolution” in collaboration with Jeff Lichtman of Harvard University and others. Released previously as a preprint, this connectome describes a piece of human brain tissue the size of half a grain of rice that nonetheless requires 1.4 petabytes (1.4 million gigabytes) to encode — including about 16,000 neurons, 32,000 glia, 8,000 blood vessel cells (for a total of ~57,000 cells) and 150 million synapses. This project revealed never-before-seen structures within the human brain that may change our understanding of how our brains work.

The first connectome was published in 1986 — before AI tools existed — for the 302 neurons in the nematode model organism Caenorhabditis elegans. It took 16 years to create it from cross-sectional microscope images of the worm. Researchers manually colored in cells from one cross-section to another to visualize the connections in this simple nervous system.

When the Connectomics team at Google launched ten years ago, we were excited about how innovations in AI and working with large datasets could enable us to move from 302 neurons to the tens of thousands or millions found in more complex organisms. Our work required novel algorithms capable of handling the tremendous amounts of data — now petabytes — these studies generate. We developed flood-filling networks to replace the manual effort of coloring in cells across images. These networks allow automated reconstruction of neurons through layers of tissue. Building on this, our SegCLR algorithm automatically identifies distinct parts of cells and cell types within these networks. We also developed TensorStore, an open-source C++ and Python software library to store and manage massive multi-dimensional datasets. This tool has realized benefits well beyond connectomics and is now widely used at Google and across the broader ML community.

We first demonstrated these algorithms when we released the connectome for the “hemibrain” of the fruit fly Drosophila melanogaster in 2020. Revealing the connections among 25,000 neurons in a central portion of the fruit fly brain, this reconstruction has been used by other researchers to make findings about learning, memory, and behavior in the fruit fly. Groups have since published hundreds of papers that build on the fruit fly connectome.

Through collaborations with researchers at the Howard Hughes Medical Institute, Harvard University, and Max Planck Institute we have also published connectomes for portions of the brains of the zebra finch and zebra fish larvae.

In the work published today, our team reports a new milestone: a synaptic-resolution reconstruction of a 1 cubic mm piece of human brain tissue. Our collaborators generated the dataset using a sample of brain tissue from the left anterior temporal lobe that was removed during brain surgery on a person with epilepsy. Lichtman’s team used a multibeam scanning electron microscope to gather high resolution images of more than 5,000 slices of tissue, each roughly 30 nanometers thick. Image acquisition alone took 326 days. Then our team’s tools stitched and aligned the image data, reconstructed the three dimensional structure of each cell, including its axons and dendrites, identified synaptic connections, and classified cell types. The reconstruction revealed several surprises.

The central collaborative tool for interacting with the brain tissue data is Neuroglancer, a browser-based platform — originally created for our reconstruction of the fruit fly brain — that allows users to explore the dataset, including the AI-generated annotations of cell type for each cell.

Another key tool is CAVE, which was developed by researchers at Princeton and the Allen Institute as an extension of Neuroglancer and allows users to proofread and update the annotated data to improve the dataset as a part of their own research. Researchers can select particular neurons or their connections and manually refine and annotate their reconstructions. CAVE has been available for smaller datasets including the fly connectome, but in this work we updated the algorithms to be able to handle a dataset of this size. The proofreading that this interface allows is a key final step in completing the connectome reconstruction.

Our hope is that the new Science paper will encourage other researchers to use the data to make additional discoveries about this part of the human brain. We are pleased that this is already happening. The preprint has been downloaded over 30,000 times and has been cited in over 200 papers. These include a recent study in Neuron that analyzed how tiny “cilia” attached to brain cells found in the dataset may play an important role in the functional organization of the human cortex, and a computational investigation into how the shape of the neurons relates to other properties of human cortex organization. The interactive viewer of the dataset has been accessed over 100,000 times.

Looking ahead, we are working to scale connectomics to the hippocampus of the mouse with partners at Harvard, Princeton, and elsewhere. Our team and partners at the Institute of Science and Technology Austria have also recently proposed a method for connectomic mapping using light microscopes that could make it much easier to collect images for connectomic analysis. The team has also begun more intensive study of tissue associated with neurological diseases, such as Alzheimer’s disease, in order to better understand the potential structural changes associated with these devastating conditions.

We thank our long-term academic collaborators in the Lichtman Lab (Harvard University), and acknowledge core contributions from the Connectomics Team at Google. We are grateful to Elise Kleeman and Jessica Marshall for their help. Thanks to Lizzie Dorfman, Michael Brenner, John Platt, Jay Yagnik and Jeff Dean for their support, coordination and leadership.