Jump to Content

Connectomics

Our goal is to leverage Google expertise and resources to advance understanding of the structure and function of the brain.

Imaging

Our goal is to leverage Google expertise and resources to advance understanding of the structure and function of the brain.

About our team

A major hypothesis in modern neuroscience is that neuron-to-neuron connectivity structure in the brain can be linked to function -- how the brain encodes memories, extracts features from perceptual stimuli, and makes decisions. However, the structure of these brain networks has remained largely unknown, due to technical difficulties involved in imaging and reconstructing the brain in 3D.

New microscopy techniques have begun to address the challenge of imaging the brain in 3D at nanometer resolution, and 4D at cellular resolution, but this has led to a huge bottleneck in the subsequent step of data analysis. Our goal is to help solve some of these data analysis problems and thus enable a high-throughput approach to studying the network architecture of the brain.

In order to facilitate this work we collaborate with the Max Planck Institute, HHMI Janelia Research Campus, Harvard University, and other organizations.

Team focus summaries

Automated 3d brain reconstruction

We develop algorithms and software for automating the process of aligning, segmenting, and annotating petabyte-scale 3d images of brain tissue.

Visualization and infrastructure

We develop software such as TensorStore and Neuroglancer which helps store, process, and visualize large n-dimensional images and volumes.

Featured publications

A connectome and analysis of the adult Drosophila central brain
Stephen M Plaza
Anne K Scott
Masayoshi Ito
Gregory SXE Jefferis
Tansy Yang
Marisa Dreher
Sari McLin
Sean M Ryan
Feng Li
Samantha Finley
Robert Svirskas
Alexander S Bates
Christopher Ordish
Christopher J Knecht
Jens Goldammer
Miatta Ndama
Jon Thomson Rymer
Nicole Neubarth
C Shan Xu
Christopher M Patrick
Jackie Swift
Dorota Tarnogorska
Gary B Huang
Shin-ya Takemura
Ashley L Scott
Satoko Takemura
Nicole A Kirk
Kenneth J Hayworth
Natalie L Smith
Michael Cook
Jolanta A Borycz
Louis K Scheffer
Gerald M Rubin
Patricia K Rivlin
Iris Talebi
SungJin Kim
Caitlin Ribeiro
Ting Zhao
Neha Rampally
Nicholas Padilla
Stephan Saalfeld
Nora Forknall
Claire Smith
Aya Shinomiya
Tanya Wolff
Vivek Jayaraman
Donald J Olbris
Marta Costa
Madelaine K Robertson
Nneoma Okeoma
Audrey Francis
Brandon S Canino
Natasha Cheatham
Alia Suleiman
Caroline Mooney
Lowell Umayam
Ian Meinertzhagen
Tyler Paterson
Khaled A Khairy
Samantha Ballinger
Reed George
Omotara Ogundeyi
Alanna Lohff
Margaret A Sobeski
Jody Clements
Bryon Eubanks
Harald F Hess
Dagmar Kainmueller
Kelsey Smith
Emily M Phillips
Kazunori Shinomiya
Philip M Hubbard
Emily Tenshaw
Dennis A Bailey
Ruchi Parekh
Eric T Trautman
Megan Sammons
William T Katz
Julie Kovalyak
Hideo Otsuna
John J Walsh
Tom Dolafi
Charli Maldonado
Kei Ito
Gary Patrick Hopkins
Jane Anne Horne
Erika Neace
Emily M Joyce
Temour Tokhi
Kelli Fairbanks
Zhiyuan Lu
Elliott E Phillips
Emily A Manley
Stuart Berg
Takashi Kawase
Chelsea X Alvarado
Shirley Lauchie
Philipp Schlegel
David Ackerman
John Bogovic
Octave Duclos
Michal Januszewski
Viren Jain
Laramie Leavitt
Jeremy Maitlin-Shepard
Peter H Li
Larry Lindsey
Tim Blakely
eLife, 9 (2020)
Preview abstract The neural circuits responsible for animal behavior remain largely unknown. We summarize new methods and present the circuitry of a large fraction of the brain of the fruit fly Drosophila melanogaster. Improved methods include new procedures to prepare, image, align, segment, find synapses in, and proofread such large data sets. We define cell types, refine computational compartments, and provide an exhaustive atlas of cell examples and types, many of them novel. We provide detailed circuits consisting of neurons and their chemical synapses for most of the central brain. We make the data public and simplify access, reducing the effort needed to answer circuit questions, and provide procedures linking the neurons defined by our analysis with genetic reagents. Biologically, we examine distributions of connection strengths, neural motifs on different scales, electrical consequences of compartmentalization, and evidence that maximizing packing density is an important criterion in the evolution of the fly's brain. View details
Denoising-based Image Compression for Connectomics
Jeff W. Lichtman
Alex Shapson-Coe
Richard L. Schalek
Michal Januszewski
Johannes Ballé
Viren Jain
David Charles Minnen
bioRxiv (2021)
Preview abstract Connectomic reconstruction of neural circuits relies on nanometer resolution microscopy which produces on the order of a petabyte of imagery for each cubic millimeter of brain tissue. The cost of storing such data is a significant barrier to broadening the use of connectomic approaches and scaling to even larger volumes. We present an image compression approach that uses machine learning-based denoising and standard image codecs to compress raw electron microscopy imagery of neuropil up to 17-fold with negligible loss of reconstruction accuracy. View details
The Mind of a Mouse
Narayanan Kasthuri
Bruce R. Rosen
John H.R. Maunsell
Davi D. Bock
David C. Van Essen
R. Clay Reid
Yann LeCun
Kristen M. Harris
Winfried Denk
Gerald M. Rubin
Adrienne L. Fairhall
David W. Tank
Doris Tsao
Catherine Dulac,
Edward M. Callaway
Liqun Luo
H. Sebastian Seung
Jeff W. Lichtman
Peter B. Littlewood
Larry F. Abbott
Moritz Helmstaedter
Terrence J. Sejnowski
Ila Fiete
Karel Svoboda
Viren Jain
Cell, 182 (2020)
Preview abstract Large scientific projects in genomics and astronomy are influential not because they answer any single ques- tion but because they enable investigation of continuously arising new questions from the same data-rich sources. Advances in automated mapping of the brain’s synaptic connections (connectomics) suggest that the complicated circuits underlying brain function are ripe for analysis. We discuss benefits of mapping a mouse brain at the level of synapses. View details
An anatomical substrate of credit assignment in reinforcement learning
Michale S. Fee
Jorgen Kornfeld
Philipp Schubert
Winfried Denk
Michal Januszewski
Viren Jain
bioRxiv (2020)
Preview abstract How is experience used to improve performance? In both biological and artificial systems, the optimization of parameters that affect behavior requires a process that determines whether a parameter affects the outcome and then modifies the parameter accordingly. Central to the recent bloom of artificial intelligence has been the error-backpropagation algorithm(Rumelhart, Hinton, and Williams 1986) , which computationally retraces the signal from the output to each synapse (weight) and allows a large number of parameters to be optimized in parallel at high learning rates. Biological systems, however, lack an obvious mechanism to retrace the signal path. Here we show, by combining high-throughput volume electron microscopy (Denk and Horstmann 2004) and automated connectomic analysis(Januszewski et al. 2018; Dorkenwald et al. 2017; Schubert et al. 2019) , that the synaptic architecture of songbird basal ganglia supports a form of local credit assessment proposed in a model of songbird reinforcement learning (M. S. Fee and Goldberg 2011). We show that three of this model’s major predictions hold true: first, cortical axons that encode exploratory motor variability terminate predominantly on dendritic shafts of spiny neurons. Second, cortical axons that encode timing seek out spines, which enable calcium-based coincidence detection (R. Yuste and Denk 1995) and appear to be capable of creating and storing eligibility traces (Yagishita et al. 2014). Third, synapse pairs that presynaptically share a cortical timing axon and post-synaptically a medium spiny dendrite are substantially more similar in size than expected, indicating a history of Hebbian plasticity (Bartol et al. 2015; Kasthuri et al. 2015) . Combined with numerical simulations these data provide strong evidence for a model of basal ganglia learning with a biologically plausible credit assignment mechanism. View details
Automated Reconstruction of a Serial-Section EM Drosophila Brain with Flood-Filling Networks and Local Realignment
István Taisz
Matthew Nichols
Alexander Shakeel Bates
Gregory S.X.E. Jefferis
Zhihao Zheng
Feng Li
Eric Perlman
Davi Bock
Michal Januszewski
Viren Jain
Laramie J Leavitt
Jeremy Maitin-Shepard
Mike Tyka
Peter Li
Larry Lindsey
Tim Blakely
bioRxiv (2019)
Preview abstract Reconstruction of neural circuitry at single-synapse resolution is an attractive target for improving understanding of the nervous system in health and disease. Serial section transmission electron microscopy (ssTEM) is among the most prolific imaging methods employed in pursuit of such reconstructions. We demonstrate how Flood-Filling Networks (FFNs) can be used to computationally segment a forty-teravoxel whole-brain Drosophila ssTEM volume. To compensate for data irregularities and imperfect global alignment, FFNs were combined with procedures that locally re-align serial sections as well as dynamically adjust and synthesize image content. The proposed approach produced a largely merger-free segmentation of the entire ssTEM Drosophila brain, which we make freely available. As compared to manual tracing using an efficient skeletonization strategy, the segmentation enabled circuit reconstruction and analysis workflows that were an order of magnitude faster. View details
High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks
Jörgen Kornfeld
Winfried Denk
Michał Januszewski
Viren Jain
Jeremy B Maitin-Shepard
Mike Tyka
Peter H. Li
Larry Lindsey
Art Pope
Tim Blakely
Nature Methods (2018)
Preview abstract Reconstruction of neural circuits from volume electron microscopy data requires the tracing of cells in their entirety, including all their neurites. Automated approaches have been developed for tracing, but their error rates are too high to generate reliable circuit diagrams without extensive human proofreading. We present flood-filling networks, a method for automated segmentation that, similar to most previous efforts, uses convolutional neural networks, but contains in addition a recurrent pathway that allows the iterative optimization and extension of individual neuronal processes. We used flood-filling networks to trace neurons in a dataset obtained by serial block-face electron microscopy of a zebra finch brain. Using our method, we achieved a mean error-free neurite path length of 1.1 mm, and we observed only four mergers in a test set with a path length of 97 mm. The performance of flood-filling networks was an order of magnitude better than that of previous approaches applied to this dataset, although with substantially increased computational costs. View details
Superhuman Accuracy on the SNEMI3D Connectomics Challenge
Jonathan Zung
H. Sebastian Seung
Kisuk Lee
Viren Jain
Peter H. Li
arXiv, abs/1706.00120 (2017)
Preview abstract For the past decade, convolutional networks have been used for 3D reconstruction of neurons from electron microscopic (EM) brain images. Recent years have seen great improvements in accuracy, as evidenced by submissions to the SNEMI3D benchmark challenge. Here we report the first submission to surpass the estimate of human accuracy provided by the SNEMI3D leaderboard. A variant of 3D UNet is trained on a primary task of predicting affinities between nearest neighbor voxels, and an auxiliary task of predicting long-range affinities. The training data is augmented by simulated image defects. The nearest neighbor affinities are used to create an oversegmentation, and then supervoxels are greedily agglomerated based on mean affinity. The resulting SNEMI3D score exceeds the estimate of human accuracy by a large margin. While one should be cautious about extrapolating from the SNEMI3D benchmark to real-world accuracy of large-scale neural circuit reconstruction, our submission inspires optimism that the goal of full automation may be realizable in the future. View details
Adversarial Image Alignment and Interpolation
Viren Jain
arXiv, abs/1707.00067 (2017)
Preview abstract Volumetric (3d) images are acquired for many scientific and biomedical purposes using imaging methods such as serial section microscopy, CT scans, and MRI. A frequent step in the analysis and reconstruction of such data is the alignment and registration of images that were acquired in succession along a spatial or temporal dimension. For example, in serial section electron microscopy, individual 2d sections are imaged via electron microscopy and then must be aligned to one another in order to produce a coherent 3d volume. State of the art approaches find image correspondences derived from patch matching and invariant feature detectors, and then solve optimization problems that rigidly or elastically deform series of images into an aligned volume. Here we show how fully convolutional neural networks trained with an adversarial loss function can be used for two tasks: (1) synthesis of missing or damaged image data from adjacent sections, and (2) fine-scale alignment of block-face electron microscopy data. Finally, we show how these two capabilities can be combined in order to produce artificial isotropic volumes from anisotropic image volumes using a super-resolution adversarial alignment and interpolation approach. View details
Combinatorial Energy Learning for Image Segmentation
Pieter Abbeel
Michal Januszewski
Viren Jain
Jeremy B. Maitin-Shepard
Peter H. Li
Advances in Neural Information Processing Systems 29 (2016)
Preview abstract We introduce a new machine learning approach for image segmentation that uses a neural network to model the conditional energy of a segmentation given an image. Our approach, combinatorial energy learning for image segmentation (CELIS) places a particular emphasis on modeling the inherent combinatorial nature of dense image segmentation problems. We propose efficient algorithms for learning deep neural networks to model the energy function, and for local optimization of this energy in the space of supervoxel agglomerations. We extensively evaluate our method on a publicly available 3-D microscopy dataset with 25 billion voxels of ground truth data. On an 11 billion voxel test set, we find that our method improves volumetric reconstruction accuracy by more than 20% as compared to two state-of-the-art baseline methods: graph-based segmentation of the output of a 3-D convolutional neural network trained to predict boundaries, as well as a random forest classifier trained to agglomerate supervoxels that were generated by a 3-D convolutional neural network. View details

Highlighted work

Some of our locations

Some of our people