Enhancing Super-Resolution Imaging and Volume Electron Microscopy with Deep Learning Algorithms Based on Diffusion Models

Background Introduction

Electron Microscopy (EM) as a high-resolution imaging tool has made significant breakthroughs in cell biology. Traditional EM techniques are primarily used for two-dimensional imaging, and although they have revealed complex nanometer-scale cellular structures, they have certain limitations in studying three-dimensional (3D) structures. Volume Electron Microscopy (VEM), as a more advanced technology, achieves 3D imaging of cells and tissues by combining serial sectioning and tomographic scanning techniques (such as Transmission Electron Microscopy TEM and Scanning Electron Microscopy SEM), allowing the extraction of nanometer-scale 3D structures of cells, tissues, and even small model organisms.

Although VEM technology has overcome the limitations of traditional two-dimensional EM, there is an inherent trade-off between imaging speed and quality, leading to limitations in imaging area and volume. Furthermore, generating isotropic data remains a significant challenge for VEM. Currently, VEM typically produces anisotropic data, and even with state-of-the-art Focused Ion Beam-Scanning Electron Microscopy (FIB-SEM) techniques, isotropic data can only be generated within a relatively small volume (approximately 300μm x 300μm x 300μm). To overcome these technical limitations, researchers have recently introduced computational methods and deep learning techniques to accelerate the imaging process and improve image quality.

Paper Source

This research was a collaborative effort between research teams from the Department of Chemistry at the University of Hong Kong, the School of Molecular Sciences at the University of Western Australia, and the Department of Electrical and Electronic Engineering at the University of Hong Kong. The paper was published in Nature Communications on May 20, 2024. The authors include Chixiang Lu, Kai Chen, Heng Qiu, Xiaojun Chen, Gu Chen, Xiaojuan Qi, and Haibo Jiang.

Research Details

Workflow

This research proposed a deep learning method based on diffusion models, named emdiffuse, aimed at enhancing the super-resolution imaging capabilities of electron microscopy and volume electron microscopy. emdiffuse includes a series of algorithms for denoising, super-resolution tasks, and generating isotropic datasets.

Denoising

The denoising part of emdiffuse is called emdiffuse-n, consisting of data acquisition, image processing, and diffusion models. First, the research team acquired electron microscopy training pairs with different acquisition times, and then used a hierarchical method to precisely align and register the noisy images with the reference images. Subsequently, a diffusion model called udim was trained to remove noise.

During the inference stage, udim can generate multiple possible denoised predictions from the input image, improving prediction accuracy and reliability. The research team validated emdiffuse-n’s outstanding performance in generating images with complex super-structural information by comparing it with three widely adopted denoising methods (including care, rcan, and pssr) and two self-supervised methods (noise2noise and noise2void).

Super-Resolution

For the super-resolution task, emdiffuse-r is used to reconstruct high-resolution images from low-resolution inputs. The research team acquired a mouse cortex super-resolution dataset containing noisy inputs and ground truth data for training and testing emdiffuse-r and other benchmark models. The experimental results showed that emdiffuse-r outperformed other methods in improving resolution and resolving detailed structures, successfully distinguishing nearby synaptic vesicles and mitochondrial cristae.

Isotropic Reconstruction

emdiffuse was further extended for isotropic reconstruction. We developed the vemdiffuse-i and vemdiffuse-a models, respectively, for generating isotropic volumes from anisotropic volumes. vemdiffuse-i generates isotropic data from small-volume isotropic training data, while vemdiffuse-a can perform the reconstruction task using only anisotropic training data without the need for isotropic training data.

Main Results

Denoising Performance of emdiffuse-n

In the experiments, emdiffuse-n not only generated ultra-high-resolution denoised images but also had the ability to self-assess the reliability of its predictions. The research team successfully improved the model’s stability and performance by introducing a prediction difficulty map during the training process, preventing difficult samples from negatively impacting the model weights.

Super-Resolution Performance of emdiffuse-r

emdiffuse-r demonstrated superior super-resolution capabilities, especially when processing low-noise input levels. With an optimized prediction generation method, emdiffuse-r could double the image resolution while providing a 36-fold increase in imaging speed.

Isotropic Data Reconstruction with vemdiffuse-i and vemdiffuse-a

vemdiffuse-i can generate high-quality data similar to the original isotropic volume from anisotropic data and accurately reconstruct the structures of organelles such as mitochondria and the endoplasmic reticulum. On the other hand, vemdiffuse-a can perform high-quality isotropic data generation without the need for isotropic training data, allowing existing large datasets to be used for studying 3D cellular structures.

Conclusion and Significance

The proposed emdiffuse not only significantly accelerated the imaging process of traditional EM and VEM but also significantly improved image quality. Particularly, in the absence of isotropic training data, it achieved the reconstruction from anisotropic data to isotropic data, a capability with practical value in many research fields. emdiffuse demonstrated its strong generalization and transfer abilities across various biological samples, requiring only one dataset for model fine-tuning. Experimental results on different biological samples showed that this method has broad application prospects and can drive the study of complex subcellular nanometer structures within large-volume biological systems, opening new avenues for scientific exploration.

Research Highlights

1. Multifunctionality: emdiffuse integrates denoising, super-resolution, and isotropic reconstruction capabilities, adapting to different EM and VEM application needs.
2. Efficiency: Through the application of diffusion models, efficient training and inference processes were achieved, significantly improving image resolution and imaging speed.
3. Reliability: With the incorporation of self-assessment functionality, it can evaluate the reliability of predictions in real-time, aiding researchers in making more accurate judgments.
4. Broad Applicability: emdiffuse has demonstrated outstanding performance across different data types and noise levels and can easily adapt to new biological datasets.

This research on emdiffuse will further drive the advancement of EM and VEM technologies, providing more powerful tools for the life sciences field and revealing the internal structures of more complex biological systems.