在透射電鏡斷層成像技術中，由于電子對材料的穿透十分有限，材料在高角度的投影數(shù)據(jù)通常無法獲取。而這些缺失的投影數(shù)據(jù)會在最終的三維成像結果中引發(fā)楔形失真。

Fig. 1 Schematics of missing wedge artifact in the conventional electron tomography and our UsiNet workflow.

該研究提出了一種基于深度學習的算法，利用卷積神經(jīng)網(wǎng)絡對缺失的投影數(shù)據(jù)進行補全，以消除透射電鏡斷層成像中的楔形失真。在同類研究中，由于透射電鏡斷層成像數(shù)據(jù)集的缺乏，標簽的獲取一直是神經(jīng)網(wǎng)絡訓練的難題。

來自美國伊利諾伊大學香檳分校材料科學與工程系的陳倩教授團隊，設計了無監(jiān)督式的神經(jīng)網(wǎng)絡訓練流程來避免訓練集標簽獲取的難題。相比于同類算法，該研究的亮點在于其完全不依賴于任何數(shù)據(jù)庫以及計算機模擬數(shù)據(jù)就能完成對模型的訓練和對缺失投影的補全。

作者首先使用計算機模擬的納米粒子電鏡投影數(shù)據(jù)對算法進行了驗證，得到了對算法性能在不同的缺失投影角度范圍下，不同噪聲影響下和不同納米粒子形貌下的定量表征。

Fig. 4 Unsupervised sinogram inpainting implemented on 3D images.?

隨后該算法被應用于實驗上獲得的真實數(shù)據(jù)并成功消除了實驗數(shù)據(jù)中的楔形失真。相比于其他的傳統(tǒng)校正算法，該算法能對楔形失真進行最徹底的去除，并還原樣品中納米粒子的真實三維形貌。

Fig. 5 Orientation-dependent missing wedge artifact and comparison between different reconstruction algorithms.

該研究是基于人工智能的圖像算法在電子顯微鏡中的一次成功應用，也是納米材料三維表征技術的重大進展。相關論文近期發(fā)布于npj?Computational Materials?10:?28?(2024)。

Fig. 6 Comparison of 3D reconstructions of experimentally synthesized NPs with and without inpainting.

Editorial Summary

In electron tomography, due to the limited penetration of electrons into materials, projection data at high angles is often inaccessible. These missing projection data can cause wedge distortion in the final three-dimensional imaging result. This study proposes a deep learning-based algorithm that utilizes convolutional neural networks to complete the missing projection data, thereby eliminating wedge distortion in electron tomography. In relevant studies, due to the lack of electron tomography datasets, obtaining labels has always been a challenge for such neural network training.?

Fig. 7 Visualizing the heterogeneity of experimentally synthesized NPs.

Professor Qian Chen’s team from the Department of Materials Science and Engineering at the University of Illinois at Urbana-Champaign has developed an unsupervised neural network training workflow to bypass the difficulty of obtaining training set labels. The highlight of this study, compared to other algorithms, is that it can train and apply the model without relying on any databases or computer-simulated data. The authors first validated the algorithm using computer-simulated electron microscopy projection data of nanoparticles and obtained quantitative evaluation of the algorithm performance under different ranges of missing projection angles, noise levels, and nanoparticle morphologies. The algorithm was then applied to experimentally obtained data and successfully eliminated wedge distortion in the experimental data. Compared to other traditional correction algorithms, this algorithm can thoroughly remove wedge distortion and restore the true three-dimensional morphology of nanoparticles. This study represents a successful application of artificial intelligence-based image algorithms in electron microscopy and a significant advancement in three-dimensional characterization techniques for nanomaterials.This?article was recently?published in?npj?Computational Materials?10:?28?(2024).

原文Abstract及其翻譯

No ground truth needed: unsupervised sinogram inpainting for nanoparticle electron tomography (UsiNet) to correct missing wedges (利用無監(jiān)督式圖像修補進行的透射電鏡斷層成像失真校正)

Lehan Yao,?Zhiheng Lyu, Jiahui Li &?Qian Chen

Abstract Complex natural and synthetic materials, such as subcellular organelles, device architectures in integrated circuits, and alloys with microstructural domains, require characterization methods that can investigate the morphology and physical properties of these materials in three dimensions (3D). Electron tomography has unparalleled (sub-)nm resolution in imaging 3D morphology of a material, critical for charting a relationship among synthesis, morphology, and performance. However, electron tomography has long suffered from an experimentally unavoidable missing wedge effect, which leads to undesirable and sometimes extensive distortion in the final reconstruction. Here we develop and demonstrate Unsupervised Sinogram Inpainting for Nanoparticle Electron Tomography (UsiNet) to correct missing wedges. UsiNet is the first sinogram inpainting method that can be realistically used for experimental electron tomography by circumventing the need for ground truth. We quantify its high performance using simulated electron tomography of nanoparticles (NPs). We then apply UsiNet to experimental tomographs, where >100 decahedral NPs and vastly different byproduct NPs are simultaneously reconstructed without missing wedge distortion. The reconstructed NPs are sorted based on their 3D shapes to understand the growth mechanism. Our work presents UsiNet as a potent tool to advance electron tomography, especially for heterogeneous samples and tomography datasets with large missing wedges, e.g. collected for beam sensitive materials or during temporally-resolved in-situ imaging.

摘要復雜的自然或合成材料，如亞細胞器、集成電路中的器件結構和具有微結構域的合金，需要三維的表征方法來研究其形態(tài)和物理性質。透射電鏡斷層成像技術在表征材料的三維形態(tài)方面具有超高的納米級分辨率，這對于建立材料合成、材料形態(tài)與材料性能三者之間的關系至關重要。然而，透射電鏡斷層成像技術長期以來一直受到實驗上不可避免的缺失楔形效應的困擾，這導致最終的三維重建中經(jīng)常出現(xiàn)圖像失真。本研究中我們開發(fā)了基于深度學習圖像修補進行的、用于納米粒子的透射電鏡斷層成像失真校正算法，且本研究是相關領域第一個無需訓練標簽就能進行的基于正弦圖補全的失真校正算法。本文中我們首先使用計算機模擬的納米粒子斷層成像數(shù)據(jù)對算法進行性能驗證，然后將此算法應用于實驗中獲得的成像數(shù)據(jù)，并取得了良好結果。我們使用本算法在對上百個十面體納米粒子進行的三維重建中成功避免了楔形失真，并使用這些三維表征數(shù)據(jù)對納米粒子的形貌進行了分析，從而取得了對其生長機理的了解。本工作是對透射電鏡斷層成像技術的一項重大推進，也是對異質材料、電子束敏感材料和原位成像等特殊樣品的三維表征的解決方案。