NeRF for 3D Reconstruction from X-ray Angiography: Possibilities and Limitations

Abstract

Neural Radiance Field (NeRF) is a promising deep learning technique based on neural rendering for three-dimensional (3D) reconstruction. This technique has overcome several limitations of 3D reconstruction techniques, such as removing the need for 3D ground truth or two-dimensional (2D) segmentations. In the medical context, the 3D reconstruction of vessels from 2D X-ray angiography is a relevant problem. For example, the treatment of coronary arteries could still benefit from 3D reconstruction solutions, as common solutions do not suffice. Challenging areas in the 3D reconstruction from X-ray angiography are the vessel morphology characteristics, such as sparsity, overlap, and the distinction between foreground and background. Moreover, sparse view and limited angle X-ray projections restrict the information available for the 3D reconstructions. Many traditional and machine learning methods have been proposed, but they rely on demanding user interactions or require large amounts of training data. NeRF could solve these limitations, given that promising results have been shown for medical (X-ray) applications. However, to the best of our knowledge, no results have been shown with X-ray angiography projections or consider the vessel morphology characteristics. This paper explores the possibilities and limitations of using NeRF for 3D reconstruction from X-ray angiography. An extensive experimental analysis is conducted to quantitatively and qualitatively evaluate the effects of the X-ray angiographic challenges on the reconstruction quality. We demonstrate that NeRF has the potential for 3D Xray angiography reconstruction (e.g., reconstruction with sparse and limited angle X-ray projections) but also identify explicit limitations (e.g., the overlap of background structures) that must be addressed in future works.