Single-image 3D modeling of the colon can enable rapid disease diagnosis

Using just a single image taken by a capsule endoscopy camera, scientists have succeeded in creating a three-dimensional model of the colon. This new method provides much better images and can help specialists detect disease more rapidly.

Our work shows that it is possible to re-create a fairly accurate three-dimensional (3D) model of the colon of some patients based on a single image taken by a capsule endoscopy camera – even if it’s a low-quality image.”

Pål Anders Floor, Researcher, Department of Computer Science, NTNU

Goal: better images, faster diagnoses

He has been working for several years on using images from capsule endoscopy cameras to reconstruct an almost identical 3D model of the intestines. These cameras were first used more than 20 years ago, but low-quality images and image noise prevented this smart technology from ever really taking off. The new study, “Single-Image-Based 3D Reconstruction of Endoscopic Images,” was recently published in the Journal of Imaging. This is part of the doctoral research being undertaken by Bilal Ahmad, supervised by Floor and Professor Ivar Farup.

The main aim of this research is to improve the images that specialists use to detect abnormalities in the gastrointestinal tracts of their patients. They can then more easily detect disease and make diagnoses faster. These are important steps on the path to the ultimate goal: fighting bowel cancer, the second most common type of cancer in Norway.

A rubber bowel, an endoscope and mathematics

Using an artificial colon, single images taken by endoscopy and a special algorithm, the NTNU researchers have managed to reconstruct a three-dimensional gut model. The endoscope does the same job as a capsule camera – with some important differences: the images have a higher resolution and the camera brightness can be adjusted manually.

The algorithm that the researchers have used is a called a shape-from-shading algorithm (SFS). This is a mathematical model that can be used to reconstruct a three-dimensional shape from a single, two-dimensional image.

True bowel copy recreated on a PC

If algorithms are fed with images that are noisy, or where the colour is distorted, the errors will spread to the 3D model.

“We show that by carefully calibrating and pre-processing the images, we can obtain a good 3D model based on just one image – even if the image is full of noise and visual distortion,” Floor said.

Great benefits for doctors

Floor and his colleagues have also been working on developing 3D models based on sequences involving several images taken by capsule cameras. This enables the models to re-create longer parts of the colon. The models should make it possible to find out exactly where in the intestines there are abnormalities and signs of disease.

“We have previously shown that doctors find this type of model useful when assessing patients, along with having a video from a capsule camera. Now we have shown that we can calculate relatively accurate 3D models using real images and technology that already exists,” Floor said.

Better, faster, more accurate

In their new work the researchers are reconstructing three-dimensional, relatively high-quality bowel images on the basis of a rather poor original. Their combined efforts are enabling them to reconstruct the geometry of an entire gut.

“The 3D model can be used to adjust the lighting in an original image so that darker areas become better illuminated and more detail can be seen,” Floor said.

“We all have a colon, but the anatomy and geometry of your bowel is probably quite different from your neighbour’s. New 3D models can be adapted to each patient. They allow specialists to examine the digestive tract from different angles on a PC and enable them to plan better and determine exactly where an intervention should take place. Doctors can also practise ahead of particularly difficult procedures, helping reduce the risk of something going wrong,” Floor says.

One step further along the path

The big question is whether or not these new methods could mean a breakthrough in the fight against bowel cancer.

“It’s too early to draw any definite conclusions. First, we need to refine what we have developed and do more research. It could take quite a bit of time to achieve results that are statistically valid.”

Top of the cancer statistics

Figures from the Cancer Registry of Norway (in Norwegian) show that Norway tops global statistics in respect of prevalence of and deaths from bowel cancer. It is the second most widespread form of cancer in the country, with 4,912 diagnoses of colorectal cancer in 2023.

Researchers initially had very high expectations about wireless capsule endoscopy (WCE). It was thought the small capsules could replace traditional colonoscopy and gastroscopy, which some people find so painful that they refuse to undergo them.

Camera in a capsule

Capsule cameras, by contrast, can investigate all the nooks and crannies in the intestines, almost imperceptibly and without pain, while the patient goes about their daily life. As soon as it is swallowed, the capsule with the camera commences its laborious journey through the digestive tract. This journey can take many hours, and along the way it transmits tens of thousands of images. As good as it is, this innovation does not work equally well for the whole digestive tract. The capsule camera is the best option for screening the small intestines. As far as the colon is concerned, there is still a way to go before it can replace a colonoscopy.

This is in part because it takes a lot of time to review and analyse such large volumes of data, which means there is a distinct possibility that signs of disease might be missed. The capsule follows the movements of the intestine, progressing in fits and starts. But sometimes the camera suddenly zooms past a polyp without “seeing” it.

Missing important details

It is therefore possible that only one image might be taken of a critical site with signs of disease. Visibility inside the bowel is also very varied, often with little light and all kinds of debris. The images also have poor resolution, and there are no guarantees that what could possibly be the most important image will be sharp and clear.

Herein lies the importance of being able to reconstruct a three-dimensional model based on only one picture, as Floor and his colleagues have now done. The new 3D models will be able to help make gastrointestinal examinations more accurate and less demanding – and ultimately lead to better treatment.

Can be used in many areas

Floor and his colleagues have made several recommendations about how the design of the technology could be improved to obtain even higher quality images. They also believe that the technique could easily be transferred to other areas where it would be helpful to generate three-dimensional shapes and models from images.

The researchers wrote that this could include cultural heritage, robotics, and medical diagnostics, to name but a few.

The effort is part of work package 2 in the Research Council of Norway’s project entitled “Improved Pathology Detection in Wireless Capsule Endoscopy Images through Artificial Intelligence and 3D Reconstruction (CAPSULE).

Floor praises the collaboration with Innlandet Hospital Trust, Gjøvik, and says it has been crucial for the progress of the project. Their specialists have played a crucial role, in both the choice of methodology and evaluating the models along the way.

“Without the feedback of these doctors, we would largely be fumbling in the dark,” he said.