Generative Adversarial Networks (GANs) have recently gained notoriety for their ability to produce “deep fakes”: images, video, and audio that can mimic reality. For example, fake social media accounts have been created with realistic profile pictures created with GANs.
The ability of artificial intelligence (AI) models to generate realistic images can be used in more productive ways, such as in their use to train AI routines for image segmentation. They can also be used to “sharpen” actual images, enhancing their detail beyond the optical resolution of the imaging system. Now, researchers from Harvard University (Cambridge, MA) and the University of Massachusetts-Dartmouth (North Dartmouth, MA) have demonstrated a GAN-based approach that enables low-cost optics to produce high resolution cellular images.
What your computer can do for you
Microscope images are essential parts of many front-line diagnostic procedures, but key information can be hidden in fine detail. Higher resolution microscopes and advanced imaging techniques can reveal these details, but at a significant cost, both in the instrument itself and in the physical infrastructure and training of personnel to support the detail. image acquisition. Ideally, an inexpensive optical instrument would produce the finely detailed images needed for clinical diagnostics. Professor Y. Shrike Zhang, from Harvard Medical School, recognized the potential value of “achieving higher resolutions approaching those provided by some high-end conventional microscopy, but without the high costs”. He turned to Daniel Shao and his team at the University of Massachusetts, looking to leverage their expertise in AI image processing.
Shao’s team rose to the challenge with a GAN. The GAN approach opposes two competing models. One, the discriminator, is trained with a set of real images. After training, when the discriminator is presented with an image file, it assigns it a sort of “realism score”, a number between 1 and 0 indicating whether it considers an image to be, respectively, real or fake. . The second model is the generator. As its name suggests, a generator creates something; in this case, the “something” is an image. This image is presented to the discriminator, which gives the generator a realism score. The generator trains, improving this score. The drive cycles alternate, with the static generator during the discriminator trains, and the static discriminator during the generator drive. The result is a generator capable of producing “realistic” images.
Machine learning approaches typically require very large databases to train a given model. To compensate for this requirement, the researchers started with an existing model, trained on standard non-cellular images. This existing model quickly and smoothly retrieves high-quality images, but its lack of training on biomedical images means it may not recover cell-specific textures or patterns. They therefore trained an analogue of this existing model on their own sets of regular microscopy images. They then combined the preexisting model and their newly trained model to create a hybrid with both smooth and fast performance and the ability to reconstruct features unique to cell-based biomedical images.
Low cost, high resolution?
The team had previously developed a stripped-down mini-microscope, built from a conventional inexpensive webcam. Spacers between the lens and the CMOS sensor vary the system’s magnification in five steps from 2X to 40X. The cost of the material is less than ten dollars. They also used an “ordinary” optical microscope, a Zeiss Axio Observer D1. With both microscopes, they acquired images of various cell types, including, for example, A549 human lung carcinoma cells and HepG2 human hepatocellular carcinoma cells. They downsampled the images and then used their hybrid GAN model to reconstruct high-resolution versions (see figure).
They used two metrics to compare the performance of their new model with other super-resolution AI methods: peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). “PSNR measures the pixel-level difference between super-resolution images and high-resolution ground-truth images,” Shao explained, “while SSIM indicates how well the system produces ‘vivid, high resolution’ images. fidelity and visually pleasing”. which are preferred by human visual systems. previous tip tested. This was found to be true for all cell types and for images acquired with both the regular microscope and the mini-microscope.
Although the team has not yet attempted to improve the resolution of the acquired images, their results are promising enough that Zhang is excited about “the possibility of obtaining high resolutions using a mini microscope very cheap ; critical in many resource constrained settings where they simply cannot afford high end scope. With further refinement of the algorithm and other hands-on demonstrations, he said, “The ultimate step could be an inexpensive and powerful system of optimal hardware/software.”