Surgery has evolved tremendously over the past few decades. One particularly interesting development has been the emergence of microsurgery. This development is especially meaningful due to the use of surgical microscopes to magnify the operating field and allow for higher precision. Combined with new types of cameras, they truly ‘make the invisible visible.’
This approach is bound to revolutionize medical treatment. Imagine surgeons being able to make decisions based on real-time information about tissues’ chemical composition at a molecular level, ‘seeing’ the oxygenation of blood flowing through arteries and vessels, or distinguishing in vivo between healthy and anomalous tissue, such as tumor cells.
It is a glimpse at a future that is much closer than one might think. One important driver is the development of spectral imaging technology directly compatible with current medical practice.
Download our white paper on how spectral imaging can advance surgical vision
When two worlds meet, magic happens
Truth be told, the idea of using hyperspectral cameras in surgical practice is not new. Many studies on this topic have already been conducted, but several issues have prevented the technology from being integrated into hospitals’ everyday workflows.
Hyperspectral imaging (HSI) solutions’ form factor is one important hurdle. They are typically quite bulky and thus incompatible with an operating theatre’s already-crammed sterile field. Another limitation has been the lack of video-rate hyperspectral cameras that can acquire real-time data, which is essential to capturing the motion and dynamics in a surgical scene. Enter imec’s hyperspectral sensors.
Contrary to conventional methods of building a hyperspectral camera, imec’s approach does not use precision optics between the lens and the image sensor to select and diffract the light. Instead, it leverages thin-film spectral filters deposited directly on top of image sensors. This translates into robust yet compact devices, very similar to standard machine vision chips and cameras. And since they build on the same wafer-level CMOS process used to create microchips, they are mass-manufacturable, cost-efficient, and highly customizable to specific customer and application needs.
Figure-1: A variety of imec hyperspectral sensors, with spectral filters deposited directly on top of image sensors, in a mosaic or striped pattern. Source: imec
One technology, two flavors
Imec has an extensive track record in the domain of HSI research and prototyping – with a portfolio that comes in two flavors, depending on spectral filters’ patterns.
A striped pattern on top of each row of pixels enables high-resolution hyperspectral imaging, comparable to classic line-scanning push-broom cameras – yet more compact, faster, and easier to use. They make up imec’s ‘SNAPSCAN’ series of hyperspectral sensors.
A mosaic pattern on top of a group of 3x3, 4x4, or 5x5 pixels comes with a somewhat lower resolution but enables video-rate spectral imaging for real-time data acquisition – even in motion. Dubbed ‘SNAPSHOT’, the cameras come in several flavors. Each variant covers a specific spectral resolution and range – from the visual (VIS), over the near-infrared (NIR), to the short-wave infrared (SWIR) ranges.
Case study: using HSI technology for the in vivo identification of low-grade gliomas
Low-grade gliomas are a diverse group of (slow-growing) brain tumors that often arise in young, otherwise healthy patients. While typically benign in origin, studies have shown that low-grade gliomas can expand at a rate of four to five millimeters annually and come with the risk of malignant transformation. The tumor’s early surgical resection is thus a much-favored treatment option – although in vivo detection of low-grade gliomas and retrieving their exact demarcations is extremely hard, even with the aid of surgical microscopes.
In other words, giving surgeons the proper tools to detect these tumors in vivo (i.e., in living tissue) would make for an important breakthrough. Hyperspectral imaging technology shows great potential to do just that – as demonstrated in a recent research project featuring imec’s snapscan VNIR 150 hyperspectral camera.
Thanks to its small form factor (10 x 7 x 6.5 cm, and weighing 645 g), and its compatibility with standard C-mount optics, imec’s SNAPSCAN can easily be mounted on a standard surgical microscope. It makes for a compact set-up that can be incorporated smoothly into hospitals’ stringent clinical workflows, contrary to the bulky systems used in previous studies.
Figure-2: Thanks to its small form factor, and its compatibility with standard C-mount optics, imec’s SNAPSCAN can easily be mounted on a standard surgical microscope. Source: imec
What’s more, the set-up proves to generate accurate clinical data that can be fed into a deep-learning neural network to convert the data into actable knowledge, allowing surgeons to discriminate between healthy and anomalous tissue. As such, it is an important first step to accommodating the in vivo detection of intrinsic brain tumors – such as low-grade gliomas.
While intraoperative use of the set-up is premature, the approach has so far been validated using a clinical dataset of six patients at Belgium’s Leuven University Hospital. Going forward, imec aims to further this project by integrating its SNAPSHOT technology, which accommodates video-rate spectral imaging. This would allow researchers to explore the real-time detection of low-grade gliomas in surgical practice.
The next challenge: making it easier to experiment with hyperspectral imaging technology
A growing number of users are investigating how hyperspectral imaging technology can enhance their product and/or service offerings – or how it can be used in (surgical) practice. For some, it is clear from the start which spectral range best suits their needs. For others, however, it is a journey that involves testing several camera options. While these camera options are available, fusing their data into one dataset from which learnings and conclusions can be drawn is a proven challenge.
That is why imec has recently expanded its snapshot portfolio with the imec multishot VNIR+RGB – a multi-sensor system that covers the VIS to NIR spectral range, complemented with a high-resolution RGB sensor. It is specifically aimed at companies and research groups engaging in HSI application development.
Three sensors in one housing for unmatched application research
A world’s first, imec’s new hyperspectral camera comes with three sensors, integrated into a single housing, and equipped with a standard F-mount lens. It will allow partners to flexibly assess the pros and cons of different spectral resolutions and ranges without needing to invest in a myriad of devices or duplicate experiments.
Figure-3: The imec multishot VNIR+RGB – the world’s first VIS to NIR spectral camera system, complemented with hi-res RGB imaging, for data acquisition at video rate. Source: imec
Inside the camera, the light is directed to three channels. Two of them are equipped with an imec off-the-shelf two-megapixel sensor – together covering the VIS and NIR spectral ranges. A third channel sports a high-resolution RGB sensor. As such, the device covers 30 bands in the 460 to 870nm range, complemented by a true-color, five-megapixel image – all working in sync at video rate speed for real-time data acquisition (even of dynamic scenes).
From assisted (micro)surgery, and machine vision to quality inspection
Similar to the other flavors in imec’s snapshot offering, the new camera particularly lends itself to supporting use cases that involve an uncontrollably dynamic scene. Examples include assisted (micro) surgery, environmental monitoring, anomaly detection, automotive vision, precision agriculture, and crop inspection – just to name a few.
So far, the underlying complexity has prevented the industry from building a fully functional multi-sensor hyperspectral camera. So that is where imec comes in, exploring and prototyping the required components and helping the industry overcome all related hurdles. Thanks to the system’s flexibility and the software that comes with it, experimenting with hyperspectral imaging technology has never been easier. It allows partners that are still in doubt about which sensor best suits their needs to focus on their core business, i.e., application discovery and development.
As a next step, imec plans to share its learnings and a complete development kit with camera builders and systems companies to support the system’s further commercialization.
This article was previously featured in Imaging and Machine Vision Europe.
