Intro
When multiple image sensors take a picture of a specific object, each from a different angle, the object can be reconstructed in three dimensions from those images by using ingenious software. This technology is called ‘multiview imaging’. The simplest form of multiview imaging involves two image sensors. We call this stereovision. Imec has conducted research into multiview imaging for many years, after which the imec.engineering team took the technology out of the lab and beyond the research stage and made it market ready. Imec’s multiview imaging systems offer major advantages, such as speed, precision and low costs. These properties make them attractive for use in industrial applications, such as in robot vision or quality control. Peter Lemmens, General Manager of imec Taiwan, describes the marketing process and explains about the services that imec offers companies.
The possibilities of multiview imaging
Cameras that use stereovision are based on the capabilities of human eyesight. Two image sensors take a picture of a specific object from various positions and angles. Algorithms compare the images and go looking for corresponding pixels. A depth map is then generated using software in which the pixels are displayed according to their mutual distance from the camera. This data is used to display the object in 3D, visualizing it in what is called a ‘point cloud’. The camera performs with greater precision when more than two image sensors are used. This is called ‘multiview imaging’.
Just as our eyesight gives us the power of depth perception, the technology is often used to define the distances to objects in an image. Stereovision – and, by extension, multiview imaging – has a number of specific advantages over other technologies for this application.
For instance – unlike time-of-flight systems – multiview imaging can be applied both indoors and outside. And because conventional, commercial image sensors can be used, multiview imaging is also relatively inexpensive (compared with something such as costly radar technology) – plus it is easy to integrate into existing systems.
From research to product
Multiview imaging – with stereovision as its simplest form – is by no means new technology. It has long been the subject of a great deal of academic research. Also at imec, the technology was at the research stage some ten years ago.
Imec.engineering, which is the team that works on system development, prototyping and production, has taken this technology out of the imec labs and made it ready for market.
The team began by optimizing the technology for robot vision. For instance, multiview imaging can be used as a visual control system for ‘bin-picking’, a process in which a robot has to collect randomly placed objects from a bin. Fast object recognition and the precise determination of distances are vital to be able to do this.
Depth maps and 3D images obtained with imec’s multiview camera.
Fast, accurate and low cost
Many multiview imaging systems use software to combine the different images (stereo-matching) and generate a 3D point cloud. In these systems, the software runs on graphical processing units (GPUs) or microcontrollers. These software-based solutions are extremely simple to integrate into a system, but their processing speed tends to be on the low side. That’s why imec offers a hardware solution in which the stereo-matching software is embedded in an FPGA (field-programmable gate array).
By co-optimizing the algorithms and the physical implementation, the imec.engineering team has succeeded in substantially increasing the speed of the image processing, up to 60 depth images per second in high definition (as many as 120 images per second are possible). And the systems achieve a very high performance.
It means that the contours of the object, which are visualized in real-time by the point cloud, are displayed with very high precision and resolution, without any ‘black holes’. This helps to increase precision in post-processing. The depth is also determined with far greater accuracy than with existing systems (depth accuracy >0.1mm), enabling objects to be identified better. Imec has finally designed its systems with an optimum cost/performance ratio in mind.
Imec’s multiview camera
Active lighting for objects with no texture
In conventional multiview imaging, algorithms search for matching pixels in two (or more) images by ‘looking’ for gradations in contrast and clarity of the pixels. These gradations are defined by the texture of the object. But what about objects that don’t have any texture on their surface, such as plastics or smooth metal objects? Also for these objects without texture imec.engineering develops systems that use ‘active lighting’ (or active multiview). Using a light source, usually one or more lasers, a (random) pattern of points is projected on to a smooth object, giving it a definite texture. By examining the point pattern, the system finds it easier to detect the corresponding pixels, making it very effective for objects with no texture.
Depth map and 3D image of a box of candy obtained using imec’s active multiview camera.
A whole range of applications
With its ultrafast and extremely precise multiview imaging systems, imec is aiming for applications where these characteristics can be used to the full. One such application is robot vision in which multiview cameras are used for determining distances and identifying objects in real-time. In these dynamic environments, speed and precision of depth are crucial factors.
Another application for multiview imaging is quality control in industrial environments. For checking simple objects, an ordinary camera will suffice. But for inspecting and checking objects that have complex shapes, highly precise distance measurements are required. This is something that can be done with multiview imaging.
Multiview imaging systems can also be put to good use for security purposes. For example, depth measurements make it possible to count people or track individuals through a fast-moving crowd of people. And the technology can also be used to monitor the safety of large industrial facilities, such as for checking whether gas is escaping from gas pipes.
Another possible industrial application is the use of automated guided vehicles, such as forklift trucks, which use multiview imaging to navigate their way around manufacturing plants.
From design to total solution
Imec.engineering provides a wide range of services to back up its unique stereovision and active multiview technology. To do so, it relies on an extensive portfolio of international patents (more than 15).
The imec offering is extremely varied, ranging from the design and modeling of optical components (such as lenses, collimators, optical filters, etc.), through the design of vision algorithms for implementation in cost/performance systems, and on to prototyping and preproduction.
Companies can come to imec for total customized system solutions, or for just one or two part aspects of the multiview imaging solution. Whatever the need, the imec.engineering team is able to draw on imec’s many years of experience in areas such as GPUs, FPGAs, processors and the integration of image sensors.
In fact, many companies come calling on imec.engineering because they are unable to find an off-the-shelf solution on the market that meets their needs, and/or because they are looking for a partner capable of resolving their specific technological difficulties.
Software companies also find their way to imec. Companies developing e.g. robot vision are often software-oriented and specialize in algorithms for applications such as pattern recognition. These companies are now increasingly looking to differentiate themselves in terms of hardware, using their own cameras that are better than the devices used by their competitors. Imec is seeing this trend not only among the major software companies, but also in small and medium-sized businesses. All of these enterprises can come to imec for unique camera technology.
The development of multiview imaging systems is a good example of how imec.engineering takes research from its imec labs to maturity and onward to the product level.
Want to know more?
- Visit our webpage about imec.engineering
- Interested? Contact imec’s account manager in your region via the local offices. Companies in Flanders can contact kris.hermus@imec.be.
