New Sensors and Machine Vision Applications Drive Advances in Optics

Apr 27, 2023

All Images Source: A3

Before any camera can capture an image of an object, the lens must collect scattered light from the object and distribute it properly over the sensor's active area. That's one reason why camera sensors and new machine vision applications tend to drive parallel advances in optics, and optics suppliers must continually evolve as machine vision technologies progress.

"We consistently see machine vision applications going to higher resolutions," says Nick Sischka, recently promoted to director of imaging product development at Edmund Optics. "We've stopped noticing pixels getting significantly smaller and instead see sensor sizes growing."

As companies like Gpixel and Sony continue to release ever-larger sensors (e.g., 152 MP), the challenge for optics suppliers is to keep pace. These lenses will be quite large — larger than full-frame lenses typically used in photography — and until new ones are developed, a photographic medium-format lens may be the best option.

"Lenses for the very large 60.6-mm diagonal sensors that offer greater than 150 MP must be developed," says Mark Peterson, cofounder and vice president, advanced technology, Theia Technologies. "Unfortunately, the cost for development, the large size of the lens, and the initial low volume will keep these lenses at a premium price."

Jason Baechler, president of MORITEX North America Inc., concurs. "When designing optics for increasingly large sensors, several challenges exist, especially if the pixel sizes/pitches are quite small," he says. "The two main challenges are controlling the size and cost of the lenses/optics to align with the cameras using such new sensors. Beyond just the challenge to address all the camera sensors out there, those two factors increase the necessity of making tradeoffs in optical and mechanical designs."

For bi-telecentric lenses, MORITEX designs the front/objective lenses to match the maximum field of view with a targeted resolution. That front portion of the lens can then be suitable for a wide range of image sensor formats, which minimizes component variation within its portfolio and cost optimizes the most expensive telecentric lens components. The back lens portion (image side), however, varies based on image format and the result is that the length is not always optimized.

For that reason, Baechler notes, "we still offer object-side telecentric lenses of different formats for applications that have tighter space requirements." However, those lenses (MORITEX MMLs) can only be used with a specific sensor format or smaller, assuming the mount size is the same. For factory automation and other non-telecentric fixed focal-length lenses (where some large or high-power objective lens doesn't drive the cost), the tradeoffs grow with the versatility and cost competitiveness of the products.

By minimizing the number of elements in a lens and simplifying the opto-mechanical systems for working distance (WD), aperture, and focus adjustment, a lens can be designed for a limited working distance range and/or aperture size. Another way to approach this is to design products to cover some range of sensor formats and offer mount adapters to match up with different cameras.

"This approach has the advantage of improved resolution as the sensor image format gets smaller versus the maximum image format of the lens," Baechler explains. "As a result, a lens designed for 62-mm image diagonal (or line) and 5 µm pixels could match a sensor with 3 µm pixels and a 43.5-mm diagonal."

In response to the growing release of larger sensors, Navitar Inc. has also developed optics optimized for use with the larger camera formats. One new product is suitable for high-end industrial applications such as semiconductor wafer inspection, FPD inspection, and MEMS, as well as life science and biomedical applications such as multi-well experiments and cell imaging.

Jeremy Goldstein, owner and CEO of Navitar, notes that new optics to take advantage of the larger format cameras "enable Navitar to provide their customers larger fields of view imaging so the customer can see more of the object at one time. This increases the processing throughput and speed of data collection, which is extremely important to all industries, both industrial and medical."

In biological research and instruments especially, capturing a larger surface area of a sample decreases the amount of time to collect meaningful data and the possibility of photo bleaching of labeled cells. Time previously spent moving the optics, the stage, or waiting for software to stitch multiple images together is greatly reduced when using the system, according to Goldstein.

Industry reports expect non-visible imaging, including ultraviolet, short-wave infrared, IR, and hyperspectral and multispectral, to grow by more than 30 percent over the next five years. As sensor/camera options have rapidly expanded in recent years, costs have come down, spurring an explosion of new applications — which, in-turn, has expanded demand for a variety of lenses required to address these new applications.

Peterson notes that Theia has developed SWIR-only lenses in the past before the new hyperspectral sensors were available. "Now we are evaluating new lens types that can work with the advanced vis-SWIR sensors that are responsive to wavelengths from 400 nm to 1700nm, to provide an ultra-wide image and remove barrel distortion without software."

He adds: "The ‘killer app' for this vis-SWIR wavelength band hasn't been identified yet, but the decreasing cost of hyperspectral sensors will necessitate the availability of lenses that complement the wide wavelength band."

In hyperspectral imaging, Sony has introduced broadband imaging sensors, but until there is an actual hyperspectral coating over it, most applications will use a narrow wavelength band across that sensor such as SWIR, according to Sischka.

"It often makes more sense to have a multi-camera solution with different sensors focusing on different wavebands rather than having one very expensive camera that can image across the whole broad spectrum," Sischka says. "An exception to this would be an application where minimizing weight is a priority, like aerial imaging from a drone."

Other advancements in optics that are significant to the machine vision market include zoom lenses, new optical materials, and improved manufacturing methods. For example, zoom lenses — which were previously thought undesirable due to concerns about vibration effects on the motorized mechanics — are being adopted in greater numbers, according to Peterson.

The increased adoption is driven by applications where the subject is not just a uniformly shaped product on an assembly line, but an object at unknown and variable distance. One example is mobile applications, where the object or the imager or both may be moving. Meanwhile, fixed robots and roving autonomous robots require the flexibility of a zoom lens to navigate and identify objects in a dynamic environment.

Peterson also says that motorized lenses with adjustable zoom and focus provide greater versatility and convenience, allowing adjustment to changing conditions, remote setup, and operation. Combined with high resolution and NIR correction, as well as AI and machine learning, this type of lens can be a powerful tool to enable higher level identification and recognition tasks, such as defect detection, optical character recognition (OCR) and automatic number plate recognition.

One such application is intelligent traffic systems (ITS), where the flexibility to optimize the field of view and focus distance after initial installation and without the need for stopping traffic is enabled by motorized zoom and focus. Image optimization is critical to getting the best performance to identify targets using OCR, AI, and machine learning.

"There is a common misunderstanding that an imaging system is only useful if it is pixel-limited and operating at the Nyquist frequency," Sischka says. "Imaging lenses can actually be very useful beyond the Nyquist frequency, meaning that pixel size is smaller than the focused spot size of the lens. For example, smearing multiple pixels across a feature is useful for intelligent traffic solutions because it allows them to notice small differences between objects that look relatively similar to each other."

As mentioned earlier, the incorporation of metamaterials will be very exciting but is not quite here yet, says Sischka. Materials science will be key to making further developments in serving larger sensors and broader wavelength ranges. Computational imaging and the ability to use diffractive elements have also led to recent advances.

Baechler agrees. "Material options have helped keep costs down while performance increases for lenses designed for IR-SWIR sensors," he says. "Liquid lenses allow for novel solutions and also increased versatility of existing base optical designs."

When Samsung, a specialist in LCD display manufacturing, was shifting its focus from making LCD panels to manufacturing QD-OLED (quantum-dot display organic light-emitting diode) panels, the company required new automated optical inspection (AOI) machines to perform the inspection. The AOI solution required a modular microscope system to perform the inspection. Samsung evaluated various objective lenses from different manufacturers but chose the Navitar 4X HR objective because it not only provided a larger field of view compared to a competitive 5X objective, but it also offered a higher NA.

Goldstein also sees the continued expansion of using "advanced assembly techniques that ensure the centering of each optical element of the lens assembly instead of relying on just simply dropping lens elements in barrels," adding that Navitar is at the forefront of this technology. "By combining the advanced assembly techniques with the expanded use of automated lens-sensor integration equipment, we improve lens performance and greatly improve the imaging performance of the optical system."

As lens makers try to keep pace with the release of new camera sensors and expanding machine vision applications where advanced solutions are now feasible, new opportunities continually emerge — including in the field of robotics and autonomous machines. Increases in dynamic range allow autonomous vehicles to be much more robust and robots are now able to multitask. In retail applications, they can simultaneously recognize product outages, safety hazards, and label discrepancies. Like most robots, they automate tedious, repetitive tasks, freeing human talent for higher value tasks requiring more nuance and discretion.

In addition to new applications, existing applications also benefit from the greater throughput achieved by larger format optics that take advantage of the larger format sensors. Providing more data to improve image quality and image processing speed should improve AI efficiency and performance, allowing further advancements in automation. Precise data and metadata on optics and sensors is critical to enabling concepts such as digital twin and Industry 4.0.

John Lewis is market development manager of Cognex Corp. (Natick, MA). For more information, call (508) 650-3000 or visit www.cognex.com.