AI for better photography
One of the most important reasons that made cameras more user-friendly was the ability to focus automatically without requiring the photographer to tilt the focus ring back and forth to get sharp image. The “magic” trick was the contrast within the frame. Traditional contrast-based autofocus systems simply had to scan the entire focus range (i.e. from the minimum focus distance to infinity) and luck the focus on the area with the highest level of contrast. This process was relatively slow, taking anywhere between 500 ~ 2000 milliseconds. Clearly, this left much room for improvement. Another problem was focusing on moving objects. The slow autofocus meant that the shot could be missed entirely while the camera “hunts” for focus.
Phase Detection Autofocus
Phase detection is a technology that utilizes special sensor to detect focus instead of relying completely on the contrast within the frame. The traditional phase detection in professional DSLRs (i.e. cameras with mirror to direct light to the optical view finder) allows some of the light to bypass the mirror (the mirror is about 5% transparent) to hit the phase detection sensor. The advantage is much better autofocusing at the expense of some light not being used for the actual taking of the photo.
Mirrorless cameras can’t have this luxury since they don’t incorporate mirrors. Instead, special photocells are added to the image sensor, which are used for autofocusing.
Phase detection has greatly improved autofocusing by “sacrificing” a small amount of light. The biggest challenge (or disadvantage if you like) is when focusing in low-light conditions. The amount of available light for focusing may be inadequate, forcing the camera to fallback to contrast-based autofocus.
Depth from Defocus
Depth from Defocus, or DfD for short is a technique that relies on determining the focus area from the out-of-focus data. The Camera takes multiple out-of-focus pictures and applies complex algorithm to work out the best focus point. The theory behind DfD emerged in the late 80’s, however, it required massive amount of computation, making it prohibitively expensive.
In 2012, Panasonic released the Lumix GH4, which was the first mirrorless camera to incorporate DfD technology. What is particularly interesting about Panasonic’s implementation is the inclusion of an AI predictive algorithm that predicts the point of focus based on the distance and velocity of the object of interest, promising extremely fast and accurate focusing. Panasonic claims that their DfD can focus within 70 milliseconds only.
Additional advantage is that by eliminating the dedicated focusing sensors, no light is sacrificed. This is especially more critical for the small Micro Four Thirds sensor. This also allows the camera to autofocus in a very dim light down to -4 EV (Exposure Value) because all the light is used for focusing.
What really made DfD possible was the multi-core, ultra-fast and power-efficient processing engine. In fact, the engine has to perform a massive amount of calculations to reliably detect the focus area in a very short amount of time before falling back to pure contrast-based autofocusing if DfD failed.
However, Panasonic’s latest Venus engine is still not capable enough to reliably focus while taking video. With a vast stream of data needed to capture video at 4K resolution, the engine falls short and the camera is forced to hunt for focus. Annoyingly, this “hunting” is visible and can ruin the shot.
By incorporating AI into photography, a new chapter of possibilities has opened. The main obstacle today is the amount of processing power a camera can have. However, if Moore’s law holds out, Depth from Defocus technology is very likely to prevail in all applications, including video and DSLRs may as well disappear. After all, why carry a large and heavy camera around when a smaller and light one can do the job, better and smarter?
Note: This article first appeared on CognitionX blog