Plenoptic cameras or light field cameras are a recent type of imaging devices that are starting to regain some popularity. The concept of plenoptic cameras was formalized in the early 20th century, but we are only starting to see commercial available cameras nowadays due to the technological limitations back then. While 2D conventional cameras project a 3D scene into a 2D image, plenoptic cameras work in a different way, since they acquire the plenoptic function, which not only contains information about the intensities of each pixel but also the amount of light that travels in every direction.
The objective of this work is clear and it has to do with depth estimation. Our goal was to develop a fully-automatic algorithm able to estimate the depth of a scene captured with a multi-focus plenoptic camera and, if possible, obtain better results than those presented by the manufacturers. Despite being formalized in an article, Raytrix algorithms are not publicly available, leading to a reverse engineering problem. We achieved our goal, obtaining similar results to the ones presented by Raytrix, which we considered to be ground truth, fact that we cannot confirm.
The main contribution of this work is a novel approach used to achieve a virtual depth estimation, where we can use the redundancy created by the fact that a single 3D point is imaged several times by the main lens. This redundancy allows us to backtrace the light rays in order to estimate the depth. The main contribution is followed by a series of minor contributions that helps to reach the goal of this work. Before we backtrace the light rays, we need to find correspondences of the image points, which we do using the SAD method, but scaled in a way to obtain better results. We developed an iterative method, a RANSAC-like method we called COMSAC (Complete Sample Consensus), to perform the referred backtrace, where we estimate the depth of each point. This iterative method allows us to eliminate bad results (outliers) and, consequently, produce robust results. The final contribution is the random growing of the sparse depth map, where we analyze a pixel without color information and use the intensity of neighboring pixels to fill the referred pixel, transforming the sparse depth map into a complete dense depth map.
Our methods were tested on public datasets availlable from Raytrix, and you can see the results on the Results section. As future work, it is intended to increase the perfomance of our methods, as well the methods produced. Also, it is intended to test our algorithms using our own Raytrix camera and also port them to standard plenoptic cameras, as Lytro.
This Master Thesis started on September 2013 and ended on September 2014, and was developed by João Barata Custódio under the supervision of Prof. Dr. Nuno Gonçalves. For furher information, download the full document here.