Menelaos - Premiere H2020 Results

Monodepth2 is a self-learning neural network that generates dense depth maps by using the Structure from Motion (SfM) approach. Monodepth2 was initially conceived for autonomous driving. We have adapted the network to the drone flight by taking advantage of the flight parameters measured by various onboard instruments.

Figure 1. Monodepth2 uses a standard, fully convolutional encoder-decoder network based on the general U-Net architecture. The encoder part of the network is ResNet18, which contains 11M parameters that are pretrained on ImageNet. The decoder part has sigmoid activation functions at the output and ELU nonlinearities elsewhere. 

The network is trained by self-learning. More precisely, two chronologically adjacent frames I1 and I2 of the video sequence are considered. As in classical supervised learning, I1 is feedforwarded through the depth network that outputs an estimate D1 of its depth map. Based on D1 and with the help of camera intrinsic matrix K and camera pose change T1,2 between the two frames, the pixels of I1 are projected onto I2. Once projected, their values are recalculated by bilinear interpolation of the neighbouring pixels in I2. The difference between the actual values and the recalculated ones is the "engine" of the learning process.

Figure 2. Three examples of frames and the corresponding depth maps obtained with the original Monodepth2 and the modified network, adapted for drone flight. The ground truth is on the last column.

DATASETS

The following table summarises a series of datasets useful for our application. For each dataset, it is provided a short description the website address from where they can be downloaded. The items 1 – 10 are videos or high resolution images captured by an UAV. The main criterion for selecting them was the availability of ground truth for depth and camera position and orientation. The items 11 – 16 are non-drone general purpose benchmarks.