LiDAR + Camera Data Fusion on KITTI

KITTI Data

KITTI offers many datasets; here we pick the raw_data (raw data) for fusion. Download link: http://www.cvlibs.net/datasets/kitti/raw_data.php

The KITTI dataset’s data-collection platform is equipped with 2 grayscale cameras, 2 color cameras, one Velodyne 64-beam 3D LiDAR, 4 optical lenses, and 1 GPS navigation system. The specific sensor parameters are as follows:

  • 2 × PointGray Flea2 grayscale cameras (FL2-14S3M-C), 1.4 Megapixels, 1/2” Sony ICX267 CCD, global shutter
  • 2 × PointGray Flea2 color cameras (FL2-14S3C-C), 1.4 Megapixels, 1/2” Sony ICX267 CCD, global shutter
  • 4 × Edmund Optics lenses, 4mm, opening angle ∼ 90◦, vertical opening angle of region of interest (ROI) ∼ 35◦
  • 1 × Velodyne HDL-64E rotating 3D laser scanner, 10 Hz, 64 beams, 0.09 degree angular resolution, 2 cm distance accuracy, collecting ∼ 1.3 million points/second, field of view: 360◦ horizontal, 26.8◦ vertical, range: 120 m
  • 1 × OXTS RT3003 inertial and GPS navigation system, 6 axis, 100 Hz, L1/L2 RTK, resolution: 0.02m / 0.1◦
Sensors on the KITTI data-collection platform
Sensors mounted on the KITTI data-collection platform: cameras, LiDAR, optical lenses, and GPS navigation system

The mounting positions of the sensors are shown below:

Diagram of sensor mounting positions
Diagram of the mounting positions of the various sensors on the KITTI data-collection vehicle

For the raw_data dataset we use, each data package (taking 2011_09_26_drive_0001, synced + rectified data as an example), after extraction, mainly contains the following parts:

  • Calibration files: calib_cam_to_cam.txt, calib_imu_to_velo.txt, calib_velo_to_cam.txt
  • Camera files: four folders image_00 through image_03, corresponding to the 4 cameras; the commonly used camera is 02
  • Point cloud files: the point files scanned by the LiDAR, in the velodyne_points folder, which includes multiple bin files and a timestamp file

Code Implementation

The KITTI dataset provides an official toolkit to make it convenient to work with the dataset. For the raw_data, it provides a dozen or so functions to ease our operations, and it offers a data fusion demo as follows:

function run_demoVelodyne (base_dir,calib_dir)

% clear and close everything
close all; dbstop error; clc;
disp('======= KITTI DevKit Demo =======');

% set depending on the function's input arguments
if nargin<1
  base_dir  = './data/2011_09_26/2011_09_26_drive_0005_sync';
end
if nargin<2
  calib_dir = './data/2011_09_26';
end
cam       = 2; % 0-based index
frame     = 137; % 0-based index

% load the calibration file; fullfile combines the path and the file
% calib is the struct matrix of camera parameters
calib = loadCalibrationCamToCam(fullfile(calib_dir,'calib_cam_to_cam.txt'));
% Tr_velo_to_cam is a parameter matrix
Tr_velo_to_cam = loadCalibrationRigid(fullfile(calib_dir,'calib_velo_to_cam.txt'));

% compute the projection matrix from the LiDAR's 3D data to the camera image
R_cam_to_rect = eye(4);
R_cam_to_rect(1:3,1:3) = calib.R_rect{1};
P_velo_to_img = calib.P_rect{cam+1}*R_cam_to_rect*Tr_velo_to_cam;

% load and display image
img = imread(sprintf('%s/image_%02d/data/%010d.png',base_dir,cam,frame));
fig = figure('Position',[20 100 size(img,2) size(img,1)]);
axes('Position',[0 0 1 1]);
imshow(img); hold on;

% load velodyne points
fid=fopen(sprintf('%s/velodyne_points/data/%010d.bin',base_dir,frame),'rb');% first open in binary mode
velo = fread(fid,[inf 4],'single');% read the file as single format, with dimensions (n,4)
velo = velo(1:5:end,:);% keep only one point out of every five, for easier display
fclose(fid);

% remove all points behind image plane (approximation
idx = velo(:,1)<5;	% find the indices of LiDAR points whose x coordinate is less than 5
velo(idx,:) = [];	% remove these points

% project to image plane (exclude luminance)
velo_img = project(velo(:,1:3),P_velo_to_img);

% plot points, in color

cols = jet;
for i=1:size(velo_img,1)
  col_idx = round(64*5/velo(i,1));
  plot(velo_img(i,1),velo_img(i,2),'o','LineWidth',4,'MarkerSize',1,'Color',cols(col_idx,:));
end

% plot points, grayscale

% cols = 1-[0:1/63:1]'*ones(1,3);
% for i=1:size(velo_img,1)
%   col_idx = round(64*5/velo(i,1));
%   plot(velo_img(i,1),velo_img(i,2),'o','LineWidth',4,'MarkerSize',1,'Color',cols(col_idx,:));
% end

The final result looks like this:

Result of fusing LiDAR and camera data
The fusion result of LiDAR point cloud projected onto the camera image; color indicates distance

The complete toolkit, data, and code can be downloaded from my GitHub: https://github.com/Lannyy/DataFusion

  • Technology