【OpenCV】双目相机计算深度图和点云

双目相机计算深度图的基本原理是通过两台相机从不同角度拍摄同一场景，然后利用视差来计算物体的距离。本文的Python实现示例，使用OpenCV库来处理图像和计算深度图。

1、数据集介绍

Mobile stereo datasets由Pan Guanghan、Sun Tiansheng、Toby Weed和Daniel Scharstein在2019-2021年期间创建的，使用了Roger Dai、Kyle Meredith、Tommaso Monaco、Nick Mosier和Daniel Scharstein在2017-2018年期间开发的结构化光采集系统。该系统采用安装在UR5机械臂上的移动设备（苹果iPod touch 6G）；地基真差是使用[5]中描述的结构化照明管道的一个子集来计算的。这些数据集包括11个场景，在许多不同的照明条件和曝光（包括闪光灯和移动设备的“火炬”照明）下，从1-3个不同的观看方向拍摄。

数据集基本描述：

Dataset description
Each dataset consists of 2 views taken under several different illuminations and exposures. The files are organized as follows:
SCENE{1,2,3}/                  -- scene imaged from 1-3 viewing directions
  ambient/                     -- directory of all input views under ambient lighting
    {F,L,T}{0,1,...}/          -- different lighting conditions (F=flash, L=lighting, T=torch)
      im0e{0,1,2,...}.png      -- left view under different exposures
      im1e{0,1,2,...}.png      -- right view under different exposures
  calib.txt                    -- calibration information
  im{0,1}.png                  -- default left and right view (typically ambient/L0/im{0,1}e2.png)
  disp{0,1}.pfm                -- left and right GT disparities
Zip files containing the above files can be downloaded here. "all.zip" contains all 24 scenes (image pair, disparities, calibration file), but not the ambient subdirectories. The latter are available in separate zip files.

Calibration file format
Here is a sample calib.txt file:
cam0=[1758.23 0 953.34; 0 1758.23 552.29; 0 0 1]
cam1=[1758.23 0 953.34; 0 1758.23 552.29; 0 0 1]
doffs=0
baseline=111.53
width=1920
height=1080
ndisp=290
isint=0
vmin=75
vmax=262
Explanation:

cam0,1:        camera matrices for the rectified views, in the form [f 0 cx; 0 f cy; 0 0 1], where
  f:           focal length in pixels
  cx, cy:      principal point

doffs:         x-difference of principal points, doffs = cx1 - cx0 (here always == 0)

baseline:      camera baseline in mm

width, height: image size

ndisp:         a conservative bound on the number of disparity levels;
               the stereo algorithm MAY utilize this bound and search from d = 0 .. ndisp-1

vmin, vmax:    a tight bound on minimum and maximum disparities, used for color visualization;
               the stereo algorithm MAY NOT utilize this information
To convert from the floating-point disparity value d [pixels] in the .pfm file to depth Z [mm] the following equation can be used:
Z = baseline * f / (d + doffs)
Note that the image viewer "sv" and mesh viewer "plyv" provided by our software cvkit can read the calib.txt files and provide this conversion automatically when viewing .pfm disparity maps as 3D meshes.

• 如果使用自己的双目相机，则需要知道相机两个摄像头的内参矩阵cam0,1、基线baseline

2、Python代码

代码使用的双目图像数据是chess2

import cv2
import numpy as np
import time
import matplotlib.pyplot as plt

def load_images(left_path, right_path):
    left_image = cv2.imread(left_path, 0)
    right_image = cv2.imread(right_path, 0)
    return left_image, right_image

#
def compute_disparity_map(left_image, right_image, cam0, cam1, doffs, ndisp, vmin, vmax, block_size=15):
    stereo = cv2.StereoSGBM_create(
        minDisparity=vmin,  # 视差从0开始
        numDisparities=ndisp,  # 视差范围数量，必须是16的倍数
        blockSize=block_size,  # 匹配块的大小，奇数
        P1=8 * 3 * block_size ** 2,  # 平滑惩罚项（第一级）
        P2=32 * 3 * block_size ** 2,  # 平滑惩罚项（第二级），通常是P1的4倍
        disp12MaxDiff=1,  # 左右视差检查的最大允许差异
        uniquenessRatio=10,  # 唯一性比率阈值
        speckleWindowSize=100,  # 斑点滤波器窗口大小
        speckleRange=32  # 斑点滤波器最大容许差异
    )
    disparity_map = stereo.compute(left_image, right_image).astype(np.float32) / 16.0
    return disparity_map

# 转成深度图
def convert_disparity_to_depth(disparity_map, focal_length_px, baseline_mm):
    # 避免除以零的情况
    depth_map = (focal_length_px * baseline_mm) / disparity_map
    depth_map[disparity_map == 0] = 0  # 设置无效区域的深度为0
    return depth_map

# 过滤最远平面
def filter_far_plane(depth_map, max_distance=1500):  
    depth_map[depth_map