\text{Stereo Vision and Depth Estimation}
\textbf{Naresh Kumar Devulapally}
\text{CSE 4/573: Computer Vision and Image Processing}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Lectures 6, 7, 8: June 12, 17, 24 - 2025}

Slides adapted from Dr. Chen Wang CSE 573 lecture slides

\text{Stereo Vision}
  • Recap of Perspective Projection
  • Introduction to Stereo Vision
  • Example Projects for Depth Estimation

\( \text{Agenda of this Lecture:}\)

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Recap - Perspective Projection}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Recap - Perspective Projection}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Recap - Perspective Projection}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Recap - Perspective Projection}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Why Stereo Vision?}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Stereo Vision}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Stereo Vision}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Stereo Vision - Why Multiple Views?}

Structure and depth are inherently ambiguous from single views.

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Stereo Vision - Why Multiple Views?}

Structure and depth are inherently ambiguous from single views.

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation cues}
  • Shading
  • Focus/Defocus
  • Texture
  • Perspective
  • Motion
  • Occlusion
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Shading}

Prados & Faugeras, 2006

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Focus/Defocus}
  • Same point of view, different camera parameters
  • 3D shape / depth estimation

H. Jin & P. Favaro, 2002

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Texture}

A.M. Loh

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Perspective Projection}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Motion}

L. Zhang

Keunhong Park, Nerifies, 2021

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Stereo photography \& Stereo Viewers}

http://www.johnsonshawmuseum.org

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Stereo Vision}

We will need to consider:

  • Info on camera pose ("Calibration")
  • Image point correspondences (feature detection/matching)
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Stereo Vision}

Two cameras, simultaneous views

Single camera moving in a static scene

Structure from Motion:

  • Shape from "motion" between two images.
  • Infer 3D shape of scene from two (multiple) images from different viewponts.
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Stereo Vision}

Stereo Correspondence:

Given a point in one of the images, where could its corresponding points be in the other images?

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Stereo Vision}

Motion (Camera Calibration):

Given a set of corresponding points in two or more images, compute the camera parameters.

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Stereo Vision}

Structure (Depth in the Scene):

Given projections of the same 3D point in two or more images, compute the 3D coordinates of that point.

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Stereo Vision}

Human eyes fixate on point in space - rotate so that corresponding images form in centers of fovea.

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - using Disparity}

Disparity occurs when eyes fixate on one object; that appears at different visual angles.

d = r - l = D - F

We will be using Disparity for depth estimation.

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Camera Parameters (Recap)}

We will assume for now that these parameters are given and fixed.

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Simple Stereo Vision}

Assuming parallel optical axes, known camera parameters (i.e., calibrated cameras).

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Simple Stereo Vision}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - using Disparity}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - using Disparity}

General

Case

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - using Disparity}

General

Case

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - using Disparity}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - using Disparity}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Epipolar Geometry}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Epipolar Geometry}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Epipolar Geometry}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Rectification}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Rectification}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Rectification}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Rectification}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Rectification}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Rectification}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Rectification}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Rectification}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Rectification}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Cross Product - Recap}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Essential and Fundamental Matrices }
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Essential and Fundamental Matrices }
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Essential and Fundamental Matrices }

\( 2T + 3R\)

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Essential and Fundamental Matrices }
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Essential and Fundamental Matrices }
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Essential and Fundamental Matrices }
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - 8 point algorithm}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - 8 point algorithm}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - 8 point algorithm}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Correspondence search}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Correspondence search}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Correspondence search}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Correspondence search}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Correspondence search}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Correspondence search}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Correspondence search}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Correspondence search}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Correspondence search}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Correspondence search}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Effect of Window size}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Effect of Window size}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Occlusion Challenge}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Ordering Challenge}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Results with Window Search}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Results with Window Search}

Use Live Code in the course website to play around with the implementation of depth estimation using window search.

Depth map \( \rightarrow \)

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Results with Window Search}

Use Live Code in the course website to play around with the implementation of depth estimation using window search.

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Results with Window Search}

Use Live Code in the course website to play around with the implementation of depth estimation using window search.

\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Better Solutions}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Better Solutions}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Epipolar Geometry - Energy Minimization}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Better Solutions}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Data-driven solutions}
\text{June 12, 2025}
\text{Naresh Kumar Devulapally}
\text{CSE 4/573: CVIP, Summer 2025}
\text{June 12, 17, 24 - 2025}
\text{Depth Estimation - Data-driven solutions - Interesting Projects}