computer vision lecture 17 segmentation & grouping ng‘19€¦ · • partial reconstruction...

1

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Computer Vision – Lecture 17

Uncalibrated Reconstruction & SfM

08.07.2019

Bastian Leibe

Visual Computing Institute

RWTH Aachen University

http://www.vision.rwth-aachen.de/

[email protected] Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Course Outline

• Image Processing Basics

• Segmentation & Grouping

• Object Recognition

• Local Features & Matching

• Deep Learning

• 3D Reconstruction

Epipolar Geometry and Stereo Basics

Camera calibration & Triangulation

Uncalibrated Reconstruction & Active Stereo

Structure-from-Motion

2

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Recap: A General Point

• Equations of the form

• How do we solve them? (always!)

Apply SVD

Singular values of A = square roots of the eigenvalues of ATA.

The solution of Ax=0 is the nullspace vector of A.

This corresponds to the smallest singular vector of A.3

B. Leibe

Ax 0

11 11 1

1

T

N

T

NN N NN

d v v

d v v

A UDV U

SVD

Singular values Singular vectors

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Recap: Camera Parameters

• Intrinsic parameters

Principal point coordinates

Focal length

Pixel magnification factors

Skew (non-rectangular pixels)

Radial distortion

• Extrinsic parameters

Rotation 𝐑

Translation 𝐭(both relative to world coordinate system)

• Camera projection matrix

General pinhole camera: 9 DoF

CCD Camera with square pixels: 10 DoF

General camera: 11 DoF 4B. Leibe

0

0

1 1 1

x x x

y y y

m f p x

K m f p y

s s

P K R | t

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Recap: Calibrating a Camera

• Goal

Compute intrinsic and extrinsic parameters

using observed camera data.

• Main idea

Place “calibration object” with known

geometry in the scene

Get correspondences

Solve for mapping from scene to image:

estimate P=PintPext

5B. LeibeSlide credit: Kristen Grauman

? P

Xi

xi

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Recap: Camera Calibration (DLT Algorithm)

• P has 11 degrees of freedom.

• Two linearly independent equations per independent 2D/3D

correspondence.

• Solve with SVD (similar to homography estimation)

Solution corresponds to smallest singular vector.

• 5 ½ correspondences needed for a minimal solution.6

B. Leibe

• Solve with …?

Slide adapted from Svetlana Lazebnik

0pA 0

P

P

P

X0X

XX0

X0X

XX0

3

2

1111

111

T

nn

TT

n

T

nn

T

n

T

TTT

TTT

x

y

x

y

? P

Xi

xi

2

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

• Stack equations and solve with …?

Recap: Triangulation – Linear Algebraic Approach

• Two independent equations each in terms of

three unknown entries of X.

• Stack equations and solve with SVD.

• This approach nicely generalizes to multiple cameras.7

B. Leibe

XPx

XPx

222

111

0XPx

0XPx

22

11

0XP][x

0XP][x

22

11

Slide credit: Svetlana Lazebnik

O1O2

x1x2

X?

R1R2

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Topics of This Lecture

• Revisiting Epipolar Geometry Calibrated case: Essential matrix

Uncalibrated case: Fundamental matrix

Weak calibration

Epipolar Transfer

• Active Stereo Kinect sensor

Structured Light sensing

Laser scanning

• Structure from Motion (SfM) Motivation

Ambiguity

Projective factorization

Bundle adjustment

8B. Leibe

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Recap: Epipolar Geometry – Calibrated Case

9B. Leibe

X

x x’


Essential Matrix

(Longuet-Higgins, 1981)

0)]([ xRtx RtExExT ][with0

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Epipolar Geometry: Calibrated Case

• E x’ is the epipolar line associated with x’ (l = E x’)

• ETx is the epipolar line associated with x (l’ = ETx)

• E e’ = 0 and ETe = 0

• E is singular (rank two)

• E has five degrees of freedom (up to scale)11

B. Leibe

X

x x’


0)]([ xRtx RtExExT ][with0

Why?

Why?

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Epipolar Geometry: Uncalibrated Case

• The calibration matrices K and K’ of the two cameras are unknown

• We can write the epipolar constraint in terms of unknownnormalized coordinates:

14B. Leibe

X

x x’


0ˆˆ xExT xKxxKx ˆ,ˆ

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19


15B. Leibe

X

x x’


Fundamental Matrix(Faugeras and Luong, 1992)

0ˆˆ xExT

xKx

xKx

ˆ

ˆ

1with0 KEKFxFx TT

3

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19


• F x’ is the epipolar line associated with x’ (l = F x’)

• FTx is the epipolar line associated with x (l’ = FTx)• F e’ = 0 and FTe = 0

• F is singular (rank two)

• F has seven degrees of freedom 16B. Leibe

X

x x’


0ˆˆ xExT 1with0 KEKFxFx TT

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Estimating the Fundamental Matrix

• The Fundamental matrix defines the epipolar geometry

between two uncalibrated cameras.

• How can we estimate F from an image pair?

We need correspondences…

17B. Leibe

xi

Xi

x0i

P? P0?

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

The Eight-Point Algorithm

• Taking 8 correspondences:

18B. Leibe

x = (u, v, 1)T, x’ = (u’, v’, 1)T

This minimizes:

2

1

)( i

N

i

T

i xFx


[u0u; u0v; u0; uv0; vv0; v0; u; v; 1]

26666666666664

F11

F12

F13

F21

F22

F23

F31

F32

F33

37777777777775

= 0

Af = 0

Solve using… ?Solve using… SVD!

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Excursion: Properties of SVD

• Frobenius norm

Generalization of the Euclidean norm to matrices

• Partial reconstruction property of SVD

Let i i=1,…,N be the singular values of A.

Let Ap = UpDpVpT be the reconstruction of A when we set

p+1,…, N to zero.

Then Ap = UpDpVpT is the best rank-p approximation of A in the

sense of the Frobenius norm

(i.e. the best least-squares approximation).19

B. Leibe

2

1 1

m n

ijFi j

A a

min( , )

2

1

m n

i

i

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19


• Problem with noisy data

The solution will usually not fulfill the constraint that F only has

rank 2.

There will be no epipoles through which all epipolar lines pass!

• Enforce the rank-2 constraint using SVD

• As we have just seen, this provides the best least-squares

approximation to the rank-2 solution.

20B. Leibe

11 11 13

22

33 31 33

T

T

d v v

F d

d v v

UDV U

SVD

Set d33 to

zero and

reconstruct F

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Problem with the Eight-Point Algorithm

• In practice, this often looks as follows:

21B. Leibe

266666666664

u01u1 u01v1 u01 u1v01 v1v

01 v01 u1 v1 1

u02u2 u02v2 u02 u2v02 v2v

02 v02 u2 v2 1

u03u3 u03v3 u03 u3v03 v3v

03 v03 u3 v3 1

u04u4 u04v4 u04 u4v04 v4v

04 v04 u4 v4 1

u05u5 u05v5 u05 u5v05 v5v

05 v05 u5 v5 1

u06u6 u06v6 u06 u6v06 v6v

06 v06 u6 v6 1

u07u7 u07v7 u07 u7v07 v7v

07 v07 u7 v7 1

u08u8 u08v8 u08 u8v08 v8v

08 v08 u8 v8 1

377777777775

26666666666664

F11

F12

F13

F21

F22

F23

F31

F32

F33

37777777777775

=

266666666664

0

0

0

0

0

0

0

0

377777777775


4

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Problem with the Eight-Point Algorithm

• In practice, this often looks as follows:

Poor numerical conditioning

Can be fixed by rescaling the data

22B. Leibe

266666666664

u01u1 u01v1 u01 u1v01 v1v

01 v01 u1 v1 1

u02u2 u02v2 u02 u2v02 v2v

02 v02 u2 v2 1

u03u3 u03v3 u03 u3v03 v3v

03 v03 u3 v3 1

u04u4 u04v4 u04 u4v04 v4v

04 v04 u4 v4 1

u05u5 u05v5 u05 u5v05 v5v

05 v05 u5 v5 1

u06u6 u06v6 u06 u6v06 v6v

06 v06 u6 v6 1

u07u7 u07v7 u07 u7v07 v7v

07 v07 u7 v7 1

u08u8 u08v8 u08 u8v08 v8v

08 v08 u8 v8 1

377777777775

26666666666664

F11

F12

F13

F21

F22

F23

F31

F32

F33

37777777777775

=

266666666664

0

0

0

0

0

0

0

0

377777777775


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

The Normalized Eight-Point Algorithm

1. Center the image data at the origin, and scale it so the

mean squared distance between the origin and the data

points is 2 pixels.

2. Use the eight-point algorithm to compute F from the

normalized points.

3. Enforce the rank-2 constraint using SVD.

4. Transform fundamental matrix back to original units: if T

and T’ are the normalizing transformations in the two

images, then the fundamental matrix in original coordinates

is TT F T’.23

B. Leibe [Hartley, 1995]Slide credit: Svetlana Lazebnik

11 11 13

22

33 31 33

T

T

d v v

F d

d v v

UDV U

SVD

Set d33 to

zero and

reconstruct F

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19


• Meaning of error

Sum of Euclidean distances between points xi and epipolar

lines Fx’i (or points x’i and epipolar lines FTxi), multiplied by

a scale factor

• Nonlinear approach for refining the solution: minimize

Similar to nonlinear minimization

approach for triangulation.

Iterative approach (Gauss-Newton,

Levenberg-Marquardt,…)24

B. Leibe

:)( 2

1

i

N

i

T

i xFx

N

i

i

T

iii xFxxFx1

22 ),(d),(d

O1O2

x1x2

X?

x’1

x’2


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Comparison of Estimation Algorithms

25B. Leibe

8-point Normalized 8-point Nonlinear least squares

Av. Dist. 1 2.33 pixels 0.92 pixel 0.86 pixel

Av. Dist. 2 2.18 pixels 0.85 pixel 0.80 pixel


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

3D Reconstruction with Weak Calibration

• Want to estimate world geometry without requiring calibrated cameras.

• Many applications: Archival videos

Photos from multiple unrelated users

Dynamic camera system

• Main idea:

Estimate epipolar geometry from a (redundant) set of point correspondences between two uncalibrated cameras.


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Stereo Pipeline with Weak Calibration

• So, where to start with uncalibrated cameras?

Need to find fundamental matrix F and the correspondences

(pairs of points (u’,v’) ↔ (u,v)).

• Procedure

1. Find interest points in both images

2. Compute correspondences

3. Compute epipolar geometry

4. Refine27

B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman

5

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19


1. Find interest points (e.g. Harris corners)

28B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19


2. Match points using only proximity

29B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Putative Matches based on Correlation Search

30B. Leibe Example from Andrew Zisserman

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

RANSAC for Robust Estimation of F

• Select random sample of correspondences

• Compute F using them

This determines epipolar constraint

• Evaluate amount of support – number of inliers

within threshold distance of epipolar line

• Iterate until a solution with sufficient support

has been found (or for max #iterations)

• Choose F with most support (#inliers)


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Putative Matches based on Correlation Search


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Pruned Matches

• Correspondences consistent with epipolar geometry


6

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Resulting Epipolar Geometry


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Epipolar Transfer

• Assume the epipolar geometry is known

• Given projections of the same point in two images, how can

we compute the projection of that point in a third image?

35B. Leibe

x1 x2

? x3


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Extension: Epipolar Transfer

• Assume the epipolar geometry is known

• Given projections of the same point in two images, how can

we compute the projection of that point in a third image?

36B. Leibe

x1 x2 x3l32l31

l31 = FT13 x1

l32 = FT23 x2

When does epipolar transfer fail?


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19




Weak calibration

Epipolar Transfer



Laser scanning


Ambiguity


Bundle adjustment

37B. Leibe

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Microsoft Kinect – How Does It Work?

• Built-in IR

projector

• IR camera for

depth

• Regular camera

for color

38B. Leibe

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Recall: Optical Triangulation

39B. Leibe

O1

x1

X?

Image plane

Camera center

3D Scene point

7

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Recall: Optical Triangulation

• Principle: 3D point given by intersection of two rays.

Crucial information: point correspondence

Most expensive and error-prone step in the pipeline…

40B. Leibe

O1 O2

x1 x2

X

Image plane

Camera center

3D Scene point

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Active Stereo with Structured Light

• Idea: Replace one camera by a projector.

Project “structured” light patterns onto the object

Simplifies the correspondence problem

41B. Leibe

O1 O2

x1 x2

X

Image plane

Camera center

3D Scene point

Projector

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

What the Kinect Sees…

42B. Leibe

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

3D Reconstruction with the Kinect

43B. Leibe

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Laser Scanning

• Optical triangulation

Project a single stripe of laser light

Scan it across the surface of the object

This is a very precise version of structured light scanning

45B. Leibe

Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/

Slide credit: Steve Seitz

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Laser Scanned Models

46B. Leibe

The Digital Michelangelo Project, Levoy et al.


http://graphics.stanford.edu/projects/mich/

8

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19


47B. Leibe



Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19


48B. Leibe



Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19


49B. Leibe



Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19


50B. Leibe



Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Poor Man’s Scanner

51Bouget and Perona, ICCV’98

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Slightly More Elaborate (But Still Cheap)

52B. Leibe

http://www.david-laserscanner.com/

Software freely available from Robotics Institute TU Braunschweig

../../../Local Settings/ANGEL.WRL

../../../Local Settings/ANGEL.WRL

http://www.david-laserscanner.com/

9

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19




Weak calibration

Epipolar Transfer



Laser scanning


Ambiguity


Bundle adjustment

53B. Leibe

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Structure from Motion

• Given: m images of n fixed 3D points

xij = Pi Xj , i = 1, … , m, j = 1, … , n

• Problem: estimate m projection matrices Pi and

n 3D points Xj from the mn correspondences xij54

B. Leibe

x1j

x2j

x3j

Xj

P1

P2

P3


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Applications

• E.g., movie special effects

Video

55B. Leibe Video Credit: Stefan Hafeneger

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Structure from Motion Ambiguity

• If we scale the entire scene by some factor k and, at the

same time, scale the camera matrices by the factor of 1/k,

the projections of the scene points in the image remain

exactly the same:

It is impossible to recover the absolute scale of the scene!

56B. LeibeSlide credit: Svetlana Lazebnik

x = PX =

µ1

kP

¶(kX)

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Structure from Motion Ambiguity

• If we scale the entire scene by some factor k and, at the

same time, scale the camera matrices by the factor of 1/k,

the projections of the scene points in the image remain

exactly the same.

• More generally: if we transform the scene using a

transformation Q and apply the inverse transformation to the

camera matrices, then the images do not change


x =PX= (PQ¡1)QX

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Reconstruction Ambiguity: Similarity

58B. LeibeSlide credit: Svetlana Lazebnik Images from Hartley & Zisserman

x =PX= (PQ¡1S )QSX

../cv-ws08/videos/MotivationFilm.mov

10

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Reconstruction Ambiguity: Affine


Affine

x =PX= (PQ¡1A )QAX

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Reconstruction Ambiguity: Projective


x=PX= (PQ¡1P )QPX

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Projective Ambiguity


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

From Projective to Affine


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

From Affine to Similarity


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Hierarchy of 3D Transformations

• With no constraints on the camera calibration matrix or on the scene, we get a projective reconstruction.

• Need additional information to upgrade the reconstruction to affine, similarity, or Euclidean. 64

B. Leibe

vTv

tAProjective

15dof

Affine

12dof

Similarity

7dof

Euclidean

6dof

Preserves intersection

and tangency

Preserves parallellism,

volume ratios

Preserves angles, ratios

of length

10

tAT

10

tRT

s

10

tRT

Preserves angles,

lengths


11

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Structure from Motion


xij = Pi Xj , i = 1, … , m, j = 1, … , n

• Problem: estimate m projection matrices Pi and

n 3D points Xj from the mn correspondences xij65

B. Leibe

x1j

x2j

x3j

Xj

P1

P2

P3


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Projective Structure from Motion


• zij xij = Pi Xj , i = 1,… , m, j = 1, … , n

• Problem: estimate m projection matrices Pi and n 3D points

Xj from the mn correspondences xij

• With no calibration info, cameras and points can only be recovered up to a 4£4 projective transformation Q:

X → QX, P → PQ-1

• We can solve for structure and motion when

2mn >= 11m +3n – 15

• For two cameras, at least 7 points are needed.


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Projective SfM: Two-Camera Case

• Assume fundamental matrix 𝐅 between the two views

First camera matrix: [I|0]Q-1

Second camera matrix: [A|b]Q-1

• Let , then

• And

• So we have

67B. Leibe

[ ]z z x A I | 0 X b Ax b

QXX ~

z z x b Ax b

( ) ( )z z x b x Ax b x

0][T Axbx

AbF ][

[ ] , [ | ]z z x I | 0 X x Ab X

b: epipole (FTb = 0), A = –[b×]F


0 ( )z Ax b x

F&P sec. 13.3.1

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Projective SfM: Two-Camera Case

• Decomposing the Fundamental Matrix

This means that if we can compute the fundamental matrix between

two cameras, we can directly estimate the two projection matrices

from F.

Once we have the projection matrices, we can compute the 3D

position of any point X by triangulation.

• How can we obtain both kinds of information at the same

time?

68B. Leibe

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Projective Factorization

• If we knew the depths z, we could factorize D to estimate

M and S.

• If we knew M and S, we could solve for z.

• Solution: iterative approach

(alternate between above two steps).69

B. Leibe

n

mmnmnmmmm

nn

nn

zzz

zzz

zzz

XXX

P

P

P

xxx

xxx

xxx

D

21

2

1

2211

2222222121

1112121111

Cameras

(3m × 4)

Points (4 × n)

D = MS has rank 4


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Sequential Structure from Motion

• Initialize motion from two images

using fundamental matrix

• Initialize structure

• For each additional view:

Determine projection matrix

of new camera using all the

known 3D points that are

visible in its image –

calibration

70B. Leibe

Cam

era

s

Points


12

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19










calibration

Refine and extend structure:

compute new 3D points,

re-optimize existing points

that are also seen by this camera –

triangulation

71B. Leibe

Cam

era

s

Points


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19










calibration

Refine and extend structure:

compute new 3D points,

re-optimize existing points

that are also seen by this camera –

triangulation

• Refine structure and motion: bundle adjustment72

B. Leibe

Cam

era

s

Points


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Bundle Adjustment

• Non-linear method for refining structure and motion

• Minimizing mean-square reprojection error

73B. Leibe

2

1 1

,),(

m

i

n

j

jiijDE XPxXP

x1j

x2j

x3

j

Xj

P1

P2

P3

P1Xj

P2Xj

P3Xj


Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

Bundle Adjustment

• Idea

Seek the Maximum Likelihood (ML) solution assuming the

measurement noise is Gaussian.

It involves adjusting the bundle of rays between each camera center

and the set of 3D points.

Bundle adjustment should generally be used as the final step

of any multi-view reconstruction algorithm.

– Considerably improves the results.

– Allows assignment of individual covariances to each measurement.

• However…

It needs a good initialization.

It can become an extremely large minimization problem.

• Very efficient algorithms available.

74B. Leibe

Pe

rce

ptu

al

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pu

ter

Vis

ion

Su

mm

er‘

19

References and Further Reading

• Background information on camera models and calibration

algorithms can be found in Chapters 6 and 7 of

• Also recommended: Chapter 9 of the same book on

Epipolar geometry and the Fundamental Matrix and

Chapter 11.1-11.6 on automatic computation of F.

75B. Leibe

R. Hartley, A. Zisserman

Multiple View Geometry in Computer Vision

2nd Ed., Cambridge Univ. Press, 2004

computer vision lecture 17 segmentation & grouping ng‘19€¦ · • partial reconstruction...

Documents