a graph-matching kernel for object categorization

42
A Graph-Matching Kernel for Object Categorization Olivier Duchenne Armand Joulin Jean Ponce Willow Lab 1

Upload: binh

Post on 22-Feb-2016

51 views

Category:

Documents


0 download

DESCRIPTION

A Graph-Matching Kernel for Object Categorization. Willow Lab. Olivier Duchenne. Armand Joulin. Jean Ponce. Goal: Object Categorization. DINOSAUR. CAT. PANDA. Comparing images. We need to design a good image similarity measure :. ≈. ?. Unaligned Images are hard to compare. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: A Graph-Matching Kernel for Object Categorization

1

A Graph-Matching Kernel for Object Categorization

OlivierDuchenne

ArmandJoulin

JeanPonce

WillowLab

duchenne
pour le panda:Une diapo pour expliquer l'example d'Ishikawa eavec L1 ?parler du non-croisementparler du fait que ca entrave l'alpha-expansion et que notre algo peut passer outre.
duchenne
slide à refaire23-27 explication de la fonction-objectf29-39 explication de lóptimisation
Page 2: A Graph-Matching Kernel for Object Categorization

2

Goal: Object Categorization

CAT

DINOSAUR

PANDA

Page 3: A Graph-Matching Kernel for Object Categorization

3

Comparing images

≈?

We need to design a good image similarity measure:

Page 4: A Graph-Matching Kernel for Object Categorization

4

Unaligned Images are hard to compare

Page 5: A Graph-Matching Kernel for Object Categorization

5

Unaligned Images are hard to compare

Page 6: A Graph-Matching Kernel for Object Categorization

6

Graph Matching

[1] A. C. Berg, T. L. Berg, and J. Malik. CVPR05

[2] M. Leordeanu and M. Hebert. ICCV05

Page 7: A Graph-Matching Kernel for Object Categorization

Feature Matching

Caputo (2004)Boiman, Shechtman & Irani (2008)Caputo & Jie (2009)

Page 8: A Graph-Matching Kernel for Object Categorization

Graph Matching

Nevatia & Binford’72Fischler & Elschlager’73Berg, Berg, & Malik’05Leordeanu & Hebert’05Cour & Shi’08Kim & Grauman’10Etc.

Page 9: A Graph-Matching Kernel for Object Categorization

• Lazebnik, Schmid and Ponce (2006)• Indyk and Thaper (2003) • Grauman and Darrell (2005)

Spatial pyramid kernels

(Courtesy of L. Lazebnik)

9

Page 10: A Graph-Matching Kernel for Object Categorization

10

Page 11: A Graph-Matching Kernel for Object Categorization

11

9 matches x 1

= 9

Page 12: A Graph-Matching Kernel for Object Categorization

12

9 matches x 1

= 9

Page 13: A Graph-Matching Kernel for Object Categorization

13

9 matches x 1 4 matches x ½

= 9 = 2

Page 14: A Graph-Matching Kernel for Object Categorization

14

9 matches x 1 4 matches x ½

= 9 = 2

Page 15: A Graph-Matching Kernel for Object Categorization

15

9 matches x 1 4 matches x ½ 2 matches x ¼

= 9 = 2 = 1/2

Total matching weight (value of spatial pyramid kernel ): 9 + 2 + 0.5 = 11.5

Page 16: A Graph-Matching Kernel for Object Categorization

16

Graph MatchingVSSpatial Pyramids

•Approximate Feature Matching•Does not take into account pair-wise relationship•Require quantization•Low Computational Coast

•Graph Matching•Take into account pair-wise relationship•Does not require quantization•High Computational Coast

Page 17: A Graph-Matching Kernel for Object Categorization

17

So, how does it perform in real life?

• Graph-Matching keeps more information, but is much slower…

• In addition, it appears to perform much worse than SPM kernel.

Caltech 101 (%)Graph Matching Based

Methods15 examples

Berg Matching [CVPR 2005] 48GBVote [Berg phD thesis] 52

Kim et Graumann [CVPR 2010] 61Spatial Pyramid MethodsBoureau et al. [CVPR10] 69.0

Yang et al.[ICCV09] 73.3

Page 18: A Graph-Matching Kernel for Object Categorization

18

Graph MatchingVSSpatial Pyramids

•Dense Features•SVM classifier•Fast•No pair-wise Information•State-of the-Artperformance

•Sparse Features•NN Classifier•Slow•Use pair-wiseInformation•Lower performance

Page 19: A Graph-Matching Kernel for Object Categorization

19

Our ApproachSpatial Pyramids

•Dense Features•SVM classifier•Fast•No pair-wise Information•State-of the-Artperformance

•As Dense•SVM Classifier•Fast enough•Use pair-wiseInformation•State-of-the-art performance

VS

Page 20: A Graph-Matching Kernel for Object Categorization

20

Outline of our method

K=

Image i

Image j

K(Imi,Imj) •As Dense•SVM Classifier•Fast enough•Use pair-wiseInformation•State-of-the-art performance

Page 21: A Graph-Matching Kernel for Object Categorization

21

≈?

Page 22: A Graph-Matching Kernel for Object Categorization

22

Similarity?≈

Sum of Local

Similarities

Deformation Smoothness

Maximized over all possible

deformations

Page 23: A Graph-Matching Kernel for Object Categorization

24

In our case: we allow each point to match to a 11x11 grid in the other image.

Page 24: A Graph-Matching Kernel for Object Categorization

25

Unary term

)(),()( nnn dnfnfdU

Page 25: A Graph-Matching Kernel for Object Categorization

26

GOOD

Binary term

Page 26: A Graph-Matching Kernel for Object Categorization

27

BAD

Binary term

Page 27: A Graph-Matching Kernel for Object Categorization

28

VERY BAD

Binary term II

otherwise 01 if ][1 if ][

),(, mnnm

mnnm

nmnm yydydyxxdxdx

ddv

Page 28: A Graph-Matching Kernel for Object Categorization

29

Optimization problem

),(),()()(),(

,),(

, nmEnm

nmnmEnm

nmVn

nn ddvddudUdE

Minimize with respect to the integer disparity vector d:

• This is (in general) an NP-hard problem…

…because there is no natural ordering on the vectors d

• We give an efficient approximation algorithm

Page 29: A Graph-Matching Kernel for Object Categorization

30

Vertical MoveThe general problem is NP Hard, but if we just allow vertical moves, it can be solved optimally.

Page 30: A Graph-Matching Kernel for Object Categorization

31

Horizontal MoveHorizontal moves can also be solved optimally.

Page 31: A Graph-Matching Kernel for Object Categorization

32

Diagonal MoveDiagonal moves can also be solved optimally.

Page 32: A Graph-Matching Kernel for Object Categorization

33

General MoveIf all points move in the same direction (e.g. dx increasing dy decreasing), it can also be solved optimally.

The allowed moves can be different for each point, as long as they are in the same direction.

Page 33: A Graph-Matching Kernel for Object Categorization

34

Optimization

• Even if the whole optimization is NP hard, for each of the sub-problems (horizontal move,…) we can find the global optimum.

• To optimize our objective function, we iterate over the different types of move, until convergence.

• We can do it using Ishikawa’s method.

Page 34: A Graph-Matching Kernel for Object Categorization

35

Optimization methodIshikawa’s method can solve:

with g convex, and the labels mapped on integer.

with

Page 35: A Graph-Matching Kernel for Object Categorization

36

Comparison with alpha-expansion

• Alpha Expansion: At each iteration, each node can choose between keeping its label, or taking the label alpha.

• Ours: At each iteration, each label can chose between K (=11) labels.

At convergence, the local minimum is better than its n neighbors:

• Alpha Expansion: • Ours:• Total number of configurations:

132102 nNlN

45010nN

lNp89910nN

lN

Page 36: A Graph-Matching Kernel for Object Categorization

37

SummaryExtract Features

Global Normalization

Match pairs of images

Make a kernel from

matching scores

Train 1vsAll SVM Test with SVM

Page 37: A Graph-Matching Kernel for Object Categorization

38

Page 38: A Graph-Matching Kernel for Object Categorization

39

Results for Graph Matching methods

Caltech 101 (%)Graph Matching Based

Methods15 examples

Berg Matching [CVPR 2005] 48GBVote [Berg PhD thesis] 52Kim and Graumann [CVPR

2010]61

Ours 75.3

+14%

Page 39: A Graph-Matching Kernel for Object Categorization

40

Results: Caltech 101Caltech 101 (%)

Feature Method 15 examples

30 examples

SingleNBNN (1 Desc.) [CVPR08] 65.0 -Boureau et al. [CVPR10] 69.0 75.7

Ours 75.3 80.3

MultipleGu et al.[CVPR09] - 77.5

Gehler et al.[ICCV09] - 77.7NBNN (5 Desc.)[CVPR08] 72.8 -

Yang et al.[ICCV09] 73.3 84.3

+5%+6%

Page 40: A Graph-Matching Kernel for Object Categorization

41

Results: Caltech 256

Caltech 256 (%)Feature Method 30 examples

Single

SPM+SVM[07] 34.1Kim et al. [CVPR10] 36.3

NBNN (1 desc.) 37.0Ours 38.1

Multiple NBNN (5 Desc.)[CVPR08]

42.0

Page 41: A Graph-Matching Kernel for Object Categorization

42

Matching Time

Method Number of Nodes TimeBerg el al.[CVPR 05] 50 5s

Leordaenu et al.[ICCV 05]

130 9s

Kim , Grauman[CVPR10]

4800 10s

Kim, Grauman[CVPR 10]

500 1s

Alpha-Expansion 500 1sTRW-S 500 10sOurs 500 0.04s

Page 42: A Graph-Matching Kernel for Object Categorization

43

Conclusion• Graph-matching can make good

performance• It is more interpretable, and it makes it

easy to visualize what is happening.• Our main contributions are: the

combination of SVM and graph matching; and the design of a fast

matching algorithm.

Thank youQuestions?