pierre bessière lppa – collège de france - cnrs cours « cognition bayésienne » 2010 bayesian...

59
Pierre Bessière — LPPA – Collège de France - CNRS Cours « Cognition bayésienne » — 2010 Bayesian Inference Algorithms Revisited

Upload: pierres-tanguy

Post on 03-Apr-2015

108 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Bayesian Inference

Algorithms Revisited

Page 2: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Inference

P Search |Known∧δ∧π( ) = P Search∧Free |Known∧δ∧π( )Free∑

=P Search∧Known∧Free|δ ∧π( )

Free∑

P Known|δ ∧π( )

=

P Search∧Free∧Known |δ∧π( )Free∑

P Search∧Free∧Known |δ∧π( )Search∧Free

=1Z

× P Search∧Free∧Known |δ∧π( )Free∑

=1

Z× P L1( ) × P Li |Ri( )[ ]

i=2

K

∏ ⎡

⎢ ⎢ ⎢

⎥ ⎥ ⎥

Free

Page 3: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

2 optimisation problems

Draw P Search |Known∧δ∧π( )( )

P Search |Known∧δ∧π( )

=1Z

× P Search∧Known∧Unknown |δ∧π( )Free∑

Page 4: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Symbolic Simplification

Page 5: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Exact symbolic simplification (example)

Page 6: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Question dependent

9x106

Page 7: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Reordering

Page 8: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Applying normalization

Page 9: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Factorizing

Page 10: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Result (1)

19+19+2=40

Page 11: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Result (2)

(21x10)+9+1=220

Page 12: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Summary1. Reorder2. Normalize3. Factorize

Page 13: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Question independent

Page 14: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Sharing parts (1)

Page 15: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Sharing parts (2)

Page 16: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Sharing parts (3)

Page 17: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Sharing parts (4)

Page 18: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Sharing parts (5)

Page 19: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Sharing parts (6)

Page 20: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Sharing parts (7)

Page 21: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Message passing algorithms

Page 22: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Example 2

Page 23: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Question dependent

Page 24: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Junction Tree Algorithm

Page 25: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Cut-Set Algorithm

Page 26: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Max-Product & Min-Sum Algorithms

Page 27: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Viterbi Algorithm

Page 28: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Approximate symbolic simplification:

Variational methods

Page 29: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Crunching numbers: Sampling methods

1. Monte Carlo (MC)1. Importance sampling2. Rejection sampling

2. Markof Chains Monte Carlo (MCMC)1. Metropolis sampling2. Gibbs sampling

Information theory, Inference and learning algorithms (2003) D. MacKay

Page 30: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Bayesian Learning Revisited

Page 31: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Data and Preliminary knowledge

Page 32: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

1324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141645551112666

How to Deal with Data?

Using Preliminary Knowledge

Page 33: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

P X1⊗ X2⊗...⊗ Xn |δ ⊗π( )

P Δ⊗Π( )=P Π( )×P Δ |Π( ) =P Δ( )×P Π |Δ( )

• Direct problem:

• Inverse problem:

P Δ |Π( ) =P Δ( )×P Π |Δ( )

P Π( )

P Π |Δ( ) =P Π( )×P Δ|Π( )

P Δ( )

Page 34: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Bernoulli's Urn (1)• Variables

Draw• Decomposition

• Parametrical Form

Preliminary Knowledge π: "We draw from an urn containing w white balls and b black balls"

P Draw=black[ ]|π( )=b

w+b

P Draw|π( )

P Draw=white[ ] |π( ) =w

w+b

Page 35: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Bernoulli's Urn (2)• Variables:

Δ =Draw1⊗Draw2⊗...⊗ Drawm

WBΠ = ′ π ⊗W⊗ B

• Decomposition:P Δ |Π( ) =P Draw1⊗...⊗Drawm | ′ π ⊗W⊗B( )

• Parametrical Form:P Drawi =white[ ]| ′ π ⊗w⊗b( ) =

ww+b

P Draw2 | Draw1 =white[ ]⊗ ′ π ⊗w⊗b( )=w−1

w+b−1

Note:

=P Draw1 | Draw2 =white[ ]⊗ ′ π ⊗w⊗b( )

P Δ |Π( ) =

ω

w

⎝ ⎜

⎠ ⎟ ×

β

b

⎝ ⎜

⎠ ⎟

m

w+b⎛ ⎝ ⎜

⎞ ⎠ ⎟

Page 36: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Bernoulli's Urn (3)P Π |Δ( ) =

P Π( )×P Δ|Π( )P Δ( )

P π1 |δ( )P π2 |δ( )

=P π1( )×P δ |π1( )P π2( )×P δ |π2( )

=P π1( )P π2( )

×

ω

w1

⎝ ⎜

⎠ ⎟ ×

β

b1

⎝ ⎜

⎠ ⎟ ×

m

w2 +b2

⎝ ⎜

⎠ ⎟

ω

w2

⎝ ⎜

⎠ ⎟ ×

β

b2

⎝ ⎜

⎠ ⎟ ×

m

w1+b1

⎝ ⎜

⎠ ⎟

w1 =5000,b1 =5000 w2 =3000,b2 =7000

m,ω

2,1 1.2

4,2 1.4

10,5 2.4

20,10 5.7

50,25 80

100,50 6728

P π1|δ( )P π2|δ( )

Page 37: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Parameters IdentificationVariables:

Δ , Π = ′ Π ⊗Ψ

Decomposition:P Δ⊗Π( )=P Δ⊗ ′ Π ⊗Ψ( ) =P ′ Π ( )×P Ψ | ′ Π ( )×P Δ |Ψ⊗ ′ Π ( )

P Ψ |δ ⊗ ′ π ( ) =P δ⊗ ′ π ⊗Ψ( )

P δ ⊗ ′ π ( )

=1ω

×P Ψ | ′ π ( )×P δ |Ψ⊗ ′ π ( )

Page 38: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Model SelectionVariables:

Δ , Π = ′ Π ⊗Ψ

Decomposition:P Δ⊗Π( )=P Δ⊗ ′ Π ⊗Ψ( ) =P ′ Π ( )×P Ψ | ′ Π ( )×P Δ |Ψ⊗ ′ Π ( )

P ′ Π |δ( )=P δ⊗Ψ ⊗ ′ Π ( )

P δ( )Ψ∑

=1ω

×P ′ Π ( )× P Ψ | ′ Π ( )×P δ |Ψ ⊗ ′ Π ( )[ ]Ψ∑

Page 39: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Summary

Preliminary

Knowledge

Do we need it?Where does it come from?

How to specify it?

Variables Pre-treatmentsPost-treatments

Decomposition

Parametrical FormsModel

Selection

Parameters values Learning

Page 40: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Entropy Principles

Content:• Entropy Principle Statement• Frequencies and Laplace succession law• Observables and Exponential laws• Wolf's dice

Page 41: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Entropy Principle Statement

H P( )=− Pi ×logPi( )[ ]i=1

q

20 000 flip of a coins: 9553 headsProbability distribution of the coin?

Page 42: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Observables and Exponential Laws

Observable:fi V( )→ ℜ

Constraint levels:∀j, j ∈ 1,...,m{ }, P V( )×f j V( )[ ]

V∑ =Fj

Maximum Entropy Distribution:

P* V( ) =1

Z λ1,...,λm( )×e

− λ j ×fj V( )[ ]j=1

m

Partition Function:Z λ1,...,λm( ) = e−λ1f1 V( )−...−λmfm V( )

V∑

Constraints differential equation:∂

∂λ j

logZ λ1,...,λm( )( )+Fj =0

proof

Page 43: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

20 000 FlipsObservable:

Constraint levels:

Maximum Entropy Distribution:

Partition Function:

Constraints differential equation:

f1 V( ) =1, if V is "Head"; else f1 V( ) =0

P V( )×f1 V( )[ ]V∑ =F1 =

955320000

P* V( ) =1

Z λ1( )×e−λ1f1 V( )

Z λ1( ) = e−λ1f1 V( )

V∑ =1+e−λ1

∂∂λ1

log Z λ1( )( )+F1 =0

∂∂λ1

log1+e−λ1( )+F1 =0

⇔ −e−λ1

1+e−λ1+F1 =0

⇔1F1

−1⎛

⎝ ⎜

⎠ ⎟ e−λ1 =1

⇔ λ1 =log1−F1

F1

⎝ ⎜

⎠ ⎟

P* V( ) =1

1+e−log

1−F1F1

⎝ ⎜

⎠ ⎟

×e−log

1−F1F1

⎝ ⎜

⎠ ⎟ f1 V( )

P* V =Head[ ]( )=F1

Page 44: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

1324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141641324154131461646541464123146464634343601161464116461164564114641641131116646414616141343511616316416114461164641141135521611614111161616111314413161414646461346464464446461131316416144164316341641416413241541314616465414641231464646343436011614641164611645641146416411311166464146161413435116163164161144611646411411355216116141111616161113144131614146464613464644644464611313164161441643163416414164132415413146164654146412314646463434360116146411646116456411464164113111664641461614134351161631641611446116464114113552161161411116161611131441316141464646134646446444646113131641614416431634164141645551112666

Page 45: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Frequencies and Laplace Succession Law

Preliminary Knowledge:1- Each of the 20 000 digit is a number2- The 20 000 data come from the same phenomenon3- A single variable V has been observed 20 000 times4- The order of these observations is not relevant

P* V =i[ ]( ) =ni

n

5- The variable V may take 6 different values

P* V =i[ ]( ) =ni +1

n+ V⎣ ⎦V=1 V=2 V=3 V=4 V=5 V=6 Total

ni 3246 3449 2897 2841 3635 3932 20000

Uniform 0,16666 0,16666 0,16666 0,16666 0,16666 0,16666 1

Laplace 0,16230 0,17245 0,14486 0,14206 0,18174 0,19659 1

Page 46: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Wolf's dice (1)H1 Hypothesis: excavations shifted the gravity center

f1 V( ) =V F1 =3.5983

P* V( ) =1Z

×e0.03372×V

V=1 V=2 V=3 V=4 V=5 V=6 Total

ni 3246 3449 2897 2841 3635 3932 20000

Uniform 0.16666 0.16666 0.16666 0.16666 0.16666 0.16666 1

Laplace 0.16230 0.17245 0.14486 0.14206 0.18174 0.19659 1

H1 0.15294 0.15818 0.16361 0.16922 0.17502 0.18103 1

Page 47: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Wolf's dice (2)H2 Hypothesis: The dice is oblong along the 1-6 direction and the excavations shifted the gravity centerf1 V( ) =V F1 =3.5983

f2 V( ) =−1 if V=1 or if V=0; else f2 V( ) =6 F2 =1

20000× ni ×f2 i( )

i=1

6

∑ =0.3589

P* V( ) =1Z

×e0.03234×V−0.1104×f2 V( )

V=1 V=2 V=3 V=4 V=5 V=6 Total

ni 3246 3449 2897 2841 3635 3932 20000

Uniform 0.16666 0.16666 0.16666 0.16666 0.16666 0.16666 1

Laplace 0.16230 0.17245 0.14486 0.14206 0.18174 0.19659 1

H1 0.15294 0.15818 0.16361 0.16922 0.17502 0.18103 1

H2 (1-6) 0,16497 0,15259 0,15760 0,16278 0,16813 0,19393 1

H3 (2-5) 0,14803 0,16808 0,15843 0,16390 0,18612 0,17543 1

H4 (3-4) 0,16433 0,16963 0,14117 0,14573 0,18656 0,19258 1

Page 48: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Wolf's dice (3)Inverse Problem:

P hi |δ( )=P hi( )×P δ |hi( )

P δ( )

P δ |hi( )=W×pi13246×pi2

3449×pi32897×pi4

2841×pi53635×pi6

3932

Laplace h4 h2 h3 h1 Uniform

P(hi) 0.99 0.01 10-32 10-35 10-45 10-59

Page 49: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Theoretical Basis

Content:• What is a good representation?•Combinatorial justification• Information theory justification • Bayesian justification• Axiomatic justification• Entropy concentration theorems justifications

Objective:Justify the use of the entropy function H

Page 50: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

What is a Good Representation?

Page 51: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Combinatorial JustificationStatistical Mechanic

q microscopic states

Macroscopic state νk = n1,...,nq{ }

ni =ni=1

q

ni ×eii=1

q

∑ =e

W νk( ) =n!

n1!×...×nq!

logW νk( )( ) ≈−n×ni

nlog

ni

n⎛ ⎝ ⎜ ⎞

⎠ ⎟

i=1

q

ν w

10000,10000 101386

9553,10447 101384

0,20000 1

Probabilistic Inference

q propositions

Distribution δk = p1,...,pq{ }

pii=1

q

∑ =1

∀j, j ∈ 1,...,m{ }; pi ×fj vi( )i=1

q

∑ =Fj

W δk( ) =n!

n×p1( )!×...× n×pq( )!

logW δk( )( ) ≈−n× pi log pi( )i=1

q

ν w

10000,10000 101386

9553,10447 101384

0,20000 1

Page 52: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Entropy Principles Preliminary Knowledge

"Exchangeability" Preliminary Knowledge: • has no meaningful order• Each "experience" in is independent from the others knowing the model and its parameters• Each "experience" in corresponds to a unique phenomenon

P Δ |ψ ⊗π( )

=P 1X⊗...⊗nX |ψ ⊗π( )

=P 1X |ψ ⊗π( )×...×P nX |ψ ⊗π( )

= P i X |ψ ⊗π( )i=1

n

= P X =xj[ ]|ψ ⊗π( )n×Fj

j=1

q

Page 53: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Maximum Entropy for Frequencies

Variables:

Decomposition:

ΟΔ=1X⊗...⊗nX Ψ Π

P Δ⊗Ψ⊗Ο⊗Π( ) =P Π( )×P Δ |Π( )×P Ψ |Δ⊗Π( )×P Ο |Δ⊗Π( )

P Ψ |ο⊗π( ) =1Z

× P Δ|π( )×P Ψ |Δ⊗π( )×P Ο |Δ⊗π( )Δ∑

=1Z

× P Δ|π( )×P Ψ |Δ⊗π( )ΔCompatible with ο

=1Z

×1

mn ×n!

n×F1( )!×...× n×Fm( )!

=1′ Z ×

1n×F1( )!×...× n×Fm( )!

−n× Fj ×logFj( )j=1

m

Proof

Page 54: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Minimum X-entropy with Observed Frequencies

Variables:Δ=1X⊗...⊗nX Ψ Π

Decomposition:P Δ⊗Ψ ⊗Π( ) =P Π( )×P Ψ |Π( )×P Δ |Ψ⊗Π( )

P Ψ |π( ) =1Z

× P π( )×P Ψ |π( )×P Δ |Ψ⊗π( )Δ∑

=1′ Z × P X =xj[ ]|Ψ⊗π( )

j=1

q

∏n×Fj⎡

⎣ ⎢

⎦ ⎥

Δ∑

=1′ Z ×

n!n×F1( )!... n×Fq( )!

× P X =xj[ ]|Ψ⊗π( )j=1

q

∏n×Fj

=1′ ′ Z

×P X =xj[ ]|Ψ⊗π( )

n×Fj( )!j=1

q

∏n×F j

n× Fj ×logFj( )

logP X =xj[ ]|Ψ⊗π( )( )j=1

m

Page 55: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Shannon’s justification

Shannon C. E. (1948) ; “A Mathematical Theory of Communication” ; Bell Systems Technical Journal ; 27

Reprinted as Shannon C.E. & Weaver (1949) “The Mathematical Theory of Communication” ; University of Illinois Press, Urbana

Page 56: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Shore’s Axiomatic Justification

Shore, J.E. & Johnson, R.W. (1980) ; “Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy” ; IEEE Transactions on Information Theory ; IT-26 26-37

Page 57: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Entropy Concentration Theorem

Robert Claudine (1990) ; “An Entropy Concentration Theorem: Applications in Artificial Intelligence and Descriptive Statistics” ; Journal of Applied Probabilities

Jaynes E.T. (1982) ; “On the rationale of Maximum Entropy Methods” ; Proceedings of the IEEE

Page 58: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010

Want to know more ?

Bayesian-Programming.org

Probabilistic reasoning and decision making in sensory-motor systems

Springer, Star Series

[email protected]

Page 59: Pierre Bessière LPPA – Collège de France - CNRS Cours « Cognition bayésienne » 2010 Bayesian Inference Algorithms Revisited

Pierre Bessière — LPPA – Collège de France - CNRSCours « Cognition bayésienne » — 2010 59