texture-based computational models of tissue in biomedical images: initial experience with digital...

50
TEXTURE-BASED COMPUTATIONAL MODELS OF TISSUE IN BIOMEDICAL IMAGES: INITIAL EXPERIENCE WITH DIGITAL HISTOPATHOLOGY Adrien Depeursinge, PhD Journées GdR ISIS “Analyse de Tissu Biologique et Histopathologie Numérique” Paris, June 23rd 2014

Upload: hes-so-haute-ecole-specialisee-de-suisse-occidentale

Post on 07-Aug-2015

60 views

Category:

Health & Medicine


0 download

TRANSCRIPT

Page 1: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

TEXTURE-BASED COMPUTATIONAL MODELS OF TISSUE IN BIOMEDICAL IMAGES:

INITIAL EXPERIENCE WITH DIGITAL HISTOPATHOLOGY

Adrien Depeursinge, PhD Journées GdR ISIS “Analyse de Tissu Biologique et Histopathologie Numérique”

Paris, June 23rd 2014

Page 2: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

BACKGROUND – RADIOMICS - HISTOPATHOLOMICS

• Personalized medicine aims at enhancing the patient’s quality of life and prognosis

• Tailored treatment and medical decisions based on the molecular composition of diseased tissue

• Current limitations [1]

• Molecular analysis of tissue composition is invasive (biopsy), slow and costly

• Cannot capture molecular heterogeneity

2

[1] Intratumor heterogeneity and branched evolution revealed by multiregion sequencing, Gerlinger et al., N Engl J Med, 366(10):883-92, 2012.

Intr atumor Heterogeneity Revealed by multiregion Sequencing

n engl j med 366;10 nejm.org march 8, 2012 887

tion through loss of SETD2 methyltransferase func-tion driven by three distinct, regionally separated mutations on a background of ubiquitous loss of the other SETD2 allele on chromosome 3p.

Convergent evolution was observed for the X-chromosome–encoded histone H3K4 demeth-ylase KDM5C, harboring disruptive mutations in R1 through R3, R5, and R8 through R9 (missense

and frameshift deletion) and a splice-site mutation in the metastases (Fig. 2B and 2C).

mTOR Functional Intratumor HeterogeneityThe mammalian target of rapamycin (mTOR) ki-nase carried a kinase-domain missense mutation (L2431P) in all primary tumor regions except R4. All tumor regions harboring mTOR (L2431P) had

B Regional Distribution of Mutations

C Phylogenetic Relationships of Tumor Regions D Ploidy Profiling

A Biopsy Sites

R2 R4

DI=1.43

DI=1.81

M2bR9

Tetraploid

R4b

R9 R8R5

R4a

R1R3R2

M1M2b

M2a

VHL

KDM5C (missense and frameshift)mTOR (missense)

SETD2 (missense)KDM5C (splice site)

SETD2 (splice site)

?

SETD2 (frameshift)

PreP

PreM

Normal tissue

PrePPreMR1R2R3R5R8R9R4M1M2aM2b

C2o

rf85

WD

R7SU

PT6H

CD

H19

LAM

A3

DIX

DC

1H

PS5

NRA

PKI

AA

1524

SETD

2PL

CL1

BCL1

1AIF

NA

R1$

DA

MTS

10

C3

KIA

A12

67.

RT4

CD

44A

NKR

D26

TM7S

F4SL

C2A

1D

AC

H2

MM

AB

ZN

F521

HM

G20

AD

NM

T3A

RLF

MA

MLD

1M

AP3

K6H

DA

C6

PHF2

1BFA

M12

9BRP

S8C

IB2

RAB2

7ASL

C2A

12D

USP

12A

DA

MTS

L4N

AP1

L3U

SP51

KDM

5CSB

F1TO

M1

MYH

8W

DR2

4IT

IH5

AKA

P9FB

XO1

LIA

STN

IKSE

TD2

C3o

rf20

MR1

PIA

S3D

IO1

ERC

C5

KLALK

BH8

DA

PK1

DD

X58

SPA

TA21

ZN

F493

NG

EFD

IRA

S3LA

TS2

ITG

B3FL

NA

SATL

1KD

M5C

KDM

5CRB

FOX2

NPH

S1SO

X9C

ENPN

PSM

D7

RIM

BP2

GA

LNT1

1A

BHD

11U

GT2

A1

MTO

RPP

P6R2

ZN

F780

AW

SCD

2C

DKN

1BPP

FIA

1THSS

NA

1C

ASP

2PL

RG1

SETD

2C

CBL

2SE

SN2

MA

GEB

16N

LRP7

IGLO

N5

KLK4

WD

R62

KIA

A03

55C

YP4F

3A

KAP8

ZN

F519

DD

X52

ZC

3H18

TCF1

2N

USA

P172

X4KD

M2B

MRP

L51

C11

orf6

8A

NO

5EI

F4G

2M

SRB2

RALG

DS

EXT1

ZC

3HC

1PT

PRZ

1IN

TS1

CC

R6D

OPE

Y1A

TXN

1W

HSC

1C

LCN

2SS

R3KL

HL1

8SG

OL1

VHL

C2o

rf21

ALS

2CR1

2PL

B1FC

AM

RIF

I16

BCA

S2IL

12RB

2

PrivateUbiquitous Shared primary Shared metastasis

Private

Ubiquitous

Lungmetastases

Chest-wallmetastasis

Perinephricmetastasis

M110 cm

R7 (G4)

R5 (G4)

R9

R3 (G4)

R1 (G3) R2 (G3)

R4 (G1)

R6 (G1)

Hilu

m

R8 (G4)

Primarytumor

Shared primaryShared metastasis

M2b

M2a

Propidium Iodide Staining

No.

of C

ells

The New England Journal of Medicine Downloaded from nejm.org at UNIVERSITE DE GENEVE on June 2, 2014. For personal use only. No other uses without permission.

Copyright © 2012 Massachusetts Medical Society. All rights reserved.

Intr atumor Heterogeneity Revealed by multiregion Sequencing

n engl j med 366;10 nejm.org march 8, 2012 887

tion through loss of SETD2 methyltransferase func-tion driven by three distinct, regionally separated mutations on a background of ubiquitous loss of the other SETD2 allele on chromosome 3p.

Convergent evolution was observed for the X-chromosome–encoded histone H3K4 demeth-ylase KDM5C, harboring disruptive mutations in R1 through R3, R5, and R8 through R9 (missense

and frameshift deletion) and a splice-site mutation in the metastases (Fig. 2B and 2C).

mTOR Functional Intratumor HeterogeneityThe mammalian target of rapamycin (mTOR) ki-nase carried a kinase-domain missense mutation (L2431P) in all primary tumor regions except R4. All tumor regions harboring mTOR (L2431P) had

B Regional Distribution of Mutations

C Phylogenetic Relationships of Tumor Regions D Ploidy Profiling

A Biopsy Sites

R2 R4

DI=1.43

DI=1.81

M2bR9

Tetraploid

R4b

R9 R8R5

R4a

R1R3R2

M1M2b

M2a

VHL

KDM5C (missense and frameshift)mTOR (missense)

SETD2 (missense)KDM5C (splice site)

SETD2 (splice site)

?

SETD2 (frameshift)

PreP

PreM

Normal tissue

PrePPreMR1R2R3R5R8R9R4M1M2aM2b

C2o

rf85

WD

R7SU

PT6H

CD

H19

LAM

A3

DIX

DC

1H

PS5

NRA

PKI

AA

1524

SETD

2PL

CL1

BCL1

1AIF

NA

R1$

DA

MTS

10

C3

KIA

A12

67.

RT4

CD

44A

NKR

D26

TM7S

F4SL

C2A

1D

AC

H2

MM

AB

ZN

F521

HM

G20

AD

NM

T3A

RLF

MA

MLD

1M

AP3

K6H

DA

C6

PHF2

1BFA

M12

9BRP

S8C

IB2

RAB2

7ASL

C2A

12D

USP

12A

DA

MTS

L4N

AP1

L3U

SP51

KDM

5CSB

F1TO

M1

MYH

8W

DR2

4IT

IH5

AKA

P9FB

XO1

LIA

STN

IKSE

TD2

C3o

rf20

MR1

PIA

S3D

IO1

ERC

C5

KLALK

BH8

DA

PK1

DD

X58

SPA

TA21

ZN

F493

NG

EFD

IRA

S3LA

TS2

ITG

B3FL

NA

SATL

1KD

M5C

KDM

5CRB

FOX2

NPH

S1SO

X9C

ENPN

PSM

D7

RIM

BP2

GA

LNT1

1A

BHD

11U

GT2

A1

MTO

RPP

P6R2

ZN

F780

AW

SCD

2C

DKN

1BPP

FIA

1THSS

NA

1C

ASP

2PL

RG1

SETD

2C

CBL

2SE

SN2

MA

GEB

16N

LRP7

IGLO

N5

KLK4

WD

R62

KIA

A03

55C

YP4F

3A

KAP8

ZN

F519

DD

X52

ZC

3H18

TCF1

2N

USA

P172

X4KD

M2B

MRP

L51

C11

orf6

8A

NO

5EI

F4G

2M

SRB2

RALG

DS

EXT1

ZC

3HC

1PT

PRZ

1IN

TS1

CC

R6D

OPE

Y1A

TXN

1W

HSC

1C

LCN

2SS

R3KL

HL1

8SG

OL1

VHL

C2o

rf21

ALS

2CR1

2PL

B1FC

AM

RIF

I16

BCA

S2IL

12RB

2

PrivateUbiquitous Shared primary Shared metastasis

Private

Ubiquitous

Lungmetastases

Chest-wallmetastasis

Perinephricmetastasis

M110 cm

R7 (G4)

R5 (G4)

R9

R3 (G4)

R1 (G3) R2 (G3)

R4 (G1)

R6 (G1)

Hilu

mR8 (G4)

Primarytumor

Shared primaryShared metastasis

M2b

M2a

Propidium Iodide Staining

No.

of C

ells

The New England Journal of Medicine Downloaded from nejm.org at UNIVERSITE DE GENEVE on June 2, 2014. For personal use only. No other uses without permission.

Copyright © 2012 Massachusetts Medical Society. All rights reserved.

Intr atumor Heterogeneity Revealed by multiregion Sequencing

n engl j med 366;10 nejm.org march 8, 2012 887

tion through loss of SETD2 methyltransferase func-tion driven by three distinct, regionally separated mutations on a background of ubiquitous loss of the other SETD2 allele on chromosome 3p.

Convergent evolution was observed for the X-chromosome–encoded histone H3K4 demeth-ylase KDM5C, harboring disruptive mutations in R1 through R3, R5, and R8 through R9 (missense

and frameshift deletion) and a splice-site mutation in the metastases (Fig. 2B and 2C).

mTOR Functional Intratumor HeterogeneityThe mammalian target of rapamycin (mTOR) ki-nase carried a kinase-domain missense mutation (L2431P) in all primary tumor regions except R4. All tumor regions harboring mTOR (L2431P) had

B Regional Distribution of Mutations

C Phylogenetic Relationships of Tumor Regions D Ploidy Profiling

A Biopsy Sites

R2 R4

DI=1.43

DI=1.81

M2bR9

Tetraploid

R4b

R9 R8R5

R4a

R1R3R2

M1M2b

M2a

VHL

KDM5C (missense and frameshift)mTOR (missense)

SETD2 (missense)KDM5C (splice site)

SETD2 (splice site)

?

SETD2 (frameshift)

PreP

PreM

Normal tissue

PrePPreMR1R2R3R5R8R9R4M1M2aM2b

C2o

rf85

WD

R7SU

PT6H

CD

H19

LAM

A3

DIX

DC

1H

PS5

NRA

PKI

AA

1524

SETD

2PL

CL1

BCL1

1AIF

NA

R1$

DA

MTS

10

C3

KIA

A12

67.

RT4

CD

44A

NKR

D26

TM7S

F4SL

C2A

1D

AC

H2

MM

AB

ZN

F521

HM

G20

AD

NM

T3A

RLF

MA

MLD

1M

AP3

K6H

DA

C6

PHF2

1BFA

M12

9BRP

S8C

IB2

RAB2

7ASL

C2A

12D

USP

12A

DA

MTS

L4N

AP1

L3U

SP51

KDM

5CSB

F1TO

M1

MYH

8W

DR2

4IT

IH5

AKA

P9FB

XO1

LIA

STN

IKSE

TD2

C3o

rf20

MR1

PIA

S3D

IO1

ERC

C5

KLALK

BH8

DA

PK1

DD

X58

SPA

TA21

ZN

F493

NG

EFD

IRA

S3LA

TS2

ITG

B3FL

NA

SATL

1KD

M5C

KDM

5CRB

FOX2

NPH

S1SO

X9C

ENPN

PSM

D7

RIM

BP2

GA

LNT1

1A

BHD

11U

GT2

A1

MTO

RPP

P6R2

ZN

F780

AW

SCD

2C

DKN

1BPP

FIA

1THSS

NA

1C

ASP

2PL

RG1

SETD

2C

CBL

2SE

SN2

MA

GEB

16N

LRP7

IGLO

N5

KLK4

WD

R62

KIA

A03

55C

YP4F

3A

KAP8

ZN

F519

DD

X52

ZC

3H18

TCF1

2N

USA

P172

X4KD

M2B

MRP

L51

C11

orf6

8A

NO

5EI

F4G

2M

SRB2

RALG

DS

EXT1

ZC

3HC

1PT

PRZ

1IN

TS1

CC

R6D

OPE

Y1A

TXN

1W

HSC

1C

LCN

2SS

R3KL

HL1

8SG

OL1

VHL

C2o

rf21

ALS

2CR1

2PL

B1FC

AM

RIF

I16

BCA

S2IL

12RB

2

PrivateUbiquitous Shared primary Shared metastasis

Private

Ubiquitous

Lungmetastases

Chest-wallmetastasis

Perinephricmetastasis

M110 cm

R7 (G4)

R5 (G4)

R9

R3 (G4)

R1 (G3) R2 (G3)

R4 (G1)

R6 (G1)H

ilum

R8 (G4)

Primarytumor

Shared primaryShared metastasis

M2b

M2a

Propidium Iodide Staining

No.

of C

ells

The New England Journal of Medicine Downloaded from nejm.org at UNIVERSITE DE GENEVE on June 2, 2014. For personal use only. No other uses without permission.

Copyright © 2012 Massachusetts Medical Society. All rights reserved.

Page 3: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

BACKGROUND – RADIOMICS - HISTOPATHOLOMICS

• Huge potential for computerized medical image analysis

• Create imaging biomarkers to predict diagnosis, prognosis, treatment response [2]

3

[2] Imaging and genomics: is there a synergy?, Jaffe et al., Radiology, 264(2):329-31, 2012 [3] Radiomics: the process and the challenges, Kumar et al., Magn Reson Imaging, 30(9):1234-48, 2012 [4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009

Radiomics [3] “Histopatholomics” [4]

Reuse existing diagnostic images radiology data digital pathology

Capture tissue heterogeneity

3D neighborhoods(e.g., 512x512x512)

large 2D regions(e.g., 15,000x15,000)

Analytic power beyond naked eyes

complex 3D tissue morphology

exhaustive characterization of 2D tissue structures

Non-invasive x

Page 4: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

BACKGROUND – RADIOMICS - HISTOPATHOLOMICS

• Huge potential for computerized medical image analysis

• Create imaging biomarkers to predict diagnosis, prognosis, treatment response

• Local quantitative image feature extraction

• Supervised machine learning

4

malignant, nonrespondermalignant, responder

benign

pre-malignant

undefined

quant. feat. #1

quan

t. fe

at. #

2

Supervised learning, big data

This could include an additional step of studying the spatial relationships between local image properties (e.g., using image graphs)

Page 5: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Shape, intensity, margin, texture, …

• Shape and margin features often require prior image segmentation

• 2D and 3D texture analysis can quantify micro- and macro- structures in biomedical images [4,6]

IMAGE FEATURES

5

GU

RC

AN

etal

.:H

IST

OPA

TH

OL

OG

ICA

LIM

AG

EA

NA

LYSI

S:A

RE

VIE

W15

5

Fig.

5.R

esul

tsof

the

auto

mat

icse

gmen

tatio

nal

gori

thm

(blu

eco

ntou

rs:l

umen

boun

dary

,bla

ckco

ntou

rs:

inne

rbo

unda

ryof

the

nucl

eiof

the

epith

elia

lce

llssu

rrou

ndin

gth

egl

and)

.Sho

wn

from

left

tori

ghta

reex

ampl

eim

ages

ofbe

nign

epith

eliu

m,i

nter

med

iate

-,an

dhi

gh-g

rade

canc

er.

and

are

2-D

Car

tesi

anco

ordi

nate

sof

.The

evol

utio

nof

isth

ende

scri

bed

bya

leve

l-se

tfor

mul

atio

nad

opte

dfr

om[7

8]

(4.1

)

whe

reth

efu

nctio

nde

fines

the

spee

dof

the

evol

utio

n.T

hecu

rve

evol

utio

nis

driv

enby

the

nucl

eilik

elih

ood

imag

e.T

hein

itial

cont

our

isin

itial

ized

auto

mat

ical

lyus

ing

the

dete

cted

lum

enar

eafr

omth

eca

ndid

ate

glan

dre

gion

s.T

hecu

rve

isev

olve

dou

twar

dfr

omth

ede

tect

edlu

men

regi

ons

inth

eco

mbi

ned

nucl

eilik

elih

ood

imag

eto

avoi

dno

ise

and

allo

wsm

ooth

erev

olut

ion

rela

tive

toth

eor

igin

alim

age.

The

inte

nsiti

esof

the

nucl

eilik

elih

ood

imag

efo

rmth

est

oppi

nggr

adie

nt.T

heal

gori

thm

isru

nun

tilth

edi

ffer

ence

inth

eco

ntou

rsin

two

con-

secu

tive

itera

tions

isbe

low

anem

piri

cally

dete

rmin

edth

resh

old.

Dur

ing

trai

ning

,siz

edi

stri

butio

nssi

mila

rto

thos

eus

edto

calc

u-la

teob

ject

likel

ihoo

dar

ecr

eate

dus

ing

the

final

cont

ours

.The

senu

clea

rbou

ndar

yba

sed

dist

ribu

tions

are

used

tore

mov

ere

gion

sth

atar

eto

ola

rge

tobe

true

glan

ds.F

inal

ly,t

helu

men

and

nu-

clea

rbou

ndar

iese

xtra

cted

from

true

glan

dre

gion

sare

pass

edon

toth

ene

xtst

epfo

rfe

atur

eex

trac

tion.

Sam

ple

resu

ltsfr

omth

eau

tom

ated

glan

dse

gmen

tatio

nal

gori

thm

are

show

nin

Fig.

5.T

helu

men

boun

dari

esar

edi

spla

yed

ina

solid

blue

cont

oura

ndth

ein

teri

ornu

clea

rbo

unda

ries

are

disp

laye

das

dash

edbl

ack

lines

.R

esul

tsof

our

glan

dse

gmen

tatio

nal

gori

thm

are

show

nfo

rsa

mpl

eim

ages

from

beni

gnep

ithel

ium

,in

term

edia

te,

and

high

-gra

depr

osta

teca

ncer

(fro

mle

ftto

righ

t).

B.

Glo

balS

cene

Segm

enta

tion

App

roac

hes

In[2

6],

aun

ified

segm

enta

tion

algo

rith

mfo

rsu

b-ce

l-lu

lar

com

part

men

taliz

atio

nw

aspr

esen

ted.

Qua

ntita

tion

ofbi

omar

kers

atsu

b-ce

llula

rre

solu

tion

requ

ires

segm

enta

tion

ofsu

b-ce

llula

rco

mpa

rtm

ents

such

asnu

clei

,mem

bran

es,a

ndcy

topl

asm

.W

hile

diff

eren

tse

gmen

tatio

nal

gori

thm

sca

nbe

used

for

each

ofth

esu

b-ce

llula

rco

mpa

rtm

ents

,an

alte

rnat

ive

isto

use

the

sam

eal

gori

thm

indi

ffer

entm

odes

.The

algo

rith

min

[26]

capt

ured

ase

tof

brig

htpi

xels

shar

ing

aco

mm

onsh

ape

dist

ribu

tion.

The

algo

rith

mus

eda

set

ofth

ree

feat

ures

,one

isth

eflu

ores

cent

emis

sion

inte

nsity

,and

the

othe

rtw

oar

eba

sed

oncu

rvat

ure

desc

ript

orst

hata

reco

mpu

ted

from

the

eige

nval

ues

ofth

eH

essi

anm

atri

x.Fo

ran

imag

e,,t

heei

genv

alue

s(

)of

the

Hes

sian

mat

rix

enco

deth

ecu

rvat

ure

info

rmat

ion

ofth

eim

age,

and

prov

ide

usef

ulcu

esfo

rde

tect

ing

ridg

e-lik

em

embr

ane

stru

ctur

es,o

rbl

ob-l

ike

nucl

eist

ruct

ures

.How

ever

,th

eei

genv

alue

sar

ede

pend

ent

onim

age

brig

htne

ss.

The

fol-

low

ing

two

curv

atur

e-ba

sed

feat

ures

are

inde

pend

ento

fim

age

brig

htne

ss:

(4.2

)

(4.3

)

and

refe

rred

toas

shap

ein

dex,

and

norm

aliz

ed-c

urva

ture

inde

x,re

spec

tivel

y.T

his

ises

sent

ially

the

sam

eas

defin

ing

the

eige

n-va

lues

ina

pola

rco

ordi

nate

syst

em.

Thi

str

ansf

orm

atio

nal

sore

sults

inbo

unde

dfe

atur

es,

,an

d.

The

estim

atio

npr

oces

sst

arts

with

the

expe

cted

dist

ribu

tions

ofth

esh

ape

inde

xfo

rth

est

ruct

ures

tobe

segm

ente

d.Fo

rex

-am

ple,

forb

righ

tmem

bran

ean

dve

ssel

like

stru

ctur

esth

esh

ape

inde

xis

clos

eto

,bec

ause

the

smal

lere

igen

valu

eis

nega

-tiv

ean

dth

ela

rger

eige

nval

ueap

proa

ches

toze

ro.O

nth

eot

her

hand

,for

the

blob

-lik

enu

clei

stru

ctur

es,t

hesh

ape

inde

xis

clos

eto

,bec

ause

both

eige

nval

ues

are

nega

tive

and

clos

ein

valu

e.Fo

rbot

hst

ruct

ures

,pos

itive

valu

esin

dica

tea

pixe

lbei

ngm

ore

like

aba

ckgr

ound

.The

seco

nstr

aint

sar

eus

edto

com

pute

the

initi

alfo

regr

ound

and

back

grou

ndse

tsfo

rm

embr

ane

and

nucl

eist

ruct

ures

.A

nin

itial

segm

enta

tion

base

don

the

shap

ein

dex

and

the

norm

aliz

ed-c

urva

ture

inde

xse

para

tes

the

imag

epi

xels

into

thre

esu

bset

s:ba

ckgr

ound

,for

egro

und,

and

inde

ter-

min

ate.

The

inde

term

inat

esu

bset

com

pris

esal

lthe

pixe

lsth

atar

eno

tinc

lude

din

the

back

grou

ndor

fore

grou

ndsu

bset

s.Fr

omth

ese

subs

ets,

the

back

grou

ndan

dfo

regr

ound

inte

nsity

dist

ri-

butio

ns,a

sw

ella

sth

ein

tens

itylo

g-lik

elih

ood

func

tions

are

es-

timat

ed.T

heal

gori

thm

keep

site

ratin

gby

usin

gtw

oou

tof

the

thre

efe

atur

esat

atim

eto

estim

ate

the

dist

ribu

tion

ofth

efe

atur

eth

atis

left

out.

Inth

efin

alst

ep,t

hese

log-

likel

ihoo

dfu

nctio

nsar

eco

mbi

ned

tode

term

ine

the

over

all

likel

ihoo

dfu

nctio

n.A

prob

abili

tym

apth

atre

pres

ents

the

prob

abili

tyof

api

xelb

eing

afo

regr

ound

isca

lcul

ated

.C

ytop

lasm

can

bede

tect

edei

ther

byus

ing

asp

ecifi

ccy

-to

plas

mic

mar

ker,

orca

nbe

dete

cted

usin

gco

mpu

tatio

nal

met

hods

usin

gth

efa

ctth

atth

ecy

topl

asm

icar

eas

are

betw

een

nucl

ear

and

mem

bran

ear

eas.

For

mos

tca

ncer

tissu

ety

pes,

itis

very

impo

rtan

tto

diff

eren

tiate

the

epith

elia

ltis

sue

from

the

stro

mal

and

conn

ectiv

etis

sue,

soth

atfo

rIH

Cst

udie

sth

eex

pres

sion

leve

lsof

mos

tm

arke

rsin

the

epith

elia

lre

gion

sca

nbe

quan

tified

.C

ompu

tatio

nal

met

hods

that

use

the

high

conn

ectiv

ityof

mem

bran

em

eshe

sca

nbe

used

todi

ffer

entia

teth

eep

ithel

ial

regi

ons.

For

the

sam

ple

imag

es,

any

conn

ecte

dco

mpo

nent

larg

erth

an80

0pi

xels

isac

cept

edas

apa

rtof

the

epith

elia

lmas

k.T

henu

clei

seti

sth

ense

para

ted

into

epith

elia

lnu

clei

and

stro

mal

nucl

eius

ing

the

epith

elia

lmas

k.

[4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009 [5] Quantifying the margin sharpness of lesions on radiological images for content-based image retrieval, Xu et al., Med Phys, 39(9):5405-18, 2012 [6] Three-dimensional solid texture analysis in biomedical imaging: review and opportunities, Depeursinge et al., Med Image Anal, 18(1):176-96, 2014 [7] Prediction of prognosis for prostatic adenocarcinoma by combined histological grading and clinical staging, Gleason et al., J Urol, 111(1):58-64, 1974 [8] HyMaP: A hybrid magnitude-phase approach to unsupervised segmentation of tumor areas in breast cancer histology images, Khan et al., J Pathol Inform, 4, 2013

[4]

5407Xuetal.:Quantifyingthemarginsharpnessoflesions5407

(a)(b)

Xo-4Xo-2XoXo+2Xo+4100

200

300

400

500

600

Sigmoid FitIntensity Values

FIG.2.(a)CroppedCTimagecontainingalungnodulewithanormallineandROIdrawn.(b)Pixelintensityvaluesalongthenormalline(solidline)andfittedsigmoid(dashedline)showingthescaleandwindowparameters.

byradiologists,butcouldalsobegeneratedautomaticallybyaCADsystemorautomatedsegmentationalgorithm.Togener-atethemarginsharpnessfeature,wefirstrepresenttheROIbyapiecewisecubicpolynomialcurve(interpolatedfrom10–20controlpointsdrawnbyaradiologist).WenextdrawnormallinesegmentsoflengthTpixelsacrosstheboundaryofthelesionatfixedintervalsarounditscircumference(thefixedintervalisthelargervalueof1pixelandthelengthoftheboundaryinpixelsdividedby200).Intensityvaluesarethenrecordedalongthesesegmentsusingbilinearinterpolation.Wenextfitasigmoidfunction(Fig.2)tothesevaluesusingarobustweightednonlinearregression.24ForeachlineI,theproblemcanbeformulatedas

argminS,W,xo,Io

!

x

"I(x)−Io−S

1+e−x−xoW

#,

wherexisthedistancealongthenormal,xoistheintersectionoftheboundarypointwiththenormal,I(x)istheintensityalongthenormalatx,andIoistheintensityoffset.

Twoparameters,scale(S)andwindow(W)fromthesig-moidarethenusedtocharacterizeeachlinesegmentI.Thescalemeasuresthedifferenceinintensitiesoutsideandinsidethelesion,andthewindowcharacterizesthemarginblurbymeasuringthetransitionfromthelesiontosurroundingnor-malorgantissueattheboundary.WesettherangeofWtobe

−2*Tto2*TandtherangeofStobe−1000HUto1000HU.Iffittedparametersareoutoftherange,wedonotincludetheresultintothefeaturevector.

II.A.2.Marginsharpnessfeatureimplementation

Themarginsharpnessofeachlesionisrepresentedasafeaturevectorcomposedof[two]30-binhistogramsofthescaleandwindowparametersobtainedfromeachnormal.Weremoveoutliersbydiscardingvaluesbelowthe5thpercentileandabovethe95thpercentileforeachparameterbeforecon-structingthehistogram.Eachhistogramisnormalizedtohaveunitarea.Thefeaturevectoristheconcatenationofthetwohistograms.

Itshouldbenotedthatincreatingthiscomputationalrep-resentationofthelesionboundary,itisassumedthateachnor-mallinesegmentincludesbothlesionsubstanceandnormalorgantissuesurroundingthelesion.However,forlesionsattheedgeofanorgan,aportionofthecircumferenceofthele-sionwillbeadjacenttosomestructureotherthanthenormalsurroundingorgan.Figure3(a)showssuchanexample.Duetothepossiblechangesinintensitybetweentheorgananditssurroundingtissue,thelinesegmentsdrawnontheborderoftheorgandonothaveenoughinformationtocharacterizethemarginofthelesion.Includingthescaleandwindowparam-etersobtainedfromthelinesegmentsfromthisportionofthe

(a)(b)(c)(d)

FIG.3.(a)CroppedlungCTimagedisplayedatstandardlungwindow-levelsetting,withROIoutlined(indarkgray)byradiologist.(b)Automaticallysegmentedlungregion(ingray)showingfillingdefectcausedbyinadequatesegmentationnearthelesionandtheROIoutline(inwhite).(c)Finallungsegmentationmask(theunionoftheautomaticallysegmentedlungmaskandthelesionmask).(d)CroppedlungCTimagewithlinesegments(inwhite)thatarecompletelyinsidethelungmask.

MedicalPhysics,Vol.39,No.9,September2012

[5]

J Pathol Inform 2013, 1:1 http://www.jpathinformatics.org/content/4/1/1

G x y g x y j f xcos ysinfθ σ π θ θ, ( , ) ( , )exp( ( ))= +2 (1)

where gσ(x,y) is a Gaussian kernel with a bandwidth of σ. The parameters f and θ represent frequency and orientation of the 2D Gabor filter, where θ varies between 0 and π in regular intervals, f ∈ F, and F denotes a set of possible frequencies, and is defined as follows.

F i N F iLi

C L( ) . / | ( ) ..= − < <−0 25 2 0 0 250 5 (2)

F i N F iHi

C H( ) . / | . ( ) ..= + < <−0 25 2 0 25 0 50 5 (3)

Where i Ncol= … ( )0 1 82, , , log / , Ncol is the width of the image in terms of the nearest power of 2. We then define the set F of possible frequencies as follows,

F F FL H= ∪ (4)

For an image with 512 columns, for example, a total of 84 Gabor filters can be used (six orientations and 14 frequencies). The hypocellular stromal features are then computed by convolving the Gabor filters Gθ,f(·) with Inorm

β (obtained from step 3 of Algorithm 1), and computing local energy on the results of the convolution.

Hypercellular Stromal FeaturesPhase information could be used as an important cue in modeling the textural properties of a region. Murtaza et al., used local frequency estimates in the Gabor domain over a range of scales and orientations to yield a signature, which was shown to efficiently characterize the texture of a village in satellite images.[16] We chose

the phase spectrum to represent the attributes of the HyperCS regions in a breast histology image, due to the recently established efficacy of the phase in textures exhibiting randomness.

Let vi(x,y) denote the ith Gabor channel for the stain normalized and smoothened version of an input image I(x,y), where i = ,1,2,...,Ng, N N Fg = ×θ | | and Nθ denotes the number of orientations. We can represent it as follows,

v x y v x y j x yi i i( , ) ( , ) exp( ( , ))= φ (5)

where |·| denotes the magnitude operator and φi x y( , ) denotes the local phase. The gradient of the local phase and its magnitude can then be computed as below,

φii

i

i

i

x yv x yv x y

v x yv x y

′′ ′

= −⎡

⎣⎢⎢

⎦⎥⎥

( , )( , )( , )

( , )( , )

(6)

and

φ φ φi

i ix yddx

ddy

′ = +( , )2 2

(7)

The phase gradient features are computed using (7) for each of the Gabor filter responses, over a window of size N × N(where N=15).

RanPEC SegmentationRanPEC is a fast, unsupervised, and data-independent framework for dimensionality reduction and clustering of high-dimensional data points.[9] The main idea of RanPEC is to project high-dimensional feature vectors onto a relatively small number of orthogonal random vectors belonging to a unit ball and perform ensemble clustering in the reduced-dimensional feature space. By getting an ensemble of projections for each feature vector and then picking a cluster for a pixel by the majority voting selection criterion, ensures stability of the results among different runs. Experimental results in[9] suggest that promising classification accuracy can be achieved by random projections when using fast matrix operations in an unsupervised manner.

RESULTS

All 35 images in the database were hand segmented by two expert pathologists. We generate all experimental results on three criteria: (1) Considering the first pathologist’s markings (P-1) as the ground truth (GT); (2) considering the second pathologist’s markings (P-2) as GT; (3) fusing P-1 and P-2 using the logical OR rule (i.e., a pixel is considered to be tumorous if any one of the two pathologists marked the pixel as tumorous), and considering the fused image as GT. Some of the HPF images contain large tumor regions with small islands of stroma here and there; however, a majority of HPF images contain a fair share of hypo- and hypercellular stroma (approximately 33%, on an average). The average

Figure 2: Overview of the proposed algorithm: HyMaP

Figure 1: A sample H & E–stained breast cancer histology image: (a) Original image, and (b) Overlaid image, with four types of contents shown in different colors. The tumor areas are shown in Red, HypoCS in Purple, and HyperCS in Green. Areas containing background or fat tissue are shown in white with black outline. Note the difference in morphology of the Hypo- and Hypercellular stromal regions

ba

[Downloaded free from http://www.jpathinformatics.org on Tuesday, June 16, 2015, IP: 128.179.146.236]

[7] [8]

Page 6: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

COMPUTERIZED TEXTURE ANALYSIS

directionsscale

6

• Image scales and directions are important for visual texture discrimination

• Most approaches are leveraging these two properties

• Explicitly: Gray-level co-occurrence matrices (GLCMs), run-length matrices (RLE), directional filterbanks and wavelets, Fourier, histograms of gradients (HOG), local binary patterns (LBP)

• Implicitly: Convolutional neural networks (CNN), scattering transform, topographic independant component analysis (TICA)

Page 7: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

COMPUTERIZED TEXTURE ANALYSIS

7

• Texture invariances: computer vision VS biomedical imaging

Computer vision Biomedical image analysis

scale scale-invariant multi-scale

rotation rotation-invariant rotation-invariant

[4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009 [9] A sparse texture representation using local affine regions, Lazebnik et al., IEEE Trans on Pattern Anal and Mach Intel, 27(8):1265-78, 2005

160 IEEE REVIEWS IN BIOMEDICAL ENGINEERING, VOL. 2, 2009

Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based representations of tissue architecture via Delaunay Triangulation, VoronoiDiagram, and Minimum Spanning tree.

Fig. 11. Digitized histological image at successively higher scales (magnifica-tions) yields incrementally more discriminatory information in order to detectsuspicious regions.

or resolution. For instance at low or coarse scales color or tex-ture cues are commonly used and at medium scales architec-tural arrangement of individual histological structures (glandsand nuclei) start to become resolvable. It is only at higher res-olutions that morphology of specific histological structures canbe discerned.

In [93], [94], a multiresolution approach has been used for theclassification of high-resolution whole-slide histopathology im-ages. The proposed multiresolution approach mimics the eval-uation of a pathologist such that image analysis starts from thelowest resolution, which corresponds to the lower magnificationlevels in a microscope and uses the higher resolution represen-tations for the regions requiring more detailed information fora classification decision. To achieve this, images were decom-posed into multiresolution representations using the Gaussianpyramid approach [95]. This is followed by color space con-version and feature construction followed by feature extractionand feature selection at each resolution level. Once the classifieris confident enough at a particular resolution level, the systemassigns a classification label (e.g., stroma-rich, stroma-poor orundifferentiated, poorly differentiating, differentiating) to theimage tile. The resulting classification map from all image tilesforms the final classification map. The classification of a whole-slide image is achieved by dividing into smaller image tiles andprocessing each image tile independently in parallel on a clusterof computer nodes.

As an example, refer to Fig. 11, showing a hierarchicalcascaded scheme for detecting suspicious areas on digitizedprostate histopathology slides as presented in [96].

Fig. 12 shows the results of a hierarchical classifier for detec-tion of prostate cancer from digitized histopathology. Fig. 12(a)

Fig. 12. Results from the hierarchical machine learning classifier. (a) Originalimage with the tumor region (ground truth) in black contour, (b) results at scale1, (c) results at scale 2, and (d) results at scale 3. Note that only areas determinedas suspicious at lower scales are considered for further analysis at higher scales.

shows the original image with tumor outlined in black. The nextthree columns show the classifier results at increasing analysisscales. Pixels classified as “nontumor” at a lower magnification(scale) are discarded at the subsequent higher scale, reducingthe number of pixels needed for analysis at higher scales. Ad-ditionally, the presence of more discriminating information athigher scales allows the classifier to better distinguish betweentumor and nontumor pixels.

At lower resolutions of histological imagery, textural analysisis commonly used to capture tissue architecture, i.e., the overallpattern of glands, stroma and organ organization. For each digi-tized histological image several hundred corresponding featurescenes can be generated. Texture feature values are assignedto every pixel in the corresponding image. 3-D statistical, gra-dient, and Gabor filters can be extracted in order to analyzethe scale, orientation, and anisotropic information of the re-gion of interest. Filter operators are applied in order to extractfeatures within local neighborhoods centered at every spatiallocation. At medium resolution, architectural arrangement ofnuclei within each cancer grade can be described via severalgraph-based algorithms. At higher resolutions, nuclei and themargin and boundary appearance of ductal and glandular struc-tures have proved to be of discriminatory importance. Many ofthese features are summarized in Tables I and II.

D. Feature Selection, Dimensionality Reduction,and Manifold Learning

1) Feature Selection: While humans have innate abilities toprocess and understand imagery, they do not tend to excel at

COMPUTERIZED TEXTURE ANALYSIS

7

• Invariances: computer vision versus biomedical imaging

Computer vision Biomedical image analysis

scale scale-invariant multi-scale

rotation rotation-invariant rotation-invariant

[4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009

160 IEEE REVIEWS IN BIOMEDICAL ENGINEERING, VOL. 2, 2009

Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based representations of tissue architecture via Delaunay Triangulation, VoronoiDiagram, and Minimum Spanning tree.

Fig. 11. Digitized histological image at successively higher scales (magnifica-tions) yields incrementally more discriminatory information in order to detectsuspicious regions.

or resolution. For instance at low or coarse scales color or tex-ture cues are commonly used and at medium scales architec-tural arrangement of individual histological structures (glandsand nuclei) start to become resolvable. It is only at higher res-olutions that morphology of specific histological structures canbe discerned.

In [93], [94], a multiresolution approach has been used for theclassification of high-resolution whole-slide histopathology im-ages. The proposed multiresolution approach mimics the eval-uation of a pathologist such that image analysis starts from thelowest resolution, which corresponds to the lower magnificationlevels in a microscope and uses the higher resolution represen-tations for the regions requiring more detailed information fora classification decision. To achieve this, images were decom-posed into multiresolution representations using the Gaussianpyramid approach [95]. This is followed by color space con-version and feature construction followed by feature extractionand feature selection at each resolution level. Once the classifieris confident enough at a particular resolution level, the systemassigns a classification label (e.g., stroma-rich, stroma-poor orundifferentiated, poorly differentiating, differentiating) to theimage tile. The resulting classification map from all image tilesforms the final classification map. The classification of a whole-slide image is achieved by dividing into smaller image tiles andprocessing each image tile independently in parallel on a clusterof computer nodes.

As an example, refer to Fig. 11, showing a hierarchicalcascaded scheme for detecting suspicious areas on digitizedprostate histopathology slides as presented in [96].

Fig. 12 shows the results of a hierarchical classifier for detec-tion of prostate cancer from digitized histopathology. Fig. 12(a)

Fig. 12. Results from the hierarchical machine learning classifier. (a) Originalimage with the tumor region (ground truth) in black contour, (b) results at scale1, (c) results at scale 2, and (d) results at scale 3. Note that only areas determinedas suspicious at lower scales are considered for further analysis at higher scales.

shows the original image with tumor outlined in black. The nextthree columns show the classifier results at increasing analysisscales. Pixels classified as “nontumor” at a lower magnification(scale) are discarded at the subsequent higher scale, reducingthe number of pixels needed for analysis at higher scales. Ad-ditionally, the presence of more discriminating information athigher scales allows the classifier to better distinguish betweentumor and nontumor pixels.

At lower resolutions of histological imagery, textural analysisis commonly used to capture tissue architecture, i.e., the overallpattern of glands, stroma and organ organization. For each digi-tized histological image several hundred corresponding featurescenes can be generated. Texture feature values are assignedto every pixel in the corresponding image. 3-D statistical, gra-dient, and Gabor filters can be extracted in order to analyzethe scale, orientation, and anisotropic information of the re-gion of interest. Filter operators are applied in order to extractfeatures within local neighborhoods centered at every spatiallocation. At medium resolution, architectural arrangement ofnuclei within each cancer grade can be described via severalgraph-based algorithms. At higher resolutions, nuclei and themargin and boundary appearance of ductal and glandular struc-tures have proved to be of discriminatory importance. Many ofthese features are summarized in Tables I and II.

D. Feature Selection, Dimensionality Reduction,and Manifold Learning

1) Feature Selection: While humans have innate abilities toprocess and understand imagery, they do not tend to excel at

[4][9]

Page 8: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Averaging texture properties over the entire lesions or images discards tissue heterogeneity [10]

• Exhaustive and non-specific features

• Limited characterization of directions [6]

BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (1/4)

8

[6] Three-dimensional solid texture analysis in biomedical imaging: review and opportunities, Depeursinge et al., Med Image Anal, 18(1):176-96, 2014 [10] Quantitative imaging in cancer evolution and ecology, Gatenby et al., Radiology, 269(1):8-15, 2013

feartures are not specific when not learned from data

Page 9: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• A global characterization of directions is not enough [11]

• local organization of image directions:

• independently from their local orientation:

• Rotation-covariance

• (local) grouped steering of the operators:

BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (2/4)

image operators: grouped steering:

9

[11] Rotation–covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014

Page 10: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (3/4)

GLCM contrast [1 0]

GLC

M c

ontra

st [0

1]

GLCMs (unaligned) Riesz (unaligned) Riesz (aligned)

10

Page 11: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (4/4)

GLCM contrast [1 0]

GLC

M c

ontra

st [0

1]

GLCMs (unaligned) Riesz (unaligned) Riesz (aligned)

11

Page 12: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

OBJECTIVES

12

• Highly adaptive texture-based computational models of biomedical tissue:

Complete coverage of image scales and directions in 2-D and 3-D

Rotation-covariance

Specificity: the models can be trained to characterize specific tissue types

Local characterization of tissue properties

Locate tissue properties in organ anatomy to create digital phenotypes

• Goal: predict survival, function, treatment response and reveal subtypes

be averaged over the folds of the CV and used to build texture models for each location along dROI (see Fig 9).The sum of the weights for all channels from each location will reveal the subregions that are specific to eachtumor subtypes, and most related to patient survival. A K–means clustering of the vectors w will be carried outfor each locations to evaluate the stability of the regional models over the folds of the CV and define homogeneousgroups among patients. A selection of the models based on stability and location importance will be carried out.

Fig. 9 Prototype tissue archi-tecture of a GBM tumor [128].

In a second step, the selected models will be locally steered to maximize theirmagnitude. The energies of the maximal magnitudes will be used to constructa final feature space for (1) predicting the tumor subtypes and (2) performingKaplan–Meier survival analysis. The performance of the proposed approach forpredicting tumor subtypes and patient survival will be compared to (1) unalignedwavelet energies and (2) average wavelet energies over the entire tumor. Whilestarting with a LOPO CV on the TCGA–TCIA dataset, the generalizability of theapproach will be further assessed by training with the TCGA–TCIA and testingwith the SU dataset.

Deliverable 2.1: Predicting tumor subtype and survival from localized of tissueproperties in GBM tumors.Task 2.2: Digital lung tissue atlases of ILD diagnoses (8 months of the PI)In this task, we will use a simple atlas of the lungs to locate texture properties and create prototype diagnosisphenotypes of ILDs. In previous work, we developed a simple 3–D atlas of the lungs with 36 subregionsthat will be used in this task [42]. In a first step, diagnosis–wise digital tissue atlases will be created bylearning 3–D texture models (i.e., the average of w over the folds of the LOPO CV) for each 36 regions ofthe lungs (see Fig. 10). The regions for which the models are most distant11 from all other diagnoses willbe highlighted and compared to previously built models of tissue patterns [25] to create 3–D prototype tissueatlases for each diagnosis. The obtained results will be validated using medical knowledge (e.g., Table 1 of [42]).

Fig. 10 Tissue atlas of the lungs.

For each diagnosis, K–means clustering of the vectors w will be carried outfor each locations to evaluate the stability of the regional models over the foldsof the CV and reveal homogeneous groups among patients (e.g., subtypes ofUIP). The most stable models will be kept for the further characterization oflung tissue types. A hierarchical clustering of the models from all diagnoseswill be carried out to define a radiomics–based hierarchy of all diagnoses, whichwill be compared to medical knowledge [140] (e.g., Fig. 1 of [40]). A largefeature space including the energies of the steered models from each of the 36localizations will be used to predict the diagnoses with uncertainty assessment(using e.g., pairwise coupling [141]). When a minimum amount of trust is notachieved when predicting a given diagnosis, the parent group of ILD diagnosesin the previously built hierarchy will be predicted instead [19].

Deliverable 2.2: Digital tissue atlases of ILD diagnoses and their subtypes.Task 2.3: Digital tissue atlases of ILDs: correlation with PFTs and survival (6 months )Digital tissue atlases will be constructed for poor versus normal/high (1) PFTs and (2) survival. Regions forwhich the models significantly di↵er between poor versus normal/high will be revealed as being of primaryimportance to evaluate pulmonary function. The links between these models and previously built models oftissue patterns will be investigated to define the combination of regions and patterns that are most responsiblefor lung function impairment. The feature space spanned by the energies of the steered models will be used topredict (1) PFT values or (2) survival with a LOPO CV, which can be evaluated using ground truth.

Deliverable 2.3: Estimating pulmonary function from digital tissue atlases of ILD diagnoses in CT images.

5.3 WP3. Imaging genomics

In this WP, we will use the texture models and digital tissue atlases developed in WP2 to learn and locatethe radiological patterns associated with prevailing meta–genes. This will allow for a deeper comprehension ofthe links between the radiological phenotypes and the expression of meta–genes. The molecular heterogeneityof GBM tumors ILD diagnoses will be studied using 3–D patch–based image analysis to enable the localcharacterization of radiogenomic properties.Task 3.1: Texture–based imaging genomics of GBM tumors (6 months of the PI)Based on the hypothesis that every meta–gene is associated with a specific visual pattern inside GBM tumorsin MR T1 images, patch–based analysis of the GBM tumors will be used to reveal local image regions thatcorrelate most with a given driver meta–gene. The gene expression is known to be di↵erent in perilesionaledema, on the tumor margin and in central necrosis [61, 130]. Gene expression patterns are also defining the

11in terms of Euclidean distance

10

8

(a) Original Image (b) Simple Cell Graph

(c) Voronoi Diagram (d) Delaunay Triangulation

(e) ECM-Aware Cell-Graph

Fig. 2 A fractured bone tissue example is shown in 2(a). Note the fracture cells in themiddle of the original image. The simple-cell-graph representation, the Voronoi diagramand the Delaunay triangulation for this sample tissue are depicted in 2(b), 2(c) and 2(d).The corresponding ECM-aware cell-graph is drawn in 2(e). The interactions between fracturecells are drawn with blue and the red cells with red color. Delaunay triangulation representsthe tissue as a single connected component and does not allow crossing of edges. Simple-cell-graphs relaxes these restrictions and allows the tissue to non-planar graph and disconnected.Likewise, ECM-aware cell-graphs do not put such restrictions on the tissue and moreovercan capture the structural organization of different cells in a tissue. Furthermore, Delaunaytriangulations are fixed representations whereas ECM-aware cell-graphs can be adjustedwith different linking thresholds.

8

(a) Original Image (b) Simple Cell Graph

(c) Voronoi Diagram (d) Delaunay Triangulation

(e) ECM-Aware Cell-Graph

Fig. 2 A fractured bone tissue example is shown in 2(a). Note the fracture cells in themiddle of the original image. The simple-cell-graph representation, the Voronoi diagramand the Delaunay triangulation for this sample tissue are depicted in 2(b), 2(c) and 2(d).The corresponding ECM-aware cell-graph is drawn in 2(e). The interactions between fracturecells are drawn with blue and the red cells with red color. Delaunay triangulation representsthe tissue as a single connected component and does not allow crossing of edges. Simple-cell-graphs relaxes these restrictions and allows the tissue to non-planar graph and disconnected.Likewise, ECM-aware cell-graphs do not put such restrictions on the tissue and moreovercan capture the structural organization of different cells in a tissue. Furthermore, Delaunaytriangulations are fixed representations whereas ECM-aware cell-graphs can be adjustedwith different linking thresholds.[12] Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high-

resolution CT, Depeursinge et al., Invest Radiol, 50(4):261-67, 2015 [13] ECM-Aware Cell-Graph Mining for Bone Tissue Modeling and Classification, Bilgin et al., Data Min Knowl Discov, 20(3):416-38, 2009

[12] [13]

Page 13: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Steerable texture models

Σassociated texture

model

EXAMPLE: PATTERN = WIGGLED CHECKERBOARD

pattern to learn:

30

Steerable texture models: our workhorse ….

13

Page 14: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Multi-directional, multi-scale and rotation-covariant image analysis is achieved using Riesz wavelets

• The components of the th-order Riesz transform of a 2-D signal are defined in the Fourier domain as [14]:

• Yields allpass filters: only phase (i.e., direction) is kept and is defined by th-order partial derivatives

THE RIESZ TRANSFORM

14

[14] Wavelet Steerability and the Higher-Order Riesz Transform, Unser et al., IEEE Trans on Imag Proc, 19(3):636-52, 2010

Page 15: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Riesz components

THE RIESZ TRANSFORM

input image

15

Page 16: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Riesz components

THE RIESZ TRANSFORM

16

Page 17: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Riesz components

THE RIESZ TRANSFORM

17

Page 18: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Multi-scale filterbanks are obtained by combining the Riesz transform with isotropic wavelets [15]

• E.g., dyadic ( )

• Systematic coverage of image scales

THE RIESZ TRANSFORM

18

UNSER et al.: STEERABLE PYRAMIDS AND TIGHT WAVELET FRAMES IN 2711

of unity) guarantees that is dense in ; it is

essential for the -completeness of the wavelet decomposition.

We now proceed with the construction of orienta-

tion-free wavelets by projecting the multiresolution Riesz

basis onto some appropriate wavelet subspace

.

B. Isotropic Bandlimited Wavelet FramesThere are a number of constructions in the literature that

fall into this category [13], [27]–[29]. Before reviewing them,

we apply the aforementioned projection strategy to obtain a

straightforward design that is in direct correspondence with

Shannon’s sampling theorem, and that is the starting point for

the specification of Meyer-type wavelets.1) Construction of Isotropic, Shannon-Type Wavelets: By

selecting in (19),

we specify the so-called Shannon multiresolution analysis of

, which consists of a sequence of embedded subspaces

that are bandlimited to

We then define some corresponding wavelet subspaces of radi-ally bandpass functions

(20)

Since is a closed subspace of , we can apply Proposition

3 to its orthogonal sinc basis to obtain the tight wavelet frame

of with

(21)

where is the impulse response of the

ideal radial bandpass filter, whose frequency response is, i.e., the indicator function

corresponding to the spectral support of . Based on the

fact that is dense in , the design procedure

yields a tight wavelet frame of . The energy-preserving

condition that ensures that the latter is true is as follows:

(22)

It is automatically fulfilled, since the sequence of ideal radialbandpass filters constitutes a tilling of the frequency domain ,

as illustrated in Fig. 1. For a complete characterization of these

wavelets, we refer to the work of Papadakis et al. [27].2) Specification of Meyer-Type Wavelets: While the afore-

mentioned construction yields a tight isotropic wavelet frame

of , it has the drawback of producing wavelets with

poor spatial decay (e.g., ), due to the

sharp cutoff in frequency domain. A remarkable observation,

which can be traced back to the early work of Daubechies and

Meyer on frames [37], [38], is that this can corrected via an

appropriate adjustment of the radial bandpass filtering functions, which need not be indicator functions, as long as they

Fig. 1. Tiling of the 2-D frequency domain using radial-bandpass filters. Theshaded area corresponds to the spectral support of the wavelet subspace ;

it is included in the spectral support of (enclosing square).

satisfy (22). This leads to the following extended definition ofthe wavelet subspaces

which is equivalent to (20) if is the impulse response of

the ideal radial bandpass filter. Since can be written as

, there exists a sequence such that

where the wavelet functions are still given by (21). This

indicates that is a frame of , albeit not neces-

sarily a tight one. Yet, if Condition (22) is satisfied, then onerecovers the tight frame property over which is the

union of the wavelet subspaces , . The condition for

the wavelet frame to be isotropic is that the restriction of the fil-tering function over be isotropic, i.e.,

.

The aforementioned functional framework accounts for all

known constructions of isotropic wavelet frames of

which are summarized in Table I. The common feature of the

wavelet profile functions in Table I is that they are compactlysupported within the frequency interval (bandlimited

property) and that they satisfy a rescaled version of the partition

of unity condition

which is the key to ensuring the tight frame property [38]. Two

remarks are in order with respect to Simoncelli’s pioneering

design which is, by far, the solution most widely used in ap-

plications. First, the response of the filter is a warped versionof a bump [cf., Fig. 2(a)] and the furthest away from an ideal

bandpass filter; this simply reflects the fact that a design objec-tive for the steerable pyramid was to approximate the behavior

of a log-Gabor filterbank, which is a well-accepted model ofthe response properties of cortical cells in the mammalian vi-

sual system [39]. Second, Simoncelli and coworkers put a lot of

emphasis on finite-impulse response filter design in their initial

[15] Steerable pyramids and tight wavelet frames in , Unser et al., IEEE Trans on Imag Proc, 20(10):2705-21, 2011

Page 19: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

THE RIESZ TRANSFORM

19

• Steerability [16]:

• Example :

• Higher-order steering can be done fully analytical

Complete coverage of the image directions

Steerability enables rotation-covariance[16] The design and use of steerable filters, Adelson et al., IEEE Trans on Pattern Anal and Mach Intel, 13(9):891-906, 1991

Page 20: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• A Riesz filterbank constitutes a dictionary of basic textures:

• Higher-level texture models are built from linear combinations of Riesz components

LEARNING TEXTURE MODELS

20

Page 21: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Σ

EXAMPLE: PATTERN = WIGGLED CHECKERBOARD

pattern to learn:

Page 22: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Σ

EXAMPLE: PATTERN = WIGGLED CHECKERBOARD

pattern to learn:

Page 23: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Σ

EXAMPLE: PATTERN = WIGGLED CHECKERBOARD

pattern to learn:

Page 24: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Σ

EXAMPLE: PATTERN = WIGGLED CHECKERBOARD

pattern to learn:

Page 25: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Σ

EXAMPLE: PATTERN = WIGGLED CHECKERBOARD

pattern to learn:

Page 26: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Σ

EXAMPLE: PATTERN = WIGGLED CHECKERBOARD

pattern to learn:

Page 27: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Σ

EXAMPLE: PATTERN = WIGGLED CHECKERBOARD

pattern to learn:

Page 28: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Σassociated texture

model

EXAMPLE: PATTERN = WIGGLED CHECKERBOARD

pattern to learn:

28

Page 29: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Support vector machines (SVM) were used to learn (multi-scale )

versus

ONE-VERSUS-ALL SUPERVISED MODEL LEARNING [11]:

texture

all others

[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014

29

Page 30: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• 2-D synthetic textures with noise ( )

Captures specific local properties in terms of image scales and directions

LEARNING TEXTURE MODELS

30

Page 31: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• 2-D tissue from interstitial lung diseases (ILD) in CT [17]

• ( , 32x32 patches)

LEARNING TEXTURE MODELS

healthy emphysema ground glass fibrosis micronodules

3011 patches, 7 patients.

407 patches, 6 patients.

2226 patches, 32 patients.

2962 patches, 37 patients.

5988 patches, 16 patients.

31

[17] Multiscale lung texture signature learning using the Riesz transform, Depeursinge et al., Med Image Comput Comput Assist Interv (MICCAI), 15(3):517-24, 2012

Page 32: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• 3-D synthetic textures ( , based on [18])

• Vertical planes

• 3-D checkerboard

• 3-D wiggled checkerboard

LEARNING TEXTURE MODELS

32

[18] 3D Steerable Wavelets and Monogenic Analysis for Bioimaging, Chenouard et al., IEEE 8th Int Symp on Biomed Imag (ISBI), 2132-5, 2011

Page 33: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Steering texture models:

STEERABLE TEXTURE MODELS [11]

The expression of the rotated texture model remains a linear combination of the initial Riesz components

[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014

Page 34: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Steering texture models:

• Rotation-covariant models: local steering of

• Keep the maximum magnitude of the model

• Local quantitative features: energies/abs values of the magnitudes in the patch

STEERABLE TEXTURE MODELS [11]

34

The expression of the rotated texture model remains a linear combination of the initial Riesz components

[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014

Page 35: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Classification of 2-D natural texturesIEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5

1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021

9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035

17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009

Fig. 5. 128 128 blocks from the 24 texture classes of the Outex database.

1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin

9) raffia 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool

Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite.

180 180 images from rotation angles 20

, 70, 90, 120,135

and 150

of the other seven Brodatz images for eachclass. The total number of images in the test set is 672.

G. Experimental setupOVA SVM models using Gaussian kernels as K(x

i

,x

j

) =

exp(

||xixj ||22

2k

) are used both to learn texture signatures andto classify the texture instances in the final feature spaceobtained after k iterations. A number of scales J = 6

was used to cover the whole spectrum of the 128 128

subimages in Outex and J = 3 for covering the spectrum of16 16 subimages in Contrib TC 00000. The angle matrixthat maximizes the response of the texture signature at thesmallest scale

1

(x) (see Eq. (11)) is used to steer Riesztemplates from all scales. The dimensionality of the initialfeature space is J(N + 1). Every texture signature

N

c,K

iscomputed using the texture instances from the training set.The coefficients from all instances are rotated to locally aligneach signature N

c,K

and are concatenated to constitute the finalfeature space. The dimensionality of the final feature space isJ (N + 1) N

c

. OVA SVM models are trained in thisfinal feature space using the training instances. The remainingtest instances obtained are used to evaluate the generalizationperformance. All data processing was performed using MAT-LAB R2012b (8.0.0.783) 64–bit (glnxa64), The MathWorks

Inc., 2012. The computational complexity is dominated by thelocal orientation of N

c

in Eq. 11, which consists of finding theroots of the polynomials defined by the steering matrix A

.It is therefore NP–hard (Non–deterministic Polynomial–timehard), where the order of the polynomials is controlled by theorder of the Riesz transform N .

III. RESULTS

The performance of our approach is demonstrated withthe Outex and the Brodatz databases. The performance oftexture classification is first investigated in Section III-A.The evolution and the convergence of the texture signatures

N

c,k

through iterations k = 1, . . . , 10 is then studied inSection III-B for the Outex TC 00010 test suite.

A. Rotation–covariant texture classification

The rotation–covariant properties of our approach are eval-uated using Outex TC 00010, Outex TC 00012 and Con-trib TC 00000 test suites. The classification performanceof the proposed approach after the initial iteration (k=1)is compared with two other approaches that are based onmultiscale Riesz filterbanks. As a baseline, the classificationperformance using the energy of the coefficients of the initialRiesz templates was evaluated. Since the cardinality of the

APPLICATIONS (1/2)

35

Page 36: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• 24 classes, 180 images/class, 9 rotation angles in

• A SVM classifier is trained with unrotated images only

• 98.4% best acc.

• aligned models ( )

• Literature: 90-99%

• E.g., scattering transform: 98.75% [18]

2-D TEXTURE CLASSIFICATION: OUTEX DATABASE [17]

Errors occur with most stochastic textures

confusion matrix:

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 6

TABLE IAVERAGE Ac OBTAINED WITH THE LOCAL ORIENTATION MAXIMIZATION OF N

c,1 . THE PERFORMANCE OF EVEN AND ODD ORDERS IS COMPARED.

Outex TC 00010 Outex TC 00012 (P 000) Outex TC 00012 (P 001) Contrib TC 00000Ac for N = 2, 4, 6, 8, 10. 98± 0.7 97.2±0.7 98± 0.4 98.1±4.2Ac for N = 1, 3, 5, 7, 9. 94.4±0.7 93.6±0.6 95.1±0.4 89.9±1.8Ac for N = 1, . . . , 10. 96.2± 2 95.4± 2 96.6±1.6 94± 5.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10

cla

ssifi

catio

n a

ccu

racy

N

aligned

aligned

initial Riesz

Nc,1

R(0,N)

Fig. 7. Classification performance with the Outex TC 00010 test suite. Thetwo rotation covariant approaches are performing much better than usingthe initial Riesz coefficients. The local orientation maximization of N

c,1

outperforms the local orientation of the first Riesz template R(0,N) asproposed in [8]. A maximum Ac of 98.4% is reached with N = 8.

classes are balanced, the classification accuracy A

c

is used asa performance measure of the methods. All performances aresummarized in Table I.

1) Outex TC 00010: The classification performance fororders N = 1, . . . , 10 is shown in Fig. 7. The performanceusing the energy of the coefficients that are maximizing theresponse of the first Riesz template (i.e., R(0,N)) at thesmallest scale was also evaluated as a first rotation–covariantapproach [8]. An average A

c

of 96.2 ± 2% is obtained withN = 1 . . . 10 and the local orientation maximization of

N

c,1

.A maximum A

c

of 98.4% is reached with N = 8.2) Outex TC 00012: The classification performance for

orders N = 1, . . . , 10 is shown in Fig. 8. Average A

c

of95.4 ± 2% (P 000) and 96.6 ± 1.6% (P 001) are obtainedwith N = 1 . . . 10 and the local orientation maximization of

N

c,1

. Maximum A

c

of 97.8% (P 000, N = 10) and 98.4%(P 001, N = 8) are reached.

3) Contrib TC 00000: The classification performance fororders N = 1, . . . , 10 is shown in Fig. 9. An average A

c

of 94 ± 5.3% is obtained with N = 1 . . . 10 and the localorientation maximization of

N

c,1

. A perfect classification ofthe test set (A

c

= 100%) is reached for orders N = 4, 6, 8, 10.

B. Convergence and iterative evolution of N

c,k

The visual appearance and convergence of the texture signa-tures N

c,k

is investigated using the Outex TC 00010 test suite.The assumption is made that after a given number of iterationsK,

N

c,k

will converge to a final template representing theessential “stitch” of the texture class. The evolutions of 8

c,k

forclasses “6) canvas009” and “15) canvas033” are represented in

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10

cla

ssifi

catio

n a

ccu

racy

N

aligned (P_000)

initial Riesz (P_000)

aligned (P_001)

initial Riesz (P_001)

Nc,1

Nc,1

Fig. 8. Classification performance with the Outex TC 00012 test suitefor P 000 and P 001. Similarly to Outex TC 00010, the local orientationmaximization of N

c,1 performs much better than using the initial Rieszcoefficients for both problems. Maximum Ac of 97.8% (P 000, N = 10)and 98.4% (P 001, N = 8) are reached, which suggests high robustness toillumination changes of the proposed approach.

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10

cla

ssifi

catio

n a

ccu

racy

N

aligned

initial Riesz

Nc,1

Fig. 9. Classification performance with the Contrib TC 00000 test suite.The importance of the rotation–covariance of the operators is highlightedonce more, where the local orientation maximization of N

c,1 also performsmuch better than using the initial Riesz coefficients. A perfect classification(Ac = 100%) is reached for orders N = 4, 6, 8, 10.

Figures 10 and 11, respectively. The convergence is assessedas the evolution of ||wkwk+1||

||w1|| . The average convergence ofall classes is shown in Fig. 12.

IV. DISCUSSIONS AND CONCLUSIONS

We developed a texture learning framework that leveragesa key property of visual pattern discrimination: the local or-ganizations of scales and directions. Localized comprehensivecharacterizations of scales and directions are enabled usingsteerable Riesz wavelets. Class–wise templates are learnedfrom the data: discriminative combinations of scales and direc-tions are revealed from one–versus–all SVMs. The local ori-entation of the obtained templates or “signatures” is optimizedto maximize their response, which is carried out analyticallywith linear combinations of the initial Riesz templates. The

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5

1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021

9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035

17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009

Fig. 5. 128 128 blocks from the 24 texture classes of the Outex database.

1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin

9) raffia 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool

Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite.

180 180 images from rotation angles 20

, 70, 90, 120,135

and 150

of the other seven Brodatz images for eachclass. The total number of images in the test set is 672.

G. Experimental setupOVA SVM models using Gaussian kernels as K(x

i

,x

j

) =

exp(

||xixj ||22

2k

) are used both to learn texture signatures andto classify the texture instances in the final feature spaceobtained after k iterations. A number of scales J = 6

was used to cover the whole spectrum of the 128 128

subimages in Outex and J = 3 for covering the spectrum of16 16 subimages in Contrib TC 00000. The angle matrixthat maximizes the response of the texture signature at thesmallest scale

1

(x) (see Eq. (11)) is used to steer Riesztemplates from all scales. The dimensionality of the initialfeature space is J(N + 1). Every texture signature

N

c,K

iscomputed using the texture instances from the training set.The coefficients from all instances are rotated to locally aligneach signature N

c,K

and are concatenated to constitute the finalfeature space. The dimensionality of the final feature space isJ (N + 1) N

c

. OVA SVM models are trained in thisfinal feature space using the training instances. The remainingtest instances obtained are used to evaluate the generalizationperformance. All data processing was performed using MAT-LAB R2012b (8.0.0.783) 64–bit (glnxa64), The MathWorks

Inc., 2012. The computational complexity is dominated by thelocal orientation of N

c

in Eq. 11, which consists of finding theroots of the polynomials defined by the steering matrix A

.It is therefore NP–hard (Non–deterministic Polynomial–timehard), where the order of the polynomials is controlled by theorder of the Riesz transform N .

III. RESULTS

The performance of our approach is demonstrated withthe Outex and the Brodatz databases. The performance oftexture classification is first investigated in Section III-A.The evolution and the convergence of the texture signatures

N

c,k

through iterations k = 1, . . . , 10 is then studied inSection III-B for the Outex TC 00010 test suite.

A. Rotation–covariant texture classification

The rotation–covariant properties of our approach are eval-uated using Outex TC 00010, Outex TC 00012 and Con-trib TC 00000 test suites. The classification performanceof the proposed approach after the initial iteration (k=1)is compared with two other approaches that are based onmultiscale Riesz filterbanks. As a baseline, the classificationperformance using the energy of the coefficients of the initialRiesz templates was evaluated. Since the cardinality of the

[17] Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, Ojala et al., IEEE Trans on Pattern Anal and Mach Intel, 24:7(971-87), 2002 [18] Combined scattering for rotation invariant texture analysis, Sifre et al., European Symposium on Artificial Neural Networks, 2012

Importance of rotation-covariance and learned models

Page 37: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

Medulloblastoma tumor classification

APPLICATIONS (2/2)

Medulloblastoma Di↵erentiation Fusing UFL and Riesz Features 5

where m stands for the number of samples, and k is the number of classes, and is the weight decay parameter that penalizes large values for parameters. Thefused representation of an unseen test tissue tile x

(p) 2 R(N+1)Jk+l is classifiedas anaplastic (or non-anaplastic) by calculating a probability:

p(yp = 1|xp;) =exp (1x

(p))P2

l=1 exp (lx(p))

.

A tile belongs to the anaplastic class if p(yp|xp;) > 0.5 and non-anaplasticotherwise.

3 Experimental Results and Discussion

The workflow of the proposed approach is summarized in Fig. 2. As first step, wecompute the UFL features learned by TICA and the supervised features learnedwith Riesz wavelets for each image as described in Sections 2.1 and 2.2. OnceTICA and Riesz wavelets are computed a final step of supervised classification ismade using the combination of the computed features in a concatenated vectoras input for a standard softmax classifier as described in Section 2.3. Parametertuning is presented in Section 3.2.

[2.4] Medulloblastoma Image Cases

Fig. 2. Flowchart for MB feature extraction and classification for both learned repre-sentations: Riesz and TICA, the details of each stage are described in subsections.

3.1 Medulloblastoma Dataset

Our MB database is from St. Jude Childrens Research Hospital in Memphiswhere a neuropathologist manually annotated the cancerous regions of 10 pathol-ogy slides, 5 diagnosed as anaplastic and 5 as non-anaplastic MB. Slides werestained with hematoxylin and eosin (H&E) and digitization was done on anAperio Scanner obtaining WSI with a resolution of 80,00080,000 pixels. Eachimage can have several cancerous regions, which were manually annotated. Fortraining, we randomly extracted a total of 7,500 square regions (750 per case) of200200 pixels of the tumor regions (3,750 anaplastic and 3,750 non-anaplastic).

steerable Riesz texture models

topographic ICA

37

Page 38: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Medulloblastoma (MB)

• Most common (i.e., 25%) pediatric brain tumor

• Major cause of death in pediatric oncology

• Histological subtypes of MB have different prognosis and treatments

• Anaplastic subtype is the worst!

• “marked nuclear pleomorphism, cell wrapping, high mitotic count, and abundant apoptotic bodies” [19]

MEDULLOBLASTOMA TUMORS [19]

38

[19] Childhood medulloblastoma: novel approaches to the classification of a heterogeneous disease, Ellison et al., Acta Neuropathol, 20(3):305-16, 2010

Pleomorphism (cytology): variability in the size and shape of cells and/or their nuclei

Page 39: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Proposed approach [20]

• Anaplastic versus non-anaplastic

• Compare/combine steerable Riesz texture models with unsupervised topographic independent component analysis (TICA)

MEDULLOBLASTOMA TUMOR CLASSIFICATION

39

[20] Combining Unsupervised Feature Learning and Riesz Wavelets for Histopathology Image Representation: Application to Identifying Anaplastic Medulloblastoma, Otálora et al., Med Image Comput Comput Assist Interv (MICCAI), 2015

Medulloblastoma Di↵erentiation Fusing UFL and Riesz Features 5

where m stands for the number of samples, and k is the number of classes, and is the weight decay parameter that penalizes large values for parameters. Thefused representation of an unseen test tissue tile x

(p) 2 R(N+1)Jk+l is classifiedas anaplastic (or non-anaplastic) by calculating a probability:

p(yp = 1|xp;) =exp (1x

(p))P2

l=1 exp (lx(p))

.

A tile belongs to the anaplastic class if p(yp|xp;) > 0.5 and non-anaplasticotherwise.

3 Experimental Results and Discussion

The workflow of the proposed approach is summarized in Fig. 2. As first step, wecompute the UFL features learned by TICA and the supervised features learnedwith Riesz wavelets for each image as described in Sections 2.1 and 2.2. OnceTICA and Riesz wavelets are computed a final step of supervised classification ismade using the combination of the computed features in a concatenated vectoras input for a standard softmax classifier as described in Section 2.3. Parametertuning is presented in Section 3.2.

[2.4] Medulloblastoma Image Cases

Fig. 2. Flowchart for MB feature extraction and classification for both learned repre-sentations: Riesz and TICA, the details of each stage are described in subsections.

3.1 Medulloblastoma Dataset

Our MB database is from St. Jude Childrens Research Hospital in Memphiswhere a neuropathologist manually annotated the cancerous regions of 10 pathol-ogy slides, 5 diagnosed as anaplastic and 5 as non-anaplastic MB. Slides werestained with hematoxylin and eosin (H&E) and digitization was done on anAperio Scanner obtaining WSI with a resolution of 80,00080,000 pixels. Eachimage can have several cancerous regions, which were manually annotated. Fortraining, we randomly extracted a total of 7,500 square regions (750 per case) of200200 pixels of the tumor regions (3,750 anaplastic and 3,750 non-anaplastic).

steerable Riesz texture models

topographic ICA

Page 40: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Unsupervised topographic independent component analysis (TICA) [21]

• Learn while minimizing cost function :

MEDULLOBLASTOMA TUMOR CLASSIFICATION

40[21] Energy correlations and topographic organization, Hyvärinen et al., Natural Image Statistics, 39:249-272, 2009

Medulloblastoma Di↵erentiation Fusing UFL and Riesz Features 3

Neural networks are the dominant approach for representation learning. How-ever there are other representation learning strategies that are able to adaptconventional image descriptors to the needs of a particular image analysis task.In [3], the authors propose a multiscale texture signature learning approach usingrotation-covariant Riesz wavelets, where most relevant combinations of orienta-tions and scales are learned directly from the data. This approach outperformedstate-of-the-art representations based on local binary patterns and grey levelcooccurrence matrices for lung tissue classification [4]. Drawbacks of the datadriven representations approaches are the amount of parameters involved thathave to be manually tuned, which requires more time in model training. Sometexture based representations fail to describe the feature patterns present intraining samples that the data-driven approach is able to find [1].

In this work, we propose a joint framework for classification of MB WSI,where the invariant properties of TICA features and the multiscale rotation-covariant properties of Riesz wavelet features complement each other. We hy-pothesize that this fusion can lead to a better classification performance. Thiswork join e↵orts of [3] and [1] in a simple manner to achieve the best accuracyreported for this histopathology WSI database.

2 Methodological Description

2.1 Topographic Independent Component Analysis

TICA is an unsupervised feature learning model, inspired by findings of the visualcortex behaviour. It groups activations of units in order to discover features thatare rotation and translation invariant [1]. These are appropriate features forhistopathology image characterization since shapes and cell organizations canbe present regardless of the position or orientation of cells. Particularly, TICAorganizes feature detectors in a square matrix for l groups such that adjacentfeature detectors activate in a similar proportion to the same stimulus. To learnsuch groups, we need to optimize the cost function:

JTICA(W) =

2

TX

i=1

WTWx

(i) x

(i)

2

2

+mX

i=1

lX

k=1

»Hk(Wx

(i))2 + (1)

where x

(i) 2 Rm is the i-th sample, T is the number of samples, W 2 Rnm isthe matrix that encodes the features in each row, and H 2 0, 1ln is the binary

topographic organization where H(j)k = 1, if the j-th feature detector, j-th row

of W , belongs to the k-th group, and 0 otherwise. This model sets H fixed whilelearning W. In addition, TICA has two main computational advantages. First,the only parameters to be tuned are the regularization hyperparameter and thesparsity controller . Second, it is an unconstrained optimization problem, whichcan be solved eciently by optimization techniques such as Limited memory-Broyden-Fletcher-Goldfarb-Shanno (L-BFGS).

11111111111111.11.11.111111..1111111111.11...11..11.11..111111111111111111.11siiáiiiisisiiisiii1111111111.iiiiiéiiáiiiiiiiái1111..111111.111111111111iiiéiiiiiiiiii®ii1111111111.11111111.1111.U11.1111iiééiiéiéviii111111111111111111.1111...11iiiiii11éisi11111111111111.11111111111111...1111111111111111.111111.111111..1111.s11.a11

\iriiiiéiiiiiiiiiiiiiio siiióiiiiïiiiiiiiii1=11.e1111MMIIIMS111111111111111MIN

11=1iiiiiiiiiiiiisiiiiiIO

ON1110111011

sRoMM

INIMI

11ErMIIM

MERIIM

Mvs

MMMMMMMMMMMMMMMMMMMiiiMUMMMiMMMiiiiiii iíiíi iiiíiiiii UWE:lMiiMME= M MEEis i iiiiíiiiiMóóóiiiMMMMMlMiMMMMMMMiMMMiMMMMMMMMMiiMMiMMiMMMiMMMiMMMiiMMMMiiiiiiiMiMMiMiMiiM\MiiMiiiiiiiiMMS MMMMiMMMMMMMiiMMMMiiiiiiiiiiiiiiitiiii

Figure 2. Local learned features with di↵erent UFL methods. Left: Sparse Autoencoders. Center: RICA. Right: T-RICAhighlighting some translational (blue), color (red), scale (yellow) and rotational (green) invariances.

2.4 Basal-cell carcinoma detection

In order to compare the proposed method with other state-of-the-art representation schemes, we fixed classifiermodel using the Softmax regression algorithm. Softmax regression is a multinomial classifier that generalizes thelogistic regression binary classifier.12 A softmax classifier was chosen because such model allows to interpret theprediction as a probability, as well as highlighting features related with each class based on learned weights.

3. EXPERIMENTAL EVALUATION

The experimental evaluation was performed over a dataset that comprises 1,407 histopathology images ofskin tissue biopsies at 10X magnification for Basal-cell carcinoma diagnosis (518 cancerous and 889 healthy)which were labeled by an expert pathologist. Experiments were performed on a 5-fold cross validation schemewith stratified sampling, and average of Sensitivity, F-Score(2 precisionrecall

precision+recall

) and Balanced Accuracy (BAC)

( sensitivity+specificity

2

) were reported. To perform result analysis, we compare our proposed method within BOFframework versus traditional DCT, Haar and Sparse Auto-Encoders descriptors for local feature representations.400 features were learned for each UFL Method (AE, RICA and T-RICA) and a 400 visual word BOF dictionarywas used for all experiments.

3.1 Topographic Local Learned Features Analysis

100,000 patches of 8 8 pixels were randomly sampled from training set, and 400 features were learned with atile size of 3 3 (group size in topography). The set of learned features is shown in Fig. 2. T-RICA learns inunsupervised way abstract concepts, like edges and texture patterns, directly from raw pixels. The topographicapproach organizes features in such a way that it discovers not only translational, but also rotational, color andscale invariances. The representation scheme described in section 2.2 exploits such organization to detect thoseinvariances in target image.

Images are described as histograms as explained in Section 2.3. The histogram representation is used as inputto a softmax classifier model. This model gives positive and negative weights to features which are proportionalto the relative importance for generating a classification. So, it is possible to map such learned weights overtopographic organization to highlight the most discriminant features as displayed in Fig. 3.

This organization suggests that T-RICA model captures and organizes relevant patterns of each class. Thisis a very interesting result since the model is totally unsupervised, i.e., the visual patterns were found usingnon-labelled samples.

Classification results are shown in Fig. 4 reporting the average of performance measures. Results show thatUFL representations are competitive with standard DCT descriptors. Though, adding topographic organizationconsiderably improves classification results in 7% with respect to traditional autoencoders, and 6% with respect

Patches were preprocessed using Zero-phase Component Analysis (ZCA) Whitening transformation. See details in13

Proc. of SPIE Vol. 8922 89220M-4

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/18/2015 Terms of Use: http://spiedl.org/terms

JTICA(W )

reconstruction penaltyL2 topographic constraint: if the patch belongs to the local group of patchesHk = 1

x

(i)

k

~scale inv. ~rotational inv.

~translational inv.

W

Page 41: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Dataset:

• 5 anaplastic, 5 non-anaplastic patients

• 750 patches (200x200) / patient

• Leave-2-patients-out cross-validation

• Softmax classifier

• Comparison with state-of-the-art [22]

• Convolutional neural networks (CNN), sparse autoencoders (sAE), bag of features (BOF), Haar, MR8, …

• Predictions over entire-slides:

MEDULLOBLASTOMA TUMOR CLASSIFICATION

41

[22] A comparative evaluation of supervised and unsupervised representation learning approaches for anaplastic medulloblastoma differentiation, Cruz-Roa et al., Proc. SPIE 9287, Int Symp Med Inf Proc Anal, 92870G, 2015

Medulloblastoma Di↵erentiation Fusing UFL and Riesz Features 7

Table 1. MB classification performance (baseline, Riesz, fusion). The measures areaveraged over the 20 test runs with standard deviation where available.

Method Accuracy Sensitivity Specificity

TICA + Riesz[N13 , N

22 , N

21 ] 0.997 ± 0.002 0.995 ± 0.004 1 ± 0

TICA [1] 0.972 ± 0.018 0.977 ± 0.021 0.967 ± 0.031

Riesz [N13 , N

22 , N

21 ] [3] 0.964 ± 0.038 0.999 ± 0.001 0.932 ± 0.07

Riesz [N13 ] [3] 0.958 ± 0.062 0.963 ± 0.05 0.916 ± 0.125

Riesz [N22 ] [3] 0.94 ± 0.02 0.94 ± 0.02 0.3 ± 0.04

2-Layer CNN [1] 0.90 ± 0.1 0.89 ± 0.18 0.9 ± 0.0.3

sAE [1] 0.90 0.87 0.93

BOF + A2NMF (Haar) [2] 0.87 0.86 0.87

Riesz [N21 ] [3] 0.85 ± 0.23 0.9 ± 0.15 0.7 ± 0.47

BOF + K - NN (Haar) [7] 0.80 - -

BOF + K - NN (MR8) [7] 0.62 - -

4 Concluding Remarks

We present a feature fusion between unsupervised feature learning and super-vised Riesz wavelet representation that captures subtle pattern of textures aswell as high level features, allowing to create a more separable feature spacewhere the di↵erentiation of medulloblastoma into anaplastic and non-anaplasticcan be made with high classification accuracy outperforming any other resultpreviously described in the literature. To our knowledge this is the first timethat a feature fusion method is presented between UFL and the Riesz waveletsin the context of histopathology image analysis showing the complementaritybetween these learned features for the challenging task of tumour di↵erentia-tion, we are currently working on extending the method to other patch-basedhistopathology image analysis problems with larger cohorts of patients.

Fig. 3. Predictions over two WSIs, non-anaplastic MB (left) and anaplastic (right).

[22]

[22]

[22]

[22]

[22]

Medulloblastoma Di↵erentiation Fusing UFL and Riesz Features 7

Table 1. MB classification performance (baseline, Riesz, fusion). The measures areaveraged over the 20 test runs with standard deviation where available.

Method Accuracy Sensitivity Specificity

TICA + Riesz[N13 , N

22 , N

21 ] 0.997 ± 0.002 0.995 ± 0.004 1 ± 0

TICA [1] 0.972 ± 0.018 0.977 ± 0.021 0.967 ± 0.031

Riesz [N13 , N

22 , N

21 ] [3] 0.964 ± 0.038 0.999 ± 0.001 0.932 ± 0.07

Riesz [N13 ] [3] 0.958 ± 0.062 0.963 ± 0.05 0.916 ± 0.125

Riesz [N22 ] [3] 0.94 ± 0.02 0.94 ± 0.02 0.3 ± 0.04

2-Layer CNN [1] 0.90 ± 0.1 0.89 ± 0.18 0.9 ± 0.0.3

sAE [1] 0.90 0.87 0.93

BOF + A2NMF (Haar) [2] 0.87 0.86 0.87

Riesz [N21 ] [3] 0.85 ± 0.23 0.9 ± 0.15 0.7 ± 0.47

BOF + K - NN (Haar) [7] 0.80 - -

BOF + K - NN (MR8) [7] 0.62 - -

4 Concluding Remarks

We present a feature fusion between unsupervised feature learning and super-vised Riesz wavelet representation that captures subtle pattern of textures aswell as high level features, allowing to create a more separable feature spacewhere the di↵erentiation of medulloblastoma into anaplastic and non-anaplasticcan be made with high classification accuracy outperforming any other resultpreviously described in the literature. To our knowledge this is the first timethat a feature fusion method is presented between UFL and the Riesz waveletsin the context of histopathology image analysis showing the complementaritybetween these learned features for the challenging task of tumour di↵erentia-tion, we are currently working on extending the method to other patch-basedhistopathology image analysis problems with larger cohorts of patients.

Fig. 3. Predictions over two WSIs, non-anaplastic MB (left) and anaplastic (right).

non-anaplastic anaplastic

TICA: unsupervised Riesz: rotation-covariant information

Small dataset

Page 42: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Highly adaptive texture-based computational models of biomedical tissue:

Complete/systematic coverage of image scales and directions in 2-D and 3-D

Rotation-covariance

Specificity: the models can be trained to characterize specific tissue types

Local characterization of tissue properties

• Locate tissue properties in organ anatomy to create digital phenotypes

• localization systems, quant. graph analysis

CONCLUSIONS

model learning

Multi-resolution, steerability

patch-based analysis

42

be averaged over the folds of the CV and used to build texture models for each location along dROI (see Fig 9).The sum of the weights for all channels from each location will reveal the subregions that are specific to eachtumor subtypes, and most related to patient survival. A K–means clustering of the vectors w will be carried outfor each locations to evaluate the stability of the regional models over the folds of the CV and define homogeneousgroups among patients. A selection of the models based on stability and location importance will be carried out.

Fig. 9 Prototype tissue archi-tecture of a GBM tumor [128].

In a second step, the selected models will be locally steered to maximize theirmagnitude. The energies of the maximal magnitudes will be used to constructa final feature space for (1) predicting the tumor subtypes and (2) performingKaplan–Meier survival analysis. The performance of the proposed approach forpredicting tumor subtypes and patient survival will be compared to (1) unalignedwavelet energies and (2) average wavelet energies over the entire tumor. Whilestarting with a LOPO CV on the TCGA–TCIA dataset, the generalizability of theapproach will be further assessed by training with the TCGA–TCIA and testingwith the SU dataset.

Deliverable 2.1: Predicting tumor subtype and survival from localized of tissueproperties in GBM tumors.Task 2.2: Digital lung tissue atlases of ILD diagnoses (8 months of the PI)In this task, we will use a simple atlas of the lungs to locate texture properties and create prototype diagnosisphenotypes of ILDs. In previous work, we developed a simple 3–D atlas of the lungs with 36 subregionsthat will be used in this task [42]. In a first step, diagnosis–wise digital tissue atlases will be created bylearning 3–D texture models (i.e., the average of w over the folds of the LOPO CV) for each 36 regions ofthe lungs (see Fig. 10). The regions for which the models are most distant11 from all other diagnoses willbe highlighted and compared to previously built models of tissue patterns [25] to create 3–D prototype tissueatlases for each diagnosis. The obtained results will be validated using medical knowledge (e.g., Table 1 of [42]).

Fig. 10 Tissue atlas of the lungs.

For each diagnosis, K–means clustering of the vectors w will be carried outfor each locations to evaluate the stability of the regional models over the foldsof the CV and reveal homogeneous groups among patients (e.g., subtypes ofUIP). The most stable models will be kept for the further characterization oflung tissue types. A hierarchical clustering of the models from all diagnoseswill be carried out to define a radiomics–based hierarchy of all diagnoses, whichwill be compared to medical knowledge [140] (e.g., Fig. 1 of [40]). A largefeature space including the energies of the steered models from each of the 36localizations will be used to predict the diagnoses with uncertainty assessment(using e.g., pairwise coupling [141]). When a minimum amount of trust is notachieved when predicting a given diagnosis, the parent group of ILD diagnosesin the previously built hierarchy will be predicted instead [19].

Deliverable 2.2: Digital tissue atlases of ILD diagnoses and their subtypes.Task 2.3: Digital tissue atlases of ILDs: correlation with PFTs and survival (6 months )Digital tissue atlases will be constructed for poor versus normal/high (1) PFTs and (2) survival. Regions forwhich the models significantly di↵er between poor versus normal/high will be revealed as being of primaryimportance to evaluate pulmonary function. The links between these models and previously built models oftissue patterns will be investigated to define the combination of regions and patterns that are most responsiblefor lung function impairment. The feature space spanned by the energies of the steered models will be used topredict (1) PFT values or (2) survival with a LOPO CV, which can be evaluated using ground truth.

Deliverable 2.3: Estimating pulmonary function from digital tissue atlases of ILD diagnoses in CT images.

5.3 WP3. Imaging genomics

In this WP, we will use the texture models and digital tissue atlases developed in WP2 to learn and locatethe radiological patterns associated with prevailing meta–genes. This will allow for a deeper comprehension ofthe links between the radiological phenotypes and the expression of meta–genes. The molecular heterogeneityof GBM tumors ILD diagnoses will be studied using 3–D patch–based image analysis to enable the localcharacterization of radiogenomic properties.Task 3.1: Texture–based imaging genomics of GBM tumors (6 months of the PI)Based on the hypothesis that every meta–gene is associated with a specific visual pattern inside GBM tumorsin MR T1 images, patch–based analysis of the GBM tumors will be used to reveal local image regions thatcorrelate most with a given driver meta–gene. The gene expression is known to be di↵erent in perilesionaledema, on the tumor margin and in central necrosis [61, 130]. Gene expression patterns are also defining the

11in terms of Euclidean distance

10

Page 43: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

CONCLUSIONS

• Importance of rotation-covariant information to model biomedical textures

• i.e., the local organization of directions

• Rotational invariance is not enough

• Called “roto-translation invariance” in [23]

TextureQbased'biomarkers:'current'limitaGons'

x  Assume'homogeneous'texture'properGes'over'the'enGre'lesion'[5]'

'

x NonQspecific'features'x  Global'vs'local'characterizaGon'of'image'direcGons'[6]'

Radiology: Volume 269: Number 1—October 2013 n radiology.rsna.org 13

REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al

with the mean signal value. By using just two sequences, a contrast-enhanced T1 sequence and a fluid-attenuated inver-sion-recovery sequence, we can define four habitats: high or low postgadolini-um T1 divided into high or low fluid-at-tenuated inversion recovery. When these voxel habitats are projected into the tu-mor volume, we find they cluster into spatially distinct regions. These habitats can be evaluated both in terms of their relative contributions to the total tumor volume and in terms of their interactions with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysis can be performed with CT scans (Fig 5).

Analysis of spatial patterns in cross-sectional images will ultimately re-quire methods that bridge spatial scales from microns to millimeters. One possi-ble method is a general class of numeric tools that is already widely used in ter-restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Species distribution models (48–51) are used to gain ecologic and evolutionary insights and to predict distributions of species or morphs across landscapes, sometimes extrapolating in space and time. They can easily be used to link the environ-mental selection forces in MR imaging-defined habitats to the evolutionary dy-namics of cancer cells.

Summary

Imaging can have an enormous role in the development and implementation of patient-specific therapies in cancer. The achievement of this goal will require new methods that expand and ultimately re-place the current subjective qualitative assessments of tumor characteristics. The need for quantitative imaging has been clearly recognized by the National Cancer Institute and has resulted in for-mation of the Quantitative Imaging Net-work. A critical objective of this imaging consortium is to use objective, repro-ducible, and quantitative feature metrics extracted from clinical images to develop patient-specific imaging-based prog-nostic models and personalized cancer therapies.

rise to local-regional phenotypic adap-tations. Phenotypic alterations can re-sult from epigenetic, genetic, or chro-mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations.

Emerging Strategies for Tumor Habitat Characterization

A method for converting images to spa-tially explicit tumor habitats is shown in Figure 4. Here, three-dimensional MR imaging data sets from a glioblastoma are segmented. Each voxel in the tumor is defined by a scale that includes its image intensity in different sequences. In this case, the imaging sets are from (a) a contrast-enhanced T1 sequence, (b) a fast spin-echo T2 sequence, and (c) a fluid-attenuated inversion-recov-ery (or FLAIR) sequence. Voxels in each sequence can be defined as high or low based on their value compared

microenvironment can be rewarded by increased proliferation. This evolution-ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec-tional images (Fig 5).

Interpretation of the subsegmenta-tion of tumors will require computa-tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex-ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character-istics. Conceptually, this approach as-sumes that regional variations in tumors can be viewed as a coalition of distinct ecologic communities or habitats of cells in which the environment is governed, at least to first order, by variations in vascular density and blood flow. The environmental conditions that result from alterations in blood flow, such as hypoxia, acidosis, immune response, growth factors, and glucose, represent evolutionary selection forces that give

Figure 4

Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions.

Radiology: Volume 269: Number 1—October 2013 n radiology.rsna.org 13

REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al

with the mean signal value. By using just two sequences, a contrast-enhanced T1 sequence and a fluid-attenuated inver-sion-recovery sequence, we can define four habitats: high or low postgadolini-um T1 divided into high or low fluid-at-tenuated inversion recovery. When these voxel habitats are projected into the tu-mor volume, we find they cluster into spatially distinct regions. These habitats can be evaluated both in terms of their relative contributions to the total tumor volume and in terms of their interactions with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysis can be performed with CT scans (Fig 5).

Analysis of spatial patterns in cross-sectional images will ultimately re-quire methods that bridge spatial scales from microns to millimeters. One possi-ble method is a general class of numeric tools that is already widely used in ter-restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Species distribution models (48–51) are used to gain ecologic and evolutionary insights and to predict distributions of species or morphs across landscapes, sometimes extrapolating in space and time. They can easily be used to link the environ-mental selection forces in MR imaging-defined habitats to the evolutionary dy-namics of cancer cells.

Summary

Imaging can have an enormous role in the development and implementation of patient-specific therapies in cancer. The achievement of this goal will require new methods that expand and ultimately re-place the current subjective qualitative assessments of tumor characteristics. The need for quantitative imaging has been clearly recognized by the National Cancer Institute and has resulted in for-mation of the Quantitative Imaging Net-work. A critical objective of this imaging consortium is to use objective, repro-ducible, and quantitative feature metrics extracted from clinical images to develop patient-specific imaging-based prog-nostic models and personalized cancer therapies.

rise to local-regional phenotypic adap-tations. Phenotypic alterations can re-sult from epigenetic, genetic, or chro-mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations.

Emerging Strategies for Tumor Habitat Characterization

A method for converting images to spa-tially explicit tumor habitats is shown in Figure 4. Here, three-dimensional MR imaging data sets from a glioblastoma are segmented. Each voxel in the tumor is defined by a scale that includes its image intensity in different sequences. In this case, the imaging sets are from (a) a contrast-enhanced T1 sequence, (b) a fast spin-echo T2 sequence, and (c) a fluid-attenuated inversion-recov-ery (or FLAIR) sequence. Voxels in each sequence can be defined as high or low based on their value compared

microenvironment can be rewarded by increased proliferation. This evolution-ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec-tional images (Fig 5).

Interpretation of the subsegmenta-tion of tumors will require computa-tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex-ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character-istics. Conceptually, this approach as-sumes that regional variations in tumors can be viewed as a coalition of distinct ecologic communities or habitats of cells in which the environment is governed, at least to first order, by variations in vascular density and blood flow. The environmental conditions that result from alterations in blood flow, such as hypoxia, acidosis, immune response, growth factors, and glucose, represent evolutionary selection forces that give

Figure 4

Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions.

[5]'QuanGtaGve'imaging'in'cancer'evoluGon'and'ecology,'Gatenby'et'al.,'Radiology,'269(1):8Q15,'2013'

5'

global'direcGonal'operators:' local'grouped'steering:'

[6]'RotaGonQcovariant'texture'learning'using'steerable'Riesz'wavelets,'Depeursinge'et'al.,'IEEE'Trans'Imag'Proc.,'23(2):898Q908,'2014.'

global directional operators local grouped steering

43

[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc, 23(2):898-908, 2014[23] Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination, Sifre et al., IEEE Conf Comp Vis and Pat Rec (CVPR), 1233-40, 2013

sentation R(x) of x is invariant to the action of G if it is notmodified by the action of any g ∈ G: R(g.x) = R(x). It iscovariant to G if R(g.x) = g.R(x), where g acts on R(x)by shifting its coefficients. A separable invariant on a groupproduct G = G1G2 combines a first operator R1, which isinvariant to the action of G1 and covariant to the action G2,with a second operator R2 which is invariant to the actionof G2. Indeed for all g1.g2 ∈ G1 G2 and all images x(u):

R2(R1(g1.g2.x)) = R2(g2.R1(x)) = R2(R1(x)) .However, such separable invariants do not capture the jointproperty of the action of G2 relatively to G1, and may loseimportant information. This is why two-dimensional trans-lation invariant representations are not computed by cascad-ing invariants to horizontal and vertical translations. It isalso important for rotations and translations. Let us considerfor example the two texture patches of Figure 1. A separa-ble product of translation and rotation invariant operatorscan represent the relative positions of the vertical patterns,and the relative positions of the horizontal patterns, up toglobal translations. However, it can not represent the po-sitions of horizontal patterns relatively to vertical patterns,because it is not sensitive to a relative shift between thesetwo sets of oriented structures. It loses the relative positionsof different orientations, which is needed to be sensitive tocurvature, crossings and corners. Such a separable invariantthus can not discriminate the two textures of Figure 1.

Figure 1: The left and right textures are not discriminatedby a separable invariant along rotations and translations, butcan be discriminated by a joint roto-translation invariant.

Several authors [6, 7, 8] have proposed to take into ac-count the joint structure of roto-translation operators in im-age processing, particularly to implement diffusion oper-ators. Computing a joint invariant between rotations andtranslations also means taking into account the joint rela-tive positions and orientations of image structures, so thatthe textures of Figure 1 can be discriminated. Section 3introduces a roto-translation scattering operator, which iscomputed by cascading wavelet transforms on the roto-translation group.

Calculating joint invariants on large non-commutativegroups may however become very complex. Keeping a sep-arable product structure is thus desirable as long as it does

not lose too much information. This is the case for scaling.Indeed, local image structures are typically spread acrossscales, with a power law decay. This is the case for con-tours, singularities and most natural textures. As a result ofthis strong correlation across scales, one can use a separa-ble invariant along scales, with little loss of discriminativeinformation.

2.2. Hierarchical Architecture

We now explain how to build an affine invariant repre-sentation, with a hierarchical architecture. We separate vari-abilities of potentially large amplitudes such as translations,rotations and scaling, from smaller amplitude variabilities,but which may belong to much higher dimensional groupssuch as shearing and general diffeomorphisms. These smallamplitude deformations are linearized to remove them withlinear projectors.

Image variabilities typically differ over domains of dif-ferent sizes. Most image representations build localized in-variants over small image patches, for example with SIFTdescriptors [15]. These invariant coefficients are then ag-gregated into more invariant global image descriptors, forexample with bag of words [10] or multiple layers of deepneural network [4, 5]. We follow a similar strategy by firstcomputing invariants over image patches and then aggregat-ing them at the global image scale. This is illustrated by thecomputational architecture of Figure 2.

xroto-trans.

patchscattering

logglobal

space-scaleaveraging

deformat.invariant

linear proj.

Figure 2: An affine invariant scattering is computed by ap-plying a roto-translation scattering on image patches, a log-arithmic non-linearity and a global space-scale averaging.Invariants to small shearing and deformations are computedwith linear projectors optimized by a supervised classifier.

Within image patches, as previously explained, one mustkeep the joint information between positions and orienta-tions. This is done by calculating a scattering invariant onthe joint roto-translation group. Scaling invariance is thenimplemented with a global scale-space averaging betweenpatches, described in Section 4. A logarithmic non-linearityis first applied to invariant scattering coefficients to linearizetheir power law behavior across scales. This is similar to thenormalization strategies used by bag of words [10] and deepneural networks [5].

Because of three dimensional surface curvature in the vi-sual scene, the image patches are also deformed. A scat-tering transform was proved to be stable to deformations[9]. Indeed, it is computed with a cascade of wavelet trans-

[23]

[11]

Page 44: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Rotation-covariance

• Local orientation of the models is computationally intensive

• 2D:

• 3D:

• Use graphics processor units (GPUs) [24]: 60x speedup

• Explore other steerable wavelet representations [25,26]

LIMITATIONS AND FUTURE WORK

44

G R(2,0,0) G R(0,2,0) G R(0,0,2) G R(1,1,0) G R(1,0,1) G R(0,1,1)

Fig. 1. Second–order Riesz kernels R(n1,n2,n3) convolved with isotropic Gaussian kernels G(x).

Support vector machines (SVM) are then used to classifybetween 9,347 normal and embolic cubic instances of lungparenchyma from 19 patients with APE and 8 control cases.

2. MATERIAL AND METHODS

2.1. Rotation–covariant texture analysis

3D multiscale Riesz filterbanks are used to characterize thetexture of the lung parenchyma in 3D at a given CT energylevel. The N–th order Riesz transform R(N) of a three–dimensional signal f(x) is defined in the Fourier domain as:

¤R(n1,n2,n3)f(!) =

…n1 + n2 + n3

n1!n2!n3!

(j!1)n1(j!2)

n2(j!3)

n3

||!||n1+n2+n3

ˆf(!),

(1)

for all combinations of (n1, n2, n3) with n1 + n2 + n3 = Nand n1,2,3 2 N. Eq. (1) yields

N+22

templates R(n1,n2,n3)

and forms multiscale steerable filterbanks when coupledwith a multi–resolution framework based on isotropic band–limited wavelets (e.g., Simoncelli) [12]. The second–orderRiesz filterbank is depicted in Fig. 1. Rotation–covariance isobtained by locally aligning the Riesz components R(n1,n2,n3)

of all scales based on the local prevailing orientation [13].The steerability property allows synthesizing the responses ofthe templates at any orientation based on a linear combinationof themselves. It therefore does not require additional convo-lutions of oriented filters [12]. Locally prevailing orientationsbased on the three Euler angles parameterizing SO(3) areestimated using a regularized version of the structure tensor,where the latter is computed over a 3D Gaussian window.The variance t of this Gaussian window determines thescale for the local orientation estimation. We use the imple-mentation of the regularized structure tensor by Chenouardet al. [11]. Locally aligned multiscale Riesz features are

denoted as ‹R(N)in the next sections.

2.2. Experimental setup

19 patients with APE and 8 control cases are used to evalu-ate the proposed approach. All 27 cases underwent a DECTscanner1 at the Department of Radiology of the University

1GE Discovery CT750 HD with two X–ray tubes and rapid peak kilovolt-age switching (gemstone spectral imaging, GSI).

Hospitals of Geneva with an inter–slice distance of 1mm, aslice thickness of 1.25mm, and a sub–millimetric resolutionin the axial plane. 11 energy levels are used from 40keV to140keV with a step of 10keV. All five lobes of each patienthave been manually segmented using the OsiriX software2.The perfusion levels of each lobe were quantified using theQanadli index (QI) on a lobe basis [14]. The QI is defined asthe sum of the scores of all arteries as: 0 if no occlusion isvisible, 1 if partially occluded, and 2 if totally obstructed.

DECT data of all energy levels are preprocessed to havean isotropic voxel resolution, which is obtained by dividingsamples along the z axis. All lobes are divided into 32

3

overlapping blocks to constitute a local instance of the lungparenchyma. A block is considered as valid when at least95% of its voxels belong to it. To obtain a sufficient numberof blocks for the middle right lobe, this rule was changed to90% due to its smaller size when compared to the other lobes.A total of 9,347 blocks are obtained for all lobes of all pa-tients. The classes of all blocks from a given lobe are definedas embolism if QI>0. Only blocks from control cases areused to represent the healthy class, as the redistribution of theblood flow in healthy lobes of embolism patients may alterparenchymal texture properties. Since the visual appearanceof the healthy parenchyma may vary from upper and lowerlobes due to gravity effects and anatomy, each lobe type(i.e., left, right, lower, middle, upper) is analyzed separately.Grey–level histograms (GLH) in Hounsfield Units (HU) ofthe extended lung window [1000; 512] HU with Nbins=50bins are used to characterize X–ray attenuations. GLH areproviding essential information to characterize hypo– andhyper–attenuated regions which respectively result fromunder–perfused oligemic areas of the lung or from shunt-ing of blood to irrigated parenchyma. The feature space iscomposed of the concatenation of the HU histogram bins andthe energies of the Riesz coefficients for all 11 DECT energylevels with a total dimensionality of 11 (Nbins +

N+22

).

SVMs with a Gaussian kernel are used to learn from themulti–energy features. Several parameters are exhaustivelyoptimized for each lobe: N , t, the cost C of the errorsof SVMs and the variance K of the associated Gaussiankernel K(xi,xj) = exp(

||xixj ||2122 ) as: N 2 [1; 4],

t = 0.75, 1.5, 3, C 2 [10

0; 10

18] and K 2 [10

0; 10

13].

A leave–one–patient–out cross–validation is used to estimatethe generalization performance of the proposed approach.

2http://www.osirix-viewer.com/, as of 31 October 2012.

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 10

RN

f

(x) =

0

BBBBBB@

R(0,N)

f

(x)

...R(n,Nn)

f

(x)

...R(N,0)

f

(x)

1

CCCCCCA

| z RNf(x)

=

0

BBBBBB@

A

0,0

. . . A

0,l

. . . A

0,N

......

...A

n,1

. . . A

n,l

. . . A

n,N

......

...A

N,1

. . . A

N,l

. . . A

N,N

1

CCCCCCA

| z A

0

BBBBBB@

R(0,N) f (x)...

R(l,Nl) f (x)...

R(N,0) f (x)

1

CCCCCCA

| z RNf(x)

.

(12)

ˆR(n,Nn)

=

N

n!(N n)!

(j(cos()!

1

+ sin()!

2

))

n

(j( sin()!

1

+ cos()!

2

))

Nn

||!||N

=

1

||!||N

N

n!(N n)!

nX

k1=0

Çn

k

1

å(cos())

k1(j!

1

)

k1(sin())

nk1(j!

2

)

nk1

NnX

k2=0

ÇN n

k

2

å(cos())

k2(j!

1

)

k2(sin())

Nnk2(j!

2

)

Nnk2

=

1

||!||N

N

n!(N n)!

nX

k1=0

NnX

k2=0

(j!

1

)

k1+k2(j!

2

)

Nk1k2

(1)

k2

Çn

k

1

åÇN n

k

2

å(cos())

Nnk2+k1(sin())

nk1+k2

=

nX

k1=0

NnX

k2=0

N

(k

1

+ k

2

)!(N k

1

k

2

)!

(j!

1

)

k1+k2(j!

2

)

Nk1k2

||!||N| z

ˆR(k1+k2,Nk1k2) (k

1

+ k

2

)!(N k

1

k

2

)!

n!(N n)!

(1)

k2n!

k

1

!(n k

1

)!

(N n)!

k

2

!(N n k

2

)!

(cos())

Nnk2+k1(sin())

nk1+k2.

(13)

ˆR(n,Nn)

(!) =

NX

l=0

N

l!(N l)!

(j!

1

)

l

(j!

2

)

Nl

||!||N| z

ˆR(l,Nl)

min(l,n)X

l1=max(0,lN+n)

(1)

ll1

l!(N l)!

l

1

!(n l

1

)!(l l

1

)!

(cos())

Nn+2l1l

(sin())

n2l1+l

| z A

n,l

.

(14)

[32] T. Leung and J. Malik. Representing and recognizing the visualappearance of materials using three–dimensional textons. InternationalJournal of Computer Vision, 43(1):29–44, 2001.

[33] L. Liu, L. Zhao, Y. Long, G. Kuang, and P. Fieguth. Extended localbinary patterns for texture classification. Image and Vision Computing,30(2):86–99, 2012.

[34] D. G. Lowe. Distinctive image features from scale-invariant keypoints.International Journal of Computer Vision, 60(2):91–110, 2004.

[35] S. G. Mallat. A theory for multiresolution signal decomposition: thewavelet representation. IEEE Transactions on Pattern Analysis andMachine Intelligence, 11(7):674–693, 1989.

[36] F. G. Meyer and R. R. Coifman. Brushlets: A tool for directional imageanalysis and image compression. Applied and Computational HarmonicAnalysis, 4(2):147–187, 1997.

[37] T. Ojala, T. Maenpaa, M. Pietikainen, J. Viertola, J. Kyllonen, andS. Huovinen. Outex – new framework for empirical evaluation oftexture analysis algorithms. In 16th International Conference on PatternRecognition, volume 1 of ICPR, pages 701–706, 2002.

[38] T. Ojala, M. Pietikainen, and T. Maenpaa. Gray scale and rotation

invariant texture classification with local binary patterns. In ComputerVision — ECCV 2000, volume 1842 of Lecture Notes in ComputerScience, pages 404–420. Springer Berlin Heidelberg, 2000.

[39] T. Ojala, M. Pietikainen, and T. Maenpaa. Multiresolution gray–scale and rotation invariant texture classification with local binary pat-terns. IEEE Transactions on Pattern Analysis and Machine Intelligence,24(7):971–987, 2002.

[40] P. Perona. Deformable kernels for early vision. IEEE Transactions onPattern Analysis and Machine Intelligence, 17(5):488–499, 1995.

[41] D. D. Y. Po and M. N. Do. Directional multiscale modeling of imagesusing the contourlet transform. IEEE Transactions on Image Processing,15(6):1610–1620, 2006.

[42] R. Porter and N. Canagarajah. Robust rotation–invariant texture classi-fication: wavelet, Gabor filter and GMRF based schemes. IEE Proc. onVision, Image and Signal Processing, 144(3):180–188, 1997.

[43] X. Qian, X.-S. Hua, P. Chen, and L. Ke. PLBP: An effective localbinary patterns texture descriptor with pyramid representation. PatternRecognition, 44(10–11):2502–2515, 2011.

[44] G. Quellec, M. Lamard, G. Cazuguel, B. Cochener, and C. Roux.

[24] GPU-accelerated texture analysis using steerable Riesz wavelets, Vizitiu et al., 11th Int Conf Par Proc and App Math (PPAM), 2015 (submitted)[25] A unifying parametric framework for 2D steerable wavelet transforms, Unser et al., SIAM Jour Imag Sci, 6(1):102-35, 2013 [26] Harmonic Singular Integrals and Steerable Wavelets in , Ward et al., App and Comp Harm Anal, 36(2):183-197, 2014

Page 45: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• Multi-scale

• Influence of surrounding objects: bandlimitedness VS compact support [27]

• Continuous band-limited scale characterization [28]

• Dyadic is not enough!

LIMITATIONS AND FUTURE WORK

45

Optimized Steerable Wavelets for Texture Analysis of Lung Tissue in 3-D CT: Classification of Usual Interstitial Pneumonia

Adrien Depeursinge1,2, Pedram Pad1, Anne S. Chin3, Ann N. Leung3, Daniel L. Rubin3, Henning Müller2, Michael Unser1

1Biomedical Imaging Group, EPFL, Switzerland, 2Institute of Information Systems, HES-SO, Switzerland, 3Department of Radiology, Stanford University, CA, USA.

Contact and more information: [email protected], http://bigwww.epfl.ch/

Background

•  An accurate identification of classic usual interstitial pneumonia (UIP) in volumetric CT images can obviate the need for invasive lung biopsies [1]

•  Identification of classic UIP:

Proposed scientific contributions

•  Automated identification of classic UIPs based on regional multi-scale texture analysis in CT

•  Balance between bandlimitedness of the texture operators and influence of surrounding objects

•  Extension of maximally localized isotropic wavelet pyramids proposed in [2] to 3-D and for an adjustable radial bandwidth

window, can be seen in aminority of otherwise typical cases ofUIP23 (Fig. 3) and should be distinguished from the non-fibrotic linear and nodular pattern of calcification seen indendriform pulmonary ossification.24 The fibrotic pattern ofUIP is quite often asymmetric but never unilateral (Fig. 6).One of themore challenging features of diagnosingUIPwith

HRCT is determining the presence of honeycombing. A studyby Watadani et al25 showed that substantial interobservervariability exists for identifying honeycombing on HRCT, withkappa values ranging from 0.40-0.58. However, it is not clearwhether this study, which scored only single images, andselectively included atypical cases of honeycombing, is appli-cable to more general assessment of honeycombing. Thedefinition of honeycombing used by the Fleischner Society(clustered cystic airspaces that are usually subpleural withwell-defined walls and often with comparable diameters of3-10 mm but occasionally as large as 2.5 cm)26 should becarefully applied. Honeycombing must be distinguished fromparaseptal emphysema (which is usually associated with largercysts and usually more prominent in the upper lungs and

Figure 2 UIP pattern in a 75-year-old man. (A and B) Axial and coronal CT images show peripheral-predominant, basal-predominant reticular abnormalitywith honeycombing, typical forUIP. The honeycombing ismore evident on the coronalreconstructions. (C and D) Axial and coronal images obtained 3 years later show substantial progression of reticularabnormality and honeycombing.

Figure 3 UIP pattern with ossification. Axial CT image shows typicalfindings of UIP with peripheral-predominant reticular abnormalityand honeycombing. Numerous punctate calcifications are presentwithin the fibrosis.

D.A. Lynch and J.M. Huckleberry14

2 Depeursinge et al.

UIP, however, are found in a substantial proportion of cases ranging from 30%to 50% [2]. In this context, candidate selection for lung biopsy requires a multi-disciplinary consensus of clinicians and radiologists with extensive experience inintersitial lung diseases.

The classic computed tomography (CT) appearances of UIP is characterizedwith basal– and peripheral–predominant reticular abnormality and honeycomb-ing [2] (see Table 1). The fibrotic pattern of UIP is often asymmetric but rarelyunilateral. Importantly, a confident CT diagnosis also requires the absence ofatypical UIP findings. For example, disease that is predominant in the lungapices is atypical for UIP and would suggest other diagnoses such as sarcoidosis,hypersensitive pneumonia, pneumoconiosis or familial pulmonary fibrosis. Pat-terns of fibrosis with anterior predominance are related to respiratory distresssyndrome. Therefore, the accurate identification of classic UIP requires a de-tailed description of the parenchymal alterations and their anatomical locations,which can only be established by experienced thoracic radiologists. The charac-terization of lung tissue types such as honeycombing, reticulation and groundglass requires the subtle appreciation of three–dimensional tissue texture proper-ties (see Fig. 1), for visual inspection has provided little reproducibility [4]. Theimportance of relating these patterns to their anatomical location adds anotherlevel of complexity and is subject to high inter-observer variation.

Table 1. Radiological criteria for UIP [1].

Classic UIP (all required) Inconsistent with UIP (any)

• Peripheral, basal predominance • Upper or mid–lung predominance

• Reticular abnormality • Peribronchovascular predominance

• Honeycombing with or withouttraction bronchiectasis

• Extensive ground glass abnormality(extent > reticular abnormality)

• Absence of features listed asinconsistent with UIP pattern

• Profuse micronodules (bilateral, predominantlyupper lobes)

• Discrete cysts (multiple, bilateral, away fromareas of honeycombing)

• Di↵use mosaic attenuation/air–trapping(bilateral, in three or more lobes)

• Consolidation in bronchopulmonary segment(s)/lobe(s)

normal ground glass reticular honeycombing

Fig. 1. Common parenchymal appearances of UIP in MDCT.

The computerized recognition of lung tissue types in chest CT has been anactive research domain to provide assistance in image interpretation and enhancediagnosis sensitivity and specificity [5]. Whereas most of the studies are based onslice–based 2–D texture analysis, few of them are fully leveraging the wealth ofmodern multiple detector CT (MDCT) protocols using 3–D solid texture anal-

i) tissue type ii) tissue location

peripheral

basal

Experiments Patients •  33 patients: 15 classic and 18 atypical UIP (biopsy proven) •  Volumetric CT resampled to 0.6x0.6x0.6mm3 (cubic spline interp.)

3-D rotation-covariant texture analysis of the parenchyma •  3-D Riesz-wavelet frames, defined in the Fourier domain as:

where is the 3-D Fourier transform of

•  can be viewed as a multiscale gradient operator when coupled with an isotropic wavelet pyramid

•  Yields a steerable filterbank •  Rotation-covariance (represented by ) is achieved by

steering the Riesz operator at each voxel to maximize its local energy [3]

Regional lung texture analysis and UIP classification •  Average energies of the coefficients computed for each 36

subregions of a basic lung atlas including peripheral and basal

•  A generalized linear model estimates the class membership of as , where denotes classic UIP

•  10-fold cross-validation (200 repetitions) to estimate classification performance (AUC and ACC based on and respectively)

References

[1] An Official ATS/ERS/JRS/ALAT Statement: Idiopathic Pulmonary Fibrosis: Evidence-based Guidelines for Diagnosis and Management, G. Raghu et al., Am J Respir Crit Care Med 2011; 183(6):788-824

[2] VOW: Variance Optimal Wavelets for the Steerable Pyramid, P. Pad et al., IEEE ICIP 2014; 2973-2977 [3] 3D Steerable Wavelets and Monogenic Analysis for Bioimaging, N. Chenouard et al., IEEE ISBI 2011;

2132-2135 [4] A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients, J. Portilla et al.,

Int Jour Comput Vision 2000; 1: 49-70 [5] Nonseparable radial frame multiresolution analysis in multidimensions and isotropic fast wavelet

algorithms, M. Papadakis et al., SPIE Wavelets 2003; 5207: 631-642 [6] Ten Lectures on Wavelets, I. Daubechies, SIAM 1992; 61

Methods

Adjustable maximally localized isotropic wavelet pyramids

•  3-D extension of [2] to be maximally localized over an adjustable radial bandwidth ,

•  Minimize , where is the radial profile

•  Constraints: •  bandwidth limited to •  generates tight frames

•  Analytical approximation in Fourier:

Results

•  The proposed texture operator achieved best AUC and ACC for and outperformed other popular isotropic wavelet pyramids

Conclusions and Perspectives

•  New family of 3D isotropic wavelet pyramids with a tunable bandwidth to balance between the bandlimitedness of the texture operators and the influence of surrounding objects

•  Importance of rotation-covariance and tuning of the wavelet bandwidth demonstrated for UIP classification

•  Future work includes the extension of the texture operators to higher orders of the Riesz transform

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

π/4 3π/8 π/2

Ω

. .

. .

. .

. .

. .

. .

[27] Optimized steerable wavelets for texture analysis of lung tissue in 3-D CT: classification of usual interstitial pneumonia, Depeursinge et al., IEEE Int Symp on Biomed Imag (ISBI), 403-6, 2015[28] Fast detection and refined scale estimation using complex isotropic wavelets, Püspöki et al., IEEE Int Symp on Biomed Imag (ISBI), 512-5, 2015

spatial domain Fourier

Page 46: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

(a) Synthetic image containing 3 visual concepts:

1) vertical lines (quadrants I and III),

2) checkerboard (quadrant II),

3) wiggled checkerboard (quadrant IV).

PCA 1

PC

A 2

101

102

103

(b) PCA visualization of 3232 overlapping blocks and clus-

ters from the left image (N = 10, J = 4, K = 3). The tem-

plates

10k corresponding to the respective visual concepts are

dislayed for scale j = 3.

Figure 5: Qualitative evaluation of the visual concepts 10

k found using K–means in the feature space spannedby the energies of the multi–scale Riesz components.

Figure 6: Classification accuracy with the Outex TC 00010 test suite. An optimal number of visual conceptsK = 20 and order N = 4 allowed an accuracy of 97.5%.

• Model learning

• Limited performance for stochastic textures with no clear multi-scale signature

• Reveal visual diversity with unsupervised learning [29,30]

LIMITATIONS AND FUTURE WORK

46

[29] Rotation-covariant visual concept detection using steerable Riesz wavelets and bags of visual words, Depeursinge et al., SPIE Wavelets and Sparsity XV, 8858:885816-885816-11, 2013 [30] Unsupervised texture segmentation using monogenic curvelets and the Potts model, Storath et al., IEEE Int Conf Imag Proc, 4348-52, 2014

(a) Synthetic image containing 3 visual concepts:

1) vertical lines (quadrants I and III),

2) checkerboard (quadrant II),

3) wiggled checkerboard (quadrant IV).

PCA 1

PC

A 2

101

102

103

(b) PCA visualization of 3232 overlapping blocks and clus-

ters from the left image (N = 10, J = 4, K = 3). The tem-

plates

10k corresponding to the respective visual concepts are

dislayed for scale j = 3.

Figure 5: Qualitative evaluation of the visual concepts 10

k found using K–means in the feature space spannedby the energies of the multi–scale Riesz components.

Figure 6: Classification accuracy with the Outex TC 00010 test suite. An optimal number of visual conceptsK = 20 and order N = 4 allowed an accuracy of 97.5%.

IEEETRANSACTIONSONIMAGEPROCESSING,VOL.XX,NO.XX,XX20135

1)canvas0012)canvas0023)canvas0034)canvas0055)canvas0066)canvas0097)canvas0118)canvas021

9)canvas02210)canvas02311)canvas02512)canvas02613)canvas03114)canvas03215)canvas03316)canvas035

17)canvas03818)canvas03919)tile00520)tile00621)carpet00222)carpet00423)carpet00524)carpet009

Fig.5.128128blocksfromthe24textureclassesoftheOutexdatabase.

1)canvas2)cloth3)cotton4)grass5)leather6)matting7)paper8)pigskin

9)raffia10)rattan11)reptile12)sand13)straw14)weave15)wood16)wool

Fig.6.16BrodatztextureclassesoftheContribTC00000testsuite.

180180imagesfromrotationangles20,70,90,120,135and150oftheothersevenBrodatzimagesforeachclass.Thetotalnumberofimagesinthetestsetis672.

G.ExperimentalsetupOVASVMmodelsusingGaussiankernelsasK(x

i

,x

j

)=

exp(

||xix

j||2

22k

)areusedbothtolearntexturesignaturesandtoclassifythetextureinstancesinthefinalfeaturespaceobtainedafterkiterations.AnumberofscalesJ=6

wasusedtocoverthewholespectrumofthe128128

subimagesinOutexandJ=3forcoveringthespectrumof1616subimagesinContribTC00000.Theanglematrixthatmaximizestheresponseofthetexturesignatureatthesmallestscale

1

(x)(seeEq.(11))isusedtosteerRiesztemplatesfromallscales.ThedimensionalityoftheinitialfeaturespaceisJ(N+1).Everytexturesignature

N

c,Kiscomputedusingthetextureinstancesfromthetrainingset.Thecoefficientsfromallinstancesarerotatedtolocallyaligneachsignature

N

c,Kandareconcatenatedtoconstitutethefinalfeaturespace.ThedimensionalityofthefinalfeaturespaceisJ(N+1)N

c.OVASVMmodelsaretrainedinthisfinalfeaturespaceusingthetraininginstances.Theremainingtestinstancesobtainedareusedtoevaluatethegeneralizationperformance.AlldataprocessingwasperformedusingMAT-LABR2012b(8.0.0.783)64–bit(glnxa64),TheMathWorks

Inc.,2012.Thecomputationalcomplexityisdominatedbythelocalorientationof

N

cinEq.11,whichconsistsoffindingtherootsofthepolynomialsdefinedbythesteeringmatrixA

.ItisthereforeNP–hard(Non–deterministicPolynomial–timehard),wheretheorderofthepolynomialsiscontrolledbytheorderoftheRiesztransformN.

III.RESULTS

TheperformanceofourapproachisdemonstratedwiththeOutexandtheBrodatzdatabases.TheperformanceoftextureclassificationisfirstinvestigatedinSectionIII-A.Theevolutionandtheconvergenceofthetexturesignatures

N

c,kthroughiterationsk=1,...,10isthenstudiedinSectionIII-BfortheOutexTC00010testsuite.

A.Rotation–covarianttextureclassification

Therotation–covariantpropertiesofourapproachareeval-uatedusingOutexTC00010,OutexTC00012andCon-tribTC00000testsuites.Theclassificationperformanceoftheproposedapproachaftertheinitialiteration(k=1)iscomparedwithtwootherapproachesthatarebasedonmultiscaleRieszfilterbanks.Asabaseline,theclassificationperformanceusingtheenergyofthecoefficientsoftheinitialRiesztemplateswasevaluated.Sincethecardinalityofthe

Page 47: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

• THANKS !

47

Matlab code [email protected]

MICCAI tutorial on Biomedical Texture Analysis: Oct 5th in Münich

https://sites.google.com/site/btamiccai2015/

Page 48: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

BIOMEDICAL TISSUE MODELING IN 2D AND 3D

• Interstitial lung diseases in CT • Lung texture classification using locally-oriented Riesz components, Depeursinge A, Foncubierta-

Rodriguez A, Van de Ville D, Müller H, Med Image Comput Comput Assist Interv. (MICCAI) 2011;14(3):231-8.

• Multiscale lung texture signature learning using the Riesz transform, Depeursinge A, Foncubierta-Rodriguez A, Van de Ville D, Müller H, Med Image Comput Comput Assist Interv. (MICCAI) 2012;15(3):517-24.

• Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high-resolution CT, Depeursinge A, Chin A, Leung A, Terrone D, Bristow M, Rosen G, Rubin D, Invest Radiol.,in press.

• Pulmonary embolism in dual-energy CT • Rotation-covariant texture analysis of 4D dual-energy CT as an indicator of local pulmonary

perfusion, Depeursinge A, Foncubierta-Rodriguez A, Vargas A, Van de Ville D, Platon A, Poletti PA,

48

Page 49: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

BIOMEDICAL TISSUE MODELING IN 2D AND 3D

• Liver lesions in CT • Predicting visual semantic descriptive terms from radiological image data: preliminary results with

liver lesions in CT, Depeursinge A, Kurtz C, Beaulieu C, Napel S, Rubin D, IEEE Trans Med Imag. 2014;33(8):1669-76.

• Brain epileptogenic lesions in MRI • Epileptogenic lesion quantification in MRI using contralateral 3D texture comparisons, Jiménez del

Toro OA, Foncubierta-Rodríguez A, Vargas Gómez MI, Müller H, Depeursinge A, Med Image Comput Comput Assist Interv. (MICCAI) 2013;16(2):353-60.

49

Page 50: Texture-Based Computational Models of Tissue in Biomedical Images: Initial Experience With Digital Histopathology

EVEN VS ODD ORDERS: 1-D

signal: Heaviside

filter: 1st (dashed) and 2nd

order Gaussian derivatives

convolution: 1st (dashed) versus 2nd

50