1 modélisation de linteraction avec objets déformables en temps-réel pour des simulateurs...

1

Modélisation de l’interaction avec objets déformables en temps-réel pour des simulateurs médicaux

Diego d’AulignacGRAVIR/INRIA Rhone-AlpesFrance

2

Medical Simulators

Motivations danger to patients cost certification

Objectives Geometric Models Physical Models

deformationinteraction

3

Problems

Simulation MUST be real-time! deformation resolution

Simulation MUST be realistic! model identification of parameters

Simulation MUST be interactive! collision detection haptic interaction

4

Plan

Deformation Models Mass-Spring vs. FEM

Real-time Resolution Techniques Static Dynamic

Echographic Simulator parameter identification

Liver Model interactive deformation

5

Deformable Object

GeometryElements

Springs [TW90] Tetrahedra FEM [OH99]

Comparison Realism Speed

6

Geometrical Model

56 surface points108 triangles57 total points120 tetrahedra230 edges

7

Mass-Spring Model

20

20

2

2L

LLE

:Strain

Initial length

Deformed length

8

Finite Element Method (FEM)

displacements

Small strain

Green’s strain

Cauchy Strain:

a

x

Deformation tensor:

Initial configuration Deformed configuration

a x

9

Strain-Stress

Snt

:Traction

EES

: stressLinearE

WS

ij

d

kijkkij

21

Lamé coefficients

force per unit area

Deformation Energy

10

Mass-Spring Model

•Springs are placed along the edges (230)

•Not very realistic: modeling a volume with springs!

•The force of each spring relatively cheap to evaluate

•globally fast

11

Finite Element Method (FEM)

•120 tetrahedra using Green’s strain tensor

•Continuum is modeled with volumetric element.

•Dilatation may be controlled

•Approximately four times slower than mass-spring network

12

Deformable Models (conclusions)

Mass-Spring One dimentional elements Unrealistic to model volume

Tetrahedral FEM Good realism for 3D continuum Control of dilatation Approximately 4 times slower to

evaluate forces

13

Contributions

Quantitative and qualitative comparison of mass-springs and tetrahedral elements

Interactive non-linear static resolutionFormal analysis of the real-time stability of

integration methods based on parameters

Identification of the parameters of a model from experimental data

Relevant medical applications

14

Plan

Deformation Models Mass-Spring vs. FEM

Real-time Resolution Techniques Static Dynamic

Echographic Simulator parameter identification

Liver Model interactive deformation

15

Real-time Resolution

Static Resolution linear resolution [Cotin97]

small displacements

Our approach: non-linear resolutionlarge displacements

Dynamic resolution explicit [Picinbono01] implicit [BW98]

externalfKuuDuM

externalfKu

16

Linear Static Resolution

Principle of virtual work:

internal and external forces are balanced

externalfKu

Linear case:• Pre-inversion (if enough space)

• No large strain

• No rotation

• No material non-linearity

17

Nonlinear Static Resolution

externalfuuK )(Non-linear case:•Stiffness matrix changes with displacement:

•geometric

•material

18

Newton Iteration

Full Newton-Rapson method:

•Reevaluation of Jacobian

•Faster convergence

Modified Newton-Rapson method:

•Constant Jacobian

•Slower Convergence

19

Dynamic Analysis

externalfKuuDuM

u

uY

u

uYf

)(

2nd order non-linear differential equation

Convertto

1st ordersystem

20

Explicit Integration

s

iijij kahYfk

10

Runge-Kutta method with s stages

0

2

4

6

8

10

1 2 3 4 5 6 7 8

Order of consistency (accuracy) vs. stages

j

s

jjkbYY

1

01

s

precision

21

Explicit Integration Stability

hM

K

M

D

M

Dhz

2

22

Im

Re Timestep is limited by the

the physical parameters!

externalfKuuDuM linearizing

22

Implicit Integation

011 )( YYhfY

)(1

)()(

0)()(

0

0

0

0 YffIh

Y

hYfYfhY

YfYfhY

YY

linearisation

Semi-implicit euler

B-stable implicit euler:

Stable for linear case (A-stable)

any timestep

any physical parameters

If you know your history, then you would know where you are coming from.

Bob Marley

Over-damped case

YYf )(

0Y

1Y

h

Y

23

Resolution (conclusions)

Static analysis non-linear resolution for large displacements

Dynamic explicit

strict stability criteria

implicitno limit on timestep, but resolution of non-linear

system

24

Contributions

Quantitative and qualitative comparison of mass-springs and tetrahedral elements

Interactive non-linear static resolutionFormal analysis of the real-time stability of

integration methods based on parameters

Identification of the parameters of a model from experimental data

Relevant medical applications

25

Plan

Deformation Models Mass-Spring vs. FEM

Real-time Resolution Techniques Static Dynamic

Echographic Simulator parameter identification

Liver Model interactive deformation

26

Thigh Echography

27

Echographic Simulator

Data AcquisitionModel of the thigh

Mass-Spring Neural

Interaction collision haptics

Generation of echographic image

baxxxf

)( axxf )(

28

Data Acquisition

64 sample points are marked on the thigh. For each, the forces for some given penetrations are measured

Two different probes

(a) Indentor shaped probe for punctual force-penetration data

(b) Probe with surface equal to that of a typical echographic probe

Two different probes

(a) Indentor shaped probe for punctual force-penetration data

(b) Probe with surface equal to that of a typical echographic probe

1- The end effector advances in small steps (2mm) in the direction normal to the surface of the thigh.

2- The force depending on the penetration distance is measured

(at LIRMM, Montpellier)

29

Data Acquisition: Data Acquisition: Experimental ResultsExperimental Results

The two probes do not offer the same resistance The two probes do not offer the same resistance difference in surface areadifference in surface area

Different curves for different pointsDifferent curves for different points different depth of soft tissuedifferent depth of soft tissue

Highly non-linear behaviour Highly non-linear behaviour

Indentor probeIndentor probe Surface probeSurface probedisplacement

Forc

e

displacement

Forc

e

[d’Aulignac et al.

MICCAI 99]

30

Echographic Simulator

Data AcquisitionModel of the thigh

Mass-Spring Neural

Interaction collision haptics

Generation of echographic image

baxxxf

)( axxf )(

Dynamic Model of the thighDynamic Model of the thigh

baxxxf

)( axxf )(

Incompressibility of the tissue

Elasticity of theepidermis

•Why mass-spring model?

•computationally efficient

•interior NOT discretized into tetrahedra

0LLx

32

Identification of Identification of thethe Parameters of a Parameters of aDynamic ModelDynamic Model

New parameters (elasticity, plasticity, collision stiffness ...)New parameters (elasticity, plasticity, collision stiffness ...)

Desired behaviourDesired behaviour

BehaviourBehaviour

ErrorError

OptimizationOptimization AlgorithmAlgorithm

ModelModel ResolutionResolution --

Measurements

For each sample point, 10-12 deformation/force values with each probe

=> Total of ~1200 measurements.

Parameter EstimationParameter Estimation

Least-squares minimisation:Least-squares minimisation:

1. 1. find (find (a,ba,b) for ) for eacheach non-linear spring non-linear spring

2. 2. find (find (a,ba,b) for ) for eacheach non-linear spring, non-linear spring, andand ( (aa) for ) for allall linear springs linear springs

Error of the model with respect to the experimental data

=> Overall error less than 5%=> Overall error less than 5%

Distribution of Nonzero Error Values

baxxxf

)(

axxf )(

(in collaboration with UC Berkeley)

[d’Aulignac et al., IROS 99]

Error (N)

=> Avoid local minima=> Avoid local minima

34

Dynamic Analysis

Explicit integration Euler stability

too small timesteps• no real-time

...or large mass• slow movement• no gravity

Implicit integration Semi-Implicit Euler

constant Jacobian100 steps per second

• h=1/100 (i.e. real time)

)(1

00 YffIh

Y YY

hYfY )( 0

35

Dynamic Resolution

100 Hz using semi-implicit integration

36

Neural Networks

Forces acting on particles: f

Displacement of particles: uufM )(

•Static Analysis

•Multi-layer perceptron is a general approximizer

•Network is trained directly on experimental data

•back-propagation64 inputs and outputs

37

Neural NetworksDisplacement (mm)

Force (N)

Experimental data Neural Model

38

Mass-Spring vs. Neural Model

Mass-spring topology chosen

based on measurements

dynamic resolution semi-implicit (100 Hz)

Neural model no assuption on topology static resolution

very fastno change of topology

baxxxf

)( axxf )(

39

Echographic Simulator

Data AcquisitionModel of the thigh

Mass-Spring Neural

Interaction collision haptics

Generation of echographic image

baxxxf

)( axxf )(

40

Interaction

Collision Detection

Collision Response

Force Feedback

xxxFcollision

41

Collision Detection

Finds polygons in the OpenGL viewing frustrum

Detects collision between simple rigid body and any other object quickly

42

Collision Response

Inter-penetration distance must be computed

Generates large forces (bad for haptics)

xxxFcollision

Penalty forces [Hunt and Crossley 1975]

43

Haptics

Haptic devices require high update frequency typically around 1kHz

….which the simulation normally can’t meet 100 Hz (dynamic

model)

44

Haptic Interaction

Local approximation of the contact simple local model

running in a separate threadfast collision detectionfast force computation

[Balaniuk 99]

Haptic loop (1kHz):

collision detection and response with local model

Simulation Loop (100Hz):

deformation

global collision detection and response

positionLocal model update

45

Haptic Feedback

time

With local model

Without local model

[d’Aulignac et al. ,

ICRA, 2000]

forc

e

46

Echographic Simulator

Data AcquisitionModel of the thigh

Mass-Spring Neural

Interaction collision haptics

Generation of echographic image

baxxxf

)( axxf )(

47

Echographic Image Generation

64 images aquired on each sample point

Voxel Map 120 Mb

Interpolation fill in the blanks

Provide image any rotation any position

[Vieira01] (in collaboration with TIMC-IMAG, France)

48

Echographic Image Deformation

Problem structures deform

differentlyveinbone, etc.

segmentation

Linear deformation Possible extension:

precalculated deformation maps [Troccaz et al, 2000]

49

A first Prototype

50

Echographic Simulator (conclusions)

Data AcquisitionModel of the thigh

Mass-Spring Neural

Interaction local model

Generation of echographic image linear deformation

baxxxf

)( axxf )(

51

Contributions

Quantitative and qualitative comparison of mass-springs and tetrahedral elements

Interactive non-linear static resolutionFormal analysis of the real-time stability of

integration methods based on parameters

Identification of the parameters of a model from experimental data

Relevant medical applications

52

Plan

Deformation Models Mass-Spring vs. FEM

Real-time Resolution Techniques Static Dynamic

Echographic Simulator parameter identification

Liver Model interactive deformation

53

Keyhole Surgery

Surgery involves soft tissues

Need to model deformation

…in real-time!

simulation

54

Human Liver

•Interior composed of parenchyma

•Surounded by elastic skin or Glisson’s capsule

•Venous network

•Approximate weight: 1.5 kg

55

Liver Model

Geometry

Physical Model

Dynamic Analysis explicit integration stability

Static Analysis non-linear resolution

56

Geometrical Model

•187 Vertices

•370 Triangles

•299 Particles

•1151 Tetrahedra

•1634 Edges

GHS3D

57

Physical Model[Boux et al., ISER, 2000]

Heterogenous Non-linear:

skin Parenchyma

Weight distributed equaly on all particles (i.e. approximately 5g each)

dddFspring

)( 23

1

Strain

Str

ess

58

Explicit Integration

280

1

105)105(2)105(2 6

2

66

KDDhz

280 steps per second

mass 5 grams

hM

K

M

D

M

Dhz

2

22

59

Stability Analysis

Im

Re

280

1

105)105(2)105(2 6

2

66

KDD

hz

60

Simulation

AchitectureSGI Onyx2

Compexity370 facets1151 tetrahedra3399 springs

Frequency150Hz

•Explicit not stable!

•...large mass

61

Static Resolution

The large deformations of the organ during operation require non-linear resolution techniques.

62

Calculate forces on nodes

Evaluate stiffness matrix K? (analytically)

Iteratively solve linear system for displacements u

Ku = f

by successive over-relaxation (SOR)

until residual forces < epsilon through Newton-Rapson iteration

Iterative Solution

63

Modified Newton-Raphson

•Accurate solution (many SOR iterations) does not allow faster solution

•Inexact Jacobian limits convergence speed

•Of special importance for strong nonlinearities

resi

dual

iterations

64

Newton-Raphson

•Less iteration to converge then modified NR

•Exact Jacobian allows faster convergence

•Global time gain when solving linear system accurately

iterationsre

sidu

al

65

Pseudo-Dynamic

Interactive resolution of the non-linear system.

66

Result

1157 tetrahedraIterative non-linear resolution

Rotational invarience

(N.B. Real-time animation)

1157 tetrahedraIterative non-linear resolution

Rotational invarience

(N.B. Real-time animation)

60 NR iterations/sec on SGI Octane 175Mhz

Pseudo-dynamic

67

Liver Model (conclusions)

Physical Model mass-springs

Dynamic Analysis explicit integration unstable

Static Analysis interactive non-linear resolution

68

Summary

Physical Models Mass-Spring or FEM?

Resolution Static

linear or non-linear?

Dynamicexplicit or implicit?

Medical Simulators The choice of numerical methods must be guided

by the application!

69

Contributions

Quantitative and qualitative comparison of mass-springs and tetrahedral elements

Interactive non-linear static resolutionFormal analysis of the real-time stability of

integration methods based on parameters

Identification of the parameters of a model from experimental data

Relevant medical applications