point clouds from calibration to classification · calibration of point clouds. integrated...

Point CloudsFrom Calibration to Classification

Norbert Pfeifer

norbert.pfeifer@geo.tuwien.ac.at

Technische Universität Wien, Department for Geodesy and GeoinformationVienna, Austria

ContributionsGottfried Mandlburger (IfP Stuttgart, Germany), Philipp Glira (Siemens, Austria), Martin Pfennigbauer (Riegl, Austria), Konrad Wenzel (nframes, Germany), Tran Giang

Point Cloud Calibration and Orientation

2

Dense Image Matching (RGB)Airborne LiDAR

• 3D color vs. mainly geometry• Differences: vegetation, crane• Piles of construction material• DIM: stereo occlucion • LiDAR: points between piles available

DIMLiDAR

How can these point cloudsbe „brought together“in order to understand their differences?

Mandlburger et al.

Point Cloud Change Detection

4 data sets taken at 4 different times Different modality

RGB, nIR, photo/lidar, GSD, point density, shadow, …

Classified Lidar point cloudsground, building, vegetation

32005 2011

Computational Geometry – Point Cloud Legacy

Design Process• Given: Point cloud pi ∈R3, i = 1, … n• Find: surface s(u,v) closely interpolating pi , i.e. minimize f ( s(ui,vi) - pi )• NURBS (…) surface, patch layout based on curvature, curvature defined in each point

5Image: learning alias

Computational Geometry – Point Cloud Legacy

Design Process• Given: Point cloud pi ∈R3, i = 1, … n• Find: surface s(u,v) closely interpolating pi , i.e. minimize f ( s(ui,vi) - pi )• NURBS (…) surface, patch layout based on curvature, curvature defined in each point

Quality Inspection• Given: CAD model of a object, discretized as points mi

• Given: point cloud of the manufactured object, data points dj

• Find: transformation of dj onto mi , but no exact correspondence• Replace exact with approximate, iteratively determined correspondence: ICP

6Image: learning alias

Computational Geometry – Point Cloud Legacy

Design Process• Given: Point cloud pi ∈R3, i = 1, … n• Find: surface s(u,v) closely interpolating pi , i.e. minimize f ( s(ui,vi) - pi )• NURBS (…) surface, patch layout based on curvature, curvature defined in each point

Quality Inspection• Given: CAD model of a object, discretized as points mi

• Given: point cloud of the manufactured object, data points dj

• Find: transformation of dj onto mi , but no exact correspondence• Replace exact with approximate, iteratively determined correspondence: ICP

Visualization• Given: Point cloud pi

• Find TIN with vertices pi that follows surface of object• No 2D solution, e.g. Delaunay criterion,

but localized processing independent of overall shape

7

Point clouds

Computational Geometry Object centered (often) Point pi ( xi , yi , zi ) Cartesian coordinate system No datum No attributes Error free (negligible errors) No measurement process

8

Geodesy and Geoinformation Billions of points Projected CS Random (0.5-30cm), systematic (up

to meter), and many gross errors Attributes: color, accuracy, echo ID,

FWF information, time, etc. Measurement process known Measurement along optical line of

sight

Point cloud research domain

Data structures kD-tree

appropriate for processing, nearestneighbor search, etc.

octreeappropriate for visualization (e.g. potree)

Data structure for manuel editing, processing and visualization of pointclouds?

9

Geodesy and Geoinformation Billions of points Projected CS Random (0.5-30cm), systematic (up

to meter), and many gross errors Attributes: color, accuracy, echo ID,

FWF information, time, etc. Measurement process known Measurement along optical line of

sight

Point cloud research domain

Data structures kD-tree

appropriate for processing, nearestneighbor search, etc.

octreeappropriate for visualization (e.g. potree)

Data structure for manuel editing, processing and visualization of pointclouds?

Processing strategiesparallelizationtile wise processing

Neighborhoodoptimal neighborhoods, parameterselection, etc.

10

Geodesy and Geoinformation Billions of points Projected CS Random (0.5-30cm), systematic (up

to meter), and many gross errors Attributes: color, accuracy, echo ID,

FWF information, time, etc. Measurement process known Measurement along optical line of

sight

Point cloud research domain

Calibration and orientation sensor modeling measurement process modeling

sensor + medium/media + object observation error models

12

Geodesy and Geoinformation Billions of points Projected CS Random (0.5-30cm), systematic (up

to meter), and many gross errors Attributes: color, accuracy, echo ID,

FWF information, time, etc. Measurement process known Measurement along optical line of

sight

Glira et al.Image: Riegl

Point cloud research domain

Applications land cover mapping topographic modeling indoor modeling engineering surveying deformation analysis change detection

13

Geodesy and Geoinformation Billions of points Projected CS Random (0.5-30cm), systematic (up

to meter), and many gross errors Attributes: color, accuracy, echo ID,

FWF information, time, etc. Measurement process known Measurement along optical line of

sight

Calibration of point clouds

Integrated calibration and orientation of lidar and image matching point clouds

Where‘s the problem? Lidar point clouds are good within surfaces and for vegetation,

not influenced by (sun) shadows Image matching point clouds are good on edges and have high resolution

physical explanation: aperture size of current laser scanners vs. photogrammetric cameras

Aimcombine the advantages and exploit the differences

Requirementprecise geo-referencing

Standard methodindependent bundle block adjustment and laser scanning strip adjustment

14

Flight block Melk (Austria)

15

Abbey and city of Melkaerial image

Overview map

Flight strips

Mandlburger et al.

Results: data acquisition Melk

17

Dense Image Matching3D RGB point cloud

nframes

Results: data acquisition Melk

18

Airborne LiDAR3D point cloud

Riegl

Elevation differences with standard method

19

Height differences „LiDAR minus DIM“ before integrated strip adjustment / AT

0 100 200 m

Mandlburger et al.

Integration of LiDAR strip adjustment and aerotriangulation

20

Equations• Direct georeferencing• Collinearity• Point to tangent plane ICP

correspondence

Glira et al.

Results: data acquisition Melk

22

Height differences „LiDAR minus DIM“ before integrated strip adjustment / AT

0 100 200 m

Mandlburger et al.

Results: data acquisition Melk

23

Height differences „LiDAR minus DIM“ after integrated strip adjustment / AT

vegetation

vegetation vegetation

Mandlburger et al.

Results: data acquisition Melk

24

Height differences „Lidar minus DIM“ Masked height differences „LiDAR minus DIM“ after hybrid adjustment

0 100 200 m

Mandlburger et al.

DSM generation from LiDAR and DIM

25

LiDARDIM: Default settings DIM: Refinedsettings LiDAR+DIM

Precision (1σ):LiDAR: 2cmDIM: 10cm = 1.67 * GSD

Mandlburger et al.

LiDAR/DIM data properties: vegetation penetration

Vegetation: DIM=top of grass surface, LiDAR=penetrates grass layer Impenetrable surfaces: negligible height differences between DIM and ALS

26

colorized DIM point cloud height difference: LiDAR-DIM

beach volleyball court

soccer fieldrunning trackA BMandlburger et al.

Classification of point clouds

Point cloud classification Principle: use measured and computed features to infer,

which class a point belongs to (features selection?)

28

Echo widthEchoRatio Sigma0 Tran et al.

Classification of point clouds

Point cloud classification Principle: use measured and computed features to infer,

which class a point belongs to (features selection?) Methods: manual decision trees, support vector machines,

random forests, Markov chains, deep convolutional neuralnetworks

Considerations: transferability of rules (across missions, scales, etc.), number of classes, run time, amout of trainingdata, representation of rarely occuring classes, classification accuracy

29

Point cloud

Nor<0.2Ground Y OffGroundN

GreenRatio<0.3393

RoadRoad

Y

Grassland

N Nor<=1.5 Buildings and TreeNLowtree+Car Y

GreenRatio<0.3393 Nor<=5.5

Car LowTree

SmallBuildings and Med Tree

Y

High Buildings and high TreeN

GreenRatio<0.348

Smallbuilding

Y

Medium Tree

N

_omnivariance>1

High Tree

Y

Tree mix building

N

Nor<17

High Tree

Y

High Building

N

Y N

Tran et al.

Change detection in point clouds

A1: classify point clouds of different epochs + compare labels (proximity?) A2: compute geometric point cloud difference + classify changed points Problem 1

change in classand change in geometry can beindependent

Problem 2different dataproperties

Problem 33D changes

30

31

Cut tree

Ground change in height >0.5m

Reconstructed building

New buildings

Unchange

New tree

Ground to another landuse

2005

2011

PC CD envisaged

result

Tran et al.

PC CD Method

Exploit the power of machine learning Features for class and features for change1. measurement process: echo ID, waveform attribute, etc.2. local neighborhood: normal vector, roughness, etc.3. approximate height (simple terrain model)4. change indicators

• distance to nearest point in PC of other epoch• point distribution feature

evaluated at location of epoch 1 pointswith points of epoch 2

32Tran et al.

PC CD Method

Exploit the power of machine learning Features for class and features for change1. measurement process: echo ID, waveform attribute, etc.2. local neighborhood: normal vector, roughness, etc.3. approximate height (simple terrain model)4. change indicators

• distance to nearest point in PC of other epoch• point distribution feature

evaluated at location of epoch 1 pointswith points of epoch 2

Sample training data, learn model (e.g. random forests),

apply model to detect all changes,done (?)

33Tran et al.

PC CD ML

34Tran et al.

PC CD ML

35Tran et al.

36Tran et al.

Conclusions

Many open pointcloud researchquestions

37

References

Tran et al., 2018 (accepted): Classification of image matching point clouds over an urban area. International Journal of Remote Sensing.

Tran et al., 2018: Integrated Change Detection and Classification in Urban Areas Based on Airborne Laser Scanning Point Clouds. Sensors 18.

Mandlburger et al., 2017: Improved Topographic Models via Concurrent Airborne LIDAR and Dense Image Matching. ISPRS Annals IV-2/W4.

Glira et al., 2016: Rigorous Strip adjustment of UAV-based laserscanning data including time-dependent correction of trajectory errors. Photogrammetric Engineering & Remote Sensing 82.

Glira et al., 2015: A Correspondence Framework for ALS Strip Adjustments based on Variants of the ICP. Photogrammetrie, Fernerkundung, Geoinformation 2015.

Pfeifer et al., 2014: OPALS–A framework for Airborne Laser Scanning dataanalysis. Computers, Environment and Urban Systems 45.

Otepka et al., 2013: Georeferenced point clouds: A survey of features and point cloud management. ISPRS International Journal of Geo-Information 2.

38