autonomous driving validation using game engine technology · autonomous driving, these vehicles...

Autonomous Driving Validation Using Game Engine Technology

[email protected] © Tata Elxsi 2019 1

Co-Authors:

Jithin Jose, Tata Elxsi

Jiljith John, Tata Elxsi

Dashlin Jeno, Tata Elxsi

Arun R, Tata Elxsi

Muhammad Asif V, Tata Elxsi

Autonomous Driving Validation using Game Engine

Technology

Jihas Khan, Tata Elxsi



TABLE OF CONTENTS

Abstract ......................................................................................................................................................... 3

Introduction .................................................................................................................................................. 4

Challenges in Real-World Validation and Testing of Autonomous Driving ................................................... 5

What Can be Done about it? ......................................................................................................................... 7

Leveraging Game Engine Technology For Autonomous Driving validation .................................................. 8

Key highlights .............................................................................................................................................. 10

Conclusion ................................................................................................................................................... 14

About Tata Elxsi ........................................................................................................................................... 15

References .................................................................................................................................................. 16



ABSTRACT

In this paper, we present the various challenges and risks associated with real-world validation of AI-based

Autonomous Vehicles (AVs) and how the prototyping and production of these vehicles would benefit from

virtual validation. This paper provides statistical and graphical analysis of the potential risks and

challenges involved in AV validation and how these risks and challenges exponentially increase when

dealing with a fleet of such vehicles. In order to seek out an effective solution, we benchmarked various

Autonomous Driving (AD) virtual validation platforms based on several criteria essential for AD validation

and realized their limitations. This paper gives you a peek into the advantages of using game engine

technology to re-create a real-world test scenario fully in a virtual manner. A virtual validation tool

enabling automatic generation of millions of scenarios and even allowing manual creation of user-specific

scenarios will provide much-needed validation of AD systems before deployment.



INTRODUCTION

With the current advancements in technologies such as Artificial Intelligence (A.I.), it is clear that a future with fully autonomous vehicles is just around the corner. A fully autonomous vehicle has many advantages over its human-driven counterpart. While reduced accidents would be the primary advantage of autonomous driving, these vehicles will also have better fuel efficiency and reduced emissions. Hence, it is of no surprise that most automobile manufacturers are leaning towards this proposed future.

Due to the relative infeasibility of real-world validation and testing of autonomous driving solutions, the automotive sector leans towards the more practical solution of virtual simulation. Currently, many platforms enable virtual validation and testing of autonomous driving. CARLA is one such platform, which uses game engine technology. CARLA [1] possesses numerous realistic vehicles, pedestrians, infrastructures and sensors models required for creating a virtual environment, with road network creation possible using tools such as RoadRunner. IPG CarMaker [2] and Mechanical Simulation CarSim [3] are two other platforms that provide similar functionality with Sil, Mil, Hil, and Dil support and MatLab/Simulink interface. dSPACE ASM [4] supports a multitude of sensor models with support to only dSPACE platforms, while TESIS veDYNA [5] can be used in any hardware but requires modification of the plan model for functionality. TASS International PreScan [6] is a simulation platform that provides real-time physics and collisions but lacks the ability to use external models. VIRES VTD [7] is another platform that provides most of these features along with realistic visuals. With support to automotive standards such as OpenDRIVE and OpenCRG, Vires VTD provides robust features. Figure 1 shows the comparison between these platforms with respect to their vehicle dynamics models, sensor support, 3D rendering, real-time physics and collisions, actor maneuvers and behaviors, model libraries and hardware dependencies.

This paper discusses the shortcomings of real-world validation of testing of AD solutions, compares various

virtual validation platforms that currently exists in the market and discusses how a game engine can be

used to provide end-to-end solution for validation of Autonomous Driving algorithm with its powerful,

adaptable and fully customizable sensor models, environment simulation, vehicle models, vehicle

Figure 1: High Level Comparison between existing virtual autonomous driving validation platforms



dynamics, static and dynamic actors, map generation, physics simulation, and photorealistic rendering.

UE4 (Unreal Engine 4) is the gaming engine used to achieve the 3D photorealistic real-world rendering,

where sensors can perceive and provide precise ground truth data for every object/property in an

environment. The user has full control over customization and creation of edge/complex scenarios instead

of testing in a real-world environment. This, in turn, helps to accelerate the development, testing, and

deployment of Autonomous Vehicles.

CHALLENGES IN REAL-WORLD VALIDATION AND

TESTING OF AUTONOMOUS DRIVING

In real-world validation and testing of autonomous driving solutions, the test vehicle with the AD

algorithm is driven through various scenarios and conditions in the real world, for long distances and

durations, in order to collect data for training and testing the AD algorithm as well as to analyze its

performance. This approach has drawbacks to it, which are described below.

In order to train and validate the A.I. based AD algorithm in these vehicles, it needs to be exposed

to millions of scenarios. The sheer volume of scenarios required makes it impractical for real-world

validation and testing

These scenarios must include traffic conditions of various intensity, different road structure and

surface types, dynamic weather phenomenon, variations in temperature and lighting and the day-

night cycle. The creation of such scenarios in a real vehicle in the real world would be expensive

and time-consuming.

Further, these scenarios must also include dynamic interaction of the test vehicle with other

dynamic actors and a multitude of rare events. Construction works, road hazards, accidents,

pedestrians that suddenly cross the road and rogue vehicles that don’t follow the traffic rules are

a few examples of the rare events. These rare events raise additional challenges, as manufacturing

these scenarios in the real world is not a viable solution.

Another cause for concern regarding real-world testing is the multitude of health and safety risks

that need to be considered. The major issue here is the one thing that makes AD vehicles what

they are – the absence of the human driver. In a conventional vehicle or even one with Advanced

Driver Assistance System (ADAS) features, the driver can intervene at times of emergency. Further,

while testing of ADAS features if the said system does fail, the driver can assume control of the

vehicle. This is not the case when the driver is the software being tested. Scenarios that push

algorithm to its limits are fundamentals in any testing procedure. Figure 2 shows the number of

disengagements in AD test vehicles in California during the period from December 2017 to

November 2018 [8]. Miles per disengagement is a measure of distance traveled by the AD vehicle

(in miles) before the AD technology disengages due to the failure of the AD technology. This shows

the frequency of disengagements in AD test vehicles and hence the risk they pose during real-

world testing.



Figure 2: Number of AD disengagements in vehicles tested in California from Dec 2017 to Nov 2018

The creation of edge conditions where the AD vehicle is about to collide with a pedestrian or with

another vehicle would have obvious risks. If Ad technology were to fail or disengage, it could lead

to catastrophic outcomes

To collect sufficient data, these vehicles must be driven for long durations and long distances while

being controlled by their AD algorithm

The challenges presented above were merely those present for the validation and testing of a

single AD vehicle. Training and validating a fleet of AD vehicles would significantly multiply these

challenges. Further, these test scenarios must be repeated until the results obtained are of

statistical significance. This further shows the impracticality of validating and testing AD

algorithms in the real world. Figure 3 compares the miles per disengagement for AD test vehicles

in California during the period from December 2017 to November 2018. This comparison along

with the number of disengagements from Figure 2 provides us with an estimate of the amount of

distance that was covered for the testing of these vehicles. This vast distance traveled and the

amount of time taken to do so acts as limitations for real-world testing.

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300



Figure 3: Miles per disengagement for AD vehicles tested in California from Dec 2017 to Nov 2018 (Source:

thelastdriverlicenseholder.com)

WHAT CAN BE DONE ABOUT IT?

A more practical solution to Autonomous Driving validation and testing would be to train, test and validate

these algorithms in a virtual vehicle in a simulated environment. This will enable the creation of test

scenarios to be created in a controlled virtual world, which would otherwise be impossible to recreate in

the real world. Further being a virtual simulation, test automation and batch execution of test scenarios

enable accelerated validation and testing.

A game engine, that blurs the line between reality and simulations, facilitates the creation of the simulated

environment. We can use game engine technology for providing reliable ground truth data for

autonomous vehicle validation [9] and hence use it in HiL, SiL, ViL, PiL, MiL and DiL applications. Game

engines enable usage of complex collisions and realistic physics in order to simulate realistic interactions

between the ego vehicle (vehicle under test) and other vehicles, pedestrians and animals. Further, this

technology enables the creation of a vast variety of environments such as cities and rural areas, with

editable terrain. Environment simulation can also be extended to dynamic weather and lighting which

enables a vast range of weather and time-of-day simulations that can influence test parameters and ego

vehicle movement. This is done using realistic lighting and particle effects. Lastly, the high-end game



engine technology enables faithful visual recreation of vehicles, pedestrians and static objects. This

enables added visual realism to be present in a virtual environment. The end result is that the AD algorithm

would be tricked into believing that it truly is in the real world.

Figure 4: A virtual scenario visualized using Unreal Engine 4

LEVERAGING GAME ENGINE TECHNOLOGY FOR

AUTONOMOUS DRIVING VALIDATION

A test scenario would be a combination of values consisting of road details, environmental conditions, and

traffic details. According to these values, the proposed tool simulates a virtual environment. The tool

facilitates the creation of scenarios in two ways:

1) Manual - Creating the scenario using a graphical user interface, where the user can interact to create

the full scenario as per his/her requirements.

2) Automatic - An intelligent scenario generation algorithm generates unique and realistic scenarios based

on some rules and preconditions set by the user.

This scenario will have a road network in a defined environment with both static and dynamic actors. The

road network can either be created by the user or loaded from a standard format such as OpenDRIVE. It

would consist of various road geometries, tunnels, bridges, lane and road markings, road and roadside

objects, and traffic signs and signals. The static actors would consist of various static infrastructures like

buildings and vegetation, while the dynamic actors consist of the Ego and other POV vehicles, pedestrians

and animals. Finally the environment would consist of the terrain information, weather, and lighting.



The trajectory of the Ego vehicle could be pre-defined by the user or could be controlled during run-time

by an external algorithm using CAN, ROS, TCP, UDP and FMI modes of data transfer. The trajectories of the

other POV vehicles, pedestrians and animals are either pre-defined by the user in manual mode or defined

by the tool in automatic mode.

Figure 5: Rendered 2D Scenario

Figure 6: Rendered 3D Scenario



The created scenario is visualized using the Unreal Engine 4 (UE4) game engine. The use of a game engine

like UE4 facilitates faithful recreation of the created test scenario. Realism is achieved by the usage of

vehicle, pedestrian and animal models that have realistic aesthetics, physics, and behavior. This enables

faithful recreation of real-world interactions between these dynamic actors as in the case of accident

scenarios or vehicle and pedestrian detections. The tool will have an inbuilt vehicle dynamics model. The

user can also interface vehicle dynamics models of the user’s choice, e.g. MatLab models via a MatLab

interface, further enabling user customizability.

UE4 also facilitates modeling of various sensors: Ultrasonic, RADAR, LIDAR, Camera, GPS, IMU, and Wheel-

tick sensors. These sensors can be used as ideal models for benchmarking real sensors or can be used as

realistic models by the introduction of errors. These errors are generated by mathematical distributions

or using UE4’s dynamic simulation environment. Semantic segmentation and data annotation for each

frame of the rendered scenario is performed. The sensor outputs can be transferred through CAN, TCP,

UDP and ROS channels. The sensor models would also be FMI compliant, enhancing modularity.

Figure 7: LIDAR point cloud data visualization

KEY HIGHLIGHTS

Our analysis of the currently existing AD virtual validation platforms has shown that there is a need for a

solution that encompasses all the merits of each of these platforms and overcome their flaws. The

proposed Tata Elxsi solution does so while also having a set of aspects unique to it. These are:

Realistic 3D modeling and simulation –

The use of Unreal Engine 4 enables faithful 3D recreation of dynamic models and the

environment. These dynamic models include vehicle models that accurately represent their real-

world counterparts.

User-friendly GUI –

The solution facilitates quick and easy creation of test scenarios. This, combined with the user-

friendly nature of the GUI, which enables any user to create test scenarios without much training,

enables accelerated test scenario generation.



Automatic scenario generation –

The solution has the ability to automatically create scenarios based on a set of rules or parameters

defined by the user. The tool does so using a generation algorithm, which is fine-tuned so that

each generated scenario is logical, realistic and unique. The parameters defined by the user

include the type, count and behavior of both static and dynamic actors, using which the

generation algorithm generates a random but logical scenario. Figure 7 describes the approach

for automatic scenario generation.

Figure 8: Approach for automatic generation of scenarios based on user-defined parameters

Real-time physics and collisions –

Unreal Engine 4 provides the ability to simulate fast and accurate physics interactions between

actors and realistic collision detections. This enables the simulated scenario to have simulated

realism without the need for external models or plug-ins.

A vast and editable library –

Users will have access to a vast library of static and dynamic actors. Further, the user has the

ability to customize these actors or import a model of their choice into the library. This added

layer of customizability promotes the flexibility of the tool for each user.

Sensor simulation –

The solution facilities realistic and ideal sensor modeling of n-number of sensors of each type –

Ultrasonic, RADAR, LIDAR, Camera, GPS, IMU, and Wheel-tick.

Integration with external vehicle dynamics models –

Users can integrate external vehicle dynamics models, such as MatLab models through a MatLab

interface. This promotes further customizability of the solution to fit the user’s requirements.



Data annotation –

The solution provides data annotation output for each frame of the rendered scene. This output

contains information regarding the bounding box, location, and orientation of each actor in the

frame. This can be used as ground truth data required for validation and testing. Figure 6 show an

example of such an output. Figure 9 shows the annotated data output from the solution.

Figure 9: Annotated data output

Test automation –

Batch execution of test scenarios coupled with automatic scenario generation leads to the

solution accelerating the validation and testing process.

Modularity –

Each fundamental component of the solution is capable of being scaled, modified and interfaced

with as per the requirements. These fundamental parts include- scenario creation, road network

creation, sensor configuration, and scenario visualization.



SAMPLE USE CASE

This sample use case shows how a version of the proposed solution has been customized as per the

requirements of a leading Tier 1. This version has been customized to automatically generate a vast

number of scenarios that would then be used to train the AI algorithm present in an AD camera ECU. The

tool generates each frame of each rendered scenario along with a semantic segmented image and

annotated data for that frame. These would then be used as ground truth data for training the AI algorithm

implemented in the AD camera ECU. During the testing phase, the algorithm output for each scenario is

compared with the ground truth by the test automation framework. The test result is generated as a

report of the detected and expected data. The architecture of the proposed version is shown in Figure 10.

Figure 10: Test system architecture for AD camera ECU training



CONCLUSION

How many miles of road testing would an Autonomous Vehicle would require? How much time would

that take? Without simulation it is difficult to test all the testing scenarios an Autonomous Driving vehicle

would find in a real dynamic environment. This paper identifies existing challenges in the validation of an

AI-based Autonomous Driving (AD) algorithm. These challenges include the sheer volume of data to be

processed, the time and monetary constraints required to train and test these algorithms in real-world

environments. Further simulating corner and rare cases in the real-world would add further difficulty in

testing and validation. Users should have the ease of option to generate millions of AD scenarios

automatically facilitating faster scenario generation, which in turn would promote workload reduction.

The use of game engine technology enables rendering realistic 3D simulation, which helps attain higher

accuracy and efficiency. Further, due to the entire simulation process being in a virtual scenario, real-

world risks would be non-existent. Further to the game engine for validating AD technology, the option of

simulating sensor models and vehicle dynamics would be required to fully validate the AD controller

algorithm or the ECU. This simulated game engine based solution would result in an accelerated,

economical, efficient, and risk-free solution to AD testing and validation. The proposed method of using

game engine technology for validation of autonomous driving technology is well accepted by OEMs and

Tier 1s, for activities that were merely achievable using existing tools in the market.

A 3D simulation solution “VDRIVE” from Tata Elxsi which uses game engine technology overcomes the

challenges discussed in this paper and is an effective tool to test the complexity of AI-based Autonomous

Driving algorithms.



ABOUT TATA ELXSI Tata Elxsi is amongst the world’s leading providers of design and technology services for product design,

engineering, solutions, electronics and software for the Automotive industry. Accredited with Automotive

SPICE Level 5 certification, premium member of the AUTOSAR consortium and other industry partnerships

help us gain a competitive advantage in the global market place.

Tata Elxsi works with leading OEMs and suppliers in the automotive and transportation industries for R&D,

design and product engineering services from architecture to launch and beyond. Tata Elxsi stays invested

in developing solutions that would help clients achieve a faster time to market. Supported global brands

in the automotive space (OEMs and Tier-1 suppliers) for testing and validation of their programs. Tata

Elxsi has designed and developed 40+ HILS for testing single and multiple ECU’s (Infotainment, BCM,

Chassis, and Powertrain ECUs).



REFERENCES

1. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio López, and Vladlen Koltun. CARLA: An

Open Urban Driving Simulator

2. CarMaker: Virtual testing of automobiles and light-duty vehicles. Accessed on: Nov. 20, 2019.

[Online]. Available: https://ipg-automotive.com/products-services/simulation-

software/carmaker/

3. CarSim Mechanical Simulation. Accessed on: Nov. 20, 2019. [Online]. Available:

https://www.carsim.com/products/carsim/index.php

4. Automotive Simulation Models (ASM): Tool suite for simulating the engine, vehicle dynamics,

electrical system, and traffic environment, 2019. [Online]. Available:

https://www.dspace.com/shared/data/pdf/2019/dSPACE_ASM_Catalog2019.pdf

5. veDYNA: More efficiency in component and controller development with comprehensive vehicle

dynamics simulation, 2018. [Online]. Available:

https://www.tesis.de/fileadmin/Downloads/Flyer_Broschueren/veDYNA/Product_overview_ved

yna.pdf

6. preScan | TASS International. Accessed on: Nov 20, 2019. [Online]. Available:

https://tass.plm.automation.siemens.com/prescan

7. VIRES VTD Details 201403, March 2014. [Online]. Available:

https://www.vires.com/docs/VIRES_VTD_Details_201403.pdf

8. Mario Herger, UPDATE: Disengagement Reports 2018 – Final Results, Feb. 13, 2019. Accessed on:

Nov 20, 2019. [Online]. Available: https://thelastdriverlicenseholder.com/2019/02/13/update-

disengagement-reports-2018-final-results/

9. S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games.

In European Conference on Computer Vision (ECCV), 2016.

https://ipg-automotive.com/products-services/simulation-software/carmaker/

https://ipg-automotive.com/products-services/simulation-software/carmaker/

https://www.carsim.com/products/carsim/index.php

https://www.dspace.com/shared/data/pdf/2019/dSPACE_ASM_Catalog2019.pdf

https://www.tesis.de/fileadmin/Downloads/Flyer_Broschueren/veDYNA/Product_overview_vedyna.pdf

https://www.tesis.de/fileadmin/Downloads/Flyer_Broschueren/veDYNA/Product_overview_vedyna.pdf

https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftass.plm.automation.siemens.com%2Fprescan&data=02%7C01%7Carun.r1%40tataelxsi.co.in%7Cd6a6c39706bc4268aabb08d76cbc283d%7Cad6a39dd96b6436882daf2ec4d92e26a%7C0%7C0%7C637097427695010381&sdata=7iBFNX%2FVd6b5MtrNoXaoZQp%2FU8hI7sFQdx8%2FgVWHMeA%3D&reserved=0

https://www.vires.com/docs/VIRES_VTD_Details_201403.pdf

https://thelastdriverlicenseholder.com/2019/02/13/update-disengagement-reports-2018-final-results/

https://thelastdriverlicenseholder.com/2019/02/13/update-disengagement-reports-2018-final-results/

autonomous driving validation using game engine technology · autonomous driving, these vehicles...

Documents