ACQUA : Application-level Quality of Experience at
Internet Access
http://team.inria.fr/diana/chadi/
Joint work with
T. Speterbroot, S. Afra, N. Auguilera, A. Al Jalam, M. Jost, D. Saucez
Chadi Barakat - 2
Introduction to ACQUA
Application for prediCting Quality of User experience at Internet Access,
https://team.inria.fr/diana/acqua/
What is Quality of Experience (QoE) ?
- A subjective measure of human experience
Good, Medium, Poor … for an audio conversation
0, 1, 2, … for a video streaming
….
- Obtained by a panel of testers
How can QoE be estimated/predicted?
- By linking it to measurable metrics (Quality of Service or QoS)
- Application level, network level, device level, etc
Chadi Barakat - 3
ACQUA main features
Application level QoE: Skype, YouTube, etc
Measurements of network and device as input (QoS)
Expected Quality of Experience as output (QoE)
An application in ACQUA is a profile, a function, or a model
For Application 1 f1 (measurements) = QoE1
For Application 2 f2 (measurements) = QoE2
….
QoE prediction thanks to direct linking to network level QoS
- No need for applications to be running
Reutilization of measurements
- Measure network once, predict QoE for many applications
Chadi Barakat - 4
QoE vs. network QoS in ACQUA
Application
e.g. Skype
Vary artificially
network
performance
Write down QoE
Model for QoE (Decision Tree,
SVM, BayesianNet)
Model for QoE
Measure
network
performance
Expected QoE
Model Calibration Phase
QoE Estimation/Prediction Phase
Ask end users
for their
feedback
Write down QoE
and network
conditions
Controlled experimentation in the lab
Crowdsourcing
Chadi Barakat - 5
Network measurements in ACQUA
Path-level measurements
- Bandwidth, delay and loss, upload and download, …
Device-level measurements
- Signal strength, type of connection, traffic in/out, …
Measurements inside and from the device
Measurements to Landmarks
- Measurement servers
- Expected QoE per landmark
- Statistics of QoE over landmarks
- Troubleshooting by landmark elimination
- Dozen of landmarks for a good span of QoE
Internet
Internet
Chadi Barakat - 6
ACQUA in a nutshell
Appli 1
Appli 2
Appli N
Classification
Co
ntr
olle
d E
xp
eri
me
nta
tio
n
Cro
wd
so
urc
ing
QoE
Model 1
QoE
Model 2
QoE
Model N
Vary network
performance
vs. Network
Measurements
Learning/Calibration
Network and
device level
measurements
Measurements re-utilization
Measurements at end-user
Expected
Quality 1
Expected
Quality 2
Expected
Quality N
Troubleshooting by
analyzing Landmarks
Chadi Barakat - 7
Model calibration by controlled experimentation
Varying network conditions in ACQUA
Space of experimentation can be huge
- One dimension per performance metric
- Complexity power of the number of metrics
A two-layer approach for space sampling
- Fourier Amplitude Sensitivity Analysis (FAST) method for a fair coverage of the
space and for sample suggestion
- Active learning for sample acceptance/rejection (Vowpal Wabbit implementation)
Only accepted samples transform into scenarios to experiment with
Chadi Barakat - 8
DummyNet
Five measurable path metrics
- Bandwidth and loss rate,
both upload and download
- Round-trip delay
QoE = Skype quality meter
- Four levels
- Good, Medium, Poor, No Call
Controlled experimental setup
- DummyNet at access point
- Both ways
- Local Skype traffic
- Around 600 experiments
A local skype call
One network
configuration
(5 values)
QoE of Skype
(Good, Medium,
Poor, No Call)
One experiment
Experimenting with Skype
Controlled experimentation
Chadi Barakat - 9
Modeling Skype Quality of Experience
A variety of machine learning techniques
- Decision Tree, Naïve Bayesian, Lazy learner, Support Vector Machine, etc.
Focus on Decision Trees for their readability
Performance metrics: Precision and Recall per Quality class
Chadi Barakat - 10
Skype QoE prediction accuracy
- Best performance for the Medium class, no particular technique outperforming
- Almost 70% prediction accuracy (or recall) on average
Recall
Recall
Pre
cis
ion
P
recis
ion
Chadi Barakat - 12
Skype Quality Rules
Rule = set of branches from root to leaf
20 rules (after pruning)
- Rule 1: Download Bandwidth > 1078, Download Delay <= 94 class “Excellent” [84.1%]
- Rule 2: Upd Bandwidth > 1903, Dwn Bandwidth > 1078 class “Excellent” [70.7%]
- Rule 3: Dwn Bandwidth <= 1078, Dwn Delay <= 665, Upd Loss > 0, Upd Loss <= 2,
Dwn Loss > 0, Dwn Loss <= 2 class “Excellent” [66.2%]
- Rule 4: Dwn Bandwidth <= 12 class “No Call” [90.6%]
- Rule 5: Upd Bandwidth <= 14, Upd Loss <= 27 class “No Call” [75.7%]
- Rule 6: Upd Delay <= 506, Upd Loss > 27, Upd Loss <= 46, Dwn Loss > 45 class “No
Call” [61.2%]
-
-
-
- Default class: Good
AR
Q/F
EC
1
2kb
s
a c
ritica
l ra
te
Skype can easily deal with
one-way losses if bandwidth is available
one-way delay up tp 400ms
Chadi Barakat - 13
Still many open issues
Consideration of other multimedia and non-multimedia applications
Scalability of network measurements
Application to network regulation and optimization
Current work focuses on
- Consolidation of the audio and video case
- ACQUA for mobiles (Inria ADT ACQUA)
- Crowdsourcing