Professional Documents
Culture Documents
Qos Test Specification 0-2475 v3
Qos Test Specification 0-2475 v3
Company / organisation
ACCESS : ; Public
Restricted
Confidential
NAME
DATE
VISA
Author
L.Poutas
Review
ERTMS/GSM-R Operators
Group
C.Spaans/J.Nooijen
Approval
ERTMS/GSM-R ERIG
February 2007
R.Sarfati
Page 1 of 1
EVOLUTION SHEET
Version
Date
Author
Modification
First draft version, based on a QoS
Working document created by GSMR/UNISIG.
0.1
3 June 2003
0.2
0.3
21 July 2003
0.4
25 July 2003
1.0
1.1
11 September
QoS Working Group
2003
16th
GSM-R QoS Working Group
September
2005
1.2
2.0
30th October
2006
2.01
3.0
22nd
OG meeting #26
November
2006
19th February
ERIG/IG/UNISIG/EEIG
2007
Page 2 of 2
Table of Contents
1
Introduction..........................................................................................................5
1.1
Purpose...........................................................................................................5
1.2
References......................................................................................................5
1.3
Abbreviations and definitions ........................................................................5
Appendixes .........................................................................................................20
6.1
Appendix A, Clarification on Statistics .......................................................20
Page 3 of 3
6.1.1
Quantiles:.............................................................................................20
6.1.2
Standard deviation:..............................................................................21
6.1.3
Confidence Intervals:...........................................................................21
6.2
Appendix B, Algorithm for Transmission Interference calculation
(informative) ............................................................................................................23
6.2.1
Sending side .........................................................................................23
6.2.2
Receiving side ......................................................................................24
6.2.3
Validation of Transmission Interference parameters ..........................26
6.3
Appendix C, Command script for GSM-R network registration delay
measurement (informative)......................................................................................27
6.3.1
Command script for inbound border crossing.....................................27
6.3.2
Command script for outbound border crossing...................................27
6.4
Appendix D, Recommendation of minimum sample sizes for ETCS QoS
parameter validation (informative) ..........................................................................28
6.4.1
Introduction..........................................................................................28
6.4.2
Summary ..............................................................................................28
6.4.3
Connection Establishment Delay (CED), Transfer Delay (TD),
Network Registration Delay (NRD) .....................................................................29
6.4.4
Connection Loss Rate (CLR) ...............................................................30
6.4.5
Transmission Interference ...................................................................33
6.4.6
Connection Establishment Error Ratio................................................35
Page 4 of 4
1 INTRODUCTION
1.1
Purpose
This document is a test specification for the Quality of Service requirements based on
[SUBSET-093] and proposes a standardized set of acceptance tests to allow
comparable QoS behaviour of GSM-R communication networks.
It contains:
- Detailed definitions of the QoS parameters for the bearer service.
- The interfaces on which the bearer service and the QoS parameters are
applicable.
- The methods of measuring and verification of QoS parameter values.
Remark: it is intended to include in a future release of this document the methods of
monitoring QoS parameters in operational situations.
When complying with the tests in this document it will be possible:
- To demonstrate that the end-to-end communication system for ETCS complies
with -requirements.
1.2
References
[SUBSET-093]
[EUR FFFIS]
1.3
Abbreviations
ASCII
ERTMS
ETCS
KPI
IMSI
LUP
MT
NT
OBU
PLMN
QoS
Page 5 of 5
RBC
TI
Definitions
Byte
Call
Connection
Confidence
Data frame
IFIX
IGSM
IUm
Octet
Probability
Rate
Bearer Service User
User
Page 6 of 6
In order to define and to clarify the different QoS aspects, a reference system
architecture is needed. Both the operational system architecture and the measurement
architecture are shown below.
Page 7 of 7
2.2
2.2.1 General
The purpose of these tests is to test the correct behaviour of the GSM-R subsystem.
EuroRadio protocol stacks according to [subset 037] are not used during testing.
Measurement applications are used to collect the QoS result.
The tests shall be performed using the same bearer service(s) as the ETCS operational
system.
Although a data frame in the operational system consists of octets, all test descriptions
use bytes, since test tools are more likely to handle bytes and ASCII characters.
For some QoS tests, 30 bytes long data frames shall be used. These frames are
providing a model for HDLC protocol which is using data frames consisting of 30
octets.
The test area is defined as the area with ETCS service. ETCS service boundaries are
usually marked by a ETCS announcement balise.
The test shall be performed under conditions similar to nominal operational
conditions. Tests shall be done using the highest commercial available train speed
(according to the line status: new line, on-going upgrades,). All equipment is in
normal operational use. Non-availability due to technical defects is excluded, unless
stated otherwise.
Measuring for a higher data transfer rate also qualifies the test area for the lower rates
(e.g. measuring at 9.6 kbps also qualifies 4.8 and 2.4)
Requirements are normally specified with a Confidence Level. This allows
calculation of the number of samples needed before a test can be considered as
completed. If feasible, all parameters shall be verified up to the Confidence Level
specified on a national or operator basis. See Appendix D, Recommendation of
minimum sample sizes for ETCS QoS parameter validation (informative) and
Appendix A, Clarification on Statistics.
Tests are fulfilling data bearer requirements, as specified in [SUBSET-093].
The network configuration and the parameters shall be described within test cases and
logged during test execution.
Page 8 of 8
The test mobile performance shall be equal to an MT2 on the relevant parameters in
the different test cases. ETCS data communication is tested using only 8W mobiles.
The MT2 test mobile must be compliant with the EIRENE specifications.
The mobile must report classmark 2 information element, with a SIM Card not
configured for subscription to VBS/VGCS services, as described in [EUR FFFIS], to
avoid disturbance of ASCI notification.
2.3
Test plan
Testing and tuning the network is usually part of larger and partially national project
efforts. One can use QoS testing for a step-by-step or layer-by-layer integration of
ETCS.
Basic testing of QoS for ETCS will be necessary or at least helpful for tuning and
optimising the network that will operate ETCS. QoS test equipment will here be used
as "test application", integration of which will show possible network or mobile
deficiencies and lead to an optimised transmission system. Fundamental as well as
local problems can be disclosed.
After this it should be easier to integrate the real ETCS application consisting of
EURORADIO equipment, RBC, OBU into the GSM-R system. Nevertheless, it can
be necessary to perform additional QoS testing in this phase, e.g. to raise the
confidence level in view of forthcoming railway operation.
Therefore the extent of testing, and for what purpose, is a decision of the national
projects.
Page 9 of 9
Phase 2 ETCS integration tests using real components like RBC, OBU and
EuroRadio protocol stack.
Phase 3 In-service continuous monitoring of the performances (not covered
by this document).
Phase 0:
RF tuning
Phase1:
E2E testing
Phase 2:
ETCS integration
QoS acceptance
(GSM-R sub system)
Phase3:
In service monitoring
QoS acceptance
(Entire system)
Page 10 of 10
3.1.1 Definition:
This is the value of the elapsed time between the connection establishment request
and the indication of successful connection establishment on the requesting side.
3.2
3.2.1 Definition:
This is the ratio of the number of unsuccessful attempts for connection establishment
to the total number of connection establishment attempts.
Page 11 of 11
3.3
3.3.1 Definition:
The number of connections released unintentionally per total accumulated connection
time.
Page 12 of 12
Notes
1. The first occurrence of such event whichever comes first is sufficient to
assume a connection loss.
2. A combination of these events occurring in sequence before starting
connection re-establishment is counted as only one connection loss.
The single connection duration for this connection established is accumulated to the
total connection duration.
3.4
3.4.1 Definition:
This is the value of the elapsed time between the request for transfer of a data frame
and the indication of successfully transferred end-to-end data frame.
Page 13 of 13
3.5
3.5.1 Definition:
The purpose of this KPI is to test the correct sending of data frames across the GSMR subsystem. The statistical distributions of transmission interferences and error-free
periods will be then built and finally assessed against [Subset093] targets using
separately uplink and downlink distributions.
Transmission recovery period TREC
TREC starts with the first error-free data frame.
TREC ends before the first erroneous data frame.
Transmission interference period TTI:
TTI starts with the first erroneous data frame. An erroneous data frame is a 30
bytes frame bearing at least one bit in the received data stream that does not
comply with the associated bits of the transmitted data frame.
TTI ends before the first error-free data frame. An error-free data frame is a 30
bytes data frame bearing no bit in the received data frame that does not comply
with the associated bits of the transmitted data frame.
In case a connection loss (see chapter 3.3.3 page 12), the last continuous interference
period prior to the connection loss shall not be taken in account for TTI cumulated
distribution function, since such an event is already counted within the connection
loss rate.
In case of a TREC period equal or less than 4 consecutive error-free data frames, this
short error free period shall be counted as one error period and shall be merged with
the adjacent Transmission interference period TTI. A TREC period always starts with 5
or more consecutive error-free data frames.
KPI assessment:
The statistical events defining the QoS targets for transmission interference
specified in [Subset093] are the transmission recovery periods and transmission
interference periods.
Page 14 of 14
The data frame pattern is set at the sending side, with the following pattern:
[7E] [random data payload] [sequence number] [CRC-16] (= 30 octets in total)
where:
[7E] is the octet flagging the start of the frame, as in HDLC
[random data payload] is a 25 octet random payload
[16 bits sequence nbr] is a 2 octets sequence number marking the frame
[CRC-16] is a 2 octets CRC-16 check sum to detect valid data frames on the
receiving application (CRC Hamming distance is very good for a 30 octets
data frame)
The receiving side shall synchronise with sent frames.
The time reference for interference duration and recovery measurement shall be on
the receiving application only (Channel latency is measured independently).
The measurement results are updated following verification of the data frame
sequence number.
The data streams must be stored to files on both sending and receiving sides (at both
IGSM(T) and IFIX(T)).
The result is either calculated by comparison and analysis of the differences in these
files or detecting CRC errors in the received data.
The transmission interference test shall be performed using separate applications for
uplink and downlink (no round-trip data).
The uplink and downlink may be tested simultaneously depending on test application
capabilities.
Cumulated distribution functions for both TREC and TTI shall be built over the
measured area.
Some typical examples are shown below:
TREC
= 8 frames
Channel
state
T REC
= 6 frames
TTI
= 2 frames
A
1
T REC
= .. frames
Error-free
Errored
TREC
= 2 frames
TTI = 2+3+1
= 6 frames
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
Frame number
Page 15 of 15
3.6
3.6.1 Definition:
The elapsed time between the registration request +COPS=1,2, network2select and
the indication of successful registration onto GSM-R network +CREG: 1 or
+CREG: 5.
Page 16 of 16
Page 17 of 17
Page 18 of 18
Page 19 of 19
6 APPENDIXES
6.1
6.1.1 Quantiles:
A statement t<5sec (95%) means that for the KPI to evaluate, the following is valid:
95% of all values are smaller than 5s.
probability
distribution function
mean
5s
KPI
Equivalent to the above statement is that the area under the distribution function is
enclosed in the interval [0s;5s].
The cumulated statistic is often used to visualise whether the KPI target is met:
cumulated
distribution function
100%
95%
50%
mean
Page 20 of 20
5s
KPI
Note
The mean value of the KPI is often different from the KPI target.
Standard
deviation
5s
KPI
Note
There is no relationship between the standard deviation and errors and
confidence intervals. The standard deviation is a property of the statistical law,
whereas the confidence interval is related to measured estimations.
probability
distribution function
P(x) = value
(Estimated statistical law)
5s
Page 21 of 21
KPI
cumulated
distribution function
100%
95%
93%
50%
mean
5s
KPI
1
.
(number of samples)
This means to increase the confidence from 90% to 99% keeping the same
interval, 100 times more samples are needed!
The knowledge of the confidence interval and the type of statistical law allow
computing the number of samples needed.
A confidence level of 90% means that there is 10% of chance to fall out of the
interval [0, measured KPI for t<5s].
It does NOT mean that the error on the measurement is 10%.
Page 22 of 22
6.2
Page 23 of 23
Startime
01:00:00.234
01:10:00.934
Endtime
01:10:00.834
01:11:08.555
FramesErroneous
0
13
FramesErrorfree
1525
5
FramesLost
0
3
EF
01:30:30.523
01:50:10.523
30127
ChannelState indicates the type of interval. The interval is either Error Free (EF) or
Transmission Interfered (TI). Startime and Endtime indicate the beginning
respectively the end of an EF- or TI-interval (format:
hour:minute:second.milisecond). FramesErroneous indicates the number of frames
received in error during the interval, FramesErrorfree indicates the number of
correctly received frames during the interval. FramesLost indicates the number of lost
frames during the interval. Lost frames occur when two consecutively received error
free frames do not have consecutive sequence numbers.
The algorithm uses the following variables:
Parameter
FrameRxTime_Actual
FrameRxTime_Previous
FrameSNR_Actual
FrameSNR_Previous
#Frames_ConsEF
EF_Threshold
ChannelState_Actual
ChannelState_Previous
Diff_SNR
#FramesLost
#FramesERR
#FramesEF
Description
Time of reception of the actual or current frame. This time is derived from the operating system of
the receiver (receiver local time).
Time of reception of the previous frame.
Sequence number of the actual or current frame. If the actual frame is not error free
FrameSNR_Actual = -1.
Sequence number of the frame received just before the actual or current frame. If the previous frame
is not error free FrameSNR_Previous = -1.
Number of consecutive received error free frames since the last erroneous or lost frame.
EF_Threshold indicates how many consecutive error free frames should have been received before
the channel state changes from TI to EF. Its default value is 5.
The actual or current state of the transmission channel. The channel resides either in TI- or EF-state.
If a lost frame is detected or an erroneous frame is received, the channel state changes to TI. If
EF_Threshold consecutive error free frames have been received, the channel state changes to EF.
The state of the transmission channel just before the reception of the actual frame (i.e. the channel
state at reception of the previous frame).
Contains the difference between the sequence numbers of the actual and previous frame.
Contains the total number of lost frames in a certain TI-interval. The value of the parameter is
incremented with Diff_SNR 1 if two consecutive error free frames are received which sequence
numbers differ more than 1. Its value is reset every time ChannelState changes.
Contains the total number of erroneous frames in a certain TI-interval. Its value is reset every time
ChannelState changes.
Contains the total number of error free frames in a certain TI- or EF-interval. Its value is reset every
time ChannelState changes.
Page 24 of 24
Start
ChannelState_Previous = TI
ChannelState_Actual = TI
FrameSNR_Actual = -1
FrameSNR_Previous = -1
EF_Threshold = 5
All other parameter values = 0
Create first record
ChannelState = ChannelState_Actual
Starttime = (local) system time of receiver
Receive next frame (30 bytes) and evaluate CRC
FrameRxTime_Actual = (local) system time of receiver
CRC is correct
(frame error
free)?
No
FrameSNR_Actual = -1
#Frames_ConsEF = 0
#FramesERR = #FramesERR + 1
Yes
Determine FrameSNR_Actual (read SNR from frame)
Diff_SNR = FrameSNR_Actual - FrameSNR_Previous
#FramesEF = #FramesEF + 1
9
9
9
No
Diff_SNR <=
FrameSNR_Actual?
Yes
No
Diff_SNR = 1?
Yes
#Frames_ConsEF = #Frames_ConsEF + 1
#Frames_ConsEF
>= EF_Threshold
#Frames_ConsEF = 1
No
Yes
#Frames_ConsEF = 1?
No
ChannelState_Actual = EF
Yes
StarttimeEF = FrameRxTime_Actual
EndtimeTI = FrameRxTime_Previous
ChannelState_Actual = TI
ChannelState_Previous = EF?
AND
ChannelState_Actual = TI?
9
9
9
ChannelState_Previous = TI?
AND
ChannelState_Actual = EF?
No
Yes
Yes
FrameSNR_Actual
= -1?
No
Yes
FrameSNR_Previous = FrameSNR_Actual
ChannelState_Previous = ChannelState_Actual
FrameRxTime_Previous = FrameRxTime_Actual
#FramesERR = 1
#FramesEF = 0
#FramesLost = 0
#FramesERR = 0
#FramesEF = 1
#FramesLost = Diff_SNR - 1
#FramesERR = 0
#FramesEF = EF_Threshold
#FramesLost = 0
No
Page 25 of 25
PercentageTI _ 0.8 =
PercentageTI _ 1 =
PercentageREC _ 7 =
PercentageREC _ 20 =
Page 26 of 26
6.3
Page 27 of 27
6.4
6.4.1 Introduction
This annex recommends sample sizes and validation procedures for performance tests
to validate ETCS QoS parameters.
Since ETCS QoS parameters differ on statistical characteristics and target values,
sample sizes for individual parameters also differ. Furthermore, the choice for a
particular sample size is to a large extent arbitrary. It is a compromise between costs
(time and effort to perform validation tests), precision (reproducibility) and
confidence level.
ETCS QoS parameters can be subdivided into four categories. Each category requires
a particular statistical approach to determine a suitable sample size. The categories as
discussed in the next paragraphs are:
1. Connection Establishment Delay (CED), Transfer Delay (TD), Network
Registration Delay (NRD)
2. Connection Loss Rate (CLR)
3. Transmission Interference (TI)
4. Connection Establishment Error Ratio (CER)
6.4.2 Summary
The table below provides an overview of ETCS QoS parameters. It lists minimum
required sample sizes and validation procedures as derived in the following
paragraphs.
Minimum
recommended
sample size (95%
confidence level)
Requirement
(Subset-093)
<8.5s (95%)
10s (100%)
100 samples
0.5s (99%)
100 samples
30s (95%)
35s (99%)
40s (100%)
<10-2/h
TI period:
<0.8s (95%)
<1s (99%)
<10-2
100 samples
(inbound border
crossing)
100 samples
(outbound border
crossing)
300 hours
Validation procedure
Parameter successfully validated if the 95th-percentile is less
than 8.5s
Parameter successfully validated if the 99th-percentile is less
than 0.5s
Note 1:
For CED and CER a sample or event is defined as a call set up (attempt). For TD and
TI a sample or event is defined as the transmission/reception of a data frame. For
NRD a sample or event is defined as a network registration session as described in
par. 6.3 (Appendix C, Command script for GSM-R network registration delay
measurement (informative).
Page 28 of 28
For CLR, the sample size is expressed as the duration of the measurement. A
connection loss should typically not occur.
Note 2:
For Transmission Interference the parameters of interest are TREC and TTI rather than
the immediate result of transmission/reception of data frames. With a sample size of
1500 frame transmissions per kilometre however, adequate reproducibility of the
average probability of a frame transmitted in error is achieved (see par. 6.4.5). The
assumption is that this sample size also implies adequate reproducibility of TREC and
TTI. These parameters are validated over a substantial ETCS area (covered by at least
5 GSM-R cells) by 1 up to 3 round-trip test runs depending on the speed of the
measurement train (see par. 6.4.5).
Note 3:
In this annex, sample sizes are calculated for a confidence level of 95%. Decreasing
the confidence level while maintaining the same confidence interval, typically results
in a decreased sample size (and hence in a reduction of measurement time and costs).
The choice for a particular confidence level and sample size is a national matter.
s
n
(1)
Page 29 of 29
s
n
(2)
st *
Solving for n yields: n =
0.1x
(3)
Measurements in the Netherlands have shown that s/ x is much smaller than 0.5 for
all parameters (i.e. CED, TD and NRD).
Taking a worst case value of s/ x = 0.5, a confidence level of 95% (two sided
confidence interval) and applying equation (3) results in n = 100.
Consequences for measurement in practice
If for validating CED and TD a single round-trip drive-test is performed, a sample
size of n = 100 is easily achieved.
NRD is defined under standstill conditions. A validation test with a sample size of n =
100 can be performed in less than 2 hours.
Validation of parameters CED, TD and NRD
Compliancy with the requirements is demonstrated by taking the 95th-percentile from
the measurement data set for CED, taking the 99th-percentile from the measurement
data set for TD and the 95th and 99th percentiles from the measurement data set for
NRD.
The k-th percentile is defined as that value of the measurement data set which
corresponds to a cumulative frequency of n.k/100. With n=100, the 95th/99thpercentile is simply the 95th/99th value of the measurement data set (when sorted in
ascending order).If a percentile of a particular parameter is less than the
corresponding value as specified in Subset-093, the parameter is successfully
validated.
e Rt
k!
(4)
Page 30 of 30
Where:
P(k) =
The probability that k failures occur within a measurement time interval
t if the Connection Loss Rate is R.
k=
Number of connection losses in measurement time interval t
R=
Connection Loss Rate (connection losses per hour)
t=
Measurement time interval (hours)
Note that for a Poisson distribution both the mean and the variance are equal to Rt.
Measurement time interval proposal
The question is then how to choose the size of the measurement time interval t.
If we choose a confidence level of 95% and define that as the probability that 1 or
more connection losses occur in a measurement time interval t under the pre-condition
that the Connection Loss Rate is truly R=0.01/hour (i.e. as specified in Subset-093),
the measurement time interval t can be calculated. The probability of having 1 or
more connection losses in a measurement time interval t is:
P(k >= 1) = 1 P(k = 0) = 1 e 0.01t = 0.95
(5)
0,25
1,0
0,20
0,8
0,15
0,6
0,10
0,4
0,05
0,2
0,00
P( # Connection Losses = k)
0,0
0
10
There is a trade off between confidence level and measurement time interval. Two
examples demonstrating this trade off are given below. In both cases the starting point
is that the true CLR = 0,01/h.
If the measurement time interval is 70 hours, the probability of no connection loss is
about 50%.
Page 31 of 31
1,0
0,50
0,8
0,40
0,6
0,30
0,4
0,20
P( # Connection Losses = k)
0,60
0,2
0,10
0,00
0,0
0
10
1,0
1,00
0,90
0,80
0,70
0,60
0,50
0,40
0,30
P( # Connection Losses = k)
Finally if the measurement time interval is 5 hours, the probability of one or more
connection losses is only about 5%.
0,20
0,10
0,00
0,8
0
10
Page 32 of 32
Page 33 of 33
Since the behaviour of downlink frame transfer and uplink frame transfer is quite
different, a sample size of 1500 is applicable to both uplink- and downlink
measurement.
Note that the assumption that the transfer of frames follows a binomial statistical law
may not be true since the a-priori probability of transferring a frame in error may not
be the same for every frame transfer (it is probably dependent on train speed and
location). During a GSM-R hand over for example, the data transfer may be disturbed
over several consecutive frames.
Therefore the area over which the 1500 samples are taken is limited to 1 km.
In any case, the results of real-live performance tests are needed to evaluate the
strength of the analysis and the underlying assumptions.
Consequences for measurement in practice
The analysis of the measurement data of the HSL line in the Netherlands also shows
that after completion of the transfer of a frame in the uplink direction, the average
time it takes before the next frame is starting to transmit is about 10 msec. It takes
(1+8+1)*30/9600*1000 = 31.25 msec. to transfer a single 30-byte frame (including
start- and stop-bit) with 9600 bps. So the average frame transfer rate is 25 per second
(i.e. actually 1/(31.25+10)*1000 = 24.24).
For the downlink the frame transfer rate is higher (it takes only 1 msec. on average in
between two frame transfers). The rate on the downlink is 1/(31.25+1)*1000 = 31
frames per second.
The table below shows the practical consequences in terms of the number of
recommended test runs:
Average speed
of test train
along the track
Duration to travel
1000 meter
(seconds)
66.66
33.33
16.67
11.11
1666.50
833.25
416.75
277.78
* 2 runs (1 return trip) is the minimum number of runs that is needed. This is due to the GSM mobility
management mechanisms that have hysteresis and therefore it is required to travel at least once both
ways to check the correct sending of data frames
Note
The test for validation of Transmission Interference shall comprise a substantial
ETCS test area including at least 5 GSM-R cells.
Example:
If you have a 100 km ETCS line with an average operational train speed of 216 km/h,
it is recommended to use a measurement train equipped with 2 measurement sets (1
for uplink- and 1 for downlink measurements) and to perform 2 round trip test runs.
Page 34 of 34
Validation of TI parameters
For a description of the validation procedure and the definition of TI parameters refer
to par. 6.2.3.
1,0
0,14
0,12
0,8
0,10
0,6
0,08
0,06
0,4
0,04
0,2
0,02
0,00
0,0
0
10
CER = 0.01
Sample size (n) = 1078
12
14
16
18
20
22
24
26
28
30
32
34
The figure above illustrates the binomial distribution for CER=0.01 and n=1078. The
red area at the tail corresponds with a cumulative probability of about 5%, which is
the probability that the number of connection establishment failures is 17 or more.
Page 35 of 35
END OF DOCUMENT.
Page 36 of 36