Professional Documents
Culture Documents
Machine Learning Approach To 5G Infrastructure Market Optimization - Bega Et Al. 2019
Machine Learning Approach To 5G Infrastructure Market Optimization - Bega Et Al. 2019
DOI: https://doi.org/10.1109/TMC.2019.2896950
Abstract—It is now commonly agreed that future 5G Networks will build upon the network slicing concept. The ability to provide virtual,
logically independent “slices” of the network will also have an impact on the models that will sustain the business ecosystem. Network
slicing will open the door to new players: the infrastructure provider, which is the owner of the infrastructure, and the tenants, which
may acquire a network slice from the infrastructure provider to deliver a specific service to their customers. In this new context, how to
correctly handle resource allocation among tenants and how to maximize the monetization of the infrastructure become fundamental
problems that need to be solved. In this paper, we address this issue by designing a network slice admission control algorithm that
(i) autonomously learns the best acceptance policy while (ii) it ensures that the service guarantees provided to tenants are always
satisfied. The contributions of this paper include: (i) an analytical model for the admissibility region of a network slicing-capable 5G
Network, (ii) the analysis of the system (modeled as a Semi-Markov Decision Process) and the optimization of the infrastructure
providers revenue, and (iii) the design of a machine learning algorithm that can be deployed in practical settings and achieves close
to optimal performance.
Index Terms—Network Slicing, Admission Control, Neural Networks, Machine Learning, 5G Networks
needs of the service being provided, adjusting the slice’s not known a priori and may vary with time. For these reasons,
operation to the service provider’s needs. For mobile network traditional solutions building on optimization techniques are
operators, network slicing allows to target new customers not affordable (because of complexity reasons) or simply im-
with specific service requirements, such as industrial sectors possible (when slice behavior is not known). Instead, machine
with stringent requirements, ultimately providing new revenue learning provides a mean to cope with such complex problems
streams coming from the new customers. while learning the slice behavior on the fly, and thus allows
This business model underlying network slicing is the to develop a practical approach to deal with such a complex
Infrastructure as a Service (IaaS), which is expected to increase and potentially unknown system.
the number of available revenue streams in 5G. This model The rest of the paper is structured at follows. In Section 2
has already been successfully applied to the cloud computing we review the relevant works related to this paper in the
infrastructure by providers such as Amazon AWS or Microsoft fields of resource allocation for network slicing aware net-
Azure. However, cloud computing platforms are selling to works, network slice admission control and machine learning
customers (i.e., tenants) cloud resources (e.g., CPU, memory, applied to 5G Networks. In Section 3 we describe our System
storage) only, while in a 5G Infrastructure market such as the Model, while the analytical formulation for the network slice
one enabled by network slicing, the traded goods also include admissibility region is provided in Section 4. In Section 5 we
network resources (e.g., spectrum, transport network). This model the decision-making process by means of a Markovian
entails a totally different problem due to the following reasons: analysis, and derive the optimal policy which we use as
(i) spectrum is a scarce resource for which over-provisioning a benchmark. In Section 6 we present a Neural Networks
is not possible, (ii) the actual capacity of the systems (i.e., approach based on deep reinforcement learning, which pro-
the resources that can actually be sold) heavily depends vides a practical and scalable solution with close to optimal
on the mobility patterns of the users, and (iii) the Service performance. Finally, in Section 7 we evaluate the proposed
Level Agreements (SLAs) with network slices tenants usually algorithm in a number of scenarios to assess its performance
impose stringent requirements on the Quality of Experience in terms of optimality, scalability and adaptability to different
(QoE) perceived by their users. Therefore, in contrast to IaaS, conditions, before concluding the paper in Section 8.
in our case applying a strategy where all the requests coming to
the infrastructure provider are admitted is simply not possible.
The ultimate goal of InPs is to obtain the highest possible 2 S TATE OF THE ART
profit from of the deployed infrastructure, thus maximizing
monetization. The design of a network slice admission control While the network slicing concept has only been proposed
policy that achieves such goal in this spectrum market is still recently [2], it has already attracted substantial attention. 3GPP
an open problem. More specifically, the network capacity bro- has started working on the definition of requirements for
ker algorithm that has to decide on whether to admit or reject network slicing and the design of a novel network architecture
new network slice requests shall simultaneously satisfy two for supporting it [4], whereas the Next Generation Mobile
different goals: (i) meeting the service guarantees requested by Networks Alliance (NGMN) identified network sharing among
the network slices admitted while (ii) maximizing the revenue slices (the focus of this paper) as one of the key issues to
of a network infrastructure provider. be addressed [5]. While there is a body of work on the
The goal of meeting the desired service guarantees needs literature on spectrum sharing [6]–[9], these proposal are not
to consider radio related aspects, as a congested network will tailored to the specific requirements of the 5G ecosystem.
likely not be able to meet the service required by a network Conversely, most of the work has focused on architectural
slice. Conversely, the goal of maximizing the revenue obtained aspects [10], [11] with only a limited focus on resource
by the admission control should be met by applying an on- allocation algorithms. In [12], the authors provide an analysis
demand algorithm that updates the policies as long as new of network slicing admission control and propose a learning
requests arrive. algorithm; however, the proposed algorithm relies on an offline
In this paper, we propose a Machine Learning approach to approach, which is not suitable for a continuously varying
the 5G Infrastructure Market optimization. More specifically, environment such as the 5G Infrastructure market. Moreover,
the contributions of this paper are: (i) we provide an analytical the aim is to maximize the overall network utilization, in
model for the admissibility region in a sliced network, that contrast to our goal here which is focused on maximizing InP
provide formal service guarantees to network slices, and revenues.
(ii) we design an online Machine Learning based admission The need for new algorithms that specifically targets the
control algorithm that maximizes the monetization of the monetization of the network has been identified in [13].
infrastructure provider. However, there are still very few works on this topic. The work
Machine learning is the natural tool to address such a in [14] analyzes the problem from an economical perspective,
complex problem. As discussed in detail along the paper, proposing a revenue model for the InP. The authors of [15]
this problem is highly dimensional (growing linearly with the build an economic model that describes the Mobile Network
number of network slices classes) with a potentially huge Operator (MNO) profit when dealing with the network slice
number of states (increasing exponentially with the number of admission control problem, and propose a decision strategy to
classes) and many variables (one for each state). Furthermore, maximize the expected overall network profit. The proposed
in many cases the behavior of the tenants that request slices is approach, however, is not on demand and requires the full
3
knowledge of arriving requests statistics, thus making it im- available services and the possible business relationships. New
practicable in real scenarios. Another work in this field is the players are expected to join the 5G market, leading to an
one of [14], with similar limitations. ecosystem that is composed of (i) users that are subscribed
Learning algorithms are in the spotlight since Mnith et to a given service provided by a (ii) tenant that, in turn,
al. [16] designed a deep learning algorithm called “deep Q- uses the resources (i.e., cloud, spectrum) provided by an (iii)
network” to deal with Atari games, and further improved infrastructure provider.1 In the remainder of the paper we use
it in [17] making the algorithm able to learn successful this high level business model as basis for our analysis. In the
policies directly from high-dimensional sensory inputs and following, we describe in details the various aspects related to
reach human-levels performance in most of Atari games. our system model.
Another approach is “AlphaGo”, which builds on deep neural Players. As mentioned before, in our system model there are
networks to play with the “Go” game [18]. Many more the following players: (i) the Infrastructure Provider, InP,
algorithms have been proposed [19]–[23], which are mostly who is the owner of the network (including the antenna
applied in games, robotics, natural language processing, image location and cloud infrastructure) and provides the tenants
recognition problems. with network slices corresponding to a certain fraction of
The application of Reinforcement and Machine learning network resources, (ii) the tenants, which issue requests to
approaches to mobile networks is also gaining popularity. To the infrastructure provider to acquire network resources, and
name a few examples, the work in [24] proposes a Q-learning use these resources to serve their users, providing them a
algorithm for improving the reliability of a millimeter wave specific telecommunication service, and finally (iii) the end-
(mmW) non-line-of-sight small cell backhaul system, while users, which are subscribers of the service provided by a tenant
in [25] Q-learning is implemented to solve the adaptive call which uses the resources of the infrastructure provider.
admission control problem. Network model. The ecosystem described above does not
Machine learning has been applied to a wide span of make any distinction on the kind of resources an InP may
applications in 5G networks, ranging from channel estima- provide to the tenants. From the various types of resources,
tion/detection for massive MIMO channel to user behav- spectrum will typically be the most important factor when
ior analysis, location prediction or intrusion/anomaly detec- taking a decision on whether to accept a request from a
tion [26]. For instance, decision tree and information-theoretic tenant. Indeed, cloud resources are easier to provision, while
regression models have been used in [27] in order to identify increasing the spectrum capacity is more complex and more
radio access networks problems. The authors of [28] em- expensive (involving an increase on antenna densification).
ploy a deep learning approach for modulation classification, Based on this, in this paper we focus on the wireless access
which achieves competitive accuracy with respect to traditional network as the most limiting factor. In our model of the
schemes. The authors of [29] apply deep neural networks wireless access, the network has a set of base stations B owned
to approximate optimization algorithm for wireless networks by an infrastructure provider. For each base station b ∈ B, we
resources management. let Cb denote the base station capacity. We further refer to
This work is an extension of the paper in [30]. In that the system capacity
∑ as the sum of the capacity of all base
paper, the problem of slice admission control for revenue stations, C = B Cb . We let U denote the set of users in the
maximization was addressed by employing Q-learning. While network.2 We consider that each user u ∈ U in the system
this provides the ability to adapt to changing environments is associated to one base station b ∈ B. We denote by fub
while achieving close to optimal performance, an inherent the fraction of the resources of base station b assigned to
drawback of Q-learning is its lack of scalability, as the learning user u, leading to a throughput for user u of ru = fub Cb .
time grows excessively when the state space becomes too We also assume that users are distributed among base stations
large. In contrast, the algorithm proposed in this paper is based according to a given probability distribution; we denote by
on Neural Networks, and it is shown to scale with the size of Pu,b the probability that user u is associated with base station
the network, quickly converging to optimal performance. b. We assume that these are independent probabilities, i.e.,
To the best of our knowledge, the work presented in this each user behaves independently from the others.
paper along with the previous version in [30] are the first Traffic model. 5G Networks provide diverse services which
ones that build on Machine Learning to address the problem are mapped to three different usage scenarios or slice cat-
of admission control for a 5G Infrastructure Market, with the egories: eMBB, mMTC and URLLC [3]. As the main bot-
aim of maximizing the InP’s revenue while guaranteeing the tleneck from a resource infrastructure market point of view
SLAs of the admitted slices. is spectrum, different slice categories need to be matched
based to their requirements in terms of the spectrum usage.
For instance eMBB-alike slices have a higher flexibility with
3 S YSTEM M ODEL respect to resource usage, and can use the leftover capacity of
As discussed in Section 1, 5G networks necessarily introduce URLLC services which have more stringent requirements on
changes in the applied business models. With the legacy
and rather monolithic network architecture, the main service 1. While some of these roles may be further divided into more refined
offered is a generic voice and best-effort mobile broadband. ones, as suggested in [31], the ecosystem adopted in this paper reflects a
large consensus on the current view of 5G networks.
Conversely, the high customizability that 5G Networks intro- 2. The users of the network are the end-users we referred to above, each
duce will enable a richer ecosystem on both the portfolio of of them being served by one of the tenants.
4
the needed capacity. to the users residing in the region. In our model, we focus on
Following the above, in this paper we focus on elastic the general case and consider network slices that span over the
and inelastic traffic as it is the main distinguishing factor entire network. However, the model could be easily extended
for spectrum usage and thus provides a fairly large level of to consider restricted geographical areas.
generality. In line with previous work in the literature [32], we Following state of the art approaches [11], network slicing
consider that inelastic users require a certain fixed throughput onboarding is an automated process that involves little or no
demand which needs to be satisfied at all times,3 in contrast human interaction between the infrastructure provider. Based
to elastic users which only need guarantees on the average on these approaches, we consider a bidding system in order
throughput, requiring that the expected average throughput to dynamically allocate network slices to tenants. With this,
over long time scales is above a certain threshold. That is, tenants submit requests for network slices (i.e., a certain
for inelastic users throughput needs to be always (or with number of users of a given service) to the infrastructure
a very high probability) above the guaranteed rate, while provider, which accepts or rejects the request according to
the throughput for elastic users is allowed to fall below the an admission control algorithm such as the one we propose in
guaranteed rate during some periods as long as the average this paper. To that aim, we characterize slices request by:
stays above this value. • Network slice duration t: this is the length of the time
We let I denote the set of classes of inelastic users; each interval for which the network slice is requested.
class i ∈ I has a different rate guarantee Ri which needs to be • Traffic type κ: according to the traffic model above, the
satisfied with a very high probability; we refer the probability traffic type of a slice can either be elastic or inelastic
that this rate is not met as the outage probability, and impose traffic.
that it cannot exceed P̄out , which is set to a very small value. • Network slice size N : the size of the network slice
We further let Ni denote the number of inelastic users of class is given by the number of users it should be able to
i ∈ I, and Pi,b be the probability that a user of class i is at accommodate.
base station b. Finally, we let Ne be the number of elastic • Price ρ: the cost a tenant has to pay for acquiring
users in the network and Re their average rate guarantee. resources for a network slice. The price is per time
At any given point in time, the resources of each base unit, and hence the total revenue obtained by accepting a
stations are distributed among associated users as follows: network slice is given by r = ρt.
inelastic users u ∈ I are provided sufficient resources to Following the above characterization, an infrastructure
guarantee ru = Ri , while the remaining resources are equally provider will have catalogs of network slice blueprinted by
shared among the elastic users. In case there are not sufficient predefined values for the tuple {κ, N, ρ}, which we refer to
resources to satisfy the requirements of inelastic users, even as network slice classes. Tenants issue requests for one of
when leaving elastic users with no throughput, we reject the slice classes available in the catalogue, indicating the total
as many inelastic users as needed to satisfy the required duration t of the network slice. When receiving a request,
throughput guarantees of the remaining ones. an infrastructure provider has two possible decisions: it can
Note that the above traffic types are well aligned with the reject the network slice and the associate revenue to keep the
slice categories defined in 3GPP, as the elastic traffic behavior resources free or it can accept the network slice and charge the
is in line with the eMBB and mMTC services, while inelastic tenant r dollars. If accepted, the infrastructure provider grants
behavior matches the requirements of URLCC services. resources to a tenant during a t-window.
Network slice model. By applying the network slicing concept To compute the profit received by the tenant, we count the
discussed in Section 1, the network is divided into different aggregated revenue resulting from all the admitted slices. This
logical slices, each of them belonging to one tenant. Thus, we reflects the net benefit of the InP as long as (i) the costs of
characterize a network slice by (i) its traffic type (elastic or the InP are fixed, or (ii) they are proportional to the network
inelastic), and (ii) its number of users (i.e., the subscribers of utilization (in the latter case, ρ reflects the difference between
a given service) that have to be served. the revenue and cost of instantiating a slice). We argue that
A network slice comes with certain guarantees provided by this covers a wide range of cases of practical interest such as
an SLA agreement between the tenant and the infrastructure spectrum resources or computational ones. Moreover, in the
provider. In our model, a tenant requests a network slice that cases where costs are not linear with the network usage, our
comprises a certain number of users and a traffic type. Then, analysis and algorithm could be extended to deal with such
as long as the number of users belonging to a network slice cases by subtracting the cost at a given state from the revenue.
is less or equal than the one included in the SLA agreement,
each of them will be provided with the service guarantees
corresponding to their traffic type.
4 B OUNDING THE ADMISSIBILITY REGION
A network slice may be limited to a certain geographical An online admission control algorithm has to decide whether
area, in which case the corresponding guarantees only apply to accept or reject a new incoming network slice request issued
by a tenant. Such a decision is driven by a number of variables
3. Note that, by ensuring that the instantaneous throughput of inelastic traf- such as the expected income and the resources available. The
fic stays above a certain threshold, it is possible to provide delay guarantees. objective of an admission control algorithm is to maximize
Indeed, as long as the traffic generated by inelastic users is not excessive, by
providing a committed instantaneous throughput we can ensure that queuing the overall profit while guaranteeing the SLA committed to
delays are sufficiently low. all tenants. A fundamental component of such an algorithm is
5
the admissibility region, i.e., the maximum number of network variable. In line with [34], where this is applied to a binomial
slices that can be admitted in the system while guaranteeing distribution and the correction factor is one half of the step
that the SLAs are met for all tenants. Indeed, if admitting a size, in our case we a set the continuity correction factor as
new network slice in the system would lead to violating the one half of the average step size, which yields
SLA of already admitted slices, then such a request should be ∑
1 j∈I Pj,b Nj Rj
rejected. In the following, we provide an analysis to determine C̃b = ∑ . (6)
the admissibility region, denoted by A, as a first step towards 2 j∈I Pj,b Nj
the design of the optimal admission algorithm. Once we have obtained Pout,b , we compute the outage
probability of an inelastic user of class i with the following
4.1 Admissibility region analysis expression: ∑
We say that a given combination of inelastic users of the Pout,i = Pi,b Pout,b . (7)
b∈B
various classes and elastic users belongs to the admissibility
region, i.e., {N1 , . . . , N|I| , Ne } ∈ A, when the guarantees Next, we compute the average throughput of an elastic user.
described in the previous section for elastic and inelastic traffic To this end, we assume that (i) in line with [32], elastic users
are satisfied for this combination of users. In the following, consume all the capacity left over by inelastic traffic, (ii) there
we compute the admissibility region A. is always at least one elastic user in each base station, and (iii)
In order to determine whether a given combination of all elastic users receive the same throughput on average.
users of different types, {N1 , . . . , N|I| , Ne }, belongs to A, we With the above assumptions, we proceed as follows. The
proceed as follows. We first compute the outage probability average committed throughput consumed by inelastic users at
for an inelastic user of class i ∈ I, Pout,i . Let Rb be the base station b is given by
throughput consumed by the inelastic users at b. The average ∑
E[Rb ] = Ni Pi,b Ri , (8)
value of Rb can be computed as
i∈I
∑
E[Rb ] = Nj Pj,b Rj , (1) which gives an average capacity left over by inelastic users
j∈I equal to Cb − E[Rb ]. This capacity is entirely used by elastic
and the typical deviation as users as long as the base station is not empty. The total
∑ capacity usage by elastic users is then given by the sum of
σb2 = 2
Nj σj,b , (2) this term over all base stations. As this capacity is equally
j∈I shared (on average) among all elastic users, this leads to the
2
where σj,b is the variance of the throughput consumed by one following expression for the average throughput of an elastic
inelastic user of class j, which is given by user: ∑
Cb − E[Rb ]
re = b∈B . (9)
2
σj,b = Pj,b (Rj − Pj,b Rj )2 + (1 − Pj,b )(Pj,b Rj )2 Ne
= Pj,b (1 − Pj,b )Rj2 . (3) Based on the above, we compute the admissibility region A
as follows. For a given number of inelastic users in each class,
Our key assumption is to approximate the distribution of Ni , i ∈ I, and of elastic users, Ne , we compute the outage
the committed throughput at base station b by a normal probability of the inelastic classes, Pout,i , and the average
distribution of mean Rb and variance σb2 , i.e., N (E[Rb ], σb2 ). throughput of the elastic users, re . If the resulting values meet
Note that, according to [33], this approximation is appropriate the requirements for all classes, i.e., Pout,i ≤ P̄out ∀i and
as long as the number of users per base station in the boundary re ≥ Re , then this point belongs to the admissibility region,
of the admissibility region is no lower than 5, which is and otherwise it does not.
generally satisfied by cellular networks (even in the extreme
case of small cells).
4.2 Validation of the admissibility region
The outage probability at base station b is given by the
probability that the committed throughput exceeds the base In order to assess the accuracy of the above analysis, we
station capacity, i.e., compare the admissibility region obtained theoretically against
the one resulting from simulations. To this end, we consider
Pout,b = P(Rb > Cb ), (4) the reference scenario recommended by ITU-T [35], which
consists of |B| = 19 base stations placed at a fixed distance
where Cb be the capacity of base station b.
of 200m. Following the system model of Section 3, we have
To compute the above probability with the normal approx-
elastic and inelastic users. All inelastic users belong to the
imation, we proceed as follows:
( ) same class, and all users (elastic and inelastic) move in the
Cb + C̃b − E[Rb,i ] area covered by these base stations following the Random
Pout,b ≈ 1 − Φ , (5) Waypoint (RWP) mobility model, with a speed uniformly
σb,i
distributed between 2 and 3 m/s.
where Φ(·) is the cumulative distribution function of the The association procedure of elastic and inelastic users with
standard normal distribution and C̃b is a continuity correction base stations is as follows. Inelastic users try to associate to
factor that accounts for the fact Rb is not a continuous the nearest base station b ∈ B, if it has at least Ri capacity left.
6
where ρi and ρe are the price per time unit paid by an inelastic
and an elastic network slice, respectively.
The ultimate goal is to find the policy π (S) that maximizes
the long term average revenue, given the admissibility region
and the network slices requests arrival process. We next devise
the Optimal Policy when the parameters of the arrival process
Fig. 2: Example of system model with the different states. are known a priori, which provides a benchmark for the best
possible performance. Later on, in Section 6, we design a
We define our space state S as follows. A state s ∈ S is a learning algorithm that approximates the optimal policy.
three-sized tuple (ni , ne , k | ni , ne ∈ A) where ni and ne are
the number of inelastic and elastic slices in the system at a
given decision time t, and k ∈ {i, e, d} is the next event that 5.2 Optimal policy
triggers a decision process. This can be either a new arrival of In order to derive the optimal policy, we build on Value
a network slice request for inelastic and elastic slices (k = i Iteration [37], which is an iterative approach to find the optimal
and k = e, respectively), or a departure of a network slice of policy that maximizes the average revenue of an SMDP-based
any kind that left the system (k = d). In the latter case, ni and system. According to the model provided in the previous
ne represent the number of inelastic and elastic slices in the section, our system has the transition probabilities P (s, a, s′ )
system after the departure. Fig. 2 shows how the space state detailed below.
S relates to the admissibility region A. Let us start with a = D, which corresponds to the action
The possible actions a ∈ A are the following: A = G, D. where an incoming request is rejected. In this case, we have
The action G corresponds to admitting the new request of an that when there is an arrival, which happens with a rate λi
elastic or inelastic slice; in this case, the resources associated and λe for inelastic and elastic requests, respectively, the
with the request are granted to the tenant and the revenue request is rejected and the system remains in the same state.
r = ρi,e t is immediately earned by the infrastructure provider. In case of a departure of an elastic or an inelastic slice, which
In contrast, action D corresponds to rejecting the new request; happens with a rate of ne µe or ni µi , the number of slices in
in this case, there is no immediate reward but the resources the system is reduced by one unit (recall that no decision is
remain free for future requests. Note that upon a departure needed when slices leave the system). Formally, for a = D
(k = d), the system is forced to a fictitious action D that and s = (ni , ne , i), we have:
involves no revenue. Furthermore, we force that upon reaching
λi ′
a state in the boundary of the admissibility region computed
υ(ni ,ne ) , s = (ni , ne , i)
in the previous section, the only available action is to reject
λe , s′ = (n , n , e)
i e
an incoming request (a = D) as otherwise we would not be P (s, a, s′ ) = υ(n i ,ne )
. (12)
ni µi
υ(ni ,ne ) , s ′
= (n − 1, ne , d)
meeting the committed guarantees. Requests that are rejected
i
ne µe , s′ = (n , n − 1, d)
are lost forever. υ(ni ,ne ) i e
The transition rates between the states identified above are
derived next. Transitions to a new state with k = i and k = e When the chosen action is to accept the request (a = G)
happen with a rate λi and λe , respectively. Additionally, states and the last arrival was an inelastic slice (k = i), the transition
with k = d are reached with a rate ni µi + ne µe depending probabilities are as follows. In case of an inelastic slice arrival,
the number of slices already in the system. Thus, the average which happens with a rate λi , the last arrival remains k = i,
time the system stays at state s, T̄ (s, a) is given by and in case of an elastic arrival it becomes k = e. The number
of inelastic slices increases by one unit in all cases except of an
1 inelastic departure (rate ni µi ). In case of an elastic departure
T̄ (s, a) = , (10)
υ (ni , ne ) (rate ne µe ), the number of elastic slices decreases by one.
Formally, for a = G and s = (ni , ne , i), we have:
where ni , and ne are the number of inelastic and elastic slices
in state s and υ (ni , ne ) = λi + λe + ni µi + ne µe . λi ′
υ(ni +1,ne ) , s = (ni + 1, ne , i)
We define a policy π (S), π (s) ∈ A, as a mapping from
λe
, s′ = (ni + 1, ne , e)
each state s to an action A. Thus, the policy determines P (s, a, s′ ) = υ(n i +1,ne )
. (13)
whether, for a given number of elastic and inelastic slices in
(ni +1)µi ′
υ(ni +1,ne ) , s = (ni , ne , d)
the system, we should admit a new request of an elastic or ne µe , s′ = (n + 1, n − 1, d)
υ(ni +1,ne ) i e
an inelastic slice upon each arrival. With the above analysis,
given such a policy, we can compute the probability of staying If the accepted slice is elastic (k = e), the system exhibits
at each of the possible states. Then, the long-term average a similar behavior to the one described above but increasing
8
τ ∑ ( ) ( )
+ P s, a, s′ Vn s′
T (s, a) ′
s
( ) ]
τ
+ 1− Vn (s) ∀s ∈ S
T (s, a)
3) Compute the boundaries
Mn = max (Vn+1 (s) − Vn (s))
s∈S Fig. 3: Example of optimal policy for elastic and inelastic
mn = min (Vn+1 (s) − Vn (s)) slices.
s∈S
6 N3AC: A DEEP LEARNING APPROACH some states by applying this knowledge to other similar states,
The Value Iteration algorithm described in Section 5.2 pro- which involves introducing a different F function. The key
vides the optimal policy for revenue maximization under the idea behind such generalization is to exploit the knowledge
framework described of Section 5.1. While this is very useful obtained from a fraction of the space state to derive the right
in order to obtain a benchmark for comparison, the algorithm action for other states with similar features. There are different
itself has a very high computational cost, which makes it generalization strategies that can be applied to RL algorithms.
impractical for real scenarios. Indeed, as the algorithm has to The most straightforward technique is the linear function
update all the V values V (s) , s ∈ S at each step, the running approximation [42]. With this technique, each state is given
time grows steeply with the size of the state space, and may as a linear combination of functions that are representative of
become too high for large scenarios. Moreover, the algorithm the system features. These functions are then updated using
is executed offline, and hence cannot be applied unless all standard regression techniques. While this approach is scalable
system parameters are known a priori. In this section, we and computationally efficient, the right selection of the feature
present an alternative approach, the Network-slicing Neural functions is a very hard problem. In our scenario, the Q-values
Network Admission Control (N3AC) algorithm, which has a associated to states with similar features (e.g., the number
low computational complexity and can be applied to practical of inelastic users) are increasingly non linear as the system
scenarios. becomes larger. As a result, linearization does not provide a
good performance in our case.
Neural Networks (NNs) are a more powerful and flexible
6.1 Deep Reinforcement Learning tool for generalization. NNs consist of simple, highly inter-
N3AC falls under category of the deep reinforcement learning connected elements called neurons that learn the statistical
(RL). With N3AC, an agent (the InP) interacts with the structure of the inputs if correctly trained. With this tool, the
environment and takes decisions at a given state, which lead design of neuron internals and the interconnection between
to a certain reward. These rewards are fed back into the agent, neurons are the most important design parameters. RL al-
which “learns” from the environment and the past decisions gorithms that employ NNs are called Deep RL algorithms:
using a learning function F. This learning function serves to N3AC belongs to this family. One of the key features of such
estimate the expected reward (in our case, the revenue). a NN-based approach is that it only requires storing a very
RL algorithms rely on an underlying Markovian system limited number of variables, corresponding to the weights and
such as the one described in Section 5. They provide the biases that compose the network architecture; yet, it is able
following features: (i) high scalability, as they learn online on to accurately estimate the F function for a very large number
an event basis while exploring the system and thus avoid a long of state/action pairs. In the rest of this Section, we review the
learning initial phase, (ii) the ability to adapt to the underlying NNs design principles (Section 6.2) and explain how these
system without requiring any a priori knowledge, as they principles are applied to a practical learning algorithm for our
learn by interacting with the system, and (iii) the flexibility system (Section 6.3).
to accommodate different learning functions F, which provide
the mapping from the input state to the expected reward when 6.2 Neural networks framework
taking a specific action.
The main distinguishing factor between different kinds of The fundamental building blocks of DRL algorithms are the
RL algorithms is the structure of the learning function F. following ones [43]:
Techniques such as Q-Learning [40] employ a lookup table for • A set of labeled data (i.e., system inputs for which
F, which limits their applicability due to the lack of scalability the corresponding outputs are known) which is used to
to a large space state [41]. In particular, Q-Learning solutions train the NN (i.e., teach the network to approximate the
need to store and update the expected reward value (i.e., the Q- features of the system).
value) for each state-action pair. As a result, learning the right • A loss function that measures the neural network perfor-
action for every state becomes infeasible when the space state mance in terms of training error (i.e., the error made when
grows, since this requires experiencing many times the same approximating the known output with the given input).
state-action pair before having a reliable estimation of the Q- • An optimization procedure that reduces the loss functions
value. This leads to extremely long convergence times that are at each iterations, making the NN eventually converge.
unsuitable for most practical applications. Additionally, storing There are many different Machine Learning (ML) schemes
and efficiently visiting the large number of states poses strong that make use of NNs, which are usually categorized as
requirements on the memory and computational footprint of supervised, unsupervised and RL. A ML system is supervised
the algorithm as the state space grows. For the specific case or unsupervised depending on whether the labeled data is
studied in this paper, the number of states in our model available or not, and it is a RL system when it interacts with the
increases exponentially with the number of network slicing environment receiving feedback from its experiences. N3AC
classes. Hence, when the number of network slicing classes falls under the latter category, and, within RL, it falls under
grows, the computational resources required rapidly become the category of deep RL. Since the seminal work in [16], deep
excessive. RL techniques have gained momentum and are nowadays one
A common technique to avoid the problems described above of the most popular approaches for RL. In spite of the bulk
for Q-learning is to generalize the experience learned from of literature available for such techniques, devising the N3AC
1
0
′
wi weigh t su singa G radientD escen tapp ro ach:t hem easu r ed
j
ij
erro ra tth eou tpu tl ay e
ri sb a
ck -p rop ag atedtoth einpu tl ay er
w ch ang ingth ew eight sv alu eso fe a chl ay erac cordingly[44 ].
wi
′′ j
Mor esp ecifi c a
lly,N3ACemp loy sth eRMSp rop[46 ]G rad ien t
Des cen ta lgo ri
thm .
Int eg ra t
ion w ithth e RLf ram ew o rk.On eo fth ec rit
i cal
requ ir em en tsfo r N3ACi stoop erat ew ithoutan yp reviou sly
knownou tpu t
,b utr atherin te
ra c
ting w i
thth een vironm entto
lea rni tsch ara cteris
ti cs
.Ind eed,i nN3ACw edono th a vean y
“groundt ru th”andt husw en eedtor el yone s
tim at
ionso ft he
s= wijxi outpu t,wh i chw i
llb e com emo rea c cur atea sw ek eepe xplo ring
ij th es ystem . Wh ileth isp roblemh asb eene xtensi ve
lys tud ied
inth el iteratur e[43 ],[47 ],w en e edtod e viseaso lu t
ionth atis
suit ablefo rth esp ecifi cp roblemadd r ess ed.InN3AC ,w et ak e
F
ig.4
:Ne
ura
lne
two
rksi
nte
rna
ls
. asth eou tpu to fth e NNth ea v erag er e v enuese xpecteda ta
givens t a
t ewh ent akingasp ecifi cd e cision .On ceth ed e cision
algor
ithmin volv
esanumbero
fde
signdec
isi
onstoadd
res
s ist ak en,th esy stemt ransitionstoth en ews tateandw em easu re
thes pecific
it
iesofo
urprob
lem
,wh
ichar
es ummar
ize
dinthe th ea ve r
ag er e venu er esu l
tingf romth ed ec
isiont aken( 0i n
following. cas eo far ejectionandρ tinc as eo fana cceptan ce).Th en ,t he
Neuronin ternalconfi gurat
ion.A nexem p
lary NNisillus- erro rb etw e enth ee st
im at edr ev enu eandth em easuredo ne
tra
tedinF igure4 ,w he
rew eh a vem ul
tip
lelayersofinter
- isu s edtot rainth e NN ,b a
ck-p rop ag atingth ise rrorin toth e
connec
t edn euronsorgani
zeda s:(i)a ninput
,(ii
)anou tpu
t weigh t s
. N3ACu sestw od if
feren t NN s:on etoe stimat eth e
and(iii)on eo r mo
rehiddenl aye
rs.An euronisan on-
linea
r re venu efo re a chs ta
t ewh enth es ele cteda ct
ioni stoa c
c ep tth e
elementthate x
ecute
sas o-
calleda c
tiva
tionfunc
tiontoth e in com ingr eq uest,andano t h
eron ewh enw erej ec
tth er eq u est
.
line
arcomb inat
iono ft hew eightedinput
s. Manyd if
feren
t Uponr ecei v
ingar equ es
t,N3ACp ollst hetw oNN sands el ects
act
iva
tionfun ctionsarea va
ilableint heli
tera
turespanning th ea ction w i
thth eh igh este xp ect edr e v enue;th en,a fterth e
fromal in
e a
rfun ct
iontom o
r ec ompl
e xonessucha stanh, tran sitiontot hen ews tatei sp erfo rm ed,th es ele c
ted NNi s
sigmoido rt h eR ec
tifiedL inearU n
it(R eLU)[44].N3AC train ed. Mo red eta
ilsabou ttheN3ACop era
tiona rep ro vid ed
employsth elatte
r. inth en e xts ection.
Exp l ora t
ion v s exp loita t
iont rad e- off. N3AC d rivesth e
Neu ralN etw orkS t ructure.O neoft hed e
sig nchoi cesthat sele ctiono fth eb esta ctiontob et ak ena te acht imes t ep.
need sto b et ak en w he nd ev
isinga NNa pproachi sthe Whil echoo s
ingth ea ctionth a
tm ax im i ze sth er evenu eate a ch
thew ayn euron sa rei nte
rconnectedam ongt hem.T h e most stepcon trib utesto m axim iz
ingth eo ver allrev enu e(re ferr ed
commons etupi sf eed-
f orward
,w he
ret hene uron
so fal ayer toa se xplo itations tep),ino rdertol e arnw ealson e
edtov i s
it
arefu llyin terconn e c
tedw i
tht heo neso ft hen ex
t. Th e
re new( st
illunknown )s ta
t ese veni fth ism aye ventu al
lyl e adto
area ls ooth e
rco nfi gu
r a
tions
,s ucha st h
ec onvolut
iona lor asubop t
im alr evenu e( ref er
redtoa se xp lora
tions tep).Th is
ther ecurrent(wh ereth eo u
tputisu sedasi n pu
tinth en ext pro cedu r
ei se sp ecia
l lyimpo rtantdu ringth einitia
lin tera ction
iteration).How ev e
r,theb e
stchoiceforas ysteml iket heon e ofth esy st em ,wh e
ree st
im ate
sa rev e ryin a
ccur ate.InN3AC ,
studi edinth isp ap eristhef e
ed-fo
rw ard
.I nd
eed,c onv o
lu t
iona
l th et rad e-offb etw eene xplo i
tat
ionande xp lorat
ioni sre gul ated
netw o rksar eu su al
l yem p
loyedforim a
ger ecogni
t i
on, wh i
le byth eγp ar am eter,wh ichind i cate sth ep robab i
lityo ft ak ing
recu rrentar eu sefulw h
e nthesyst
emi np u
tan dtheou tputh ave ane xplor at
ions t ep.Inth es e
tupu sedinth isp ap er
,w et ak e
ac e r
t ainde greeo fm u
tualrela
tion.N oneoft hesem a
t chour γ=0. 1.On ceth eNN sa r efullyt r ain ed,th esy stemgo e sin to
syst em , whichi sm em ory
lessasiti sb a
sedo na M a
rk ovi
an exp loit a
tionon ly,comp letelyom ittingth ee xplorationp art.
app ro ach.F urth erm ore
,o ur NNd es
i gnrel
i eso nas ing
le
hidd enl ayer.Su chad esignchoiceisd r
ivenb ythefo llowing
tw oob se
rv a
tion s:( i
)ith asbeenp rovent hatispo s
sibleto 6
.3 A
lgo
ri
thmdesc
rip
tion
app rox imatean yf u n
ctionu s
ing NN w ithas ingleh idden Inth efo l
l ow ing,w ed esc
ribethep roposedN3ACa lgori
thm.
lay er[45 ],and( ii)w hileal argern umbero fhi ddenl ayers Thisa lgorithmb uildsonth eN euralN etw orksf r
am ework
mayimp rov eth ea ccu
r acyoft he NN ,ita lsoin vo
lv esa descr
ib edabo ve,e xploi
tingRLtot r a
int h
ea lgorithmw ithouta
high erc omp lexitya ndlo nge
rtrain
ingp er
iod;a sar esult,one groundt ruths equ en ce
.Th ealgor
ithmcon s
istso fth efollowing
shou ldemp lo yth er equ
ir ednum b
ero fhiddenl ay
ersb uta vo
id highle velst eps( seeA lgori
thm2fo rth ep seudo code )
:
buildingal arg e
rn etw o
r kthanstr
ictlynecessary
. •St ep1 ,a cceptanced eci
sion:Ino rdertod ecidewh ethe
r
Back-propagationalgor
ithms el
ect
ion.I ncla
ssic
al MLap - toa c cepto rr e
jec
tanin comingr equ es
t,w elooka tthe
pl
icat
ions,theNNi st r
ainedusingalabeledda
tase
t:theNN e xpect eda v e
r ager ev
enuesr esu
ltingfroma cceptingand
isfedw iththeinpu
tsa ndthedif
feren
ceam ongtheest
im a
ted r ejec
tingar equestinth etw o NN s, wh ich w er e
ferto
outpu
ta ndthel ab
eli se v
alua
ted withtheerro
rf unc
tion. asth eQ -valu es
.Sp ec
ifical
ly,w ed efineQ( s,a )a sthe
Then,thee r
rori sproce
ssedt oadju
stt h
e NN w e
ight
sand e xpect edcumu l
ati
v erewardwh enstar
tingf romac e
rta
in
thusr educeth eer
rori nt h
en extite
rat
ion.N 3ACad jus
ts st at
esw i
tha ctiona,comp aredtoab a
selineσg ivenby
11
the optimal policy reward when starting from state 0, i.e., Algorithm 2 N3AC algorithm.
1) Initialize the Neural Network’s weights to random values.
∑
D(t)
2) An event is characterized by: s, a, s′ , r, t (the starting state,
Q (s, a) = E lim Rn − σt|s0 = s, a0 = a (17) the action taken, the landing state, the obtained reward and
t→∞
n=0 the transition time).
3) Estimate Q (s′ , a′ ) for each action a′ available in state s′
where D(t) is the number of requests received in a period through the NN.
4) Build the target value with the new sample observation as
t, Rn is the revenue obtained with the nth request and
∑D(t) follows:
σ = E[limt→∞ t n=0 Rn |s0 = 0] under the optimal
1
( ) ( )
policy. Then, we take the decision that yields the highest target = R s, a, s′ − σtn + max ′
Q s ′ , a′ (19)
a
Q-value. This procedure is used for elastic slices only, as
where tn is the transition time between two subsequent states
inelastic slices shall always be accepted as long as there is s and s′ after action a.
sufficient room. When there is no room for an additional 5) Train the NNs through RMSprop algorithm:
slice, requests are rejected automatically, regardless of 5.1 If s ̸∈ admissibility region boundary, train the NN with
their type. the error given by the difference between the above target
• Step 2, evaluation: By taking a decision in Step 1, the value (step 4) and the measured one.
5.2 Otherwise, train the NN corresponding to accepted re-
system experiences a transition from state s at step n, to
quests by applying a “penalty” and train the NN corre-
state s′ at step n+1. Once in step n+1, the algorithm has sponding to rejected requests as in step 5.1.
observed both the reward obtained during the transition
R (s, a) and a sample tn of the transition time. The • Step 4, learning finalization: Once the learning phase is
algorithm trains the weights of the corresponding NN over, the NN training stops. At this point, at a given
based on the error between the expected reward of s state we just take the the action that provides the highest
estimated at step n and the target value. This step relies expected reward.
on two cornerstone procedures: We remark that the learning phase of our algorithm does not
– Step 2a, back-propagation: This procedure drives the require specific training datasets. Instead, the algorithm learns
weights update by propagating the error measured back from the real slice requests on the fly, during the real operation
through all the NN layers, and updating the weights of the system; this is the so-called exploration phase. The
according to their gradient. The convergence time is training corresponding to such an exploration phase terminates
driven by a learning rate parameter that is used in the when the algorithm has converged to a good learning status,
weight updates. and is triggered again when the system detects changes in the
– Step 2b, target creation: This procedure is needed to system that require new training.
measure the accuracy of the NNs estimations during
the learning phase. At each iteration our algorithm
computes the observed revenue as follows: 7 P ERFORMANCE E VALUATION
In this section we evaluate the performance of N3AC via
ω = R (s, a, s′ ) − σtn + max
′
Q (s′ , a′ ) , (18) simulation. Unless otherwise stated, we consider a scenario
a
with four slice classes, two for elastic traffic and two for
where R (s, a, s′ ) is the revenue obtained in the transi- inelastic. Service times follow an exponential distribution with
tion to the new state. As we do not have labeled data, µ = 5 for all network slices classes, and arrivals follow a
we use ω to estimate the error, by taking the difference Poisson process with average rates equal to λi = 2µ and
between ω and the previous estimate Qn+1 (s, a) and λe = 10λi for the elastic and inelastic classes, respectively.
using it to train the NN. When the NN eventually We consider two network slice sizes, equal to C/10 and C/20,
converges, ω will be close to the Q-values estimates. where C is the total network capacity. Similarly, we set the
• Step 3, penalization: When a state in the boundary of the throughput required guarantees for elastic and inelastic traffic
admissibility region is reached, the system is forced to to Ri = Re = Cb /10. Two key parameters that will be
reject the request. This should be avoided as it may force employed throughout the performance evaluation are ρe and
the system to reject potentially high rewarding slices. To ρi , the average revenue per time unit generated by elastic
avoid such cases, N3AC introduces a penalty on the Q- and inelastic slices, respectively (in particular, performance
values every time the system reaches the border of the depends on the ratio between them). The rest of the network
admissibility region. With this approach, if the system is setup (including the users’ mobility model) is based on the
brought to the boundary through a sequence of highly scenario described in Section 4.2.
rewarding actions, the penalty will have small effect Following the N3AC algorithm proposed in the previous
as the Q-values will remain high even after applying section, we employ two feed-forward NNs, one for accepted
the penalty. Instead, if the system reaches the boundary requests and another one for rejected. Each neuron applies
following a chain of poorly rewarding actions, the impact a ReLU activation function, and we train them during the
on the involved Q-values will be much higher, making it exploration phase using the NNs RMSprop algorithm imple-
unlikely that the same sequence of decisions is chosen in mentation available in Keras (https://keras.io/); the learning
the future. parameter of the RMSprop Gradient Descent algorithm [46] is
12
PDF
PDF
N3AC N3AC
0.6 0.7 0.8 0.9 1.0 0.6 0.7 0.8 0.9 1.0
Relative revenue Relative revenue
ACKNOWLEDGMENTS
The work of University Carlos III of Madrid was sup-
ported by the H2020 5G-MoNArch project (Grant Agreement
Fig. 10: Revenue vs. ρi /ρe . No. 761445) and the 5GCity project of the Spanish Ministry
of Economy and Competitiveness (TEC2016-76795-C6-3-R).
The work of NEC Laboratories Europe was supported by the
scenario that includes the traffic types corresponding to the
5G-Transformer project (Grant Agreement No. 761536).
four service classes defined by 3GPP [48] (hereafter we refer
to them as class 1 to class 4, where class 1 is the one with
most stringent delay requirements). In line with the analysis R EFERENCES
of Section 4, for this scenario with 4 different traffic types we [1] D. Bega, M. Gramaglia, C. J. Bernardos Cano, A. Banchs, and X. Costa-
Perez, “Toward the network of the future: From enabling technologies
take the admissibility region A∗ given by (i) |T1 | ≤ T1max , to 5G concepts,” Transactions on Emerging Telecommunications Tech-
(ii) |T1 | + |T2 | ≤ T2max , (iii) |T1 | + |T2 | + |T3 | ≤ T3max , and nologies, vol. 28, no. 8, pp. 3205–3216, Jun. 2017.
(iv) |T1 |+|T2 |+|T3 |+|T4 | ≤ T4max . For this scenario, we run [2] NGMN Alliance, “Description of Network Slicing Concept,” Public
Deliverable, 2016.
the same experiment as in Section 7.3, varying the price ratio [3] ITU-R, “Minimum requirements related to technical performance for
ρ among different classes as follows: rk = ρ · rk+1 ∀k, where imt-2020 radio interface(s),” Technical Report, 2017.
rk is the revenue generated by class k. Figure 10 compares the [4] 3GPP, “Study on Architecture for Next Generation System,” TS 23.799,
v2.0.0, Dec. 2016.
performance provided by the N3AC algorithm in this scenario [5] NGMN Alliance, “5G White Paper,” White Paper, Feb. 2015.
against the one provided by naive policies which only accept [6] A. Gudipati, L. Li, and S. Katti, “RadioVisor: A Slicing Plane for Radio
a subset of classes. We can observe that N3AC provides very Access Networks,” in Proc. ACM HotSDN, Chicago, Illinois, Aug. 2014,
pp. 237–238.
high gains when compared to all the naive policies, which [7] I. Malanchini, S. Valentin, and O. Aydin, “Generalized resource sharing
confirms that our approach can be successfully applied to for multiple operators in cellular wireless networks,” in Proc. IEEE
scenarios with more traffic types, such as, e.g., the 3GPP IWCMC, Nicosia, Cyprus, Aug. 2014, pp. 803–808.
[8] R. Mahindra, M. A. Khojastepour, H. Zhang, and S. Rangarajan, “Radio
service classes. Access Network sharing in cellular networks,” in Proc. IEEE ICNP,
Göttingen, Germany, Oct. 2013, pp. 1–10.
8 C ONCLUSION [9] S. Rathinakumar and M. Marina, “GAVEL: Strategy-proof Ascending
Bid Auction for Dynamic Licensed Shared Access,” in Proc. ACM
Network Slicing will be one of the pillars of future 5G MobiHoc, Paderborn, Germany, Jul. 2016, pp. 121–130.
networks. It is expected that this new paradigm will bring new [10] X. Zhou, R. Li, T. Chen, and H. Zhang, “Network slicing as a service:
players to the business: Infrastructure Providers will sell their enabling enterprises’ own software-defined cellular networks,” IEEE
Commun. Mag., vol. 54, no. 7, pp. 146–153, Jul. 2016.
resources to tenants which, in turn, provide a service to their [11] K. Samdanis, X. Costa-Perez, and V. Sciancalepore, “From network
users. An open problem within this model is how to admit sharing to multi-tenancy: The 5G network slice broker,” IEEE Com-
requests from the tenants, ensuring that the corresponding mun. Mag., vol. 54, no. 7, pp. 32–39, Jul. 2016.
[12] V. Sciancalepore, K. Samdanis, X. Costa-Perez, D. Bega, M. Gramaglia,
SLAs will be satisfied while maximizing the monetization and A. Banchs, “Mobile traffic forecasting for maximizing 5G network
of the Infrastructure Provider. In this paper we propose a slicing resource utilization,” in Proc. IEEE INFOCOM, Atlanta, USA,
machine learning approach to address this problem. To this May 2017, pp. 2583–2591.
[13] M. A. Habibi, B. Han, and H. D. Schotten, “Network Slicing in 5G
aim, we first present a model based on SMDP for the decision- Mobile Communication Architecture, Profit Modeling, and Challenges,”
making process and formulate the optimal revenue problem. arXiv:1707.00852, 2017.
15
[14] B. Han, S. Tayade, and H. D. Schotten, “Modeling profit of sliced [44] F. Chollet, Deep learning with Python. Manning Publications, 2018.
5G networks for advanced network resource management and slice [45] K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward
implementation,” in Proc. IEEE ISCC, Heraklion, Greece, Jul. 2017, networks are universal approximators,” Neural networks, vol. 2, no. 5,
pp. 576–581. pp. 359–366, Jan. 1989.
[15] B. Han, D. Feng, L. Ji, and H. D. Schotten, “A Profit-Maximizing [46] T. Tieleman and G. Hinton, “RMSprop gradient optimization,” http://
Strategy of Network Resource Management for 5G Tenant Slices,” www.cs.toronto.edu/tijmen/csc321/slides/lecture slides lec6.pdf, 2014.
arXiv:1709.09229, 2017. [47] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521,
[16] V. Mnih et al., “Playing atari with deep reinforcement learning,” no. 7553, pp. 436–444, May 2015.
arXiv:1312.5602, 2013. [48] 3GPP, “Technical specification group services ans system aspects; policy
[17] V. Mnih et al., “Human-level control through deep reinforcement learn- and charging control architecture,” TS 23.203, v15.1.0, Dec. 2017.
ing,” Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015.
[18] D. Silver, et al., “Mastering the game of Go with deep neural networks
and tree search,” Nature, vol. 529, no. 7587, pp. 484–489, Jan. 2016.
[19] A. Graves et al., “Hybrid computing using a neural network with
dynamic external memory,” Nature, vol. 538, no. 7626, pp. 471–476,
Oct. 2016.
[20] V. Mnih et al., “Asynchronous methods for deep reinforcement learning,”
in Proc. International Conference on Machine Learning, New York,
USA, Jun. 2016, pp. 1928–1937.
[21] A. Tamar, Y. Wu, G. Thomas, S. Levine, and P. Abbeel, “Value iteration
networks,” in Proc. NIPS, Barcelona, Spain, Dec. 2016, pp. 2154–2162.
[22] P. Mirowski et al., “Learning to navigate in complex environments,”
arXiv:1611.03673, 2016.
[23] P. H. Su et al., “On-line active reward learning for policy optimisation
in spoken dialogue systems,” arXiv:1605.07669, 2016.
[24] B. Malila, O. Falowo, and N. Ventura, “Intelligent NLOS Backhaul for
5G Small Cells,” IEEE Commun. Lett., vol. 22, no. 1, pp. 189–192,
Jan. 2018.
[25] H. Tong and T. X. Brown, “Adaptive call admission control under
quality of service constraints: a reinforcement learning solution,” IEEE
J. Sel. Areas Commun., vol. 18, no. 2, pp. 209–221, Feb. 2000.
[26] C. Jiang, H. Zhang, Y. Ren, Z. Han, K. C. Chen, and L. Hanzo,
“Machine Learning Paradigms for Next-Generation Wireless Networks,”
IEEE Wireless Commun., vol. 24, no. 2, pp. 98–105, Dec. 2017.
[27] A. P. Iyer, L. E. Li, and I. Stoica, “Automating Diagnosis of Cellular
Radio Access Network Problems,” in Proc. ACM MobiCom, Salt Lake
City, USA, Oct. 2017, pp. 79–87.
[28] T. O’Shea and J. Hoydis, “An introduction to deep learning for the
physical layer,” IEEE Trans. on Cogn. Commun. Netw., vol. 3, no. 4,
pp. 563–575, Dec. 2017.
[29] H. Sun, X. Chen, Q. Shi, M. Hong, X. Fu, and N. D. Sidiropoulos,
“Learning to optimize: Training deep neural networks for wireless
resource management,” arXiv:1705.09412, 2017.
[30] D. Bega, M. Gramaglia, A. Banchs, V. Sciancalepore, K. Samdanis, and
X. Costa-Perez, “Optimising 5G Infrastructure Markets: The Business of
Network Slicing,” in Proc. IEEE INFOCOM, Atlanta, USA, May 2017,
pp. 910–918.
[31] Will Townsend, “Will 5G Change Lives? Infrastructure Providers,
Carriers, Even Dell EMC And HPE Are Betting It Will,” Forbes, 2017.
[Online]. Available: https://www.forbes.com/sites/moorinsights/2017/
06/21/will-5g-change-lives-infrastructure-providers-carriers-even-dell-
emc-and-hpe-are-betting-it-will/
[32] S. Shenker, “Fundamental design issues for the future Internet,” IEEE
J. Sel. Areas Commun., vol. 13, no. 7, pp. 1176–1188, Sep. 1995.
[33] M. Berenson, D. Levine, K. A. Szabat, and T. C. Krehbiel, Basic
business statistics: Concepts and applications. Pearson higher education
AU, 2014.
[34] J. L. Devore, Probability and Statistics for Engineering and the Sciences.
Cengage learning, 2011.
[35] ITU-R, “Guidelines for evaluation of radio interface technologies for
imt-advanced,” Report ITU-R M.2135-1, 2016.
[36] R. Bellman, “A Markovian decision process,” DTIC, Tech. Rep., 1957.
[37] R. Howard, Dynamic Programming and Markov Processes. Technology
Press-Wiley, 1960.
[38] S. Lippman, “Applying a New Device in the Optimization of Exponential
Queuing Systems,” Operation Research, vol. 23, no. 4, pp. 687–710,
Aug. 1975.
[39] H. Tijms, A First Course in Stochastic Models. J. Wiley & Sons, 2003.
[40] C. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, no. 3-
4, pp. 279–292, 1992.
[41] S. J. Russell, P. Norvig, J. F. Canny, J. M. Malik, and D. D. Edwards,
Artificial Intelligence: a Modern Approach. Prentice hall Upper Saddle
River, 2003, vol. 2, no. 9.
[42] F. S. Melo and M. I. Ribeiro, “Q-learning with linear function approx-
imation,” in Proc. COLT, San Diego, USA, Jun. 2007, pp. 308–322.
[43] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep learning.
MIT press Cambridge, 2016.