Professional Documents
Culture Documents
01 ISP Network Design PDF
01 ISP Network Design PDF
ISP Workshops
2
Point of Presence
Topologies
3
PoP Topologies
p Core routers high speed trunk
connections
p Distribution routers and Access routers
high port density
p Border routers connections to other
providers
p Service routers hosting and servers
p Some functions might be handled by a
single router
4
PoP Design
p ModularDesign
p Aggregation Services separated according
to
n connection speed
n customer service
n contention ratio
n security considerations
5
Modular PoP Design
Other ISPs
Web Cache
ISP Services Hosted Services &
(DNS, Mail, News, Datacentre
FTP, WWW)
Network
Core
Consumer cable,
Consumer xDSL and
Dial Access wireless Access
8
PoP Modules
p Low Speed customer connections
n ADSL2, Cable, Public Wireless, PSTN access
n Low bandwidth needs
n Low revenue, large numbers
9
PoP Modules
p Highband Business customer connections
n From 100Mbps to over 1Gbps
n Ethernet, Fibre, or high end SDH
n High bandwidth needs
n High revenue, low numbers
10
PoP Modules
p PoP Core
n Two dedicated routers
n High Speed interconnect
n Backbone Links ONLY
n Do not touch them!
p Border Network
n Dedicated border router to other ISPs
n The ISPs front door
n Transparent web caching?
n Two in backbone is minimum guarantee for
redundancy 11
PoP Modules
p ISP Services
n DNS (cache, secondary)
n News (still relevant?)
n Mail (POP3, Relay, Anti-virus/anti-spam)
n WWW (server, proxy, cache)
p Hosted Services/DataCentres
n Virtual Web, WWW (server, proxy, cache)
n Information/Content Services
n Electronic Commerce
12
PoP Modules
p Network Operations Centre
n Consider primary and backup locations
n Network monitoring
n Statistics and log gathering
n Direct but secure access
13
PSTN & Broadband Access Module
Web Cache
DSLAM
Telephone Network BRAS
Access Network
Gateway Routers
IP, ATM
Cable RAS
To Core Routers
Cable System
Primary Rate
T1/E1
Aggregation Edge
Channelised T1/E1
Metro Ethernet
To Core Routers
15
High Speed Access Module
Aggregation Edge
Metro Ethernet
Channelised OC3/OC12
16
ISP Services Module
To core routers
Service Network
Gateway Routers
17
Hosted Services Module
To core routers
Hosted Network
Gateway Routers
18
Border Module
Network
Border Routers
To core routers
19
NOC Module
Critical Services
To core routers Module
Corporate LAN
Out of Band
Hosted Network
Management Network Gateway Routers Firewall
Billing, Database
and Accounting
Systems
NetFlow TACACS+ SYSLOG Primary DNS
Analyser server server
Out of Band
Management Network
Router
consoles
Terminal server
To the NOC
NetFlow
enabled
routers
NetFlow
Collector
21
Backbone Network
Design
22
Backbone Design
p Routed Backbone
p Switched Backbone
n ATM/Frame Relay core network
n Now obsolete
p Point-to-point circuits
n nx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48,
10GigE, OC192, OC768, 100GE
p ATM/Frame Relay service from telco
n T3, OC3, OC12, delivery
n Easily upgradeable bandwidth (CIR)
n Almost vanished in availability now
23
Distributed Network Design
p PoP design standardised
n operational scalability and simplicity
p ISP essential services distributed around
backbone
p NOC and backup NOC
p Redundant backbone links
24
Distributed Network Design
Customer
ISP Services connections
Backup
Operations Centre POP Two
Customer Customer
connections connections
ISP Services
ISP Services
External External
connections connections
Operations Centre
25
Backbone Links
p ATM/Frame Relay
n Virtually disappeared due to overhead, extra
equipment, and shared with other customers
of the telco
n MPLS has replaced ATM & FR as the telco
favourite
p Leased Line/Circuit
n Most popular with backbone providers
n IP over Optics and Metro Ethernet very
common in many parts of the world
26
Long Distance Backbone Links
p These usually cost more
p Important to plan for the future
n This means at least two years ahead
n Stay in budget, stay realistic
n Unplanned emergency upgrades will be
disruptive without redundancy in the network
infrastructure
27
Long Distance Backbone Links
p Allow
sufficient capacity on alternative
paths for failure situations
n Sufficient can depend on the business strategy
n Sufficient can be as little as 20%
n Sufficient is usually over 50% as this offers
business continuity for customers in the case
of link failure
n Some businesses choose 0%
p Very short sighted, meaning they have no spare
capacity at all!!
28
Long Distance Links
POP Two
Long distance link
Alternative/Backup Path
29
Metropolitan Area Backbone Links
p Tend to be cheaper
n Circuit concentration
n Choose from multiple suppliers
p Think big
n More redundancy
n Less impact of upgrades
n Less impact of failures
30
Metropolitan Area Backbone Links
POP Two
Metropolitan Links
Metropolitan Links
32
Transits
p Transit provider is another autonomous system
which is used to provide the local network with
access to other networks
n Might be local or regional only
n But more usually the whole Internet
p Transit providers need to be chosen wisely:
n Only one
p no redundancy
n Too many
p more difficult to load balance
p no economy of scale (costs more per Mbps)
p hard to provide service quality
p Recommendation: at least two, no more
than three
Common Mistakes
p ISPs sign up with too many transit providers
n Lots of small circuits (cost more per Mbps than larger
ones)
n Transit rates per Mbps reduce with increasing transit
bandwidth purchased
n Hard to implement reliable traffic engineering that
doesnt need daily fine tuning depending on customer
activities
p No diversity
n Chosen transit providers all reached over same satellite
or same submarine cable
n Chosen transit providers have poor onward transit and
peering
Peers
p A peer is another autonomous system with which
the local network has agreed to exchange locally
sourced routes and traffic
p Private peer
n Private link between two providers for the purpose of
interconnecting
p Public peer
n Internet Exchange Point, where providers meet and
freely decide who they will interconnect with
p Recommendation: peer as much as possible!
Common Mistakes
p Mistaking a transit providers Exchange
business for a no-cost public peering point
p Not working hard to get as much peering
as possible
n Physically near a peering point (IXP) but not
present at it
n (Transit sometimes is cheaper than peering!!)
p Ignoring/avoiding
competitors because
they are competition
n Even though potentially valuable peering
partner to give customers a better experience
Private Interconnection
p Twoservice providers agree to
interconnect their networks
n They exchange prefixes they originate into the
routing system (usually their aggregated
address blocks)
n They share the cost of the infrastructure to
interconnect
p Typically each paying half the cost of the link (be it
circuit, satellite, microwave, fibre,)
p Connected to their respective peering routers
PR ISP2
PR
ISP1
p PR = peering router
n Runs iBGP (internal) and eBGP (with peer)
n No default route
n No full BGP table
n Domestic prefixes only
p Peering router used for all private interconnects38
Public Interconnection
p Service
provider participates in an
Internet Exchange Point
n It exchanges prefixes it originates into the
routing system with the participants of the IXP
n It chooses who to peer with at the IXP
p Bi-lateral peering (like private interconnect)
p Multi-lateral peering (via IXPs route server)
39
Public Interconnection
Upstream
ISP6-PR
ISP5-PR
ISP2-PR
ISP3-PR
IXP
ISP1-PR1 ISP1
ISP4-PR
ISP2-PR
ISP3-PR
43
Local Transit Provider
AR ISP1
BR
Transit
AR1
ISP1
Transit BR
AR2
47
Distant Transit Provider
AR1
ISP1
Transit BR1
AR2
BR2
p Upgrade scenario:
n Provision two routers
n Two independent circuits
n Consider second transit provider and/or turning up at
an IXP
48
Summary
p Design considerations for:
n Private interconnects
p Simple private peering
n Public interconnects
p Router co-lo at an IXP
n Local transit provider
p Simple upstream interconnect
n Long distance transit provider
p Router remote co-lo at datacentre or Transit
premises
49
Upstream Connectivity
and Peering Case Study
How Seacom chose their
international peering locations
and transit providers
50
Objective
p Obtain high grade Internet connectivity for
the wholesale market in Africa to the rest
of the world
p Emphasis on:
n Reliability
n Interconnectivity density
n Scalability
51
Metrics Needed in Determining
Solution (1)
p Focusing on operators that cover the destinations
mostly required by Africa
n i.e., English-speaking (Europe, North America)
p Include providers with good connectivity into
South America and the Asia Pacific.
p Little need for providers who are strong in the
Middle East, as demand from Africa for those
regions is very, very low.
52
Metrics Needed in Determining
Solution (2)
p Split the operators between Marseille (where the
SEACOM cable lands) and London (where there is
good Internet density)
n To avoid outages due to backhaul failure across Europe
n And still maintain good access to the Internet
p Look at providers who are of similar size so as
not to fidget too much (or at all) with BGP tuning.
p The providers needed to support:
n 10Gbps ports
n Bursting bandwidth/billing
n Future support for 100Gbps or N x 10Gbps
53
Metrics Needed in Determining
Solution (3)
p Implement peering at major exchange points in
Europe
n To off-set long term operating costs re: upstream
providers.
54
Implementing Solution
p Connected to Level(3) and GT-T (formerly
Inteliquent, formerly Tinet) in Marseille
p Connected to NTT and TeliaSonera in London
p Peered in London (LINX)
p Peered in Amsterdam (AMS-IX)
p BGP setup to prefer traffic being exchanged at
LINX and AMS-IX
p BGP setup to prefer traffic over the upstreams
that we could not peer away
p No additional tuning done on either peered or
transit traffic, i.e., no prepending, no de-
aggregation, etc. All traffic setup to flow naturally55
End Result
p 50% of traffic peered away in less than 2x
months of peering at LINX and AMS-IX
p 50% of traffic handled by upstream providers
p Equal traffic being handled by Level(3) and GT-T
in Marseille
p Equal traffic being handled by TeliaSonera and
NTT in London
p Traffic distribution ratios across all the transit
providers is some 1:1:0.9:0.9
p This has been steady state for the last 12x
months
n No BGP tuning has been done at all 56
Addressing
57
Where to get IP addresses and AS
numbers
p Your upstream ISP
p Africa
n AfriNIC http://www.afrinic.net
p Asia and the Pacific
n APNIC http://www.apnic.net
p North America
n ARIN http://www.arin.net
p Latin America and the Caribbean
n LACNIC http://www.lacnic.net
p Europe and Middle East
n RIPE NCC http://www.ripe.net/info/ncc
58
Internet Registry Regions
59
Getting IP address space
p Take part of upstream ISPs PA space
or
p Become a member of your Regional Internet
Registry and get your own allocation
n Require a plan for a year ahead
n General policies are outlined in RFC2050, more
specific details are on the individual RIR website
p There is no more IPv4 address space at IANA
n APNIC and RIPE NCC are now in their final /8 IPv4
delegation policy phase
n Limited IPv4 available
n IPv6 allocations are simple to get in most RIR regions
60
What about RFC1918 addressing?
p RFC1918 defines IPv4 addresses reserved for
private Internets
n Not to be used on Internet backbones
n http://www.ietf.org/rfc/rfc1918.txt
p Commonly used within end-user networks
n NAT used to translate from private internal to public
external addressing
n Allows the end-user network to migrate ISPs without a
major internal renumbering exercise
p ISPs must filter RFC1918 addressing at their
network edge
n http://www.cymru.com/Documents/bogon-
list.html 61
What about RFC1918 addressing?
p There is a long list of well known problems:
n http://www.rfc-editor.org/rfc/rfc6752.txt
p Including:
n False belief it conserves address space
n Adverse effects on Traceroute
n Effects on Path MTU Discovery
n Unexpected interactions with some NAT implementations
n Interactions with edge anti-spoofing techniques
n Peering using loopbacks
n Adverse DNS Interaction
n Serious Operational and Troubleshooting issues
n Security Issues
p false sense of security, defeating existing security 62
techniques
Private versus Globally Routable IP
Addressing
p Infrastructure Security: not improved by using
private addressing
n Still can be attacked from inside, or from customers, or
by reflection techniques from the outside
p Troubleshooting: made an order of magnitude
harder
n No Internet view from routers
n Other ISPs cannot distinguish between down and broken
p Summary:
n ALWAYS use globally routable IP addressing for ISP
Infrastructure
63
Addressing Plans ISP
Infrastructure
p Address block for router loop-back interfaces
p Address block for infrastructure
n Per PoP or whole backbone
n Summarise between sites if it makes sense
n Allocate according to genuine requirements, not historic
classful boundaries
p Similar allocation policies should be used for IPv6
as well
n ISPs just get a substantially larger block (relatively) so
assignments within the backbone are easier to make
64
Addressing Plans Customer
p Customers are assigned address space
according to need
p Should not be reserved or assigned on a
per PoP basis
n ISP iBGP carries customer nets
n Aggregation not required and usually not
desirable
65
Addressing Plans ISP Infrastructure
p Phase One
223.10.0.0/21
223.10.0.1
223.10.5.255 /24 /24
223.10.15.255
67
Addressing Plans (contd)
p Document infrastructure allocation
n Eases operation, debugging and management
p Document customer allocation
n Contained in iBGP
n Eases operation, debugging and management
n Submit network object to RIR Database
68
Routing Protocols
69
Routing Protocols
p IGP Interior Gateway Protocol
n Carries infrastructure addresses, point-to-point
links
n Examples are OSPF, ISIS,...
70
Why Do We Need an IGP?
p ISP backbone scaling
n Hierarchy
n Modular infrastructure construction
n Limiting scope of failure
n Healing of infrastructure faults using dynamic
routing with fast convergence
71
Why Do We Need an EGP?
p Scaling to large network
n Hierarchy
n Limit scope of failure
p Policy
n Control reachability to prefixes
n Merge separate organizations
n Connect multiple IGPs
72
Interior versus Exterior Routing
Protocols
p Interior p Exterior
n Automatic neighbour n Specifically configured
discovery peers
n Generally trust your IGP n Connecting with outside
routers networks
n Prefixes go to all IGP n Set administrative
routers boundaries
n Binds routers in one AS n Binds ASs together
together
73
Interior versus Exterior Routing
Protocols
p Interior p Exterior
n Carries ISP n Carries customer
infrastructure addresses prefixes
only n Carries Internet
n ISPs aim to keep the prefixes
IGP small for efficiency n EGPs are independent
and scalability of ISP network topology
74
Hierarchy of Routing Protocols
Other ISPs
BGP4
BGP4
and OSPF/ISIS
BGP4 Static/BGP4
Customers
IXP
75
Routing Protocols:
Choosing an IGP
p OSPF and ISIS have very similar properties
n Review the ISIS vs OSPF presentation
p Which to choose?
n Choose which is appropriate for your operators
experience
n In most vendor releases, both OSPF and ISIS have
sufficient nerd knobs to tweak the IGPs behaviour
n OSPF runs on IP
n ISIS runs on infrastructure, alongside IP
n ISIS supports both IPv4 and IPv6
n OSPFv2 (IPv4) plus OSPFv3 (IPv6)
76
Routing Protocols:
IGP Recommendations
p Keep the IGP routing table as small as possible
n If you can count the routers and the point-to-point links
in the backbone, that total is the number of IGP entries
you should see
p IGP details:
n Should only have router loopbacks, backbone WAN
point-to-point link addresses, and network addresses of
any LANs having an IGP running on them
n Strongly recommended to use inter-router
authentication
n Use inter-area summarisation if possible
77
Routing Protocols:
More IGP recommendations
p To fine tune IGP table size more, consider:
n Using ip unnumbered on customer point-to-
point links saves carrying that /30 in IGP
p (If customer point-to-point /30 is required for
monitoring purposes, then put this in iBGP)
n Use contiguous addresses for backbone WAN
links in each area then summarise into
backbone area
n Dont summarise router loopback addresses
as iBGP needs those (for next-hop)
n Use iBGP for carrying anything which does not
contribute to the IGP Routing process
78
Routing Protocols:
iBGP Recommendations
p iBGP
should carry everything which
doesnt contribute to the IGP routing
process
n Internet routing table
n Customer assigned addresses
n Customer point-to-point links
n Access network dynamic address pools,
passive LANs, etc
79
Routing Protocols:
More iBGP Recommendations
p Scalable iBGP features:
n Use neighbour authentication
n Use peer-groups to speed update process and
for configuration efficiency
n Use communities for ease of filtering
n Use route-reflector hierarchy
p Route reflector pair per PoP (overlaid clusters)
80
Security
81
Security
p ISP Infrastructure security
p ISP Network security
p Security is not optional!
p ISPs need to:
n Protect themselves
n Help protect their customers from the Internet
n Protect the Internet from their customers
p The following slides are general recommendations
n Do more research on security before deploying any
network
82
ISP Infrastructure Security
p Router & Switch Security
n Use Secure Shell (SSH) for device access &
management
p Do NOT use Telnet
n Device management access filters should only
allow NOC and device-to-device access
p Do NOT allow external access
n Use TACACS+ for user authentication and
authorisation
p Do NOT create user accounts on routers/switches
83
ISP Infrastructure Security
p Remote access
n For Operations Engineers who need access
while not in the NOC
n Create an SSH server host (this is all it does)
p Or a Secure VPN access server
n Ops Engineers connect here, and then they can
access the NOC and network devices
84
ISP Infrastructure Security
p Other network devices?
n These probably do not have sophisticated security
techniques like routers or switches do
n Protect them at the LAN or point-to-point ingress (on
router)
p Servers and Services?
n Protect servers on the LAN interface on the router
n Consider using iptables &c on the servers too
p SNMP
n Apply access-list to the SNMP ports
n Should only be accessible by management system, not
the world
85
ISP Infrastructure Security
p General Advice:
n Routers, Switches and other network devices
should not be contactable from outside the AS
n Achieved by blocking typical management
access protocols for the infrastructure address
block at the network perimeter
p E.g. ssh, telnet, http, snmp,
n Use the ICSI Netalyser to check access levels:
p http://netalyzr.icsi.berkeley.edu
n Dont block everything: BGP, traceroute
and ICMP still need to work!
86
ISP Network Security
p Effective filtering
n Protect network borders from traffic which
should not be on the public Internet, for
example:
p LAN protocols (eg netbios)
p Well known exploit ports (used by worms and
viruses)
p Drop traffic arriving and going to private and non-
routable address space (IPv4 and IPv6)
n Achieved by packet filters on border routers
p Remote trigger blackhole filtering
87
ISP Network Security RTBF
p Remote trigger blackhole filtering
n ISP NOC injects prefixes which should not be accessible
across the AS into the iBGP
n Prefixes have next hop pointing to a blackhole address
n All iBGP speaking backbone routers configured to point
the blackhole address to the null interface
n Traffic destined to these blackhole prefixes are dropped
by the first router they reach
p Application:
n Any prefixes (including RFC1918) which should not have
routability across the ISP backbone
88
ISP Network Security RTBF
p Remote trigger blackhole filtering example:
n Origin router:
router bgp 64509
redistribute static route-map black-hole-trigger
!
ip route 10.5.1.3 255.255.255.255 Null0 tag 66
!
route-map black-hole-trigger permit 10
match tag 66
set local-preference 1000
set community no-export
set ip next-hop 192.0.2.1
!
91
What is uRPF?
FIB:
172.16.1.0/24 fa0/0
192.168.1.0/24 se0/1
src=172.16.1.1 fa0/0 se0/1
router
src=192.168.1.1
93
Out of Band Management
94
Out of Band Management
p Not optional!
p Allows access to network equipment in
times of failure
p Ensures quality of service to customers
n Minimises downtime
n Minimises repair time
n Eases diagnostics and debugging
95
Out of Band Management
p OoB Example Access server:
n modem attached to allow NOC dial in
n console ports of all network equipment
connected to serial ports
n LAN and/or WAN link connects to network
core, or via separate management link to NOC
p Full
remote control access under all
circumstances
96
Out of Band Network
Equipment Rack Equipment Rack
Router, switch
and ISP server
consoles
Modem access
to PSTN for out of
band dialin
Ethernet
97
to the NOC
Out of Band Management
p OoB Example Statistics gathering:
n Routers are NetFlow and syslog enabled
n Management data is congestion/failure
sensitive
n Ensures management data integrity in case of
failure
p Full
remote information under all
circumstances
98
Test Laboratory
99
Test Laboratory
p Designed to look like a typical PoP
n Operated like a typical PoP
p Used to trial new services or new software
under realistic conditions
p Allows discovery and fixing of potential
problems before they are introduced to
the network
100
Test Laboratory
p Some ISPs dedicate equipment to the lab
p Other ISPs purchase ahead so that
todays lab equipment becomes
tomorrows PoP equipment
p Other ISPs use lab equipment for hot
spares in the event of hardware failure
101
Test Laboratory
p Cant afford a test lab?
n Set aside one spare router and server to trial
new services
n Never ever try out new hardware, software or
services on the live network
p Everymajor ISP in the US and Europe has
a test lab
n Its a serious consideration
102
Operational
Considerations
103
Operational Considerations
105
Operational Considerations
Support
p Differentiate
between customer support
and the Network Operations Centre
n Customer support fixes customer problems
n NOC deals with and fixes backbone and
Internet related problems
p Network Engineering team is last resort
n They design the next generation network,
improve the routing design, implement new
services, etc
n They do not and should not be doing support!
106
Operational Considerations
NOC Communications
p NOC should know contact details for
equivalent NOCs in upstream providers
and peers
p Or consider joining the INOC-DBA system
n Voice over IP phone system using SIP
n Runs over the Internet
n www.pch.net/inoc-dba for more information
107
ISP Network Design
Summary
108
ISP Design Summary
p KEEP IT SIMPLE & STUPID ! (KISS)
p Simple is elegant is scalable
p Use Redundancy, Security, and
Technology to make life easier for yourself
p Above all, ensure quality of service for
your customers
109
ISP Network Design
ISP Workshops
110