Service Description

Cisco

Cisco's philosophy appears similar to that of some PC OS vendors: enable as many protocols and features as possible by default, so the device works out-of-the-box in most situations. Unfortunately, this means that many unnecessary features are turned on which, while harmless in LAN or corporate environments, may cause undesired traffic on an Internet exchange.
Typical items that need to be disabled are:

  • autoconfiguration protocols (DHCP, BOOTP, TFTP config download over the ECIX interface),
  • CDP
  • DEC MOP
  • IP redirects
  • IP directed broadcasts
  • proxy ARP-
  • IPv6 Router Advertisements
  • keepalive

Intermediate switches or hybrid devices will also need to disable VTP, STP, etc.

Global Config

! Do not run a DHCP server/relay agent
no service dhcp
! Older IOS versions require this instead of the above.
no ip bootp server
! Do not download configs through TFTP
no service config
! Do not run CDP
no cdp run
5.2. Interface Config
! Don't do redirects  
no ip redirects
! Don't run proxy ARP on your ECIX interface
no ip proxy-arp
! Don't run CDP on your ECIX interface
no cdp enable
! Directed broadcasts are evil.
no ip directed-broadcast
! Disable the DEC drek if you haven't done so globally yet.
no mop enable
! For (Fast)Ethernet: no auto-negotiation on your connection.
! no negotiation auto
! duplex half
duplex full
! L2 keepalives are useless on the ECIX
no keepalive

Layer 2 Config

It is difficult to give a complete guide for Cisco products, owing to the many different types of devices and (IOS) software versions. When in doubt, consult your documentation.

29xx and 35xx Series

If you use a Cisco Layer 2 device (such as the 2900 and 3500 series), you have to turn off VTP (VLAN Trunking Protocol), DTP (Dynamic Trunking Protocol), LLDP, and UDLD.

In global config mode:

vtp mode transparent
!
no spanning-tree vlan 9033
! If you don't need LLDP, disable globally
no lldp run
! If you don't need CDP, disable globally
no cdp run
!
vlan 9033
 name ECIX
!
interface /IfIdent/
 description Interface to ECIX
 switchport access vlan 9033
 switchport mode access
 switchport nonegotiate
 no keepalive
 speed nonegotiate
 no udld enable
 ! If CDP has not been disabled globally:
 no cdp enable
 ! If LLDP has not been disabled globally:
 no lldp receive
 no lldp transmit
 ! If you do not want to shut off STP:
 spanning-tree bpdufilter enable
end

7600 Series

Members should be advised not to run 12.2(33)SRC on their Cisco 7600's with a sup720. This software release does not always send or forward replies to solicit requests; even when it's acting as a pure layer2 switch between a member router and the ECIX fabric.

To render a cisco 7600 switch 'silent' the following configuration seems to work:

no service dhcp
no ip bootp server
vtp mode transparent
spanning-tree mode pvst
spanning-tree extend system-id
no spanning-tree vlan XX
!
vlan XX
 name ECIX 
 exit
!
interface GigabitEthernet6/0/0
 description to-ECIX switchport
 switchport access vlan XX
 switchport mode access
 switchport nonegotiate
 no mls qos trust
 no cdp enable
 spanning-tree bpdufilter enable
 exit
!

Vlan XX was also removed from the allow list on all dot1q trunk ports not related to the setup, in this case every dot1q trunk port in the chassis.

Catalyst 6500 Series

CatOS and IOS are different beasts, so for Catalyst switches, the following applies:

set vtp mode off
set port name /IfIdent/ My ECIX Port
set cdp disable /IfIdent/
set udld disable /IfIdent/
set trunk /IfIdent/ off dot1q
set spantree bpdu-filter /IfIdent/ enable
set vlan 9033 name My_ECIX_Vlan
set vlan 9033 /IfIdent

If, for some reason, you cannot afford to turn off VTP globally, the only way to turn it off on individual ports seems to be by using l2pt

set port l2protocol-tunnel /IfIdent/ vtp enable

Depending on your CatOS platform, you may or may not be able to do this.

CRS (IOS-XR)

CDP, Proxy ARP, Directed Broadcast, Link Auto Negotiation, and ICMP redirects are disabled by default in IOS-XR.
ICMP redirect messages are disabled by default on the interface unless the Hot Standby Router Protocol (HSRP) is configured.

Other Devices

For other devices, some or all of the above may apply. Check your documentation for details.

Cisco Aggregated Links

Catalyst 6500 Series

Configure the port-channel as active. Please do not not configure any forms of negotiate or desirable as the ECIX switches do not speak PAgP.

Load-balancing over four ports may result in an unequal distribution due to bug CSCsg80948.

Here is an example configuration:

interface GigabitEthernet1/1
 description ECIX Link 1
 no ip address
 no ip redirects
 no ip proxy-arp
 no keepalive
 no cdp enable
 channel-group 1 mode active
!
interface GigabitEthernet1/2
 description ECIX Link 2
 no ip address
 no ip redirects
 no ip proxy-arp
 no keepalive
 no cdp enable
 channel-group 1 mode active
!
interface Port-channel1
 description ECIX aggregated link
 ip address 194.146.118.x 255.255.255.0
 no ip redirects
 no ip proxy-arp
 no keepalive
!

Here are examples of LACP configurations:

Cisco IOS 65xx/76xx:
interface GigabitEthernet1/1 
 description ECIX Link 1 
 channel-group 10 mode active
!  (12.2(18)SXF2  or  (12.2(33)SRC) upwards) 

 lacp rate fast   
! 
interface GigabitEthernet1/2 
 description ECIX Link 2 
 channel-group 10 mode active
! 
interface Port-channel10
 description ECIX aggregated link 
 no switchport 
 ip address 194.146.118.x 255.255.255.0
! 
Cisco IOS-XR:
interface Bundle-Ether 10
 description ECIX aggregated link 
 ipv4 address 194.146.118.x 255.255.255.0
!
interface GigabitEthernet 1/0/0/0
 description ECIX Link 1
 bundle-id 10 mode active
!  (3.2 upwards) 

 lacp period short 
!
interface GigabitEthernet 1/0/1/0
 description ECIX Link 2
 bundle-id 10 mode active
!

(don't forget to commit)

Cisco NX-OS:
feature lacp
!
interface ethernet 2/1
 description ECIX Link 1 
 channel-group 10  mode active
 lacp rate fast
!
interface ethernet 2/2
 description ECIX Link 2
 channel-group 10 mode active
!
interface port-channel 10
 description ECIX aggregated link 
 ip address 194.146.118.x 255.255.255.0
!  

GSR Series

Do not set a static MAC address on the Port-channel interface. This causes CEF inconsistencies and other assorted failures.
Link aggregation and IPv6 do not seem to play well together. Cisco advises against trying this.

Some changes will result in a different MAC address getting chosen for the aggregated link (likely such as reloading a linecard, if it contains the first port in the bundle). This will keep your ports dysfunctional due to port security on the ECIX switches and you will have to contact the ECIX in such cases to fix this.

Some restrictions apply to what features are supported on link bundles (e.g. sampled NetFlow only on ISE/Engine4+; no uRPF). Also not all line cards support link bundling, and if traffic towards ECIX comes in on such an interface you will experience suboptimal load-balancing. Please see the Cisco documentation for more details.

Support for link bundling on Engine 5 linecards will come in 12.0(33)S.
Cisco Engineering have a special train called "Phase 3" (lb-eft-ph3) that is purported to also provide functionality such as MAC address accounting for Port-Channel interfaces. This seems to have been integrated into 12.0(32)S, but IPv6 does not seem to be supported yet.

Below follows a list of Cisco Bug IDs (ddts) related to link aggregation that you need to consider when choosing an appropriate IOS image

CSCee27396

present in 12.0(26)S1; fixed in 12.0(26)S3, 12.0(27)S2, 12.0(28)S1, 12.0(30)S
Symptoms: Over 90% CPU usage by CEF Scanner on all linecards and %TFIB-7-SCANSABORTED errors occur when configuring a link bundle. Also, the router sends traffic to MAC addresses taken from its ARP table seemingly at random, instead of to the appropriate next-hop's MAC address.

CSCef12828

present in post-CSCee27396; fixed in 12.0(26)S4, 12.0(27)S3, 12.0(28)S1, 12.0(30)S
Symptoms: When traffic passes through a router, the router blocks traffic for certain prefixes behind a port-channel link.

CSCdz33664

present in 12.0(25)S3, 12.0(26)S1, 12.0(27)S2, 12.0(28)S; fixed in 12.0(25)S4
Symptoms: An HSRP state change on any Engine2 interface causes a microcode bundle flap on all other Engine2 linecards, preventing load balancing to work due to vanilla microcode getting loaded.

CSCee81071

present in 12.0(26)S3, 12.0(27)S2, 12.0(29)S
Symptoms: Router sends Ethernet frames with a source MAC address of beef.f00d.beef and destination MAC address f00d.beef.f00d (which is the pattern scribbled in unallocated memory in linecards), with what looks to be a legitimate payload of transit traffic. This is one of the symptoms of CSCee27396

CSCeb38014

present in 12.0(26)S5; fixed in 12.0(26)S5, 12.0(27)S
Symptoms: The BGP Router process flushes the BGP tables for each peer when you change one neighbor's description. This pegs the GRP CPU at 99% for quite a while.

CSCeg31951

present in 12.0(31)S; fixed in 12.0(31)S2 (CSCei53226) IOS (at least in the PRP code) places each individual public peer in its own update-group if remove-private-as is configured on a peer. Needless to say, this scales badly for a router connected to an Internet exchange. (Try "show ip bgp replication".) A collection of hearsay follows for recent IOS images for the GSR PRP regarding link aggregation. ECIX does not run any GSRs. Please take this information with appropriately-sized grains of salt.

  • 12.0(24)S2 is not advisable (not many specifics known but they include CSCef89562 and CSCee33045)
  • 12.0(24)S6 boots but load-balancing is completely off 12.0(25)S until S3 have CSCdz33664
  • 12.0(26)S until S4 have CSCef89562, where Engine4+ linecards can have continuously flapping interfaces, but is also somewhat required for Quadra linecards
  • 12.0(26)S3 has CSCee27396 integrated but not CSCef12828, which leads to traffic blackholing 12.0(27)S until S3 have CSCef89562 as well
  • 12.0.(27)S1 has a problem where it sends traffic to random destinations 12.0(27)S2 has CSCee27396 integrated but not CSCef12828
  • 12.0(27)S4 reportedly works reasonably well on PRP2s
  • 12.0(28)S1 has problems with Engine2 linecards (CSCef78098) and Engine4+ (CSCef89562)
  • 12.0(28)S2 reportedly works better but still sometimes emits beef.f00d.beef frames on normal ports with only an IPv6 address configured
  • 12.0(30)S has only been observed to exhibit CSCef12828-like symptoms in conjunction with broken hardware, and also to still sometimes emit frames from MAC beef.f00d.beef.

Routers occasionally still send out frames with beef.f00d.beef as MAC source address on interfaces with an IPv6 but no IPv4 address configured, even on regular links.
Due to the massive amount of feature requests there will be both a 12.0(32)S and a new 12.0(32)SY train.
You can check for incorrect next-hops by attaching to the linecard and executing show controllers rewrite and show adjacency internal and comparing the two rewrite strings for a certain peer's IPv4 address (suffix the commands with | begin 80.249.20a.b). The first six bytes of the returned long hex string should be the peer's MAC address, and equal for all three occurrences.

An example configuration follows:

!
interface Port-channel1
 description ECIX Aggregated Link
 ip address 194.146.118.x 255.255.255.0
 no ip redirects
 no ip directed-broadcast
 no ip proxy-arp
 channel-group minimum active 1
 no channel-group bandwidth control-propagation
 hold-queue 150 in
!
interface GigabitEthernet1/2/1
 no keepalive
 no negotiation auto
 channel-group 1
 no cdp enable
!
interface GigabitEthernet1/2/2
 no keepalive
 no negotiation auto
 channel-group 1
 no cdp enable
!

Specifying a value is optional, but setting it to the amount of ports in an aggregated link multiplied by 75 is advised.
Show interfaces Port-channel 1 will display keepalives enabled even though they are not; also, the BIA (burnt-in address, shown as 0000.0000.0000) can be ignored.
Please contact the ECIX NOC if you disable autonegotiation on Gigabit Ethernet ports as we may have to explicitly configure our switch for this.

CRS (IOS-XR)

interface Bundle-Ether1
 description Aggregated interface to ECIX Peering LAN
 ipv4 address 194.146.118.x 255.255.255.0
 ipv6 nd suppress-ra
 ipv6 address 2001:07F8:8::A50a:bcde:1/64
 ipv6 enable
 bundle minimum-active links 1
!
interface TenGigE0/0/0/0
 description interface to ECIX Peering LAN #1
 bundle id 1 mode on
!
interface TenGigE0/0/0/1
 description interface to ECIX Peering LAN #2
 bundle id 1 mode on
!

IPv6 Config

Responses on a ICMPv6 multicast listener queries result in bursts of ICMPv6 multicast listener reports. To prevent this configure no ipv6 mld router in interface context. Some other per-interface commands we recommend on a Cisco device:

! disable ICMPv6 multicast listener reports
no ipv6 mld router

! disable IPv6 multicast forwarding
no ipv6 mfib forwarding

! v6 ND-RA is unnecessary and undesired
ipv6 nd suppress-ra
! on IOS version 12.2(33)SRC it is the following syntax:
ipv6 nd ra suppress

! on even more later IOS/IOS-XE versions the "all" option is needed to also 
! suppress responses to Router Solicitation messages besides periodic RAs:
ipv6 nd ra supress all

! disable PIM on a specified interface
no ipv6 pim

! disable MLD snooping on hybrid devices and intermediate layer-2 devices
no ipv6 mld snooping

MTU Config

On newer Cisco IOS/IOS-XR versions, the interface IP MTU is automatically set, based on the presence or absence of 802.1q tags. For more details, please consult this document.