Cisco's philosophy appears similar to that of some PC OS vendors: enable as many protocols and features as possible by default, so the device works out-of-the-box in most situations. Unfortunately, this means that many unnecessary features are turned on which, while harmless in LAN or corporate environments, may cause undesired traffic on an Internet exchange.
Typical items that need to be disabled are:
- autoconfiguration protocols (DHCP, BOOTP, TFTP config download over the ECIX interface),
- DEC MOP
- IP redirects
- IP directed broadcasts
- proxy ARP-
- IPv6 Router Advertisements
Intermediate switches or hybrid devices will also need to disable VTP, STP, etc.
! Do not run a DHCP server/relay agent no service dhcp ! Older IOS versions require this instead of the above. no ip bootp server ! Do not download configs through TFTP no service config ! Do not run CDP no cdp run 5.2. Interface Config ! Don't do redirects no ip redirects ! Don't run proxy ARP on your ECIX interface no ip proxy-arp ! Don't run CDP on your ECIX interface no cdp enable ! Directed broadcasts are evil. no ip directed-broadcast ! Disable the DEC drek if you haven't done so globally yet. no mop enable ! For (Fast)Ethernet: no auto-negotiation on your connection. ! no negotiation auto ! duplex half duplex full ! L2 keepalives are useless on the ECIX no keepalive
Layer 2 Config
It is difficult to give a complete guide for Cisco products, owing to the many different types of devices and (IOS) software versions. When in doubt, consult your documentation.
29xx and 35xx Series
If you use a Cisco Layer 2 device (such as the 2900 and 3500 series), you have to turn off VTP (VLAN Trunking Protocol), DTP (Dynamic Trunking Protocol), LLDP, and UDLD.
In global config mode:
vtp mode transparent ! no spanning-tree vlan 9033 ! If you don't need LLDP, disable globally no lldp run ! If you don't need CDP, disable globally no cdp run ! vlan 9033 name ECIX ! interface /IfIdent/ description Interface to ECIX switchport access vlan 9033 switchport mode access switchport nonegotiate no keepalive speed nonegotiate no udld enable ! If CDP has not been disabled globally: no cdp enable ! If LLDP has not been disabled globally: no lldp receive no lldp transmit ! If you do not want to shut off STP: spanning-tree bpdufilter enable end
Members should be advised not to run 12.2(33)SRC on their Cisco 7600's with a sup720. This software release does not always send or forward replies to solicit requests; even when it's acting as a pure layer2 switch between a member router and the ECIX fabric.
To render a cisco 7600 switch 'silent' the following configuration seems to work:
no service dhcp no ip bootp server vtp mode transparent spanning-tree mode pvst spanning-tree extend system-id no spanning-tree vlan XX ! vlan XX name ECIX exit ! interface GigabitEthernet6/0/0 description to-ECIX switchport switchport access vlan XX switchport mode access switchport nonegotiate no mls qos trust no cdp enable spanning-tree bpdufilter enable exit !
Vlan XX was also removed from the allow list on all dot1q trunk ports not related to the setup, in this case every dot1q trunk port in the chassis.
Catalyst 6500 Series
CatOS and IOS are different beasts, so for Catalyst switches, the following applies:
set vtp mode off set port name /IfIdent/ My ECIX Port set cdp disable /IfIdent/ set udld disable /IfIdent/ set trunk /IfIdent/ off dot1q set spantree bpdu-filter /IfIdent/ enable set vlan 9033 name My_ECIX_Vlan set vlan 9033 /IfIdent
If, for some reason, you cannot afford to turn off VTP globally, the only way to turn it off on individual ports seems to be by using l2pt
set port l2protocol-tunnel /IfIdent/ vtp enable
Depending on your CatOS platform, you may or may not be able to do this.
CDP, Proxy ARP, Directed Broadcast, Link Auto Negotiation, and ICMP redirects are disabled by default in IOS-XR.
ICMP redirect messages are disabled by default on the interface unless the Hot Standby Router Protocol (HSRP) is configured.
For other devices, some or all of the above may apply. Check your documentation for details.
Cisco Aggregated Links
Catalyst 6500 Series
Configure the port-channel as active. Please do not not configure any forms of negotiate or desirable as the ECIX switches do not speak PAgP.
Load-balancing over four ports may result in an unequal distribution due to bug CSCsg80948.
Here is an example configuration:
interface GigabitEthernet1/1 description ECIX Link 1 no ip address no ip redirects no ip proxy-arp no keepalive no cdp enable channel-group 1 mode active ! interface GigabitEthernet1/2 description ECIX Link 2 no ip address no ip redirects no ip proxy-arp no keepalive no cdp enable channel-group 1 mode active ! interface Port-channel1 description ECIX aggregated link ip address 194.146.118.x 255.255.255.0 no ip redirects no ip proxy-arp no keepalive !
Here are examples of LACP configurations:
Cisco IOS 65xx/76xx:
interface GigabitEthernet1/1 description ECIX Link 1 channel-group 10 mode active ! (12.2(18)SXF2 or (12.2(33)SRC) upwards) lacp rate fast ! interface GigabitEthernet1/2 description ECIX Link 2 channel-group 10 mode active ! interface Port-channel10 description ECIX aggregated link no switchport ip address 194.146.118.x 255.255.255.0 !
interface Bundle-Ether 10 description ECIX aggregated link ipv4 address 194.146.118.x 255.255.255.0 ! interface GigabitEthernet 1/0/0/0 description ECIX Link 1 bundle-id 10 mode active ! (3.2 upwards) lacp period short ! interface GigabitEthernet 1/0/1/0 description ECIX Link 2 bundle-id 10 mode active !
(don't forget to commit)
feature lacp ! interface ethernet 2/1 description ECIX Link 1 channel-group 10 mode active lacp rate fast ! interface ethernet 2/2 description ECIX Link 2 channel-group 10 mode active ! interface port-channel 10 description ECIX aggregated link ip address 194.146.118.x 255.255.255.0 !
Do not set a static MAC address on the Port-channel interface. This causes CEF inconsistencies and other assorted failures.
Link aggregation and IPv6 do not seem to play well together. Cisco advises against trying this.
Some changes will result in a different MAC address getting chosen for the aggregated link (likely such as reloading a linecard, if it contains the first port in the bundle). This will keep your ports dysfunctional due to port security on the ECIX switches and you will have to contact the ECIX in such cases to fix this.
Some restrictions apply to what features are supported on link bundles (e.g. sampled NetFlow only on ISE/Engine4+; no uRPF). Also not all line cards support link bundling, and if traffic towards ECIX comes in on such an interface you will experience suboptimal load-balancing. Please see the Cisco documentation for more details.
Support for link bundling on Engine 5 linecards will come in 12.0(33)S.
Cisco Engineering have a special train called "Phase 3" (lb-eft-ph3) that is purported to also provide functionality such as MAC address accounting for Port-Channel interfaces. This seems to have been integrated into 12.0(32)S, but IPv6 does not seem to be supported yet.
Below follows a list of Cisco Bug IDs (ddts) related to link aggregation that you need to consider when choosing an appropriate IOS image
present in 12.0(26)S1; fixed in 12.0(26)S3, 12.0(27)S2, 12.0(28)S1, 12.0(30)S
Symptoms: Over 90% CPU usage by CEF Scanner on all linecards and %TFIB-7-SCANSABORTED errors occur when configuring a link bundle. Also, the router sends traffic to MAC addresses taken from its ARP table seemingly at random, instead of to the appropriate next-hop's MAC address.
present in post-CSCee27396; fixed in 12.0(26)S4, 12.0(27)S3, 12.0(28)S1, 12.0(30)S
Symptoms: When traffic passes through a router, the router blocks traffic for certain prefixes behind a port-channel link.
present in 12.0(25)S3, 12.0(26)S1, 12.0(27)S2, 12.0(28)S; fixed in 12.0(25)S4
Symptoms: An HSRP state change on any Engine2 interface causes a microcode bundle flap on all other Engine2 linecards, preventing load balancing to work due to vanilla microcode getting loaded.
present in 12.0(26)S3, 12.0(27)S2, 12.0(29)S
Symptoms: Router sends Ethernet frames with a source MAC address of beef.f00d.beef and destination MAC address f00d.beef.f00d (which is the pattern scribbled in unallocated memory in linecards), with what looks to be a legitimate payload of transit traffic. This is one of the symptoms of CSCee27396
present in 12.0(26)S5; fixed in 12.0(26)S5, 12.0(27)S
Symptoms: The BGP Router process flushes the BGP tables for each peer when you change one neighbor's description. This pegs the GRP CPU at 99% for quite a while.
present in 12.0(31)S; fixed in 12.0(31)S2 (CSCei53226) IOS (at least in the PRP code) places each individual public peer in its own update-group if remove-private-as is configured on a peer. Needless to say, this scales badly for a router connected to an Internet exchange. (Try "show ip bgp replication".) A collection of hearsay follows for recent IOS images for the GSR PRP regarding link aggregation. ECIX does not run any GSRs. Please take this information with appropriately-sized grains of salt.
- 12.0(24)S2 is not advisable (not many specifics known but they include CSCef89562 and CSCee33045)
- 12.0(24)S6 boots but load-balancing is completely off 12.0(25)S until S3 have CSCdz33664
- 12.0(26)S until S4 have CSCef89562, where Engine4+ linecards can have continuously flapping interfaces, but is also somewhat required for Quadra linecards
- 12.0(26)S3 has CSCee27396 integrated but not CSCef12828, which leads to traffic blackholing 12.0(27)S until S3 have CSCef89562 as well
- 12.0.(27)S1 has a problem where it sends traffic to random destinations 12.0(27)S2 has CSCee27396 integrated but not CSCef12828
- 12.0(27)S4 reportedly works reasonably well on PRP2s
- 12.0(28)S1 has problems with Engine2 linecards (CSCef78098) and Engine4+ (CSCef89562)
- 12.0(28)S2 reportedly works better but still sometimes emits beef.f00d.beef frames on normal ports with only an IPv6 address configured
- 12.0(30)S has only been observed to exhibit CSCef12828-like symptoms in conjunction with broken hardware, and also to still sometimes emit frames from MAC beef.f00d.beef.
Routers occasionally still send out frames with beef.f00d.beef as MAC source address on interfaces with an IPv6 but no IPv4 address configured, even on regular links.
Due to the massive amount of feature requests there will be both a 12.0(32)S and a new 12.0(32)SY train.
You can check for incorrect next-hops by attaching to the linecard and executing show controllers rewrite and show adjacency internal and comparing the two rewrite strings for a certain peer's IPv4 address (suffix the commands with | begin 80.249.20a.b). The first six bytes of the returned long hex string should be the peer's MAC address, and equal for all three occurrences.
An example configuration follows:
! interface Port-channel1 description ECIX Aggregated Link ip address 194.146.118.x 255.255.255.0 no ip redirects no ip directed-broadcast no ip proxy-arp channel-group minimum active 1 no channel-group bandwidth control-propagation hold-queue 150 in ! interface GigabitEthernet1/2/1 no keepalive no negotiation auto channel-group 1 no cdp enable ! interface GigabitEthernet1/2/2 no keepalive no negotiation auto channel-group 1 no cdp enable !
Specifying a value is optional, but setting it to the amount of ports in an aggregated link multiplied by 75 is advised.
Show interfaces Port-channel 1 will display keepalives enabled even though they are not; also, the BIA (burnt-in address, shown as 0000.0000.0000) can be ignored.
Please contact the ECIX NOC if you disable autonegotiation on Gigabit Ethernet ports as we may have to explicitly configure our switch for this.
interface Bundle-Ether1 description Aggregated interface to ECIX Peering LAN ipv4 address 194.146.118.x 255.255.255.0 ipv6 nd suppress-ra ipv6 address 2001:07F8:8::A50a:bcde:1/64 ipv6 enable bundle minimum-active links 1 ! interface TenGigE0/0/0/0 description interface to ECIX Peering LAN #1 bundle id 1 mode on ! interface TenGigE0/0/0/1 description interface to ECIX Peering LAN #2 bundle id 1 mode on !
Responses on a ICMPv6 multicast listener queries result in bursts of ICMPv6 multicast listener reports. To prevent this configure no ipv6 mld router in interface context. Some other per-interface commands we recommend on a Cisco device:
! disable ICMPv6 multicast listener reports no ipv6 mld router ! disable IPv6 multicast forwarding no ipv6 mfib forwarding ! v6 ND-RA is unnecessary and undesired ipv6 nd suppress-ra ! on IOS version 12.2(33)SRC it is the following syntax: ipv6 nd ra suppress ! on even more later IOS/IOS-XE versions the "all" option is needed to also ! suppress responses to Router Solicitation messages besides periodic RAs: ipv6 nd ra supress all ! disable PIM on a specified interface no ipv6 pim ! disable MLD snooping on hybrid devices and intermediate layer-2 devices no ipv6 mld snooping
On newer Cisco IOS/IOS-XR versions, the interface IP MTU is automatically set, based on the presence or absence of 802.1q tags. For more details, please consult this document.