Allowed Traffic & Configurations
The Technical Specifications state the following:
Only three ethertypes are allowed:
- 0x0800 - IPv4
- 0x0806 - ARP
- 0x86dd - IPv6
This implies IEEE 802.3 compliance, not 802.2, so no LLC encapsulation!
Only one MAC address is allowed on a port: all frames sent towards the ECIX should have exactly one unique MAC address
The only non-unicast traffic allowed is:
- Broadcast ARP
- Multicast ICMPv6 Neighbour Discovery (ND) packets.
(NOTE: this does not include Router Advertisement (ND-RA) packets!)
ECIX customer equipment should only reply to ARP queries for IP addresses of their directly connected ECIX interface. In other words, proxy ARP is not allowed.
Traffic for link-local protocols is not allowed, except for ARP and IPv6 ND (see above).
IP packets addressed to ECIX's Peering LAN's directed broadcast address shall not be automatically forwarded to ECIX ports.
The speed and duplex setting of 1000BASE ports should be statically configured, i.e. auto-negotiation must be disabled.
The ECIX platform is designed to carry Ethernet frames with a payload of up to 1500 bytes. MTU settings must be configured accordingly.
Physical L2 Topology
ECIX rules dictate that only one MAC address is allowed behind a port per VLAN. This means that you have to be extremely careful when connecting a device that can act as a L2 device.
Why do we allow only one MAC address? Simple: we don't want additional devices behind the ECIX ports. Extended L2 networks are not under the control of ECIX, but instabilities in a L2 network behind the ECIX switches can, and typically do, negatively impact the entire exchange. Forwarding loops and spanning tree topology changes are good examples of this. By enforcing the one-MAC-address-per-port-per-vlan rule, we effectively prevent forwarding loops and STP traffic from intermediate L2 devices.
In short, an intermediate L2 device may only bridge frames from the member's router to the ECIX port (so we see only one MAC address) and should otherwise be completely invisible. No connected device should bridge frames from other devices onto the ECIX Peering LANs, or speak STP on its ECIX interface.
Connecting a L3 Device
This is your best chance of not leaking MAC addresses or STP traffic and it greatly increases the stability of the network.
Connecting Through a L2 Device
We neither recommend nor encourage connecting your router through a L2 device, but if you do so, keep the following in mind:
- You must make absolutely sure that only traffic to/from your L3 router's interface goes to/from the ECIX port.
- You must make absolutely sure that all legitimate traffic to/from your L3 router's interface goes to/from the ECIX port.
- MLD snooping may block legitimate ICMPv6 neighbour solicitations.
- You must disable spanning tree on your link to ECIX.
On all intermediate L2 devices, consider using explicitly defined port-based VLANs for production ports. It forces you to understand your topology and reduces the chances of a nasty surprise further down the road. In particular, we strongly recommend using a dedicated VLAN for the path from your router to ECIX.
Connecting a L2/L3 Hybrid
The L2/L3 hybrid switch/router requires careful configuration in order to prevent unwanted traffic from leaking onto the exchange. As with intermediate L2 devices, you need to keep the following in mind:
You must make absolutely sure that your ECIX port is configured as a 'router only' port.
You must disable Spanning Tree on your link to ECIX!
On a L2/L3 hybrid device, it is advisable to place the ECIX connected interface (untagged) in a separate (non-default) port-based VLAN without spanning tree and with no other ports in it. This is the best way to ensure that no traffic from other ports will be bridged onto the ECIX port.
Commonly Seen Illegal Traffic and Setup
Any traffic other than the types mentioned in the previous section is deemed to be illegal traffic. In this section we will list some of the more common types of violations we see at ECIX and provide some argumentation as to why it is considered unwanted.
Multiple MAC addresses
Since ECIX operates on the principle of 'one router per port per vlan', there should be only one MAC address visible behind each port in each VLAN. Some members connect through intermediate switches or use a L2/L3 hybrid device. If these devices are not configured properly, they can cause forwarding loops, STP instabilites, and lots of unwanted traffic on the exchange. There is no excuse for these devices to leak traffic, and there is no necessity to talk STP on the link to ECIX. Hence, by enforcing the one-MAC-address rule, we also enforce these issues.
Spanning Tree (STP)
This point is closely related to the previous point. The device(s) connected to the ECIX port are not allowed to be visible as L2 bridges. This means that they should not speak STP (spanning tree) or any other (proprietary) L2 specific protocol.
Routing protocols: EIGRP, OSPF, RIP, IS-IS
The only routing protocol allowed on the ECIX Peering VLANs is BGP. There is no valid reason for interior routing protocols to appear on the shared medium. These protocols only cause unnecessary multicast and broadcast traffic.
By default, Cisco routers and switches periodically test their (Fast) Ethernet links by sending out Loopback frames (ethertype 0x9000) addressed to themselves. (An 'L2 self-ping'). In a switched environment, this can be used to test the functionality of the switch and/or keep the router's MAC address in the switch's address table.
In the ECIX environment, this is not useful since we use MAC timeouts that are larger than the typical BGP and/or ARP timeouts.
Discovery protocols: CDP, EDP, etc.
Various vendors (such as Extreme, Cisco) tend to ship their boxes as gregarious devices: by default they announce their existence from all of their interfaces and try to find family members. CDP (Cisco) and EDP (Extreme) are examples of this, but there are others.
The only reason for running discovery protocols is to support certain types of auto configuration. Auto configuration on an Internet Exchange is a very bad idea. Hence, there is absolutely no reason to run discovery protocols on your ECIX interface. Discovery protocols typically cause unwanted broadcast or multicast traffic.
Non-unicast IPv4: IGMP, DHCP, TFTP
On the ECIX Peering VLANs the only non-unicast traffic that is allowed is the ARP query.
Sometimes we see equipment trying to get a configuration through broadcast TFTP, or configure themselves through DHCP. We will leave it to the reader to consider why this is a bad idea.
Other equipment has IGMP turned on by default (or by accident). The Peering LAN is for unicast IP traffic only, so there is no point in configuring multicast on the ECIX interface.
Since traffic over the ECIX Peering VLANs is exchanged based on BGP routes, there is no reason to answer ARP queries for any other IP address(es) than those that are configured on your ECIX interface.
Unfortunately, some vendors (e.g. Cisco) ship their products with proxy ARP enabled by default.
Proxy ARP is not only sloppy, it can lead to unwanted traffic on your network. Consider that if you have it enabled at ECIX, it is also likely to be enabled at other peering points, allowing parties on both sides to use you as a transit.
Proxy ARP is not allowed.
Non-unicast IPv6: IPv6 ND-RA
IPv6 router advertisements are not allowed: they generate a lot of unnecessary traffic, since IPv6 hosts on the ECIX are not autoconfigured, and besides, you don't want to be the default router for the whole Peering VLAN.
Miscellaneous non-IP: DEC MOP, etc.
Some vendors enable protocols other than IP by default. Cisco, for example ships certain versions of IOS with DEC MOP enabled by default. This non-IP traffic has no place at ECIX.