Tag: EVPN Redundancy

EVPN – Single-active redundancy

In the previous 2 posts I looked at the basics of EVPN including the new BGP based control-plane, later I looked at the integration between the layer-2 and layer-3 worlds within EVPN. However – all the previous examples were shown with basic single site networks with no link or device redundancy, this this post I’m going to look at the first and simplest EVPN redundancy mode.

First – consider the new lab topology:

Capture4

The topology and configuration remains pretty much the same, except that MX-1 and MX-2 each connect back to EX4200-1, for VLAN 100 and VLAN 101, with the same IRB interfaces present on each MX router, essentially a very basic site with 2 PEs for redundancy.

Let’s recap the EVPN configuration on each MX1, I’ve got the exact same configuration loaded on MX-2 and MX-3, the only differences being the interface numbers and a unique RD for each site.

MX-1: 

  1. tim@MX5-1> show configuration routing-instances
  2. EVPN-100 {
  3.     instance-type virtual-switch;
  4.     route-distinguisher 1.1.1.1:100;
  5.     vrf-target target:100:100;
  6.     protocols {
  7.         evpn {
  8.             extended-vlan-list 100-101;
  9.             default-gateway do-not-advertise;
  10.         }
  11.     }
  12.     bridge-domains {
  13.         VL-100 {
  14.             vlan-id 100;
  15.             interface ge-1/1/5.100;
  16.             routing-interface irb.100;
  17.         }
  18.         VL-101 {
  19.             vlan-id 101;
  20.             interface ge-1/1/5.101;
  21.             routing-interface irb.101;
  22.         }
  23.     }
  24. }
  25. VPN-100 {
  26.     instance-type vrf;
  27.     interface irb.100;
  28.     interface irb.101;
  29.     route-distinguisher 100.100.100.1:100;
  30.     vrf-target target:1:100;
  31.     vrf-table-label;
  32. }
  33. tim@MX5-1>

 

 

Essentially, each site is configured exactly the same, except for a unique RD per site, and differences in the interface numbering.

In terms of providing active/standby redundancy at the main site, for layer-2 and layer-3 simultaneously, we would historically use VPLS combined with VRRP on the IRB interfaces to provide connectivity.

However this isn’t a perfect solution, for the following reasons:

  1. Unlike EVPN – VPLS needs unique IPv4 GW/MAC addresses at each site, inside the same VPN, so the only way to do active-standby redundancy is with VRRP.
  2. VRRP designs can become complex, ensuring that everything is tracked and monitored – partial failures can be hard to track and things can get over-complicated.
  3. Traffic tromboning can occur where VRRP is used

Regarding point 3

Imagine a scenario where each PE is providing a layer-3 default gateway for each VLAN on each PE, where MX1 is active for VLAN 100 and MX2 is active for VLAN 101

Capture5

It looks simple enough, but traffic tromboning can occur quite easily – due to the reliance on VRRP, for example if host-1 in VLAN 100 wants to send traffic to host-2 in VLAN 101, connected to the same switch – the following things happen:

  1. The packet hits the VRRP active VLAN 100 IRB interface on MX1
  2. Because VLAN 101 is in standby mode on MX1 – it can’t be switched locally
  3. MX1 forwards the packet towards the MPLS network, because there’s a BGP route coming from MX2 (because it’s VRRP active for VLAN 101)
  4. Rather than being routed locally, the packet has to traverse the MPLS network, in order to route between VLANs:

Capture6

Things like this are a pain, and can be mitigated by design and awareness from the start – but in my opinion these sorts of scenarios are good examples of why EVPN was invented, because VPLS never properly solved the basic problems that we get in day to day designs, for simple bread and butter problems like routing between VLANs you end up having a nightmare.

So how does EVPN do it differently?

First, lets look at the configuration required to convert the lab topology into EVPN active-standby, it’s pretty simple:

MX-1: 

  1. tim@MX5-1# run show configuration interfaces ge-1/1/5
  2. flexible-vlan-tagging;
  3. encapsulation flexible-ethernet-services;
  4. esi {
  5.     00:11:22:33:44:55:66:77:88:99;
  6.     single-active;
  7. }
  8. unit 100 {
  9.     encapsulation vlan-bridge;
  10.     vlan-id 100;
  11. }
  12. unit 101 {
  13.     encapsulation vlan-bridge;
  14.     vlan-id 101;
  15. }
  16. [edit]
  17. tim@MX5-1#

 

MX-2:

  1. tim@MX5-2# run show configuration interfaces ge-1/0/5
  2. flexible-vlan-tagging;
  3. encapsulation flexible-ethernet-services;
  4. esi {
  5.     00:11:22:33:44:55:66:77:88:99;
  6.     single-active;
  7. }
  8. unit 100 {
  9.     encapsulation vlan-bridge;
  10.     vlan-id 100;
  11. }
  12. unit 101 {
  13.     encapsulation vlan-bridge;
  14.     vlan-id 101;
  15. }
  16. [edit]
  17. tim@MX5-2#

 

In basic EVPN where sites are single-homed, the “ESI” (Ethernet segment identifier) remains at zero, however whenever you have single-active multi-homing or active-active multi-homing, the ESI value  must be configured to a non-default value. It’s purpose is to identify an Ethernet segment and as such it identifies the entire “site” or “data-centre” to other PE routers on the network, it’s configured under the physical Ethernet interface and must be the same across the segment, in this case for MX1 and MX2 access-facing interfaces

Secondly, under the ESI configuration the PE interfaces are configured to operate in “single-active” mode, which should be self explanatory to most readers 🙂

How does this alter the EVPN control-plane? lets have a more detailed look at the EVPN instance on MX-1

 

  1. tim@MX5-1> show evpn instance extensive
  2. Instance: EVPN-100
  3.   Route Distinguisher: 1.1.1.1:100
  4.   Per-instance MAC route label: 299776
  5.   MAC database status                Local  Remote
  6.     Total MAC addresses:                 2       2
  7.     Default gateway MAC addresses:       2       0
  8.   Number of local interfaces: 2 (2 up)
  9.     Interface name  ESI                            Mode             Status
  10.     ge-1/1/5.100    00:11:22:33:44:55:66:77:88:99  single-active    Up    
  11.     ge-1/1/5.101    00:11:22:33:44:55:66:77:88:99  single-active    Up    
  12.   Number of IRB interfaces: 2 (2 up)
  13.     Interface name  VLAN ID  Status  L3 context
  14.     irb.100         100      Up      VPN-100
  15.     irb.101         101      Up      VPN-100
  16.   Number of bridge domains: 2
  17.     VLAN ID  Intfs / up    Mode             MAC sync  IM route label
  18.     100          1   1     Extended         Enabled   302080
  19.     101          1   1     Extended         Enabled   301872
  20.   Number of neighbors: 2
  21.     10.10.10.2
  22.       Received routes
  23.         MAC address advertisement:              0
  24.         MAC+IP address advertisement:           0
  25.         Inclusive multicast:                    2
  26.         Ethernet auto-discovery:                1
  27.     10.10.10.3
  28.       Received routes
  29.         MAC address advertisement:              2
  30.         MAC+IP address advertisement:           2
  31.         Inclusive multicast:                    2
  32.         Ethernet auto-discovery:                0
  33.   Number of ethernet segments: 1
  34.     ESI: 00:11:22:33:44:55:66:77:88:99
  35.       Status: Resolved by IFL ge-1/1/5.100
  36.       Local interface: ge-1/1/5.100, Status: Up/Forwarding
  37.       Number of remote PEs connected: 1
  38.         Remote PE        MAC label  Aliasing label  Mode
  39.         10.10.10.2       301008     0               single-active
  40.       Designated forwarder: 10.10.10.1
  41.       Backup forwarder: 10.10.10.2
  42.       Advertised MAC label: 301232
  43.       Advertised aliasing label: 301232
  44.       Advertised split horizon label: 0
  46. Instance: __default_evpn__
  47.   Route Distinguisher: 10.10.10.1:0
  48.   VLAN ID: None
  49.   Per-instance MAC route label: 299808
  50.   MAC database status                Local  Remote
  51.     Total MAC addresses:                 0       0
  52.     Default gateway MAC addresses:       0       0
  53.   Number of local interfaces: 0 (0 up)
  54.   Number of IRB interfaces: 0 (0 up)
  55.   Number of bridge domains: 0
  56.   Number of neighbors: 1
  57.     10.10.10.2
  58.       Received routes
  59.         Ethernet auto-discovery:                0
  60.         Ethernet Segment:                       1
  61.   Number of ethernet segments: 0
  63. tim@MX5-1>

 

 

A couple of things to note:

  • EVPN is running in single-active mode, for ge-1/1/5.100 and ge-1/0/5.101
  • The access-interface (ge-1/1/5) on MX1 is shown to be up/forwarding, making this the active PE
  • MX1 is operating in single-active mode
  • The designated forwarder is MX1 (10.10.10.1)
  • The backup designated forwarder is MX2 (10.10.10.2)

Because MX-1 is the active PE, lets take a look at BGP on MX-3 to see what routes are advertised from the redundant site, to a remote site:

(Note – I currently have 2Mbps of IXIA traffic flowing bi-bidirectionally between each site, in each VLAN)

  1. EVPN-100.evpn.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
  2. + = Active Route, – = Last Active, * = Both
  4. 1:1.1.1.1:100::112233445566778899::0/304
  5.                    *[BGP/170] 04:17:27, localpref 100, from 10.10.10.1
  6.                       AS path: I, validation-state: unverified
  7.                     > to 192.169.100.15 via ge-1/1/0.0, Push 300912
  8. 1:10.10.10.1:0::112233445566778899::FFFF:FFFF/304
  9.                    *[BGP/170] 04:17:27, localpref 100, from 10.10.10.1
  10.                       AS path: I, validation-state: unverified
  11.                     > to 192.169.100.15 via ge-1/1/0.0, Push 300912
  12. 1:10.10.10.2:0::112233445566778899::FFFF:FFFF/304
  13.                    *[BGP/170] 13:50:18, localpref 100, from 10.10.10.2
  14.                       AS path: I, validation-state: unverified
  15.                     > to 192.169.100.15 via ge-1/1/0.0, Push 300848
  16. 2:1.1.1.1:100::100::00:00:2e:18:6d:e1/304
  17.                    *[BGP/170] 04:17:23, localpref 100, from 10.10.10.1
  18.                       AS path: I, validation-state: unverified
  19.                     > to 192.169.100.15 via ge-1/1/0.0, Push 300912
  20. 2:1.1.1.1:100::101::00:00:2e:e6:77:95/304
  21.                    *[BGP/170] 04:17:23, localpref 100, from 10.10.10.1
  22.                       AS path: I, validation-state: unverified
  23.                     > to 192.169.100.15 via ge-1/1/0.0, Push 300912
  24. 2:1.1.1.1:100::100::00:00:2e:18:6d:e1::192.168.100.10/304
  25.                    *[BGP/170] 04:17:23, localpref 100, from 10.10.10.1
  26.                       AS path: I, validation-state: unverified
  27.                     > to 192.169.100.15 via ge-1/1/0.0, Push 300912
  28. 2:1.1.1.1:100::101::00:00:2e:e6:77:95::192.168.101.10/304
  29.                    *[BGP/170] 04:17:23, localpref 100, from 10.10.10.1
  30.                       AS path: I, validation-state: unverified
  31.                     > to 192.169.100.15 via ge-1/1/0.0, Push 300912
  32. 3:1.1.1.1:100::100::10.10.10.1/304
  33.                    *[BGP/170] 04:17:26, localpref 100, from 10.10.10.1
  34.                       AS path: I, validation-state: unverified
  35.                     > to 192.169.100.15 via ge-1/1/0.0, Push 300912
  36. 3:1.1.1.1:100::101::10.10.10.1/304
  37.                    *[BGP/170] 13:50:26, localpref 100, from 10.10.10.1
  38.                       AS path: I, validation-state: unverified
  39.                     > to 192.169.100.15 via ge-1/1/0.0, Push 300912
  40. 3:1.1.1.2:100::100::10.10.10.2/304
  41.                    *[BGP/170] 13:50:18, localpref 100, from 10.10.10.2
  42.                       AS path: I, validation-state: unverified
  43.                     > to 192.169.100.15 via ge-1/1/0.0, Push 300848
  44. 3:1.1.1.2:100::101::10.10.10.2/304
  45.                    *[BGP/170] 13:50:18, localpref 100, from 10.10.10.2
  46.                       AS path: I, validation-state: unverified
  47.                     > to 192.169.100.15 via ge-1/1/0.0, Push 300848
  49. tim@MX5-3>

 

We covered type-2 and type-3 routes in the previous labs, but here we have a new type-1 route being received on MX-3, what’s that all about? lets take a deeper look:

  1. tim@MX5-3> show route protocol bgp table EVPN-100.evpn.0 extensive
  3. EVPN-100.evpn.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
  4. 1:1.1.1.1:100::112233445566778899::0/304 (1 entry, 1 announced)
  5.         *BGP    Preference: 170/-101
  6.                 Route Distinguisher: 1.1.1.1:100
  7.                 Next hop type: Indirect
  8.                 Address: 0x2a7b880
  9.                 Next-hop reference count: 16
  10.                 Source: 10.10.10.1
  11.                 Protocol next hop: 10.10.10.1
  12.                 Indirect next hop: 0x2 no-forward INH Session ID: 0x0
  13.                 State: <Secondary Active Int Ext>
  14.                 Local AS:   100 Peer AS:   100
  15.                 Age: 4:21:25    Metric2: 1
  16.                 Validation State: unverified
  17.                 Task: BGP_100.10.10.10.1+179
  18.                 Announcement bits (1): 0-EVPN-100-evpn
  19.                 AS path: I
  20.                 Communities: target:100:100
  21.                 Import Accepted
  22.                 Route Label: 301232
  23.                 Localpref: 100
  24.                 Router ID: 10.10.10.1
  25.                 Primary Routing Table bgp.evpn.0
  26.                 Indirect next hops: 1
  27.                         Protocol next hop: 10.10.10.1 Metric: 1
  28.                         Indirect next hop: 0x2 no-forward INH Session ID: 0x0
  29.                         Indirect path forwarding next hops: 1
  30.                                 Next hop type: Router
  31.                                 Next hop: 192.169.100.15 via ge-1/1/0.0
  32.                                 Session Id: 0x0
  33.             10.10.10.1/32 Originating RIB: inet.3
  34.               Metric: 1           Node path count: 1
  35.               Forwarding nexthops: 1
  36.                 Nexthop: 192.169.100.15 via ge-1/1/0.0
  38. 1:10.10.10.1:0::112233445566778899::FFFF:FFFF/304 (1 entry, 1 announced)
  39.         *BGP    Preference: 170/-101
  40.                 Route Distinguisher: 10.10.10.1:0
  41.                 Next hop type: Indirect
  42.                 Address: 0x2a7b880
  43.                 Next-hop reference count: 16
  44.                 Source: 10.10.10.1
  45.                 Protocol next hop: 10.10.10.1
  46.                 Indirect next hop: 0x2 no-forward INH Session ID: 0x0
  47.                 State: <Secondary Active Int Ext>
  48.                 Local AS:   100 Peer AS:   100
  49.                 Age: 4:21:25    Metric2: 1
  50.                 Validation State: unverified
  51.                 Task: BGP_100.10.10.10.1+179
  52.                 Announcement bits (1): 0-EVPN-100-evpn
  53.                 AS path: I
  54.                 Communities: target:100:100 esi-label:single-active (label 0)
  55.                 Import Accepted
  56.                 Localpref: 100
  57.                 Router ID: 10.10.10.1
  58.                 Primary Routing Table bgp.evpn.0
  59.                 Indirect next hops: 1
  60.                         Protocol next hop: 10.10.10.1 Metric: 1
  61.                         Indirect next hop: 0x2 no-forward INH Session ID: 0x0
  62.                         Indirect path forwarding next hops: 1
  63.                                 Next hop type: Router
  64.                                 Next hop: 192.169.100.15 via ge-1/1/0.0
  65.                                 Session Id: 0x0
  66.             10.10.10.1/32 Originating RIB: inet.3
  67.               Metric: 1           Node path count: 1
  68.               Forwarding nexthops: 1
  69.                 Nexthop: 192.169.100.15 via ge-1/1/0.0
  71. 1:10.10.10.2:0::112233445566778899::FFFF:FFFF/304 (1 entry, 1 announced)
  72.         *BGP    Preference: 170/-101
  73.                 Route Distinguisher: 10.10.10.2:0
  74.                 Next hop type: Indirect
  75.                 Address: 0x2a7ae54
  76.                 Next-hop reference count: 6
  77.                 Source: 10.10.10.2
  78.                 Protocol next hop: 10.10.10.2
  79.                 Indirect next hop: 0x2 no-forward INH Session ID: 0x0
  80.                 State: <Secondary Active Int Ext>
  81.                 Local AS:   100 Peer AS:   100
  82.                 Age: 13:54:16   Metric2: 1
  83.                 Validation State: unverified
  84.                 Task: BGP_100.10.10.10.2+179
  85.                 Announcement bits (1): 0-EVPN-100-evpn
  86.                 AS path: I
  87.                 Communities: target:100:100 esi-label:single-active (label 0)
  88.                 Import Accepted
  89.                 Localpref: 100
  90.                 Router ID: 10.10.10.2
  91.                 Primary Routing Table bgp.evpn.0
  92.                 Indirect next hops: 1
  93.                         Protocol next hop: 10.10.10.2 Metric: 1
  94.                         Indirect next hop: 0x2 no-forward INH Session ID: 0x0
  95.                         Indirect path forwarding next hops: 1
  96.                                 Next hop type: Router
  97.                                 Next hop: 192.169.100.15 via ge-1/1/0.0
  98.                                 Session Id: 0x0
  99.             10.10.10.2/32 Originating RIB: inet.3
  100.               Metric: 1           Node path count: 1
  101.               Forwarding nexthops: 1
  102.                 Nexthop: 192.169.100.15 via ge-1/1/0.0

 

The Type-1 route is known as an AD or Auto-Discovery route, and it’s broken up into two distinct chunks:

  • A per-EVI AD route (line 4
  • A per-ESI AD route (lines 71 and 87)

The first route (line 4) is known as a per-EVI route, and contains what’s known as the “aliasing label” technically this isn’t required in an active-standby situation, as it exists to ensure that traffic can be forwarded equally where you have multiple PEs in an active-active setup. It solves the problem of traffic polarisation caused by a CE hashing traffic on one egress link only – resulting in that being replicated in the control-plane, so return traffic is also polarised, the aliasing label gets around this simply because a remote PE treats it like a regular MAC/IP route, but more on that in the next blog 🙂

The other two routes (line 71 and 87) are Per-ESI AD routes, and contain the ESI of the site, advertised from PE1 and PE2, you notice that the community is set as “target:100:100 esi-label:single-active” and has a label-value of 0. This is essentially telling MX3 that the ESI is running in single-active mode, if it was running in active-active mode – then a non-zero MPLS label would be present – in order to cater for split horizon and BUM traffic. In this case the setup is single-active and so there will only ever be one route at a time back to site 1.

These routes also speed up convergence, if you’re advertising 1000s of MAC/IP routes and you get a link failure, rather than a PE having to send BGP messages to withdraw all those routes, it can simply withdraw the Ethernet AD routes – which speeds up convergence.

Next lets take a look at what’s going on at the main site, and see what MX1 is advertising to MX2:

 

  1. tim@MX5-1> show route advertising-protocol bgp 10.10.10.2 evpn-esi-value 00:11:22:33:44:55:66:77:88:99 detail
  3. VPN-100.inet.0: 8 destinations, 14 routes (8 active, 0 holddown, 0 hidden)
  5. EVPN-100.evpn.0: 16 destinations, 16 routes (16 active, 0 holddown, 0 hidden)
  6. * 1:1.1.1.1:100::112233445566778899::0/304 (1 entry, 1 announced)
  7.  BGP group iBGP-PEs type Internal
  8.      Route Distinguisher: 1.1.1.1:100
  9.      Route Label: 301232
  10.      Nexthop: Self
  11.      Flags: Nexthop Change
  12.      Localpref: 100
  13.      AS path: [100] I
  14.      Communities: target:100:100
  16. __default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
  18. * 1:10.10.10.1:0::112233445566778899::FFFF:FFFF/304 (1 entry, 1 announced)
  19.  BGP group iBGP-PEs type Internal
  20.      Route Distinguisher: 10.10.10.1:0
  21.      Nexthop: Self
  22.      Flags: Nexthop Change
  23.      Localpref: 100
  24.      AS path: [100] I
  25.      Communities: target:100:100 esi-label:single-active (label 0)
  27. * 4:10.10.10.1:0::112233445566778899:10.10.10.1/304 (1 entry, 1 announced)
  28.  BGP group iBGP-PEs type Internal
  29.      Route Distinguisher: 10.10.10.1:0
  30.      Nexthop: Self
  31.      Flags: Nexthop Change
  32.      Localpref: 100
  33.      AS path: [100] I
  34.      Communities: es-import-target:22-33-44-55-66-77

 

You can see that there’s a new “type-4” route being advertised, this is known as an “Ethernet Segment (ES) route” and is advertised by PE routers which are configured with non-zero ESI values. Essentially, it’s a special extended community (ES-Import-target) that each PE router will import if they both have the same ESI configured, it means that two PE routers remote from one another, know that they’re both connected to the same Ethernet segment, all other PE routers with default, or non-zero ESI values filter these advertisements.

So a quick recap – we’ve looked at the new route types, the control-plane and the configuration, the next step is to see how well it works, first a quick recap of the diagram:

Capture7

I’ve created a flow of IXIA traffic bi-bidirectionally between the top site and the bottom site, if I go to MX-1 and look at the MPLS facing interface, we should see the traffic:

Physical interface: ge-1/1/0, Enabled, Physical link is Up
Interface index: 147, SNMP ifIndex: 525
Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
Flow control: Enabled, Auto-negotiation: Enabled, Remote fault: Online
Pad to minimum frame size: Disabled
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x0
Link flags : None
CoS queues : 8 supported, 8 maximum usable queues
Current address: a8:d0:e5:5b:7c:90, Hardware address: a8:d0:e5:5b:7c:90
Last flapped : 2016-06-10 20:15:19 UTC (5d 19:13 ago)
Input rate : 5599000 bps (500 pps)
Output rate : 5583408 bps (499 pps)

So it’s clear that traffic is being forwarded by MX-1, because I’m sending packets at an exact rate of 1000pps we should be able to measure how quickly fail-over occurs by counting the number of lost packets, for example – at 1000pps, if I lose 50 packets, that yields a fail-over time of 50ms.

First an easy failure – I’ll shut down ge-0/0/0 on EX4200-1, this will put the interface down/down on MX-1 and we’ll measure how long it takes to recover:

imtech@ex4200-1# set interfaces ge-0/0/0 disable
{master:0}[edit]
imtech@ex4200-1# commit
configuration check succeeds
commit complete
{master:0}[edit]
imtech@ex4200-1#

Lets look at much traffic was lost:

Fail1

Frames delta = 1077, so just a fraction longer than 1 second to failover, which isn’t THAT bad, we might be able to improve it later..

Lets check the EVPN instance to see how things have changed:

on MX1:

  1. im@MX5-1> show evpn instance extensive
  2. Instance: EVPN-100
  3.   Route Distinguisher: 1.1.1.1:100
  4.   Per-instance MAC route label: 299776
  5.   MAC database status                Local  Remote
  6.     Total MAC addresses:                 0       3
  7.     Default gateway MAC addresses:       0       0
  8.   Number of local interfaces: 2 (0 up)
  9.     Interface name  ESI                            Mode             Status
  10.     ge-1/1/5.100    00:11:22:33:44:55:66:77:88:99  single-active    Down  
  11.     ge-1/1/5.101    00:11:22:33:44:55:66:77:88:99  single-active    Down  
  12.   Number of IRB interfaces: 2 (0 up)
  13.     Interface name  VLAN ID  Status  L3 context
  14.   irb.100         100      Down    VPN-100                          
  15.     irb.101         101      Down    VPN-100      
  16.   Number of bridge domains: 2
  17.     VLAN ID  Intfs / up    Mode             MAC sync  IM route label
  18.     100          1   0     Extended         Enabled
  19.     101          1   0     Extended         Enabled
  20.   Number of neighbors: 2
  21.     10.10.10.2
  22.       Received routes
  23.         MAC address advertisement:              1
  24.         MAC+IP address advertisement:           1
  25.         Inclusive multicast:                    2
  26.         Ethernet auto-discovery:                2
  27.     10.10.10.3
  28.       Received routes
  29.         MAC address advertisement:              2
  30.         MAC+IP address advertisement:           2
  31.         Inclusive multicast:                    2
  32.         Ethernet auto-discovery:                0
  33.   Number of ethernet segments: 1
  34.     ESI: 00:11:22:33:44:55:66:77:88:99
  35.       Status: Resolved by NH 1048582
  36.   Local interface: ge-1/1/5.100, Status: Down
  37.       Number of remote PEs connected: 1
  38.         Remote PE        MAC label  Aliasing label  Mode
  39.         10.10.10.2       301008     301008          single-active
  40.       Designated forwarder: 10.10.10.2
  41.       Advertised MAC label: 301232
  42.       Advertised aliasing label: 301232
  43.       Advertised split horizon label: 0
  44. Instance: __default_evpn__
  45.   Route Distinguisher: 10.10.10.1:0
  46.   VLAN ID: None
  47.   Per-instance MAC route label: 299808
  48.   MAC database status                Local  Remote
  49.     Total MAC addresses:                 0       0
  50.     Default gateway MAC addresses:       0       0
  51.   Number of local interfaces: 0 (0 up)
  52.   Number of IRB interfaces: 0 (0 up)
  53.   Number of bridge domains: 0
  54.   Number of neighbors: 1
  55.     10.10.10.2
  56.       Received routes
  57.         Ethernet auto-discovery:                0
  58.         Ethernet Segment:                       1
  59.   Number of ethernet segments: 0
  60. tim@MX5-1>

 

So it’s pretty clear that things have gone down, and MX2 is the new active PE router, lets check it out:

  1. tim@MX5-2> show evpn instance extensive
  2. Instance: EVPN-100
  3.   Route Distinguisher: 1.1.1.2:100
  4.   Per-instance MAC route label: 299776
  5.   MAC database status                Local  Remote
  6.     Total MAC addresses:                 1       2
  7.     Default gateway MAC addresses:       2       0
  8.   Number of local interfaces: 2 (2 up)
  9.     Interface name  ESI                            Mode             Status
  10.     ge-1/0/5.100    00:11:22:33:44:55:66:77:88:99  single-active    Up    
  11.     ge-1/0/5.101    00:11:22:33:44:55:66:77:88:99  single-active    Up    
  12.   Number of IRB interfaces: 2 (2 up)
  13.     Interface name  VLAN ID  Status  L3 context
  14.     irb.100         100      Up      VPN-100                          
  15.     irb.101         101      Up      VPN-100      
  16.   Number of bridge domains: 2
  17.     VLAN ID  Intfs / up    Mode             MAC sync  IM route label
  18.     100          1   1     Extended         Enabled   302272
  19.     101          1   1     Extended         Enabled   302224
  20.   Number of neighbors: 1
  21.     10.10.10.3
  22.       Received routes
  23.         MAC address advertisement:              2
  24.         MAC+IP address advertisement:           2
  25.         Inclusive multicast:                    2
  26.         Ethernet auto-discovery:                0
  27.   Number of ethernet segments: 1
  28.     ESI: 00:11:22:33:44:55:66:77:88:99
  29.       Status: Resolved by IFL ge-1/0/5.100
  30.       Local interface: ge-1/0/5.100, Status: Up/Forwarding
  31.       Designated forwarder: 10.10.10.2
  32.       Advertised MAC label: 301008
  33.       Advertised aliasing label: 301008
  34.       Advertised split horizon label: 0
  36. Instance: __default_evpn__
  37.   Route Distinguisher: 10.10.10.2:0
  38.   VLAN ID: None
  39.   Per-instance MAC route label: 299808
  40.   MAC database status                Local  Remote
  41.     Total MAC addresses:                 0       0
  42.     Default gateway MAC addresses:       0       0
  43.   Number of local interfaces: 0 (0 up)
  44.   Number of IRB interfaces: 0 (0 up)
  45.   Number of bridge domains: 0
  46.   Number of neighbors: 0
  47.   Number of ethernet segments: 0
  49. tim@MX5-2>

 

 

If we look at the MPLS facing interface on MX2, we should see that all traffic is being sent and received via the MPLS network:

tim@MX5-2> show interfaces ge-1/1/0
Physical interface: ge-1/1/0, Enabled, Physical link is Up
Interface index: 147, SNMP ifIndex: 526
Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
Flow control: Enabled, Auto-negotiation: Enabled, Remote fault: Online
Pad to minimum frame size: Disabled
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x0
Link flags : None
CoS queues : 8 supported, 8 maximum usable queues
Current address: a8:d0:e5:5b:75:90, Hardware address: a8:d0:e5:5b:75:90
Last flapped : 2016-06-10 20:08:17 UTC (5d 19:42 ago)
Input rate : 5605824 bps (502 pps)
Output rate : 5584392 bps (501 pps)

 

The solution itself is a lot more elegant than traditional FHRP (First hop routing protocols) such as VRRP or HSRP.

  • Because MX1 and MX2 automatically learn about each other via the MPLS network and the type-4 Ethernet-Segment route, and NOT the LAN (like HSRP) – if there’s any problem with the MPLS side connected to the active router, it transitions to standby and the solution fails over.

If I fail the MPLS interface on the “P” router connected to MX1, we get failover in less than 1 second:

Axians@m10i-1# set interfaces ge-0/0/2 disable
[edit]
Axians@m10i-1# commit
commit complete

Then check the packet loss in IXIA:

Fail2

The solution recovers from the failure in 912ms.

This is pretty great, not least because it works reliably – but most of this functionality is built directly into the protocol, I haven’t had to do any crazy tracking of routes, I haven’t needed to go anywhere near IP SLA or any of that horror that is a massive pain when designing this sort of thing, with EVPN – things are pretty simple and work reliably.

It’s not perfect however, unlike HSRP or VRRP which form an adjacency over a LAN via Multicast, EVPN doesn’t do this – all information about other PEs is sent and received via BGP. If you have a complex LAN environment and a failure leaves the PEs isolated – you don’t get a traditional split-brain scenario like you would with HSRP or VRRP, the solution simply doesn’t fail at all, the basic triggers for failure are that the physical interface goes down, the MPLS side goes down, or the entire PE goes down.

This can easily be demonstrated by breaking the logical interface on EX4200-1 whilst leaving the physical interface up/up:

imtech@ex4200-1# set interfaces ge-0/0/0.0 disable
{master:0}[edit]
imtech@ex4200-1# commit
configuration check succeeds
commit complete

The whole solution breaks, and stays broken forever:

Fail3

So you still need to be careful with the design and the different way in which EVPN operates, incidentally you can use things like Ethernet OAM to get around this problem:

Just for laughs, lets apply a basic Ethernet OAM config to MX1, MX2 and the EX4200:

OAM template (shown just on MX-1):

  1. oam {
  2.     ethernet {
  3.         connectivity-fault-management {
  4.             action-profile bring-down {
  5.                 event {
  6.                     interface-status-tlv down;
  7.                     adjacency-loss;
  8.                 }
  9.                 action {
  10.                     interface-down;
  11.                 }
  12.             }
  13.             maintenance-domain “IEEE level 4” {
  14.                 level 4;
  15.                 maintenance-association PE1 {
  16.                     short-name-format character-string;
  17.                     continuity-check {
  18.                         interval 100ms;
  19.                         interface-status-tlv;
  20.                     }
  21.                     mep 1 {
  22.                         interface ge-1/1/5.100;
  23.                         direction down;
  24.                         auto-discovery;
  25.                         remote-mep 2 {
  26.                             action-profile bring-down;
  27.                         }
  28.                     }
  29.                 }
  30.             }
  31.         }
  32.     }

 

Just for clarity, the OAM configuration ensures that if there’s a problem with connectivity between MX1 – EX4200-1 and MX2 – EX4200-1 but the physical interfaces remain up/up, OAM will detect the connectivity loss, and automatically tear the line-protocol of the interface to the down/down status, and force EVPN to fail-over,

lets repeat the exact same test again, with the OAM configuration applied to the PEs and the switch:

imtech@ex4200-1# set interfaces ge-0/0/0.0 disable
{master:0}[edit]
imtech@ex4200-1# commit
configuration check succeeds
commit complete

and check the packet-loss with IXIA:

Fail4

Not bad! 612 packets lost, equals failure and convergence in 624ms, which is a lot better than the original 1077ms when failing the physical interface, and a hell of a lot better than it being down forever, if the network experiences a non-direct failure, (software/logical fail)

Anyway I hope you’ve found this useful, there’s a few bits I’ve skipped over – but I’ll cover those in more detail when I do all-active redundancy in the next blog 🙂