BGP Flowspec redirect with ExaBGP

I’ve been busy as hell since the summer, not had much time to work on blog posts – but it’s all been good work! I also got a new job working for Riot Games, (Makers of the worlds largest online multiplayer game – league of legends) which has been totally fantastic.

This post is about BGP Flowspec, specifically how we can now more easily redirect traffic to a scrubbing appliance, it’s common for a device such as an Arbor TMS, or some other type of filtering box, to be installed close to the network edge, it could be a linux box full of filters, a DPI box, anything that might be useful in terms of performing traffic verification or enforcement.

In the event that a DDOS event occurs, it’s possible to redirect suspect traffic, or traffic to a specific victim host, through an appliance where it can be dropped or permitted.

Traditionally this has been done with Layer-3 VPNS, where ingress traffic from the internet is punted into a “Dirty VRF” it’s then forced through a mitigation appliance where it’s either dropped, or permitted – where it returns back into the same router but in a new “Clean VRF”

It looks something like this;

• DDOS Traffic from Lizardsquad ingresses through the edge router aimed at the victim 1.2.3.4/32
• A BGP host route of 1.2.3.4/32 is injected into the edge router via the Dirty VRF with a next-hop of the mitigation appliance
• RIB Groups or route leaking is used to punt traffic aimed at 1.2.3.4 from GRT into the Dirty VRF
• Suspect traffic is either dropped or forwarded back into the same edge router via the “Clean VRF”
• It flows towards the destination, where it’s then leaked back into GRT ahead of reaching the final destination

There are many different permutations of this design, but the main flaw with it is the reliance on having to provision a clean VRF everywhere, along with route-leaking.

The only real reason for the existence of the Clean VRF in this scenario, is to prevent forming a routing-loop on the edge router – if traffic is returned back into GRT, or back through the Dirty VRF – it’ll encounter the 1.2.3.4/32 mitigation route, and be looped back into the mitigation appliance for infinity;

It’s always seemed a bit of a waste to me, to have to provision VPNs everywhere simply because we can’t put clean traffic into the same router, without causing a huge routing loop, because unless we resort to horrific things like policy-routing – we’re stuck with regular forwarding logic.

The only other alternative to this is to route clean traffic back into a physically different router that doesn’t maintain a mitigation route, that way the returned clean traffic can just follow the regular path to the destination;

Obviously, the above scenario only really applies if you absolutely can’t run L3VPNs in your network, but still need DDOS mitigation.

Thankfully now that BGP Flowspec is now widely available, we can simplify everything and have a much more streamlined design.

Flowspec is standardised under RFC 5575 https://tools.ietf.org/html/rfc5575

The main principle of Flowspec is actually based on policy routing, in that we can apply a match criteria to ingress traffic, packets that match any of the specific criteria can then be subject to specific actions – in pretty much exactly the same way as with policy-routing, with one major difference – we can program it through BGP rather than through the CLI.

There are some obvious benefits to using BGP for this task;

• Most networks and their operators already run and understand BGP – to turn on an additional AFI/SAFI to support flow routes, is pretty easy
• It’s far easier to automate – programatic networking using APIs to inject routes is far less laborious, than having to do things like policy-routing, or creating gigantic clunky scripts.
• Flowspec supports new extended “redirect” communities that allow a router to automatically forward traffic directly to a different IP next-hop, or directly to a VRF without requiring much configuration
• There are a number of open source BGP daemons that are fully programmable and support Flowspec – such as ExaBGP and GoBGP, they’re also free!

Like with policy-routing and indeed QoS there’s a whole host of specific criteria that we can use to match packets;

• Type 1 – IPv4 / IPv6 Destination prefix
• Type 2 – IPv4 / IPv6 Source Prefix
• Type 3 – IP protocol
• Type 4 – Source / Destination Port
• Type 5 – Destination Port
• Type 6 – Source Port
• Type 7 – ICMP Type
• Type 8 – ICMP Code
• Type 9 – TCP Flags
• Type 10 – Packet Length
• Type 11 – DSCP
• Type 12 – Fragment encoding

Likewise, once we’ve matched our packet – there’s a number of highly useful things that we can do to it using new BGP extended communities;

• Type 0x8006 – Drop, or police; (traffic-rate 0 , or traffic-rate <rate> )
• Type 0x8007 – Traffic action; (apply sampling)
• Type 0x8008 – Redirect to VRF; (punt traffic into a VRF based on route-target)
• Type 0x8009 – Traffic marking; (Set a DSCP value)
• Type 0x0800 – Redirect to IP NH; (creates a policy that forces traffic towards the specified next-hop) currently supported on Cisco and Nokia – but not Juniper 😦

The beauty of Flowspec, is that all of this is done directly in hardware – if you’re using modern silicon there’s practically no hit to performance, even with small packets – you should be able to run Flowspec rules at line rate – but as always, make sure you read the docs, test it AND speak to your vendor, 😉

Furthermore, the configuration is really quite basic – once you’ve enabled the BGP Flowspec AFI/SAFI and have a working session, simply go ahead and inject your mitigation routes – most of the work and config takes place on the controller.

Lets take a quick look at the lab topology;

Cisco / Juniper mix, assume basic ISIS/MPLS internal connectivity with iBGP between loopbacks, all other basic settings at default.

Lets take a look at the basic ExaBGP config;

(ExaBGP can be installed from git; https://github.com/Exa-Networks/exabgp)

Pretty simple stuff;

• Lines 1 through 6 take care of basic BGP neighbour establishment
• Line 7 specifies the AFI/SAFI as “Flow” enabling Flowspec
• Lines 9 and 10 signify the match criteria;
  • Match anything from 30.30.30.100/32
• Lines 12 and 13 attach an extended community to the Flow route advertisement
  • redirect 666:666 corresponds to the “Dirty VRF” route-target on the edge router

Now lets take a look at the relevant config snippets on “edge1” the router which will receive and install the Flow route, from ExaBGP;

• Lines 1 through 11 relate to standard iBGP internal peering
• Lines 11 through 18 relate to the upstream eBGP peering to the upstream router in AS1
• lines 20 through 26 run an internal iBGP session between the edge1 router and the ExaBGP controller;
  • Family inet for “flow” is enabled
  • It references a policy-statement called FSPEC which contains the extended community and the VRF we wish to use for redirecting to the “Dirty VRF”
  • The “No validate” command disables the route-validation procedure if the packets match a specific policy
  • The community “ON-RAMP” is used to match the Flow route coming from ExaBGP, tying it to the policy

The “Dirty VRF” configuration is shown below, essentially it’s just a VRF with the same route-target that ExaBGP is advertising flow routes for, a default route just punts all traffic directly into the mitigation appliance;

There’s also one really important behaviour that should be understood when employing Flowspec, as it differs between vendors.

Remember nearer the start of the post, where I talked about the problem relating to routing-loops, where traffic matching a mitigation route will loop for infinity if it’s routed back into the same device.

This occurs on a Juniper router, because when you enable Flowspec – implicitly applies that flowspec filter to every single interface on the router, so if your packet re-enters the router on any interface, it’ll match the flow-route every time and have the same action applied to it.

To get around this problem, Juniper added the ability to exclude interfaces from Flowspec processing, the config looks like this;

In the above snippet, we’re excluding interface ge-0/0/1 from any form of further Flowspec processing or filtering, this allows return traffic to flow naturally southbound towards Edge2 inside GRT

Note; this is not needed on some other platforms such as a Nokia 7750 – where Flowspec is embedded inside a packet filter, and so Flowspec is only ever applied to whichever interfaces the packet filter is applied to, rather than to every single interface on the router – as is the case with Juniper. Always read the documentation – especially Nokia as they have a tendency to completely change things from one release to the next 😀 

Lets see it in action;

I’m using the Ostinato traffic generator inside Eve-NG to send a small amount of traffic from the external generator in AS1 behind the Cisco router “peering” from the IP address 30.30.30.100, to the internal endpoint in AS65001, behind the Cisco router “Edge2, targeted at 192.168.100.2

Traffic flows normally from north to south;

If we look at the mitigation interface (ge-0/0/0) on Edge1, we can see that nothing is being punted to the mitigation device, traffic is just flowing normally, out of the southbound ge-0/0/2 interface towards Edge2;

So lets go ahead and turn on the Flowspec advertisement, firstly by switching on the ExaBGP process and advertising the Flow route to Edge1;

So we can see some relevant information, such as the connection parameters and the successful connection to Edge1, lets look on Edge1 to see what’s being received;

The flow route received by ExaBGP contains some interesting information;

• Line 15 shows a regex against the prefix of *,30.30.30.100 – this means anything “from” 30.30.30.100, compared to normal destination based routing, if you remember from the ExaBGP config – we’re matching the source of the traffic for inspection.
• Line 26 specifies the Announcement bits as 0-Flow
• Line 28 includes the special Flowspec “redirect community” of 666:666

Flowspec in Juniper uses the firewall filter architecture, it doesn’t add any configuration to the device, instead it uses the BGP advertisement to automatically construct a firewall filter, from the flow route advertisement;

We can see that the firewall filter has been added, it’s matching packets so hopefully those packets should be flowing out of the “Dirty-VRF” towards the mitigation appliance, (remember before, they were flowing straight down from north to south)

We can see that the traffic rate on ge-0/0/0 has gone up to 98pps, meaning we’re sending traffic towards the mitigation appliance. That very same traffic returns clean on ge-0/0/1.

In the case of the lab, the mitigation appliance is just a Cisco CSR with a default route pointing back at the ge-0/0/1 interface on the Juniper, but whether it’s an Arbor TMS or a Linux box full of filters, the principle remains the same.

In many cases, vendor supported DDOS Mitigation appliances such as Arbor SP/TMS have built in support for Flowspec, so you can trigger mitigation flow routes automatically if certain things get detected.

Previously such appliances had no other way of redirecting traffic without advertising hundreds, or thousands of /32 victim host routes, in order to break regular best-path routing, Thanks to Flowspec – we can now specify traffic sources, ports, protocols, you name it.

It’s also pretty easy to rate limit, if we look at the ExaBGP config, we can use the “rate-limit” extended community, to create a packet policer directly in the forwarding plane, all built from a BGP advertisement;

In the above config, I’ve simply removed the redirect community and replaced it with “rate-limit” this instead encodes the rate-limit action into the flow route advertisement, in this case 1000Bps

If we go back to the router and see what’s happening and look at the Flowspec filter;

We can see the Flowspec BGP flow route being received, with the “traffic-rate:0:1000” community being received.

We can also see that the firewall filter now has two entries, one for matching the source and a second for rate-limiting traffic that exceeds the configured speed, but there’s a mismatch – can you see it?

If you look closely at the Firewall filter – it’s converted the rate to “8K” rather than the ExaBGP configured value of 1k.

The reason for this, is that there appears to be a mismatch between the RFC and the Juniper implementation, RFC 5575 specifies that the rate should be specified in Bytes per second, however Juniper convert that value to bps (bits per second) inside their firewall filter;

From the RFC 5575;

The remaining 4 octets carry the rate information in IEEE floating point [IEEE.754.1985] format, units being bytes per second. A traffic-rate of 0 should result on all traffic for the particular flow to be discarded.

The fact that Juniper convert the Value to bps isn’t a problem, it’s just something to be aware of and explains the differences in the show commands.

Hope you found this useful 🙂