MPLS over GRE with ASR9000

Posted by

I recently have been working with a customer on a solution that utilizes the “Internet” as a backup link for a linear MPLS network. By utilizing this configuration the customer would be able to save thousands of dollars a month on a dedicated 10G circuit.  Yes, I understand the downside to doing this solution but figured it was something I’d at least mock up and make work in the lab.

I  built the following network for testing.  PE1 and PE2 are ASR901 routers acting as MPLS edge routers. P3 and P4 are the ASR9000 routers with a point-to-point non-MPLS connection between them.  This connection is acting as the “Internet” and a GRE tunnel has been configured between their Loopback1 interfaces.

TruVista GRE Lab Diagram Base

P3 is an ASR9010 running IOS-XR 4.3 with Trident cards, P4 is an ASR9001 running IOS-XR 4.3, PE1 is an ASR901 running 15.2(2)SNI, PE2 is also an ASR901 running 15.2(2)SNI. Technically the Trident cards do not support MPLS over GRE functionality but I was able to confirm with Cisco TAC that this configuration would in fact work if I had Typhoon cards.  IOS-XR version 4.3 is required for this functionality as well.  I’ll detail the troubleshooting that took place to figure out the Trident card issue.

Here is the current testing results.  I’ve shutdown the direct connection between PE1 and PE2 forcing all traffic between those routers across the GRE tunnel path: PE1<->P3<->P4<->PE2.

From PE1:

LAB-ASR901-1#ping 10.10.2.2 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.2.2, timeout is 2 seconds:
Packet sent with a source address of 10.10.1.1
.....
Success rate is 0 percent (0/5)

LAB-ASR901-1#ping 10.10.2.2 so Vlan1             
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.2.2, timeout is 2 seconds:
Packet sent with a source address of 10.10.13.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

I’m unable to ping from Loopback0 of PE1 and Loopback0 of PE2. If I source the ping from 10.10.13.1 (Vlan1,G0/0) instead of the 10.10.1.1 (Lo0) on PE1 address it is successful. I believe it is a label issue with the return packets. 10.10.13.1 is directly connected to P3 so P4 will not impose a label across the GRE tunnel.  However, 10.10.1.1 is a loopback on PE1 so PE2->P4->P3->PE1 will have labels on the packets across the GRE. This doesn’t appear to work. However, in the forward path for the ping we have no issues going from PE1->P3->P4->PE2.

From PE2:

LAB-ASR901-2#ping 10.10.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.1.1, timeout is 2 seconds:
Packet sent with a source address of 10.10.2.2
.....
Success rate is 0 percent (0/5)
LAB-ASR901-2#ping 10.10.13.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.13.1, timeout is 2 seconds:
Packet sent with a source address of 10.10.2.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

From PE2 we can source it from the loopback 10.10.2.2 but can’t get to 10.10.1.1 but can get to 10.10.13.1. So this verifies the same issue we see from PE1.  Only the traffic with labels going from P3 to P4 seems to be work, labeled traffic from P4 to P3 appears to be broke. The mpls forwarding tables look correct though.

P3:

RP/0/RSP0/CPU0:LAB-ASR9010-1#show mpls forwarding
Tue Apr 23 18:40:52.567 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes      
Label  Label       or ID              Interface                    Switched   
------ ----------- ------------------ ------------ --------------- ------------
16000  Pop         10.10.10.10/32     Gi0/1/0/39   10.10.13.1      0          
289985 Pop         10.10.1.1/32       Gi0/1/0/39   10.10.13.1      13834      
289986 16004       10.10.2.2/32       ti1          10.10.43.2      0          
289987 Pop         10.10.4.4/32       ti1          10.10.43.2      0          
289989 Pop         10.10.24.0/30      ti1          10.10.43.2      0          
289990 Unlabelled  4.4.4.4/32         Gi0/1/0/38   10.10.34.2      0

P4:

RP/0/RSP0/CPU0:LAB-ASR9001-1#sh mpls forwarding
Wed Apr 24 02:54:37.201 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes      
Label  Label       or ID              Interface                    Switched   
------ ----------- ------------------ ------------ --------------- ------------
16000  Unlabelled  3.3.3.3/32         Gi0/0/1/1    10.10.34.1      0          
16001  289985      10.10.1.1/32       ti1          10.10.43.1      4686       
16002  Pop         10.10.3.3/32       ti1          10.10.43.1      0          
16003  Pop         10.10.13.0/30      ti1          10.10.43.1      2612       
16004  Pop         10.10.2.2/32       Gi0/0/1/0    10.10.24.2      18652      
16005  16000       10.10.10.10/32     ti1          10.10.43.1      0

In these tables traffic from 10.10.1.1 to 10.10.2.2 has a proper LSP. Same from 10.10.2.2 to 10.10.1.1. From a control plane perspective everything looks as it should.

I wanted to dig into this a little more so I turned on explicit-nulls on P4 and P3’s forwarding table now looks like this:

RP/0/RSP0/CPU0:LAB-ASR9010-1#show mpls forwarding
Tue Apr 23 18:47:46.698 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes      
Label  Label       or ID              Interface                    Switched   
------ ----------- ------------------ ------------ --------------- ------------
16000  Pop         10.10.10.10/32     Gi0/1/0/39   10.10.13.1      0          
289985 Pop         10.10.1.1/32       Gi0/1/0/39   10.10.13.1      14578      
289986 16004       10.10.2.2/32       ti1          10.10.43.2      0          
289987 Exp-Null-v4 10.10.4.4/32       ti1          10.10.43.2      0          
289989 Exp-Null-v4 10.10.24.0/30      ti1          10.10.43.2      0          
289990 Unlabelled  4.4.4.4/32         Gi0/1/0/38   10.10.34.2      0

Notice the Exp-Null-v4 label. Pinging from 10.10.3.3 to 10.10.4.4 still functions as it should. I then enabled explicit-null on P3 and P4’s forwarding table now looks like this:

RP/0/RSP0/CPU0:LAB-ASR9001-1#show mpls forwarding
Wed Apr 24 03:02:48.022 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes      
Label  Label       or ID              Interface                    Switched   
------ ----------- ------------------ ------------ --------------- ------------
16000  Unlabelled  3.3.3.3/32         Gi0/0/1/1    10.10.34.1      0          
16001  289985      10.10.1.1/32       ti1          10.10.43.1      4686        
16002  Exp-Null-v4 10.10.3.3/32       ti1          10.10.43.1      0          
16003  Exp-Null-v4 10.10.13.0/30      ti1          10.10.43.1      0          
16004  Pop         10.10.2.2/32       Gi0/0/1/0    10.10.24.2      19856      
16005  16000       10.10.10.10/32     ti1          10.10.43.1      0

Pings between 10.10.3.3 and 10.10.4.4 stop working at this point.  It appears anytime the ASR9001 has to send a packet across the GRE tunnel with MPLS labels it doesn’t work. Its either the ASR9001 swapping labels or the ASR9010 not receiving them correctly that is the issue I believe. IOS-XR config between P3 and P4 is pretty much the same except for IP addresses and such.

This issue was confirmed by Cisco TAC and configs verified as OK

Hello Matt,

I hope you are doing great!

I am sending you this email to update you on this Service Request.

Your configuration is Ok. I made a research on your setup and found that MPLS over GRE is NOT supported on Trident cards. It is only supported on Typhoon cards from 4.3.0 onwards.

So this would be the reason why the labeling/forwarding is broken.

I hope you found this information useful, let me know if there is anything else I can do to assist you.

Have a nice day.

Best regards,

Sergio

Configuration Files:

PE1: https://dl.dropboxusercontent.com/u/38814102/GRE%20Configs/PE1.txt

PE2:https://dl.dropboxusercontent.com/u/38814102/GRE%20Configs/PE2.txt

P3:https://dl.dropboxusercontent.com/u/38814102/GRE%20Configs/P3.txt

P4:https://dl.dropboxusercontent.com/u/38814102/GRE%20Configs/P4.txt

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s