IPNET

Automating EVPN VLAN Provisioning with Cisco NSO

8 min read Automation

Key Takeaways

  • Every EVPN service on a dual-homed server is built from the same three blocks: the server attachment (bundle or single interface, plus its Ethernet Segment), the VLAN (subinterface and bridge-domain), and an optional L3 gateway (anycast BVI in a VRF).
  • The server attachment is the same for every VM on that server, so it's effectively configured once per server. Deploying a new VM only adds a VLAN, and optionally a gateway.
  • A Cisco NSO template service builds all three blocks from four inputs: evpn-name, evi-id, server and vlan-id. It pushes them to both PEs in one transaction, so both PEs get the change or neither does.
  • The LACP decision (All-Active bundle or Single-Active interface) comes from the server's record, so the operator never has to choose it per VM.
  • One bridge-domain, one EVI, one BVI. Each VLAN gets its own bridge-domain with its own EVI and BVI. A bridge group can hold as many bridge-domains as you need.
  • NSO talks to the PEs through the cisco-iosxr CLI NED over SSH, so the routers receive ordinary IOS-XR CLI.

Prerequisites

This article builds on EVPN Multi-Homing over SR-MPLS, which explains All-Active and Single-Active multi-homing. You should be comfortable with basic NSO operation: devices onboarded, sync-from done, and a working cisco-iosxr NED.

The Service Flow

The whole service fits on one page. The operator orders an EVPN service with four inputs. NSO checks whether the server is attached by an LACP bundle, builds Block 1 for that case, adds the VLAN in Block 2, and adds the gateway in Block 3 if one was requested. The teal values in the diagram are the variables NSO fills in.

Block 1: Server Attachment

This block connects the server to both PEs. Which version you get depends on how the server is cabled.

LACP bundle (All-Active). Both PEs forward for the server at the same time:

interface Bundle-Ether10
 description srv01
 mtu 9216
 load-interval 30
 lacp system mac 0200.0000.0001
!
interface TenGigE0/0/0/1
 bundle id 10 mode active
!
evpn
 evi 110
  advertise-mac
 !
 interface Bundle-Ether10
  ethernet-segment
   identifier type 0 00.11.22.33.44.55.66.77.88

Single interface (Single-Active). Used when the server can't form a bundle across two PEs. One PE forwards, and the other takes over on failure:

interface TenGigE0/0/0/1
 description srv01
!
evpn
 evi 110
  advertise-mac
 !
 interface TenGigE0/0/0/1
  ethernet-segment
   identifier type 0 00.11.22.33.44.55.66.77.88
   load-balancing-mode single-flow-active
   convergence
    mac-mobility

Three values here must be right, and all three belong to the server, not to the VLAN:

  • The ESI is unique per server and identical on both PEs.
  • The LACP system MAC is identical on both PEs, so the server sees one LACP partner instead of two.
  • The bundle number and member ports are fixed per server.

Every VM on the same server produces exactly the same Block 1. NSO keeps a reference count on configuration shared by several services. The first VM's service creates the bundle, later services just add a reference to it, and it's only removed when the last service using it is deleted.

Block 2: VLAN

This is the part that changes with every VM: a subinterface for the VM's VLAN, placed in its own bridge-domain and EVI.

interface Bundle-Ether10.110 l2transport
 description web01
 encapsulation dot1q 110
 rewrite ingress tag pop 1 symmetric
!
l2vpn
 bridge group EVPN
  bridge-domain bd-110-110
   interface Bundle-Ether10.110
   !
   evi 110

For a Single-Active server, the subinterface is TenGigE0/0/0/1.110 instead, and everything else is identical.

The bridge-domain name bd-<evi>-<vlan> makes every bridge-domain easy to trace back to its service. Each bridge-domain has its own EVI. Don't reuse one EVI across several bridge-domains.

Block 3: L3 Gateway (Optional)

If the VM needs its default gateway on the PEs, the service adds an anycast BVI in the tenant VRF and attaches it to the bridge-domain:

interface BVI110
 host-routing
 vrf TENANT-A
 ipv4 address 10.110.0.1 255.255.255.0
 mac-address 0200.0a0a.0001
!
l2vpn
 bridge group EVPN
  bridge-domain bd-110-110
   routed interface BVI110

The IP and MAC are identical on both PEs. That's what makes it an anycast gateway: the VM's ARP entry stays valid whichever PE its traffic hashes to.

The VRF itself (RD and route-targets) comes from your existing VPN service's shared template. The EVPN service only references it. Each bridge-domain gets its own BVI, and one BVI is never shared between bridge-domains.

By Hand vs NSO

Configured by handNSO service
EffortLog in to PE1, then PE2, type three blocks twiceOne command in NSO
ConsistencyESI, LACP MAC and gateway typed separately on each PEOne template renders both PEs from the same values
Failure halfwayPE1 changed, PE2 notOne transaction: both or neither
Review before changeRe-read your notescommit dry-run outformat native
RemovalFind and delete the lines on both PEsno evpn-service web01

The Service Model (YANG)

The model has two lists. evpn-server describes each server once. evpn-service is the service you order for every VM.

module evpn-service {
  yang-version 1.1;
  namespace "http://ipnet.io/evpn-service";
  prefix evpn;

  import tailf-ncs { prefix ncs; }

  // Described once per server. Nothing is pushed until a service uses it.
  list evpn-server {
    key name;
    leaf name { type string; }
    leaf lacp-bundle { type boolean; default true; }
    leaf esi {
      type string;
      mandatory true;
      description "Unique per server, identical on both PEs.";
    }
    leaf lacp-system-mac {
      when "../lacp-bundle = 'true'";
      type string;
    }
    list pe {
      key device;
      min-elements 2;
      max-elements 2;
      leaf device {
        type leafref { path "/ncs:devices/ncs:device/ncs:name"; }
      }
      leaf bundle-id {
        when "../../lacp-bundle = 'true'";
        type uint16;
      }
      leaf-list member {
        when "../../lacp-bundle = 'true'";
        type string;
        description "TenGigE port ids, e.g. 0/0/0/1.";
      }
      leaf interface {
        when "../../lacp-bundle = 'false'";
        type string;
        description "TenGigE port id, e.g. 0/0/0/1.";
      }
    }
  }

  // Ordered once per VM VLAN.
  list evpn-service {
    key evpn-name;
    unique evi-id;
    uses ncs:service-data;
    ncs:servicepoint evpn-service-servicepoint;

    leaf evpn-name { type string; }
    leaf evi-id {
      type uint32 { range "1..65534"; }
      mandatory true;
    }
    leaf server {
      type leafref { path "/evpn:evpn-server/evpn:name"; }
      mandatory true;
    }
    leaf vlan-id {
      type uint16 { range "2..4094"; }
      mandatory true;
    }
    leaf bridge-group { type string; default "EVPN"; }

    container l3 {
      presence "Add an anycast gateway";
      leaf vrf { type string; mandatory true; }
      leaf gateway { type string; mandatory true; }
      leaf mask { type string; default "255.255.255.0"; }
      leaf anycast-mac { type string; mandatory true; }
    }
  }
}

A few choices that do the real work:

  • unique evi-id: two services can never claim the same EVI, which enforces one EVI per bridge-domain.
  • server is a leafref: you can only order a service for a server that has been described, and the ESI, LACP MAC and ports come from that one record.
  • Exactly two PEs per server: a multi-homed server on one PE is always a mistake, so the model refuses it.

The Template

The template follows the flowchart box by box, and needs no Python. Its outline:

<?foreach {deref(/server)/../pe}?>        one pass per PE of the server
  <?if {../lacp-bundle = 'true'}?>          the LACP decision
     Block 1: Bundle-Ether, members, ethernet-segment
     Block 2: Bundle-Ether<be>.<vlan> in bd-<evi>-<vlan>
  <?else?>
     Block 1: interface, ethernet-segment single-flow-active
     Block 2: <intf>.<vlan> in bd-<evi>-<vlan>
  <?end?>
  <?if {/l3}?>                               optional
     Block 3: BVI<evi>, routed interface in the bridge-domain
  <?end?>
<?end?>

{/vlan-id} and {/evi-id} read the service's inputs. Inside the foreach, {device}, {bundle-id} and {../esi} read the current PE and its server. As an example, here is Block 2 for the bundle case:

<interface xmlns="http://tail-f.com/ned/cisco-ios-xr">
  <Bundle-Ether-subinterface>
    <Bundle-Ether>
      <id>{bundle-id}.{/vlan-id}</id>
      <mode>l2transport</mode>
      <description>{/evpn-name}</description>
      <encapsulation>
        <dot1q><vlan-id>{/vlan-id}</vlan-id></dot1q>
      </encapsulation>
      <rewrite>
        <ingress><tag><pop>1</pop><mode>symmetric</mode></tag></ingress>
      </rewrite>
    </Bundle-Ether>
  </Bundle-Ether-subinterface>
</interface>
<l2vpn xmlns="http://tail-f.com/ned/cisco-ios-xr">
  <bridge>
    <group>
      <group-name>{/bridge-group}</group-name>
      <bridge-domain>
        <bridge-domain-name>bd-{/evi-id}-{/vlan-id}</bridge-domain-name>
        <interface><name>Bundle-Ether{bundle-id}.{/vlan-id}</name></interface>
        <evi><id>{/evi-id}</id></evi>
      </bridge-domain>
    </group>
  </bridge>
</l2vpn>

The easiest way to get the other blocks right for your NED version is to let NSO generate them. Configure one PE by hand inside NSO without committing, run show configuration | display xml-template, and replace the literal values with {...}expressions. Element names can differ slightly between NED releases, so always start from your own output.

Using It

Once per server, describe it. Nothing is sent to the PEs yet:

admin@ncs(config)# evpn-server srv01 esi 00.11.22.33.44.55.66.77.88 lacp-system-mac 0200.0000.0001
admin@ncs(config-evpn-server-srv01)# pe PE1 bundle-id 10 member [ 0/0/0/1 ]
admin@ncs(config-pe-PE1)# exit
admin@ncs(config-evpn-server-srv01)# pe PE2 bundle-id 10 member [ 0/0/0/1 ]
admin@ncs(config-pe-PE2)# commit

For every new VM, order the service:

admin@ncs(config)# evpn-service web01 evi-id 110 server srv01 vlan-id 110
admin@ncs(config-evpn-service-web01)# l3 vrf TENANT-A gateway 10.110.0.1 anycast-mac 0200.0a0a.0001
admin@ncs(config-evpn-service-web01)# commit dry-run outformat native
admin@ncs(config-evpn-service-web01)# commit

commit dry-run outformat native shows the exact IOS-XR commands NSO will send to each PE over SSH, without sending anything. You should see the blocks from the flowchart, identical on PE1 and PE2. Leave out the l3 line for a pure Layer 2 service.

When the VM is removed:

admin@ncs(config)# no evpn-service web01
admin@ncs(config)# commit

This removes that VM's VLAN, bridge-domain, EVI and gateway on both PEs. The bundle stays as long as another service on the same server still uses it.

Verification

In NSO:

admin@ncs# evpn-service web01 check-sync
admin@ncs# evpn-service web01 get-modifications

in-sync true means both PEs still match what the service configured, and get-modifications lists exactly what the service owns on each PE.

On each PE:

show evpn ethernet-segment interface Bundle-Ether10 detail

Both PEs should list each other on the same ESI, in All-Active mode for a bundle or Single-Active for a single interface.

show l2vpn bridge-domain bd-name bd-110-110

The bridge-domain should be up, with the subinterface and, if configured, the BVI attached.

show evpn evi vpn-id 110 detail

The EVI should be up and advertising MAC routes into BGP EVPN.

Troubleshooting

  • commit fails with a validation error: usually a duplicate EVI, a server that hasn't been described, or a VLAN out of range. The model caught it before anything reached the PEs.
  • Traffic only uses one PE on a bundle: compare the ESI and LACP system MAC on both PEs. They must be identical. If they are, check that the server really runs LACP towards both PEs.
  • The PE rejects the BVI or EVI: the same BVI or EVI is being attached to more than one bridge-domain. Give every bridge-domain its own.
  • Tests pass in netsim but fail on a real PE: netsim checks that the configuration fits the NED's model; it doesn't enforce IOS-XR's own rules. Always test on a real IOS-XR device, or on XRd, before production.
  • check-sync reports out-of-sync: someone changed a PE by hand. Run evpn-service web01 re-deploy dry-run to see what NSO would restore, then re-deploy.

EVPN Multi-Homing over SR-MPLS explains All-Active and Single-Active multi-homing and the Ethernet Segment in more depth. Segment Routing (SR-MPLS) Fundamentals covers the transport that carries the EVPN traffic between the PEs.