Crossbow Experiements and Elation

Posted on December 31, 2008

I wanted to play a little deeper with Crossbow, and in particular get my mind around Etherstubs and inter-stub routing. So I devised the following experimental architecture:

Etherstub0
        |----> vnic0    ---> zone001
        |----> vnic1    ---> zone002
        +----> vnic2  --
Etherstub1               +-> router01
        |----> vnic3  --/
        |----> vnic4    ---> zone003
        +----> vnic5    ---> zone004

The idea is to have 2 zones on one etherstub (virtual switch) on one subnet, 2 on another, and then an additional zone that sits on both acting as a router.

So I set forth to do this. Create a template zone, cloned it out and brought them all up. I created all the vnic’s assigned to the appropriate etherstubs and gave them to the zones as exclusive-ip interfaces and then configured each zones networking stack by plumbing and ifconfig’ing.

root@quadra ~$ dladm create-etherstub etherstub0
root@quadra ~$ dladm create-etherstub etherstub1
root@quadra ~$
root@quadra ~$ dladm create-vnic -l etherstub0 vnic0
root@quadra ~$ dladm create-vnic -l etherstub0 vnic1
root@quadra ~$ dladm create-vnic -l etherstub0 vnic2
root@quadra ~$ dladm create-vnic -l etherstub1 vnic3
root@quadra ~$ dladm create-vnic -l etherstub1 vnic4
root@quadra ~$ dladm create-vnic -l etherstub1 vnic5
root@quadra ~$
root@quadra ~$ dladm show-link
LINK        CLASS    MTU    STATE    OVER
e1000g1     phys     1500   up       -- 
e1000g2     phys     1500   down     -- 
e1000g0     phys     1500   unknown  --
etherstub0  etherstub 9000  unknown  --
etherstub1  etherstub 9000  unknown  --
vnic0       vnic     9000   up       etherstub0
vnic1       vnic     9000   up       etherstub0
vnic2       vnic     9000   up       etherstub0
vnic3       vnic     9000   up       etherstub1
vnic4       vnic     9000   up       etherstub1
vnic5       vnic     9000   up       etherstub1

Here is the zone configuration:

zonecfg:template0> info
zonename: template0
zonepath: /quadra/zones/template0
brand: native
autoboot: false
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
inherit-pkg-dir:
        dir: /lib
inherit-pkg-dir:
        dir: /platform
inherit-pkg-dir:
        dir: /sbin
inherit-pkg-dir:
        dir: /usr
inherit-pkg-dir:
        dir: /opt
net:
        address not specified
        physical: vnic0
        defrouter not specified

I then decided on the following IP scheme:

IPs:
vnic0   10.0.90.10      /24
vnic1   10.0.90.11
vnic2   10.0.90.12
vnic3   10.0.91.12
vnic4   10.0.91.11
vnic5   10.0.91.10

Zones up, and it looks like this:

root@quadra ~$ zoneadm list -vc
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              native   shared
   3 zone001          running    /quadra/zones/zone001          native   excl
   4 zone002          running    /quadra/zones/zone002          native   excl
   5 zone003          running    /quadra/zones/zone003          native   excl
   6 zone004          running    /quadra/zones/zone004          native   excl
   7 router01         running    /quadra/zones/router01         native   excl
   - template0        installed  /quadra/zones/template0        native   excl

Now we play!

First things first… can I touch an interface other than the one explicit assigned to it? And, do dladm commands work in a zone?

root@zone001 ~$ dladm show-vnic
root@zone001 ~$ dladm show-vnic vnic0
dladm: invalid vnic name 'vnic0': object not found
root@zone001 ~$ dladm show-vnic vnic1
dladm: invalid vnic name 'vnic1': object not found
root@zone001 ~$ dladm show-vnic vnic2
dladm: invalid vnic name 'vnic2': object not found
root@zone001 ~$ dladm show-ether
root@zone001 ~$ dladm show-usage
dladm: show-usage requires a file
root@zone001 ~$ dladm create-etherstub zonestub0
dladm: etherstub creation failed: object not found

root@template0 ~$ ifconfig vnic2 plumb 
ifconfig: cannot open link "vnic2": DLPI link does not exist
root@template0 ~$ ifconfig vnic1 plumb

Ok, so dladm is useless and I can’t plumb an interface not assigned. Good.

Now, to setup our router. All we should have to do is enable IPv4 Forwarding on a zone with 2 interfaces, one on each network:

root@router01 ~$ routeadm -e ipv4-forwarding
root@router01 ~$ routeadm -u
root@router01 ~$ routeadm
              Configuration   Current              Current
                     Option   Configuration        System State
---------------------------------------------------------------
               IPv4 routing   enabled              enabled
               IPv6 routing   disabled             disabled
            IPv4 forwarding   enabled              enabled
            IPv6 forwarding   disabled             disabled

           Routing services   "route:default ripng:default"

Routing daemons:

                      STATE   FMRI
                   disabled   svc:/network/routing/legacy-routing:ipv4
                   disabled   svc:/network/routing/legacy-routing:ipv6
                   disabled   svc:/network/routing/ripng:default
                   disabled   svc:/network/routing/ripng:quagga
                     online   svc:/network/routing/ndp:default
                   disabled   svc:/network/routing/zebra:quagga
                   disabled   svc:/network/routing/rip:quagga
                   disabled   svc:/network/routing/ospf:quagga
                   disabled   svc:/network/routing/ospf6:quagga
                   disabled   svc:/network/routing/bgp:quagga
                     online   svc:/network/routing/route:default
                   disabled   svc:/network/routing/rdisc:default

root@router01 ~$ ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vnic2: flags=201100843 mtu 9000 index 2
        inet 10.0.90.12 netmask ffffff00 broadcast 10.0.90.255
        ether 2:8:20:27:5f:6
vnic3: flags=201100843 mtu 9000 index 3
        inet 10.0.91.12 netmask ffffff00 broadcast 10.0.91.255
        ether 2:8:20:e9:65:94
lo0: flags=2002000849 mtu 8252 index 1
        inet6 ::1/128

Easy enough. In the old days you would enable the “ROUTER” flag on each interface and such, but now its all nicely wrapped by routeadm. Yeah!

I won’t bore you with the ping scenario details, but thanks to in.routed running in each zone by default the gateway just appeared auto-magically:

root@zone004 ~$ netstat -nr

Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface 
-------------------- -------------------- ----- ----- ---------- --------- 
default              10.0.91.12           UG        1          0 vnic5     
10.0.91.0            10.0.91.10           U         1          1 vnic5     
127.0.0.1            127.0.0.1            UH        1          0 lo0       

Routing Table: IPv6
  Destination/Mask            Gateway                   Flags Ref   Use    If   
--------------------------- --------------------------- ----- --- ------- ----- 
::1                         ::1                         UH      1       0 lo0   
root@zone004 ~$ ping -s 10.0.90.10
PING 10.0.90.10: 56 data bytes
64 bytes from 10.0.90.10: icmp_seq=0. time=0.549 ms
64 bytes from 10.0.90.10: icmp_seq=1. time=0.091 ms
^C
----10.0.90.10 PING Statistics----
2 packets transmitted, 2 packets received, 0% packet loss
round-trip (ms)  min/avg/max/stddev = 0.091/0.320/0.549/0.324
root@zone004 ~$ traceroute 10.0.90.10
traceroute to 10.0.90.10 (10.0.90.10), 30 hops max, 40 byte packets
 1  10.0.91.12 (10.0.91.12)  0.087 ms  0.041 ms  0.034 ms
 2  10.0.90.10 (10.0.90.10)  0.086 ms  0.056 ms  0.052 ms

How kool is that! I could take this further by adding in a public interface to the router and routing it as well, but I’d need to bring IP NAT into the mix and I’m not terribly interesting in that tonight.

Of course, one other test of interest is will snoop work properly? We know it works with IP Instances, but still work fine with vnic’s and etherstubs? Yes!

root@zone001 ~$ snoop
Using device vnic0 (promiscuous mode)
  10.0.91.10 -> zone001      ICMP Echo request (ID: 496 Sequence number: 5)
     zone001 -> 10.0.91.10   ICMP Echo reply (ID: 496 Sequence number: 5)
  10.0.91.10 -> zone001      ICMP Echo request (ID: 496 Sequence number: 6)
     zone001 -> 10.0.91.10   ICMP Echo reply (ID: 496 Sequence number: 6)

Furthermore, Etherstub does act as a switch. Other zones on the same etherstub will not see traffic unless its addressed to it.

As a sidenote, you’ll notice that Etherstub’s default to JumboFrame. You should be able to modify this, however the link-property shows as read-only… I’ll look into that later.

Ever wanted to roll out a functioning, routing, VLAN’ed, multicast network of hundreds of nodes to test your dream setup but only have a laptop? Now you can. All my test zones are consuming only 12MB of disk each, and I’ve got 300GB free on my home SATA RAIDZ2… so do that math. 🙂

BTW…. I did all this from architect to implementation and fully tested in 1 hour, including the time it took to install and configure all the zones. Solaris rules.

Can’t resist… lets try IP Filter within the Zone just to see that its happy. I’ll use a simple ruleset that blocks everything but SSH:

root@zone001 ~$ cat /etc/ipf/ipf.conf 
#
# ipf.conf
pass in quick proto tcp from any to 10.0.90.10/32 port = 22
block in log from any to 10.0.90.10/32

root@zone001 ~$ svcadm enable ipfilter

Now we’ll test from another node:

root@zone004 ~$ ssh 10.0.90.10
The authenticity of host '10.0.90.10 (10.0.90.10)' can't be established.
RSA key fingerprint is 2e:fc:c7:36:33:70:db:16:d7:74:35:04:1a:3f:02:bb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.90.10' (RSA) to the list of known hosts.
Password: 

root@zone004 ~$ ftp 10.0.90.10
^C

Sweet. Now just a look at the IP Filter stats to make sure its not a fluke:

root@zone001 ~$ ipfstat 
bad packets:            in 0    out 0
 IPv6 packets:          in 0 out 0
 input packets:         blocked 5 passed 25 nomatch 8 counted 0 short 0
output packets:         blocked 0 passed 15 nomatch 15 counted 0 short 0
 input packets logged:  blocked 5 passed 0

Perfect! Its actually blocking the packets. IP Filter works as you expect it too, in a zone on a vnic. Super sweet.