Crossbow Experiements and Elation

Posted on December 31, 2008

I wanted to play a little deeper with Crossbow, and in particular get my mind around Etherstubs and inter-stub routing. So I devised the following experimental architecture:

        |----> vnic0    ---> zone001
        |----> vnic1    ---> zone002
        +----> vnic2  --
Etherstub1               +-> router01
        |----> vnic3  --/
        |----> vnic4    ---> zone003
        +----> vnic5    ---> zone004

The idea is to have 2 zones on one etherstub (virtual switch) on one subnet, 2 on another, and then an additional zone that sits on both acting as a router.

So I set forth to do this. Create a template zone, cloned it out and brought them all up. I created all the vnic’s assigned to the appropriate etherstubs and gave them to the zones as exclusive-ip interfaces and then configured each zones networking stack by plumbing and ifconfig’ing.

root@quadra ~$ dladm create-etherstub etherstub0
root@quadra ~$ dladm create-etherstub etherstub1
root@quadra ~$
root@quadra ~$ dladm create-vnic -l etherstub0 vnic0
root@quadra ~$ dladm create-vnic -l etherstub0 vnic1
root@quadra ~$ dladm create-vnic -l etherstub0 vnic2
root@quadra ~$ dladm create-vnic -l etherstub1 vnic3
root@quadra ~$ dladm create-vnic -l etherstub1 vnic4
root@quadra ~$ dladm create-vnic -l etherstub1 vnic5
root@quadra ~$
root@quadra ~$ dladm show-link
e1000g1     phys     1500   up       -- 
e1000g2     phys     1500   down     -- 
e1000g0     phys     1500   unknown  --
etherstub0  etherstub 9000  unknown  --
etherstub1  etherstub 9000  unknown  --
vnic0       vnic     9000   up       etherstub0
vnic1       vnic     9000   up       etherstub0
vnic2       vnic     9000   up       etherstub0
vnic3       vnic     9000   up       etherstub1
vnic4       vnic     9000   up       etherstub1
vnic5       vnic     9000   up       etherstub1

Here is the zone configuration:

zonecfg:template0> info
zonename: template0
zonepath: /quadra/zones/template0
brand: native
autoboot: false
ip-type: exclusive
        dir: /lib
        dir: /platform
        dir: /sbin
        dir: /usr
        dir: /opt
        address not specified
        physical: vnic0
        defrouter not specified

I then decided on the following IP scheme:

vnic0      /24

Zones up, and it looks like this:

root@quadra ~$ zoneadm list -vc
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              native   shared
   3 zone001          running    /quadra/zones/zone001          native   excl
   4 zone002          running    /quadra/zones/zone002          native   excl
   5 zone003          running    /quadra/zones/zone003          native   excl
   6 zone004          running    /quadra/zones/zone004          native   excl
   7 router01         running    /quadra/zones/router01         native   excl
   - template0        installed  /quadra/zones/template0        native   excl

Now we play!

First things first… can I touch an interface other than the one explicit assigned to it? And, do dladm commands work in a zone?

root@zone001 ~$ dladm show-vnic
root@zone001 ~$ dladm show-vnic vnic0
dladm: invalid vnic name 'vnic0': object not found
root@zone001 ~$ dladm show-vnic vnic1
dladm: invalid vnic name 'vnic1': object not found
root@zone001 ~$ dladm show-vnic vnic2
dladm: invalid vnic name 'vnic2': object not found
root@zone001 ~$ dladm show-ether
root@zone001 ~$ dladm show-usage
dladm: show-usage requires a file
root@zone001 ~$ dladm create-etherstub zonestub0
dladm: etherstub creation failed: object not found

root@template0 ~$ ifconfig vnic2 plumb 
ifconfig: cannot open link "vnic2": DLPI link does not exist
root@template0 ~$ ifconfig vnic1 plumb

Ok, so dladm is useless and I can’t plumb an interface not assigned. Good.

Now, to setup our router. All we should have to do is enable IPv4 Forwarding on a zone with 2 interfaces, one on each network:

root@router01 ~$ routeadm -e ipv4-forwarding
root@router01 ~$ routeadm -u
root@router01 ~$ routeadm
              Configuration   Current              Current
                     Option   Configuration        System State
               IPv4 routing   enabled              enabled
               IPv6 routing   disabled             disabled
            IPv4 forwarding   enabled              enabled
            IPv6 forwarding   disabled             disabled

           Routing services   "route:default ripng:default"

Routing daemons:

                      STATE   FMRI
                   disabled   svc:/network/routing/legacy-routing:ipv4
                   disabled   svc:/network/routing/legacy-routing:ipv6
                   disabled   svc:/network/routing/ripng:default
                   disabled   svc:/network/routing/ripng:quagga
                     online   svc:/network/routing/ndp:default
                   disabled   svc:/network/routing/zebra:quagga
                   disabled   svc:/network/routing/rip:quagga
                   disabled   svc:/network/routing/ospf:quagga
                   disabled   svc:/network/routing/ospf6:quagga
                   disabled   svc:/network/routing/bgp:quagga
                     online   svc:/network/routing/route:default
                   disabled   svc:/network/routing/rdisc:default

root@router01 ~$ ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet netmask ff000000
vnic2: flags=201100843 mtu 9000 index 2
        inet netmask ffffff00 broadcast
        ether 2:8:20:27:5f:6
vnic3: flags=201100843 mtu 9000 index 3
        inet netmask ffffff00 broadcast
        ether 2:8:20:e9:65:94
lo0: flags=2002000849 mtu 8252 index 1
        inet6 ::1/128

Easy enough. In the old days you would enable the “ROUTER” flag on each interface and such, but now its all nicely wrapped by routeadm. Yeah!

I won’t bore you with the ping scenario details, but thanks to in.routed running in each zone by default the gateway just appeared auto-magically:

root@zone004 ~$ netstat -nr

Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface 
-------------------- -------------------- ----- ----- ---------- --------- 
default               UG        1          0 vnic5             U         1          1 vnic5              UH        1          0 lo0       

Routing Table: IPv6
  Destination/Mask            Gateway                   Flags Ref   Use    If   
--------------------------- --------------------------- ----- --- ------- ----- 
::1                         ::1                         UH      1       0 lo0   
root@zone004 ~$ ping -s
PING 56 data bytes
64 bytes from icmp_seq=0. time=0.549 ms
64 bytes from icmp_seq=1. time=0.091 ms
---- PING Statistics----
2 packets transmitted, 2 packets received, 0% packet loss
round-trip (ms)  min/avg/max/stddev = 0.091/0.320/0.549/0.324
root@zone004 ~$ traceroute
traceroute to (, 30 hops max, 40 byte packets
 1 (  0.087 ms  0.041 ms  0.034 ms
 2 (  0.086 ms  0.056 ms  0.052 ms

How kool is that! I could take this further by adding in a public interface to the router and routing it as well, but I’d need to bring IP NAT into the mix and I’m not terribly interesting in that tonight.

Of course, one other test of interest is will snoop work properly? We know it works with IP Instances, but still work fine with vnic’s and etherstubs? Yes!

root@zone001 ~$ snoop
Using device vnic0 (promiscuous mode) -> zone001      ICMP Echo request (ID: 496 Sequence number: 5)
     zone001 ->   ICMP Echo reply (ID: 496 Sequence number: 5) -> zone001      ICMP Echo request (ID: 496 Sequence number: 6)
     zone001 ->   ICMP Echo reply (ID: 496 Sequence number: 6)

Furthermore, Etherstub does act as a switch. Other zones on the same etherstub will not see traffic unless its addressed to it.

As a sidenote, you’ll notice that Etherstub’s default to JumboFrame. You should be able to modify this, however the link-property shows as read-only… I’ll look into that later.

Ever wanted to roll out a functioning, routing, VLAN’ed, multicast network of hundreds of nodes to test your dream setup but only have a laptop? Now you can. All my test zones are consuming only 12MB of disk each, and I’ve got 300GB free on my home SATA RAIDZ2… so do that math. 🙂

BTW…. I did all this from architect to implementation and fully tested in 1 hour, including the time it took to install and configure all the zones. Solaris rules.

Can’t resist… lets try IP Filter within the Zone just to see that its happy. I’ll use a simple ruleset that blocks everything but SSH:

root@zone001 ~$ cat /etc/ipf/ipf.conf 
# ipf.conf
pass in quick proto tcp from any to port = 22
block in log from any to

root@zone001 ~$ svcadm enable ipfilter

Now we’ll test from another node:

root@zone004 ~$ ssh
The authenticity of host ' (' can't be established.
RSA key fingerprint is 2e:fc:c7:36:33:70:db:16:d7:74:35:04:1a:3f:02:bb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '' (RSA) to the list of known hosts.

root@zone004 ~$ ftp

Sweet. Now just a look at the IP Filter stats to make sure its not a fluke:

root@zone001 ~$ ipfstat 
bad packets:            in 0    out 0
 IPv6 packets:          in 0 out 0
 input packets:         blocked 5 passed 25 nomatch 8 counted 0 short 0
output packets:         blocked 0 passed 15 nomatch 15 counted 0 short 0
 input packets logged:  blocked 5 passed 0

Perfect! Its actually blocking the packets. IP Filter works as you expect it too, in a zone on a vnic. Super sweet.