Crossbow Experiements and Elation
Posted on December 31, 2008
I wanted to play a little deeper with Crossbow, and in particular get my mind around Etherstubs and inter-stub routing. So I devised the following experimental architecture:
Etherstub0
|----> vnic0 ---> zone001
|----> vnic1 ---> zone002
+----> vnic2 --
Etherstub1 +-> router01
|----> vnic3 --/
|----> vnic4 ---> zone003
+----> vnic5 ---> zone004
The idea is to have 2 zones on one etherstub (virtual switch) on one subnet, 2 on another, and then an additional zone that sits on both acting as a router.
So I set forth to do this. Create a template zone, cloned it out and brought them all up. I created all the vnic’s assigned to the appropriate etherstubs and gave them to the zones as exclusive-ip interfaces and then configured each zones networking stack by plumbing and ifconfig’ing.
root@quadra ~$ dladm create-etherstub etherstub0 root@quadra ~$ dladm create-etherstub etherstub1 root@quadra ~$ root@quadra ~$ dladm create-vnic -l etherstub0 vnic0 root@quadra ~$ dladm create-vnic -l etherstub0 vnic1 root@quadra ~$ dladm create-vnic -l etherstub0 vnic2 root@quadra ~$ dladm create-vnic -l etherstub1 vnic3 root@quadra ~$ dladm create-vnic -l etherstub1 vnic4 root@quadra ~$ dladm create-vnic -l etherstub1 vnic5 root@quadra ~$ root@quadra ~$ dladm show-link LINK CLASS MTU STATE OVER e1000g1 phys 1500 up -- e1000g2 phys 1500 down -- e1000g0 phys 1500 unknown -- etherstub0 etherstub 9000 unknown -- etherstub1 etherstub 9000 unknown -- vnic0 vnic 9000 up etherstub0 vnic1 vnic 9000 up etherstub0 vnic2 vnic 9000 up etherstub0 vnic3 vnic 9000 up etherstub1 vnic4 vnic 9000 up etherstub1 vnic5 vnic 9000 up etherstub1
Here is the zone configuration:
zonecfg:template0> info
zonename: template0
zonepath: /quadra/zones/template0
brand: native
autoboot: false
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
inherit-pkg-dir:
dir: /lib
inherit-pkg-dir:
dir: /platform
inherit-pkg-dir:
dir: /sbin
inherit-pkg-dir:
dir: /usr
inherit-pkg-dir:
dir: /opt
net:
address not specified
physical: vnic0
defrouter not specified
I then decided on the following IP scheme:
IPs: vnic0 10.0.90.10 /24 vnic1 10.0.90.11 vnic2 10.0.90.12 vnic3 10.0.91.12 vnic4 10.0.91.11 vnic5 10.0.91.10
Zones up, and it looks like this:
root@quadra ~$ zoneadm list -vc ID NAME STATUS PATH BRAND IP 0 global running / native shared 3 zone001 running /quadra/zones/zone001 native excl 4 zone002 running /quadra/zones/zone002 native excl 5 zone003 running /quadra/zones/zone003 native excl 6 zone004 running /quadra/zones/zone004 native excl 7 router01 running /quadra/zones/router01 native excl - template0 installed /quadra/zones/template0 native excl
Now we play!
First things first… can I touch an interface other than the one explicit assigned to it? And, do dladm commands work in a zone?
root@zone001 ~$ dladm show-vnic root@zone001 ~$ dladm show-vnic vnic0 dladm: invalid vnic name 'vnic0': object not found root@zone001 ~$ dladm show-vnic vnic1 dladm: invalid vnic name 'vnic1': object not found root@zone001 ~$ dladm show-vnic vnic2 dladm: invalid vnic name 'vnic2': object not found root@zone001 ~$ dladm show-ether root@zone001 ~$ dladm show-usage dladm: show-usage requires a file root@zone001 ~$ dladm create-etherstub zonestub0 dladm: etherstub creation failed: object not found root@template0 ~$ ifconfig vnic2 plumb ifconfig: cannot open link "vnic2": DLPI link does not exist root@template0 ~$ ifconfig vnic1 plumb
Ok, so dladm is useless and I can’t plumb an interface not assigned. Good.
Now, to setup our router. All we should have to do is enable IPv4 Forwarding on a zone with 2 interfaces, one on each network:
root@router01 ~$ routeadm -e ipv4-forwarding
root@router01 ~$ routeadm -u
root@router01 ~$ routeadm
Configuration Current Current
Option Configuration System State
---------------------------------------------------------------
IPv4 routing enabled enabled
IPv6 routing disabled disabled
IPv4 forwarding enabled enabled
IPv6 forwarding disabled disabled
Routing services "route:default ripng:default"
Routing daemons:
STATE FMRI
disabled svc:/network/routing/legacy-routing:ipv4
disabled svc:/network/routing/legacy-routing:ipv6
disabled svc:/network/routing/ripng:default
disabled svc:/network/routing/ripng:quagga
online svc:/network/routing/ndp:default
disabled svc:/network/routing/zebra:quagga
disabled svc:/network/routing/rip:quagga
disabled svc:/network/routing/ospf:quagga
disabled svc:/network/routing/ospf6:quagga
disabled svc:/network/routing/bgp:quagga
online svc:/network/routing/route:default
disabled svc:/network/routing/rdisc:default
root@router01 ~$ ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
vnic2: flags=201100843 mtu 9000 index 2
inet 10.0.90.12 netmask ffffff00 broadcast 10.0.90.255
ether 2:8:20:27:5f:6
vnic3: flags=201100843 mtu 9000 index 3
inet 10.0.91.12 netmask ffffff00 broadcast 10.0.91.255
ether 2:8:20:e9:65:94
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
Easy enough. In the old days you would enable the “ROUTER” flag on each interface and such, but now its all nicely wrapped by routeadm. Yeah!
I won’t bore you with the ping scenario details, but thanks to in.routed running in each zone by default the gateway just appeared auto-magically:
root@zone004 ~$ netstat -nr Routing Table: IPv4 Destination Gateway Flags Ref Use Interface -------------------- -------------------- ----- ----- ---------- --------- default 10.0.91.12 UG 1 0 vnic5 10.0.91.0 10.0.91.10 U 1 1 vnic5 127.0.0.1 127.0.0.1 UH 1 0 lo0 Routing Table: IPv6 Destination/Mask Gateway Flags Ref Use If --------------------------- --------------------------- ----- --- ------- ----- ::1 ::1 UH 1 0 lo0 root@zone004 ~$ ping -s 10.0.90.10 PING 10.0.90.10: 56 data bytes 64 bytes from 10.0.90.10: icmp_seq=0. time=0.549 ms 64 bytes from 10.0.90.10: icmp_seq=1. time=0.091 ms ^C ----10.0.90.10 PING Statistics---- 2 packets transmitted, 2 packets received, 0% packet loss round-trip (ms) min/avg/max/stddev = 0.091/0.320/0.549/0.324 root@zone004 ~$ traceroute 10.0.90.10 traceroute to 10.0.90.10 (10.0.90.10), 30 hops max, 40 byte packets 1 10.0.91.12 (10.0.91.12) 0.087 ms 0.041 ms 0.034 ms 2 10.0.90.10 (10.0.90.10) 0.086 ms 0.056 ms 0.052 ms
How kool is that! I could take this further by adding in a public interface to the router and routing it as well, but I’d need to bring IP NAT into the mix and I’m not terribly interesting in that tonight.
Of course, one other test of interest is will snoop work properly? We know it works with IP Instances, but still work fine with vnic’s and etherstubs? Yes!
root@zone001 ~$ snoop
Using device vnic0 (promiscuous mode)
10.0.91.10 -> zone001 ICMP Echo request (ID: 496 Sequence number: 5)
zone001 -> 10.0.91.10 ICMP Echo reply (ID: 496 Sequence number: 5)
10.0.91.10 -> zone001 ICMP Echo request (ID: 496 Sequence number: 6)
zone001 -> 10.0.91.10 ICMP Echo reply (ID: 496 Sequence number: 6)
Furthermore, Etherstub does act as a switch. Other zones on the same etherstub will not see traffic unless its addressed to it.
As a sidenote, you’ll notice that Etherstub’s default to JumboFrame. You should be able to modify this, however the link-property shows as read-only… I’ll look into that later.
Ever wanted to roll out a functioning, routing, VLAN’ed, multicast network of hundreds of nodes to test your dream setup but only have a laptop? Now you can. All my test zones are consuming only 12MB of disk each, and I’ve got 300GB free on my home SATA RAIDZ2… so do that math. 🙂
BTW…. I did all this from architect to implementation and fully tested in 1 hour, including the time it took to install and configure all the zones. Solaris rules.
Can’t resist… lets try IP Filter within the Zone just to see that its happy. I’ll use a simple ruleset that blocks everything but SSH:
root@zone001 ~$ cat /etc/ipf/ipf.conf # # ipf.conf pass in quick proto tcp from any to 10.0.90.10/32 port = 22 block in log from any to 10.0.90.10/32 root@zone001 ~$ svcadm enable ipfilter
Now we’ll test from another node:
root@zone004 ~$ ssh 10.0.90.10 The authenticity of host '10.0.90.10 (10.0.90.10)' can't be established. RSA key fingerprint is 2e:fc:c7:36:33:70:db:16:d7:74:35:04:1a:3f:02:bb. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.0.90.10' (RSA) to the list of known hosts. Password: root@zone004 ~$ ftp 10.0.90.10 ^C
Sweet. Now just a look at the IP Filter stats to make sure its not a fluke:
root@zone001 ~$ ipfstat bad packets: in 0 out 0 IPv6 packets: in 0 out 0 input packets: blocked 5 passed 25 nomatch 8 counted 0 short 0 output packets: blocked 0 passed 15 nomatch 15 counted 0 short 0 input packets logged: blocked 5 passed 0
Perfect! Its actually blocking the packets. IP Filter works as you expect it too, in a zone on a vnic. Super sweet.