The Blog of Ben Rockwood

Crossbow for Christmas

29 Dec '08 - 09:20 by benr

After 2 years of waiting, Project Crossbow has arrived! It integrated into Nevada Build 105 on Dec 4th, and BFU's became available around the middle of the month. SX:CE isn't available just yet, but should be up in about a week I hope. Crossbow is huge. This is a monumental improvement to Solaris and continues to push the bar out of reach of its competitors.

Simply put, Crossbow redefines the nature of network virtualization. To date, virtualization was limited to creating traditional "virtual interfaces" like so:

root@quadra ~$ ifconfig e1000g1:1 plumb 10.0.0.50 netmask 255.255.255.0 up
root@quadra ~$ ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
e1000g1: flags=201000843 mtu 1500 index 2
        inet 10.0.0.18 netmask ffffff00 broadcast 10.0.0.255
        ether 0:1b:21:25:3e:7b 
e1000g1:1: flags=201000843 mtu 1500 index 2
        inet 10.0.0.50 netmask ffffff00 broadcast 10.0.0.255

Creating virtual interfaces like this gets the job done but has a number of drawbacks, all based on the fact that its not a real interface. Stats are screwed up, you can't snoop the interface, you can't tune it, etc.

Crossbow changes all that. Now we can create Virtual NIC's (vnic's) which are, for all intents and purposes, real interfaces. They have their own network stack and queues, they can be tuned, the can be snooped, they can be VLAN'ed, etc. Anything you can do to a real interface you can do to a VNIC.

While VNICs are handy things to have in the globalzone, they really shine when used with virtualization such as Solaris Containers (zones) or Xen guests, because we now can hand off interfaces that are fully controllable from within the virtual environment without having to dedicate a physical NIC to each one. The result is virtualized environments that feel way more like real servers.

If you're not already familiar with the dladm command its time for you to get acquainted. dladm is short for "Data Link Administration", and now compliments ifconfig. For some time now its been used for managing WIFI, 802.11ad Link Aggregation ("teaming" or "trunking", depending on your pedigree), and more recently VLANs. its even replacing the old (and crappy) ndd with dladm's "link properties"... a welcome improvement.

As of snv_105 several new options are available, namely sub-commands for creating VNICs and Etherstubs. A VNIC is a virtual network interface with all the trimmings of a real network interface. For the moment, it appears the max number of vnic's is 799, but thats not set in stone, and frankly if you need more than that you need to re-architect. Etherstubs are in-software switches which can be used in concert with VNIC's to create entirely virtualized in-software networks! In short, a standard VNIC will be associated with a physical GLDv3 network adapter, but we can also create a VNIC associated with an Etherstub to keep anything from ever touching the wire.

Lets ponder this. Why would you want a VNIC that uses a software switch (etherstub)? Seems completely useless right? Not entirely. On a traditional network you would create a DMZ with firewall and other goodies which routes to a private internal network... imagine that you can now do that all inside a single system!

Ok, so lets get cracking. Once you have snv_105 installed, we'll create a VNIC associated with physical e1000g1, then an etherstub and 3 more VNICs that are internal using that etherstub:

root@quadra ~$ dladm show-link
LINK        CLASS    MTU    STATE    OVER
e1000g1     phys     1500   up       --
e1000g2     phys     1500   down     --
e1000g0     phys     1500   unknown  --

root@quadra ~$ dladm create-vnic -l e1000g1 vnic0
root@quadra ~$ dladm create-etherstub etherstub0
root@quadra ~$ dladm create-vnic -l etherstub0 vnic1
root@quadra ~$ dladm create-vnic -l etherstub0 vnic2
root@quadra ~$ dladm create-vnic -l etherstub0 vnic3
root@quadra ~$ dladm show-link
LINK        CLASS    MTU    STATE    OVER
e1000g1     phys     1500   up       --
e1000g2     phys     1500   down     --
e1000g0     phys     1500   unknown  --
vnic0       vnic     1500   up       e1000g1
etherstub0  etherstub 9000  unknown  --
vnic1       vnic     9000   up       etherstub0
vnic2       vnic     9000   up       etherstub0
vnic3       vnic     9000   up       etherstub0

So we have a variety of VNIC's at our disposal. We now treat these like regular interfaces, using ifconfig to plumb them and assign IP's:

root@quadra ~$ ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
e1000g1: flags=201000843 mtu 1500 index 2
        inet 10.0.0.18 netmask ffffff00 broadcast 10.0.0.255
        ether 0:1b:21:25:3e:7b 

root@quadra ~$ ifconfig vnic0 plumb 10.0.0.19 up
root@quadra ~$ ifconfig vnic1 plumb 10.100.0.2 netmask 255.255.255.0 up
root@quadra ~$ ifconfig vnic2 plumb 10.100.0.3 netmask 255.255.255.0 up
root@quadra ~$ ifconfig vnic3 plumb 10.100.0.4 netmask 255.255.255.0 up

root@quadra ~$ ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
e1000g1: flags=201000843 mtu 1500 index 2
        inet 10.0.0.18 netmask ffffff00 broadcast 10.0.0.255
        ether 0:1b:21:25:3e:7b 
vnic0: flags=201000843 mtu 1500 index 7
        inet 10.0.0.19 netmask ff000000 broadcast 10.255.255.255
        ether 2:8:20:3a:70:5a 
vnic1: flags=201000843 mtu 9000 index 8
        inet 10.100.0.2 netmask ffffff00 broadcast 10.100.0.255
        ether 2:8:20:f2:56:4d 
vnic2: flags=201000843 mtu 9000 index 9
        inet 10.100.0.3 netmask ffffff00 broadcast 10.100.0.255
        ether 2:8:20:bc:b1:a1 
vnic3: flags=201000843 mtu 9000 index 10
        inet 10.100.0.4 netmask ffffff00 broadcast 10.100.0.255
        ether 2:8:20:55:11:56

Please notice that they all have individual MAC addresses! There are severla methods for how the MAC is chosen, but I won't go into them here.

If you are using Solaris Containers these VNIC's would be given to a Zone as an "IP-Instance" (exclusive mode), a feature which was added some time ago but untill now only usable by dedicating a physical interface. The same should apply to Xen or other virtualization tools.

Finally, in our whirlwind tour of this amazing technology, lets look at my favorite feature of Crossbow.

Crossbow is both Network Virtualization (we looked at that above) and Network Resource Control. With Crossbow we have a real network resource control capability that is free from the terror that is IPQoS.

There are three types of resource controls at present: max bandwidth (rate limiting), priority (relative to other traffic), and cpu's. Please note that these controls are not cumulative, but rather apply to any given point in time. These controls can be applied either to an entire link (NIC or VNIC) or alternatively to a particular network flow.

Let me pause here. If your not familiar with a "network flow", it is a defined collection of network communication. For instance, a flow might refer to all HTTP (port 80) traffic to a given IP address, or perhaps all TCP traffic, or perhaps a combination of FTP, SMTP, and HTTP ports. If you've worked with firewall rules your familiar with the concept, a flow simply allows us a way to apply some action to a specific flow of traffic.

Crossbow adds the new command flowadm to define and control network flows. Here is an example:

root@quadra ~$ flowadm add-flow -l vnic0 -a transport=tcp,local_port=80 httpflow
root@quadra ~$ flowadm add-flow -l vnic0 -a transport=tcp,local_port=443 httpsflow
root@quadra ~$ flowadm show-flow
FLOW        LINK        IP ADDR                        PROTO  PORT    DSFLD
httpflow    vnic0       --                             tcp    80      --
httpsflow   vnic0       --                             tcp    443     --

flowadm relies on attributes that describe a flow, and properties which assign some resource control. We'll add bandwith control to the flows above by modifying the "maxbw" property:

root@quadra ~$ flowadm show-flowprop
FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
httpflow     maxbw              50          --             50M 
httpflow     priority        --             --             
httpsflow    maxbw              80          --             80M 
httpsflow    priority        --             --

Here the maxbw is specified in Mbps. Docs show that percentages, Kbps, etc are supported, but they don't seem to work right now.

maxbw will rate limit to the specified throughput, priority can be set "low", "normal", "high" or "rt" (real time). Using these controls carefully you can partition off bandwidth pretty nicely.

In addition to all this, extended accounting has been extended to incorporate accounting based on links or flows, but I'll save that for another day.

Congrats to everyone on the Crossbow team. This is a major achievement and an amazing technological advance!

- - C O M M E N T S - -
This sounds great. As a Joyent customer, I would love to have detailed network statistics for my zones.

A.

Adam (Email) (URL) - 29 December '08 - 15:12

I have been following crossbow for awhile, very excited about it. I know OpenSolaris has ipf integrated but if OpenBSD’s pf were integrated this would make me switch my firewall systems to OpenSolaris too….

Zachary Schneider (URL) - 29 December '08 - 16:19

Ben,

Great post. However, could you put the following into a container?

using ifconfig to plumb them and assign IP’s:
root@quadra ~$ ifconfig -a lo0
Please notice that they all have individual MAC addresses!

Thanks,
alan

Alan Pae (Email) (URL) - 29 December '08 - 19:31

Sorry,

Make that an html “pre” container

Alan Pae (Email) (URL) - 29 December '08 - 19:32

Fixed.

benr - 29 December '08 - 19:51

this is outright crazy – i didnt know i had gotten something that great for christmas :)

darkfader (Email) - 30 December '08 - 17:56

Great Overview on Crossbow Ben! And thanks everyone for
trying it out. You can find more details relating to architecture
or building advanced virtual networks aka Virtual Wire at
[[http://blogs.sun.com/sunay]]

Sunay Tripathi (URL) - 05 February '09 - 07:52

FYI, Jonathon Schwartz just linked to this post (second last paragraph):

[[http://blogs.sun.com/jonathan/entry/q2..]]

David Magda (URL) - 07 February '09 - 02:07

Great, yet even more complexity added to an already cryptic OS! I’m sorry but Solaris is becoming ever more hard to follow, constant updates and changes.

Mike - 12 February '09 - 13:00

“and now compliments ifconfig.” By offering flowery praise? Bowing?

You want “complements” – and no it’s not just a spelling error. The terms complement and supplement have well defined, well understood, meanings in economics that get borrowed in usage like this. People who use “compliments” instead are demonstrating that they don’t know, and don’t care.

FYI:
A long time ago two people I knew started an economics consultancy – and described their skills as complimentary in their marketing materials. Go figure :)

Paul Murphy (Email) (URL) - 16 February '09 - 20:11

know OpenSolaris has ipf integrated but if OpenBSD’s pf were integrated this would make me switch my firewall systems to OpenSolaris too..

Chris (Email) (URL) - 10 May '09 - 07:06

OpenSolaris has ipf integrated but if OpenBSD’s pf were integrated this would make me switch my firewall systems to OpenSolaris too….

samax (Email) (URL) - 10 August '09 - 16:47

With Crossbow you indeed have a real network resource control capability that is free from the terror that is IPQoS.

how to grow taller 101 (Email) (URL) - 31 August '09 - 15:18

wow, that looks quite complicated!

Grow Taller (Email) (URL) - 11 October '09 - 15:45

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

Christian louboutin shoes (Email) (URL) - 27 October '09 - 01:31

Thank you very much!

cheap links of london (Email) (URL) - 07 November '09 - 01:06

yeah ,i think so

christian louboutin (Email) (URL) - 15 November '09 - 11:39

Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very good！
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.

links of london (Email) (URL) - 17 November '09 - 03:25

christian louboutin shoes (Email) (URL) - 29 November '09 - 06:29

good post,I think so!

christian louboutin shoes (Email) (URL) - 29 November '09 - 06:30

i want to give something intresting to introduce you,come to look these christian louboutin shoesit is a beautiful world

wrwr (Email) (URL) - 08 December '09 - 03:53

links of london sweetie ring
links of london ring

liubaiying (Email) (URL) - 08 December '09 - 08:28

Your articles make me very comfortable, so, I introduce a good thing to you ：a lot of pleated and ruched shoes and bags this fall. And here’s another example from Christian Louboutin.We hope you enjoy！

christian louboutin shoes (Email) (URL) - 10 December '09 - 06:32

christian louboutin shoes (Email) (URL) - 24 December '09 - 01:57

I was very pleased to find this site.I wanted to thank you for this great read!! I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you post.
regards,

christian louboutin (Email) (URL) - 28 December '09 - 11:14

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

christian louboutin boots (Email) (URL) - 28 December '09 - 11:15

thanks your commetns,this is good job!It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. christian louboutin christian louboutin street lighting .I will be your loyal reader. Thanks again.

christian louboutin shoes (Email) (URL) - 28 December '09 - 11:15

links of london sweetie bracelet
links of london sweetie

Pauline (Email) (URL) - 29 December '09 - 09:37

This is a great blog! I would really like to keep reading your blogs. Cant wait for the next post. Good blogs are hard to find!thank you,and I also like the christian louboutin shoes,and it has a new style and good quality.

christian louboutin (Email) (URL) - 30 December '09 - 06:36

thank you for sharing it .and the article is very inspiring us to working hard in the company which sells christian louboutin shoes online.please come and see it.

christian louboutin (Email) (URL) - 31 December '09 - 08:40

Just one question: how to add your blog into my rrs reader, thanks so much.

christian louboutin (Email) (URL) - 03 January '10 - 06:22

thanks for sharing!I like this artic.

louboutin (Email) (URL) - 03 January '10 - 07:19

I was very pleased to find this site.I wanted to thank you for this great read!! I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you post.
regards,

christian louboutin (Email) (URL) - 04 January '10 - 12:18

It is hard for women to resist the temptation of jewelries and women’s jewelry box is like an abysm which is never full.

liubaiying (Email) (URL) - 05 January '10 - 03:19

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

christian louboutin (Email) (URL) - 06 January '10 - 03:11

We supplier the most complete ugg boots collections at favorable price,chose your ugg boots here.our ugg boots is your best choice.

uggs (Email) (URL) - 06 January '10 - 10:04

I was very pleased to find this site.I wanted to thank you for this great read!! I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you post.

christian louboutin (Email) (URL) - 07 January '10 - 12:17

thank you ,I love it very much

christian louboutin shoes (Email) (URL) - 08 January '10 - 03:11

This may be a perfect example of information asymmetry and adverse selection in insurance. I hope all textbook authors and legislators notice.

christian louboutin (Email) (URL) - 10 January '10 - 06:59

Thank you for the sensible critique. Me & my neighbour were preparing to do some research about that. We got a good book on that matter from local library and most books where not as influensive as your information. I am very glad to see such information which I was searching for a long time.This made very glad Smile..

louboutin (Email) (URL) - 13 January '10 - 02:09

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

christian louboutin (Email) (URL) - 14 January '10 - 11:48

Good post! Thanks you for your information! China Wholesale Wholesale China Wholesalers Wholesale Game Accessories Wholesale Iphone Accessories Video Game Accessories Wholesale Wholesale Wii Accessories Wholesale Xbox 360 Accessories Wholesale Xbox 360 Games Wholesale Video Games Cheap Video Games Cheap Ps3 Games Cheap Xbox 360 Games Wholesale Computers Wholesale Laptop Computers Wholesale Laptops Discount Computers Cheap Computers Wholesale Iphones Wholesale Iphone Wholesale Iphones 3g Hiphones Wholesale Hiphone Wholesale Hiphones Wholesale Nokia Wholesale Nokia 8800 Wholesale Nokia n97 wholesale blackberry wholesale blackberry phones wholesale blackberry 9700 wholesale blackberry 9600 wholesale blackberry 9500

raging bull (Email) (URL) - 15 January '10 - 06:26

Good post! Thanks for your information! ed hardy ed hardy ed hardy clothing ed hardy clothing ed hardy swimwear ed hardy swimwear ed hardy jeans ed hardy jeans ed hardy hoodies ed hardy hoodies ed hardy shoes ed hardy shoes ed hardy uk ed hardy uk ed hardy bags ed hardy bags ed hardy shirts ed hardy shirts christian audigier christian audigier ed hardy mens ed hardy mens ed hardy womens ed hardy womens ed hardy kids ed hardy kids ed hardy

ed hardy (Email) (URL) - 15 January '10 - 07:28

Once she was satisfied, she jumped out of her chair and headed to the comm shack links of london jewellery, pushing past Rynyan, who was loitering in the hatchway and clearly relished the close contact links of london. I’m caUing for the beam. Sooner you two dreamers are out of my hair, the better. And out of my head Links of London sale .

charms (Email) (URL) - 16 January '10 - 09:09

Heat can’t distract from pure fabulosity… Love those shoes.When do they go on sale again? and please tell me they will be in size 11… The big foot girls also love choosing christian louboutin too! Because they are so fit you——girls,they will be your best choice!

christian louboutin (Email) (URL) - 19 January '10 - 12:20

Thank you for sharing your brushes! I’m having fun playing with them.

linksjuy (Email) (URL) - 20 January '10 - 08:54

thanks

kamagra (Email) (URL) - 21 January '10 - 01:08

good read thanks

M65 Jacket (Email) (URL) - 21 January '10 - 01:08

many thanks

viagra cialis (Email) (URL) - 21 January '10 - 01:53

What is unique to Pandora bracelets is that every each style is unique. Belles can even organize each string of beads, diamonds and pendants according to their own tastes.Rolling slightly with movements of the wrist as beads in the bracelet have a certain room to slide，Pandora bracelet enables girls to be the eye-catching ones.And the well selected diamonds and ultimate manual carving ornamentations bring much charm and a sense of fashion to those smooth and round gemstones with wild colors, which represent a colorful beauty.The various sweet and lovely jewelry designs awake the sweet and lovely heart of girls，shaping their ideal and romantic temperament perfectly. It’s so touching and irresistible.

Omnilens (Email) (URL) - 21 January '10 - 07:04

I agree and I will come back and follow more of your posts.

american drew furniture (Email) (URL) - 21 January '10 - 20:29

thank you for sharing, it is great.

charms (Email) (URL) - 22 January '10 - 07:19

Faithful links of london ladies are always looking for discount links of london collections among UK, and we can get access to the discount collections like links of london bracelets and cheap links of london charms on the internet.

links of London (Email) (URL) - 22 January '10 - 08:20

good post

abercrombie clothing (Email) (URL) - 22 January '10 - 08:29

I agree and I will come back and follow more of your posts.

charms (Email) (URL) - 22 January '10 - 08:40

Thank you for the valuable information. I have used most of these services. Keep up the good work.

discount links of london (Email) (URL) - 22 January '10 - 09:15

Oh,you said right,i like you!

tiffany (Email) (URL) - 23 January '10 - 03:58

Thank you for sharing.Nice post.

christian louboutin (Email) (URL) - 25 January '10 - 11:50

Good post! Thanks for your information!

As Seen On TV (Email) (URL) - 27 January '10 - 06:12

Everything will be all right,I am behind you.

iphone (Email) (URL) - 29 January '10 - 07:31

New battery review module surface is black, not because of the positive electrode exists, the module is likely to be integrated back contact silicon solar cells, silicon, where the role is often to extend the service life of equipment.Mitsubishi motors in October 2009 28-30 at the yokohama exhibition center held the Pacific “GreenDevice2009″ displayed on the high efficiency solar camera battery 19.1% respectively. In the square 15cm in polysilicon solar cells, realized the highest efficiency. The research achievements in 2009 September 2009 and the application of physics to the international society in October EUPVSEC “, “said in a statement.

battery review (Email) (URL) - 30 January '10 - 03:29

Good post! Thanks for your information! As Seen On TV

As Seen On TV (Email) (URL) - 30 January '10 - 06:00

GOOD

links of london (Email) (URL) - 05 February '10 - 06:23

[[http://www.ladyshoesstore.com/]]

manolo (Email) (URL) - 06 February '10 - 08:54

There are many Tiffany jewellery renowned designers of engagement rings from different countries. As usual, I loveTiffany Charms the simplicity of the design. This dress has a bodice that is strapless, and has a very subtle pleating. 00. tiffanyTiffany Bracelet rings are Tiffany Sets exceptional Tiffany Rings and extremely beautiful. The Tiffany Earrings company Tiffany Necklace will have a much tougher battle in Europe, however, where tiffany has opened six stores to sell tiffany necklace and Tiffany & co since 1986. Everyone has many accessories whether men or women. Make sure that you buy it early because “Tiffany” dresses Tiffany Pendant usually sell out pretty quickly. It’s not too fancy for everyday and Tiffany Accessories not to casual when dressed up.

happy23 (Email) - 07 February '10 - 02:13

Your article is very useful!Thank you for sharing.Nice post.

laptop battery club (Email) (URL) - 08 February '10 - 08:38