The Blog of Ben Rockwood

Understanding ZFS: Transaction Groups & Disk Performance

23 Jan '09 - 06:42 by benr

I've been deeply concerned about the number of people who continue to use iostat as the means to universally judge IO as "good" or "bad". Before I explain why, lets review iostat.

# iostat -xnM c0t1d0 1  
                    extended device statistics              
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
   19.9  240.9    2.1   18.4  0.0 13.7    0.2   52.4   4  56 c0t1d0
  127.2    0.0   14.7    0.0  0.0  1.0    0.1    7.7   2  78 c0t1d0
  116.0  375.0   13.6   21.6  0.1  3.2    0.1    6.5   7  82 c0t1d0
   27.0  407.0    2.6   30.8  0.1  9.3    0.1   21.5   6  99 c0t1d0
   95.0    6.0   11.7    0.2  0.0  0.9    0.1    8.6   1  78 c0t1d0
^C

The first 4 columns we can agree on, reads and writes per second. The following 6 columns get the concern. These are 3 different ways of viewing essentially the same data, the active queue (sent to device) and wait queue (waiting to be sent). Note that all these are based on Kstats and can be easily re-formulated into custom tools.

Universal "rules of thumb" regarding queues are very dangerous. I've heard such ridiculous suggestions as anything over 5% busy is a problem. The busy time simply denotes the quantity of time that IO was active... thus 100% busy means your doing a lot of IO, not that its slow. Over simplifying the interpretation is as old skool as suggesting that a CPU more than 75% busy needs to be upgraded, which is moronic (but sells a lot of servers).

When it comes to iostat you must carefully balance the numbers of IOs, size of IOs together with the service time and come to a conclusion based on that. If your doing a large streaming write you would expect to see 100% busy but very high throughput. Naturally, the great enemy of storage performance is random workloads that require a lot of head movement, in which case those seek times will kill you and things start to back up, in those cases tune your app... application caching is always a bigger win than adding spindles.

In interactive shared server environments the closest to a rule I've ever provided was that, in my experience, active services times below 30ms are optimal, between 30ms and 100ms worry some and higher than 100ms means that someone out there is probly unhappy. When IO's are regularly taking more than 100ms to complete is likely that the next fellow to type "ls" in an uncached directory is going to be pissed off.

Now, all this gets more complicated with ZFS in the mix. You probly have heard that ZFS is transactional and as a result is always consistent-on-disk. But few really spend time thinking that through. Similar to an OLTP database, transactions are created, work is done, and then finally committed. This commit sends the transaction (tx) into a transaction group (txg) for "sync" to disk. At any given time there are 3 transaction groups: one in an open state accepting transactions, one is a quiescence state ready for sync, and one being sync'ed to disk. (For the sake of simplicity I'll leave discussion regarding O_DSYNC synchronous writes and ZIL out of this discussion, for now.)

Between these transaction groups gathering writes in memory for orderly flush to disk and the ARC filesystem cache, most of your run of the mill IO is going back and forth between memory. I have a great many machines servicing more than 100,000 read ops per second without a single resulting physical read IO. ZFS efficiency is truly incredible. So in this way, the physical metrics can have little to nothing to do with the actual user-experience making typical tuning based on iostat highly suspect, if not entirely meaningless.

So, first things first. On a ZFS system never look at iostat alone. Always open 2 terminals side-by-side and in one terminal watch fsstat zfs 1 and in another watch iostat -xn 1 (or 10 seconds, whatever your happy with). By watching both of these you'll get a better idea of whats really going on, and I expect that you'll be impressed by what you see.

As for async writes. What I really would like to see is how these transaction groups are doing. How often are transaction groups sync'ing to disk, how much are they sync'ing, and how much time is there in between. Prior to snv_87 transaction groups would flush upon fullness (1/8th of system memory) or a txg_time tunable defaulted to 5 seconds. As a result, if you've looked at a system running something earlier than snv_87 and saw IO "spike" every 5 seconds, this is why... its normal and healthy. In snv_87 a new ZFS Write Throttle was introduced and among the changes the sync timer got pushed out to 30 seconds. So, likewise, if you have a box post-87 that "spikes" every 30 seconds you have a very healthy system.

Knowing this is all good and well, but I'd like to see it. After spending a good amount of time in the code I realized that spa_sync() is the function to watch. Its whats responsible for actually sync'ing the txg to disk (God Bless DTrace stack[] aggregations!). With this knowledge I wrote up a Dscript that I was proud of, but, of course, now that I knew what to look for, found that Roch wrote up essentially the same thing 2 years ago.... never the less, I made a tweek and here it is:

#!/usr/sbin/dtrace -qs

/*
 * spa_sync.d - ROCH http://blogs.sun.com/roch/entry/128k_suffice 
 * mods by benr
 * 
 * Measure I/O throughput as generated by spa_sync 
 * Between the spa_sync entry and return probe
 * I count all I/O and bytes going through bdev_strategy.
 * This is a lower bound on what the device can do since
 * some aspects of spa_sync are non-concurrent I/Os.
 */

BEGIN {
        tt = 0; /* timestamp */
        b = 0; /* Bytecount */
        cnt = 0; /* iocount */
} 

spa_sync:entry/(self->t == 0) && (tt == 0)/{
        b = 0; /* reset the I/O byte count */
        cnt = 0;
        tt = timestamp; 
        self->t = 1;
        printf("%Y", walltimestamp);
}

spa_sync:return
/(self->t == 1) && (tt != 0)/
{
        this->delta = (timestamp-tt);
        this->cnt = (cnt == 0) ? 1 : cnt; /* avoid divide by 0 */
        printf("t: %d MB; %d ms of spa_sync; avg sz : %d KB; throughput %d MB/sn",
                b / 1048576,
                this->delta / 1000000, 
                b / this->cnt / 1024,
                (b * 1000000000) / (this->delta * 1048676)); 
        tt = 0;
        self->t = 0;
}

/* We only count I/O issued during an spa_sync */
bdev_strategy:entry 
/tt != 0/
{ 
        cnt ++;
        b += (args[0]->b_bcount);
}

Here is a sample output, pre-snv_87:

# ./spa_sync.d 
2009 Jan 23 06:12:28    : 44 MB; 743 ms of spa_sync; avg sz : 68 KB; throughput 59 MB/s
2009 Jan 23 06:12:33    : 81 MB; 1716 ms of spa_sync; avg sz : 79 KB; throughput 47 MB/s
2009 Jan 23 06:12:38    : 45 MB; 736 ms of spa_sync; avg sz : 65 KB; throughput 61 MB/s
2009 Jan 23 06:12:43    : 41 MB; 700 ms of spa_sync; avg sz : 67 KB; throughput 59 MB/s
2009 Jan 23 06:12:48    : 56 MB; 1287 ms of spa_sync; avg sz : 63 KB; throughput 43 MB/s
2009 Jan 23 06:12:53    : 35 MB; 668 ms of spa_sync; avg sz : 65 KB; throughput 52 MB/s
2009 Jan 23 06:12:58    : 61 MB; 1147 ms of spa_sync; avg sz : 62 KB; throughput 53 MB/s
2009 Jan 23 06:13:03    : 41 MB; 624 ms of spa_sync; avg sz : 60 KB; throughput 67 MB/s
2009 Jan 23 06:13:08    : 37 MB; 658 ms of spa_sync; avg sz : 60 KB; throughput 56 MB/s
2009 Jan 23 06:13:13    : 59 MB; 1035 ms of spa_sync; avg sz : 68 KB; throughput 57 MB/s
^C

Notice it hitting the 5 second mark nicely. This output is significantly more telling and encouraging than simply looking at iostat alone.

This of course also reminds us... Roch Rocks.

- - C O M M E N T S - -
i was surprised to see that the dtrace script completely hung my thumper with NexentaStor installed on it after running for 2 minutes.

Mike (Email) - 23 January '09 - 23:09

i verified with latest v1.1.5 of just released NexentaStor software – it runs OK

erast (Email) - 27 February '09 - 22:34

Well….i was surprised to see that the dtrace script completely hung my thumper with NexentaStor installed on it after running for 2 minutes.

Chris (Email) (URL) - 10 May '09 - 07:37

The examples you give of ZFS, are in fact perfect for illustrating this point.

apri (Email) (URL) - 12 July '09 - 17:18

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

Christian louboutin shoes (Email) (URL) - 27 October '09 - 02:41

Thank you very much!

ed hardy clothing (Email) (URL) - 09 November '09 - 00:32

Thanks!!!

Tiffany jewellery (Email) (URL) - 14 November '09 - 02:34

yeah ,i think so

christian louboutin (Email) (URL) - 15 November '09 - 07:31

Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very good！
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.

links of london (Email) (URL) - 17 November '09 - 03:16

Then do you know the christian louboutin,so come to see these christian louboutin shoes.

christian louboutin (Email) (URL) - 18 November '09 - 12:03

Nice post. Publishing, consulting and regular life all kind of

merge on the web and it is sometimes hard to know which hat you

have on.

Christian Louboutin Shoes (Email) (URL) - 20 November '09 - 06:12

Thanks for your sharing things! They’re gorgeous! We’re getting ready to launch our new online Christian Louboutin for smaller busted women, so I can definately appreciate all the hard work you’ve put into expanding your business! I know it’s quite a fete!!! Congratulations! Looking forward to seeing your shoes at more events and websites!
Heat can’t distract from pure fabulosity… Love those shoes.When do they go on sale again? and please tell me they will be in size 11… The big foot girls also love choosing christian louboutin too! Because they are so fit you——girls,they will be your best choice!

christian louboutin shoes (Email) (URL) - 06 December '09 - 02:25

cheap ugg
ugg outlet
ugg boots
cheap uggs online
cheap ugg online
cheap ugg boots online
[[http://www.cheapuggsonline.net]]

cheap uggs online (Email) (URL) - 17 December '09 - 06:39

[..]]]]]]]] Uggs boots(shoes)Store-free shipping! Discount40%-50% off.
Welcome to have a look!
ugg fashion shoes—classic,cove,nightfall,sundance,mini…

UGGBOOTS-free shipping! Discount40%-50% off (Email) (URL) - 30 December '09 - 07:43

Just one question: how to add your blog into my rrs reader, thanks so much.

christian louboutin (Email) (URL) - 03 January '10 - 06:27

they will be in size 11… The big foot girls also love choosing christian louboutin too!

brand clothing (Email) (URL) - 03 January '10 - 07:42

I love reading about all the success and accomplishments you have achieved. You’re truly an inspiration.And the same as christian louboutin, it also a good style for you and I believe you will love it .

christian louboutin (Email) (URL) - 05 January '10 - 07:29

This may be a perfect example of information asymmetry and adverse selection in insurance. I hope all textbook authors and legislators notice.

christian louboutin shoes (Email) (URL) - 06 January '10 - 08:50

Good work! Your post/article is an excellent example of why I keep comming back to read your excellent quality content that is forever updated. Thank you!roulette onlinepoker sitesblackjack onlinevideo poker onlinedivx movie downloads

christian louboutin (Email) (URL) - 08 January '10 - 09:03

links of london

links of london (Email) (URL) - 11 January '10 - 11:23

Thank you for the sensible critique. Me & my neighbour were preparing to do some research about that. We got a good book on that matter from local library and most books where not as influensive as your information. I am very glad to see such information which I was searching for a long time.This made very glad Smile..

louboutin shoes (Email) (URL) - 13 January '10 - 03:48

influensive as your information. I am very glad to see such information which I was searching for a long time.This made very glad Smile..

MBT Shoes Sale (Email) (URL) - 14 January '10 - 08:58

many thanks

M65 Jacket (Email) (URL) - 21 January '10 - 02:01

[[http://www.buykamagra.com]] buy kamagra
[[http://www.viagracialis.com]] viagra cialis

M65 Jacket (Email) (URL) - 21 January '10 - 02:05

very good evden eve nakliyat
evden eve nakliyat

evden eve nakliyat - 27 January '10 - 03:45

[[http://www.evdenevenakliyatt.net/]]
[[http://www.seckinnakliyat.com/]]

evden eve nakliyat - 27 January '10 - 03:47

New battery review module surface is black, not because of the positive electrode exists, the module is likely to be integrated back contact silicon solar cells, silicon, where the role is often to extend the service life of equipment.Mitsubishi motors in October 2009 28-30 at the yokohama exhibition center held the Pacific “GreenDevice2009″ displayed on the high efficiency solar camera battery 19.1% respectively. In the square 15cm in polysilicon solar cells, realized the highest efficiency. The research achievements in 2009 September 2009 and the application of physics to the international society in October EUPVSEC “, “said in a statement.

battery review (Email) (URL) - 30 January '10 - 03:34

thank you

evden eve nakliyat istanbul (Email) (URL) - 05 February '10 - 19:22

nice post here, if you want to know more about best mbts and ugg boots, just click here
[[http://www.bestmbtshoes.com]]
[[http://www.uggbootsroom.com]]

best mbt (Email) (URL) - 06 February '10 - 03:27

thank you

puma sneaker (Email) (URL) - 06 February '10 - 07:27

thank you very muchh very good post

sohbet (Email) (URL) - 07 February '10 - 09:19

nice post here, if you want to know more about best mbts and ugg boots, just click here

tatil otelleri (Email) (URL) - 24 February '10 - 21:44

Ed Hardy Hats were later launched with much fanfare.

ed hardy (Email) (URL) - 25 February '10 - 03:40

çok mükemmel

halÄ±yÄ±kama (Email) (URL) - 26 February '10 - 04:30

nowGoogle.com adalah Multiple Search Engine Popular memiliki Fungsi Multiple Search Engine hadir di Catatan Si Bongo dengan nama nowGoogle.com

nowGoogle.com adalah Multiple Search Engine Popular (Email) (URL) - 09 March '10 - 16:57

if you want to know more about best mbts and ugg boots, just click here

links of london (Email) (URL) - 11 March '10 - 01:42

the christian louboutin shoes is high quality and the price is low.
christian louboutin

christian louboutin (Email) (URL) - 30 March '10 - 07:32

we supply high quality louboutin shoes at low price.
[[http://www.onsale-ghd.com]]
[[http://www.mychristianlouboutinshoes.c..]]

louboutin boots (Email) (URL) - 07 April '10 - 05:56

This is just a theory but I think it’s pretty sound. [[http://www.dragonkicks.com]]

air jordan shoes (Email) (URL) - 07 April '10 - 07:08

I read your article with great pleasure. This wonderful article thank you for sharing with us

antalya otelleri (Email) (URL) - 07 April '10 - 07:28

We’re getting ready to launch our new online Christian Louboutin for smaller busted women, so I can definately appreciate all the hard work you’ve put into expanding your business

womens high heels (Email) (URL) - 08 April '10 - 02:22

Sounds cool! polo T-shirts

saylor (Email) (URL) - 10 April '10 - 04:04

i agree with you

ghd hair straighteners (Email) (URL) - 19 April '10 - 03:01

I very intersted in the article

cheap mbt shoes (Email) (URL) - 23 April '10 - 03:26

thank you,I learn a lot

supra skytop (Email) (URL) - 27 April '10 - 09:32

I’m a little late finding this article, but I loved it.

As we know, now sunglasses are loved by more and more people, and everyone would like to have a pair of comfortable sunglasses,save up to 47%.welcome to [[http://www.eyewear-rayban.com]]

rayban sunglasses (Email) (URL) - 29 April '10 - 07:17

Free shipping and top quality,comfortable fit

Nike air max shoes sale (Email) (URL) - 30 April '10 - 19:28

great post
[[http://www.sellnikesbs.com]]

mbt sheos (Email) (URL) - 01 May '10 - 01:54

discount mbt shoes [[http://www.thesuitshoes.com]] mbt shoes on sale.

www.thesuitshoes.com (Email) (URL) - 05 May '10 - 03:26

P90x .It really is not expensive if you factor in the cost

of a gym membership,P90x workout . The cost for P90X is

about three months of a paid gym membership but you get to

keep the program foreverP90x . You can try many of the

online sites, but it will be the same as buying from the

company or a Beachbody Coach. Make sure you are getting

original DVD’s. People are selling copies all over. The

problem is how long will they last, P90x workout ,and you

truly need the exercise and nutrition guide to even follow

the program. You can go to any site or you can go to

[[http://www.p90xmall.com]] and click on products. P90x dvd You can

order directly from the site,P90x dvd.

p90x (Email) (URL) - 05 May '10 - 05:46

[[http://www.gucci-shoes-bags.com]]
[[http://www.guccinewshop.com]]
Dear friends welcome to our store: we have a specail offer now which is once you buy any product you can get a free gift as well, these gifts are in the gift area, you can free to choose and please add to the shopping cart, we will send it together with your purchased product. Thanks!

gucci (Email) (URL) - 06 May '10 - 06:47

The new high can browse your article

towatches (Email) (URL) - 07 May '10 - 03:52

The new high can browse your article
[[http://www.towatches.com/Discount-Watc..]]

towatches (Email) (URL) - 07 May '10 - 03:53

china wholesale usb mp3 player
[[http://www.iaamart.com]]

china wholesale usb mp3 player (Email) (URL) - 11 May '10 - 09:51

hi there, thanks for your great post, it help me so much! now i wonder if you linke the mbt shoes from my site: the url is [[http://www.thesuitshoes.com]]

mbt shoes (Email) (URL) - 12 May '10 - 07:03

high heel bootsale Louboutin’s shoes at shops across the worldWomens high heel shoes

Louboutin Shoes (Email) (URL) - 12 May '10 - 12:12

If you are looking for the best brand new laptop batteries at the most preferred price, you have come to the right place. [[http://www.adapterlist.com/toshiba/sat..]] We provide the highest quality Battery at the lowest price with the highest level of service, all in a secure and convenient platform.

laptop battery (Email) (URL) - 14 May '10 - 08:47

thank you for this outstanding article.I thought certino was the best technologh for laptop battery performance.We specialize in [[http://www.globallaptopbattery.co.uk/d..]] ,laptop AC adapters. All our products are brand new, with the excellent service from our laptop battery of customer service team.

laptop batteries (Email) (URL) - 14 May '10 - 08:48

thanks good post

sohbet odalarÄ± (Email) (URL) - 21 May '10 - 18:32

thanks good post

bÃ¼yÃ¼ bozma (Email) (URL) - 22 May '10 - 09:20

Helo, your blog is really good, I like it very much!By the way, if you like nike chaussures tn you can come here to have a look!
[[http://www.tnchaussurescom.com]]
[[http://www.sunglassesol.net]]

tnchaussures (Email) (URL) - 26 May '10 - 06:18

Buy Nike Air Max 90 Shoes just $45-55 USD in[[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

nike air max 90 shoes (Email) (URL) - 28 May '10 - 02:49

Buy Nike Air Max 90 Shoes just $45-55 USD in [[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

nike air max 90 shoes (Email) (URL) - 28 May '10 - 05:49

7 days herbal slim7 days herbal slim

lose weight (Email) (URL) - 31 May '10 - 03:31

thanks for this artice very good

sohbet odalari (Email) (URL) - 31 May '10 - 19:09

thanks for the info

sohbet odalari (Email) (URL) - 31 May '10 - 19:09

[[http://www.air]] jordan shoes cheap.com/

air jordan shoes (Email) (URL) - 02 June '10 - 01:46

URL) - 03 June '10 - 03:36

[[http://www.vibramfivefinger.us/]] vibram five fingers

vibram five fingers (Email) (URL) - 10 June '10 - 08:31

[URL]www.nike-shox.net[/URL] nike shox

nike shox (Email) (URL) - 21 June '10 - 01:43

nike shoes

nike shoes (Email) (URL) - 21 June '10 - 01:44

[url=[[http://www.2buybags.com/replica-gucci-..]] Gucci Purses[/url]

replica Gucci Purses (Email) (URL) - 21 June '10 - 01:45

[[http://www.ushoes.net]] : Nike Air Force Ones

Nike Air Force Ones (Email) (URL) - 21 June '10 - 01:47

[[http://www.sxhgz.com]] : nike shoes

puma shoes (Email) (URL) - 21 June '10 - 01:53

It’s very nice! I love what you wrote.
I think we can make friends.

vibram five fingers (Email) (URL) - 22 June '10 - 08:09

[[http://www.mbtshoeslatest.com]]
[[http://www.nikeairmaxshoe.com]]

Nike air max shoe (Email) (URL) - 24 June '10 - 07:42

I totally love this article.

ed hardy (Email) (URL) - 02 July '10 - 08:43

American classic leather goods brand Coach, the style of which is simple and durable and are popular with most people. Coach Bags has good reputation, which are embraced by the high society and the common people. The original inspiration of designing Coach Handbags is from one softball glove. The first Coach Brand founder, Miles caha watch the softball game and found the special features of its gloves Coach Handbags 2010, which is convenient and durable and then he design Coach bags at once. As a result, it is popular by most consumers. Furthermore, these Cheap Coach Bags are at a competitive price which can easily accept by large people and now there are many Coach Outlet set up among many countries so you can buy it very easily.
[[http://www.bag-salon.com]]

coach bags (Email) (URL) - 04 July '10 - 08:07

bursa evden eve nakliyat firmalari, bursa evdeneve nakliyat yapan firmalar.

evden eve nakliyat (Email) (URL) - 12 July '10 - 05:27

I know you probably get a lot of comments like this Air Jordan 2010, but just wanted to let you know that I really appreciate the work you have put into the blog. I was wondering if I could put a link on my blog because I am sure my followers would love to read it Air jordan 1. Let me know.

Air Jordan (Email) (URL) - 03 August '10 - 03:59

Couldn’t be written any better. Reading this post reminds me of my old room mate! He always kept talking about this Michael Jordan Shoes. I will forward this article to him. Pretty sure he will have a good read. Thanks for sharing!Welcome First time jumped here on your site, found on Bing. It really helps to have sombody to talk to other then my divorced parents Cheap Jordan Shoes. Thanks again.

Air jordan 1 (Email) (URL) - 03 August '10 - 06:49

Your post is awesome, but why not take a look at our site: [[http://www.p90xwork.com]]

discount p90x (Email) (URL) - 06 August '10 - 15:18

The first layer of MBT Shoes means a high quality MBT Footwear cowhide after processing of machine. The main feature of MBT first layer cowhide: Masai Shoes surface layer are made of grain materials with close woven fibers; MBT Sandals feature smooth feeling, good strength and great abrasive resistance. MBT Medical Shoes lining are composed of suede with thick fiber, big diameter and flocking wool on the surface. Comparing with second layer of Discount MBT Sheos, first layer of MBT Shoes Cheap with more smooth surface and better quality, can be worn longer time. Our website launches MBT Outlet with first layer cowhide, so you can feel free to shop here and get satisfied MBT Shoes Clearance, including MBT Sawa, MBT Sirima, MBT Kaya, MBT Changa and so on .
Welcome to our website: [[http://www.mbtshoesmasai.com]]

mbtshoesmasai (Email) (URL) - 09 August '10 - 02:50

Buy a piece of ghd for yourself. Come and join us [[http://www.ghdiron-outlet.com/]] to win the cheap ghd.

ghd outlet (Email) (URL) - 13 August '10 - 02:57

As we know, now GHD are loved by more and more people, which will save up to 45%.welcome to [[http://www.ghdoutlet-au.com/.]].

ghd straighteners (Email) (URL) - 13 August '10 - 02:58

Come and join us [[http://www.ghdoutlet-uk.com/index.php]] to win the ghd iv styler.

ghd hair straighteners (Email) (URL) - 13 August '10 - 03:00

Your article are great, will visit very soon. there is a site: [[http://www.cheapinsanitydvds.com/]]

buy cheap insanity dvds (Email) (URL) - 14 August '10 - 14:42

[[http://www.avitoipadconverter.com]] AVI to iPad Converter is just the most suitable tool for iPad which let iPad user freely convert various video or audio files to iPad just with simple clicks.
[[http://www.magicdvdtoipad.com]]

AVI to iPad (Email) (URL) - 16 August '10 - 05:26

NFL jerseys wholesale
[[http://www.jerseycaptain.com/index.asp..]]
Cheap jerseys sale
[[http://www.jerseycaptain.com/index.asp..]] with best quality

50th patch jerseys (Email) (URL) - 19 August '10 - 03:26

thank you ,I have learned a lot. [[http://www.lovemypursemall.com]]

designer bags (Email) (URL) - 19 August '10 - 13:28

Cheap UGG Boots sale
Cheap UGG Sheepskin Boots wholesale[[http://www.2010-uggsnowboots.com/index..]]
With best quality and cheap price
2010 ugg boots [[http://www.2010-uggsnowboots.com/index..]]
ugg classic short

2010 ugg boots sale (Email) (URL) - 23 August '10 - 12:24

Love those shoes.When do they go on sale again? and please tell me they will be in size 11… The big foot girls also love choosing christian louboutin too! Because they are so fit you——girls,they will be your best choice!

save fuel (Email) (URL) - 24 August '10 - 15:05

very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff, [[http://www.ucoolstuff.com]] cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you

cool stuff (Email) (URL) - 25 August '10 - 07:05

excellent article , I added you to my [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!

China Wholesale Directory (Email) (URL) - 25 August '10 - 07:08

Once we get there, so many wonderful dreams will come true and the pieces of our black nike air max shoes lives will fit together like a completed jigsaw puzzle. [[http://www.sellnikeairmax.com/]]

air max shoes (URL) - 26 August '10 - 03:16

Pretty good post, this is one of the best posts that I’ve ever seen! This is a great site and I have to congratulate you on the content. It’s so nice Article. I appreciate it.[[http://www.mbtshoppes.com]]

MBT Shoes On Sale (Email) (URL) - 30 August '10 - 03:47

We are traveling by train. Out timberland hiking boots windows, we drink in the passing scene of cars on nearby highways, of children timberland shoe company waving at a crossing, of cattle grazing on a distant timberland boots hillside, of smoke pouring from a power plant, of row upon row of corn and wheat, of flatlands and timberland wheat shoes valleys, of mountains and rolling classic 3 eye timberland boat hillsides, of city skylines and village halls.But uppermost in our black timberland boots minds is the final destination. [[http://www.timberland4you.co.uk/sitema..]]

timberland for you (Email) (URL) - 31 August '10 - 10:01

The town’s poor seem to me often to live the most independent air max classic womens shoes lives of any. May be they are simply great nike air max enough to receive without misgiving. [[http://www.sellnikeairmax.com/]] LIJ

air max shoes (URL) - 01 September '10 - 06:18