My relationship with DTrace has been a rocky and strange sort of round trip. Like most folks, when it first arrived I built up a stash of one-liners, generally to expand upon output from vmstat and the like, such as to show execname for each page out or something. With Brendan Gregg’s DTrace Toolkit I found several fun well written script and used them where appropriate. But frankly, serious DTrace use seemed to me to be akin to mdb use… most people should only bother scratching the surface, only kernel engineers need descend deeper. And, I’m not a kernel engineer.
All that changed when I needed answers to questions about emerging Solaris technologies (SMF, ZFS, COMSTAR, etc.) that were too new to get help from others. I’m not a kernel engineer… but DTrace can make me look like one by pairing up the source code with DTrace stack aggregations. A whole new world opened up and DTrace and I have been best friends since.
I’ve now moved out of the exploratory era and into the sustained monitoring era with DTrace. Not only do I want to run DTrace scripts interactively, not only do I want to run them periodically via some management agent, rather I want to run persistent traces.
Starting a DTrace session that just runs forever presents two initial problems:
- Is the tracing going to suck up resources or let buffers grow incrementally over days or weeks until the trace impacts the running environment?
- How do you collect the data? Pipe to file?
So far, in my experience, the first concern is put to rest. I’ve got several traces running that I keep a regular watch on for any growth… and day after day they remain unchanged, sturdy as a rock. This makes me happy. Please note that I am not using aggregations or the like.
Collecting the data in some rational way seemed more difficult. Sure, I could pipe the output somewhere or use the dtrace -o some/file but that just doesn’t seem very flexible or professional to me. So I came up with a solution I like much better: Syslog!
Syslog is an amazing thing that is all too often under appreciated. Messages are sent to the daemon, which then routes them locally to a file or across the network to another syslog daemon which does the same. Combined with Splunk there isn’t much that Syslog can’t do. Syslog rules. To use syslog with DTrace there are two things to understand:
Firstly, using the Solaris logger command you can write messages to syslog from anywhere. This is super handy to embed in scripts, such as to signify that a backup is complete or some service is restarting or whatever. Here is an example:
root@quadra ~$ logger -p daemon.err 'Nightly Backup Failed!' root@quadra ~$ dmesg | tail -1 Mar 14 01:56:08 quadra root: [ID 702911 daemon.error] Nightly Backup Failed! root@quadra ~$ tail -1 /var/adm/messages Mar 14 01:56:08 quadra root: [ID 702911 daemon.error] Nightly Backup Failed!
Now, DTrace has the beautiful ability to run System Commands from within a clause. The trick is that this is considered a “destructive action”, meaning it actually can modify the running state rather than simply observe it. When enabling a script to envoke destructive actions you should be very careful and trust no one, read the script before running! You can enable these actions on the command line with the -w option or by using the destructive pragma. The system() function in DTrace works much like printf() so variable substitution is easy.
So now lets look at a script that puts these two together. The script here enabled destructive actions so that we can use the system() function and calls out to logger to syslog output:
#!/usr/sbin/dtrace -s
/* Syslogging DTrace Script; benr. */
#pragma D option quiet
#pragma D option destructive
fbt::*_fsync:entry
/ execname != "logger" /
{
self->caller = execname;
self->start = timestamp;
}
fbt::*_fsync:return
/self->start/
{
this->end = timestamp;
this->duration = (this->end - self->start)/1000000;
system("/bin/logger -p local1.info %s by %s in %d ms", probefunc, self->caller, this->duration);
self->start = 0;
}
If we then setup syslog by adding some the following lines to /etc/syslog.conf and restarting syslog…
## Dtrace: local1.info /var/log/dtrace "/etc/syslog.conf" 62 lines, 1902 characters root@quadra ~$ touch /var/log/dtrace root@quadra ~$ svcadm restart system-log
Now we start the script and tail that new logfile:
root@quadra ~$ tail -f /var/log/dtrace Mar 14 02:20:35 quadra root: [ID 702911 local1.info] fop_fsync by ps in 0 ms Mar 14 02:20:35 quadra root: [ID 702911 local1.info] fop_fsync by auditd in 0 ms Mar 14 02:21:08 quadra last message repeated 7805 times Mar 14 02:21:08 quadra root: [ID 702911 local1.info] fop_fsync by nscd in 0 ms Mar 14 02:21:08 quadra last message repeated 3 times Mar 14 02:21:08 quadra root: [ID 702911 local1.info] fop_fsync by auditd in 0 ms Mar 14 02:21:17 quadra last message repeated 2282 times Mar 14 02:21:17 quadra root: [ID 702911 local1.info] fop_fsync by nscd in 0 ms Mar 14 02:21:17 quadra last message repeated 3 times Mar 14 02:21:18 quadra root: [ID 702911 local1.info] fop_fsync by auditd in 0 ms
If you need a refresher in syslog, please refer to my older post: Centralizing Syslog on Solaris.
Please understand! This is ONLY advisable for very low frequency probes! Running a system command from the script is not a lightweight thing and probes can fire a hell of a lot faster than shell commands can be evoked. Even the above example was a bad one, once I realized that auditing is a little overzealous. NEVER EVER EVER do this in a production environment without thoroughly testing it in your test environment. Moreover, if you misuse the system() function, don’t blame DTrace!
Now, this isn’t the only way or even the best way to syslog dtrace. Some other options exist such as piping to logger or even using an SMF wrapper to do so, such as Sam Falkner wrote some time ago. I implemented this way for two reasons: 1) because it works, and 2) because it demonstrates a way to extend your DTrace scripts in new ways using system() commands. My hope is that this excites in your mind some new and interesting uses.
It’s an interesting problem, and I think there are a few different ways of solving both (1) and (2). Your approach seems one that follows the ‘shrink to fit’ idea, it’s small, effective and does exactly what it needs to do and nothing more. I’ve started writing libdtrace(3) consumers for functional testing when prototyping code changes to the kernel. Basically an app that spawns worker threads and watches them with DTrace. You could potentially do something similar here, having a DTrace app that will trace what you want to trace and also periodically observe how that tracing is affecting the system, toning it down if negatively. You’d also be able to syslog without the destructive option, have better resource control of the tracing app, … But clearly not a shrink to fit solution.
Thanks for this, Ben. How about some examples of monitors that you’ve found useful?
Thanks!
Thank you very much!
Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very good！
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.
Good post! Thanks you for your information! China Wholesale Wholesale China Wholesalers Wholesale Game Accessories Wholesale Iphone Accessories Video Game Accessories Wholesale Wholesale Wii Accessories Wholesale Xbox 360 Accessories Wholesale Xbox 360 Games Wholesale Video Games Cheap Video Games Cheap Ps3 Games Cheap Xbox 360 Games Wholesale Computers Wholesale Laptop Computers Wholesale Laptops Discount Computers Cheap Computers Wholesale Iphones Wholesale Iphone Wholesale Iphones 3g Hiphones Wholesale Hiphone Wholesale Hiphones Wholesale Nokia Wholesale Nokia 8800 Wholesale Nokia n97 wholesale blackberry wholesale blackberry phones wholesale blackberry 9700 wholesale blackberry 9600 wholesale blackberry 9500
http://www.buykamagra.com buy kamagra
http://www.viagracialis.com viagra cialis
good post,I think so!
[url=http://www.mbttrainersshoes.com/]MBT trainers shoes[/url]
[url=http://www.mbttrainersshoes.com/]MBT Walking shoes[/url]
nice post here， if you want to know more about best mbt shoes antd&ugg, just click http://www.bestmbtshoes.com
http://www.uggbootsroom.com
Your article is very useful!
Welcome to http://www.mbttrainersshoes.com.
mbt trainers shoes are physiological walking shoes. Masai Barefoot Technology，or MBT for short and their manufacturer boasts great benefits of wearing and using MBT walking shoes properly
Wearing MBT trainers shoes can activate muscle groups needed to coordinate and stabilize the body both when walking and standing. The muscular system is exercised all the way up to the neck, stretched and relaxed again,
the body balanced – with every step. Below you will find much benefits of MBT walking Shoes.
MBT SHOES on Sale up to 50% off and you can get free gifts mbt on sale gone through a popular and unpopular fashion cycle, and now it is again leading the forefront role of fashion. You may surprisely found that mbt outlet have become the most comfortable and never out-of-date reserved single products
casual dresses shoes suppliers
gp car
designer handbags
http://www.buynikeair.com | Nike Air | Nike Air Shoes | Buy Nike Air | http://www.buynikeair.com
People in the world take the home loans from various banks, because this is easy and fast.
Your site is a refreshing change from the majority of sites I have visited. When I first started visiting web sites I was excited by the potential of the internet as a resource and was very disappointed initially. You have restored my enthusiasm and I thank you for your efforts to share your insights and help the world become a better place.
ugg outlet
[url=http://www.ugg-outlet-store.com]ugg outlet[/url]
ugg outlet http://www.ugg-outlet-store.com
rc toys
i would say it is wonderful, thanks!
welcome to buy mbt healthy shoes
GHD hair straightener and chi flat iron , welcome to buy
mbt shoes in stock
rc toys your hobbies ,welcome to order .big discount
coach handbags
The Puma has been one of the most trusted names worldwide. Puma shoes sale on http://www.pumashoesale.com Offer the perfect Puma Shoes with Cheap Price and Top Quality Guarrantee.
puma shoes men on sales
thank you very much !!
Welcome to visit our MBT shoes website:
http://www.ecalibabashop.com
Hello everyone.
In our online shop,you can purchase different designer mbt shoes, and we offer the commodity are high in quality and reasonable at price now.
These are our mbt shoes on sale.
Shop’s homepage http:\www.mbtshoeshop-online.com
mbt women
mbt men
thank you
welcome to buy mbt shoes, .cheaper mbt shoes on sale,discount mbt shoes,worldwide mbt shoes,free shipping,buy mbt kaya,largest selection,high quality http://www.mbtshoes2sale.com/products/?MBT-Sport-Shoes-c115_p1.html
welcome to our site and choose the best handbag, we offer the best service, fashion and vrious styles for choosing,high quality,free shipping. http://www.b2chandbag.com.
Hello everyone.
In our online shop, you can purchase different designer mbt shoes, and we offer the commodities are high in quality and reasonable at price now.
These are our mbt shoes on sale.
Shop’s homepage http:\www.mbtshoeshop-online.com
mbt women
mbt men
i agree with you
welcome to buy mbt healthy shoes
rc toys your hobbies ,welcome to order .big discount
mbt shoes in stock
and Space Administration and the supra cuban United States National Seismographic Network, with real supras shoes the epicenter being right in Nanjing.
A 25-year-old woman, who wholesale supra sneaker spread the rumor after discount supra suprano hi reading it on a friend’s QQ zone,
a social networking site, was discount supra skytop arrested for creating public disturbance.
great post
http://www.sellnikesbs.com
I like oakley sunglasses wholesale
GHD straighteners was kmown as ghd flat iron, which was authorized online seller provides all kinds of hair straighteners,pink ghd,purple ghd,babyliss. By visiting ghd australia , you will find what you want and made yourself more beautiful.If you miss it ,you miss beauty.Buy a piece of ghd for yourself. Come and join us http://www.ghdoutlet-au.com/ to win the cheap ghd.
discount watches
http://www.towatches.com/Discount-Watches/
for best web online store http://www.salepumashoes.com
MBT Huge Selection. Shop Visually. MBT Shoes no tax. Low price guarantee,MBT Habari Easily achieve your fitness goals. http://www.mbts-mbtshoes.com/
We launched the 2010 latest and most fashionable True Religion jeans on our shop, you are welcome to patronize.Here are many discounts for you.
Our aim is to provide high-quality products and excellent after-sales service.
http://www.jeanushop.com
Welcome to Moncler Jackets online shop.The 2010 latest and most fashionable Jackets,save up to 60% off. 7 days delivery to worldwide with free shipping.
Our aim is to provide customers high-quality products and excellent after-sales service.
http://www.monclerustore.com
Air Jordan Air Jordan
A good idea! http://www.offerreplicawatches.com.They sell Chanel Watches, Replica BRM Watches,offer Cartier Watches, Copy Ebel Watches, Replica Hublot Watches, Fake Tag Heuer Watches, wholesale Richard Mille Watches.
They also sell some very fashion watches, such as Montblanc Watches for sale,Chopard Watches, Dior Watches and Gucci Watches. Rolex watches are hot sell. If u are Armani Watches and Breitling Watches fan. You can still look at Omega Watches and Longines Watches, many new arrivals watches just come here.You can have a try on 58338312391273913. :[url=http://www.offerreplicawatches.com]www.offerreplicawatches.com[/url].
written by shoes-mbt, May 13, 2010
Wearing MBT shoes provides many pro-body benefits: this is the result of an increasing number of international studies supporting the numerous health benefits of the “anti-shoe “. Here is a excited place—–discount MBT shoes http://www.shoesmbt.org
Dell inspiron 9400 Battery http://www.adapterlist.com/dell/inspiron-9400.htm
They also sell some very fashion watches, such as Montblanc Watches [url=http://www.mydiscountjordanshoes.com/air-jordan-xv-c-79.html] air-jordan-xv
[/url]
MBT Maliza Shoes or MBT Tembea Shoes.
A good MBT shoes shop online- http://www.trade161.com , cheap MBT shoes, Special Sale Offering,Real Brand,Authentic Quality & Package,Free Shipping 1 Week To Your Door!
With the MBT shoes fitness shoes purple unique structure, the wearer can be in a naturally stable state, but through the balance of movement, that is by increasing muscle activity, be eliminated.
MBT shoes classify
Dear friends, welcome to bagonhand.com give my sincere recommendations.
http://www.ebaysource.comNFL jerseys
http://www.china-jerseys.com wholesale jerseys
http://www.china-jerseys.com/products/NFL-Football-Jerseys-c3_p1.html Wholesale nfl jerseys
[http://www.china-jerseys.com/products/..]] NBA Jerseys Wholesale
every good i
http://www.ed-hardy.cc ed hardy
http://www.ed-hardy.cc swimsuit trends 2010
MBT shoes had been popular all over the world,about 100 million pairs needed every year. MBT shoes has many kinds,suah as MBT Changa shoes, MBT Chapa GTX, MBT Fanaka GTX ，MBT Lami shoes, MBT M.WALK, MBTShuguli , MBT Sports shoes, MBT Tariki , MBT Habrai Sandals , MBT Barabara Shoes , MBT Denim Shoes ,MBT Ema Shoes , MBT Karani shoes , MBT Kaya shoes , MBT Kasumu shoes , MBT Panda shoes , MBT Salama MBT Tagata shoes , MBT Tembea shoes , MBT Tunisha shoes , MBT Voi . All MBt shoes are good for your health.It is your best choice.For more information,pls visit http://www.trade161.com
jordan retro iii jordan retro iii jordan retro iii
http://www.mydiscountairjordanshoes.com/
jordan retro iii [url="http://www.mydiscountjordanshoes.com/air-jordan-iii-c-67.html" title="jordan retro iii"]jordan retro iii[/url] jordan retro iii
i love ed hardy design
http://www.godlovembt.com
Thank you for visiting http://www.godloveugg.com/
Air max http://www.cheap-shoesonline.com/nike-air-max-c-115.html even
to grandfathered
discount Mbt shoes can be found at our web, Mbtsport black and Mbt men’s shoes are good choices.
http://www.discountmbt.com/
http://www.discountmbt.com/mbt-womens-shoes-c-7.html
NFL jerseys new arrival coming and football jerseys are all available,safe and fast delivery,cheap jerseys are in stock.Welcome to purchase http://www.mvpjersey.com/
http://www.sunglasses1987.com wholesale sunglasses
http://www.sunglasses1987.com wholesale replica sunglasses
Do you want to become the beautiful scenery this summer? or do you want to attract eyeball among the crowd?now,listen carefully,your wishes will come true here http://www.ed-hardy.cc/ .we support all different kinds of beautiful leisure production,such as sunglasses,t-shiirts and swim trunks et.from here,you will find your favorite things.What are you waiting for?go and choose the now!
I totally love this article.
http://www.discounteasytone.com/
reebok easytone shoes create natural instability. reebok easy tone increases muscle activity. discount easy tone shoes will be available
http://www.coach-handbags-outlet.com/
ed-hardy.cc is one of the largest shoe retailers on the internet. The company focuses on all types of footwear and related apparel. The company partners itself with over 850 manufacturers in order to over 700,000 products from some of today’s most well known brands. A consumer can find thousands of shoes and styles for any occasions or events that they need to attend. The website divides all its products under its women, men and kids shoe department.
ed hardy
[url=http://www.ed-hardy.cc/ed-hardy-clothing.html]ed hardy clothing[/url]
http://www.ed-hardy.cc/ed-hardy-shoes.html ed hardy shoes
[url=http://www.nfljerseysbiz.com]nba jerseys wholesale from china[/url]
As we all know, silly bandz now is the best selling goods in every store all over the US country, and also selling well through network.
http://www.cheapjerseyschina.com
http://www.jerseyscloset.com
I know you probably get a lot of comments like this Air Jordan 2010, but just wanted to let you know that I really appreciate the work you have put into the blog. I was wondering if I could put a link on my blog because I am sure my followers would love to read it Air jordan 1. Let me know.
Couldn’t be written any better. Reading this post reminds me of my old room mate! He always kept talking about this Michael Jordan Shoes. I will forward this article to him. Pretty sure he will have a good read. Thanks for sharing!Welcome First time jumped here on your site, found on Bing. It really helps to have sombody to talk to other then my divorced parents Cheap Jordan Shoes. Thanks again.
I really liked reading your post!. Quallity content. With such a valuable blog i believe you deserve to be ranking even higher in the search engines
. Check out the link in my name. That links to a tool that really helped me rank high in google. This way even more people can enjoy your posts and nothing beats a big audiance 
http://www.cheapjerseyschina.comhttp://www.cheapjerseyschina.com
Mbt zapatosMe encerré, dejando sólo un balcónMbt zapatos
Abrir la puerta cuando se enfrenta el aturdimiento noche oscura
Mbt zapatosMira a escena tras escena pasadoMbt zapatos
Le muestra mi amor otra lMbt zapatosvez
Your article are great, will visit very soon. there is a site: http://www.lovemypursemall.com
excellent article , I added you to my http://www.china-wholesale-directory.com Top China Wholesalers category.. thanks for sharing the article!
very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for http://www.ucoolstuff.com cool stuff http://www.ucoolstuff.com cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you
百度
[url=http://www.sina.com]sina[/url]
ugg bailey boots
gey classsic uggswansan15zy
This is totally awesome. Thanks!
Best new era caps, new era hats wholesale, Monster Energy Hats
,Dc Shoes Hats,red bull hats,New Era Caps,NFL Hats ,Famous Hats ,new era 59fifty hats,jordan hats and coogi hats,etc are in stock now. Our site provide first-class service and reliable quanlity garantee,do not hesitate to shake hands with us and go with the tide as soon as possible!
Dear friends, thank you for visiting our website ,we are an international trade company,which specializes in http://www.hatmvp.com.We http://www.hatmvp.com rel=”nofollow”>red bull hats at competitive price,providing a huge range of hats of different teams,such as Arizona Cardinal,Atlanda Falcons ,Baltimore Ravens,etc.You can buy cheap caps. Welcome to visist here.
The largest New era hats store online. from here you can find the newest monster energy hats,new era caps,dc shoes hats and other famous brand hats in cheap price
I hope this allowed, I have never used this website before so I wasn’t really sure what this was going to do. So this is just a test post. I really like this forum, it has some excellent discussions that take place
Best new era caps ,new era hats, delicate monster energy hats, magical nfl hats, one industries hats, rockstar energy hats, Red Bull Caps, The Hundreds Hats, Supreme Hats, DC Comics new era caps
are in stock now. Our site provide first-class service and reliable quanlity garantee, do not hesitate to shake hands with us and go with the tide as soon as possible!