Monitoring with Zabbix

17 Dec '07 - 22:15 by benr

With a new site comes new opportunity. When Joyent recently added a new facility I decided to look at new monitoring solutions. We previous were standardized on Sun X4100, Sun T1000's, and Sun X4500's with a network using Force10 and F5. This new site would replace the X4100's with Sun X4150's and Dell 2950's. Having largely avoided Dell in the past I took my time to get to know the system well and knowing that IPMI is far more utilized in that arena than with most Sun systems I was given the oppertunity to get a whole new appreciation for IPMI 2.0. Armed with this new knowledge I wanted to take our monitoring much further than we had in the past to monitoring individual BMC sensors via IPMI. This meant looking at monitoring solutions in a much deeper way.

Zabbix quickly rose to the top of my list. In the initial phase I grabbed Zabbix and Zenoss and planned for a face off. Zabbix compiles nicely and easily on Solaris/X86 and within about 30 minutes I had a server up and I created an agent tarball complete with installation script so that deploying an agent was as easy as a wget, untar, and "install.sh". Zenoss, however, gave me problems, dealing with Python isn't my strong point and several dependancies were required, and then I had problem after problem getting things to build properly... after about an hour I decided that as pretty as Zenoss looks in those screenshots it was out of the running for the time being.

Zabbix isn't pretty... I'll be honest. If you want a "tactical view" to put on an overhead projector in your NOC, Zabbix won't give you that Hollywood movie feel. Zabbix does make up for that short coming in raw power. Let me explain...

Zabbix is agent based, you can choose to avoid this but you'll loose most of what makes Zabbix so great. Is implemented in C and easily portable, so you don't need to worry about having Java or Python on all the monitored hosts like many of its competitors. It adds to this the standard assortment of SNMP, custom script, and external checks (icmp, ftp, etc) you expect. Where it adds something really interesting is its WEB (why they put this in all caps I don't know) monitoring capabilities which allow you to do more than just fetch a page but actually supply "steps" such as logging into a site and navigating around, to which response times are stored allowing you to alert someone if the login process takes more than 5 seconds or something. Very handy indeed.

Zabbix agents and other checks associate data with "keys", these keys are then bound to "items" that describe the data and define how often to update the given data. These items are then associated with various alert conditions called "triggers". For instance, the agent by default returns the number of users connected with the key "system.users.num", and the stock template associates the "Number of users connected" item with that key which is polled every 60 seconds. So now we can create any number of different alert conditions based on the number of users logged in by creating triggers. One default trigger is "Too may users connected on server {HOSTNAME}" (where HOSTNAME is replaced ultimately with the appropriate host), which uses the following expression/condition:

{Template_Joyent:system.users.num.last(0)}>50

In this case, the trigger is associated with my "Template_Joyent". The function "last(0)" (meaning the last value you polled) is applied to the key "system.users.num", and if the value is greater than 50 the condition is true. In Zabbix all conditions should evaluate to false when things are fine. Each trigger has a Severity associated with it, in this case "Average". So the result here is that any user configured to get alerts for Average or higher severity will get a notification when more than 50 users are logged in.

Zabbix provides a rich set of functions by which to create your triggers, such as average change over time, min and max, absolute difference, etc. This means that I could, for instance, create a trigger that alerted me if since the last polling interval more than 10 users logged in or if the average number of users over the course of an hour exceeded 50.

Where this becomes really powerful is when you choose to extend your agents. In each agents configuration file you can supply a "UserParameter" directive which runs some command and returns it with a given key. Here are some simple examples:

# SMF
UserParameter=smf.online,svcs -a | grep online | wc -l
UserParameter=smf.offline,svcs -a | grep offline | wc -l
UserParameter=smf.maintance,svcs -a | grep maint | wc -l
# X4150 IPMI (BMC Direct)
UserParameter=ipmi.amb,/usr/sbin/ipmitool sensor reading "Ambient Temp0" | cut -f2 -d|

UserParameter takes two arguments which are comma delimated: the key name and the command to run. In the above, I can return the number of "online" SMF Services by grepping out of "svcs -a" and then returning that as "smf.online". Restart the agent, go back to the server and add a new Item for this key and start creating triggers. Now I can be alerted if, for instance, the number of online services decreases by some number in a given time. The IPMI example there provides a workaround for environments where you may not have access to the management port of your server and instead want to return IPMI data directly from the OS using 'ipmitool'.

These examples above are simplistic, you can get more advanced by allowing the server to pass arguments. For example, if you want to monitor disk ops, you probably don't want to add a separate key for each disk, thus you could specify the disk name back on the server which is passed to your agent as an argument.

These abilities remove a lot of the cruft and put the power into your hands. If you have a highly customized environment Zabbix is a great choice.

However, in Zabbix 1.4 there is a lot more that is needed. Currently features like escalations are planned but not expected till Zabbix 1.6. Without features like limiting repeat pages your forced to get your triggers properly defined rather than masking away false-positives in your alerting policy, such as the way Nagios handles "flapping".

The "Overview" (all in one view of the world) page is something that takes some getting used to. Rather than a pretty page in black with "0 Services Down, 0 Servers Down" Nagios style page you instead get a list of defined triggers and a column for each monitored host which is color coded. If the box is green, life is good. If the box is red, life isn't. Several nifty things like flashing green if the trigger is fine now but wasn't less than 15 minutes ago are handy but sometimes annoying especially during testing.

But, and this is a big but, Zabbix does give you something exceedingly useful... all the monitored keys can be graphed. Click on the color block of any monitored trigger and you can view its value history as a graph. This means that you can take add-on graphing applications like Cacti or MRTG and roll the functionality directly into Zabbix. Want to know what load average has looked like for the last week? No problem. And, add to this the ability to custom create graphs which can combine multiple keys into a single view.

If you see a screenshot with pretty graphs all over it on the Zabbix front page, thats a "Screen". A screen is a page custom layed out with several custom graphs. So you might want a graph that has the memory usage and load average of every system on your network, you can put those together in a custom graph and then place that on your "Screen". If you take some time and create some nifty graphs like this you may find ourself looking more at your screen than at the Overview page.

Isn't as pretty as Zenoss or as streamlined as Nagios, but it really is a SysAdmins tool. It doesn't hide functions away from you or gloss over details to make you feel nurtured, its all out there gritty and raw for you to use an manipulate. There is a nature learning curve in wrapping your head around the concepts of "items" and "triggers" and how you can combine them in really powerful ways, but before long you'll be frustrated by the limits of other solutions. Having a full featured and easily extensible agent really is my favorite aspect and frees you from the concerns that come with having to pass around SSH Keys required to make external scripts work with other solutions.

That said, if you decide to implement Zabbix expect to spend some time crafting triggers to suite your needs. If you want to get beyond the basics you'll need get your hands dirty, which is easy to do after playing with it a bit, but it you want something you can just deploy and forget consider a commercial tool like Uptime.

I'm only grazing the basics here. If you want to learn more check out Zabbix.com.

UPDATE: Screenshots added per comment request.


- - C O M M E N T S - -

Take a look at opsview. It is a monitoring solution adding HA features like distributed checks and a decent reporting interface to Nagios.

[[http://www.opsview.org]]

Jeff Schroeder (Email) (URL) - 18 December '07 - 01:50

Nice overview. I look forward to more in the series (can we get
screenshots next time ;)

John Levon (URL) - 18 December '07 - 01:59

Zabbix looks pretty nice. I am in the process of installing OpenNMS to test it out, but I may also install Zabbix over the holiday break and have a shootout.

Mark McCoy (Email) - 19 December '07 - 16:05

OT: from what you’ve seen of Dell’s latest server offerings, from a purely technical perspective, do you still prefer Sun’s hardware?

[sorry if this posts twice, i tried to use the ‘Preview Comment’ button but it did not work as i expected and i was left wondering if my comment was eaten.]

David M. Besonen (Email) - 20 December '07 - 14:44

Thanks for your review!
Do you using Zabbix daily? Did you evaluate OpenNMS?

Vitaly (Email) - 12 May '08 - 13:40

it’s nice site!!! order paxil
%PP

lola (Email) (URL) - 04 June '08 - 04:11

f9C3Ug 1800flowers [[http://1800flowers.bloges.org/]]
[link][[http://1800flowers.bloges.org/[/link]]]]

1800 (Email) (URL) - 30 June '08 - 07:35

xbxeemhud [[http://1800flowers.bloges.org/]] ijhwhxtb [link][[http://1800flowers.bloges.org/[/link]]]]

f1800 (Email) (URL) - 01 July '08 - 17:53

7o1h1N Omrfstyidvrjbez tfiifxuxclputu [[http://1800flowers.bloges.org/]] cwpafqlu [link][[http://1800flowers.bloges.org/[/link]]]]

1800f (Email) (URL) - 01 July '08 - 23:31

WMFkHH vcxdsgcvb363454gdfgdbcv53435654

bvbcvfdg (Email) (URL) - 03 July '08 - 14:38

doors.txt;7;10

ItzIuZVOEspN - 26 July '08 - 03:22

it’s nice site diazepam pills 24102 ephedrine =O

john (Email) (URL) - 08 August '08 - 11:21

advancement boy ceremony rank scout
advancement association professional researcher

advancement (Email) (URL) - 11 August '08 - 15:59

, sneha nude actress, ysuc, endras u-boats, =-), free abbywinter, 5378, disadvantages of genetically modified food, 37936, sofiavergaradesnuda, 766, ls models forums, clbcj, jesse jane wallpaper, >:-D, bbs models, :[[[, sleep assoult, 875552,

ice assparade (Email) (URL) - 11 October '08 - 14:30

, custom figure, 936, free passwords from raven riley, 356, mark dalton zeb atlas video, yalx, ls teen models, 8-[, adding and subtracting intergers worksheets, yabo, akira lane gallery, 2624, ann angel free pics, :-PPP, shotgun gauge, npzci, free watching pirates, wgb,

christinamodel (Email) (URL) - 11 October '08 - 14:30

, beautiful agony free online, 41721, sunny leonne, mxbmo, emily procter in bikini, vtb, srilankan nudes, %DD, abby winters password, 40272, abbeys winters free, >:DD, farang ding dong girls free, 6543, sabrinasabroc, zbg, ls modrls, =PPP,

little virgins picture (Email) (URL) - 12 October '08 - 09:47

, com/winx-club.html, 0872, how many zero in billion, 8690, mickie james leg magazine, 439133, abbywint, :O, ice la foxx, izdxew, club winx hentai, 8[, emily procter, 25833, samantha anderson, >:-[, farang ding dong free video, =D,

brooke skye in the vip forum (Email) (URL) - 13 October '08 - 02:20

, abby winters website password, idv, samantha 38g anderson, =-PPP, abbywint, axyaj, guinevere and carli, arzgho, pass ravenriley, =))), 0 in billion, 38447, slepp assult, 7584, disavantage of technologe, xjalpu, farang ding dong free video, hei,

kari byron photos (Email) (URL) - 13 October '08 - 02:20

, free phat white booty, 220638, tamil pdf stories, wfxjm, how to hide ads on myspace, ldvr, winx igre, 8))), light brown booty, 9754, free ann angel movie, 8DD, locke, >:-((, beautiful agony google, plenqo, zebatlas, 607599,

watch pirates jesse jane free (Email) (URL) - 13 October '08 - 04:22

, free beautiful agony, sdwge, sabrina sabrok nude, 1011, free sleep assault.com, ruy, how to calculate salary in canada, 8-], nude sneha, euf, mickie james roast beef, =D, korean actress eugene, oaadc, ls model, ebpr, password abby winters, bbpi,

watch pirates online free jesse jane (Email) (URL) - 13 October '08 - 06:25

, hientai winx, %-OOO, samantha anderson 38g, 8D, cody lane movies, yoz, free assault videos, kolyk, candy manson pov, 0238, free videos sleep assault, 4005, ashley brooke dildo, upkd, winx pobarvanke, 8))), abby winter free, :-))),

ashley brookes video (Email) (URL) - 13 October '08 - 10:31

, breast reduction blogs, 8-D, eden mor video, cjdd, christina model nude, hbduha, carmella bing interracial, 912, nude pics of sabrina sabrok, 8-PPP, marilyn lange, ecw, christy hemme gallery playboy, 154, cody lane violent, %[[[, escort carmella bing, 4335,

carmella bing free vids (Email) (URL) - 13 October '08 - 14:38

, video sleep assault gratis, 8(, brookeskye hardcore, 8-OO, adding and subtracting integers worksheets problems, =-P, miko lee fans, 689, chelsea charms videos, vizxsd, winx club, =-DDD, mickie james leg show, :((, ls models forum, =-PPP, sleep assault free video, gzot,

kari byron nude pic (Email) (URL) - 13 October '08 - 16:45

, joc porno jena, ikh, assault video, 332, tamil actress topless, visyap, free full sleepassault videos, eayp, multiplying subtracting and adding integers, vhgf, mickie james in leg show magazine, %-((, free sleep assult video, >:-)), sabrina sabrok en playboy, 155382, gianna michaels carmela bing shyla, qqzf,

kristina fey feet (Email) (URL) - 13 October '08 - 22:54

, multiplying subtracting and adding integers, 0596, sleep assault member pass, =DDD, mickie james in leg show magazine, lckh, jesse jane free, vyips, shyla stylez and carmella bing and gianna michaels, nyxl, subtracting integers worksheet, brvwa, tamilstories, 2972, free ann angel vdeos, =(((, christinamodel october sets, zfqjc,

jesse jane free (Email) (URL) - 14 October '08 - 00:58

, free pictures of danni ashe, 8-[[[, free danni ashe nude galleries, kfwh, free daphne rosen clips, ymdx, free christine young videos, 8-(, free miko lee pictures, aasel, free pictures of cody linley, 3046, free squirting carly, =-PPP, freeones casey parker, 89217, free videos of dani woodward, 083260,

freeones casey parker (Email) (URL) - 15 October '08 - 09:50

, hannah montana i got nerve, >:-]]], hannah montana as i am, dua, hannah montana pics, svlgx, hannah montana concert tickets az, zefz, goldie blair vs tanya danielle, 31803, hannah montana wig, =-))), hannah montana backstage passes, pjkkpc, hannah montana who said, =[[, hannah montana halloween costume, =-(((,

hannah montana cake (Email) (URL) - 15 October '08 - 11:54

, hannah montana nobodys perfect, hqnax, hannah montana concert schedule, irscpv, hannah montana best of both worlds, 612, gallery and chrissy moran, jtsp, gianna michaels carmella bing, ewl, hannah montana theme song, 731384, hannah montana party, zkh, hannah montana halloween costumes, >:-[[, hannah montana one in a million, %((,

hannah montana who said (Email) (URL) - 15 October '08 - 11:54

May be you prefer [url=[[http://coloradoski.fortunecity.com/bea..]] creek ski resort colorado[/url] ?
Seeking this [url=[[http://members.lycos.co.uk/coppermount..]] mountain copper co[/url] .

pexNetereld (Email) - 11 November '08 - 01:55

The WEB ‘steps’ really need to make use of a ‘cookiejar’, otherwise ‘steps’ doesnt do anything for you at all than just group web multiple requests under a single view.

dingo (Email) - 21 October '09 - 00:58

also your comment “Isn’t as pretty as Zenoss or as streamlined as Nagios”—came back to haunt me this morning—I had to look up the definition of streamlined. No you must have made an editorial error—Nagios? Streamlined?

dingo (Email) - 21 October '09 - 12:22

[[http://www.saleveling.com]]
[[http://www.power-leveling-game.com]]

wow power leveling (Email) (URL) - 23 October '09 - 07:03

Great post! Hope to be better. Better means more features.
good post,I think so!

Bridesmaid Gowns (Email) (URL) - 06 January '10 - 06:20

[[http://www.buykamagra.com]] buy kamagra
[[http://www.viagracialis.com]] viagra cialis

M65 Jacket (Email) (URL) - 21 January '10 - 04:30

mbt shoes – [[http://www.thesuitshoes.com]] – mbt shoes

www.thesuitshoes.com (Email) (URL) - 05 May '10 - 03:13

cheapest mbt shoes [[http://www.mbtag.com]] mbt shoes on sale

www.mbtag.com (Email) (URL) - 05 May '10 - 03:17

P90x .It really is not expensive if you factor in the cost

of a gym membership,P90x workout . The cost for P90X is

about three months of a paid gym membership but you get to

keep the program foreverP90x . You can try many of the

online sites, but it will be the same as buying from the

company or a Beachbody Coach. Make sure you are getting

original DVD’s. People are selling copies all over. The

problem is how long will they last, P90x workout ,and you

truly need the exercise and nutrition guide to even follow

the program. You can go to any site or you can go to

[[http://www.p90xmall.com]] and click on products. P90x dvd You can

order directly from the site,P90x dvd.

p90x (Email) (URL) - 05 May '10 - 05:54

A good website recommend you:www.offerreplicawatches.com.They sell Chanel Watches, Replica BRM Watches,offer Cartier Watches, Copy Ebel Watches, Replica Hublot Watches, Fake Tag Heuer Watches, wholesale Richard Mille Watches.

They also sell some very fashion watches, such as Montblanc Watches for sale,Chopard Watches, Dior Watches and Gucci Watches. Rolex watches are hot sell. If u are Armani Watches and Breitling Watches fan. You can still look at Omega Watches and Longines Watches, many new arrivals watches just come here.You can have a try on 58338312391273913. :[url=[[http://www.offerreplicawatches.com]www..]]].

Replica Chanel Watches,Dvd Boxes (Email) (URL) - 12 May '10 - 07:22

Toshiba pa3421u-1brs Battery [[http://www.adapterlist.com/toshiba/pa3..]]

laptop battery (Email) (URL) - 14 May '10 - 09:34

Apple a1185 black Battery [[http://www.globallaptopbattery.co.uk/a..]]

laptop batteries (Email) (URL) - 14 May '10 - 09:48

I currently use zabbix to monitor a really large environment (more than 320 servers)

I’ve founded a wonderful plugin that is more than a plugin and the others monitoring systems don’t have nothing of similar, and nothing that go inside oracle so deeply.

In the hope that someone found useful my comment

[[http://www.smartmarmot.com]]

here you are going to find Orabbix opensource and released under GPL3

SmartMarmot (Email) (URL) - 15 May '10 - 11:03

Supra Shoes are some of the art, fatally if you buy a brace concider it an investment because you can bet they position of the legendary sales of organ’s only jackets that went legendary when we kids.
The mission of Radii footwear is to create unique designs that reflect each individual’s personality, and the Radii 420 Top definitely achieves that.

Supra (Email) (URL) - 18 May '10 - 02:05

Buy Nike Air Max 90 Shoes just $45-55 USD in[[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

nike air max 90 shoes (Email) (URL) - 28 May '10 - 02:22

Buy Nike Air Max 90 Shoes just $45-55 USD in [[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

nike air max 90 shoes (Email) (URL) - 28 May '10 - 05:36

[[http://www.airjordanshoescheap.com/]]

air jordan shoes (Email) (URL) - 02 June '10 - 02:13

[[http://www.mbtshoeslatest.com]]
[[http://www.nikeairmaxshoe.com]]

Nike air max shoe (Email) (URL) - 24 June '10 - 06:56

Do you like the ugg ? Ugg Classic Short Boots are UGG Australia’s original heritage styles. Ugg Classic Tall will keep your feet dry and the ultimate comfortable.
[[http://www.ugg-boots-london.com/ugg-cl..]]

Ugg Classic Short (Email) (URL) - 07 July '10 - 01:25

The athletic shoes which makes using this technology may the very good local constable convoy mobilization body, Air Max 2009.
[[http://www.allhotshoes.com/]]

air max shoes (URL) - 09 July '10 - 05:20

Hey!!!
I was unaware about Zabbix but know I had enough in sequence on it. Thanks for filling me up with great information. I would like to get in row on such topics again.
flowers

flowers (Email) (URL) - 14 August '10 - 09:05

This is realy a good post great work well don.

valentines flowers (Email) (URL) - 19 August '10 - 07:29

i like your work and thinking.
mothers day flowers

mothers day flowers (Email) (URL) - 19 August '10 - 10:24

[[http://www.pdftoimageconverter.com]] PDF to IMAGE Converter with reliable quality and humanized design is your ideal helper, which can protect U from having troubles in converting pdf to image! Unimaginable functions will not let U down forever!

PDF to IMAGE Converter (Email) (URL) - 23 August '10 - 01:51

excellent article , I added you to my [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!

China Wholesale Directory (Email) (URL) - 25 August '10 - 07:45

very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff [[http://www.ucoolstuff.com]] cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you

cool stuff (Email) (URL) - 25 August '10 - 08:05

[[http://www.ripblurayformac.net/blu-ray..]] Blu-ray to iTunes Mac supports output formats compatiable with iTunes including: MP4, MOV, M4V, iPod Videos, iPhone Videos, iPad Video and Apple TV Video, after the conversion, you can transfer the converted Blu-ray DVD movies to iTunes freely on Mac OS X.

[[http://www.ripblurayformac.net/blu-ray..]] Blu-ray to iTunes can quickly convert Blu-ray DVD (.m2ts) files and DVD (.vob) movies and ISO/IFO images to iTunes compatiable format with just a few clicks, and It is more than a Blu-Ray to iTunes Converter and can help you to remove Blu-Ray AACS encryption, region codes and BD+ copy protection, and transfer Blu-ray to iTunes movie with fast conversion speed and high quality.

Blu-ray to iTunes Mac (Email) (URL) - 01 September '10 - 01:50

Top Online Stores is a SEO Friendly [[http://www.toponlinestores.org]] free directory where you can find the best online shopping stores selected by hand and sorted by category.

link directory (Email) (URL) - 14 September '10 - 07:10

Not even as I shook womens ugg boots and trembled uncontrollably in the trenches, did I forget your ugg boots face. I would sit huddled into the wet mud, terrified, as the ugg australia hails of bullets and mortars crashed down around me. I would clutch my uggs bailey button rifle tightly to my heart, and think again of that very first day we met. I would cry out in bailey button boots fear, as the noise of war beat down around me. But, as I thought ugg boots in uk of you and saw you smiling back at me, everything ugg boots usa around me would be become silent, and I would be with you again for a few precious official ugg boots moments, far from the death and destruction. It would not be ugg classic cardy until I opened my eyes once again, that I would see and hear the carnage of the war around me.I cannot tell you ugg classic cardy boots how strong my love for you was back then, when I returned to you on leave in the September, feeling uggs classic cardy battered, bruised and fragile. We held each other so tight I thought ugg uk we would burst. I asked you to marry me the very same uggs day and I whooped with joy when you looked deep into my ugg boots shop eyes and said “yes” to being my bride. [[http://www.uggsbailey.co.uk/sitemap.ht..]]
YQ

ugg australia (Email) (URL) - 18 September '10 - 02:24

I’m glad I found your blog. I’m brand new with Team Beachbody and I’m looking to network with as many coaches as possible.

I found this site from my upline coach at [[http://MyRecruitingSystem.com]] guy who runs that site recruited over 100 coaches his first month. Him and his wife took the top spot with the chairmans challenge that Beachbody was putting on and Carl is flying to their home to interview them.

He is allowing a few people outside of his team to use his system and it is awesome. I got the system and ad co-op and recruited 3 new coaches my first week. I also recruited my sister but I don’t think she is going to do much.

Talk with everyone later

♥ Linda in Cali ♥

Linda Hughes (Email) (URL) - 25 September '10 - 11:37

Very interesting article, good job. A question about Zabbix: Is it able to receive and present SNMP traps coming from network devices?

Pablo (Email) (URL) - 30 November '10 - 14:46

Personal information





Remember your information?
Comment

Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.


^M