Crashing is the wrong title actually. We’re talking about panics. Its sort of like saying “hacking” when you mean “cracking”. A “crash” is when an OS preforms some operation that typically causes the system to reboot. Solaris is very unique from rival Linux in that 99% of the time such an event will be caught by the OS and handled as a “panic” instead of an uncontrolled crash.
While its not a sexy feature of Solaris, panics are extremely important things. The reality is that sh*t happens, it just does. When it does happen you want to collect as much information about the event as possible to fix the problem that resulted in badness. Therefore, the advantage of a panic over a crash is a crash dump which can be analyzed for the cause. This is why panics are a very good thing indeed. As I like to say, “If you’ve gotta crash, fine, but you’d better give me a reason!”
This blog entry is interested in the 1% of Solaris issues that don’t result in a panic, but you wish would. In between the concepts of a “crash” and a “panic” is that dreaded situation we call “hung”. A “hung” or “locked” or “wedged” system is one that is still running, technically, but is otherwise unusable. Typically this is the result of the kernel being ok but the userland stack being trashed beyond repair. Perhaps the two most common causes of this are abuse cases such as memory/swap exhaustion or fork bombs, where the kernel is still running but processes can’t spawn to let you see whats happening. The normal way to deal with this is to reboot the system, either via IPMI or sending some poor soul (typically, you) down to the data center to unceremoniously press the big button.
So here’s the problem with rebooting a system in a bad way…. it doesn’t panic, meaning you might get the system back up post-reboot but you have no idea what happened unless something in the logs tips you off. If there are no relevant logs entries you have to simply shrug and pray that it doesn’t happen again. So what can you do about it? On SPARC systems you’d hit “STOP-A” and sync the box. But, what about Solaris/X86…. what do you do then?
Thankfully there are three very handy ways of dealing with such situations. They can be combined or used individually. Lets take them in turn.
Please note! This entry applies only to Solaris/X86!
Panic on NMI
Intel introduced a concept of Non-Maskable Interrupts a long while back. These NMI’s are extremely high priority and can not be blocked by the OS. While I’ve had trouble fully researching them, the most common use is to kick an otherwise unresponsive system into a “diagnostic” mode. On some systems its implemented as a jumper, others a button, yet other IPMI or SP commands. In the case of Solaris/X86 there are two tunables that can cause the OS to react to NMI; one causes a panic, yet another causes the system to drop into a kmdb session for live debugging. By default Solaris will simply output a message to console saying an NMI was received but otherwise do nothing.
The first of these two tunables can be added to /etc/system, it is not dynamic. You must reboot for it to take effect, it is:
set pcplusmp:apic_panic_on_nmi=1
Once this is added and the system rebooted the receipt of an NMI will cause an immediate panic followed by reboot. The most common way to invoke this behavior would be via the IPMI command: ipmitool -I lanplus -H somehost -U root chassis power diag. Remember, if you get a seriously stuck system your going to reboot the box anyway, typically via IPMI, so instead of using “power cycle” you choose “power diag” and have some tasty data to dig through (or send to Sun).
I like to call this feature “panic on demand”. 
Enter kmdb On NMI
In addition to, or instead of, the panic option above, we can use the following tunable to drop the system into the kmdb kernel debugger on NMI receipt:
set pcplusmp:apic_kmdb_on_nmi=1
If it follow the panic option above it’ll panic and then drop into the debugger, which is the best option. But please know that this only will work if you have kmdb loaded at boot time by adding the -k kernel argument via your bootloader (ie: GRUB). If your working with a production system this might not be something you want hanging over your head all the time and thus a more developer oriented solution in my mind. If your writting drivers you’ll certainly want to keep kmdb loaded, but everyone else will more likely prefer the “panic and reboot” option above.
Snooping; The Deadman Watchdog Timer
Ever watch or read The Abyss? There was a mini-sub that would automatically float to the surface with surveillance tapes and such, if a timer wasn’t reset every 12 hours, in hopes that if anything went wrong there would at least be a partial record of events. Watch dog timers are a similar concept… the kernel uses some means to determine function and if it ceases to function the white flag is waived and it panics.
While its rare, the comment in code (line 163) is the best explanation: “Setting “snooping” to a non-zero value will cause a deadman panic if snoop_interval microseconds elapse without lbolt increasing. The default snoop_interval is 50 seconds.”
So we simply add the following tuning(s) to /etc/system:
set snooping=1 set snoop_interval=90000000
The first line enables snopping. The second line changes the default 50 second interval to 90 seconds. I don’t see any reason that 50 seconds isn’t long enough, but if you want to be paranoid and use a 5 minute interval you can, just change 300 seconds to microseconds and reboot to lock it in.
Snooping has been safe in all my testing to date, but obviously it will feel risky to the casual sysadmin, so this is not something I’d enable by default. If, however, you have a system that mysteriously goes into a dead hang in the middle of the night, this is a better option than being woken up just to testify that it did actually reboot and you still have no idea why.
Poking NMI
As stated, the best way to poke NMI once your ready for it is via IPMI (tested on Sun and Dell):
# ipmitool -I lanplus -H somebox -U root chassis power diag
If you have a moderately recent version of ILOM you can poke the /SP/diag/generate_host_nmi value like so:
-> cd /SP/diag
/SP/diag
-> show
/SP/diag
Targets:
snapshot
Properties:
generate_host_nmi = (Cannot show property)
state = disabled
Commands:
cd
set
show
-> set generate_host_nmi=true
Set 'generate_host_nmi' to 'true'
What Next?
One you’ve got a dump things are on the uptick, but given that you may not be a kernel developer, and if your wise have no desire to become one, you have some options. The first is to send the panic to Sun and hope they come back with a good answer. The second is to use this as your opportunity to dig deep into the guts of Solaris and learn something interesting. I recommend reading my blog entry some time ago: Solaris Core Analysis, Part 1: mdb and Part 2: Solaris CAT, reading The Solaris Operating System on x86 Platforms: Crashdump Analysis Operating System Internals (PDF), getting the latest SolarisCAT, and if possible attending one of Max Bruning’s courses.
Finally, a shout out to my p33ps in #opensolaris IRC for their help in assisting with my testing and research on NMI, in particular “LONGCAT”.
I’m a little puzzled by your comment about Linux — I haven’t seen many uncommanded reboots of Linux that weren’t accompanied by kernel panics, except in cases of major hardware issues. (In my experience, these are often the result of power supply faults, which the OS obviously can’t do anything about.)
Linux can produce dumps on kernel panics, although it doesn’t seem to be common practice to do so.
Maybe I’m just missing the point.
You said it, “although it doesn’t seem to be common practice to do so”. If a panic doesn’t result in a dump its unfruitful.
Fair enough. It’s really a configuration issue — the kernel is fully capable of producing dumps, but most distributions don’t configure it to do so. I suspect this is because most Linux users aren’t kernel hackers and wouldn’t know what to do with a dump if they had one. It’s a bit like the situation with “core” files, which on probably 90% of systems do nothing but sit around and take up disk space.
Thanks for posting the article, was certainly a great read!
This describes exactly the symptoms one of our production boxes is having. And it is handling a lot of tasks with not much memory.
Everything ought to be beautiful in a human being: face, and dress, and soul, and ideas.
You’ve got your deep-sea underwater science-fiction films mixed up. “Sphere” was the film where they had to reset the mini-sub every 12 hrs.
Thanks for your nice sharing first.I don’t know if you are shopping online or not.I always do this.
I think it is a very convenience and save time way to buy goods.I often buy jordan shoes , UGG Boots, Tiffany jewelry on the Internet.you can seach Air Jordan ,Nike Shoes to find the jordan shoes, and I also like Nike Shoes, they are good wear. Anyway,I like Tiffany jewelry best.It can make me more confident and buty.It’s really a good shopping way,if you haven’t do that,try it.You will enjoy yourself.
Your article very interesting, I have introduced a lot of friends look at this article, the content of the articles there will be a lot of attractive people to appreciate, I have to thank you such an article.
Thanks for all of this great information. I can see you worked hard and did a great job. The techies of the world appreciate it.
Very thought provoking. I like the sort of ending that leaves it opn to personal input. Makes it work for just about everyone I think. Nicely done! I’ll subscribe.
Hey, your posts have inspired me! – I love the way you directly get to the point, and then work outwards. I’ve been trying to do figure out what I want to say about ,that would allow me to do exactly the same thing.
Thanks for your post.
I recently came accross your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.
nike shoes jordan shoes air jordans air jordan jordan shoes air jordans
We offer wholesale sport jerseys(NFL,NBA,NHL,MLB),brand-name shoes, T-Shirt, bags, jeans, etc at a low price. If you
are interested in our products,Please kindly visit our website .”Best Service, Highest Quality and Special Price”is our company principle
welcome to our website:
http://www.tradebeneficiary.com
thanks.
thanks for your post
I agree, Linux can produce dumps on kernel panics, although it doesn’t seem to be common practice to do so.
However, they do not only for the fleece actually but the primary purpose is the mutton for table eating.
http://www.usa-jordan.com
This is what I look for a long time information, are very grateful.CHI Flat Iron Blue In Love
I think all of the woman must love jewelry,but not all of them can buy it for it’s high price,even the top brand like tiffany and gucci,but ,now,you can buy it from our website for lower price,come on.
Thank you very much!
Air Max shoes
It is a contoured base that conforms,memoryfoam-like,to your foot so that you almost feel like the shoes was made for you.It is then reinforced to provide extra support.The bottom tier is a combination of th “Classis” and the “Pods” that are seperated in the flat version of the shoes.
Retail UGG australia boots on sale ，we are retail ugg boots seller,we offer original australia ugg boots spend less money to got a comfortable and warm winter.UGG classic cardy,ugg classic mini,UGG nightfall are still sales hot the years.for a fashion life style,uggs are worth to buy!
However, they do not only for the fleece actually but the primary purpose is the mutton for table eating. http://www.ugg-mall.net ugg boots on sale
There are several Gucci shoes that are considered to be elegant and classic designs.When it comes to footwear for wearingout on the town they have a great collection of suede sandals,satin mules,Gucci pumps and more.
Ugg online
Cheap ugg boots
Discount uggs boots
Ugg ultra short
Ugg classic
Cheap ugg
Ugg sale
Ugg classic tall
Shopping for ugg botts
I think I will try to recommend this post to my friends and family, cuz it’s really helpful.
ugg classic tall
Thank you so much my mother is going to love the ugg classic mini.
good
[url=http://www.uggbootshome.net]Ugg Boots[/url]
http://www.uggbootshome.net Wholesale Ugg Boots
This is what I look for a long time information, are very grateful.ps:welcome to my site
ugg knightsbridge boots
Gucci is considered one of the most influential handbag and shoes designers in the fashion industry. The label is synonymous with opulence and quality, and is widely regarded as a status symbol.
http://www.cheapuggsonline.net/
ugg boots
http://www.nikestarsonsale.com/
basketball shoes
I like this ugg ultra short, What is your refund/returns policy?
Look at this ugg ultra tall that i bought last week, Pretty! Fast service. Good Seller
hair products in the market and 100% Quality Guarantee
http://www.cheapuggsonline.net cheap ugg
http://www.cheapuggsonline.net ugg outlet
The sale of fake Gucci shoes is now a common event. But, the online shopping platform adds a fresh layer of deception. Individuals buying the fashionable shoes do not understand that the products they are buying are fake.
Ugg Boots,also called Ug or Ugg boots,are nowadays one of the most popular boots.They existed already in 1930′s and became popular in 1960′s when Australian surfers found out,that they can warm their feet after coming out from cold water.They have everything you need-comfort and warmth.You will never be cold in those boots because 100% sheepskin will warm you even in the coldest days.The natural thermal properties of sheepskin make possible that Ugg Boots will warm you in the winter but in the summer your feet will not be wet.It is true that Ugg is shorter from ugly,but if you know even little bit about fashion you will know that Ugg boots are not ugly.We can see famous people like Pamela Anderson or Kate Hudson wearing Ugg Boots.
Nowadays we have new improved Ugg boots for sale.You can get them for anybody.We have women boots,children boots,slippers and casuals.
Our UGG Australia Boots are all authentic.Their properties and features as below:
*In cold temperatures,the fleece works as an insulating layer of warmth and keeps the warmth of body heat inside of the boot.
*In the heat,the wool fibers keep the air circulating; thus keeping your fee dry and pulling the perspiration away from your skin.Sheepskin also contains lanolin,which moisturizes your feet when worn barefoot.
*ugg boots have reinforced toes,which are double stitched and extra durable heels.
*feature a natural,thick fleece on a removable in-sole for arch support.
*The sole is usually durable in most terrain,lightweight and provides traction as well.
welcome to http://www.bootscardy.com
gemahrv0225
Replica Louis Vuitton
Replica Watches
All people deserve good life and loans or just auto loan will make it better. Just because freedom is based on money.
heyI appreciate the concern which is been jobs. The things need to be sorted out because it is about the individual but it can be with everyone.
Job Sites
Are you want to buy uggs?There are many good ugg boots
Well in this situation farming for the jordan dmp can only be made easy if Discount UGG Boots jordan dmp saleyou have right World of Warcraft tools in your hand. One of the options open before you is to.258479852
Louzhu you write the article is my favorite model
there are so many people like ugg boots, but the price for them are too high , it is a pity, if one can not get one, but now, I would like to recomend you one site where Wholesale Ugg Boots
Discount Ugg Boots
if you want to buy here is some good Website
for another you can see this any more
This is just a theory but I think it’s pretty sound.http://www.bigtimekicks.com
There you can find many different kinds of cheap Air Jordan Shoes .We are a wholesaler of the Air Jordan Shoes
Supra Shoes are always one of the top skateboarding shoes on the block. Overload Skateshop carries a wide range of Supra Shoes to fit your 9-stair kickflips. If you are looking for a pair of Supra Shoes…you have came to the right place. I will introduce supra shoes at next time.cheap supra sneakers.
we supply cheap jordan shoes, we believe that to bring you a lot of surprises.All our products are free shipping,non-tax,7days to your home,incredible discounted price for top quality products.
Hello – I must say, I’m impressed with your site. I had no trouble navigating through all the tabs and information was very easy to access. I found what I wanted in no time at all. http://www.mbtsalestore.com
Dear friend,
Weclome to our website at http://www.aj-family.com
We supply various fashionable and comfortable A.J. 1 to A.J.23 shoes with A1 quality and original packing at good prices.
Wholesale and sample order are both accepted.We also do drop shipping.
Shipping methods are EMS,TNT and DHL etc and the delivery time is approximately 5 days.
We’ll do our best to meet your needs according to your request and also guarantee fast and safe ship door to door to you.
If you have any interest of them,just feel free to contact us at any time.
Sincerely hope we can start a good business relationship on the basis of equality and mutual benefit.Thank you in advance.
it is interesting,god bless you!
it is interesting,god bless you
I very intersted in the article
Index futures contracts have been traded actively since their debut in Shanghai on April 16. But analysts said the current high turnover rate and small holding size indicated that the market was dominated by retail investors.
Reaching the Chinese consumer has been something of a holy grail for international businesses, even when the country had only 400 million people in the 1930s.
Most of the students have been discharged from hospital, but eight remained for further medical observation, according to the spokesman.
Japan has no such ability, while Russia has no such motivation and India is more worried about China, Yang explained.
An Ultra-orthodox Jewish family look at a construction site in the east Jerusalem neighborhood of Ramat Shlomo, Monday, April 26, 2010. [Agencies]
“The government ordered the Interior Ministry immediately after the Biden incident to not even talk about new construction for Jewish homes in east Jerusalem,” said Meir Margalit, a city councilman who said the information came from top Jerusalem officials intimately involved with construction projects.
http://www.gucci-shoes-bags.com
http://www.guccinewshop.com
Dear friends welcome to our store: we have a specail offer now which is once you buy any product you can get a free gift as well, these gifts are in the gift area, you can free to choose and please add to the shopping cart, we will send it together with your purchased product. Thanks!
Do you knowtales of pirates gold?if you play the online game,you will knowbuy tales of pirates goldis the game gold.
thank you ,I have learned a lot.
As you may get to know something more about the jordan retro iii
Shoes
jordan retro iii
First of all, do not hurry when buying Coach Bags, Footwear is the matter of fashion and style.
it is good!
the blog is informative.
the article is useful.
that is great.
Buy Nike Air Max 90 Shoes just $45-55 USD in
http://www.iofferitems.com, 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!
Buy Nike Air Max 90 Shoes just $45-55 USD in http://www.iofferitems.com, 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!
UGG created fashion lines such as the Carolinas and Boots Stella. These boots have a sheepskin found the same as in UGG classic boots other hardships of winter, but with the heel provide a touch of fashion. Other varieties are in the style of UGG classic boots Cove and Knightsbridge, the soles have secure grip and top with unique and elegant.
http://www.jerseyscloset.com
I really liked reading your post!. Quallity content. With such a valuable blog i believe you deserve to be ranking even higher in the search engines
. Check out the link in my name. That links to a tool that really helped me rank high in google. This way even more people can enjoy your posts and nothing beats a big audiance 
Amazing write-up! This could aid plenty of people find out more about this particular issue. Are you keen to integrate video clips coupled with these? It would absolutely help out. Your conclusion was spot on and thanks to you; I probably won’t have to describe everything to my pals. I can simply direct them here!
that is very kind of you to do this for us, thank you very much. you can have a look at the [url]http://www.monclershop.org [/url]
[url]http://www.monclerjacke.org [/url]
that is very kind of you to post this for us, that is very good. you can have a look at the [url]http://www.uggbootsoutlet.biz [/url]
[url]http://www.ugg-sale.biz [/url]
ugg clothing
ugg clothing boots
women’s ugg clothing boots – free shipping http://www.uggclothingboots.org
ugg outlet,ugg boots outlet,ugg boots store http://www.ugg-boots-outlet.org
new ugg
new uggs
new ugg boots – free shipping http://www.new-ugg-boots.org
We always look for something can’t reach, walk around the real truth and can’t achieve the goals we expect, therefore, we still not know how to get it, do it and make it.
Thanks for the valuable information. This was just the thing I was looking for; I really like how it includes the actual curved shape flight paths. Keep posting. I will be visiting back soon.
Five Fingers,Vibram Five Fingers,Vibram Five Fingers Shoes
http://www.cheapvibramfivefingersshoes.com
vibram 5 fingers,5 fingers shoes,vibram 5 fingers shoes http://www.ibram-5-fingers.org
thank you.is good
http://www.airjordan-sale.com/michael-jordan-shoes-c-25.html Michael Jordan Shoes
that is very kind of you to do this for us, thank you very much. you can have a look at the cheap ugg boots.
百度
[url=http://www.sina.com]sina[/url]
In probing and examining the oceanic surface of the world named Solaris from a hovering research station the oblivious human scientists are, in turn, being studied by the sentient planet itself. In due course, Solaris scientifically probes for and examines the secret, guilty thoughts of the human beings who are analyzing it. Solaris has the ability to manifest their secret, guilty concerns in human form, for each scientist to personally confront, while the self-aware planet studies their responses to its psychological experiments.
Solaris is pervaded by a powerful, poetic sense of the physical remoteness of outer space. The sense of loneliness that this engenders is among Lem’s philosophic explorations of man’s anthropomorphic limitations. First published in Warsaw in 1961, the 1970 Polish-to-French-to-English translation of Solaris is the best-known of Lem’s English-translated works.
Very happy to see your article, I very much to like and agree with your point of view. Thank you for sharing. At the same time,i love
Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with more information? It is extremely helpful for me.
Its nice to read some post about this nowadays. Its meaningful and somehow sharing this to readers like me, makes me want to surf the web to be able to get more wonderful ideas.Thank youfor posting this…
http://www.uggonlinebootssale.com/
Days are getting colder.Are you looking for a pair of great boots to keep your feet warm?The answer for you is ugg boots.
Love is frail at birth, but it grows stronger with age if it is properly fed.
http://www.jacket-buy.com
Do you like vibram five finger shoes, now it is very popular and has many fans, if you like it, don’t wait but choose it. http://www.vibramfivefingersshoesusa.com/