Solaris is one of the most stable operating systems available… but lets face it, stuff happens. Solaris does panic, but I want everyone to be clear, a “panic”, despite the seemingly contradictory name, is by its nature a controlled event. When the kernel encounters behavior that is uncorrectable and will cause irreparable harm to the running system or, even worse, corrupt data, the system will voluntarily tap out using the panic system call to get the system down quickly, hopefully leaving a core dump in its wake for post-mortem analysis.
In this blog entry we’ll discuss core dumps and panic’s in general. In part 2 we’ll discuss a tool to make life just a little easier, the Solaris Crash Analysis Tool, or “Solaris CAT”.
I want to point out that post-mortem core analysis is really the task of a kernel engineer. The fact is, way less than 1% of us who ever engage in core analysis are actually going to have any real idea of what the hell we’re doing. And thats ok! I guarantee that you’ll post something from an analysis to a mailing list and you’ll get some asshole who forgets that he’s been paid to work on the Solaris kernel for the last 20 years while you work a job which is now on hold because of said core dump, with replies like “We can clearly see that due to the memory address in this register that you are a moron….” The point here is, if you don’t know what your doing, don’t be discouraged. What we, mere mortals, are trying to do is not necessarily solve the problem but provide clues which will help us guide our search, either by posting a stack trace to a mailing list, or send the dump to Sun Support, or to take a panic string and search the bug database or Google for. The cuddletech rule of crashes is:
An unexpected crash is unacceptable; An unexplained crash is inexcusable.
If you’re reading this you’ve probably lived through a panic before, but lets recap. The best explanation of a “crash” event and resulting dump can be found in the dumpadm(1M) man page:
A crash dump is a disk copy of the physical memory
of the computer at the time of a fatal system error. When a
fatal operating system error occurs, a message describing
the error is printed to the console. The operating system
then generates a crash dump by writing the contents of phy-
sical memory to a predetermined dump device, which is typi-
cally a local disk partition. The dump device can be config-
ured by way of dumpadm. Once the crash dump has been writ-
ten to the dump device, the system will reboot.
Fatal operating system errors can be caused by bugs in the
operating system, its associated device drivers and loadable
modules, or by faulty hardware. Whatever the cause, the
crash dump itself provides invaluable information to your
support engineer to aid in diagnosing the problem. As such,
it is vital that the crash dump be retrieved and given to
your support provider. Following an operating system crash,
the savecore(1M) utility is executed automatically during
boot to retrieve the crash dump from the dump device, and
write it to a pair of files in your file system named unix.X
and vmcore.X, where X is an integer identifying the dump.
Together, these data files form the saved crash dump. The
directory in which the crash dump is saved on reboot can
also be configured using dumpadm.
I encourage you to read both the savecore(1M) and dumpadm(1M) man pages. You’ll find that with savecore -L you can create a dump of a live system, so if you don’t have a crashed system around to play with, use that. Alternatively, you can use reboot -d to dump a core and reboot.
At this point we’ll assume you have a dump available. By default you’ll find them in /var/crash/hostname/, you’ll have dumps in pairs: vmcore.0 and unix.0. We feed these two files to mdb, the (-k, kernel) Modular DeBugger, to preform our analysis like so:
# mdb -k unix.0 vmcore.0 Loading modules: [ unix krtld genunix specfs dtrace cpu.AuthenticAMD.15 uppc pcplusmp ufs ip sctp usba lofs zfs random ipc md fcip fctl fcp crypto logindmux ptm nfs ] >
You are now free to move about the dump. mdb commands are strange and unusual at first, it takes a lot of time to get comfortable with it, but there are a couple of debugger commands that can give us the essence of what we need. Lets walk through them.
The ::status command will display high level information regarding this debugging session. Of usefulness here is the dumps “panic message” and OS release.
> ::status debugging crash dump vmcore.0 (64-bit) from hostname operating system: 5.11 snv_43 (i86pc) panic message: BAD TRAP: type=e (#pf Page fault) rp=fffffe80000ad3d0 addr=0 occurred in module "unix" due to a NULL pointer dereference dump content: kernel pages only
The ::stack command will prove you with a stack trace, this is the same thing trace you would have seen in syslog or the console.
> ::stack atomic_add_32() nfs_async_inactive+0x55(fffffe820d128b80, 0, ffffffffeff0ebcb) nfs3_inactive+0x38b(fffffe820d128b80, 0) fop_inactive+0x93(fffffe820d128b80, 0) vn_rele+0x66(fffffe820d128b80) snf_smap_desbfree+0x78(fffffe8185e2ff60) dblk_lastfree_desb+0x25(fffffe817a30f8c0, ffffffffac1d7cc0) dblk_decref+0x6b(fffffe817a30f8c0, ffffffffac1d7cc0) freeb+0x89(fffffe817a30f8c0) tcp_rput_data+0x215f(ffffffffb4af7140, fffffe812085d780, ffffffff993c3c00) squeue_enter_chain+0x129(ffffffff993c3c00, fffffe812085d780, fffffe812085d780, 1, 1) ip_input+0x810(ffffffffa23eec68, ffffffffaeab8040, fffffe812085d780, e) i_dls_link_ether_rx_promisc+0x266(ffffffff9a4c35f8, ffffffffaeab8040, fffffe812085d780) mac_rx+0x7a(ffffffffa2345c40, ffffffffaeab8040, fffffe812085d780) e1000g_intr+0xf6(ffffffff9a4b2000) av_dispatch_autovect+0x83(1a) intr_thread+0x50()
The ::msgbuf command will output the message buffer at the time of crash; the message buffer is most commonly used by sysadmins through the “dmesg” command.
> ::msgbuf MESSAGE .... WARNING: IP: Hardware address '00:14:4f:xxxxxxx' trying to be our address xxxx WARNING: IP: Hardware address '00:14:4f:xxxx' trying to be our address xxxx panic[cpu0]/thread=fffffe80000adc80: BAD TRAP: type=e (#pf Page fault) rp=fffffe80000ad3d0 addr=0 occurred in module "unix" due to a NULL pointer dereference sched: #pf Page fault Bad kernel fault at addr=0x0 .... blah blah, snipped for brevity.
The ::panicinfo command will give you lots of fun cryptic counter information, of most interest is the first 3 lines, which contain the CPU on which the panic occured, the running thread, and the panic message. You’ll notice these are commonly repeated and the most useful pieces of information.
> ::panicinfo
cpu 0
thread fffffe80000adc80
message BAD TRAP: type=e (#pf Page fault) rp=fffffe80000ad3d0 addr=0 occurred in module "unix" due to a NULL pointer dereference
rdi 0
rsi 1
rdx fffffe80000adc80
rcx 0
r8 0
r9 fffffe80dba125c0
rax 0
rbx fffffe8153a36040
rbp fffffe80000ad4e0
r10 3e0
r10 3e0
r11 ffffffffaeab8040
r12 ffffffffb7b4cac0
r13 0
r14 fffffe820d128b80
r15 ffffffffeff0ebcb
fsbase ffffffff80000000
gsbase fffffffffbc27850
ds 43
es 43
fs 0
gs 1c3
trapno e
err 2
rip fffffffffb838680
cs 28
rflags 10246
rsp fffffe80000ad4c8
ss 0
gdt_hi 0
gdt_lo defacedd
idt_hi 0
idt_lo 80300fff
ldt 0
task 60
cr0 80050033
cr2 0
cr3 10821b000
In my opinion, the koolest command is ::cpuinfo -v. Truth be told, if you run multiple applications on a server the most common question people (especially managers) want answered is “which application did it?”, being translated into geek-esse “who do I blame?” This command will help you determine that by displaying, complete with beautiful ASCII art, the threads and process names running on each CPU (NRUN). In the following example, we know the event occured on CPU 0, thus thats the one we want to look at. Note that the “sched” process should be interpreted as “kernel”.
> ::cpuinfo -v
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
0 fffffffffbc2f370 1b 1 0 165 no no t-1 fffffe80000adc80 sched
| | |
RUNNING <--+ | +--> PIL THREAD
READY | 6 fffffe80000adc80
EXISTS | - fffffe80daab6a20 ruby
ENABLE |
+--> PRI THREAD PROC
99 fffffe8000b88c80 sched
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
1 ffffffff983b3800 1f 1 0 59 yes no t-0 fffffe80daac2f20 smtpd
| |
RUNNING <--+ +--> PRI THREAD PROC
READY 99 fffffe8000bacc80 sched
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
2 ffffffff9967a800 1f 2 0 -1 no no t-0 fffffe8000443c80
(idle)
| |
RUNNING <--+ +--> PRI THREAD PROC
READY 99 fffffe8000b82c80 sched
QUIESCED 60 fffffe80018f8c80 sched
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
3 ffffffff9967a000 1f 1 0 -1 no no t-0 fffffe8000535c80
(idle)
| |
RUNNING <--+ +--> PRI THREAD PROC
READY 60 fffffe8000335c80 zsched
QUIESCED
EXISTS
ENABLE
The ::ps command allows us to see all running processes. Several flags are supported, including -z to display Zone ID’s.
> ::ps -z S PID PPID PGID SID ZONE UID FLAGS ADDR NAME R 0 0 0 0 0 0 0x00000001 fffffffffbc25900 sched R 3 0 0 0 0 0 0x00020001 ffffffff9970d928 fsflush R 2 0 0 0 0 0 0x00020001 ffffffff9970e558 pageout R 1 0 0 0 0 0 0x42004000 ffffffff9970f188 init R 20534 1 20533 20533 24 1006 0x42010400 ffffffffb246f9b8 ruby R 20532 1 20531 20531 24 1006 0x42010400 fffffe8109674308 ruby R 20529 1 20528 20528 24 1006 0x42010400 fffffe80dc5602f0 ruby ...
We can use ::pgrep to search for processes and use the appropriate address for further digging. In the following example I’ll find a Java process and then determine which zone that process was running in:
> ::pgrep java S PID PPID PGID SID UID FLAGS ADDR NAME R 3628 1 3620 3574 0 0x42004400 fffffe80deeb3240 java > fffffe80deeb3240::print proc_t p_zone->zone_name p_zone->zone_name = 0xffffffffae0cef00 "testzone03"
There are many more tools and way to dig into your dumps using mdb. It can be confusing because you need to reference things by address, but you get more comfortable with it as you play around. If you are interested in learning more I highly recommend reading Eric Lowe’s “Examining the Anatomy of a Process”, which digs into the topic of process examination via mdb.
One thing you’ll notice in all this is that the messages at the time of crash on the console or in syslog contain almost everything you need to know without digging too deeply. Therefore, assuming you have those messages, the most useful thing most people will extract from the core files is the output of the ::cpuinfo command to see what process was on the offending CPU at the time of the crash. Knowing what processes, zones, etc, were running at the time of crash are interesting but rarely mean much if they weren’t directly involved in the panic.
As I said, once you start getting into referencing memory addresses to deepen your analysis things get sticky and tricky very very quickly… thats where Solaris CAT comes in, which we’ll talk about in part 2.
very insightful article ! a nice introduction as a guideline which lets you want to know more. thanks again for the entry – and nice humour too
Wow, this is a very succinct but dense article.
I remember when I had my first kernel crash (apparently due to a corrupted kernel module on disk) I got stuck with a little console and very little idea of what to do.
It’s going to take a while to really understand all the tools available in Solaris, and hopefully I won’t have many chances to try them out on real crash dumps.
Looking forward to Part 2.
Hi Ben, did you ever used the mdebug tool ( http://developers.sun.com/solaris/articles/mdebug/mdebug.html ) ? It outputs a file which give you a summary of the core dump analysis. Very useful when you need to give a quick answer to the management why the system crash.
It also give you the time of boot and the time of crash. It happen to me one time that the savecore never finished and dumped an old kernel core. From now on, I always check the time of crash, to be really sure that I got the right core dump!
Thanks for the article Ben. Great resource. Submitting on BigAdmin now.
first national card 1
face carved pendant
college books buy
http://scolin.007sites.com/of2921.html charm jewelry phoenix
http://scolin.hostaim.com/of7131.html cheap flights mariel
http://rec-video-rb.tripod.com/of1785.html antique restoration trunk
cheap hotels rakovnik
custom casting jewelry
diamond pave settings
http://scolin.my3gb.com/of1851.html pandora cord bracelet
http://scolin.my3gb.com/the5919.html huge silver bowl
http://scolin.007sites.com/of7669.html gold vermeil findings
diamond head history
parts with platinum
cheap fares flights
http://msc-flash-hl.tripod.com/of4289.html cheap truck rims
http://scolin.my3gb.com/of3046.html tiffany jewelry replicas
http://scolin.freeweb7.com/of311.html weber grill silver
buy gas scooter
buy inflatable screens
diamond pave hoops
http://rec-video-rb.tripod.com/of6240.html claddagh gold bracelet
http://jobakter.9cy.com/of249.html floor cleaning business
http://nba-video-ty.tripod.com/of8225.html kuon game buy
business license llc
costume jewelry sale
k-cup coffee cheap
http://lyo-games-ke.tripod.com/of6595.html x2vga buy
http://nba-video-ty.tripod.com/of8620.html designer charm bracelet
http://scolin.serverocean.com/of934.html christmas angel necklace
ganga calling card
[url=http://jobakter.wtcsites.com/of9833.html]silver creek clip [/url]
usa wrestling card
http://scolin.007sites.com/of750.html macromedia buy dreamweaver
http://scolin.mycyberway.com/in9849.html phpnuke platinum themes
http://scolin.serverocean.com/of2980.html gold cup football
usb2 pci card
buy a rifle
debit card online
http://hcy-reasonings.tripod.com/in311.html purity platinum eagles
http://ckm-meier.tripod.com/of4236.html cheap labret jewlery
http://scolin.gigazu.com/in5441.html better business bearuea
rainbow lodge california
silver id tag
original airport card
http://scolin.freeweb7.com/in2735.html pagan e card
http://bjf-prawo.tripod.com/of1374.html itam jewelry manufacturer
http://scolin.fusedtree.com/in8999.html buy hash oil
platinum topaz ring
white diamond university
buy refrigerator parts
http://ckm-meier.tripod.com/of9657.html wood personalized necklace
http://volkov.orgfree.com/the5381.html birthday card fiishing
http://volkov.orgfree.com/of3789.html buy chess clocks
purchase cultured pearls
jay diamond radio
video card scan
http://ckm-meier.tripod.com/of475.html juniors cheap tees
http://scolin.fizwig.com/of8986.html acme printing boston
http://scolin.001webs.com/in2547.html chair cheap leather
baja california tijuana
carolina gold mines
wool ornament pattern
http://bebre.orgfree.com/of3091.html gold piercing jewelry
http://ckm-meier.tripod.com/of4201.html business card designs
http://scolin.aokhost.com/in3622.html ball pendant watch
max life bracelet
gold bracelet mens
buy one rim
http://scolin.freeweb7.com/of1364.html credit card philadelphia
http://scolin.hostbot.com/in3533.html cheap flights marseille
http://scolin.sitebooth.com/of42.html cheap disney vavcations
debbie diamond gallery
melt silver coins
topps card 519 mcmahon
http://scolin.hostevo.com/of3373.html cheap crotchless panties
http://scolin.8tt.org/for6727.html quicksbooks cheap ebay
http://scolin.hostbot.com/for5845.html fork r7 platinum
frog greeting cards
shoebox holiday card
grace samodal jewellery
http://scolin.fizwig.com/in6710.html southern charms jilly
http://scolin.sitebooth.com/for3908.html card friday good
http://scolin.seitenclique.net/in7720.html cheap surfboard socks
9 gold ankle braclets
diamond shaped paperweights
robins diamonds deleware
http://scolin.aokhost.com/of3271.html baseball card gold
http://scolin.seitenclique.net/of9394.html explorers in america
http://scolin.gigazu.com/of3375.html vanna k jewelry
cheap airfares potgietersrus
cheap flights qingyang
natural cosmetic recepie
http://onferra.hothostcity.com/in9187.html building credit card
http://onferra.hothostcity.com/the1674.html diamond ring sale
http://onferra.mycyberway.com/of928.html buy sabian machine
mastercard corparte card
viedo capture card
cheap flights didyr
http://onferra.8tt.org/of7005.html office index cards
http://onferra.007sites.com/of8850.html om diamond pendant
http://onferra.seitenclique.net/of481.html lb gold ring
siam silver jewelry
amy diamond nuts
rocel mexican jewelry
http://onferra.mycyberway.com/of5368.html business economics notes
http://onferra.gigazu.com/of7616.html ecount debit card
http://onferra.007sites.com/of1266.html diamond innovations dublin
silvercast diamond delta
cozumel diamonds international
octopus craft jewellery
http://onferra.justfree.com/in4574.html business manager career
http://onferra.serverocean.com/of2132.html diamond grading sdj
http://hnn-bookstall-ro.tripod.com/in2114.html australia broker business
bear jewelry navajo
sterling silver jewelers
cheap summer vacation
http://onferra.mycyberway.com/of5368.html white bracelet charity
http://onferra.mycyberway.com/in6342.html brylane home jewelry
http://onferra.hostevo.com/of8363.html amber ankle bracelet
jewelry casting supplies
hostel budapest cheap
cheap airfares aketi
http://onferra.hothostcity.com/of6525.html pr card lost
http://onferra.seitenclique.net/in1383.html windsor collection jewelry
http://onferra.sitebooth.com/of4719.html military nex card
plastic bangle jewelry
card hate mother
valley silver bullet
http://onferra.freeweb7.com/of8276.html stern moon pendant
http://onferra.fizwig.com/of81.html wsop bracelet picture
http://jzf-ytou-cg.tripod.com/in7130.html jewelry current research
video254 [url=http://my.opera.com/skazzi9/blog/2008/10/13/2]video254[/url]
video48 [url=http://my.opera.com/skazzi9/blog/2008/10/13/6]video48[/url]
video241 [url=http://my.opera.com/wearerobots/blog/2008/10/13/10]video241[/url]
video482 [url=http://my.opera.com/skazzi9/blog/2008/10/13/20]video482[/url]
video602 [url=http://my.opera.com/skazzi9/blog/2008/10/13/14]video602[/url]
video157 [url=http://my.opera.com/wearerobots/blog/2008/10/13/10]video157[/url]
video389 [url=http://my.opera.com/skazzi9/blog/2008/10/13/11]video389[/url]
video614 [url=http://my.opera.com/wearerobots/blog/2008/10/13/5]video614[/url]
video229 [url=http://my.opera.com/skazzi9/blog/2008/10/13/19]video229[/url]
video349 [url=http://my.opera.com/skazzi9/blog/2008/10/13/13]video349[/url]
video141 [url=http://my.opera.com/skazzi9/blog/2008/10/13/15]video141[/url]
video20 [url=http://my.opera.com/skazzi9/blog/2008/10/13/20]video20[/url]
video38 [url=http://my.opera.com/wearerobots/blog/2008/10/13/17]video38[/url]
video266 [url=http://my.opera.com/skazzi9/blog/2008/10/13/14]video266[/url]
video233 [url=http://my.opera.com/wearerobots/blog/2008/10/13/2]video233[/url]
video191 [url=http://my.opera.com/wearerobots/blog/2008/10/13/2]video191[/url]
video597 [url=http://my.opera.com/skazzi9/blog/2008/10/13/9]video597[/url]
video560 [url=http://my.opera.com/skazzi9/blog/2008/10/13/14]video560[/url]
video392 [url=http://my.opera.com/skazzi9/blog/2008/10/13/14]video392[/url]
video267 [url=http://my.opera.com/skazzi9/blog/2008/10/13/15]video267[/url]
video120 [url=http://my.opera.com/wearerobots/blog/2008/10/13/15]video120[/url]
video201 [url=http://my.opera.com/wearerobots/blog/2008/10/13/12]video201[/url]
video9 [url=http://my.opera.com/skazzi9/blog/2008/10/13/9]video9[/url]
video393 [url=http://my.opera.com/skazzi9/blog/2008/10/13/15]video393[/url]
video518 [url=http://my.opera.com/skazzi9/blog/2008/10/13/14]video518[/url]
video514 [url=http://my.opera.com/skazzi9/blog/2008/10/13/10]video514[/url]
video155 [url=http://my.opera.com/wearerobots/blog/2008/10/13/8]video155[/url]
video382 [url=http://my.opera.com/skazzi9/blog/2008/10/13/4]video382[/url]
video7 [url=http://my.opera.com/skazzi9/blog/2008/10/13/7]video7[/url]
video468 [url=http://my.opera.com/skazzi9/blog/2008/10/13/6]video468[/url]
video221 [url=http://my.opera.com/skazzi9/blog/2008/10/13/11]video221[/url]
video380 [url=http://my.opera.com/skazzi9/blog/2008/10/13/2]video380[/url]
video498 [url=http://my.opera.com/wearerobots/blog/2008/10/13/15]video498[/url]
video42 [url=http://my.opera.com/wearerobots/blog/2008/10/13/21]video42[/url]
video606 [url=http://my.opera.com/skazzi9/blog/2008/10/13/18]video606[/url]
video198 [url=http://my.opera.com/wearerobots/blog/2008/10/13/9]video198[/url]
video205 [url=http://my.opera.com/wearerobots/blog/2008/10/13/16]video205[/url]
video225 [url=http://my.opera.com/skazzi9/blog/2008/10/13/15]video225[/url]
video28 [url=http://my.opera.com/wearerobots/blog/2008/10/13/7]video28[/url]
video655 [url=http://my.opera.com/wearerobots/blog/2008/10/13/4]video655[/url]
video367 [url=http://my.opera.com/wearerobots/blog/2008/10/13/10]video367[/url]
video322 [url=http://my.opera.com/wearerobots/blog/2008/10/13/7]video322[/url]
video232 [url=http://my.opera.com/wearerobots/blog/2008/10/13/1]video232[/url]
video219 [url=http://my.opera.com/skazzi9/blog/2008/10/13/9]video219[/url]
video544 [url=http://my.opera.com/wearerobots/blog/2008/10/13/19]video544[/url]
fine gem jewelry
cali logan cards
animated greetings cards
http://onferra.yourfreehosting.net/in9295.html cheap flights ukraine
http://onferra.serverocean.com/of7389.html antique sapphire jewelry
http://onferra.hostbot.com/of3967.html jewelry studio layout
nautical gift basket
gift enclosure cards
jewelry mystic topaz
http://onferra.hostevo.com/of3536.html gold ring distributors
http://onferra.justfree.com/in1447.html antique exterior doors
http://onferra.8tt.org/of2755.html bibliography card maker
rap music about maine
sondheim musicals tickets 2007
flat stanley musical austin
http://pliogha.freehostia.com/music2755.html sheet music for trios
http://mmonno.freehostia.com/music193.html music resources
http://onferra.fizwig.com/music1041.html most violent music video
books musicabona cz reference
nightcastle music
anime music video newsgroups
http://foslok.orgfree.com/music7653.html larry sigmon music
http://yre-prov.tripod.com/music1056.html tribute music original band
http://onferra.007sites.com/music8905.html hawk nelson music codes
country music helping
jordanian music
free music instantly
http://vvbne5.orgfree.com/music3347.html sheet music for granada
http://klosters.freehostia.com/music3299.html music line search
http://aswere.freehostia.com/music309.html tba music
music store programs
renaissance lute music database
peter gabriel music msn
http://onferra.serverocean.com/music1986.html sanfransico music
http://foslok.orgfree.com/music5391.html download free music flamenco
http://mnegor.freehostia.com/music1332.html censorship in rap music
silver cake plataue
22 karat gold jewelery
thomas gold hypnotized
http://hma-hinder.tripod.com/music3730.html mukta silver jaipur
http://aswere.freehostia.com/music4321.html silver dragon store
http://onferra.007sites.com/music3374.html gold crown trophy
hempstead gold club
gold deposits indiana
gran turismo platinum
http://onferra.cedhost.com/music9654.html platinum practice jerseys
http://zertopyt.freehostia.com/music7532.html gold value history
http://onferra.freeweb7.com/music3844.html gold spinner charms
platinum fountain pens
map pokemon silver
gold purification mechinery
http://pliogha.freehostia.com/music1786.html silver spoons eating
http://klosters.freehostia.com/music1292.html earring peridot silver
http://foslok.orgfree.com/music7744.html going for gold
silver rehoboth delaware
silver coffee table
silver chocolate lab
http://klosters.freehostia.com/music1633.html nymex gold options
http://hma-hinder.tripod.com/music6760.html antenna orinoco gold
http://cretija.freehostia.com/music2867.html jewls gold pirate
silver turtle bank
gold leaf studios
gold angel pins
http://foslok.orgfree.com/music5918.html clinque silver pink
http://onferra.mycyberway.com/music832.html meriden silver caster
http://zertopyt.freehostia.com/music2877.html italian gold bracelets
essentials gold membership
14 gold k watch
silver bunny jewlery
http://shuare.hothostcity.com/gold1264.html sin gold cheats
http://shuare.hostevo.com/gold77.html silver state gaming
http://shuare.hostaim.com/gold4718.html xenium gold manual
family of platinum
cheap silver rings
silver ring jewelry
http://shuare.hostevo.com/gold3910.html enya silver mp3
http://gnr-misheard.tripod.com/gold6331.html postbank gold card
http://nrm-eights-ls.tripod.com/gold7252.html leo’s gold lion
gold canyon elementary
gold stationary set
dimensions gold petites
http://shuare.007sites.com/gold5907.html mexico mine silver
http://nrm-eights-ls.tripod.com/gold3130.html platinum chain bracelet
http://shuare.mycyberway.com/gold1665.html platinum stud earring
silver rocker switches
liebenzeller gold rosin
gold plated 7 16 wrench
http://ipt-scalia.tripod.com/gold4407.html scripture jewelry silver
http://shuare.hostbot.com/gold126.html single gold atom
http://shuare.hothostcity.com/gold639.html gold koran jewelry
gold figaro bracelet
baby silver plate
og himalayan gold
http://shuare.sitebooth.com/gold3405.html jpeg 1901 s $5.00 gold
http://jeh-misheard.tripod.com/gold7151.html the silver apples
http://shuare.serverocean.com/gold461.html gold glove wiki
silver anniversary invites
koning trouble silver
gold coast beaches
http://shuare.8tt.org/gold7080.html wallace silver ware
http://shuare.mycyberway.com/gold3892.html finland gold marka
http://shuare.8tt.org/gold3825.html crack dvdfab platinum 3.2.0.0
gold jewerly sale
gold women’s boots
hex code gold
http://nrm-eights-ls.tripod.com/gold8690.html gold food charms
http://jeh-misheard.tripod.com/gold8971.html microscopic gold co
http://ipt-scalia.tripod.com/gold1301.html gold heart lock
pics426 [url=http://my.opera.com/miramiar/blog/2008/10/14/18]pics426[/url]
pics444 [url=http://my.opera.com/miramiar/blog/2008/10/14/2]pics444[/url]
pics26 [url=http://my.opera.com/miramiar/blog/2008/10/14/26]pics26[/url]
pics403 [url=http://my.opera.com/miramiar/blog/2008/10/14/29]pics403[/url]
pics122 [url=http://my.opera.com/miramiar/blog/2008/10/14/20]pics122[/url]
pics154 [url=http://my.opera.com/miramiar/blog/2008/10/14/18]pics154[/url]
pics474 [url=http://my.opera.com/miramiar/blog/2008/10/14/32]pics474[/url]
pics178 [url=http://my.opera.com/miramiar/blog/2008/10/14/8]pics178[/url]
pics349 [url=http://my.opera.com/miramiar/blog/2008/10/14/9]pics349[/url]
pics533 [url=http://my.opera.com/miramiar/blog/2008/10/14/23]pics533[/url]
pics16 [url=http://my.opera.com/miramiar/blog/2008/10/14/16]pics16[/url]
pics172 [url=http://my.opera.com/miramiar/blog/2008/10/14/2]pics172[/url]
pics69 [url=http://my.opera.com/miramiar/blog/2008/10/14/1]pics69[/url]
pics71 [url=http://my.opera.com/miramiar/blog/2008/10/14/3]pics71[/url]
pics199 [url=http://my.opera.com/miramiar/blog/2008/10/14/29]pics199[/url]
pics370 [url=http://my.opera.com/miramiar/blog/2008/10/14/30]pics370[/url]
pics149 [url=http://my.opera.com/miramiar/blog/2008/10/14/13]pics149[/url]
pics244 [url=http://my.opera.com/miramiar/blog/2008/10/14/6]pics244[/url]
pics121 [url=http://my.opera.com/miramiar/blog/2008/10/14/19]pics121[/url]
pics481 [url=http://my.opera.com/miramiar/blog/2008/10/14/5]pics481[/url]
pics96 [url=http://my.opera.com/miramiar/blog/2008/10/14/28]pics96[/url]
pics85 [url=http://my.opera.com/miramiar/blog/2008/10/14/17]pics85[/url]
pics241 [url=http://my.opera.com/miramiar/blog/2008/10/14/3]pics241[/url]
pics490 [url=http://my.opera.com/miramiar/blog/2008/10/14/14]pics490[/url]
pics233 [url=http://my.opera.com/miramiar/blog/2008/10/14/29]pics233[/url]
pics529 [url=http://my.opera.com/miramiar/blog/2008/10/14/19]pics529[/url]
pics204 [url=http://my.opera.com/miramiar/blog/2008/10/14/34]pics204[/url]
pics106 [url=http://my.opera.com/miramiar/blog/2008/10/14/4]pics106[/url]
pics67 [url=http://my.opera.com/miramiar/blog/2008/10/14/33]pics67[/url]
pics62 [url=http://my.opera.com/miramiar/blog/2008/10/14/28]pics62[/url]
pics467 [url=http://my.opera.com/miramiar/blog/2008/10/14/25]pics467[/url]
pics402 [url=http://my.opera.com/miramiar/blog/2008/10/14/28]pics402[/url]
pics466 [url=http://my.opera.com/miramiar/blog/2008/10/14/24]pics466[/url]
pics163 [url=http://my.opera.com/miramiar/blog/2008/10/14/27]pics163[/url]
pics130 [url=http://my.opera.com/miramiar/blog/2008/10/14/28]pics130[/url]
pics543 [url=http://my.opera.com/miramiar/blog/2008/10/14/33]pics543[/url]
pics392 [url=http://my.opera.com/miramiar/blog/2008/10/14/18]pics392[/url]
pics408 [url=http://my.opera.com/miramiar/blog/2008/10/14/34]pics408[/url]
pics195 [url=http://my.opera.com/miramiar/blog/2008/10/14/25]pics195[/url]
pics513 [url=http://my.opera.com/miramiar/blog/2008/10/14/3]pics513[/url]
pics168 [url=http://my.opera.com/miramiar/blog/2008/10/14/32]pics168[/url]
pics99 [url=http://my.opera.com/miramiar/blog/2008/10/14/31]pics99[/url]
pics307 [url=http://my.opera.com/miramiar/blog/2008/10/14/1]pics307[/url]
pics23 [url=http://my.opera.com/miramiar/blog/2008/10/14/23]pics23[/url]
pics389 [url=http://my.opera.com/miramiar/blog/2008/10/14/15]pics389[/url]
silver bracelets dlf
vat gold scotch
i’ve found gold
http://shuare.mycyberway.com/gold4549.html white yellow gold
http://shuare.freeweb7.com/gold7003.html silver seas criuse
http://shuare.serverocean.com/gold2236.html idaho gold towns
identify silver mark
silver’s current value
mexico gold panning
http://shuare.fusedtree.com/silver3694.html silver coin margarita
http://shuare.yourfreehosting.net/gold141.html gold weight 20 franc
http://shuare.fusedtree.com/gold5376.html silver cloud stasdium
gold stocks kennecot
[url=http://shuare.fusedtree.com/gold1252.html]mens silver neckless [/url]
purple-and gold magazine
http://shuare.hostaim.com/gold8795.html gold resources reed
http://shuare.freeweb7.com/gold3755.html gold angel necklaces
http://shuare.yourfreehosting.net/gold273.html diggers vote gold
bruce oldfield jewellery
cran actin cheap
paypal accepted cards
http://vidagh.007sites.com/card8579.html course italy jewelry
http://vidagh.mycyberway.com/card9012.html cheap exotic jewelry
http://vidagh.8tt.org/card7379.html recipe cards online
cheap covert gps
gold chandelier earings
platinum card concierge
http://vidagh.hostevo.com/card1853.html 512mb sd card
http://vidagh.hothostcity.com/card5351.html wiggle fish necklace
http://vidagh.8tt.org/card138.html buy precriptions online
sirius radio sp-tk2r
st vincents radiology school
ksxt am radio
http://kovnpaq.s-enterprize.com/in1766.html satellite radio antenna
http://ieocbidu.servetown.com/of5582.html golf car accessories lights radios
http://nggrgof.100freemb.com/of8274.html me radio stations
bbc radio
ask an attorney radio show
radioactive nuclear decay elements
http://hiowqlz.rbhoward.com/of3170.html cb radio repair and parts
http://vidagh.fusedtree.com/of6856.html aol radio free
http://vidagh.seitenclique.net/in658.html free music internet radio
Buy Viagra Online
Buy Levitra Online
Buy Cialis Online
Buy Propecia Online
npr radio streaming
radio stations rhode island 920 am
talk radio contact information
http://cvzyrut.jamminweb.com/of1698.html @virus reclame radio
http://eugoudx.freecities.com/the8792.html radiohead lyric
http://ilxyzea.stinkdot.org/in7549.html pepsi cap radio
sherlock holmes radio shows
fm radio stationsphiladelphia
radio stations in ireland awards
http://tiqokhi.b-w-h.com/of7100.html wls chicago radio 890 laura miles hosts
http://erophpz.igojo.com/of2409.html radiotavshed
http://zitgcpc.angelcities.com/of1749.html toyota tercel radio
Royal Tropical Institute. Royal Tropical Institute middot Home middot News middot Latest news middot Agenda middot Newsletter middot Press releases middot International Events cheap levitra LEVITRA side effects, dosage, and drug interactions. All accurate, up-to-date information is written for the consumer by healthcare professionals.
Levitra (vardenafil) is used to treat erectile dysfunction. Includes Levitra side effects, interactions and indications. order levitra Vardenafil (INN) is a PDE5 inhibitor used for treating impotence (erectile dysfunction) that is sold under the trade name Levitra (Bayer AG).
Buy Viagra Cialis Levitra online for impotence treatment. Buy Acomplia Xenical online for obesity treatment. Buy Chantix Propecia Tamiflu online at generic cialis Generic Cialis is known to as effective as within 30 minutes of taking it with effects stay long for as long as 36 hours.
Get All medicines with or without prescription,controlled drugs ,viagra and many more all at skyrxshop.com. viagra online We’ve all heard of Viagra. The wonder pill that treats erectile dysfunction. Erectile dysfunction or impotence as it is often called, affects men of all
Viagra – 1305 results like the GENERIC Viagra Price per Pill, GENERIC BRAND VIAGRA Sildenafil Citrate 100 Mg 35 Pills, PFIZER Viagra 100mg – 30 Tablets, order viagra Rezeptpflichtige Medikamente viagra levitra levitra propecia Viagra levitra levitra Apotheker verschreibungspflichtige Arzneimittel Cialis Kaufen
atwater radio
pa radio stations
wdad radio
http://euaitrm.jvl.com/of9976.html mankato lawn mower radio
http://nouibic.freewebpages.org/of4369.html radio broadcasting swear words
http://vidagh.007sites.com/in799.html atwater reed radio
oWRvMM evurwuzjebfm, [url=http://cbdwyuifqxck.com/]cbdwyuifqxck[/url], [link=http://pujpchiovbqk.com/]pujpchiovbqk[/link], http://poxiubekvgkw.com/
http://floristic.friko.pl/flomaxafd.html flomax pbG http://floristic.freehostingz.com/flomaxd81.html flomax P flomax [ http://members.lycos.nl/floristic/flomax3d5.html ] flomax [link=http://members.lycos.nl/floristic/flomax3d5.html] flomax [/link] e)%0 http://floristic.007sites.com/geodonf14.html geodon CKEA!lE(4 http://floristic.hostaim.com/flomax328.html flomax
________________________________________________
<>http://recipe-for.chelendger.co.cc/ ]recipe for corned beef
HC4hC6 bjeukfooylie, [url=http://kqpfbrgjyran.com/]kqpfbrgjyran[/url], [link=http://agxozegnvjol.com/]agxozegnvjol[/link], http://okmfaafrrtua.com/
Hi! I read your post and found it really interesting. Concierge services like office concierge and corporate concierge are really a great demand these days by corporations especially the executives. Hope this service doesn’t become a habit.
Hello… I read the post and it is very good. Very helpful
Thanks alot
Good article, thanks to the author to share. and i want to share ugg boots with you.
uggs boots on sale in this ugg store.
Good!
Great,thanks for sharing this.
The fact is, way less than 1% of us who ever engage in core analysis are actually going to have any real idea of what the hell we’re doing. And thats ok!
very good, thanks for post.
thanks for post.
thanks
good read thanks
many thanks
Good post! Thanks for your information! As Seen On TV
Good post! Thanks for your information!
Thank you for your info ,welcome to read this info: http://www.goodpuma.com
style Classic Tall Uggs oh, so larruping of the ugg classic short 5825 i have seen. the feature of knit uppers (composed of a wool blend) ugg boots sale and a sheepskin sock liner for ultra comfort makes me so crazy and want ugg sale to mobilize all my desires, the functional buttons allow the shaft to be worn tall or folded over for a cute cuffed look and decorative buttons make this ugg boots more fashion and easier to suit your dress,
lighting designer Adam Silverman. Paul Steinberg’s corrugated metal sets are fine, wholesale jordan shoes even though the first act wholesale jordan shoes looks an awful lot like the first act set for Alden’s recent wholesale nike shoes ENO Jenfa. wholesale jordan shoes Grime’s precariously perched, too tidy hut is particularly handsome, and the act two beach scene wholesale jordan shoes is beautiful in wholesale jordan shoes an Edward Hopperish fashion.
Hi! Google gerade im Web und bin auf dieser Homepage gelandet.
Respekt, haben Sie toll gemacht.
Ich wünsche Ihnen weiterhin viel Erfolg mit Ihre Seite und natürlich viele Besucher.
Hier habe ich mich sehr wohl gefühlt und werde wieder hierher zurückkommen.
Freue mich auf einen Gegenbesuch auf meiner Homepage
thank you ,I have learned a lot.
Hi! Google gerade im Web und bin auf dieser Homepage gelandet.
Respekt, haben Sie toll gemacht.
http://www.trademic.com/product/index.tm.keyword=KOSS&bid=.html
http://www.gucci-shoes-bags.com
http://www.guccinewshop.com
Dear friends welcome to our store: we have a specail offer now which is once you buy any product you can get a free gift as well, these gifts are in the gift area, you can free to choose and please add to the shopping cart, we will send it together with your purchased product. Thanks!
thank you for sharing the article.
The easy-to-use Video Converter for Mac lets you to enjoy your videos on all sorts of palyback including PSP, iPod, Mobile Phone, Zune, iPhone, Apple TV and MP4/MP3 player.
Free download supported. http://www.videoconverterformac.com
Your article are great, will visit very soon. there is a site: http://www.lovemypursemall.com