The Blog of Ben Rockwood

Solaris Core Analysis, Part 1: mdb

06 Sep '08 - 07:47 by benr

Solaris is one of the most stable operating systems available... but lets face it, stuff happens. Solaris does panic, but I want everyone to be clear, a "panic", despite the seemingly contradictory name, is by its nature a controlled event. When the kernel encounters behavior that is uncorrectable and will cause irreparable harm to the running system or, even worse, corrupt data, the system will voluntarily tap out using the panic system call to get the system down quickly, hopefully leaving a core dump in its wake for post-mortem analysis.

In this blog entry we'll discuss core dumps and panic's in general. In part 2 we'll discuss a tool to make life just a little easier, the Solaris Crash Analysis Tool, or "Solaris CAT".

I want to point out that post-mortem core analysis is really the task of a kernel engineer. The fact is, way less than 1% of us who ever engage in core analysis are actually going to have any real idea of what the hell we're doing. And thats ok! I guarantee that you'll post something from an analysis to a mailing list and you'll get some asshole who forgets that he's been paid to work on the Solaris kernel for the last 20 years while you work a job which is now on hold because of said core dump, with replies like "We can clearly see that due to the memory address in this register that you are a moron...." The point here is, if you don't know what your doing, don't be discouraged. What we, mere mortals, are trying to do is not necessarily solve the problem but provide clues which will help us guide our search, either by posting a stack trace to a mailing list, or send the dump to Sun Support, or to take a panic string and search the bug database or Google for. The cuddletech rule of crashes is:

An unexpected crash is unacceptable; An unexplained crash is inexcusable.

If you're reading this you've probably lived through a panic before, but lets recap. The best explanation of a "crash" event and resulting dump can be found in the dumpadm(1M) man page:

     A crash dump is a disk copy of the physical memory
     of  the computer at the time of a fatal system error. When a
     fatal operating system error occurs,  a  message  describing
     the  error  is  printed to the console. The operating system
     then generates a crash dump by writing the contents of  phy-
     sical  memory to a predetermined dump device, which is typi-
     cally a local disk partition. The dump device can be config-
     ured  by way of dumpadm.  Once the crash dump has been writ-
     ten to the dump device, the system will reboot.

     Fatal operating system errors can be caused by bugs  in  the
     operating system, its associated device drivers and loadable
     modules, or by faulty  hardware.  Whatever  the  cause,  the
     crash  dump  itself  provides invaluable information to your
     support engineer to aid in diagnosing the problem. As  such,
     it  is  vital  that the crash dump be retrieved and given to
     your support provider. Following an operating system  crash,
     the  savecore(1M)  utility  is executed automatically during
     boot to retrieve the crash dump from the  dump  device,  and
     write it to a pair of files in your file system named unix.X
     and vmcore.X, where X is an integer  identifying  the  dump.
     Together,  these  data  files form the saved crash dump. The
     directory in which the crash dump is  saved  on  reboot  can
     also be configured using dumpadm.

I encourage you to read both the savecore(1M) and dumpadm(1M) man pages. You'll find that with savecore -L you can create a dump of a live system, so if you don't have a crashed system around to play with, use that. Alternatively, you can use reboot -d to dump a core and reboot.

At this point we'll assume you have a dump available. By default you'll find them in /var/crash/hostname/, you'll have dumps in pairs: vmcore.0 and unix.0. We feed these two files to mdb, the (-k, kernel) Modular DeBugger, to preform our analysis like so:

# mdb -k unix.0 vmcore.0 
Loading modules: [ unix krtld genunix specfs dtrace cpu.AuthenticAMD.15 uppc pcplusmp ufs ip sctp usba lofs zfs random ipc md fcip fctl fcp crypto logindmux ptm nfs ]
>

You are now free to move about the dump. mdb commands are strange and unusual at first, it takes a lot of time to get comfortable with it, but there are a couple of debugger commands that can give us the essence of what we need. Lets walk through them.

The ::status command will display high level information regarding this debugging session. Of usefulness here is the dumps "panic message" and OS release.

> ::status
debugging crash dump vmcore.0 (64-bit) from hostname
operating system: 5.11 snv_43 (i86pc)
panic message: BAD TRAP: type=e (#pf Page fault) rp=fffffe80000ad3d0 addr=0 occurred in module "unix" due to a NULL pointer dereference
dump content: kernel pages only

The ::stack command will prove you with a stack trace, this is the same thing trace you would have seen in syslog or the console.

> ::stack
atomic_add_32()
nfs_async_inactive+0x55(fffffe820d128b80, 0, ffffffffeff0ebcb)
nfs3_inactive+0x38b(fffffe820d128b80, 0)
fop_inactive+0x93(fffffe820d128b80, 0)
vn_rele+0x66(fffffe820d128b80)
snf_smap_desbfree+0x78(fffffe8185e2ff60)
dblk_lastfree_desb+0x25(fffffe817a30f8c0, ffffffffac1d7cc0)
dblk_decref+0x6b(fffffe817a30f8c0, ffffffffac1d7cc0)
freeb+0x89(fffffe817a30f8c0)
tcp_rput_data+0x215f(ffffffffb4af7140, fffffe812085d780, ffffffff993c3c00)
squeue_enter_chain+0x129(ffffffff993c3c00, fffffe812085d780, fffffe812085d780, 1, 1)
ip_input+0x810(ffffffffa23eec68, ffffffffaeab8040, fffffe812085d780, e)
i_dls_link_ether_rx_promisc+0x266(ffffffff9a4c35f8, ffffffffaeab8040, fffffe812085d780)
mac_rx+0x7a(ffffffffa2345c40, ffffffffaeab8040, fffffe812085d780)
e1000g_intr+0xf6(ffffffff9a4b2000)
av_dispatch_autovect+0x83(1a)
intr_thread+0x50()

The ::msgbuf command will output the message buffer at the time of crash; the message buffer is most commonly used by sysadmins through the "dmesg" command.

> ::msgbuf
MESSAGE                                                               
....
WARNING: IP: Hardware address '00:14:4f:xxxxxxx' trying to be our address xxxx
WARNING: IP: Hardware address '00:14:4f:xxxx' trying to be our address xxxx

panic[cpu0]/thread=fffffe80000adc80: 
BAD TRAP: type=e (#pf Page fault) rp=fffffe80000ad3d0 addr=0 occurred in module "unix" due to a NULL pointer dereference

sched: 
#pf Page fault
Bad kernel fault at addr=0x0
.... blah blah, snipped for brevity.

The ::panicinfo command will give you lots of fun cryptic counter information, of most interest is the first 3 lines, which contain the CPU on which the panic occured, the running thread, and the panic message. You'll notice these are commonly repeated and the most useful pieces of information.

> ::panicinfo
             cpu                0
          thread fffffe80000adc80
         message BAD TRAP: type=e (#pf Page fault) rp=fffffe80000ad3d0 addr=0 occurred in module "unix" due to a NULL pointer dereference
             rdi                0
             rsi                1
             rdx fffffe80000adc80
             rcx                0
              r8                0
              r9 fffffe80dba125c0
             rax                0
             rbx fffffe8153a36040
             rbp fffffe80000ad4e0
             r10              3e0
             r10              3e0
             r11 ffffffffaeab8040
             r12 ffffffffb7b4cac0
             r13                0
             r14 fffffe820d128b80
             r15 ffffffffeff0ebcb
          fsbase ffffffff80000000
          gsbase fffffffffbc27850
              ds               43
              es               43
              fs                0
              gs              1c3
          trapno                e
             err                2
             rip fffffffffb838680
              cs               28
          rflags            10246
             rsp fffffe80000ad4c8
              ss                0
          gdt_hi                0
          gdt_lo         defacedd
          idt_hi                0
          idt_lo         80300fff
             ldt                0
            task               60
             cr0         80050033
             cr2                0
             cr3        10821b000

In my opinion, the koolest command is ::cpuinfo -v. Truth be told, if you run multiple applications on a server the most common question people (especially managers) want answered is "which application did it?", being translated into geek-esse "who do I blame?" This command will help you determine that by displaying, complete with beautiful ASCII art, the threads and process names running on each CPU (NRUN). In the following example, we know the event occured on CPU 0, thus thats the one we want to look at. Note that the "sched" process should be interpreted as "kernel".

>  ::cpuinfo -v
 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  0 fffffffffbc2f370  1b    1    0 165   no    no t-1    fffffe80000adc80 sched
                       |    |    |
            RUNNING <--+    |    +--> PIL THREAD
              READY         |           6 fffffe80000adc80
             EXISTS         |           - fffffe80daab6a20 ruby
             ENABLE         |
                            +-->  PRI THREAD           PROC
                                   99 fffffe8000b88c80 sched

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  1 ffffffff983b3800  1f    1    0  59  yes    no t-0    fffffe80daac2f20 smtpd
                       |    |
            RUNNING <--+    +-->  PRI THREAD           PROC
              READY                99 fffffe8000bacc80 sched
           QUIESCED
             EXISTS
             ENABLE

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  2 ffffffff9967a800  1f    2    0  -1   no    no t-0    fffffe8000443c80
 (idle)
                       |    |
            RUNNING <--+    +-->  PRI THREAD           PROC
              READY                99 fffffe8000b82c80 sched
           QUIESCED                60 fffffe80018f8c80 sched
             EXISTS
             ENABLE

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  3 ffffffff9967a000  1f    1    0  -1   no    no t-0    fffffe8000535c80
 (idle)
                       |    |
            RUNNING <--+    +-->  PRI THREAD           PROC
              READY                60 fffffe8000335c80 zsched
           QUIESCED
             EXISTS
             ENABLE

The ::ps command allows us to see all running processes. Several flags are supported, including -z to display Zone ID's.

> ::ps -z
S    PID   PPID   PGID    SID  ZONE    UID      FLAGS             ADDR NAME
R      0      0      0      0     0      0 0x00000001 fffffffffbc25900 sched
R      3      0      0      0     0      0 0x00020001 ffffffff9970d928 fsflush
R      2      0      0      0     0      0 0x00020001 ffffffff9970e558 pageout
R      1      0      0      0     0      0 0x42004000 ffffffff9970f188 init
R  20534      1  20533  20533    24   1006 0x42010400 ffffffffb246f9b8 ruby
R  20532      1  20531  20531    24   1006 0x42010400 fffffe8109674308 ruby
R  20529      1  20528  20528    24   1006 0x42010400 fffffe80dc5602f0 ruby
...

We can use ::pgrep to search for processes and use the appropriate address for further digging. In the following example I'll find a Java process and then determine which zone that process was running in:

> ::pgrep java
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R   3628      1   3620   3574      0 0x42004400 fffffe80deeb3240 java
> fffffe80deeb3240::print proc_t p_zone->zone_name
p_zone->zone_name = 0xffffffffae0cef00 "testzone03"

There are many more tools and way to dig into your dumps using mdb. It can be confusing because you need to reference things by address, but you get more comfortable with it as you play around. If you are interested in learning more I highly recommend reading Eric Lowe's "Examining the Anatomy of a Process", which digs into the topic of process examination via mdb.

One thing you'll notice in all this is that the messages at the time of crash on the console or in syslog contain almost everything you need to know without digging too deeply. Therefore, assuming you have those messages, the most useful thing most people will extract from the core files is the output of the ::cpuinfo command to see what process was on the offending CPU at the time of the crash. Knowing what processes, zones, etc, were running at the time of crash are interesting but rarely mean much if they weren't directly involved in the panic.

As I said, once you start getting into referencing memory addresses to deepen your analysis things get sticky and tricky very very quickly... thats where Solaris CAT comes in, which we'll talk about in part 2.

- - C O M M E N T S - -
very insightful article ! a nice introduction as a guideline which lets you want to know more. thanks again for the entry – and nice humour too

Jean-Claude Vangonghen (Email) (URL) - 06 September '08 - 19:24

Wow, this is a very succinct but dense article.

I remember when I had my first kernel crash (apparently due to a corrupted kernel module on disk) I got stuck with a little console and very little idea of what to do.

It’s going to take a while to really understand all the tools available in Solaris, and hopefully I won’t have many chances to try them out on real crash dumps.

Looking forward to Part 2.

drew (Email) - 08 September '08 - 17:26

Hi Ben, did you ever used the mdebug tool ( [[http://developers.sun.com/solaris/arti..]] ) ? It outputs a file which give you a summary of the core dump analysis. Very useful when you need to give a quick answer to the management why the system crash.

It also give you the time of boot and the time of crash. It happen to me one time that the savecore never finished and dumped an old kernel core. From now on, I always check the time of crash, to be really sure that I got the right core dump!

Geoffroy Doucet (Email) (URL) - 09 September '08 - 03:39

Thanks for the article Ben. Great resource. Submitting on BigAdmin now. :)

Robert Weeks - 18 September '08 - 19:34

first national card 1
face carved pendant
college books buy
[[http://scolin.007sites.com/of2921.html]] charm jewelry phoenix
[[http://scolin.hostaim.com/of7131.html]] cheap flights mariel
[[http://rec-video-rb.tripod.com/of1785...]] antique restoration trunk

JasonGR (Email) (URL) - 11 October '08 - 09:53

cheap hotels rakovnik
custom casting jewelry
diamond pave settings
[[http://scolin.my3gb.com/of1851.html]] pandora cord bracelet
[[http://scolin.my3gb.com/the5919.html]] huge silver bowl
[[http://scolin.007sites.com/of7669.html]] gold vermeil findings

JasonGR (Email) (URL) - 11 October '08 - 11:15

diamond head history
parts with platinum
cheap fares flights
[[http://msc-flash-hl.tripod.com/of4289...]] cheap truck rims
[[http://scolin.my3gb.com/of3046.html]] tiffany jewelry replicas
[[http://scolin.freeweb7.com/of311.html]] weber grill silver

JasonGR (Email) (URL) - 11 October '08 - 15:48

buy gas scooter
buy inflatable screens
diamond pave hoops
[[http://rec-video-rb.tripod.com/of6240...]] claddagh gold bracelet
[[http://jobakter.9cy.com/of249.html]] floor cleaning business
[[http://nba-video-ty.tripod.com/of8225...]] kuon game buy

JasonGR (Email) (URL) - 11 October '08 - 17:04

business license llc
costume jewelry sale
k-cup coffee cheap
[[http://lyo-games-ke.tripod.com/of6595...]] x2vga buy
[[http://nba-video-ty.tripod.com/of8620...]] designer charm bracelet
[[http://scolin.serverocean.com/of934.ht..]] christmas angel necklace

JasonGR (Email) (URL) - 11 October '08 - 18:15

ganga calling card
[url=[[http://jobakter.wtcsites.com/of9833.ht..]] creek clip [/url]
usa wrestling card
[[http://scolin.007sites.com/of750.html]] macromedia buy dreamweaver
[[http://scolin.mycyberway.com/in9849.ht..]] phpnuke platinum themes
[[http://scolin.serverocean.com/of2980.h..]] gold cup football

JasonGR (Email) (URL) - 11 October '08 - 19:25

usb2 pci card
buy a rifle
debit card online
[[http://hcy-reasonings.tripod.com/in311..]] purity platinum eagles
[[http://ckm-meier.tripod.com/of4236.htm..]] cheap labret jewlery
[[http://scolin.gigazu.com/in5441.html]] better business bearuea

DominiqueFO (Email) (URL) - 11 October '08 - 21:21

rainbow lodge california
silver id tag
original airport card
[[http://scolin.freeweb7.com/in2735.html]] pagan e card
[[http://bjf-prawo.tripod.com/of1374.htm..]] itam jewelry manufacturer
[[http://scolin.fusedtree.com/in8999.htm..]] buy hash oil

DominiqueFO (Email) (URL) - 12 October '08 - 00:01

platinum topaz ring
white diamond university
buy refrigerator parts
[[http://ckm-meier.tripod.com/of9657.htm..]] wood personalized necklace
[[http://volkov.orgfree.com/the5381.html]] birthday card fiishing
[[http://volkov.orgfree.com/of3789.html]] buy chess clocks

DominiqueFO (Email) (URL) - 12 October '08 - 02:43

purchase cultured pearls
jay diamond radio
video card scan
[[http://ckm-meier.tripod.com/of475.html]] juniors cheap tees
[[http://scolin.fizwig.com/of8986.html]] acme printing boston
[[http://scolin.001webs.com/in2547.html]] chair cheap leather

DominiqueFO (Email) (URL) - 12 October '08 - 04:00

baja california tijuana
carolina gold mines
wool ornament pattern
[[http://bebre.orgfree.com/of3091.html]] gold piercing jewelry
[[http://ckm-meier.tripod.com/of4201.htm..]] business card designs
[[http://scolin.aokhost.com/in3622.html]] ball pendant watch

DominiqueFO (Email) (URL) - 12 October '08 - 05:23

max life bracelet
gold bracelet mens
buy one rim
[[http://scolin.freeweb7.com/of1364.html]] credit card philadelphia
[[http://scolin.hostbot.com/in3533.html]] cheap flights marseille
[[http://scolin.sitebooth.com/of42.html]] cheap disney vavcations

DominiqueFO (Email) (URL) - 12 October '08 - 06:42

debbie diamond gallery
melt silver coins
topps card 519 mcmahon
[[http://scolin.hostevo.com/of3373.html]] cheap crotchless panties
[[http://scolin.8tt.org/for6727.html]] quicksbooks cheap ebay
[[http://scolin.hostbot.com/for5845.html]] fork r7 platinum

JacobYH (Email) (URL) - 12 October '08 - 08:44

frog greeting cards
shoebox holiday card
grace samodal jewellery
[[http://scolin.fizwig.com/in6710.html]] southern charms jilly
[[http://scolin.sitebooth.com/for3908.ht..]] card friday good
[[http://scolin.seitenclique.net/in7720...]] cheap surfboard socks

JacobYH (Email) (URL) - 12 October '08 - 10:10

9 gold ankle braclets
diamond shaped paperweights
robins diamonds deleware
[[http://scolin.aokhost.com/of3271.html]] baseball card gold
[[http://scolin.seitenclique.net/of9394...]] explorers in america
[[http://scolin.gigazu.com/of3375.html]] vanna k jewelry

JacobYH (Email) (URL) - 12 October '08 - 13:00

cheap airfares potgietersrus
cheap flights qingyang
natural cosmetic recepie
[[http://onferra.hothostcity.com/in9187...]] building credit card
[[http://onferra.hothostcity.com/the1674..]] diamond ring sale
[[http://onferra.mycyberway.com/of928.ht..]] buy sabian machine

TylerBN (Email) (URL) - 12 October '08 - 18:06

mastercard corparte card
viedo capture card
cheap flights didyr
[[http://onferra.8tt.org/of7005.html]] office index cards
[[http://onferra.007sites.com/of8850.htm..]] om diamond pendant
[[http://onferra.seitenclique.net/of481...]] lb gold ring

TylerBN (Email) (URL) - 12 October '08 - 20:26

siam silver jewelry
amy diamond nuts
rocel mexican jewelry
[[http://onferra.mycyberway.com/of5368.h..]] business economics notes
[[http://onferra.gigazu.com/of7616.html]] ecount debit card
[[http://onferra.007sites.com/of1266.htm..]] diamond innovations dublin

TylerBN (Email) (URL) - 12 October '08 - 21:35

silvercast diamond delta
cozumel diamonds international
octopus craft jewellery
[[http://onferra.justfree.com/in4574.htm..]] business manager career
[[http://onferra.serverocean.com/of2132...]] diamond grading sdj
[[http://hnn-bookstall-ro.tripod.com/in2..]] australia broker business

TylerBN (Email) (URL) - 12 October '08 - 23:44

bear jewelry navajo
sterling silver jewelers
cheap summer vacation
[[http://onferra.mycyberway.com/of5368.h..]] white bracelet charity
[[http://onferra.mycyberway.com/in6342.h..]] brylane home jewelry
[[http://onferra.hostevo.com/of8363.html]] amber ankle bracelet

TylerBN (Email) (URL) - 13 October '08 - 02:00

jewelry casting supplies
hostel budapest cheap
cheap airfares aketi
[[http://onferra.hothostcity.com/of6525...]] pr card lost
[[http://onferra.seitenclique.net/in1383..]] windsor collection jewelry
[[http://onferra.sitebooth.com/of4719.ht..]] military nex card

TylerBN (Email) (URL) - 13 October '08 - 03:06

plastic bangle jewelry
card hate mother
valley silver bullet
[[http://onferra.freeweb7.com/of8276.htm..]] stern moon pendant
[[http://onferra.fizwig.com/of81.html]] wsop bracelet picture
[[http://jzf-ytou-cg.tripod.com/in7130.h..]] jewelry current research

MMSean (Email) (URL) - 13 October '08 - 05:13

Aline (Email) (URL) - 13 October '08 - 10:05

fine gem jewelry
cali logan cards
animated greetings cards
[[http://onferra.yourfreehosting.net/in9..]] cheap flights ukraine
[[http://onferra.serverocean.com/of7389...]] antique sapphire jewelry
[[http://onferra.hostbot.com/of3967.html]] jewelry studio layout

MMSean (Email) (URL) - 13 October '08 - 11:04

nautical gift basket
gift enclosure cards
jewelry mystic topaz
[[http://onferra.hostevo.com/of3536.html]] gold ring distributors
[[http://onferra.justfree.com/in1447.htm..]] antique exterior doors
[[http://onferra.8tt.org/of2755.html]] bibliography card maker

MMSean (Email) (URL) - 13 October '08 - 13:19

rap music about maine
sondheim musicals tickets 2007
flat stanley musical austin
[[http://pliogha.freehostia.com/music275..]] sheet music for trios
[[http://mmonno.freehostia.com/music193...]] music resources
[[http://onferra.fizwig.com/music1041.ht..]] most violent music video

JamesUA (Email) (URL) - 13 October '08 - 13:29

books musicabona cz reference
nightcastle music
anime music video newsgroups
[[http://foslok.orgfree.com/music7653.ht..]] larry sigmon music
[[http://yre-prov.tripod.com/music1056.h..]] tribute music original band
[[http://onferra.007sites.com/music8905...]] hawk nelson music codes

JamesUA (Email) (URL) - 13 October '08 - 14:44

country music helping
jordanian music
free music instantly
[[http://vvbne5.orgfree.com/music3347.ht..]] sheet music for granada
[[http://klosters.freehostia.com/music32..]] music line search
[[http://aswere.freehostia.com/music309...]] tba music

JamesUA (Email) (URL) - 13 October '08 - 17:11

music store programs
renaissance lute music database
peter gabriel music msn
[[http://onferra.serverocean.com/music19..]] sanfransico music
[[http://foslok.orgfree.com/music5391.ht..]] download free music flamenco
[[http://mnegor.freehostia.com/music1332..]] censorship in rap music

JamesUA (Email) (URL) - 13 October '08 - 18:26

silver cake plataue
22 karat gold jewelery
thomas gold hypnotized
[[http://hma-hinder.tripod.com/music3730..]] mukta silver jaipur
[[http://aswere.freehostia.com/music4321..]] silver dragon store
[[http://onferra.007sites.com/music3374...]] gold crown trophy

DSSebastian (Email) (URL) - 13 October '08 - 21:33

hempstead gold club
gold deposits indiana
gran turismo platinum
[[http://onferra.cedhost.com/music9654.h..]] platinum practice jerseys
[[http://zertopyt.freehostia.com/music75..]] gold value history
[[http://onferra.freeweb7.com/music3844...]] gold spinner charms

DSSebastian (Email) (URL) - 13 October '08 - 23:00

platinum fountain pens
map pokemon silver
gold purification mechinery
[[http://pliogha.freehostia.com/music178..]] silver spoons eating
[[http://klosters.freehostia.com/music12..]] earring peridot silver
[[http://foslok.orgfree.com/music7744.ht..]] going for gold

DSSebastian (Email) (URL) - 14 October '08 - 02:11

silver rehoboth delaware
silver coffee table
silver chocolate lab
[[http://klosters.freehostia.com/music16..]] nymex gold options
[[http://hma-hinder.tripod.com/music6760..]] antenna orinoco gold
[[http://cretija.freehostia.com/music286..]] jewls gold pirate

DSSebastian (Email) (URL) - 14 October '08 - 05:00

silver turtle bank
gold leaf studios
gold angel pins
[[http://foslok.orgfree.com/music5918.ht..]] clinque silver pink
[[http://onferra.mycyberway.com/music832..]] meriden silver caster
[[http://zertopyt.freehostia.com/music28..]] italian gold bracelets

DSSebastian (Email) (URL) - 14 October '08 - 06:21

essentials gold membership
14 gold k watch
silver bunny jewlery
[[http://shuare.hothostcity.com/gold1264..]] sin gold cheats
[[http://shuare.hostevo.com/gold77.html]] silver state gaming
[[http://shuare.hostaim.com/gold4718.htm..]] xenium gold manual

JohnIS (Email) (URL) - 14 October '08 - 06:54

family of platinum
cheap silver rings
silver ring jewelry
[[http://shuare.hostevo.com/gold3910.htm..]] enya silver mp3
[[http://gnr-misheard.tripod.com/gold633..]] postbank gold card
[[http://nrm-eights-ls.tripod.com/gold72..]] leo’s gold lion

JohnIS (Email) (URL) - 14 October '08 - 08:05

gold canyon elementary
gold stationary set
dimensions gold petites
[[http://shuare.007sites.com/gold5907.ht..]] mexico mine silver
[[http://nrm-eights-ls.tripod.com/gold31..]] platinum chain bracelet
[[http://shuare.mycyberway.com/gold1665...]] platinum stud earring

JohnIS (Email) (URL) - 14 October '08 - 09:19

silver rocker switches
liebenzeller gold rosin
gold plated 7 16 wrench
[[http://ipt-scalia.tripod.com/gold4407...]] scripture jewelry silver
[[http://shuare.hostbot.com/gold126.html]] single gold atom
[[http://shuare.hothostcity.com/gold639...]] gold koran jewelry

JohnIS (Email) (URL) - 14 October '08 - 11:34

gold figaro bracelet
baby silver plate
og himalayan gold
[[http://shuare.sitebooth.com/gold3405.h..]] jpeg 1901 s $5.00 gold
[[http://jeh-misheard.tripod.com/gold715..]] the silver apples
[[http://shuare.serverocean.com/gold461...]] gold glove wiki

JohnIS (Email) (URL) - 14 October '08 - 15:25

silver anniversary invites
koning trouble silver
gold coast beaches
[[http://shuare.8tt.org/gold7080.html]] wallace silver ware
[[http://shuare.mycyberway.com/gold3892...]] finland gold marka
[[http://shuare.8tt.org/gold3825.html]] crack dvdfab platinum 3.2.0.0

JohnIS (Email) (URL) - 14 October '08 - 16:56

gold jewerly sale
gold women’s boots
hex code gold
[[http://nrm-eights-ls.tripod.com/gold86..]] gold food charms
[[http://jeh-misheard.tripod.com/gold897..]] microscopic gold co
[[http://ipt-scalia.tripod.com/gold1301...]] gold heart lock

JohnIS (Email) (URL) - 14 October '08 - 18:01

tvolkmann80 (Email) (URL) - 14 October '08 - 18:50

silver bracelets dlf
vat gold scotch
i’ve found gold
[[http://shuare.mycyberway.com/gold4549...]] white yellow gold
[[http://shuare.freeweb7.com/gold7003.ht..]] silver seas criuse
[[http://shuare.serverocean.com/gold2236..]] idaho gold towns

ReginaldYC (Email) (URL) - 14 October '08 - 22:37

identify silver mark
silver’s current value
mexico gold panning
[[http://shuare.fusedtree.com/silver3694..]] silver coin margarita
[[http://shuare.yourfreehosting.net/gold..]] gold weight 20 franc
[[http://shuare.fusedtree.com/gold5376.h..]] silver cloud stasdium

ReginaldYC (Email) (URL) - 15 October '08 - 00:57

gold stocks kennecot
[url=[[http://shuare.fusedtree.com/gold1252.h..]] silver neckless [/url]
purple-and gold magazine
[[http://shuare.hostaim.com/gold8795.htm..]] gold resources reed
[[http://shuare.freeweb7.com/gold3755.ht..]] gold angel necklaces
[[http://shuare.yourfreehosting.net/gold..]] diggers vote gold

ReginaldYC (Email) (URL) - 15 October '08 - 03:16

bruce oldfield jewellery
cran actin cheap
paypal accepted cards
[[http://vidagh.007sites.com/card8579.ht..]] course italy jewelry
[[http://vidagh.mycyberway.com/card9012...]] cheap exotic jewelry
[[http://vidagh.8tt.org/card7379.html]] recipe cards online

VTJoe (Email) (URL) - 15 October '08 - 06:44

cheap covert gps
gold chandelier earings
platinum card concierge
[[http://vidagh.hostevo.com/card1853.htm..]] 512mb sd card
[[http://vidagh.hothostcity.com/card5351..]] wiggle fish necklace
[[http://vidagh.8tt.org/card138.html]] buy precriptions online

VTJoe (Email) (URL) - 15 October '08 - 11:42

sirius radio sp-tk2r
st vincents radiology school
ksxt am radio
[[http://kovnpaq.s-enterprize.com/in1766..]] satellite radio antenna
[[http://ieocbidu.servetown.com/of5582.h..]] golf car accessories lights radios
[[http://nggrgof.100freemb.com/of8274.ht..]] me radio stations

LYPaul (Email) (URL) - 15 October '08 - 13:21

bbc radio
ask an attorney radio show
radioactive nuclear decay elements
[[http://hiowqlz.rbhoward.com/of3170.htm..]] cb radio repair and parts
[[http://vidagh.fusedtree.com/of6856.htm..]] aol radio free
[[http://vidagh.seitenclique.net/in658.h..]] free music internet radio

LYPaul (Email) (URL) - 15 October '08 - 16:08

Buy Viagra Online
Buy Levitra Online
Buy Cialis Online
Buy Propecia Online

ppaelkos (Email) (URL) - 15 October '08 - 16:53

npr radio streaming
radio stations rhode island 920 am
talk radio contact information
[[http://cvzyrut.jamminweb.com/of1698.ht..]] @virus reclame radio
[[http://eugoudx.freecities.com/the8792...]] radiohead lyric
[[http://ilxyzea.stinkdot.org/in7549.htm..]] pepsi cap radio

LYPaul (Email) (URL) - 15 October '08 - 17:28

sherlock holmes radio shows
fm radio stationsphiladelphia
radio stations in ireland awards
[[http://tiqokhi.b-w-h.com/of7100.html]] wls chicago radio 890 laura miles hosts
[[http://erophpz.igojo.com/of2409.html]] radiotavshed
[[http://zitgcpc.angelcities.com/of1749...]] toyota tercel radio

LYPaul (Email) (URL) - 15 October '08 - 18:48

Royal Tropical Institute. Royal Tropical Institute middot Home middot News middot Latest news middot Agenda middot Newsletter middot Press releases middot International Events cheap levitra LEVITRA side effects, dosage, and drug interactions. All accurate, up-to-date information is written for the consumer by healthcare professionals.
Levitra (vardenafil) is used to treat erectile dysfunction. Includes Levitra side effects, interactions and indications. order levitra Vardenafil (INN) is a PDE5 inhibitor used for treating impotence (erectile dysfunction) that is sold under the trade name Levitra (Bayer AG).
Buy Viagra Cialis Levitra online for impotence treatment. Buy Acomplia Xenical online for obesity treatment. Buy Chantix Propecia Tamiflu online at generic cialis Generic Cialis is known to as effective as within 30 minutes of taking it with effects stay long for as long as 36 hours.
Get All medicines with or without prescription,controlled drugs ,viagra and many more all at skyrxshop.com. viagra online We’ve all heard of Viagra. The wonder pill that treats erectile dysfunction. Erectile dysfunction or impotence as it is often called, affects men of all
Viagra – 1305 results like the GENERIC Viagra Price per Pill, GENERIC BRAND VIAGRA Sildenafil Citrate 100 Mg 35 Pills, PFIZER Viagra 100mg – 30 Tablets, order viagra Rezeptpflichtige Medikamente viagra levitra levitra propecia Viagra levitra levitra Apotheker verschreibungspflichtige Arzneimittel Cialis Kaufen

braskevasmade (Email) - 15 October '08 - 19:07

atwater radio
pa radio stations
wdad radio
[[http://euaitrm.jvl.com/of9976.html]] mankato lawn mower radio
[[http://nouibic.freewebpages.org/of4369..]] radio broadcasting swear words
[[http://vidagh.007sites.com/in799.html]] atwater reed radio

LYPaul (Email) (URL) - 15 October '08 - 20:08

oWRvMM evurwuzjebfm, [url=[[http://cbdwyuifqxck.com/]cbdwyuifqxck[..]]], [link=[[http://pujpchiovbqk.com/]pujpchiovbqk[..]]], [[http://poxiubekvgkw.com/]]

ozkhwv (Email) (URL) - 12 November '08 - 15:26

[[http://floristic.friko.pl/flomaxafd.ht..]] flomax pbG [[http://floristic.freehostingz.com/flom..]] flomax P flomax [ [[http://members.lycos.nl/floristic/flom..]] ] flomax [link=[[http://members.lycos.nl/floristic/flom..]]]] flomax [/link] e)%0 [[http://floristic.007sites.com/geodonf1..]] geodon CKEA!lE(4 [[http://floristic.hostaim.com/flomax328..]] flomax

__
[[http://recipe-for.chelendger.co.cc/]] ]recipe for corned beef

Aflofesogy (Email) (URL) - 12 November '08 - 22:49

HC4hC6 bjeukfooylie, [url=[[http://kqpfbrgjyran.com/]kqpfbrgjyran[..]]], [link=[[http://agxozegnvjol.com/]agxozegnvjol[..]]], [[http://okmfaafrrtua.com/]]

axrgonplm (Email) (URL) - 13 November '08 - 19:42

It’s going to take a while to really understand all the tools available in Solaris, and hopefully I won’t have many chances to try them out on real crash dumps.

Chris (Email) (URL) - 10 May '09 - 07:45

Hi! I read your post and found it really interesting. Concierge services like office concierge and corporate concierge are really a great demand these days by corporations especially the executives. Hope this service doesn’t become a habit.

Concierge (URL) - 29 June '09 - 03:54

Hello… I read the post and it is very good. Very helpful
Thanks alot

Nidhish (Email) - 31 July '09 - 08:13

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

Christian louboutin shoes (Email) (URL) - 27 October '09 - 02:44

Thank you very much!

ed hardy clothing (Email) (URL) - 09 November '09 - 00:34

Good article, thanks to the author to share. and i want to share ugg boots with you.
uggs boots on sale in this ugg store.

Billsmith (Email) (URL) - 15 November '09 - 14:32

Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very good！
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.

tiffany and co (Email) (URL) - 19 November '09 - 01:24

Good!

Abercrombie and Fitch (Email) (URL) - 25 December '09 - 00:53

Great,thanks for sharing this.

Aladdin (Email) (URL) - 30 December '09 - 01:45

The fact is, way less than 1% of us who ever engage in core analysis are actually going to have any real idea of what the hell we’re doing. And thats ok!

butterfly valves (Email) (URL) - 30 December '09 - 04:12

very good, thanks for post.

butterfly valves (Email) (URL) - 30 December '09 - 05:11

thanks for post.

brand clothing (Email) (URL) - 03 January '10 - 07:16

thanks

kamagra (Email) (URL) - 21 January '10 - 01:39

good read thanks

M65 Jacket (Email) (URL) - 21 January '10 - 01:39

many thanks

viagra cialis (Email) (URL) - 21 January '10 - 01:58

Good post! Thanks for your information! As Seen On TV

As Seen On TV (Email) (URL) - 30 January '10 - 06:05

Good post! Thanks for your information!

puma boots (Email) (URL) - 06 February '10 - 06:48

Thank you for your info ,welcome to read this info: [[http://www.goodpuma.com]]

Puma shoes (Email) (URL) - 22 February '10 - 07:46

style Classic Tall Uggs oh, so larruping of the ugg classic short 5825 i have seen. the feature of knit uppers (composed of a wool blend) ugg boots sale and a sheepskin sock liner for ultra comfort makes me so crazy and want ugg sale to mobilize all my desires, the functional buttons allow the shaft to be worn tall or folded over for a cute cuffed look and decorative buttons make this ugg boots more fashion and easier to suit your dress,

links of london (Email) (URL) - 10 March '10 - 08:07

lighting designer Adam Silverman. Paul Steinberg’s corrugated metal sets are fine, wholesale jordan shoes even though the first act wholesale jordan shoes looks an awful lot like the first act set for Alden’s recent wholesale nike shoes ENO Jenfa. wholesale jordan shoes Grime’s precariously perched, too tidy hut is particularly handsome, and the act two beach scene wholesale jordan shoes is beautiful in wholesale jordan shoes an Edward Hopperish fashion.

air jordans (Email) - 11 March '10 - 09:03

Hi! Google gerade im Web und bin auf dieser Homepage gelandet.
Respekt, haben Sie toll gemacht.
Ich wünsche Ihnen weiterhin viel Erfolg mit Ihre Seite und natürlich viele Besucher.
Hier habe ich mich sehr wohl gefühlt und werde wieder hierher zurückkommen.
Freue mich auf einen Gegenbesuch auf meiner Homepage

Volker (Email) (URL) - 21 March '10 - 19:04

thank you ,I have learned a lot.

Nike air max shoes sale (Email) (URL) - 30 April '10 - 20:24

Hi! Google gerade im Web und bin auf dieser Homepage gelandet.
Respekt, haben Sie toll gemacht.
[[http://www.trademic.com/product/index...]]

china wholesale (Email) (URL) - 05 May '10 - 02:30

P90x .It really is not expensive if you factor in the cost

of a gym membership,P90x workout . The cost for P90X is

about three months of a paid gym membership but you get to

keep the program foreverP90x . You can try many of the

online sites, but it will be the same as buying from the

company or a Beachbody Coach. Make sure you are getting

original DVD’s. People are selling copies all over. The

problem is how long will they last, P90x workout ,and you

truly need the exercise and nutrition guide to even follow

the program. You can go to any site or you can go to

www.p90xmall.com and click on products. P90x dvd You can

order directly from the site,P90x dvd.

p90x (Email) (URL) - 05 May '10 - 07:46

[[http://www.gucci-shoes-bags.com]]
[[http://www.guccinewshop.com]]
Dear friends welcome to our store: we have a specail offer now which is once you buy any product you can get a free gift as well, these gifts are in the gift area, you can free to choose and please add to the shopping cart, we will send it together with your purchased product. Thanks!

gucci (Email) (URL) - 07 May '10 - 03:38

hp nx6100 battery [[http://www.adapterlist.com/hp/nx6100.h..]]

laptop battery (Email) (URL) - 15 May '10 - 06:10

Dell d630 Battery [[http://www.globallaptopbattery.co.uk/d..]]

laptop batteries (Email) (URL) - 15 May '10 - 06:17

Buy Nike Air Max 90 Shoes just $45-55 USD in[[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

nike air max 90 shoes (Email) (URL) - 28 May '10 - 02:23

Buy Nike Air Max 90 Shoes just $45-55 USD in [[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

nike air max 90 shoes (Email) (URL) - 28 May '10 - 05:36

thank you for sharing the article.
The easy-to-use Video Converter for Mac lets you to enjoy your videos on all sorts of palyback including PSP, iPod, Mobile Phone, Zune, iPhone, Apple TV and MP4/MP3 player.
Free download supported. [[http://www.videoconverterformac.com]]

video converter for mac (Email) (URL) - 28 May '10 - 05:46

[[http://www.sxhgz.com]] : nike shoes

puma shoes (Email) (URL) - 21 June '10 - 01:57

[[http://www.mbtshoeslatest.com]]
[[http://www.nikeairmaxshoe.com]]

Nike air max shoe (Email) (URL) - 24 June '10 - 07:55

Your article are great, will visit very soon. there is a site: [[http://www.lovemypursemall.com]]

designer bags (Email) (URL) - 07 August '10 - 14:03

The first layer of MBT Shoes means a high quality MBT Footwear cowhide after processing of machine. The main feature of MBT first layer cowhide: Masai Shoes surface layer are made of grain materials with close woven fibers; MBT Sandals feature smooth feeling, good strength and great abrasive resistance. MBT Medical Shoes lining are composed of suede with thick fiber, big diameter and flocking wool on the surface. Comparing with second layer of Discount MBT Sheos, first layer of MBT Shoes Cheap with more smooth surface and better quality, can be worn longer time. Our website launches MBT Outlet with first layer cowhide, so you can feel free to shop here and get satisfied MBT Shoes Clearance, including MBT Sawa, MBT Sirima, MBT Kaya, MBT Changa and so on .
Welcome to our website: [[http://www.mbtshoesmasai.com]]

mbtshoesmasai (Email) (URL) - 09 August '10 - 02:53

[url=[[http://www.cheapjerseyschina.com]disco..]] nfl jerseys from china[/url]
[url=[[http://www.cheapjerseyschina.com]disco..]] nfl jerseys wholesale[/url]

tanglnog (Email) (URL) - 10 August '10 - 02:12

Buy a piece of ghd for yourself. Come and join us [[http://www.ghdiron-outlet.com/]] to win the cheap ghd.

ghd outlet (Email) (URL) - 13 August '10 - 03:39

As we know, now GHD are loved by more and more people, which will save up to 45%.welcome to [[http://www.ghdoutlet-au.com/.]].

ghd straighteners (Email) (URL) - 13 August '10 - 03:40

Come and join us [[http://www.ghdoutlet-uk.com/index.php]] to win the ghd iv styler.

ghd hair straighteners (Email) (URL) - 13 August '10 - 03:41

very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff [[http://www.ucoolstuff.com]] cool gifts, unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you

cool stuff (Email) (URL) - 25 August '10 - 08:25

excellent article , I added you to my [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!

China Wholesale Directory (Email) (URL) - 25 August '10 - 08:27

Top Online Stores is a SEO Friendly [[http://www.toponlinestores.org]] free directory where you can find the best online shopping stores selected by hand and sorted by category [[http://www.china-wholesale-directory.c..]] china wholesale .

link directory (Email) (URL) - 14 September '10 - 07:35

There are various kinds of ghd hair straighteners on hot sell. GHD styler specialized in ghd australia market and ghd outlet. It provides many cheap ghd, ghd iv styler, pink ghd, purple ghd, which save up to 50%, with free shipping!

ghd outlet (Email) (URL) - 25 October '10 - 05:22