Solaris Core Analysis, Part 2: Solaris CAT

06 Sep '08 - 09:58 by benr

In Part 1 we discussed core analysis in general and some basic mdb commands for high level investigation. When you dig deeper things can get confusing and complex because everything is referenced by address. This is where the Solaris Crash Analysis Tool comes in.

Solaris CAT has been around for a long time, but only as of version 5.0 released on June 18th of this year has it been available for Solaris X86/X64. You can find the Solaris CAT 5.0 Release Notes here.

To get started, download CAT 5.0, uncompress and install the package:

# bunzip2 SUNWscat5.0-GA-i386.pkg.bz2
# pkgadd -G -d ./SUNWscat5.0-GA-i386.pkg 

The following packages are available:
  1  SUNWscat     Solaris Crash Analysis Tool (5.0 GA SV4622M)
                  (i386) 5.0

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]: 1

Processing package instance  from 

Solaris Crash Analysis Tool (5.0 GA SV4622M)(i386) 5.0
...

The package will, by default, install into /opt/SUNWscat. There are two binaries we're really interested in, found in the bin/ directory: scat and blast. The scat tool is the CLI interface to Solaris CAT and provides a shell which is a human friendly re-implementation of mdb (no "::" prefixing commands, etc.) The blast tool is a really nice Java GUI interface to the CLI which adds a lot of "just click here" functionality and is excellent for testing and playing around. I highly recommend you point your browser at /opt/SUNWscat/docs/index.html, which includes some minimal but extremely useful HTML documentation.

Authors note: I'm resisting a "scat" joke with amazing strength. Seriously... resisting.... so.... hard....

We'll focus on the CLI here. Invocation is a little unusual; add /opt/SUNWscat/bin to your path and then change to the directory containing your dumps (usual /var/crash/hostname/), for the .0 dumps use "scat 0", for the .1 dumps use "scat 1", and so on. You'll fine the "online help" within the CLI exceptional, lets look:

# export PATH=$PATH:/opt/SUNWscat/bin
# cd /var/crash/ev2-r01-s10/
# ls -l
total 14205330
-rw-r--r--   1 root     root           2 Aug 25 07:49 bounds
-rw-r--r--   1 root     root     1444762 Aug 25 07:43 unix.0
-rw-r--r--   1 root     root     7268106240 Aug 25 07:49 vmcore.0
# scat 0

  Solaris[TM] CAT 5.0 for Solaris 11 64-bit x86
    SV4622M, Jul  3 2008

  Copyright © 2008 Sun Microsystems, Inc. All rights reserved.
  Use is subject to license terms.

  Feedback regarding the tool should be sent to SolarisCAT_Feedback@Sun.COM
  Visit the Solaris CAT blog at http://blogs.sun.com/SolarisCAT

opening unix.0 vmcore.0 ...dumphdr...symtab...core...done
loading core data: modules...symbols...ctftype: unknown type struct panic_trap_info
CTF...done

core file:      /var/crash/xxxxxxxx/vmcore.0
user:           Super-User (root:0)
release:        5.11 (64-bit)
version:        snv_67
machine:        i86pc
node name:      xxxxxxxxxxxxxxxxxx
system type:    i86pc
hostid:         xxxxxxxx
dump_conflags:  0x10000 (DUMP_KERNEL) on /dev/dsk/c0t0d0s1(24.0G)
time of crash:  Mon Aug 25 07:41:00 GMT 2008 (core is 13 days old)
age of system:  91 days 22 hours 49 minutes 50.97 seconds
panic CPU:      1 (8 CPUs, 31.9G memory)
panic string:   page_free pp=ffffff0007243bd8, pfn=11228e, lckcnt=0, cowcnt=0 slckcnt = 0

sanity checks: settings...vmem...
WARNING: FSS thread 0xffffff097d1e3400 on CPU2 using 99%CPU
WARNING: FSS thread 0xffffff09fddbab40 on CPU3 using 99%CPU
sysent...clock...misc...
NOTE: system has 54 non-global zones
done
SolarisCAT(vmcore.0/11X)> 

When CAT is unleashed on a dump several "sanity checks" are run which can point out glaring known issues. There is an HTML document in the docs/ directory which outlines all the various sanity checks. These checks alone make CAT a must-have tool! Sanity check output will come in two varieties, "WARNING" which indicates something out of whack that may have been the cause or contributor to the crash, and "NOTE" which is unlikely the cause but of interest. We can see in the example above two warnings telling me that 2 threads were consuming 99% of a CPU... thats handy! It also notes that I'm running 54 zones.

The available commands a broken down into categories which you can see using the "help" command. The first group are for "Initial Investigation:" and include: analyze, coreinfo, msgbuf, panic, stack, stat, and toolinfo. Lets look at the "analyze" commands output:

SolarisCAT(vmcore.0/11X)> analyze

core file:      /var/crash/xxxxxx/vmcore.0
user:           Super-User (root:0)
release:        5.11 (64-bit)
version:        snv_67
machine:        i86pc
node name:      xxxxxx
system type:    i86pc
hostid:         xxxxx
dump_conflags:  0x10000 (DUMP_KERNEL) on /dev/dsk/c0t0d0s1(24.0G)
time of crash:  Mon Aug 25 07:41:00 GMT 2008 (core is 13 days old)
age of system:  91 days 22 hours 49 minutes 50.97 seconds
panic CPU:      1 (8 CPUs, 31.9G memory)
panic string:   page_free pp=ffffff0007243bd8, pfn=11228e, lckcnt=0, cowcnt=0 slckcnt = 0


==== panic thread: 0xfffffffef4ce5dc0 ==== CPU: 1 ====
==== panic user (LWP_SYS) thread: 0xfffffffef4ce5dc0  PID: 10156  on CPU: 1 ====
cmd: /opt/local/sbin/httpd -k start
t_procp: 0xffffffff06595e50
  p_as: 0xffffffff093490e0  size: 47374336  RSS: 3125248
  hat: 0xffffffff092a9480  cpuset: 1
  zone: address translation failed for zone_name addr: 8 bytes @ 0x3

t_stk: 0xffffff00486bcf10  sp: 0xffffff00486bc880  t_stkbase: 0xffffff00486b8000
t_pri: 3(FSS)  pctcpu: 0.380035
t_lwp: 0xfffffffefe61ab60  lwp_regs: 0xffffff00486bcf10
  mstate: LMS_SYSTEM  ms_prev: LMS_SYSTEM
  ms_state_start: 2 minutes 31.229022230 seconds earlier
  ms_start: 2 minutes 31.343582414 seconds earlier
psrset: 0  last CPU: 1  
idle: 0 ticks (0 seconds)
start: Mon Aug 25 07:41:00 2008
age: 0 seconds (0 seconds)
syscall: #131 memcntl(, 0x0) ()
tstate: TS_ONPROC - thread is being run on a processor
tflg:   T_PANIC - thread initiated a system panic
        T_DFLTSTK - stack is default size
tpflg:  TP_MSACCT - collect micro-state accounting information
tsched: TS_LOAD - thread is in memory
        TS_DONT_SWAP - thread/LWP should not be swapped
        TS_RUNQMATCH
pflag:  SMSACCT - process is keeping micro-state accounting
        SMSFORK - child inherits micro-state accounting

pc:      unix:vpanic_common+0x13b:  addq   $0xf0,%rsp

unix:vpanic_common+0x13b()
unix:panic+0x9c()
unix:page_free+0x22e()
unix:page_destroy+0x100()
genunix:fs_dispose+0x2e()
genunix:fop_dispose+0xdc()
genunix:pvn_getdirty+0x1f0()
zfs:zfs_putpage+0x129()
genunix:fop_putpage+0x65()
genunix:segvn_sync+0x39f()
genunix:as_ctl+0x1f2()
genunix:memcntl+0x709()
unix:_syscall32_save+0xbf()
-- switch to user thread's user stack --

This output provides a vast array of useful details, including:

  • System summary, including OS release and version, architecture, hostname, and hostid; as well as number of CPU's and memory
  • Time of crash and previous uptime ("age of system")
  • The panic string and CPU that it occurred on
  • The thread that caused the panic and its details, including the command (argc &argv), its memory footprint (size & rss), and zone
  • The threads state information, run time, start time, current syscall
  • The call stack

As noted in Part 1, what most people are really looking for when doing core analysis is to determine which application was responsable, and this output provides that data in great clarity. Lets dig into it a bit more explicitly... based on the above "analyze" output we can see that....

  • The system is an 8CPU X86 box running snv_67 (Solaris Nevada Build 67) in 64bit mode with 32GB of RAM.
  • System crashed on Aug 25th at 7:41AM GMT, it was previously up for 91 days
  • System paniced on "page_free" call, on CPU 1
  • The running thread was "httpd -k start"... an Apache worker process.
  • The process had the PID 10156, consumed 3.1MB of Physical Memory (RSS) and had a virtual size of 47MB
  • The process was using less than 1% (pctcpu) of CPU 1, was using the Fair Share Scheduler (FSS), on Processor Set (psrset) 0.
  • The process started on Aug 25th at 7:41AM GMT, it was 0 seconds old when it crashed... possibly a forked worker gone bad.

For many administrators this might be as much as you wanted to know, right there. But lets look at a couple more commands.

You'll recall that during the sanity checks at startup it noted 2 threads consuming full CPU's. We can feed the thread address to the "thread" command to get details on them:

SolarisCAT(vmcore.0/11X)> thread 0xffffff097d1e3400
==== user (LWP_SYS) thread: 0xffffff097d1e3400  PID: 27446  on CPU: 2 ====
cmd: nano svn-commit.tmp
t_procp: 0xffffffff2e908ab0
  p_as: 0xffffffff10402ee0  size: 2772992  RSS: 1642496
  hat: 0xffffffff102f6b48  cpuset: 2
  zone: address translation failed for zone_name addr: 8 bytes @ 0x2

t_stk: 0xffffff004e47ef10  sp: 0xffffff003d3fcf08  t_stkbase: 0xffffff004e47a000
t_pri: 26(FSS)  pctcpu: 99.306175
t_lwp: 0xffffffff202a78b0  lwp_regs: 0xffffff004e47ef10
  mstate: LMS_SYSTEM  ms_prev: LMS_USER
  ms_state_start: 2 minutes 31.228983791 seconds earlier
  ms_start: 39 days 19 hours 11 minutes 8.989252296 seconds earlier
psrset: 0  last CPU: 2  
idle: 9 ticks (0.09 seconds)
start: Wed Jul 16 12:30:07 2008
age: 3438653 seconds (39 days 19 hours 10 minutes 53 seconds)
syscall: #98 sigaction(, 0x0) ()
tstate: TS_ONPROC - thread is being run on a processor
tflg:   T_DFLTSTK - stack is default size
tpflg:  TP_TWAIT - wait to be freed by lwp_wait
        TP_MSACCT - collect micro-state accounting information
tsched: TS_LOAD - thread is in memory
        TS_DONT_SWAP - thread/LWP should not be swapped
        TS_RUNQMATCH
pflag:  SMSACCT - process is keeping micro-state accounting
        SMSFORK - child inherits micro-state accounting

pc:      unix:panic_idle+0x23:  jmp    -0x2     (unix:panic_idle+0x23)

unix:panic_idle+0x23()
0xffffff003d3fcf60()
-- error reading next frame @ 0x0 --

So using the "thread" command we can get full granularity on a given thread. In fact, using the "tlist" command you can dump this information for every thread on the system at the time of crash.

Another nifty command is "tunables". This will display the "current value" (at time of the dump) and the default value. If someone's been experimenting on the production systems this will clue you in.

SolarisCAT(vmcore.0/11X)> tunables   
    Tunable Name     Current   Default Value  Units      Description
                     Value                               
    physmem          8386375   *              pages      Physical memory 
                                                         installed in system.
    freemem          376628    *              pages      Available memory.
    avefree          338943    *              pages      Average free memory 
                                                         in the last 30 seconds
.........

Using the "dispq" command we can look at the dispatch queues (run queue). This answers "what other processes were running on CPU at the time of the crash", again, using the thread address we can dig into them with "thread":

SolarisCAT(vmcore.0/11X)> dispq
      CPU                  thread               pri        PID cmd
  0 @ 0xfffffffffbc26bb0   0xffffff003d005c80    -1            (idle)
               pri  60 -=> 0xffffff004337dc80    60          0 sched
  1 @ 0xfffffffec6634000 P 0xfffffffef4ce5dc0 P   3      10156 /opt/local/sbin/httpd -k start
  2 @ 0xfffffffec662f000   0xffffff097d1e3400    26      27446 nano svn-commit.tmp
  3 @ 0xfffffffec66f4800   0xffffff09fddbab40    25      21329 java -jar xxxxx.jar --ui=console
  4 @ 0xfffffffec66ea800   0xffffff003d414c80    -1            (idle)
               pri  60 -=> 0xffffff0048b12c80    60          0 sched
  5 @ 0xfffffffec6770800   0xffffff003d4b0c80    -1            (idle)
  6 @ 0xfffffffec6770000   0xffffff003d53bc80    -1            (idle)
  7 @ 0xfffffffec6762000   0xffffff003d58fc80    -1            (idle)

      part                 thread               pri        PID cmd
  0 @ 0xfffffffffbc4eef0

There are far too many to go through in a blog entry... but lets look at my personal favorite, "zfs". The "zfs" command can show us the pool(s), their configuration, read/write/checksum/error stats, and even ARC stats!

SolarisCAT(vmcore.0/11X)> zfs -e
ZFS spa @ 0xfffffffec6c21540
    Pool name: zones
    State: ACTIVE
       VDEV Address      State    Aux   Description
    0xfffffffec0a9e040  FAULTED    -       root

            READ   WRITE   FREE   CLAIM   IOCTL  
    OPS        0      0     0      0      0 
    BYTES      0      0     0      0      0 

    EREAD       0
    EWRITE      0
    ECKSUM      0

            VDEV Address      State    Aux     Description
         0xfffffffec0a9eac0  FAULTED    -    /dev/dsk/c0t1d0s0

                  READ      WRITE     FREE   CLAIM   IOCTL  
         OPS     74356305  578263155     0      0      0 
         BYTES       757G      10.4T     0      0      0 

         EREAD       0
         EWRITE      0
         ECKSUM      0
SolarisCAT(vmcore.0/11X)> zfs arc

ARC (Adaptive Replacement Cache) Stats:

    hits                       77708247444
    misses                         1930348
    demand_data_hits           74303514929
    demand_data_misses             1325511
    demand_metadata_hits         620388795
    demand_metadata_misses          160708
    prefetch_data_hits          1361651307
....

I hope this helps you get an idea of how easy it is to really dig deeply into your core dumps using Solaris CAT to hide the oddities of mdb from you. Its a powerful and robust tool, and I'm glad that we have it.

Happy dump divin'! You'll be amazed how much you'll learn about your system.


- - C O M M E N T S - -

Unless things have changed majorly, SCAT is not an mdb wrapper, but a re-implementation.

John Levon (Email) - 07 September '08 - 13:44

John: Noted, fixed.

benr - 07 September '08 - 16:24

Good article … this toolkit is really useful for a first assessment of a crash

BTW: I wrote an tutorial about crash dump/core dump handling a few weeks ago: [[http://www.c0t0d0s0.org/archives/4391-..]]

Joerg M. (Email) (URL) - 07 September '08 - 17:42

Can this be used for core files other than those resulting from kernel panics. I often find mysterious core files in my home directory, and don’t notice that anything has stopped working. I usually end up deleting them – I know prolly not the best idea. It would be cool to be able to figure out what exactly did in fact dump core.

If a core file shows up in a user’s directory, does it also show up in /var/crash or no?

Joseph Sniderman (URL) - 07 September '08 - 19:58

You can configure a more speaking name with coreadm … please look into the tutorial mentioned before for more information.

Joerg M. (Email) (URL) - 08 September '08 - 09:46

I would be a little conservative about saying “finding out the offending process”

All panic must happen from kernel, so which user space processes are running is, at most, a small clue of what could be happending or may be irrelavent at all.

In your example from the kernel stack trace looks like it crashed from ZFS ( VFS??) operatiions, so the “fork gone bad” is mostly likely wrong.

Sean - 08 September '08 - 15:26

Joerg M. asked about core files in the user home dir. Perhaps I misundstand the question, but those are most likely application cores. Just use mdb name-of-core-file (usually just core unless you force one out with gcore).
in mbd just use ::status and you will see where that core came from.
Example:

[root@host:~]# mdb core.13873
Loading modules: [ libc.so.1 libuutil.so.1 ld.so.1 ]
> ::status
debugging core file of httpd (32-bit) from host
file: /path/to/httpd
initial argv: /path/to/httpd
threading model: multi-threaded
status: process core file generated with gcore(1)
>

TimM (Email) - 08 September '08 - 21:01

на DVD купить фильмы сериалы новинки видео игры mp3 диски, магазин
[[http://dvd-market.com.ua/]]

[url=[[http://dvd-market.com.ua]новинк..]]]новинки mp3 видео[/url]
[url=[[http://dvd-market.com.ua]хентай,..]]]]

dvdsghglks (Email) (URL) - 11 October '08 - 20:59

video235 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video349 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video137 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video286 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video510 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video114 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video566 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video397 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video668 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video586 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video153 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video196 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video520 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video653 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video559 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video420 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video214 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video209 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video417 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video122 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video231 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video157 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video43 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video38 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video48 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video393 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video87 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video567 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video147 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video475 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video194 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video428 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video4 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video299 [url=[[http://my.opera.com/skazzi9/blog/show...]]]]
video341 [url=[[http://my.opera.com/skazzi9/blog/show...]]]]
video20 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video100 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video95 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video171 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video78 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video583 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video431 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video337 [url=[[http://my.opera.com/skazzi9/blog/2008/..]]]]
video329 [url=[[http://my.opera.com/wearerobots/blog/2..]]]]
video593 [url=[[http://my.opera.com/skazzi9/blog/show...]]]]

Amelia (Email) (URL) - 13 October '08 - 14:10

pics479 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics132 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics268 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics417 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics69 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics335 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics276 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics224 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics184 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics201 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics20 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics26 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics265 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics54 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics433 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics10 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics22 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics487 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics05 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics90 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics525 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics193 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics212 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics454 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics399 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics310 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics87 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics473 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics169 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics74 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics21 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics334 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics91 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics78 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics132 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics495 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics149 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics103 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics267 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics56 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics52 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics198 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics02 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics19 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]
pics328 [url=[[http://my.opera.com/miramiar/blog/2008..]]]]

Calvin (Email) (URL) - 14 October '08 - 23:04

Good post.

Perdita (Email) (URL) - 07 November '08 - 10:06

Keep up the good work.

Marilyn (Email) (URL) - 09 November '08 - 19:29

H47q06 pseuyrxytcnv, [url=[[http://megbycqwlqvr.com/]megbycqwlqvr[..]]], [link=[[http://vhbkpbywyduc.com/]vhbkpbywyduc[..]]], [[http://szhfqosfidab.com/]]

xnrwomp (Email) (URL) - 12 November '08 - 19:16

Hi, Great post! I felt great reading your blog post. I’m working with my friend in a small internet marketing business as a web developer. When I’m free I go around for some IT info

Jeff Paul Big League Players Club (Email) (URL) - 15 April '09 - 07:05

All panic must happen from kernel, so which user space processes are running is, at most, a small clue of what could be happending or may be irrelavent at all.

Chris (Email) (URL) - 10 May '09 - 07:40

most people are really looking for when doing core analysis is to determine which application was responsable, and this output provides that data in great clarity

apri (Email) (URL) - 12 July '09 - 19:08

[[http://www.saleveling.com]]
[[http://www.power-leveling-game.com]]

wow power leveling (Email) (URL) - 23 October '09 - 07:13

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

Christian louboutin shoes (Email) (URL) - 27 October '09 - 03:11

Thank you very much!

ed hardy clothing (Email) (URL) - 09 November '09 - 00:32

Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very good!
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.

tiffany and co (Email) (URL) - 19 November '09 - 01:22

Thank you so much!

Aladdin (Email) (URL) - 30 December '09 - 02:14

When I’m free I go around for some IT info

ED Hardy (Email) (URL) - 04 January '10 - 06:21

Thank you so much!

UGG Ultra Tall boots (Email) (URL) - 14 January '10 - 09:05

thanks

kamagra (Email) (URL) - 21 January '10 - 01:38

good read thanks

M65 Jacket (Email) (URL) - 21 January '10 - 01:38

many thanks

viagra cialis (Email) (URL) - 21 January '10 - 01:58

New battery review module surface is black, not because of the positive electrode exists, the module is likely to be integrated back contact silicon solar cells, silicon, where the role is often to extend the service life of equipment.Mitsubishi motors in October 2009 28-30 at the yokohama exhibition center held the Pacific “GreenDevice2009″ displayed on the high efficiency solar camera battery 19.1% respectively. In the square 15cm in polysilicon solar cells, realized the highest efficiency. The research achievements in 2009 September 2009 and the application of physics to the international society in October EUPVSEC “, “said in a statement.

battery review (Email) (URL) - 30 January '10 - 03:34

good sharing !!!
nike air max

nike airmax (Email) - 12 March '10 - 07:25

good sharing !!!
nike air max
[url= [[http://http://www.airmax-shox.com/en/c..]]]]
[[http://www.airmax-shox.com/en/category..]] nike air max

nike airmax (Email) - 12 March '10 - 07:25

MTS Converter software can convert MTS to video formats (AVI, MP4, WMV, MOV, 3GP, FLV, DivX, XviD, MPEG, etc.), audio formats (MP3,

WMA, WAV, AAC, AC3, OGG, etc.), HD video formats (HD MOV, HD VOB, HD MP4, HD AVI, etc.) and various devices (iPod, iPhone, Apple

TV, PSP, PS3, Xbox, Zune, Wii, BlackBerry, mobile phone, etc.)

sunny (Email) (URL) - 14 April '10 - 06:52

I very intersted in the article

cheap mbt shoes (Email) (URL) - 23 April '10 - 03:35

thank you ,I have learned a lot.

Nike air max shoes sale (Email) (URL) - 28 April '10 - 17:12

P90x .It really is not expensive if you factor in the cost

of a gym membership,P90x workout . The cost for P90X is

about three months of a paid gym membership but you get to

keep the program foreverP90x . You can try many of the

online sites, but it will be the same as buying from the

company or a Beachbody Coach. Make sure you are getting

original DVD’s. People are selling copies all over. The

problem is how long will they last, P90x workout ,and you

truly need the exercise and nutrition guide to even follow

the program. You can go to any site or you can go to

[[http://www.p90xmall.com]] and click on products. P90x dvd You can

order directly from the site,P90x dvd.

p90x (Email) (URL) - 05 May '10 - 05:57

A good idea! [[http://www.offerreplicawatches.com.The..]] sell Chanel Watches, Replica BRM Watches,offer Cartier Watches, Copy Ebel Watches, Replica Hublot Watches, Fake Tag Heuer Watches, wholesale Richard Mille Watches.

They also sell some very fashion watches, such as Montblanc Watches for sale,Chopard Watches, Dior Watches and Gucci Watches. Rolex watches are hot sell. If u are Armani Watches and Breitling Watches fan. You can still look at Omega Watches and Longines Watches, many new arrivals watches just come here.You can have a try on 58338312391273913. :[url=[[http://www.offerreplicawatches.com]www..]]].

Replica Chanel Watches,Dvd Boxes (Email) (URL) - 12 May '10 - 07:35

They also sell some very fashion watches, such as Montblanc Watches
[url=[[http://www.mydiscountjordanshoes.com/a..]]]] air-jordan-xv

[/url]

air jordan xv (Email) (URL) - 15 May '10 - 03:35

dell gd761 battery [[http://www.adapterlist.com/dell/gd761...]]

laptop battery (Email) (URL) - 15 May '10 - 05:33

Toshiba pa3465u-1brs Battery [[http://www.globallaptopbattery.co.uk/t..]]

laptop batteries (Email) (URL) - 15 May '10 - 05:55

Buy Nike Air Max 90 Shoes just $45-55 USD in[[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

nike air max 90 shoes (Email) (URL) - 28 May '10 - 02:22

Buy Nike Air Max 90 Shoes just $45-55 USD in [[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

nike air max 90 shoes (Email) (URL) - 28 May '10 - 05:37

thank you for sharing the article.
The easy-to-use Video Converter for Mac lets you to enjoy your videos on all sorts of palyback including PSP, iPod, Mobile Phone, Zune, iPhone, Apple TV and MP4/MP3 player.
Free download supported. [[http://www.videoconverterformac.com]]

video converter for mac (Email) (URL) - 28 May '10 - 05:46

thank you very much !!

wholesale sunglasses (Email) (URL) - 23 June '10 - 15:36

[[http://www.mbtshoeslatest.com]]
[[http://www.nikeairmaxshoe.com]]

Nike air max shoe (Email) (URL) - 24 June '10 - 07:52

Do you like the ugg ? Ugg Classic Short Boots are UGG Australia’s original heritage styles. Ugg Classic Tall will keep your feet dry and the ultimate comfortable.
[[http://www.ugg-boots-london.com/ugg-cl..]]

Ugg Classic Short (Email) (URL) - 07 July '10 - 01:25

The DS R4 is the most popular version and the R4 card,r4i card,m3i zero,nintendo ds r4 card,r4i dsi,r4 card for ds has the capacity to store 2GB,4GB,8GB. This R4 card with amazing price and easy to use

r4cardsohot (Email) (URL) - 29 July '10 - 02:55

Your article are great, will visit very soon. there is a site: [[http://www.lovemypursemall.com]]

designer bags (Email) (URL) - 07 August '10 - 14:02

The first layer of MBT Shoes means a high quality MBT Footwear cowhide after processing of machine. The main feature of MBT first layer cowhide: Masai Shoes surface layer are made of grain materials with close woven fibers; MBT Sandals feature smooth feeling, good strength and great abrasive resistance. MBT Medical Shoes lining are composed of suede with thick fiber, big diameter and flocking wool on the surface. Comparing with second layer of Discount MBT Sheos, first layer of MBT Shoes Cheap with more smooth surface and better quality, can be worn longer time. Our website launches MBT Outlet with first layer cowhide, so you can feel free to shop here and get satisfied MBT Shoes Clearance, including MBT Sawa, MBT Sirima, MBT Kaya, MBT Changa and so on .
Welcome to our website: [[http://www.mbtshoesmasai.com]]

mbtshoesmasai (Email) (URL) - 09 August '10 - 02:53

[url=[[http://www.cheapjerseyschina.com]disco..]] nfl jerseys from china[/url]
[url=[[http://www.cheapjerseyschina.com]disco..]] nfl jerseys wholesale[/url]

tanglnog (Email) (URL) - 10 August '10 - 02:13

Buy a piece of ghd for yourself. Come and join us [[http://www.ghdiron-outlet.com/]] to win the cheap ghd.

ghd outlet (Email) (URL) - 13 August '10 - 03:39

As we know, now GHD are loved by more and more people, which will save up to 45%.welcome to [[http://www.ghdoutlet-au.com/.]].

ghd straighteners (Email) (URL) - 13 August '10 - 03:40

Come and join us [[http://www.ghdoutlet-uk.com/index.php]] to win the ghd iv styler.

ghd hair straighteners (Email) (URL) - 13 August '10 - 03:41

excellent article , I added you to my [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!

China Wholesale Directory (Email) (URL) - 25 August '10 - 08:15

very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff [[http://www.ucoolstuff.com]] cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you

cool stuff (Email) (URL) - 25 August '10 - 08:16

Top Online Stores is a SEO Friendly [[http://www.toponlinestores.org]] free directory where you can find the best online shopping stores selected by hand and sorted by category [[http://www.china-wholesale-directory.c..]] china wholesale .

link directory (Email) (URL) - 14 September '10 - 07:25

We wholesle many styles of bodybuilding.They are very comfotable and easy to macth with all kinds of clothes.
We wholesle many styles of bodybuilding.They are very comfotable and easy to macth with all kinds of clothes.
[url= [[http://blog.bodybuilding.com/]bodybuil..]]]] are very suitable for men who like doing sport.

bodybuilding (Email) (URL) - 14 September '10 - 09:15

discount tiffany jewelry

discount tiffany jewelry (Email) (URL) - 02 October '10 - 00:00

Find cheap tiffany Jewelry at [[http://www.silver-bulk.com]]

discount tiffany jewelry (Email) (URL) - 02 October '10 - 00:01

There are various kinds of ghd hair straighteners on hot sell. GHD styler specialized in ghd australia market and ghd outlet. It provides many cheap ghd, ghd iv styler, pink ghd, purple ghd, which save up to 50%, with free shipping!

ghd outlet (Email) (URL) - 25 October '10 - 05:22

very cool article ,thanks for sharing the article

Gurudatta (Email) - 18 January '11 - 16:02

Personal information





Remember your information?
Comment

Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.


^M