Hate to say it, but thats hard!

When we understand to our satisfaction how something works and we can do it quickly we say “Thats easy.” It may or not be easy for anyone else, but in your mind it is. The converse is true, when something just doesn’t make sense or seems to take more work than you think it should we say “Thats hard.” As SysAdmin’s we are eager to say the the former and embarrassed to say the latter. General this embarrassement gives way to frustration and anger, and if we didn’t like whatever it is in the first place we just say that its stupid or poorly designed. Given that, theoretically, a sysadmins job is to use and implement technology, not to re-engineer it, our frustrations usually have more merit than vendors give us credit. The reality is that our jobs are too often about decrypting technology and there are two parties to blame: engineering (for not making it simpler in the first place) and techpubs (for not explaining plainly how to use the damn thing).

Within Solaris this epic struggle between engineering and techpubs is easy to see. My favorite example is network configuration in Solaris… its actually very elegant and well designed, but its counter-intuitive to most people, so bad bad tech pubs. Another example is ndd, its just crap, its difficult to use and easy to make mistakes, so bad bad engineering. Sometimes it comes down in the middle, such as Resource Control… different commands, funky syntax, no way to just dump out “here are my settings!” without digging through piles of man pages and mailing lists.

Given the option of re-engineering it or writing better docs, the latter is much easier to do and less time consuming. And thus, I ask….

What do you feel is “hard” in Solaris? What are those things you think you should understand but never really have?

I’ll admit that I struggled heavily with RBAC, with resource control, with auditing, and honestly I still can’t make any sense out of extended accounting. Even those of us who are called “guru’s” have plenty of weak spots, and that shouldn’t be supprising to anyone, Solaris is a really, really big operating system, as is any UNIX system.

So, have at it… what is too hard? With a good list those of us in the OpenSolaris Docs community can target our efforts to make the world a slightly better place to live and work in.

92 Responses to “Hate to say it, but thats hard!”

  1. acull says:

    Sometimes I wish sys-unconfig was more indepth ie: a place to set default DNS etc when I asked the open solaris IRC where I could find the code I had a silence that frustrated me. Sure I could wad through code for the rest of my life but I dont know where to start.

    I am slowly learning solaris 10 and find the most frustrating thing is you CAN NOT READ PDF’s from docs.sun under a default install of solaris 10. This is a oversite and needs to be fixed. Solaris 10 is heading in the right direction though in my opinion.

    Lack of documentation or poorly searched information on solaris 10 prevents most of the hardware problems to be solved. The community needs to band together and improve the information contained in the HCL including any known work arounds. Detailed information would have to include the small things that irritate most people like sys-suspend not working etc.

    ZFS is great but I have had errors that ZFS could not fix and produced a set of numbers that are useless to begining system admins. Hopefully this will be fixed soon and include more information on how to distroy the corruption. I just did a copy to ufs and all the bad files were skiped.

    have fun..
    (oh and your my inspiration at times)

  2. noyb says:

    Sun Cluster. Bleh. Why is it so hard? It could be one of the best things out there for Sun if it went like: Download, install on 2 or more boxes, pull wires out watch the the cluster NOT crash… but no.

    I believe the merits of Kerberos have been under-appreciated simply due to the fact that “it’s hard.” There are just lots of little things to trip over and the whole thing doesn’t work correctly if we miss just one tiny piece of the puzzle. That’s bad.

    LDAP is hard. Custom at every site it seems. That’s just lame. NIS and it’s big sister NIS+ are still in wide use because of this. (Shhhh… even at Sun.)

    Jumpstart. Tedious. With mysterious spells (keywords and custom scripts) to make certain install-time questions go away. Wasn’t Puppet supposed to fix all of this?

    Java and all of it’s layers of technologies, components, and plug-ins that are supposed to make things so much easier. Whatever. But then, Rouman Strobl could probably sell sand to Arabs.

    I thought Linear Algebra was a bit hard. Sorry.

  3. David Gwynne says:

    i wish patching wasnt so goddamn hard.

    ive written a driver for solaris, but the ins and out of actually running a machine on solaris are still largely magical to me. oh well, i’ll learn it eventually.

  4. Steff Davies says:

    Ooh, I’ve been waiting for this opportunity and I just didn’t know it until now. Here’s a short list (I come from a Debian and FreeBSD background, working mostly on small machines):

    LDAP-based login is a pain to set up. Despite it apparently working in the same way that it does on a Linux box, libnss-ldap doesn’t seem to be packaged anywhere, leading to a violent orgy of yak-shaving for what I’m used to being a ten-minute job.

    Package management generally seems to be a weakness. The Sun package format appears to do roughly what dpkg or half of the ports system does, but I’ve been unable to find an equivalent to apt or the dependency-following of ports. I’m aware that this is me being a lazy sod, but it feels like a retrograde step for simple installations of packaged software to become time-consuming as one traverses the dependency tree manually.

    Jumpstart I’ve barely touched, principally because I’ve watched it causing pain to several people vastly more clueful than me and am thus terrified of it.

    Setting up a root mirror wasn’t too bad, but did seem like a pretty fiddly experience for what is presumably an extremely common configuration – the debian installer makes a nice job of this.

    OTOH, Zones have been the easiest and nicest VM/Jail experience I have ever had, in part because of some nicely brief and informative docs.

  5. benr says:

    I’ve got to whole heartly agree with most of the points here. LDAP is one of those things you at some point have a “lightbulb” moment when you finally decript the OU’s and CN’s and such. Kerberos is just a nightmare.

    Now Sun Cluster is evil in this regard. It “seems” easy but isn’t. Especially when something fails; rebuilding a cluster after somethings failed is just painful. SC is one of those things that you have to do as a full time job or you forget all the syntax, ins and outs, really quickly.

    Root mirroring in SVM… hell, SVM in general is hard. Thats one of those technologies that was probly a dream come true back in the day, but has just become too archiac. “Submirror”? WTF does that mean? I know SVM really well and I _still_ think its too hard. Who can remember what d40 is? And how many times have you gone to use it, started to get excited and then realized you needed raw partitions for the db replica’s. Thankfully ZFS saved us from the torture.

  6. rob says:

    NIS+ is hard. Just too big and too complicated.

    RBAC is hard. Way too many things to configure, too scattered around, and arcane in every way it could be.

    projects and resouece controls are hard. See RBAC.

    Clustering can be hard because the cluster system takes some control away from you. It’s like another admin logged in from another box, who hates you.

    dtrace is hard because it’s so powerful. It’s difficult to know how to get the information you want, and how to interpret the information that comes back.

    SMF is hard at first because, for all its power and good features, it’s a hugely complicated, long-winded way of doing a simple thing we already knew how to do. All those svcprop commands can get very ugly, and the documentation was thin in the early days.

    Jumpstart’s easy, but it’s fragile and opaque. I always thought SVM was simple too, but I’ve struggled with the backwaters of VxVM over the years. With great power comes great complexity!

  7. Others have already got my favorites: RBAC and NIS+. Can’t agree with the one about dtrace though. It has the perfect learning curve; it just takes a little knowledge to get started, always more to learn and always worth it to learn more.

  8. BSM has been the hardest thing to use. It works well if you know what you are doing. The biggest problem is that there just isn’t much info out there. Not many people need to use it so it gets little attention. Trusted extensions would be another issue for me. I know Sun has a class coming out soon. I have lived in TSol-8 for a while, but somethings are a little different.

  9. Tom says:

    LDAP for sure.
    DHCP – the ISC server is very easy. Why go binary?
    SMF – I find it ok, but I’m not seeing faster startups.
    Account locking – NIS doesn’t. LDAP does. /etc/shadow on Sol 10 does but for > 2 systems it’s a pain
    luxadm/cfgadm – too many tools!
    ndd – too hardware specific w/ too little docs
    Solaris needs a standard apt/yum type repo. Sunfreeware does its packages in /usr/local. CSW has pkg-get that install into /opt/csw and the Companion disc is /opt/sfw. I might have 3 copies/versions of gcc and libraries that a user might use. Converge already,
    Combine pkgadd/rm/info/chk into something like rpm. And rpm -ql

  10. Karl says:

    Shared Memory drives me bananas. I have lost a lot of brain cells trying to figure out what needs to be set in /etc/system for multiple applications running on the same box.

    BUT sounds like Bill Moore is working on that.

  11. Kevin says:

    Solaris networking shows a lot of the warts, most apparently the reliance on ndd for /dev/tcp (rather than being able to use /etc/system) and the lack of a standard mechanism for specifying static routes. This makes for a perfect example of where Solaris tends to get far too focused on major overhauls and misses basics — for all of the (very worthwhile) effort in SMF, basic startup parameters like these two still go unaddressed.

  12. Frank says:

    Resource control is brutal. The docs are bad, there seem to be conflicting commands, and nothing feels at all organized…as if a bunch of stuff was just thrown in to hit bullet-points on a feature list.

    It definitely feels designed by committee…it’s the Pontiac Aztek of the Solaris world.

  13. Andrew says:

    I’ll chime in and agree that most things that have anything to do with package management are hard. Also, patching seems more difficult than it needs to be. Not difficult like SMF, or volume management, but still…

    I’ll throw in a few others too. Finding stuff on Solaris is hard. An application might be in /opt/sfw, /opt/csw, /usr, /usr/local/, /usr/xpg4. /usr/xpg6, etc. Using the Sun website to find documentation is hard; not terrible, but more difficult than it needs to be.

  14. Cass says:

    ACL’s not enough info, i continue to have to refer to a doc i wrote once i got it working for shared folders, why is man so sparse for getfacl/setfacl

    RBAC as already mentioned although i have succeeded in implementing some basic stuff, not the easiest ..

    Auditing, discovered praudit and love the way i can see what was run when and by whom, could be easier imho

    DTrace as someone already mentioned, damn its like a whole other language :-)

  15. Koteswara Rao says:

    Resource Management is difficult to grasp. I have been at it for the past 7 days and am just beginning to understand it…

  16. Paul says:

    Kerberos is actually pretty easy once you work out what’s going on at a reasonably high level. O’Reilly book makes it simple enough if you don’t need to know the internals (and the RFC’s pretty readable, not that I’m a Keb nerd or anything). The hard parts are the interoperability issues. Feel free to email me if you have Kerb questions.

    Yeah, LDAP is a PITA. Sun’s implementation doubly so. Hopefully they’ll start to fix it up as it roles out internally. This OSS tools are annoying, but beautiful in comparison.

    luxadm, now there’s a tool I love to hate. Brings me onto the subject of disks. Finding out what disk physically matches a logical disk is a really pain. So, c12tWWNd0 just died. I’ve got 2 FCAL paths to the disk and finding out which array and then which slot in the array can take a significant chunk of someones afternoon. It’s easy enough if you have the disks on a RAID array, but a JBOD isn’t pleasant.

    JET fixes a lot of Jumpstarts issues, but has (fewer, more minor) issues of it’s own.

    Patch and package management are huge issues. SCH is looking into those: http://www.opensolaris.org/jive/thread.jspa?threadID=39185&tstart=0

    Paul

  17. eike says:

    We use a lot of Solaris x86 with HP Servers, Solaris dont know the raidcontroller’s from HP. Thats not a big problem, but they made many modifications to the bootprocess in the last Sol10 Versions and if you use a driver not delivered with solaris, you will get many painfull problems with booting your server after updating Sol or HPdrivers. The Docs at Sun, were to old and descibed the old process.

    ZFS was easy with good docs but I never had a real failure.

    multipathd for network: documentation is a pain, configuring the options is hard. bonding on linux ist better.

  18. Luke says:

    I agree that package-management is a weakness on Solaris. The apt-get (Debian) and yum (Centos) models work very nicely as a unified package/patch management system. I guess I can use Nexenta if I really want that feature, but that might be a hard-sell at work.

    Both zones and ZFS are a real strengths and completely rock . The branded zones look like they will be extremely valuable too, especially given the weakness of the package management. With the branded zone, it looks like I can have ZFS and zones — and present a nice Centos-ish system with lots of prebuilt-vendor-supported-patched-daily open-source packages (like Dovecot+spamassassin+clamav) to the Open Internet. For a lightly maintained system like my computer at home, there really is no substitute for daily automatic patching — the cobbler’s children and everything.

    Also, since I located all of my zones on a ZFS pool on my machine at home(for safety and separation between the global zone and “my” data), I’m still doing my homework to see if there’s a Safe and easy way to do the upgrade to u4 — and my test-machine has been hanging, partway through the installation, when I try to boot it from the Solaris installation DVD. More often than not, searches for this kind of inforomation turn up links to “The Blog of Ben Rockwood”. :-)

  19. Ceri Davies says:

    Extended accounting is, indeed, mystifying. Collecting reams of data seems easy enough, but getting anything useful out of it…

    ndd sucks, but hard? Not really, although I now resent having to do anything that requires writing a script to go in rc.d/ (on which note, @kevin, static routes go in /etc/inet/static_routes although it’s a private interface and subject to change).

    IPMP was a little difficult, but that’s mainly because everyone claims that it requires things to be set up that it actually doesn’t, so I used to end up dragging out snoop half of the time.

  20. RNC says:

    tersly …

    there are web/gui based tools delivered with solaris, but finding them is hard. spattered across docs.

    all the add-on admin tools (inc N1) which ought to be delivered, streamlined and de-maxed (as it’s hard to figure the overlaps from the docs).

    and i’d agree with the other commenters on all points they make.

    how about product registry vs packages. what is/was that about ;-)

    implementation languages for tools. we *don’t* need java for that, despite the stock ticker. use shell or perl. we can read the scripts. the unix way.

    CIM management – are there tools, or should I get Windows for it ;-)

    could go on . . .

  21. Rich Teer says:

    The bane of my Solaris sysadmin life is setting up printing (from the command line)!

  22. rob says:

    Sorry for the two-part post! Feel free to delete. Anyway, as I was saying, admintool was a simple little chunk of C that did the job (for adding a printer at least.) Why we needed a towering edifice of Java to replace one small part of it is beyond me. The amount of packages you need to get a Solaris box printing these days is mind boggling too.

    As the moan of the day is packages, what particularly annoys me is how things like header files can only be installed in a global zone? Drivers I can understand, but header files? I want those in my workstation/compiler zone, why does my minimal SSH zone have to have them too?

  23. FWIW, the package registry was attempt #542 to “fix the packaging system” I think Indiana’s project is attempt #9721 or so…

  24. RBAC, because it’s a little arcane and keeps sudo in business. One could *write* a sudo replacement in the time it takes to learn RBAC properly, and that isn’t good.

    LDAP authentication, because you can follows the documentation to the letter and have it not work. And the docs change between Solaris releases.

    Don’t really think this is a documentation issue though, more like something that needs a new interface.

    As for other people’s comments on SunCluster – it *is* easy for a complex item of software.

  25. dominik says:

    Despite other peoples opinion, Ilike SunCluster and don’t think it’s very difficult. On the other hand I worked in the SC team at Sun for several years… :)

    Negatives are RBAC and resource management. They work quite well after having read docs.sun.com and lots of google search results for a few days but are very very un-intuitive. Especially Sun’s own documentation does not give examples for real-world implementations but wastes (virtual) pages to describe some basics.

    Setting up a non-postscript printer is a PITA first class. Won’t comment on this any more as it wakes up some very bad memories.

    SVM is a simple but nicely working tool. It just needs some system in naming the submirrors. I usually use d10 and d20 for the submirrors and d0 for the root mirror, d11/d21 for d1 etc etc. So the last digit in the submirror’s number represents the slice on the disk. You get the idea. Unfortunately, this system breaks when using LiveUpgrade (a very cool tool) to upgrade to a disk which is already prepared for SVM.

  26. spohl says:

    I find it particularly hard/tedious to create a minimized Solaris installation. If you start out with the smallest option available in the installation program and start adding the stuff you want, you have to manually sort out the dependencies. Only once I had the patience to completely do this. The system i ended up with even worked correctly… except for the smpatch stuff. But then again smaptch failed on me even on fully installed Solaris boxen.

  27. Adam says:

    LDAP: What I didn’t like the most was that sun told me that I could not be a client of myself. It took a few weeks and some playing and came up with this http://forum.java.sun.com/thread.jspa?threadID=5205825
    I haven’t had a problem doing it since. The documentation sucks, especially for Trusted Extensions and Ldap which is what I use primarily.

  28. David Strom says:

    Agree with the admin comments — Sun’s admin approach seems designed to sell more systems (Jumpstart, N1, etc.). Sun Management Console was not documented, and not supported, and I heard a rumor that it’s dead. Sun does not seem to care about smaller organizations, SMEs, workgroups & the like. What does it take to provide an easy, light (doesn’t require an app server), simple interface to admin users/groups/etc. for LDAP, so we can replace NIS? And a command line LDAP version of “useradd”?

    How come it’s “svcadm disable ” vs. “inetadm -d “? Is there something wrong with being consistent?

    The good — a Flash Archive install is fast, just create one minimal server config, use it to make a Flash Archive, and you’re good to do manual installs fast. And, the new install option — don’t run network services except sshd is great, and way overdue.

    Never had too much confusion with SVM, but I started out with the old DiskSuite version back in Solaris 2.1 days. Just be consistent with naming, use md.tab, and don’t forget the “set md:mirrored_root_flag=1″ line in /etc/system if you only have 2 disks.


    David

  29. nimelryes says:

    welcome to buy the good world of warcraft service. enjoy your wow powerleveling service here. you will be satisfy in wow power leveling.

  30. Mel26 says:

    Your parents, they give you your life, but then they try to give you their life.
    http://allmednews.org/med/pills/adipex/ – adipex http://allmednews.org/med/pills/ambien/ – ambien

  31. Google says:

    If you do not wish to receive similar messages please inform us on it by mail ban.site[dog]gmail.com

  32. williams brother pharmacy williams brother pharmacy [link=http://geraldinesboutique.com/about/media/2/news-233-20080612.html]williams brother pharmacy[/link]

  33. amc movie theater clifton commons amc movie theater clifton commons [link=http://geraldinesboutique.com/about/media/2/news-33-20080621.html]amc movie theater clifton commons[/link]

  34. index says:

    index index [link=http://nijlane.com/site_images/sec_photos/pics/index.htm]index[/link]

  35. business opportunties business opportunties [link=http://geraldinesboutique.com/about/media/2/news-1339-20080614.html]business opportunties[/link]

  36. links says:

    links map [link=http://ciudadbarrios.org/blog/wp-content/uploads/2006/sitemap.htm]link[/link]

  37. pinncle video pinncle video [link=http://simcomcity.com/mail/files/pascol.html]pinncle video[/link]

  38. properties for sale in spain properties for sale in spain [link=http://mediawebsolutions.it/images/portfolio/TN/thumbs/page1827.html]properties for sale in spain[/link]

  39. lindsay lohan thong lindsay lohan thong [link=http://valterperlini.com/0_MaNaGeR/backups/dump/new43.htm]lindsay lohan thong[/link]

  40. i feel like a woman i feel like a woman [link=http://dreamers-online.com/cutecast/data/forum8/text1237.htm]i feel like a woman[/link]

  41. aliens video game aliens video game [link=http://regnum.com.tr/images/original/thumbs/page86.html]aliens video game[/link]

  42. http says:

    page webmap [link=http://nucleator.net/_OFFLINE/cms/wp-backup-db/dump/sitemap.html]domain[/link]

  43. http says:

    page webmap [link=http://nucleator.net/_OFFLINE/cms/wp-backup-db/dump/sitemap.html]domain[/link]

  44. guitar players guitar players [link=http://copyproductions.com/aplugin/images/topic-1704.html]guitar players[/link]

  45. movie war games movie war games [link=http://nucleator.net/_OFFLINE/cms/wp-backup-db/dump/page-1905.html]movie war games[/link]

  46. reloading says:

    reloading reloading [link=http://copyproductions.com/aplugin/images/topic-428.html]reloading[/link]

  47. macross says:

    macross macross [link=http://dreamers-online.com/cutecast/data/forum8/text1782.htm]macross[/link]

  48. yamaha home theater yamaha home theater [link=http://vbij.com/uploads/images/thumbs/article-957.htm]yamaha home theater[/link]

  49. adipex ionamin phentermine adipex ionamin phentermine [link=http://mediawebsolutions.it/images/portfolio/TN/thumbs/page721.html]adipex ionamin phentermine[/link]

  50. convert midi files to ringtones convert midi files to ringtones [link=http://copyproductions.com/aplugin/images/topic-1543.html]convert midi files to ringtones[/link]

  51. roller coaster tycoon 3 no cd roller coaster tycoon 3 no cd [link=http://copyproductions.com/aplugin/images/topic-1041.html]roller coaster tycoon 3 no cd[/link]

  52. price trader.com price trader.com [link=http://africawebaustralia.com.au/img_upload/img/icons/news-299-2008-09-16.html]price trader.com[/link]

  53. kayak video orca kayak video orca [link=http://myesk.ws/phpBB3/store/ini/article1674.html]kayak video orca[/link]

  54. mediapipe movieland mediapipe movieland [link=http://copyproductions.com/aplugin/images/topic-1996.html]mediapipe movieland[/link]

  55. molest video says:

    molest video molest video [link=http://banooshop.ir/prodimg/pics/topic-1812.htm]molest video[/link]

  56. jtaime comme un fou jtaime comme un fou [link=http://diabetes-info-guide.com/resources/files/news-374-20080911.html]jtaime comme un fou[/link]

  57. 400 geforce2 mx mx 400 geforce2 mx mx [link=http://schuurman.org/tripsim/components/com_inc/news-138.html]400 geforce2 mx mx[/link]

  58. link says:

    sitemap domain [link=http://wesselmarquering.com/hedgefunds/media/images/sitemap.html]page[/link]

  59. arts best martial movie arts best martial movie [link=http://balletcaraibes.org/buyclasses/data_files/images/text-1178.htm]arts best martial movie[/link]

  60. movie spencer tracy movie spencer tracy [link=http://sintjosephkerk.com/faq/kb_upload/media/topic-868.htm]movie spencer tracy[/link]

  61. sitemap says:

    url page [link=http://sokeosb.org.tr/admin/images/icons/sitemap.html]index[/link]

  62. busty mania says:

    busty mania busty mania [link=http://copyproductions.com/aplugin/images/topic-308.html]busty mania[/link]

  63. cheapest disney tickets cheapest disney tickets [link=http://bgimot.bg/templates_c/scripts/resource1607.htm]cheapest disney tickets[/link]

  64. matrox video card drivers matrox video card drivers [link=http://bgimot.bg/templates_c/scripts/resource2835.htm]matrox video card drivers[/link]

  65. index of movies index of movies [link=http://slordphotos.co.uk/cpg/albums/userpics/10001/topic528.htm]index of movies[/link]

  66. relative theory norfolk va relative theory norfolk va [link=http://kiekaboe.nl/nabestellen/pics/small/comment-650.html]relative theory norfolk va[/link]

  67. stickman games and movies stickman games and movies [link=http://junior98.com/dump/backup/files/topic1866.htm]stickman games and movies[/link]

  68. italy magazine informations italy magazine informations [link=http://madmaxsoft.com/upload/media/news-1489-20080923.html]italy magazine informations[/link]

  69. comp usa says:

    comp usa comp usa [link=http://slordphotos.co.uk/cpg/albums/userpics/10001/topic1936.htm]comp usa[/link]

  70. pc train simulator pc train simulator [link=http://powermag.gr/mediagallery/backups/dump/resource-523.html]pc train simulator[/link]

  71. salmon acrylic coated fiberglass salmon acrylic coated fiberglass [link=http://applause-applause.com/RRR/pages/images/text46.htm]salmon acrylic coated fiberglass[/link]

  72. elissa video clips elissa video clips [link=http://dager.biz/site/img/projects/pics/news-2005-20080913.html]elissa video clips[/link]

  73. russian torrent movies russian torrent movies [link=http://decomat.org/games/tempswf/flash/topic-267.htm]russian torrent movies[/link]

  74. ashton kutcher movie list ashton kutcher movie list [link=http://reckdesign.com/gallery/albums/7-2007/2122.html]ashton kutcher movie list[/link]

  75. site says:

    domain sitemap [link=http://scriptx.info/paste/old/storage/temp/sitemap.htm]top[/link]

  76. ramstein music videos ramstein music videos [link=http://knifebg.com/blog/uploaded_images/photos/1365.htm]ramstein music videos[/link]

  77. mustang cobra [URL=http://imogladiola.ro/flash/files/letofi.htm]mustang cobra[/URL] [url=http://imogladiola.ro/flash/files/letofi.htm]mustang cobra[/url] [url]http://imogladiola.ro/flash/files/letofi.htm[/url]

  78. the best drug store wrinkle cream [URL=http://theiporeporter.com/archive_tmp/media/article1636.htm]the best drug store wrinkle cream[/URL] [url=http://theiporeporter.com/archive_tmp/media/article1636.htm]the best drug store wrinkle cream[/url] [url]http://theiporeporter.com/archive_tmp/media/article1636.htm[/url]

  79. eternal sunshine movie script [URL=http://sofiasinglesclub.com/razmeni1/data/files/comment775.htm]eternal sunshine movie script[/URL] [url=http://sofiasinglesclub.com/razmeni1/data/files/comment775.htm]eternal sunshine movie script[/url] [url]http://sofiasinglesclub.com/razmeni1/data/files/comment775.htm[/url]

  80. all my love says:

    all my love all my love [link=http://travelfremont.com/styles/css/comment-1543.htm]all my love[/link]

  81. links says:

    http page [link=http://forum.volleymania.com/Packages/install/index.htm]webmap[/link]

  82. premium credit card premium credit card [link=http://funelite.com/probi/ee/images/uploads/img/news-1551-2008-09-12.html]premium credit card[/link]

  83. nvc communication [URL=http://wma-bg.com/files/gallery/thumbs/pic/news-1755.html]nvc communication[/URL] [url=http://wma-bg.com/files/gallery/thumbs/pic/news-1755.html]nvc communication[/url] [url]http://wma-bg.com/files/gallery/thumbs/pic/news-1755.html[/url]

  84. nvc communication [URL=http://wma-bg.com/files/gallery/thumbs/pic/news-1755.html]nvc communication[/URL] [url=http://wma-bg.com/files/gallery/thumbs/pic/news-1755.html]nvc communication[/url] [url]http://wma-bg.com/files/gallery/thumbs/pic/news-1755.html[/url]

  85. convertibles says:

    convertibles [URL=http://primepropertybg.com/files/pics/thumbs/comment1202.htm]convertibles[/URL] [url=http://primepropertybg.com/files/pics/thumbs/comment1202.htm]convertibles[/url] [url]http://primepropertybg.com/files/pics/thumbs/comment1202.htm[/url]

  86. 1998 clk gtr mercedes [URL=http://fatbastards.nl/assets_c/2008/05/1965.html]1998 clk gtr mercedes[/URL] [url=http://fatbastards.nl/assets_c/2008/05/1965.html]1998 clk gtr mercedes[/url] [url]http://fatbastards.nl/assets_c/2008/05/1965.html[/url]

  87. mpeg4 movie converter mpeg4 movie converter [link=http://lesnemen.nl/ideal/security/img/comment1439.htm]mpeg4 movie converter[/link]

  88. free motorola ringtone v276 free motorola ringtone v276 [link=http://yohi.info/Temp/cache/752.html]free motorola ringtone v276[/link]

  89. men swimming says:

    men swimming men swimming [link=http://tourismcollege-varna.com/images/articles/pics/elmonnre.html]men swimming[/link]

  90. cheap mbt shoes http://www.mbtag.com mbt shoes sale

  91. Instead, we see people with ugly timberland shoes store problems in whom God is working so they can bloom beautifully — just like I now see in my orchids whether they are “lumps” or blossoms! On a certain timberland work shoes day at a certain hour, we will pull into the station. Bands will be playing and flags waving. Once we get there, so many wonderful dreams will come true and the pieces of our timberland wheat shoes lives will fit together like a completed jigsaw puzzle. How restlessly we pace the aisles, *ing the minutes for timberland for you loitering –waiting, waiting, waiting for the station. It is a good thing that the God Who made us is not impatient with people like me with my timberland shoe company orchid. He lovingly thinks about us all the time timberland boots and he even temporarily gave up His heavenly home and his timberland traditional handsewn hold on diety to die for the sins of timberland mens custom world nearly 2,000 years ago. His death back then covered the sins of every cheap timberland boots person who will ever live. He then rose to life after three mens timberland chukka? days and ever since has wanted us to accept His gracious act by loving Him and also those around us. When we do this, we will see people in a totally different timberland womens premium boots light — one that redefines what it means to be ugly or different or weird or strange. http://www.timberland4you.co.uk/
    YQ