An Accelerated Introduction to Solaris 10: Part 106 Mar '06 - 16:04 by benr
Times have changed. 10+ years ago most peoples first experience with UNIX was SunOS/Solaris. But thats changed, from about 1995 untill now the first UNIX experience is typically with Linux because of its popularity and easy availability. So now there are a huge number of *NIX experts and advanced users who are starting to look at Solaris but are feeling like n00bs again. I'm really proud of the fact that several such people have private contacted me to ask some basic questions about Solaris. These are people who know their shit, very compitent and intellegent administrators and developers who just need some help adjusting to Solaris, not a hand holding session. To help those people who fit into this description but aren't as bold as those that I've talked to we're going to take a whirl wind tour of Solaris from the ground up. Hopefully this will fill in just the right gaps to get you on your way on this wonderful platform.
Solaris Filesystem Layout
The most basic new questions are "where is f00?", "why can't I find f00?", and "what the f**k is that?" First, lets look at the high level root directory structure:
- /etc, /usr, /var: You know what they are.
- /opt: Similar to Linux, this is where we tend to store most commercial/distributed software
- /home, /net: These are, by default, automount (an automatic, on-demand NFS mounting service) point. More on automount later.
- /export: This is a common directory on commercial UNIX varients which servers both as a place for data to be shared via NFS and also, most commonly, as a general purpose dumping ground for things that don't go anywhere else. More on this with the automount discussion.
- /proc: Contains data about each process, with directory structures named after the process id. Used by the "Ptools".
Now, looking at /usr. There are a bunch of odd looking directories in there, and at first glance, a huge ammount of duplication. You've got to understand that Solaris prides itself on adherence to a wide range of industry standards, such as XPG4, UNIX98, POSIX, etc. This is one reason Solaris is often considered superior to Linux for enterprise applications and development. In order to satisfy everyone some tools are kept in their own locations to allow for diferent tools to fill diffrent needs and standards. Lets look at some of the directories in /usr:
- /usr/bin, /usr/sbin: You know these. Standard tools in bin and "system" tools in sbin.
- /usr/5bin: Location for System V binaries, in Solari 2.x systems this is just a symlink to /usr/bin
- /usr/adm: Symlink to /var/adm, the Solaris equiv to the Linux /var/log
- /usr/aset: Automated Security Enhancement Tool (ASET) tools. Seperated because the directory is root owned 700.
- /usr/ccs: CC's (?) directory containing Sun/SystemV development tools, including ar, m4, make, nm, ld, yacc, lex, elfdump, etc. Do not confused these for GNU tools.
- /usr/dt: CDE and related tools. I can't find an exact source for the definition of "dt" but anytime you see it just think "cde".
- /usr/games: Ya, you wish.
- /usr/gnome: GNOME. Well, JDS anyway.
- /usr/kernel: Various drivers and bits that act as secondary to /kernel.
- /usr/kvm: This is a dead directory, used to be used but its just a historical placeholder now. Stands for what you think it does.
- /usr/openwin: The old OpenWindows tools, including Sun's X server (Xsun), OpenLook libs, tools, and window manager, etc.
- /usr/platform: Platform specific stuff, most commonly known for the tools prtdiag, used for finding hardware status, and eeprom for interfacing with take a guess. On SPARC there are lots of directories in here for all the diffrent architectures and system specifics.
- /usr/proc: The Solaris "ptools" (pfiles, pmap, pwdx, pstack, etc). The tools just symlink to /usr/bin.
- /usr/sadm: Solaris admin tools, such as smc, the "Solaris Management Console" (GUI admin tool), and others.
- /usr/sfw: SFW stands for "Sun FreeWare". This is where you'll find GCC, GIMP, MySQL, and lots more. Don't bitch that Solaris doesn't have any tools untill you look for your GNU favorites here! Please note that to keep GNU tools distinct from Solaris system tools common names are preceeded with a "g" (for GNU), so "gmake" for GNU Make, "gld" for GNU LD, "gas" for GNU AS, etc.
- /usr/snadm: Solstice Stuff, historical, ignore it.
- /usr/ucb, /usr/ucbinclude, /usr/ucblib: BSD Tools! UCB stands for "University of California Berkeley". You'll find the BSD versions of common tools here, so if you prefer the BSD varient of ps (ie: "ps aux") over the SysV one (ie: "ps -ef") you'll want this in your path!
- /usr/xpg4, /usr/xpg6: Tools that comply with the XPG4 and XPG6 standards respectively. These are X/Open specs that are part of larger standards like the Single Unix Specification.
- /usr/X: Symlink to /usr/openwin
- /usr/X11, /usr/X11R6: Xorg X Server and associated tools.
For more information on standards that Solaris adhers to please see this page which details them.
If you think that some of these directories are stupid or unneeded I'll say simply that, A) many people agree with you, B) many strongly disagree with you. The most hotly debated directory in /usr is /usr/sfw which many people want to see renamed to /usr/gnu.
As for /var I'll only say on Solaris we typically store logs in /var/adm, not /var/log, like Linux, but this is slowly changing. You'll now find logs in both directories.
One side note, a large number of Solaris users consider /usr/local to be a sin against nature and taboo. I personally use it, but its not unusual for folks to want to keep /usr "pure". I've seen some shops use /usr2, /opt/local, and /export/local instead, but personally that seems stupid to me. Let your concious be your guide. These are purely matters of personal style.
Solaris was designed to live on a corporate LAN. This goes with Sun's "The Network Is The Computer"(tm) mantra. No where is this more evidence than Solaris's (over) reliance on the automounter. Linux admins and users may not even know what an automounter is, because its so uncommonly used by Linux systems, although almost every Linux system ships with one. The automounter is a method of mounting NFS shares "on demand". The most common usage is for home directories, in which you put all the user home directories on an NFS server, and then configure Solaris to automount /home, so that if you tried to access, for instance, /home/benr and it wasn't mounted it would be automatically mounted for you. Its a handy tool if you use it.
By default, when you install Solaris, the automounter, which you'll see as "autofs", defines both /home and /net as autofs mount points. (/net is effectively the Solaris equivelent to the Windows "My Network" destination, so that if you wanted to see the NFS shares on system f00, you'd just 'cd /net/f00/'.) Because /home is an autofs mount point you cannot create directories there! I'm really suprised that we don't get asked about this more often. If you want to create /home/benr you must either, a) disable the automounter (svcadm disable autofs), or b) remove the '/home' entry from /etc/auto_master and restart autofs (svcadm restart autofs). I recommend doing both to avoid problems in the future. Once thats done you can use /home just like you normally would on a Linux or BSD system.
Getting back to /export, traditionally on a Solaris system you would create directories which would likely be shared in here. This is why traditional long-time Solaris users will create home directories in /export/home instead of /home like non-Solaris users would expect. The idea is that, even on a local system, you create home directories in /export/home and then use the automounter to mount them to /home, which seems odd locally but makes things more transparent on the network. This is actually lost on a lot of users who simply do everything out of /export/home and simply ignore /home all together. You can use /export for whatever you wish, however, and often becomes a general purpose dumping ground for things that don't really fit anywhere else. For example, I always create zones in /export/zones.
Sudo and Root
You might wonder: "Where is /root?" We don't have one. This is (imho) because most Solaris users don't believe that the root user should be treated like a user, and furthermore the root user shouldn't be doing things that require a home directory. Roots default home directory is /. If you really want a /root, go ahead and create it, but we'd recommend that if you really want to do a lot of root-ish things you look at Role Based Access Control (RBAC) to give certain root privs to standard user accounts or special user "roles".
The common "sudo" utility can be used to either allow a user to run commands as root or to allow certain specific commands as root. In Solaris you don't find "sudo" because we have RBAC. For details on RBAC see my blog entry: Using RBAC on (Open)Solaris.
NFS Shares and /etc/exports
NFS sharing is a little diffrent on Solaris. On many OS's you would share a filesystem by editing /etc/exports and either starting the NFS server (ie: /etc/init.d/nfs.server start) or using exportfs -a. On Solaris you need to add NFS shares to /etc/dfs/dfstab. DFS stands for "Distributed File System". There are examples in the file on how to format it properly, but contains the commands as you'd use them on the CLI, which is...
To export filesystems you can edit /etc/dfs/dfstab and run exportfs -a (or svcadm restart svc:/network/nfs/server) or you can use the share command. share allows you to quickly export a filesystem, so if you wanted to NFS share /opt you could just execute share /opt and your done. More or less, share on Solaris works like you'd expect exportfs to work on Linux, although we have both.
Solaris doesn't have top. But, we have something better yet similar: prstat. Its just like top but uses less resources and is far more powerful when used properly (read the man page). Solaris users also heavily rely on vmstat, mpstat (like vmstat but per processor), iostat, and netstat. Tools like these are availible for Linux but more often top is used instead.
As mentioned before, both SysV ps (/usr/bin/ps -ef) and BSD ps (/usr/ucb/ps aux) are available on Solaris. In addition, you can use the Solaris "ptools" (process tools) to learn more about a process, such as pstack `pgrep cron` to see the call stack of the cron process, or pfiles `pgrep firefox-bin` to see details about every file that Firefox has open. This also answers the "Where's lsof?" question.
While these tools are really nifty, they only give you a high level understanding of the system. vmstat and mpstat will show you the number of interrupts and system calls per second and even on which CPU they are hitting, but if you want to know what their doing and, more importantly, where their coming from, you need Dtrace (short for "Dynamic Tracing"). Dtrace allows you to delve deep into the system in a simplistic and elegant manner and ask questions which leads to more questions and more questions, all the time understanding and redefining your perceptions of the system your working with. Dtrace is, of course, out of the scope of this entry, but The Solaris Dynamic Tracing Guide is really good and will answer most of your questions. Google will fill in the rest.
Software and Packages
Solaris uses the standard System V PKG package format. Packages come in two varieties: filesystem format and datastream. A filesystem format package is what you'll find on Sun CD's and is really just a directory stucture containing all the various elements and files of the package. A datastream package is a filesystem format package thats rolled into a single file making it easy to compress and distribute over the net. Packages use a common naming convension of ORGsoftware, such as SUNWspro, where SUNW is Sun's ticker symbol and "spro" is the historical name for Sun Studio.
Both types of packages are installed using the pkgadd command (ie: pkgadd -d ./CUDLgcc-4.0.1.pkg for datastream and pkgadd -d . for filesystem format). You can list all installed packages on the system using the pkginfo tool, and the same tool will detail information about the installed packages (ie: pkginfo -l SUNWspro). pkgrm can be used to remove, and pkgtrans can translate packages from one format to another.
Several sites on the net provide software in pkg format, but the two most popular are SunFreeware.com and Blastwave. At SunFreeware you can download a large number of popular packages and install them with the standard system tools, but because PKG doesn't resolve dependancies automatically this can be a pain in the butt. Blastwave is the Solaris equivilent to Debian's "apt-get", and allows you to pull packages over the network with a single command and resolve all the dependancies (ie: pkg-get -i vim). The primary criticism of Blastwave is that there is a huge amount of duplication with the system, but this is intential, with the idea that Blastwave packages should never depend on the system installation or make assumptions about what is or isn't installed. Currently, Blastwave is the most popular method of package retrival on Solaris.
Solaris 10 did away with the old standard RC scripts that your used to having on Linux and other OS's. They are still available but suppliment the primary system now, the Solaris Management Facility (SMF). Use svcs to view services, svcadm to administer them (start, stop, etc), and svccfg to add or change them. SMF's distinct advantage is the ability to create dependancies between services (ie: if Apache isn't up, don't bother starting MySQL), automatically restart services, startup in paralell, and fine grained control of services.
SMF services are described in XML manifests, find the default system manifests in /var/svc/manifest. Manifests describe a service, what its dependancies are, supply optional metadata, and provide methods to start, stop, refresh, or restart a service. A method can be a single line supplied in the manifest itself or an external shell script that can be as complex and extravegant as you'd like. The default system manifests are found in /lib/svc/method.
Disks can be poked, proded and partitioned using format. Solaris calls a partition a "slice". The standard Solaris naming convension for disks is: c0t0d0s0, that is to say: controller 0, target 0, LUN 0, slice 0. Solaris tradition is that slice 2 always represents the full disk and is not used for other purposes. The format tool, however, can not modify X86 partition information so instead you'll need to use fdisk. See the man pages for each tool before using them.
Removable devices such as, USB storage devices, are treated diffrently and can be seen using rmformat (rm for Removable Media). Typically removable media will be automatically mounted (if possible) by the volume management daemon: vold. CD's and DVD's will be automatically mounted to /cdrom. If you need to manually mount a CD or DVD remember to mount it using the "hsfs" (High Siera File System), not iso9660 like on Linux. In the event that vold is being lazy you can poke it to check for new devices with volcheck.
To view error counters on a device you can use iostat -En. Monitor IO throughput using iostat -xn 1.
Here you can leave your mark I like this site!