4. Volume Lesson1: A "Simple" RAID Volume

In this lesson we're going to build a Simple RAID. We're going to create it from our 4 9G drives which we added earlier. We're going to build this volume object by object, and we're going to put the objects together until we've built out volume. If it sounds like I'm talking about Lego's, it's not too far from the truth. It's very similar.

First, we need to great SubDisks from our VM Disks. We'll need these SubDisks to build a plex, later. As always, I'm going to start by verifying my current configuration with "vxprint" and then I'm going to create my subdisks. This creation will be done with one-line per subdisk creation. To do this creation I'm going to use, our hero, VxMake. The syntax for building a SubDisk from a VM Disk using VxMake is this:

vxmake sd <subdiskname> <VMDiskName>,<offset>,<len>

The syntax is pretty easy. Vxmake is the command, "sd" specifies that a subdisk is being created. "subdiskname" is the name that the new subdisk will be know as; typically this name is the name of the VM Disk is was created from (for reference) with a "-01" after it. The "-0X" would be number sequentially for each new subdisk created from that VM Disk. The "vmdiskname" is the name of the VM Disk we want to create this subdisk from. "offset" would be used for, um, setting offset. We'll talk more about this later; generally you won't need it. "len" is the length of the disk in sectors. To allocate the whole VM Disk to the new subdisk you could just "cut-and-paste" the length from the vxprint output. That's it! That's all you need to know and do to create a subdisk. Let me interject now that creating subdisks is the Veritas way to "partition" disks. For instance, if you wanted to create 2 volumes, striped across 4 disks, and you only had 4 disks you could make 2 subdisks from each VM Disk and then use the "disk0?-01" subdisks for the first volume, and the "disk0?-02" subdisks for the second. I hope this helps clarify subdisks.

Now, let's see what it looks like to create our subdisks, I'm going to create 4 subdisks one after the other, after checking vxprint. Here it is:

# vxprint
Disk group: rootdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg rootdg       rootdg       -        -        -        -        -       -

dm disk01       c2t0d0s2     -        17678493 -        -        -       -
dm disk02       c2t1d0s2     -        17678493 -        -        -       -
dm disk03       c2t2d0s2     -        17678493 -        -        -       -
dm disk04       c2t6d0s2     -        17678493 -        -        -       -
# vxmake sd disk01-01 disk01,0,17678493
# vxmake sd disk02-01 disk02,0,17678493
# vxmake sd disk03-01 disk03,0,17678493
# vxmake sd disk04-01 disk04,0,17678493
#

That's it! We've now made our subdisks! Notice the "len" is identical to the "Length" from the vxprint output. Also notice the naming conventions. Very simple huh? Further, I want to point out that vxmake doesn't output anything in return. If the operation was successful it will return a prompt. If there is an error you'll get the error message. When creating several (more than 6) subdisks at one time, make sure to check vxprint often in case you make a typo that is "syntactically correct". With that said, lets look at what vxprint outputs now:

# vxprint
Disk group: rootdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg rootdg       rootdg       -        -        -        -        -       -

dm disk01       c2t0d0s2     -        17678493 -        -        -       -
dm disk02       c2t1d0s2     -        17678493 -        -        -       -
dm disk03       c2t2d0s2     -        17678493 -        -        -       -
dm disk04       c2t6d0s2     -        17678493 -        -        -       -

sd disk01-01    -            ENABLED  17678493 -        -        -       -
sd disk02-01    -            ENABLED  17678493 -        -        -       -
sd disk03-01    -            ENABLED  17678493 -        -        -       -
sd disk04-01    -            ENABLED  17678493 -        -        -       -

See? We're just building and building. Next, we need to build our plex. The plex is actually the trickiest part, because striping and RAID5 is done at the plex level, not the volume level (remember that the volume is just a dumb container for plexes.... plexes do the work). Because we're going to build a "simple" plex we don't need to give any fancy options. In this case, the syntax will be:

vxmake plex plexname

Vxmake is the command we're using. "plex" tells vxmake that we want to build a new plex. And "plexname" is the name of our plex. You can name the plex anything you want. However, normally 95% of people name plexes as the volume name, dash 01, after the volume they will be used in. This is completely voluntary and is merely helpful to increase readability when troubleshooting. In our example I'm going to name our plex "dataplex1". Here's what it looked like when I build the plex, followed by the vxprint output:

# vxmake plex dataplex1
# vxprint
Disk group: rootdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg rootdg       rootdg       -        -        -        -        -       -

dm disk01       c2t0d0s2     -        17678493 -        -        -       -
dm disk02       c2t1d0s2     -        17678493 -        -        -       -
dm disk03       c2t2d0s2     -        17678493 -        -        -       -
dm disk04       c2t6d0s2     -        17678493 -        -        -       -

sd disk01-01    -            ENABLED  17678493 -        -        -       -
sd disk02-01    -            ENABLED  17678493 -        -        -       -
sd disk03-01    -            ENABLED  17678493 -        -        -       -
sd disk04-01    -            ENABLED  17678493 -        -        -       -

pl dataplex1    -            DISABLED 0        -        -        -       -
#

See the new plex in the vxprint output? Notice that there are no subdisks in it. With vxmake I can actually specify which subdisks should be assigned to the plex, but I want you to do this the long way. This is 100% manual!

To add the subdisks to our new plex, we're going to use the "vxsd" command, which is the subdisk control tool. Vxsd has all kinds of options and functions, but we're going to use it for "subdisk association". As the name suggests, we're merely means we're "associating" the subdisks, which implies we can later "disassociate" the subdisks. Get the idea? We'll look at this more later. The syntax used to associate disk using vxsd is:

vxsd assoc <plexname> <subdisk> <subdisk> .....

"vxsd" is the command. "assoc" means we're associating disks to a plex. "plexname" is the name of the plex to associate the subdisks with. And then the rest of the command is the list of subdisks we're associating, separated by spaces. I'm going to be associating all 4 of our disks, and then check my change with vxprint. Here's what it looks like:

# vxsd assoc dataplex1 disk01-01 disk02-01 disk03-01 disk04-01
# vxprint
Disk group: rootdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg rootdg       rootdg       -        -        -        -        -       -

dm disk01       c2t0d0s2     -        17678493 -        -        -       -
dm disk02       c2t1d0s2     -        17678493 -        -        -       -
dm disk03       c2t2d0s2     -        17678493 -        -        -       -
dm disk04       c2t6d0s2     -        17678493 -        -        -       -

pl dataplex1    -            DISABLED 70713972 -        -        -       -
sd disk01-01    dataplex1    ENABLED  17678493 0        -        -       -
sd disk02-01    dataplex1    ENABLED  17678493 17678493 -        -       -
sd disk03-01    dataplex1    ENABLED  17678493 35356986 -        -       -
sd disk04-01    dataplex1    ENABLED  17678493 53035479 -        -       -
#

Looking good so far! Now that we're got a plex built, we only need to attach the now completed plex to a volume. We'll build a volume using vxmake, and the following syntax:

vxmake -U <usagetype> vol <volname> plex=<plexname>

This one is alittle more confusing. "vxmake" is the command we're using. Here we'll use an argument "-U" after which we specify the "usage type". We won't talk about usage types now, but for a volume to contain a file system you'll need to use the "fsgen" usage type. "vol" tells vxmake we're building a new volume. "volname" is the name we want our new volume to have. And this time, we are going to let vxmake attach the plex for us, instead of having to do it ourselves. We do this with "plex=" followed by the name(s) of the plex(es) we want to attach. We could attach multiple plexes by listing them separated by coma's (then you'd have a mirror!), but we're only going to attach our "dataplex1" plex. That's it. So lets build the volume and then run vxprint. Notice I'm going to run vxprint this type with the "-hrt" (which I remember as "dash hurt") which shows us much more detail, than normal vxprint output does. Let's look at it, and then we'll discus it:

# vxmake -U fsgen vol datavol plex=dataplex1
# vxprint -hrt
Disk group: rootdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg rootdg       default      default  0        952738334.1025.nexus6

dm disk01       c2t0d0s2     sliced   3590     17678493 -
dm disk02       c2t1d0s2     sliced   3590     17678493 -
dm disk03       c2t2d0s2     sliced   3590     17678493 -
dm disk04       c2t6d0s2     sliced   3590     17678493 -

v  datavol      fsgen        DISABLED EMPTY    70713972 ROUND     -
pl dataplex1    datavol      DISABLED EMPTY    70713972 CONCAT    -        RW
sd disk01-01    dataplex1    disk01   0        17678493 0         c2t0d0   ENA
sd disk02-01    dataplex1    disk02   0        17678493 17678493  c2t1d0   ENA
sd disk03-01    dataplex1    disk03   0        17678493 35356986  c2t2d0   ENA
sd disk04-01    dataplex1    disk04   0        17678493 53035479  c2t6d0   ENA
#

Kool huh? Don't let all the vxprint headers generated because of the "-hrt" scare you. They are keys for each lines output. The two letter designations should start looking familiar, with the exception of the "SV" which is something you'll rarely see, and is for the advanced course. Anyway, What I wanted you to see is that with the "-hrt" on vxprint we see on the "pl" (plex) line of the output vxprint reports that "dataplex1" is a "CONCAT" or concatenated RAID (aka Simple RAID). Also notice that each time we put objects together that they regroup themselves in the vxprint output. This just makes everything easier and easier to read and understand quickly.

We've now got a volume! We now need to start the volume and create a filesystem on it, after which we can mount it. We can start a volume with the "vxvol" command (which is used for vx volume control, and also has tons of kool uses). The syntax is just:

vxvol start <volname>

I won't even bore you with the analysis, this is self explanatory. So let's just do it! We're going to start the volume and then look at vxprint (YES, AGAIN!!!!):

# vxvol start datavol
# vxprint -hrt
Disk group: rootdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg rootdg       default      default  0        952738334.1025.nexus6

dm disk01       c2t0d0s2     sliced   3590     17678493 -
dm disk02       c2t1d0s2     sliced   3590     17678493 -
dm disk03       c2t2d0s2     sliced   3590     17678493 -
dm disk04       c2t6d0s2     sliced   3590     17678493 -

v  datavol      fsgen        ENABLED  ACTIVE   70713972 ROUND     -
pl dataplex1    datavol      ENABLED  ACTIVE   70713972 CONCAT    -        RW
sd disk01-01    dataplex1    disk01   0        17678493 0         c2t0d0   ENA
sd disk02-01    dataplex1    disk02   0        17678493 17678493  c2t1d0   ENA
sd disk03-01    dataplex1    disk03   0        17678493 35356986  c2t2d0   ENA
sd disk04-01    dataplex1    disk04   0        17678493 53035479  c2t6d0   ENA
#

Here's the big key to notice. The "KSTATE" (Kernel State) and "STATE" changed from "DISABLED" and "EMPTY", to "ENABLED" and "ACTIVE". By starting the volume it's suddenly jumped to life.

Having fun yet? I just love Veritas. Isn't this logical? If your sailing through this, your doing better than you image. Remember, this is the "hard" way!

Okey. It's show time. We've got a live volume. We need to create a filesystem on it, and mount it.

Like many other volume managers, the volumes are accessible via a device file in the "/dev" file tree. Veritas volumes can be found in /dev/vx/dsk. Let me show you:

# ls -al /dev/vx/dsk
total 6
drwxr-xr-x   3 root     root         512 Mar 29 23:37 .
drwxr-xr-x   6 root     other        512 Mar 10 17:32 ..
brw-------   1 root     root      67,  5 Mar 29 23:37 datavol
drwxr-xr-x   2 root     root         512 Mar 29 23:37 rootdg
#

Wondering about the "rootdg" showing up? Because our volume is in the "default" disk group (rootdg) we see it in the /dev/vx/dsk directory. If we created a new disk group, and built a volume in it, the volume would be in /dev/vx/dsk/dgname/volname. Simple enough.

Just like any other volume manager, we can access a volume just like a disk, and we see that /dev/vx/dsk/datavol is a block device, so we'll create our filesystem on it. I'm going to use newfs (which defaults UFS) to create the file system on the volume:

# newfs /dev/vx/dsk/datavol
newfs: construct a new file system /dev/vx/rdsk/datavol: (y/n)? y
/dev/vx/rdsk/datavol:   70713972 sectors in 34529 cylinders of 32 tracks, 64 sectors
        34528.3MB in 705 cyl groups (49 c/g, 49.00MB/g, 6144 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 100448, 200864, 301280, 401696, 502112, 602528, 702944, 803360, 903776,
 1004192, 1104608, 1205024, 1305440, 1405856, 1506272, 1606688, 1707104,
 1807520, 1907936, 2008352, 2108768, 2209184, 2309600, 2410016, 2510432,
	snipped out
#

Newfs created the filesystem, just like it would if we used a normal partitioned disk. Notice that the size of this filesystem is 34.5G, and we used 4 9G disks, so this is about right. Now we're ready to mount the filesystem, which is just like normal. If we wanted to mount our new volume to "/vxrocks", we'd use the mount command like this:

# mount /dev/vx/dsk/datavol /vxrocks
#

That's it!!!! We're all done! I have a harder time getting dressed in the morning! Again, this was "the hard way". Even though you might not always build volumes this way (I do...) you will at least now have a better understanding of how things are put together, and how objects work with each other.