3. Mirror Juggling

Right up from I want to say that I DO NOT recommend this practice for the squeamish. If your like me and like loosing sleep at night, and like scaring the pants off Sun and Veritas tech support, then go for it.

Mirror Juggling is the term I dreamed up for one of my favorite pastimes. The idea is to rebuild existing volumes by attaching a plex (mirror) to the volume, sync'ing the plexes, and then detaching the original plex from the volume. At that point I'd destroy the original plex, and rebuild it from scratch to my design. Once the plex was rebuilt the way I wanted it, I'd attach the plex back to the volume, and let it sync. After the sync is done I can remove my temporary plex. See the idea here? The mental image I get when doing this is of water (representing the data) being rolled around in a box (representing the volume). As I roll the volume I slosh the data along with it, rebuilding behind it.

This process is pretty simple in concept, but horribly scary... on a production system this isn't for people with heart problems. But, if you hadn't guessed, the reason I'd do something as risky as this is because I can completely rebuild volumes from scratch on a hot system. NO DOWNTIME! Generally, in the environments I've worked, the systems running VxVM are the production systems that are expected to deliver 99.999% annual uptime, which doesn't lend for fixing old problems, and design flaws. Mirror Juggling is the solution I came up with.

The key that we're exploiting through this process is Veritas's ability to sync and contain plexes that aren't identical. I once used this method to fix a plex that contained 10 subdisks, yet only was striped to only use 5 columns. In this situation I had 5 disks that we're being utilized. Rather than use a cheap hack I want to rebuild, so I took on of my mirrors offline, removed the plex, then the subdisks until I had the plex broken down into it's VM Disks. From there I created subdisks, and then specified the striping model I wanted (10 columns with a 256k stripe width) to create the plex. Once my plex was the way I wanted it I reattached it to the volume and let it sync. Once it was sync'ed I took the other mirror offline and did the same to it. Afterwords I had only minimal clean up to do (which in this case involved expanding the filesystem to use all the extra space I had!). The point was that this was a production system, if it went down people went home, and I wasn't scheduled to get a downtime for 6 weeks. I did everything during a slow time with 0 downtime.