The Cuddletech Veritas Volume Manager Series Veritas Krash Kourse: The Who's Who of Vx Land by: B. Rockwood benr@cuddletech.com Welcome to VERITAS! ------------------- This course is the first step to learning the wild and wonderful world of VERITAS Volume Manager. This course is absolutely necessary for you to continue on to the future courses, and if you already know this stuff it's at least a good review. Before we start, there are a couple of things you need to know about me (your humble narrator). Why I'm Doing This ------------------ You may, or may not, be curious as to why I'd waste part of my life writing these courses. You may also have already noticed (from the preceding course) that I am not a professional instructor. Well, you're right: I'm not a professional instructor. I am, however, a UNIX systems admin who specializes in Sun Enterprise Storage Systems. I won't bore you with the story of how I got to where I am, but for several months I wanted to learn VERITAS, and no one could teach me more, and I couldn't afford classes. I begged my company to get me materials or a class on the subject to no avail. So, I learned VERITAS the hard way. I downloaded the manuals from Sun and studied them for a couple of weeks, before I got a contract that would let me actually experience VERITAS first hand. The bottom line here is that some of the concepts took days and weeks of rolling over and over in my head before I fully understood them. These documents, and the courses I teach, are my way of helping you get the most out of VERITAS and the VERITAS manuals the FIRST time you read them. Hopefully, you'll be understanding VERITAS in days or hours instead of weeks. This is my way of "giving back" to the community. I am not an "expert", nor is anyone an "expert". I'm just sharing the knowledge I've gained during the last year. My teachings have done well for many others that I've taught, so I hope you'll pick up a lot from these writings. Now, on with the show! VERITAS Volume Manager Documentation and Reference -------------------------------------------------- These tutorials directly apply to the following docs: - VERITAS Volume Manager: Installation Guide (P/N: 100-001121) - VERITAS Volume Manager: Getting Started Guide (P/N: 100-001123) - VERITAS Volume Manager: Command Line Interface Administrators Guide (P/N: 100-001124) - VERITAS Volume Manager: Storage Administrator: Administrators Guide (P/N: 100-000954) - VERITAS Volume Manager: Administrators Reference Guide (P/N: 100-001125) These are Sun part numbers. At the time of this writing these documents are not available on the web, and only available with the VERITAS VM 3.0.1 software itself. HOWEVER, you can find the VERITAS VM 2.6 documentation on docs.sun.com. To get the docs (available in HTML or PDF), go to docs.sun.com. Then click on "Browse By Collection". Then search down the list (toward the bottom) until you find "Sun StorEdge Volume Manager 2.6 AnswerBook". Click on that. Then you'll see three titles: - Sun StorEdge Volume Manager 2.6 User's Guide - Sun StorEdge Volume Manager 2.6 Storage Administrator User's Guide - Sun StorEdge Volume Manager 2.6 System Admin Guide Notice, while you're there, the "Download PDF" option in the top right hand corner; great for printing. VERITAS Volume Manager: An Overview ----------------------------------- Here's an excerpt from the Sun web page (www.sun.com) regarding VERITAS: "VERITAS Volume Manager software is an advanced, system-level disk and storage array solution that alleviates downtime during system maintenance by enabling easy, online disk administration and configuration. The product also helps ensure data integrity and high availability by offering fast failure recovery and fault tolerant features. VERITAS Volume Manager software provides easy-to-use, online storage management for enterprise computing and emerging Storage Area Network (SAN) environments. Through the support of RAID redundancy techniques, VERITAS Volume Manager software helps protect against disk and hardware failures, while providing the flexibility to extend the capabilities of existing hardware. By providing a logical volume management layer, VERITAS Volume Manager overcomes the physical restriction imposed by hardware disk devices." So, there you have it. But let me briefly list some of the things they totally missed. - Support for: Simple, RAID0, RAID1, RAID0+1, RAID1+0, RAID5 - Dynamic Multipathing (DMP): Load balancing and redundant I/O paths to disk arrays supporting multi-controller attachment. - Online ReLayout: VERITAS allows you to change the layout of a VERITAS volume while it is live and mounted. Change a RAID0 to a RAID5 without a second of downtime! - Snapshoting: Take a snapshot of your data, creating a "shadow" of it which you can use for online backups. - Hot Relocation: Designate "spares" which will take the place of failed disks on-the-fly. - Dirty Region Logging (DRL): Volume transaction logs which provide fast recoveries after system crashes. - More...... more than I can list! So lets start by building on all that. VERITAS Volume Manager is a product of VERITAS Software Corporation (www.VERITAS.com), which is used by Sun and also sold to Sun customers under a Sun part number. VERITAS is also can "Vx" or "VRTS" (the prefix of VERITAS packages, and VERITAS' ticker symbol). VERITAS Volume Manager is often abbreviated "VxVM". Sun, before VERITAS 3.x, had its own marketing names for VERITAS, the names you'll see and the version numbers are: <= Vx 2.5: Sun Enterprise Volume Manager (SEVM) = Vx 2.6: Sun StorEdge Volume Manager (Sun SEVM) > Vx 3.x: VERITAS Volume Manger (VxVM) As for OS support: SEVM 2.5: Solaris 2.5.1 and older. SEVM 2.6: Solaris 2.6 and older. VxVM 3.0.2: Solaris 7 and older. VxVM 3.0.3: Solaris 8 and older. Let me add, when Sun and/or VERITAS says "OS xxx isn't supported", they don't really mean "Ummm, we haven't really tried it yet", like most other companies in the world. VERITAS actually installs kernel modules into the system, so if you run SEVM 2.6 on Solaris 7, you'll be running Solaris 2.6 kernel modules in a Solaris 7 kernel. It doesn't work, I tried it! (and Sun was really pissed when they found out I tried it ) VERITAS Volume Manager and Vx Objects ------------------------------------- VERITAS is a logical volume manager. The operative word here is "logical". So, in order to keep things in order we have some basic pieces which are used in different ways for building volumes. The better you understand these guys, the better you'll understand VERITAS. Here's the list of objects: Disk: This is a normal physical disk with a SCSI id. (c0t0d0...) VM Disk (dm): A disk that is put under Vx control. Sub Disk (sd): A section of VM disk used to build plexes. Plex (pl): A mirror. Volume (v): A virtual disk, which can contain data. Now, lets talk about these objects a bit. A disk is nothing new or magical. When we want to use a disk in VERITAS we need to turn it over to VERITAS control. A disk turned over to VERITAS control is then given a VERITAS name (like disk01). After doing this the disk is no longer available to the system for anything outside of use inside VERITAS. Also, when we turn disks over to VERITAS control we don't turn over a partition (c0t0d0s0), but the whole disk itself (c0t0d0). Now that we've got a VM Disk we then create a Sub Disk from the VM Disk. Think of a subdisk as a VERITAS partition. We could make one big subdisk from the VM Disk and use the whole disk, or we could create a bunch of smaller subdisks. You can divide VM Disks into subdisks however you like. From subdisks (one or more) we create what is called a plex. Plexes are confusing so let's talk about these in some length. Say the following out loud: "A Plex is a Mirror. A Plex is a Mirror. A Plex is a Mirror." Maybe you should do that a couple times. You may feel silly, but plexes are a sort of cornerstone in VERITAS. A Volume is a container, in VERITAS. The volume is made up of one or more plexes. See the catch? A Plex is a mirror, yes, however you can make a volume with only one plex. Is a volume made with only one plex mirrored? No. We'll explain this more later, but for the time being keep it in your head. So, subdisks are grouped together into a plex. The interesting thing is that in VERITAS the plexes do all the work, so lets say you wanted to create a Striped volume. You would actually create a striped plex, and then attach that striped plex to a volume. The volume doesn't care, it's just a container. See the beauty here? Let's put all this stuff together and build an imaginary volume in VERITAS. We're going to build a striped (RAID0) volume from 2 9G disks. We'll say that the first disk is c1t0d0, and the second is c1t1d0. First, we need to put them in VERITAS control, so we create VM disks. The VM disks are then named disk01, and disk02. Next, we'll create subdisks using these two disks. Let's use the whole disks and just create 2 subdisks, one for each VM disk. We'll call disk01's subdisk "disk01-01", and disk02's subdisk "disk02-01". Now, it's plex time! We're going to build a striped plex using our two subdisks, which we'll call "myplex". (Don't worry yet about how we create the plex, just get the concepts now.) So now we've got a plex, which contains the subdisks "disk01-01" and "disk02-01". Now, we create a volume named "myvol" using "myplex". And bingo! We've got a striped 18G volume ready to create a file system on! Maybe, if we used the short names mentioned earlier (with the list of objects) and make an outline it'd look something like this: dm disk01 c1t0d0 <-- VM Disk named "disk01" made from "c1t0d0" dm disk02 c1t1d0 <-- VM named "disk02" made from "c1t1d0" sd disk01-01 disk01 <-- Subdisk named "disk01-01" made from "disk01" sd disk02-01 disk02 <-- Subdisk named "disk02-01" made from "disk02" pl myplex striped <-- Striped Plex named "myplex" sd disk01-01 <-- Made using subdisk "disk01-01" sd disk02-01 <-- and subdisk "disk02-01" v myvol <-- Volume made from.... pl myplex striped <-- the striped plex named "myplex", made from... sd disk01-01 <-- Subdisk "disk01-01", and... sd disk02-01 <-- "disk02-01" Look OK? Because if it does, take a look at this, real output from VERITAS, from a real volume I created on my test machine: v myvol fsgen ENABLED 35356957 - ACTIVE - - pl myplex myvol ENABLED 35357021 - ACTIVE - - sd disk01-01 myplex ENABLED 17678493 0 - - - sd disk02-01 myplex ENABLED 17678493 0 - - - Does that make any sense? Any at all? I hope it does. And if it does, you're ready for VERITAS. We're going to explain more, much more, as we roll along, but at least understand that: Volumes are made up of plexes. Plexes are made up of subdisks. Subdisks are made up of VM Disks. VM Disks are made up of (real) Disks. and, Disks can be turned into VM Disks. Vm Disks can be turned into Subdisks. Subdisks can be grouped into Plexes. Plexes can be grouped into a Volume. Good. One more note about plexes before we move on. Here's the groovy thing about plexes. Because plexes are "mirrors", we can mirror the volume we built earlier simply by building another plex identical to the first one, using two more subdisks (which means we need 2 more vmdisks, etc,etc). Once we build the second plex, we attach it to the volume (myvol) and presto chango! We're mirrored! This is really kool... mirrors are something magical, they are a part of everything. If you have only one mirror you may see yourself, but you'll need a second mirror to see yourself holding the mirror. Hmmm? VERITAS Object Creation ----------------------- In VERITAS there are two ways to create objects. There's the "vxmake" way and the "vxassist" way. Here's the difference... VxMake is an object creation tool that allows you to specifically build objects, one at a time, by your specific specs. The methods we mentioned before (add a disk, make a VM disk, create a subdisk, then...) would be an example of how we use vxmake. Piece by piece, we'd build out components, and assemble them. This is kool because there is no "guess work", meaning we are explicitly saying "build this!" and it's built. We'll learn more about vxmake, in detail, in the next course. VxAssist is the other way. As you've been reading this course, have you said "this sure looks like a lot of work!"? Well, here's your answer: vxassist. When we want to build a volume, but we don't want to specify every little detail, we could add disks to make VM disks and then use vxassist to do the rest. So instead of using the "build this, and only this!" philosophy of vxmake, we turn to the vxassist mind set that sounds something like "OK, here are some disks, build me a RAID5 that can house 10G of data... I don't care how you do it, just hurry up." Vxassist is handy for a jam, or when you really just don't care exactly how things are set up. Again, in our following course we'll discuss object creation at length using vxassist. Don't get any ideas yet about which one is better. When we get to the courses applicable to these tools and you see how they work with your own eyes (yup, you'll see lots of examples) then you can get defensive about your favorite. But also remember, you can use them both interchangeably in some cases, and often with use both. Keep your eyes open for these guys though, because they are very different ways of doing things. Special Notes on RAID0 and RAID5: Layouts ----------------------------------------- I want to briefly discuss layouts. This is a topic will delve into more thoroughly later when we talk about plex creation, but because the "VERITAS Getting Started Guide" talks about them, I'll give you the highlights. This stuff may seem pretty basic now, but the more you work will VERITAS the more you will come to appreciate layouts. Layout can be thought of as the defining difference between the different types of RAID. Simple RAID's don't really have a layout, you simple write linearly starting at the beginning of the first disk until you hit the end, at this point you start writing the second disk, and so on. Striped RAID's have special type of layout, however. Layouts use the following terms: stripe width (stwidth): The Amount of data to write to a disk before moving to the next. (Generally 32k by default) columns (ncolumn): A subdisk. In the case of a Striped RAID, each subdisk would have a column number. A RAID0 with 4 disks, would have 4 columns. The first disk would be column 0 (remember: everything starts with 0), the second would be column 1, the third would be column 2, and the fourth disk would be column 3. Now, when we build a striped RAID we specify both stripe width (which VERITAS calls "stwidth"), and the number of columns (which VERITAS calls "ncolumns") which is the number of subdisks that are going to have data on them. So, as mentioned earlier, we've got 4 disks, and we know the column numbers for each subdisk. If we assigned a stripe width of 32k, and we wrote some data to the volume, we'd write the data 32k at a time, writing each new 32k chunk to the next column, and wrapping around from the 4th column (column 3) to the 1st column (column 0), untill all the data is written. Refer to the "RAID Theory" course for more on this. If you've got this down, but feel foggy, just roll with it, it'll click later as we build plexes. Now, RAID5 is very similar because it's striped, it's just a different type of stripe. RAID5 uses a layout called "Left Symmetric Layout", or more commonly as simply "RAID5 layout". It would take me too much time to explain this, so check the book. Read through the "RAID Theory" course if you need help, and then go back to the book. You may have to look twice, but you'll get it. Volumes and Devices ------------------- So now that you understand how a volume is built, and you understand a little about how it works lets talk about how you use them. As mentioned earlier, a volume is a container. A "bit bucket" is you will. The advantage of a volume over a normal disk partition, is that volumes aren't static. You can grow, shrink, move, rearrange, and manipulate volumes in ways you just can't with a partition. You can also do all this in a mounted and live environment with no reboots. When you don't have enough disk space in a filesystem you can just grow the volume that the filesystem is on, and then grow the filesystem. Very easy. But... you're saying to yourself, "how do I actually use a volume?" Well, that's easy enough. You use a volume just like a disk block device. Once VERITAS is installed and set up, you'll see a new directory in /dev. You'll see /dev/vx/, which is where all of your volume block devices are. If you created a volume named "myvol", and wanted to "newfs" it, you could do it like this: # newfs /dev/vx/rdsk/myvol or you could mount it like this: # mount /dev/vx/dsk/myvol /myvol Kool huh? Nothing magical. Poke around in the /dev/vx file tree and see where things are, everything is fairly well named, and self explanatory. If you're new to device trees, then just know that you can access your volumes from /dev/vx/dsk/ as though it were a normal disk. --------------------------------------------------------------- (Side Note: See how simple and logical VERITAS is? This is want makes VERITAS so kool. There's no voodoo whoodoo. Everything is clearly laid out, and organized. I love VxVM and I'm hoping that you're starting to as well.) ---------------------------------------------------------------- Kernels and Daemons ------------------- One thing that separates VERITAS from other volume managers is its integration with the host operating system. VERITAS actually utilizes kernel modules, which are the backbone of the application. You can see some of the driver loads in /etc/system. You'll see several driver forceloads. This is important to note because on most systems you'll see "WARNING: Forceload of dev/XXX Failed!" messages at boot up. Don't worry about these, it just means that VERITAS is trying to load a driver for a piece of hardware you don't have. I wanted to talk about VERITAS Installation, but I'm going to leave that to the book, but be VERY careful on VERITAS and Solaris upgrades to a system using VERITAS. Remember that because VERITAS uses kernel modules, that if you should wipe out and/or replace the kernel you could loose VERITAS.... which means you'll get a very intimate session alone with the books and the system while you learn how to recover botched VERITAS installations. Please, be informed about this, I've killed a couple machines this way, and luckily they were all "test" machines and not production servers. VERITAS also has several daemons running all the time to keep watch over your volumes, and to ensure smooth operation. To look for them try typing "ps -ef | grep vx" on a machine running VERITAS. You generally will see 3 common daemons, which are: vxiod : The VERITAS IO Manager, which managers reads and writes to volumes. vxconfigd : The Vx Object Configuration Manager. vxrelocd : The Vx Hot Swap Manager, which reacts to disk failures with spares. Maybe, by now your asking "Hurry up, just tell me where the VERITAS configuration file is!". Whelp, there isn't one! Everything we do to VERITAS is watch by the vxconfigd. The vxconfigd then updates VERITAS's object database, and keeps watching. As for the other daemons, they are pretty self explanatory. Vxiod makes sure that data is written to the volume the way it's supposed to be (so if the volume is striped, vxiod makes sure the data is written to the stripe the way we told it too, etc..). Vxrelocd just looks for disk failures and if it finds one it puts a spare (if there is one) in it's place. Intro to Advanced Features -------------------------- As we wrap this up, lets talk about VERITAS's truly special, and more "advanced" features. Please don't for an instant think this is the complete and definitive list. Rather, this is a small list of features that I really think separate VxVM from the other volume managers available. All of these topics will be covered in detail in the "Advanced VERITAS Theory" course later on. Dynamic Multi-Pathing: Also know as DMP, this feature allows for arrays which can be attached to multiple controllers (such as the Sun A5000 family of arrays) to leverage them for both load balancing and redundancy. So if your disks are attached to both c1 and c2, and your using a 100Mb/s fiber connection, with DMP you'll access the disks at 200Mb/s. And if c1 dies, for any reason, you still have access to the disk through c2, which means zero downtime for your volume. DMP can really save your butt if there is a failure. Hot Spares: You can allocate VM Disks as spares, so that if a disk in one of your plexes dies for some reason the spare will immediately take it's place and again, zero downtime. You can have as many hot spares as you like, and VM Disks allocated as spares can are independent of any particular disk group, plex or volume, so you don't need to have different spares for different volumes. Spares are managed by the vxrelocd as mentioned earlier. Dirty Region Logging: Also known as DRL, this feature can be thought of as a volumes diary. Before writes are made to a DRL'ed volume VERITAS makes a note of the write in the DRL, marking that region "dirty". After the note is made in the log, the write occurs. The idea here is that if the system crashes, when the system comes back online it won't know what writes were in progress. So, if a DRL log exists for the volume, it checks the log and then sync's dirty regions from the other mirror(s). In this way, we don't have to worry about corrupted data. DRL only applies to mirrored volumes, even though DRL can be added to unmirrored volumes (it just won't do anything useful). Snapshotting: Users of NetApp Filers will be familiar with this. Snapshotting is used for online backups. When you "snapshot" a volume, you are saying "I want to create a static picture of what the volume looks like right now". Then once the snapshot is created it appears as a new temporary volume, which can be mounted and backed up. Once the backup is done you "release" the snapshot. The kool thing is that while the snapshot exists the volume is still read and writable to everyone else. Zero downtime backups! OK, I'm going to leave the list at that for now. Don't get frustrated if your thinking "Hey, that's kool! But your not telling me anything useful!", because all of that will come later in the "Advanced VERITAS Theory" course, where I will talk about these topics and more in detail. For the meantime, just know that they exists and can be used. The Wrap Up ----------- Congratulations! You now are ready to actually start playing with a VERITAS system and have a good idea of what's actually going on. The next course is split into two camps, one for VxMake and one for VxAssist. Choose either, but you should read both. At this point you should read, or re-read the "VERITAS Getting Started Guide" for you 3.x users, and for SEVM 2.6 or older users, read or re-read the first chapter of your SEVM manual. Make sure that everything makes sense, and then continue on. -------------------------------------------------------------------------------- - Feel Free to Comment on this course to: benr@cuddletech.com ------------------ ------------------------------------------------- cuddletech.com ---------------