Solaris and iSCSI are two technologies that share something in common: they are amazing technologies that have had to slug their way into the data center. Solaris puts insane amounts of power at your fingertips and iSCSI puts insane amounts of flexibility into your architecture. And so, I think its time that we start looking closely at the Solaris iSCSI Target and what it can do for you. We’ll start at the beginning and work our way forward.

The purpose of iSCSI is simple: make a block storage device, such as a disk or a volume, accessible over a conventional IP network. When iSCSI was introduced it looked neat and nifty but since 1Gbps Ethernet has become the standard baseline for network throughput across the board and 10Gbps Ethernet is a reality, iSCSI has been elevated from “interesting” to “koooooooool”. The immediate advantage of iSCSI is clear: no-cost/low-cost storage networking. However, once you get past the cost aspects you realize that the true advantage of iSCSI is the flexibility it provides as a product of the cost, systems that you would never have imagined putting a Fibre Channel HBA in are now prime candidates for iSCSI. When combined with Ethernet technologies like aggregation (“trunking”, “teaming”, etc) and JumboFrames, or storage technologies like thin provisioning or ZFS we create some unique and compelling options that were either impossible on Fibre Channel or just far too costly to be attainable by the masses.

Targets and Initiators: Old Concepts in a New World

iSCSI systems come in two varieties: Targets and Initiators. The relationship is similar to that of Server and Client. An iSCSI Target is a storage resource you wish to share. An iSCSI Initiator is an end-point that wishes to access an iSCSI Target to use that storage. This terminology comes straight from traditional SCSI. On a SCSI bus you have various targets (remember those old toggle switches on the back of SCSI devices to set the “id”?), each of those target devices (up to 15 typically) can have LUN’s, and the controller in your computer which the SCSI chain is attached to is known as the Initiator. Think of that and refer to the Solaris standard naming scheme:

c1t2d0s2 == controller 1, target 2, LUN 0, slice 2

Use this scheme to keep you grounded as you explore iSCSI. Things can seem strange and diffrent in this IP storage world, but we’re still using the same old tried and true concepts: some system acts as an initiator (instead of a SCSI controller card in a PCI slot), which connects to one or more iSCSI Targets (instead of target devices on 80 pin cable), which can have one or more LUN’s (sometimes disk arrays would present each drive as a LUN, where we get the “d0” from for “disk”), which can be partitioned into some number of slices. Its really not that new a thing at all when you think about it, we’re just replacing things that used to be hardware with software and using ethernet to get from here to there.

Targets and Initiators can both be implemented in hardware or software. The most common hardware based solution is to buy an iSCSI Host Bus Adapter (HBA) such as those sold by QLogic. These HBA’s offer the ability to offload TCP and iSCSI processing, but surprisingly benchmarks comparing hardware vs software initiators have been shockingly close in performance, leaving the primary advantage of a $800 HBA the ability to boot an iSCSI target.

IQN’s: Knowing What is What

Fibre Channel devices are represented as World Wide Numbers (WWN), insanely long unique identifiers. In the iSCSI world we have the same concept, but instead of WWN’s we use iSCSI Qualified Names: IQN’s. These IQN’s are given to Targets and Initiators to uniquely identify each component in the storage architecture. Becuase we could potentially have thousands of targets and initiators on a single network, its important that each device is both descriptive and unique. IQN’s look something like this:

iqn.1986-03.com.sun:02:df204646-7956-e0a9-bfc1-c47759d0d93b
iqn.1986-03.com.sun:01:e00000000000.4505d02b
iqn.1997-06.com.cisco:01.54197b8ffa8

For more details on IQN’s and the format, please refer to this page.

Targets and LUN’s: Bundling Up

As we stated earlier, iSCSI Targets are network accessible block storage devices. Just like a traditional SCSI device, a target will provide one or more LUNs. The first LUN is “LUN 0”. Traditionally LUN’s allowed SCSI chains to grow beyond the limitations of the number of target devices on the chain, however today they provide a useful grouping mechanism. For instance, you might want to share 3 filesystems all of which are for Oracle databases (one for data, one for the binaries, another for backups), you could either create 3 targets or you could create a single target with 3 LUN’s. The advantage of using multiple LUN’s is that all 3 would only have a single Target IQN making management much easier.

ACL’s: For Your Eyes Only

iSCSI Targets can be assigned Access Control Lists (ACL’s). These ACL’s are nothing more than a list of IQN’s that are allowed to access a given target. ACL’s are one way to enable security in our iSCSI environment and ensure that only the initiators that should see a given target can. We’ll discuss more security techniques shortly.

Target Portal Group Tags: Chose a lane, any lane

iSCSI Targets can also be assigned to Target Portal Groups using a Target Portal Group Tag (TPGT). This sounds a lot more complicated and fancy than it is: quite simply a TPGT is a numeric representation of which IP addresses a target is allowed to bind on. Its a lot like the “Listen: 127.0.0.1” paremeter in your web server or mail server configuration. Without defining a TPG initiators can connect to your server on any interface, which is fine if you only have 1, but if you have multiple NIC’s in a machine connected to multiple networks (a public and private for instance, or a private and a dedicated storage network) you might want to force that target to only be accessable on specific interfaces, this is what TPGT’s are for. Here’s an example:

[thumper3:/] root# /sbin/ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
aggr1: flags=201000843 mtu 1500 index 2
        inet 172.165.92.100 netmask ffff0000 broadcast 172.165.255.255
e1000g0: flags=201000843 mtu 1500 index 3
        inet 4.4.4.100 netmask ffffff00 broadcast 4.4.4.255
e1000g1: flags=201000843 mtu 1500 index 4
        inet 10.10.115.100 netmask ffff0000 broadcast 10.10.255.255

On the above system I have setup “aggr1” as a 2Gbps aggregate as a dedicated storage network. When I create targets I don’t want them accessable on the public 4.x interface and I don’t want them congesting or competing with my other traffic on the slower 1Gbps 10.x network, therefore I’ll create a TPGT 0 with the IP address of the interface I want to restrict access to: 172.165.92.100. Now whenever I create a new iSCSI Target I assign it to TPGT 0 and life is good.

Target Portals: Your Gateway to Kickin’ Storage

An iSCSI Target Portal, or simply “a portal”, is an access point to one or more targets. Simply put its an IP address and port number to which you access targets. The TPGT’s that we just spoke about are ways to bind them together.

Backing Stores: The Real Deal

As we said earlier, iSCSI is all about making a block device (or what seems like one) accessable over the network. A Target is a way to put it out there, but its really just a connector between the initiator and some storage on the Target system. This storage is properly referred to as a “backing store”. Backing stores don’t have to be real block storage devices actually, in many cases the “backing store” is just a big pre-allocated file thats treated like a block storage device thanks to emulation in the target. This requires targets to have types.

The Solaris iSCSI Target Implementation currently has 4 types available, which are:

• disk: This type emulates a disk device. Backing store can be a single pre-allocated file or a disk volume accessed by its raw device (/dev/rdsk/myvol). 99% of all iSCSI targets are this type.
• tape: This type emulates a tape device. Backing store can be a single pre-allocated file, disk volume, etc. This type is handy for using as an iSCSI accessible virtual tape device.
• osd: This type is for Object-Based Storage Devices (OSD). For now, just ignore it, its basically a place-holder for the future. Learn more about OSD here.
• raw: This type might better be refered to as “pass-through”. This type does no emulation, it just passes SCSI commands to the backing store using the Solaris uscsi(7I) interface. This can only be used for real SCSI/FC disks. If your using volume managers, RAID, etc, you do not want this type.

Global Unique ID’s

As we stated earlier, each iSCSI Target can have one or more LUN’s. IQN’s are defined for Targets but not LUN’s, so how do we know which is which? On most implementations that I’ve seen this is simply done by specifying both the Target IQN and the LUN ID (iqn.1992-08.com.netapp:sn.33582139.0, where .0 represents the LUN). Solaris instead assigns each LUN a Global Unique ID, or GUID, which is guaranteed as unique. This GUID is composed of the Target Portal MAC address and a timestamp and looks like this: 010000144f20fd1a00002a00454329b. (0:14:4f:20:fd:1a + 00002a00454329b)

GUID’s are important for really only one reason: the GUID is used on the initiator side as the device identifier. Lets look at targets and what they look like on an initiator using the format command:

[thumper3:/] root# iscsitadm list target -v
Target: mytarget
    iSCSI Name: iqn.1986-03.com.sun:02:216a97f3-95fa-e064-f418-bd7c1309055f.mytarget
    Connections: 1
        Initiator:
            iSCSI Name: iqn.1986-03.com.sun:01:e00000000000.4505d02b
            Alias: atlantis
    ACL list:
    TPGT list:
    LUN information:
        LUN: 2
            GUID: 010000144f20fd1a00002a00457fd7ac
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 1.0G
            Status: online
        LUN: 1
            GUID: 010000144f20fd1a00002a00457fd7a5
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 1.0G
            Status: online
        LUN: 0
            GUID: 010000144f20fd1a00002a00457fd373
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 1.0G
            Status: online

So here is a single target with no ACL’s defined, not TPGT’s associated, having 3 LUN’s. Each of those LUN’s are 1GB files using the “disk” type. You can see that “atlantis” has a connection to the target. Lets see what it looks like on that side:

[atlantis:/] root# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t2d0 
          /pci@0,0/pci1022,7450@2/pci1000,3060@3/sd@2,0
       1. c0t3d0 
          /pci@0,0/pci1022,7450@2/pci1000,3060@3/sd@3,0
       2. c4t010000144F20FD1A00002A00457FD7A5d0 
          /scsi_vhci/disk@g010000144f20fd1a00002a00457fd7a5
       3. c4t010000144F20FD1A00002A00457FD7ACd0 
          /scsi_vhci/disk@g010000144f20fd1a00002a00457fd7ac
       4. c4t010000144F20FD1A00002A00457FD373d0 
          /scsi_vhci/disk@g010000144f20fd1a00002a00457fd373
Specify disk (enter its number): 

Okey, this is why GUID’s are important. Notice that long ass number used for the target number? Thats the GUID seen above in the target output and is defined by the target the first time its accessed (newly created unused targets may have a GUID of “0x0”, which will change the first time you use it). This solves an important problem for us: how do I know what is what? If you understand what GUID’s are and where to find them you can look at an iSCSI physical disk path on an initiator and know exactly what it is.

This also illustrates a shortfall in in the Solaris iSCSI Target Implementation. Notice that even though there is 1 target with 3 LUN’s, each LUN is represented as a separate target on the initiator side. If you look at the output of iscsiadm list target on the initiator you’ll only see 1 target, but using a disk command like format you’ll see 3. Whether this will change in the future or not I don’t know, but you definitely should be aware of it.

Summing Up

So, with any luck, if you’ve read all that above, the following output should make sense to you:

[thumper3:/] root# iscsitadm list tpgt -v
TPGT: 1
    IP Address: 172.16.165.100
[thumper3:/] root# iscsitadm list target -v
Target: mytarget
    iSCSI Name: iqn.1986-03.com.sun:02:216a97f3-95fa-e064-f418-bd7c1309055f.mytarget
    Connections: 1
        Initiator:
            iSCSI Name: iqn.1986-03.com.sun:01:e00000000000.4505d02b
            Alias: atlantis
    ACL list:
        Initiator: atlantis
    TPGT list:
        TPGT: 1
    LUN information:
        LUN: 2
            GUID: 010000144f20fd1a00002a00457fd7ac
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 1.0G
            Status: online
        LUN: 1
            GUID: 010000144f20fd1a00002a00457fd7a5
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 1.0G
            Status: online
        LUN: 0
            GUID: 010000144f20fd1a00002a00457fd373
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 1.0G
            Status: online

Hopefully we’ve laid some foundation on which we can now build and learn. In future posts we’ll look at how to use these concepts to actually build targets, configure them, access them, secure them, and more.