Some time ago I wrote a big blog entry entitled Simplifying Zone Management with Kerberos. Since that time (3 years ago!?!) several improvements and simplifications have come along to make life far more pleasant. If your new to Kerberos, please refer to my previous entry before proceeding with this one.

Kerberos is still something I struggle to really wrap my head around. Thats probably due to the fact that I have never used it in a full production deployment… the problem with using something only in the lab is that you never are “done” and you never hit real world problems that push you beyond that which you dream up.

There are classicly two reasons people get interested in Kerberos: Single Sign On (SSO) and secure NFS. Once upon a time general purpose encryption support for tools like telnet, rlogin, and FTP were a great draw, but today between SSH and SSL/TLS support in almost everything there just isn’t a reason to jump down the Kerberos path. SSO is a reason, but given that the defacto UNIX remote management protocol is SSH and SSH supports passwordless (authorized keys) login, its easy to mimic the benefits of SSO…. particularly if you use OpenSSH LPK, which allows you to store public keys in LDAP. As for NFS, well, there you probly still wanna stick with Kerberos unless you are willing to go the NFSv4/IPsec route.

The point here being, prior to SSH’s rise to glory Kerberos was a vitally important enterprise solution, whereas today its a hard sell. When it comes down to it, there is really only one real standout reason to seriously consider Kerberos: passwords are never on the wire, period. That is indeed a kool thing, but I doubt it will convince most people enough to really get them to go at it. In the end, Kerberos is only really exciting when its implemented in a way such as ActiveDirectory, where you don’t even know its there, you just turn it on and reap the benefits. So you can almost conclude that Kerberos is as important as ever, its just no longer worth the expenditure of effort.

To this end, I advise anyone interested in LDAP/Kerberos get familiar with ApacheDS, the work their doing is truly amazing. …but more about that some other time.

As I was saying, Kerberos is now easier than ever to get up and running on Solaris, thanks to the kdcmgr and kclient tools.

Thanks to the kdcmgr command we no longer need to hand edit files and use the kinit command to initialize a Kerberos KDC. Here is an example of kdcmgr in action:

$ kdcmgr -a benr/admin -r CUDDLETECH.COM create master

Starting server setup
---------------------------------------------------

Setting up /etc/krb5/kdc.conf.

Setting up /etc/krb5/krb5.conf.

Initializing database '/var/krb5/principal' for realm 'CUDDLETECH.COM',
master key name 'K/M@CUDDLETECH.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify: 

Authenticating as principal root/admin@CUDDLETECH.COM with password.
WARNING: no policy specified for benr/admin@CUDDLETECH.COM; defaulting to no policy
Enter password for principal "benr/admin@CUDDLETECH.COM":
Re-enter password for principal "benr/admin@CUDDLETECH.COM":
Principal "benr/admin@CUDDLETECH.COM" created.

Setting up /etc/krb5/kadm5.acl.

---------------------------------------------------
Setup COMPLETE.

Done! If you look around you’ll see this has created the necessary config and keytabs, as well as started the appropriate SMF services:

$ svcs -a | grep security
disabled       May_30   svc:/network/security/krb5_prop:default
online         May_30   svc:/network/security/ktkt_warn:default
online          3:11:34 svc:/network/security/krb5kdc:default
online          3:11:36 svc:/network/security/kadmin:default
$ ls -alh /etc/krb5/
total 40
drwxr-xr-x   2 root     sys          512 Jul 27 03:11 .
drwxr-xr-x  96 root     sys         4.5K Jul 25 14:58 ..
-rw-r--r--   1 root     sys           52 Jul 27 03:11 kadm5.acl
-rw-r--r--   1 root     root         998 Jul 27 03:11 kadm5.acl.sav
-rw-------   1 root     root        1.5K Jul 27 03:11 kadm5.keytab
-rw-r--r--   1 root     sys          430 Jul 27 03:11 kdc.conf
-rw-r--r--   1 root     root        1.3K Jul 27 03:11 kdc.conf.sav
-rw-r--r--   1 root     sys          968 Mar 22 10:43 kpropd.acl
-rw-r--r--   1 root     sys          367 Jul 27 03:11 krb5.conf
-rw-r--r--   1 root     root        2.0K Jul 27 03:11 krb5.conf.sav
-rw-------   1 root     root         383 Jul 27 03:11 krb5.keytab
-rw-r--r--   1 root     sys         1.1K Mar 22 10:43 warn.conf

As you can see, its all there and in order. You can immediately use kinit to get a ticket and use kadmin to create additional principles, no kadmin.local required. Very refreshing indeed.

As for the clients, manual configuration is so simple as to require very little improvement, never the less we can use the kclient command to standardize it:

# kclient -a benr/admin -k kdc1.cuddletech.com -R CUDDLETECH.COM

Starting client setup

---------------------------------------------------
Do you want to use DNS for kerberos lookups ? [y/n]: n
        No action performed.
kdc1.cuddletech.com

Note, this system and the KDC's time must be within 5 minutes of each other for Kerberos to function.  Both systems should run some form of time synchronization system like Network Time Protocol (NTP).

Setting up /etc/krb5/krb5.conf.
Obtaining TGT for benr/admin ...
Password for benr/admin@CUDDLETECH.COM:
localhost: RPC: Rpcbind failure - RPC: Success
kinit:  no ktkt_warnd warning possible

host/zone1.cuddletech.com entry ADDED to KDC database.
host/zone1.cuddletech.com entry ADDED to keytab.

---------------------------------------------------
Setup COMPLETE.

Easy as that. Please note that the RPC failure did suceed and didn’t effect the configuration.

I want to note explicitly that in the above examples the hosts involved, namely the KDC, were present in DNS. Trying to do the same configuration above without DNS is not so easy and if you need to do so I recommend avoiding kdcmgr and using the manual procedure in my previous Kerberos post.

With regard to SSH, in Solaris 10 there is no need to make any changes to accomidate SSH. You do not need to edit /etc/pam.conf or even change the /etc/ssh/sshd_config file. Setup a KDC as above, setup a client as above (a zone for instance), get a ticket (“kinit”) and SSH… your rockin’ and rollin’.

If you need help with your installation or want to get connected with other Solaris Kerberos fans, visit the OpenSolaris Kerberos Project.

Today is everyones favorite day of the year: SysAdmin Day! To all my fellow admins, from myself and Joyent, a very warm pat on the back and “thank you” for all that hard work that no one else acknowledges. So kick back, enjoy a cold pint, and bask in the glory that is UNIX Systems Administration!

This is very important, WarGames, the most important geek film ever made, is in theaters TONIGHT at 7:30pm. ONE SHOWING ONLY.

This film has been hugely influential for me, and many others. That awesome unbuttoned shirt over tshirt look that we emulate to this day, decades of trying to get text-to-voice to sound like the film (which was an actor reading the lines word by word in reverse), and who didn’t fall in love with Ally Sheedy!!!

Don’t be dumb!!! Go! Tonight! 7:30PM!!!

Recently introduced (snv_92) is the first piece of the DTrace Network Providers, the DTrace IP Provider. Here is a taste:

root@ultra include$ dtrace -qn 'ip:ip:*:receive{ printf("Packet recieved from %s: %d byte packetn", args[2]->ip_saddr, args[4]->ipv4_length ); }'
Packet recieved from 74.125.15.85: 40 byte packet
Packet recieved from 74.125.15.85: 40 byte packet
Packet recieved from 8.11.47.20: 88 byte packet
Packet recieved from 8.11.47.20: 216 byte packet
Packet recieved from 8.11.47.20: 200 byte packet
Packet recieved from 8.11.47.20: 136 byte packet
Packet recieved from 8.11.47.20: 104 byte packet
^C

Pretty soon snoop and tcpdump will be nothing more than unpleasant memories.

A big thank you to the DTrace Team!!!

In this entry we’ll build on our our IPsec Basics discussed last time and actually create an IPsec connection.

IPsec can be used for direct system-to-system access known as “transport mode” or to create a virtual pipeline into which everything is encrypted, known as “tunnel mode”. We’re going to look at transport mode, which is an excellent solution for encrypting otherwise unencrypted protocols, such as SNMPv1/2 or telnet.

When encrypting and decrypting data we need keys. This can be done using PKI certificates or IKE generated one-time keys, but in this examples for simplicity sake we’ll create our own “static” keys which will be used on both ends of the connection, thus said to be “pre-shared”.

Creating Keys

Using the ipsecalgs command we can see the available algorithms, including DES, 3DES, AES, Blowfish, SHA and MD5. Different alogithms require different key lengths, for instance 3DES requires a 192 bit key, whereas Blowfish can use a key anywhere from 32bits up to 448 bits.

For interoperability reasons (such as OSX or Linux), you may with to create keys that are both ASCII and hex. This is done by choosing a string and converting it to hex. To know how long a string should be, divide the number of bits required by 8, this is the number of ASCII chars you need. The hex value of that ASCII string will be double the number of ASCII chars. Using the od utility we can convert ASCII-to-hex. Here I’ll create 2 keys, one for AH which is a SHA1 160bit key (20 ASCII chars) and another for ESP which is a Blowfish 256bit key (32 ASCII chars):

benr@ultra ~$ echo "my short ah password" | od -t x1
0000000 6d 79 20 73 68 6f 72 74 20 61 68 20 70 61 73 73
0000020 77 6f 72 64 0a
0000025
benr@ultra ~$ echo "this is my long blowfish esp pas" | od -t x1
0000000 74 68 69 73 20 69 73 20 6d 79 20 6c 6f 6e 67 20
0000020 62 6c 6f 77 66 69 73 68 20 65 73 70 20 70 61 73
0000040 0a
0000041

To ensure proper length, I like using a little text-rule like you see below in vi:

         1         2    2    3 3       4         5         6   6     7
1234567890 234567890 234567890 234567890 234567890 234567890 234567890
--------------------------------------------------------------------------------------------------------------------
my short ah password
6d792073686f72742061682070617373776f7264

this is my long blowfish esp pas
74686973206973206d79206c6f6e6720626c6f77666973682065737020706173

If you don’t require interoperability by knowing the ASCII equivilent, just grab a random set of hex chars (head /dev/random | od -t x1).

Now that we have a key, lets use it.

Configuring IPsec Policies

IPsec policies are rules that the IP stack uses to determine what action should be taken. Actions include:

• bypass: Do nothing, skip the remaining rules if datagram matches.
• drop: Drop if datagram matches.
• permit: Allow if datagram matches, otherwise discard. (Only for inbound datagrams.)
• ipsec: Use IPsec if the datagram matches.

As you can see, this sounds similar to a firewall rule, and to some extent can be used that way, but you ultimately find IPFilter much better suited to that task. When you plan your IPsec environment consider which rules are appropriate in which place.

IPsec policies are defined in the /etc/inet/ipsecinit.conf file, which can be loaded/reloaded using the ipsecconf command. Lets look at a sample configuration:

benr@ultra inet$ cat /etc/inet/ipsecinit.conf
##
##  IPsec Policy File:
##

# Ignore SSH
{ lport 22 dir both } bypass { }

# IPsec Encrypt telnet Connections to 8.11.80.5
{ raddr 8.11.80.5 rport 23 } ipsec { encr_algs blowfish encr_auth_algs sha1 sa shared }

Our first policy explicitly bypasses connections in and out (“dir both”, as in direction) for the local port 22 (SSH). Do I need this here? No, but I include it as an example. You can see the format, the first curly block defines the filter, the second curly block defines parameters, the keyword in between is the action.

The second policy is what we’re interested in, its action is ipsec, so if the filter in the first curly block matches we’ll use IPsec. “raddr” defines a remote address and “rport” defines a remote port, therefore this policy applies only to outbound connections where we’re telnet’ing (port 23) to 8.11.80.5. The second curly block defines parameters for the action, in this case we define the encryption algorithm (Blowfish), encryption authentication algorithm (SHA1), and state that the Security Association is “shared”. This is a full ESP connection, meaning we’re encrypting and encapsulating the full packet, if we were doing AH (authentication only) we would only define “auth_algs”.

Now, on the remote side of the connection (8.11.80.5) we create a similar policy, but rather than “raddr” and “rport” we use “laddr” (local address) and “lport” (local port). We could even go so far as to specify the remote address such that only the specified host would use IPsec to the node. Here’s that configuration:

##  IPsec Policy File:
##

# Ignore SSH
{ lport 22 dir both } bypass { }

# IPsec Encrypt telnet Connections to 8.11.80.5
{ laddr 8.11.80.5 lport 23 } ipsec { encr_algs blowfish encr_auth_algs sha1 sa shared }

To load the new policy file you can refresh the ipsec/policy SMF service like so: svcadm refresh ipsec/policy. I recommend avoiding the ipsecconf command except to (without arguments) display the active policy configuration.

So we’ve defined policies that will encrypt traffic from one node to another, but we’re not done yet! We need to define a Security Association that will association keys with our policy.

Creating Security Associations

Security Associations (SAs) can be manually created by either using the ipseckeys command or directly editing the /etc/inet/secret/ipseckeys file, I recommend the latter, I personally find the ipseckeys shell very intimidating.

Lets look at a sample file and then discuss it:

add esp spi 1000 src 8.15.11.17 dst 8.11.80.5 auth_alg sha1 authkey 6d792073686f72742061682070617373776f7264 encr_alg blowfish encrkey 6d792073686f72742061682070617373
add esp spi 1001 src 8.11.80.5 dst 8.15.11.17 auth_alg sha1 authkey 6d792073686f72742061682070617373776f7264 encr_alg blowfish encrkey 6d792073686f72742061682070617373

It looks more intimidating that it is. Each line is “add”ing a new static Security Association, both are for ESP. The SPI is the “Security Parameters Index”, is a simple numeric value that represents the SA, nothing more, pick any value you like. The src and dst define the addresses to which this SA applies, note that you have two SA’s here, one for each direction. Finally, we define the encryption and authentication algorithms and full keys.

I hope that looking at this makes it more clear how policies and SA’s fit together. If the IP stack matches a datagram against a policy who’s action is “ipsec”, it takes the packet and looks for an SA who’s address pair matches, and then uses those keys for the action encryption.

Note that if someone obtains your keys your hosed. If you pre-shared keys in this way, change the keys from time-to-time or consider using IKE which can negotiate keys (and thus SAs) on your behalf.

To apply your new SA’s, flush and then load using the ipseckeys command:

$ ipseckey flush
$ ipseckey -f /etc/inet/secret/ipseckeys

Is it working? How to Test

All this is for nothing if you don’t verify that the packets are actually encrypted. Using snoop, you should see packets like this:

$ snoop -d e1000g0
Using device e1000g0 (promiscuous mode)
ETHER:  ----- Ether Header -----
ETHER:
ETHER:  Packet 1 arrived at 9:52:4.58883
ETHER:  Packet size = 90 bytes
ETHER:  Destination = xxxxxxxxxxx,
ETHER:  Source      = xxxxxxxxxx,
ETHER:  Ethertype = 0800 (IP)
ETHER:
IP:   ----- IP Header -----
IP:
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 72 bytes
IP:   Identification = 36989
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 61 seconds/hops
IP:   Protocol = 50 (ESP)
IP:   Header checksum = ab9c
IP:   Source address = XXXXXXXXX
IP:   Destination address = XXXXXXXXXXXX
IP:   No options
IP:
ESP:  ----- Encapsulating Security Payload -----
ESP:
ESP:  SPI = 0x3e8
ESP:  Replay = 55
ESP:     ....ENCRYPTED DATA....

And there you go. You can no encrypt communication transparently in the IP stack. Its a little effort to get going, but once its running your done… just remember to rotate those keys every so often!

Why do I care about this again?

In this modern era where SSH is the standard for communication its easy to get jaded. Either you can communicate via SSH or easily create a tunnel to get the job done. But lets face it, SSH is massively overused, and in many cases SSH tunneling is just downright ghetto. With IPsec we can as easily encrypt 100 ports as 1, whereas with SSH thats very ugly. Furthermore, there are many instances in which you want a secure communications channel thats as transparent as possible, such as a network database connection that doesn’t offer native encryption or perhaps an SCM or even SNMPv1/2.

While most applications today provide some type of encryption capability it surprising how few people leverage them unless they are the default. In situations where its difficult or impractical to use encryption in the application, IPsec can be a really sweet solution.

IPsec is a technology widely known. Created for IPv6 and backported to IPv4, it adds a security layer into the IP stack. Prior to IPsec we needed to encrypt data before sending it down the stack and then decrypt it on the other side once it came back out using technologies like SSH or SSL/TLS. IPsec simplifies this by transparently handling encrypt/decrypt as well as header authentication.

But for all its intended simplicity and transparency it is to the administrator anything but. If you google around you find piles of email on various lists of those who tried to get IPsec working and gave up in frustration and instead used OpenVPN, on both Linux and Solaris.

Part of the reason IPsec is so complex to manage is the number of technologies that all must work together properly in order to get something functional. This includes acronyms like AH, ESP, IKE, DH, RSA, SA, SAD, etc. In many respects its similar to the frustration encountered the first time you approach LDAP and are overwhelmed with OU=, CN=, DN=, etc. Just like LDAP, with a little practice you get it sorted out and start making ground. We’ll start high level and zoom in… please please, read this before just copying examples! (I know you won’t, but it’ll save you time.)

IPsec is fundamentally similar to a firewall, in that you specify IPsec Policies that determines how datagrams are handled. If a policy matches a datagram its handled… if no policy matches it acts like normal.

Thats where the firewall analogy stops, like a firewall the policy says what to do and when to do it…. but we need to know something else for encrypting or decrypting data, we also need to know how to do it. The information that explains the how is known as a Security Association (SA). This is maintained in a special database named, unremarkably, the Security Association Database (SAD or SADB, depending on who you ask, same thing though).

It’s the SA that actually contains the keys used for encrypting, decrypting and authenticating IPsec datagrams. So a packets goes through the stack, a policy says “If the datagram source is 1.2.3.4 and the destination is 5.6.7.8 use IPsec.” Now that the stack knows to use IPsec to encrypt, it goes looking in the SADB for an SA that contains the keys. There is a lot of duplication here because they are independent things, so you specify the authentication algorithm, encryption algorithm, source and destination addresses in both the IPsec policy and the SA. Its strange at first.

Now, just as IPsec is supposed to make life simple and transparent, so is IKE (Internet Key Exchange). An IKE daemon runs on both sides of a connection and negotiates SA’s for you. This provides a variety of benefits, but simplicity isn’t exactly one of them because you still need to configure IKE rules which are similar to SA’s. In other words, you can’t just create an IPsec policy, and then enable IKE and be done.

Before we get into the examples, lets looks at the various files and commands involved:

Files:

    * IPsec:
          o /etc/inet/ipsecinit.conf: IPsec Policy Definitions
          o /etc/inet/secret/ipseckeys: IPsec SA Definitions
    * IKE:
          o /etc/inet/ike/config: IKE Global Configuation and Rules
          o /etc/inet/secret/ike.preshared: File containing Preshared Key Definitions
          o /etc/inet/secret/ike.privatekeys/: Directory containing IKE Private Keys
          o /etc/inet/ike/publickeys/: Directory containing IKE Public Keys
          o /etc/inet/ike/crls/: Directory containing IKE Certificates 

Commands:

    * IPsec:
          o ipsecconf: Load or Display IPsec Policy Configuration
          o ipsekkey: Manually manipulate IPsec Security Association Database (SADB)
          o ipsecalgs: Display available IPsec ESP/AH Algorithms
    * IKE:
          o in.iked: IKE Daemon
          o ikeadm: Manipulate IKE parameters and state (flush, add, get, set, ...)
          o ikecert: Manipulate IKE's on-filesystem public-key certificate databases

The simplest configuration would be to “pre-share” keys, meaning create a key and manually put it on both systems. In my next blog entry we’ll step through actually creating an IPsec policy, creating keys, creating Security Associations, and testing an IPsec connection.