It snuck up on me this year… I guess thats a good sign that we’re all working hard. Happy SA Day! May your coffee cups stay full, donuts fresh, and systems stable.
Archive for July, 2007
Happy SA Day!
Friday, July 27th, 2007OSCON Sun/Joyent Party Tonight!
Wednesday, July 25th, 2007Sun and Joyent are hosting a party tonight in the Portland DoubleTree Garage… yes, garage, because ballrooms are for debutantes and wimps. We’ll have a DJ, free b33r, and a good time. Such Sun 3l173 as David Comey, Sara Dornsiff, Brenden Gregg, Simon Phipps, Adam Leventhal, and others will be present in addition to Jason Hoffman, Luke Crawford, and of course myself.
If your at OSCON be there! Rock the g4r4ge h4rdc0r3 with Sun and Joyent.
Eighth Wedding Anniversary
Wednesday, July 18th, 2007Its hard to believe, but Tamarah and I have been married for 8 years. Add our 5 years of dating and we’ve been side-by-side every day for more than 13 years.
We celebrated by having dinner at CIA in Napa and spending the evening at a nearby bed and breakfast, The Chanric Inn, in Calistoga.
This was the first time we’ve spent a night away from our children together, from time to time we get a baby sitter and go out (very rarely, unfortunately) but never for the night. We called too often, but everything went well. They stayed with one of the pastors of our church and it so happens that Nova’s best friend is his daughter, so it was sorta like a sleep-over slumber party for her.
A very relaxing evening I’m glad to say. No busy city, not packed and crammed boring restaurants. Great food, great service, and a quiet, restful night. Gotta love it.
Getting out of the office
Friday, July 13th, 2007SysAdmin Mentoring, Organizations, and Growing Our Craft
Sunday, July 8th, 2007This is a very important year for me. I’m going through an incredible amount of personal growth, personally and professionally. I’ve hit that point in my life where I’m out of culturally imposed goals… the midlife lull I’ll call it. When your a kid you have clear cut goals… namely, move the hell out. When you do, you have more clear cultural goals: get an education, good job, nice car, nice home (whatever “home” means to you), wife, kids, build a career, build a chain of accomplishments, etc. The midlife lull is where you find yourself once you’ve reached all those goals. You have a good job, nice car, wife and kids, etc, and are now into that “just keep doing it till you retire” phase of life. I’m asking new questions strikingly similar to those when your young… “Am I doing what I want to?”, “Where do I really want to go next?”, “What is the purpose of my life (aside from or in addition to any religious beliefs)?” The big review.
All of these questions have jolted me free of my youthful belief that I’m alone in the universe and no one else can help. I’m reading books on every topic of personal development and growth looking for both answers and perhaps most importantly the right questions to ask of myself. I’m glad to report that its paying massive dividends so far, and I still have a long way to go.
Along with all this, my coming onto Joyent as a Director has given me a new career direction that I hadn’t previous planned on. Initially just adding a yet larger title to my resume was a draw (along with the ability to solve just about every problem known to man using iSCSI). As time went on and I started building out a team I found that what I really enjoyed, even more than the work, was working with other sysadmins in a managerial role. Thats something I never would have imagined.
We’ve all worked for managers that didn’t understand our roles, talents, abilities, or jobs. I’ve grown very tired of quarterly reviews that would follow the template “Ben is excellent technically, but has trouble being ontime to meetings, interacting with co-workers, ….” The focus always being on everything but the job. I’m not trying to shrug off my non-technical skills, indeed I’ve worked very hard to improve in those areas so as not overshadow my other abilities, but the point is that the parts of our jobs that we really work hard at commonly get smoothed over because managers simply don’t know how to evaluate performance in technical areas. I crave constructive feedback. I’m self-analytical and want to constantly improve myself and my performance and that requires input. Targeted, dispassionate criticism is just as useful as praise and accolades; both are essential in equal quantities.
This extends into something even more important to me… career growth. We, as a profession, don’t tend to stay anywhere long (being defined as more than 4 years). Promotion almost always occurs due to changing companies. Very few companies that I’ve worked with promote from within. A lot of companies don’t even think about promotion, I think, they simply heap more responsibility on employees. The mail admin wants to be more involved with storage and so starts working with the storage team here and there and maybe takes over more control of the storage systems related to his area, but rarely do admins get shifted between operational groups, given a new office, title, etc. Those that do are very large corps, but it still doesn’t happen enough in those organizations, its too easy to take someone from the outside than train someone up, even if that just means personally challenging them.
Most admins are, therefore, islands unto themselves, reinventing the wheel all the time. We are our own career advisers, training specialists, and head hunters. Why? Because we’re also very ego driven people. We pride ourselves on going it alone and doing it ourselves. For me this has come and gone in waves… I crave a job where I’m alone in the data center able to set my pace, set my priorities, do it my way. After a time I get very lonely and crave a team. I then transition to a team environment and enjoy sharing ideas, learning from peers, and being collaborative… but this to gets old after awhile due to frustrations with certain members of the team, ignorant management, or a host of other issues that come up in a big team environment. I’ve gone back and forth enough times that I got to see the patterns emerge in myself. I’m extremely thankful to Taos and my time there that moved me between gigs frequently enough that I was able to learn these things about myself, it really was a gift, most people will work for 10 or more years before having that size dataset from which to analyze themselves and their performance.
I really enjoy thinking about members on my team, who they are, what they are strong in, where they have potential that is potentially unknown even to themselves, and working with them to draw out that potential in a way that will benefit both the organization and individual long term. Being a technical guide or at least sounding board. Being able to task them outside of their comfort zone to challenge them. Being able to acknowledge their work, commitment, and provide the constant feedback which improves moral and personal satisfaction. Its vitally important, I think, that sysadmins know that they are doing a good job, are appreciated, and just know where they stand. Leaving “the office” on Friday knowing that you’ve done the best job that you can, whether it was a good week or bad, is really important. Stress builds up when you have that nagging voice in your head saying “You’re not doing enough!”, “They are gonna fire you any day now!”, “You’re a fraud, you’ve got them fooled now, but next week you’ll hit a problem you can solve and you’re done!” Being able to just answer those fears on a weekly basis from your management can go a long, long way to making your life a lot less stressful and more enjoyable both on and off the job.
Now… I’ve watched organizations like SAGE and now LOPSA for years. I like that feeling of belonging to an organization. But I constantly ask myself why I should get behind these organizations. I’ll be perfectly honest, in my mind at least, they are bureaucratic discount clubs. You save a couple bucks on books from this place or that, they organize conferences that I can’t afford to attend, and …. um… provides meaningless mailing lists? Whats the point? They “advance the profession of systems administration”. Um, how? By doing conferences? LISA and USENIX are great conferences, I’m told, but I’ve got a family that I don’t want to leave for a week, a job that demands constant attention, and paying for the conference, travel, hotels, etc, is only something thats possible by working for a company that is willing to flip the bill. So when a conference isn’t running, what value do I get for my membership fees? For this reason I basically join up when I spruce up my resume simply to have it on there. I’ll admit my own hypocritical view that I look down on any job applicant that isn’t a member of one of these. At least IEEE and ACM give you access to some great resources (the ACM library is awesome).
Now… we pair these things up. I’ve grown more and more convinced that a useful thing to have would be a really solid mentoring program at some admin organization. A “Career Counselor In The Sky” of sorts. A formal program of peers which could provide long term support for admins in both technical and non-technical aspects of the job. Mentors that could answer day-to-day questions, but also ask the “Where do you want to go with your career?” questions, and help create personalized paths for admins to take regardless of where they work. Applicants would be paired up with a mentor, and mentors would themselves have mentors. If a solid program was in place that really helped people I think it could actually achieve the goals that these organizations espouse.
Such a program would have to be very structured. Not just an informal “You email this guy if you have questions” hookup service. Rather, every participant should have a long term file to allow a stateful transition over time and chart progress. Weekly discussions by phone would provide the mentoring people need and dispassionate feedback that fuels growth. Periodic full skills reviews could allow participants to see their own growth over time and hone in on areas that they want to grow, rather than just floundering day-by-day based on immediate needs. This needs to be more than a “Sendmail is b0rked, any idea why?” Rather this is a higher level “What would you do in my situation?” ongoing conversation that provides that external viewpoint that helps individuals overcome roadblocks due to perspective. It provides that “You wanted to grow into Network Administration, but you keep growing your database skills, are you looking for opportunities to re-focus your energy? Are you growing into the wrong field?” check.
SA Mentoring isn’t a new concept. Long before I’d even considered such a thing discussions where flying. As recently as 2006, it was a topic at LOPSA, and apparently a very passionate one. In that note, Some thoughts on “mentoring”, its stated that its been tried and failed, apparently several times, in the past. What those attempts were and how they failed I don’t know. Its certainly not a simple task, your going to have to go through all the same hurtles that a new business would go through. You’ve got to bootstrap and organization, implement procedures, bring in participants, connect everyone up, track their progress, and ensure that things don’t break down after the initial “Seems like a good idea!” buzz wears off. There needs to be incentives and rewards for participation, and they need to be recognized in the field. You have to build a track record and respect in the industry and become a stable facility that managers and peers can send people to who might benefit from. You’ve got to really publicize it, so that the people who need the most help, those that feel alone and isolated, find and have the opportunity to embrace it. Its no small task to be sure, and not something that can be done half heartedly.
Is such a thing even possible? I think so. It would require a lot of commitment and a lot of time, but I think its something that, if well executed, could become an invaluable resource that would really take massive strides to really bringing the field together. Most consulting firms, such as Taos, already have systems like this in place for their consultants, it’s not a unique or novel venture, it just isn’t done on such a massive scale.
Will it happen? I don’t know. Ego is a big problem and a tough one to over come. Mentoring only works when there is real trust and building that trust takes time and requires participants to drop their guard to open themselves up. Commitment comes from results, otherwise its just another time sync, so you have to be improving peoples lives in a measurable way, not just giving them one more person to answer to. Can people open up? A better question is, because of the ego thing, would such a program have far more volunteers to be mentors rather than be mentored? Because of that I think you have to ensure that mentors are themselves mentored. And would potential participants be willing to take that first step? All that gets easier over time as you build a reputation around the program, but initially its really gonna be hard.
If something like this happens I’ll be thrilled… but until then, I’ll just continue to try to develop myself to be the best possible technical manager I can be and do everything I can for my team.
Giving Solaris Systems iSCSI IQNs That Make Sense
Tuesday, July 3rd, 2007When working with iSCSI nothing can be more confusing and frustrating than dealing with WWN-like cryptic IQN’s (unique strings that identify an initiator or target). I mean, how meaningful is this?: iqn.1986-03.com.sun:02:a182b87b-41f0-661f-c161-8de3fe28c5ef Not very.
SysAdmin Rule #102: Take cryptic non-sensical string, add some sleep deprivation, insert into production environment, result? Disaster.
Thankfully changing that random string is easy. (see target.h for how that string is determined)
First, find out what your initiators IQN is by default:
root@aeon ~$ iscsiadm list initiator-node Initiator node name: iqn.1986-03.com.sun:01:e00000000000.464d05f9
Now change it like so:
root@aeon ~$ iscsiadm modify initiator-node --node-name iqn.1998-07.com.cuddletech:aeon root@aeon ~$ iscsiadm list initiator-node Initiator node name: iqn.1998-07.com.cuddletech:aeon ...
Sweet and simple. Just remember to craft your IQN’s by the rules:
See Section 3.2.6.3.1 of RFC 3720 – Internet Small Computer Systems Interface (iSCSI): “Type “iqn.” (iSCSI Qualified Name)”
Please note that Target IQN’s may not be changed. Each Target is given a unqiue IQN and while you can change the Alias of a Target, you can’t change the IQN. This isn’t really a problem most of the time, however, because unlike NetApp Filers where a single storage system has a single IQN which is LUN masked, each Target has its distinct IQN. Besides that, the string used for the SCSI Target ID (ie: c2t0912309812039810983d0) is going to be funky anyway, nothing you can do about that. (That funky string is actually the targets GUID.)

