Solaris Extended Accounting
Posted on February 9, 2009
Extended Accounting is one of the many under appreciated features in Solaris, and quite possibly the worst documented to boot. So, it’s time to fix that.
Accounting, in general, is a means of recording data about resource utilization, CPU in particular, with the intention to be used primarily for reporting and billing purposes. Conceptually it could be confused with Auditing (see I See You!: Solaris Auditing (BSM)), in that they both record activity, except that the scope is vastly different. With BSM you can see every syscall/process/etc. that is executed, by who, when, with what privileges, etc; but the purpose of auditing is to determine what happened, when it happened, how it happened, and by who in the context of security. By contrast, Accounting is interested in what ran, how long it ran, how much resource it consumed, and some basic related information in the context of resource utilization.
When used in conjunction with Solaris’s array of observability tools, from BSM Auditing, to Kstats, to Dtrace, it provides a useful low impact way or maintaining a historical view of resource utilization more detailed than Kstats but more permanent than DTrace and more targeted than auditing. With kstats you can graph CPU usage over time, but with Extended Accounting you can determine the breakdown of CPU usage within the same period, for example.
The first thing you need to know about accounting on Solaris is that there are TWO separate accounting systems. The documentation did a poor job of making this clear for a long time. The UNIX Accounting system, we’ll call it, has been around for over a decade and is a hodgepodge of ancient scripts. Its still there and documented in the Solaris Admin Guide. Like mainframes or NIS+, if you’re not already using it you shouldn’t start now. In this blog entry we focus purely on the newer Solaris specific Extended Accounting facility.
Extended Accounting was added to Solaris in Solaris 9 along with the excellent microstate accounting enhancements. Solaris maintains all sorts of amazing stats about processes that can be explored with proc tools, and the many *stat tools (mpstat, vmstat, etc.) via options uncommonly used. The extended accounting facility really just provides a framework in which these stats can be stored in a datafile for later use, rather than be discarded when a process terminated. When exit() is run a call is made to extended accounting, and if enabled for tasks or processes the data is dumped into a record.
The framework is extensible for use in a variety of ways. For instance, when using Solaris IPQoS you can use extended accounting to record network flow data. As part of Crossbow a wonderful suprise was a new non-IPQoS flow extended accounting resource which I previously discussed here: Crossbow for Christmas.
Getting start with Extended Accounting (exacct for short) is easily done using the acctadm command:
$ acctadm
Task accounting: inactive
Task accounting file: none
Tracked task resources: none
Untracked task resources: extended
Process accounting: active
Process accounting file: /var/adm/exacct/proc
Tracked process resources: extended
Untracked process resources: host
Flow accounting: inactive
Flow accounting file: none
Tracked flow resources: none
Untracked flow resources: extended
Net accounting: active
Net accounting file: /var/adm/exacct/net
Tracked net resources: extended
Untracked net resources: none
Here you can see the various accounting types available: Task, Process, Flow (IPQoS), and Net (Crossbow). Each accounting type has a datafile to store records in which is user defined. A variety of different resources can be tracked for each accounting type, which can either individually selected or specified as either “extended” or “basic”, which are groupings of resources. To see what they are use acctadm -r:
$ acctadm -r process: extended pid,uid,gid,cpu,time,command,tty,projid,taskid,ancpid,wait-status,zone,flag,memory,mstate basic pid,uid,gid,cpu,time,command,tty,flag task: extended taskid,projid,cpu,time,host,mstate,anctaskid,zone basic taskid,projid,cpu,time flow: extended saddr,daddr,sport,dport,proto,dsfield,nbytes,npkts,action,ctime,lseen,projid,uid basic saddr,daddr,sport,dport,proto,nbytes,npkts,action net: extended name,ehost,edest,vlan_pid,vlan_tci,sap,priority,bwlimit,devname,src_ip,dst_ip,src_port,dst_port,protocol,dsfield,curtime,ibytes,obytes,ipkts,opkts,ierrpkts,oerrpkts basic name,devname,ehost,edest,vlan_pid,vlan_tci,sap,priority,bwlimit,curtime,ibytes,obytes,ipkts,opkts,ierrpkts,oerrpkts
To enable a given accounting type specify the file, resources, and then enable it with the -E option, like so:
root@quadra ~$ acctadm -f /var/adm/exacct/task task
root@quadra ~$ acctadm -e extended task
root@quadra ~$ acctadm -E task
root@quadra ~$ acctadm task
Task accounting: active
Task accounting file: /var/adm/exacct/task
Tracked task resources: extended
Untracked task resources: none
All the above is well documented and straight forward. The tricky part is actually using the data thats collected and dealing with long running processes.
As I stated before, the accounting records are recorded at process termination. That means that if you start a long running process, such as a MySQL daemon that runs for 6 months, you won’t see a record for it until its stopped. That sort of defeats the usefulness of accounting. Therefore, using the wracct command you can cause an individual process or task to write a “partial record”. In this case, partial differentiates if from a full record written at exit. So if you want to get accounting data for such processes, you’ll want a cron job that executes wracct against the process(es) at some given interval.
# wracct -i `pgrep arkeiad` -t partial process #
There are two tools provided with Solaris to examine extended accounting data files. One is the lastcomm command which is like a shells “history” command, showing a history of commands run based on the process accounting data. The second is actually a demo C application found in /usr/demo/libexacct called exdump. This demo app is intended to help developers learn the extended accounting library (libexacct), but is used all too commonly by administrators to output the contents of an accounting file.
If you really want to take advantage of accounting data you’ll need to build your own custom tools using either the Exacct PERL module (written by Sun, found in CPAN, and part of the Solaris PERL distribution) or the libexacct C library.
Getting started with the PERL modules can be confusing initially. You can find documentation for the module on CPAN and some examples in the System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.
The problem with the examples in the manual is that they mask some of the inner workings via the convenience “dump” function. Thus I present to you a modified version that is a little more explicit:
#!/usr/bin/perl
##
## This is a modified version of the 'dumpexacct' example from the
## 'System Administration Guide: Solaris Containers-Resource Management and Solaris Zones'
## In this version, rather than use the "dump()" convience function I instead navigate
## directly... this helps better illustrate how to actually use the Exacct PERL Module.
#
## benr@cuddletech.com - 2/7/09
use strict;
use warnings;
use Sun::Solaris::Exacct qw(:EXACCT_ALL);
die("Usage is $0 n") unless (@ARGV == 1);
# Open the exact file and display the header information.
my $ef = ea_new_file($ARGV[0], &O_RDONLY) || die(error_str());
printf("Creator: %sn", $ef->creator());
printf("Hostname: %snn", $ef->hostname());
my $obj_counter = 0;
# Dump the file contents
while (my $obj = $ef->get()) { ## Get an object from the EA File and reposition to next.
print("---------------- OBJECT $obj_counter -----------------------n");
$obj_counter++;
my $objectType = $obj->type(); ## Return the object type (group or item)
my $objectCatalogObj = $obj->catalog(); ## Return a catalog object for this object
my ($a, $b, $c) = $objectCatalogObj->value(); ## Breakout the catalog triplet
printf("Object is: %s - Catalog: %s %s %sn", $objectType, $a, $b, $c ); ## Output the catalog and object type for the object at hand.
if ( $objectType == EO_GROUP && $c == EXD_GROUP_PROC ) { ## If this is a PROC Group....
my @objectList = $obj->value(); ## Return the sub-group which will contain actual items.
foreach my $item (@objectList) {
my $itemsType = $item->type(); ## Get type for new object.. is it a EO_ITEM or EO_GROUP?
if ($itemsType == EO_ITEM){
my $itemCatalog = $item->catalog(); ## Return the "Catalog Object" for this object.
my ($itemType, $itemGroup, $itemId) = $itemCatalog->value(); ## Breakout the catalog triple
my $itemValue = $item->value(); ## Get the item value itself.
print("ttId: $itemId tValue: $itemValuen"); ## Now print just the catalog item id and item value.
} else {
print("ttERROR: Expected Item, got a group instead.n");
}
}
}
}
# Report any errors
if (ea_error() != EXR_OK && ea_error() != EXR_EOF) {
printf("nERROR: %sn", ea_error_str());
exit(1);
}
The example opens an accounting file and returns a file object. Using that file object we can get some metadata and via the get() method we can walk the file, because after each get() it repositions to the next object. An accounting file is a collection of objects, which come in the form of groups and items. Groups contain yet other groups or items. Items contain data. Each object has an associated catalog which is a triplet that describe the object in the form: type, group, and id.
When implementing your accounting program you’ll generally be walking the objects, then navigating groups and items to get to the data you want. To get your footing as to the structure of groups and types of data available in each item use the demo exdump tool and then start playing around. You’ll find the /usr/include/sys/exacct_catalog.h header useful in deciphering the catalog values. The heavy use of double-typed variables can be confusing at first, but the header will help it make sense.
As you can see, utilizing extended accounting isn’t for the casual user… but if you can find value in the data it collects you’ll reap the benefits of your labor learning the PERL module. If you are serious, I’d recommend taking the example above to break out selected items and formatting or sorting them in a more palatable way. Here is a simple modified version that outputs one line per record for a small handful of items:
use strict;
use warnings;
use Sun::Solaris::Exacct qw(:EXACCT_ALL);
die("Usage is $0 n") unless (@ARGV == 1);
# Open the exact file and display the header information.
my $ef = ea_new_file($ARGV[0], &O_RDONLY) || die(error_str());
printf("Creator: %sn", $ef->creator());
printf("Hostname: %snn", $ef->hostname());
## Print Header:
printf("%10s %4s %4s %5s %30sn", "Counter", "UID", "GID", "PID", "CMD");
my $idxCounter = 1;
# Dump the file contents
while (my $obj = $ef->get()) { ## Get an object from the EA File and reposition to next.
$idxCounter++; ## Count records.
my $objectType = $obj->type(); ## Return the object type (group or item)
my $objectCatalogObj = $obj->catalog(); ## Return a catalog object for this object
my ($a, $b, $c) = $objectCatalogObj->value(); ## Breakout the catalog triplet
#printf("Object is: %s - Catalog: %s %s %sn", $objectType, $a, $b, $c ); ## Output the catalog and object type for the object at hand.
if ( $objectType == EO_GROUP && $c == EXD_GROUP_PROC ) { ## If this is a PROC Group....
my @objectList = $obj->value(); ## Return the sub-group which will contain actual items.
my $xxPID = "";
my $xxCMD = "";
my $xxUID = "";
my $xxGID = "";
foreach my $item (@objectList) {
my $itemsType = $item->type(); ## Get type for new object.. is it a EO_ITEM or EO_GROUP?
if ($itemsType == EO_ITEM){
my $itemCatalog = $item->catalog(); ## Return the "Catalog Object" for this object.
my ($itemType, $itemGroup, $itemId) = $itemCatalog->value(); ## Breakout the catalog triple
my $itemValue = $item->value(); ## Get the item value itself.
#print("ttId: $itemId tValue: $itemValuen"); ## Now print just the catalog item id and item value.
$xxPID = $itemValue if ( $itemId == EXD_PROC_PID );
$xxUID = $itemValue if ( $itemId == EXD_PROC_UID );
$xxGID = $itemValue if ( $itemId == EXD_PROC_GID );
$xxCMD = $itemValue if ( $itemId == EXD_PROC_COMMAND );
} else {
print("ttERROR: Expected Item, got a group instead.n");
}
}
## Now output pretty formated data:
printf("%10d %4d %4d %5d %30sn", $idxCounter, $xxUID, $xxGID, $xxPID, $xxCMD);
}
}
# Report any errors
if (ea_error() != EXR_OK && ea_error() != EXR_EOF) {
printf("nERROR: %sn", ea_error_str());
exit(1);
}
Here is what the above looks like when run:
$ ./prettyproc.pl /var/adm/exacct/proc | head
Creator: SunOS
Hostname: quadra
Counter UID GID PID CMD
2 0 0 6878 acctadm
3 0 0 6879 acctadm
4 1004 999 6883 db2set
5 1004 999 6882 sh
6 1004 999 6881 db2fm
7 0 0 6880 sh
Please note that there should a lot of additional checking in the above, and you should differentiate between items that are full versus partial, but this should help you get started down the extended accounting road. 🙂