Centralizing Syslog on Solaris07 May '06 - 04:21 by benr
Syslog is a very simple, but all too often ignored tool. As sysadmins we constantly rely on it, but rarely customize and modify the setup, we just use it as is. But it need not be, with a couple minutes of your life you'll become a syslog expert and find it far more useful.
When a program sends a message to syslog, its given to the syslog daemon (syslogd) and then routed according to the /etc/syslog.conf configuration file. By default most of the useful output from syslog is placed into /var/adm/messages.
When messages are sent to syslog they are given a "facility" (whats sending the message) and a "level" (how important is it). If you look in /etc/syslog.conf you'll see a variety of facility.level pairs and then for each of these a destination for that type of message. Facility can be named (daemon, kern, ...) or can be a wildcard (*). Level must be named (alert, crit, notice, ..), but a given level will also report all levels above it, so if you use the "notice" level you'll also get crit's as well, for instance. Lets look at the available facilities and levels on Solaris 10:
To illustrate these, lets look at some messages in /var/adm/messages to jog your memory, notice the facility and level noted in each message:
May 6 00:28:53 aeon mac: [ID 543131 kern.info] NOTICE: nge0/0 registered May 6 00:28:55 aeon nge: [ID 742679 kern.notice] NOTICE: nge0: link 100Mbps Full-Duplex up May 5 12:48:47 aeon su: [ID 810491 auth.crit] 'su root' failed for benr on /dev/pts/4 May 5 02:41:35 aeon httpd: [ID 874347 user.error] libpkcs11: open /var/run/kcfd_door: No such file or directory
Of the facilities and levels, note that two are special, the wildcard (*) facility and the "none" level. Using the wildcard for the facility means what you'd think, all of them. The "none" severity level isn't useful on its own, but can be handy when creating compound statements for several facilities and levels. For instance, because a severity level will also pass anything more important that the one specified, saying "*.debug" is effectvely "pass along absolutely everything". If we wanted to pass along everything except for printer (lpr) messages we could use "*.debug;lpr.none".
So lets look at some sample lines from /etc/syslog.conf:
So this line passes along anything more sever than an error, any kernel notices or more sever, daemon notices or more sever, and critical or more sever mail messages, and routes these into the /var/adm/messages file. If we wanted to put all mail messages into a log file named "/var/adm/maillog" we could use something like "mail.debug /var/adm/maillog".
Putting log messages into a file is handy, but we can also send those messages across the network to a centralized syslog server. Simply use "@mysyslogserver" instead of a filename. So, "auth.notice @syslogcentral" will send all authorization notices (or more sever) to the syslog daemon running on a system called "syslogcentral".
Syslog on Solaris is setup by default to accept messages both locally and over the network. Reguardless of how a message comes into the syslogd daemon, its routed according to the syslog.conf configuration. So you could put "*.notice @syslogcentral" into the syslog.conf of each of your clients and "*.notice /var/adm/centralized_messages" in the syslog.conf of syslogcentral and wamo bamo, you'd have a centralized syslog infrastructure!
One thing to note when editing /etc/syslog.conf, you can't use spaces, you must use tabs. Also, after changing the configuration, make sure that you restart syslogd, on Solaris 10 Update 1 or newer you'd use svcadm restart svc:/system/system-log:default.
On a related note, since we're talking about syslog we might as well also talk about log rotation. If you look in /var/adm/messages chances are that you not only see messages, but you also see messages.0, messages.1 and so on. Log rotation on Solaris is done by /usr/sbin/logadm. If you look at the root crontab (/var/spool/cron/crontabs/root, or crontab -l as root) you'll see that its in there by default to run everyday at 3:10am. When logadm is executed with no args, it relies on /etc/logadm.conf which lists how to properly rotate each log. By default /var/adm/messages is rotated weekly assuming the file isn't empty and up to 8 copies (0-7) are kept. To modify this behaviour study the logadm man page and edit /etc/logadm.conf.
One thing to note however if you’re in a heterogenous environment is that even if all unices seems to log in equal fashion this is not nescessarly so. In particular I remember one time when my logserver never seemed to receieve any log from a True64 machine, after some debugging the reason for this turned out to be that the facility names didnt map in a 1 to 1 fashion (ie local0 == local1 on other machine or something similiar).
Gustaf - 07 May '06 - 12:46Syslog is an important tool and I agree it makes sense to take a long hard look at it. Personally however I’ve switched my machines over to using syslog-ng instead. You can do more fun stuff with it, as well as send the logs using TCP instead of UDP for increased reliability and having your logs centralized is a good idea because you can then run just one process on the log as it comes in to send alerts whenever something alert-worthy comes in.
Of course, it’s a bigger hassle to create a syslog-ng configuration, to say nothing of installing it on all your machines.
Kimmo (Email) - 08 May '06 - 03:39I’m a big fan of centralised syslog, and it even works well for Windows machines on your network.
In my book it has two major problems though: security and reliability. Most syslog servers accept anything sent to them on port 514, with no regard to the authenticity of the log message. Similarly, as syslog uses UDP, there’s always the risk that log messages can get lost.
There are solutions to both problems (including using IPSec), but none of them are widespread in their adoption. Perhaps you can cover them in a future blog Ben? If not, I guess I’ll have to :) Syslog-ng, logging over TCP and optionally adding Stunnel to your machines will give you reliable logging in an encrypted format. The Stunnel site even has an example for that specific scenario.
Kimmo (Email) - 16 May '06 - 10:10