Overview of Syslog Priorities, rsyslog rules, log rotation and basic troubleshooting using syslog

Logging events to the system

Many programs use the syslog protocol to log events to the system. Each log message is categorized by a facility (the type of message) and a priority (the severity of the message). Available facilities are documented in the rsyslog.conf(5) man page.

The following table lists the standard eight syslog priorities from highest to lowest.

CODE	PRIORITY	SEVERITY
0	emerg	System is unusable
1	alert	Action must be taken immediately
2	crit	Critical condition
3	err	Non-critical error condition
4	warning	Warning condition
5	notice	Normal but significant event
6	info	Informational event
7	debug	Debugging-level message

The rsyslog service uses the facility and priority of log messages to determine how to handle them. This is configured by rules in the /etc/rsyslog.conf file and any file in the /etc/rsyslog.d directory that has a file name extension of .conf. Software packages can easily add rules by installing an appropriate file in the /etc/rsyslog.d directory.

Each rule that controls how to sort syslog messages is a line in one of the configuration files. The left side of each line indicates the facility and severity of the syslog messages the rule matches. The right side of each line indicates what file to save the log message in (or where else to deliver the message). An asterisk (*) is a wildcard that matches all values.

For example, the following line would record messages sent to the authpriv facility at any priority to the file /var/log/secure:

authpriv.*                  /var/log/secure

Log messages sometimes match more than one rule in rsyslog.conf. In such cases, one message is stored in more than one log file. To limit messages stored, the keyword none in the priority field indicates that no messages for the indicated facility should be stored in the given file. Instead of logging syslog messages to a file, they can also be printed to the terminals of all logged-in users. The rsyslog.conf file has a setting to print all the syslog messages with the emerg priority to the terminals of all logged-in users.

Sample Rules of rsyslog

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

Log File Rotation

The logrotate tool rotates log files to keep them from taking up too much space in the file system containing the /var/log directory. When a log file is rotated, it is renamed with an extension indicating the date it was rotated. For example, the old /var/log/messages file may become /var/log/messages-20190130 if it is rotated on 2019-01-30. Once the old log file is rotated, a new log file is created and the service that writes to it is notified.

After a certain number of rotations, typically after four weeks, the oldest log file is discarded to free disk space. A scheduled job runs the logrotate program daily to see if any logs need to be rotated. Most log files are rotated weekly, but logrotate rotates some faster, or slower, or when they reach a certain size. For more information, see the logrotate(8) man page.

Analyzing a syslog entry

Log messages start with the oldest message on top and the newest message at the end of the log file. The rsyslog service uses a standard format while recording entries in log files. The following example explains the anatomy of a log message in the /var/log/secure log file.

1. The time stamp when the log entry was recorded
2. The host from which the log message was sent
3. The program or process name and PID number that sent the log message
4. The actual message sent

Monitoring Logs

Monitoring one or more log files for events is helpful to reproduce problems and issues. The tail -f /path/to/file command outputs the last 10 lines of the file specified and continues to output new lines in the file as they get written.

For example, to monitor for failed login attempts, run the tail command in one terminal and then in another terminal, run the ssh command as the root user while a user tries to log in to the system. In the first terminal, run the following tail command:

[root@host ~]# tail -f /var/log/secure

In the second terminal, run the following ssh command:

[root@host ~]# ssh root@localhost
root@localhost's password: mypass
...output omitted...
[root@host ~]#

Return to the first terminal and view the logs.

...output omitted...
Feb 10 09:01:13 host sshd[2712]: Accepted password for root from 172.25.254.254 port 56801 ssh2
Feb 10 09:01:13 host sshd[2712]: pam_unix(sshd:session): session opened for user root by (uid=0)

Sending syslog messages manually

The logger command can send messages to the rsyslog service. By default, it sends the message to the user facility with the notice priority (user.notice) unless specified otherwise with the -p option. It is useful to test any change to the rsyslog service configuration.

To send a message to the rsyslog service that gets recorded in the /var/log/boot.log log file, execute the following logger command:

[root@host ~]# logger -p local7.notice "Log entry created on host"