How to configure the system journal to preserve the record of events when a server is rebooted (CentOS/RHEL)

Storing the system journal permanently

By default, the system journals are kept in the /run/log/journal directory, which means the journals are cleared when the system reboots. You can change the configuration settings of the systemd-journald service in the /etc/systemd/journald.conf file to make the journals persist across reboot.

The Storage parameter in the /etc/systemd/journald.conf file defines whether to store system journals in a volatile manner or persistently across reboot. Set this parameter to persistent, volatile, or auto as follows:

• persistent: stores journals in the /var/log/journal directory which persists across reboots. If the /var/log/journal directory does not exist, the systemd-journald service creates it.
• volatile: stores journals in the volatile /run/log/journal directory. As the /run file system is temporary and exists only in the runtime memory, data stored in it, including system journals, do not persist across reboot.
• auto: rsyslog determines whether to use persistent or volatile storage. If the /var/log/ journal directory exists, then rsyslog uses persistent storage, otherwise it uses volatile storage.

This is the default action if the Storage parameter is not set. The advantage of persistent system journals is that the historic data is available immediately at boot. However, even with a persistent journal, not all data is kept forever. The journal has a built-in log rotation mechanism that triggers monthly. In addition, by default, the journals are not allowed to get larger than 10% of the file system it is on, or leave less than 15% of the file system free. These values can be tuned for both the runtime and persistent journals in /etc/systemd/ journald.conf. The current limits on the size of the journal are logged when the systemdjournald process starts. The following command output shows the journal entries that reflect the current size limits:

[user@host ~]$ journalctl | grep -E 'Runtime|System journal'
Feb 25 13:01:46 localhost systemd-journald[147]: Runtime journal (/run/log/ journal/ae06db7da89142138408d77efea9229c) is 8.0M, max 91.4M, 83.4M free.
Feb 25 13:01:48 remotehost.lab.example.com systemd-journald[548]: Runtime journal (/run/log/journal/73ab164e278e48be9bf80e80714a8cd5) is 8.0M, max 91.4M, 83.4M free.
Feb 25 13:01:48 remotehost.lab.example.com systemd-journald[548]: System journal (/var/log/journal/73ab164e278e48be9bf80e80714a8cd5) is 8.0M, max 3.7G, 3.7G free.
Feb 25 13:01:48 remotehost.lab.example.com systemd[1]: Starting Tell Plymouth To
Write Out Runtime Data...
Feb 25 13:01:48 remotehost.lab.example.com systemd[1]: Started Tell Plymouth To Write Out Runtime Data.

Configuring Persistent System Journals

To configure the systemd-journald service to preserve system journals persistently across reboot, set Storage to persistent in the /etc/systemd/journald.conf file. Run the text editor of your choice as the superuser to edit the /etc/systemd/journald.conf file.

[Journal]
Storage=persistent
...output omitted...

After editing the configuration file, restart the systemd-journald service to bring the configuration changes into effect.

[root@host ~]# systemctl restart systemd-journald

If the systemd-journald service successfully restarts, you can see that the /var/log/journal directory is created and contains one or more subdirectories. These subdirectories have hexadecimal characters in their long names and contain *.journal files. The *.journal files are the binary files that store the structured and indexed journal entries.

[root@host ~]# ls /var/log/journal
73ab164e278e48be9bf80e80714a8cd5

[root@host ~]# ls /var/log/journal/73ab164e278e48be9bf80e80714a8cd5
system.journal  user-1000.journal

While the system journals persist across a reboot, you get an extensive number of entries in the output of the journalctl command that includes entries from the current system boot as well as the previous ones. To limit the output to a specific system boot, use the -b option with the journalctl command retrieves the entries limited to the first system boot:

[root@host ~]# journalctl -b 1
...output omitted...

The following journalctl command retrieves the entries limited to the second system boot. The following argument is meaningful only if the system has been rebooted for more than twice:

[root@host ~]# journalctl -b 2

The following journalctl command retrieves the entries limited to the current system boot:

[root@host ~]# journalctl -b