Troubleshooting - Custom Systemd Service That Stops Immediately after Starting (CentOS/RHEL)

The Problem

Service started with custom systemd script stops straight after starting. For example:

# cat /etc/systemd/system/oracle.service
[Unit]
Description=Oracle Application Server Startup
After=network.target
[Service]
User=oracle
Group=oinstall
Type=simple
WorkingDirectory=/oracle/bin
ExecStart=/oracle/bin/apsinit_start.sh
ExecStop=/oracle/bin/apsinit_stop.sh
Restart=on-failure
RestartSec=1

[Install]
WantedBy=multi-user.target

Corresponding logs:

# tailf /var/log/messages
...
server1 systemd[1]: Startingd Oracle Application Server Startup.
server1 systemd[1]: Starting Oracle Application Server Startup...
server1 apsinit_start.sh[41176]: Starting RDBMS: REPOS
server1 apsinit_start.sh[41176]: Starting Domain: RMIBSDomain
server1 apsinit_start.sh[41176]: Starting Component: ohs1 (RMIBSDomain)
server1 apsinit_start.sh[41176]: Starting Component: RepServerMibsNeu2 (RMIBSDomain)
server1 apsinit_stop.sh[43727]: Stopping Component: ohs1 (RMIBSDomain)
server1 apsinit_stop.sh[43727]: Stopping Component: RepServerMibsNeu2 (RMIBSDomain)
server1 apsinit_stop.sh[43727]: Stopping Domain: RMIBSDomain
server1 apsinit_stop.sh[43727]: Stopping RDBMS: REPOS

The Solution

This kind of problem is usually due to using “Type=simple” (which is the default value for Type) in the systemd service script. In this case the first process to start (/oracle/bin/apsinit_start.sh in this example) is considered the main service process. When this process exits (presumably after having started the application server process(es)), the whole service is considered to have stopped. systemd service files have multiple Types and ‘simple’ assumes a single, non-forking process. From the systemd documentation:

Example 3. Stoppable oneshot service Similarly to the oneshot services, there are sometimes units that need to execute a program to set up something and then execute another to shut it down, but no process remains active while they are considered “started”. Network configuration can sometimes fall into this category. Another use case is if a oneshot service shall not be executed each time when they are pulled in as a dependency, but only the first time. For this, systemd knows the setting RemainAfterExit=yes, which causes systemd to consider the unit to be active if the start action exited successfully. This directive can be used with all types, but is most useful with Type=oneshot and Type=simple. With Type=oneshot, systemd waits until the start action has completed before it considers the unit to be active, so dependencies start only after the start action has succeeded. With Type=simple, dependencies will start immediately after the start action has been dispatched. The following unit provides an example for a simple static firewall.

Since the unit is considered to be running after the start action has exited, invoking systemctl start on that unit again will cause no action to be taken.

The solution here is to change “Type=simple” to “Type=oneshot” and add a line “RemainAfterExit=yes” to the [Service] section in the custom systemd script.

Then ensure that systemd picks up the changes:

# systemctl daemon-reload