We have two servers, svc-notes and svc-userauth, configured so we can run the two services making up the Notes application stack. A big task remaining is to ensure the Node.js processes are properly installed as background processes.
To see the problem, start another command window and run these commands:
$ multipass restart svc-userauth
$ multipass restart svc-notes
The server instances were running under Multipass, and the restart command caused the named instance to stop and then start. This emulates a server reboot. Since both were running in the foreground, you’ll see each command window exit to the host command shell, and running multipass list again will show both instances in the Running state. The big takeaway is that both services are no longer running.
There are many ways to manage server processes, to ensure restarts if the process crashes, and so on. We’ll use PM2 (http://pm2.keymetrics.io/) because it’s optimized for Node.js processes. It bundles process management and monitoring into one application.
Let’s now see how to use PM2 to correctly manage the Notes and user authentication services as background processes. We’ll start by familiarizing ourselves with PM2, then creating scripts to use PM2 to manage the services, and finally, we’ll see how to integrate it with the OS so that the services are correctly managed as background processes.
1. Familiarizing ourselves with PM2
To get ourselves acquainted with PM2, let’s set up a test using the svc-userauth server. We will create a directory to hold a pm2-userauth project, install PM2 in that directory, then use it to start the user authentication service. Along the way, we’ll learn how to use PM2.
Start by running these commands on the svc-userauth server:
ubuntu@svc-userauth:~$ sudo su –
root@svc-userauth:~#
root@svc-userauth:~# cd /opt
root@svc-userauth:/opt# mkdir pm2-test
root@svc-userauth:/opt# cd pm2-test
root@svc-userauth:/opt/pm2-test# npm init
This utility will walk you through creating a package.json file.
… answer questions
root@svc-userauth:/opt/pm2-test# npm install pm2 –save
… installation output
root@svc-userauth:/opt/pm2-test# node_modules/.bin/pm2
… help output
The result of these commands is an npm project directory containing the PM2 program and a package.json file that we can potentially use to record some scripts.
Now let’s start the user authentication server using PM2:
root@svc-userauth:/opt/pm2-test# cd ../userauth/
root@svc-userauth:/opt/userauth# DEBUG=users:* PORT=5858
REST_LISTEN=0.0.0.0 SEQUELIZE_CONNECT=sequelize-mysql.yaml /opt/pm2-
test/node_modules/.bin/pm2 start ./user-server.mjs
…
[PM2] Spawning PM2 daemon with pm2_home=/root/.pm2
[PM2] PM2 Successfully daemonized
[PM2] Starting /opt/userauth/user-server.mjs in fork_mode (1 instance)
[PM2] Done.
…
This boils down to running pm2 start ./user-server.mjs, except that we are adding the environment variables containing configuration values, and we are specifying the full path to PM2. This runs our user server in the background.
We can repeat our test of using cli.mjs to list users known to the authentication server:
root@svc-userauth:/opt/userauth# node cli.mjs list-users
[
{ … }, { … }
]
Since we had previously launched this service and tested it, there should be user IDs already in the authentication server database. The server is running, but because it’s not in the foreground, we cannot see the output. Try this command:
root@svc-userauth:/opt/userauth# /opt/pm2-test/node_modules/.bin/pm2 logs user-server
… log output
Because PM2 captures the standard output from the server process, any output is saved away. The logs command lets us view that output.
Some other useful commands are as follows:
- pm2 status: Lists all the commands PM2 is currently managing, and their status
- pm2 stop SERVICE: Stops the named service
- pm2 start SERVICE or pm2 restart SERVICE: Starts the named service
- pm2 delete SERVICE: Makes PM2 forget about the named service
There are several other commands, and the PM2 website contains complete documentation for them. https://pm2.keymetrics.io/docs/usage/pm2-doc-single- page/
For the moment, let’s shut it down and delete the managed process:
root@svc-userauth:/opt/userauth# /opt/pm2-test/node_modules/.bin/pm2 stop user-server
root@svc-userauth:/opt/userauth# /opt/pm2-test/node_modules/.bin/pm2 delete user-server
root@svc-userauth:/opt/userauth# rm -rf /opt/pm2-test
We have familiarized ourselves with PM2, but this setup is not quite suitable for any kind of deployment. Let’s instead set up scripts that will manage the Notes services under PM2 more cleanly.
2. Scripting the PM2 setup on Multipass
We have two Ubuntu systems onto which we’ve copied the Notes and user authentication services, and also configured a MySQL server for each machine. On these systems, we’ve manually run the services and know that they work, and now it’s time to use PM2 to manage these services as persistent background processes.
With PM2 we can create a file, ecosystem.json, to describe precisely how to launch the processes. Then, with a pair of PM2 commands, we can integrate the process setup so it automatically starts as a background process.
Let’s start by creating two directories, multipass/pm2-notes and multipass/pm2- userauth. These will hold the scripts for the corresponding servers.
In pm2-notes, create a file, package.json, containing the following:
{
“name”: “pm2-notes”,
“version”: “1.0.0”,
“description”: “PM2 Configuration for svc-notes”, “scripts”: {
“start”: “pm2 start ecosystem.json”,
“stop”: “pm2 stop ecosystem.json”,
“restart”: “pm2 restart ecosystem.json”,
“status”: “pm2 status”,
“save”: “pm2 save”,
“startup”: “pm2 startup”,
“monit”: “pm2 monit”,
“logs”: “pm2 logs”
},
“dependencies”: {
“pm2”: “^4.4.x”
}
}
This records for us the dependency on PM2, so it can easily be installed, along with some useful scripts we can run with PM2.
Then in the same directory, create an ecosystem.json file, containing the following:
{
“apps” : [ {
“name”: “Notes”,
“script”: “app.mjs”,
“cwd”: “/opt/notes”, “env”: {
“REQUEST_LOG_FORMAT”: “common”, “PORT”: “80”,
“SEQUELIZE_CONNECT”: “models/sequelize-mysql.yaml”,
“NOTES_MODEL”: “sequelize”,
“USER_SERVICE_URL”: “http://172.23.83.119:5858”,
“TWITTER_CALLBACK_HOST”: “http://172.23.89.142”
},
“env_production”: {
“NODE_ENV”: “production”
}
} ]
}
The ecosystem.json file is how we describe a process to be monitored to PM2.
In this case, we’ve described a single process, called Notes. The cwd value declares where the code for this process lives, and the script value describes which script to run to launch the service. The env value is a list of environment variables to set.
This is where we would specify the Twitter authentication tokens. But since this file is likely to be committed to a source repository, we shouldn’t do so. Instead, we’ll forego Twitter login functionality for the time being.
The USER_SERVICE_URL and TWITTER_CALLBACK_HOST variables are set according to the multipass list output we showed earlier. These values will, of course, vary based on what was selected by your host system.
These environment variables are the same as we set in notes/package.json – except, notice that we’ve set PORT to 80 so that it runs on the normal HTTP port. To successfully specify port 80, PM2 must execute as root.
In pm2-userauth, create a file named package.json containing the folllowing:
{
“name”: “pm2-userauth”,
“version”: “1.0.0”,
“description”: “PM2 Configuration for svc-userauth”, “scripts”: {
“start”: “pm2 start ecosystem.json”,
“stop”: “pm2 stop ecosystem.json”,
“restart”: “pm2 restart ecosystem.json”,
“status”: “pm2 status”,
“save”: “pm2 save”,
“startup”: “pm2 startup”,
“monit”: “pm2 monit”,
“logs”: “pm2 logs”
},
“dependencies”: {
“pm2”: “^4.4.x”
}
}
This is the same as for pm2-notes, with the names changed.
Then, in pm2-userauth, create a file named ecosystem.json containing the following:
{
“apps” : [{
“name”: “User Authentication”, “script”: “user-server.mjs”,
“cwd”: “/opt/userauth”, “env”: {
“REQUEST_LOG_FORMAT”: “common”,
“PORT”: “5858”,
“REST_LISTEN”: “0.0.0.0”,
“SEQUELIZE_CONNECT”: “sequelize-mysql.yaml”
},
“env_production”: {
“NODE_ENV”: “production”
}
}]
}
This describes the user authentication service. On the server, it is stored in the /userauth directory and is launched using the user-server.mjs script, with that set of environment variables.
Next, on both servers create a directory called /opt/pm2. Copy the files in pm2- notes to the /opt/pm2 directory on svc-notes, and copy the files in pm2- userauth to the /opt/pm2 directory on svc-userauth.
On both svc-notes and svc-userauth, you can run these commands:
ubuntu@svc-userauth:/opt/pm2$ sudo npm install
… much output
ubuntu@svc-userauth:/opt/pm2$ sudo npm start
> pm2-userauth@1.0.0 start /pm2
> pm2 start json
[PM2] Spawning PM2 daemon with pm2_home=/home/ubuntu/.pm2
[PM2] PM2 Successfully daemonized
[PM2][WARN] Applications User Authentication not running, starting…
[PM2] App [User Authentication] launched (1 instances)
ubuntu@svc-userauth:/opt/pm2$ sudo npm run status
> pm2-userauth@1.0.0 status /opt/pm2
> pm2 status
…
ubuntu@svc-userauth:/opt/pm2$ sudo npm run logs
> pm2-userauth@1.0.0 status /opt/pm2
> pm2 logs
…
Doing so starts the service running on both server instances. The npm run logs command lets us see the log output as it happens. We’ve configured both
services in a more DevOps-friendly logging configuration, without the DEBUG log enabled, and using the common log format.
For testing, we go to the same URL as before, but to port 80 rather than port 3000.
Because the Notes service on svc-notes is now running on port 80, we need to update the Twitter application configuration again, as follows:
This drops the port 3000 from the URLs on the server. The application is no longer on port 3000, but on port 80, and we need to tell Twitter about this change.
3. Integrating the PM2 setup as persistent background processes
The Notes application should be fully functioning. There is one remaining small task to perform, and that is to integrate it with the OS.
The traditional way on Unix-like systems is to add a shell script in a directory in /etc. The Linux community has defined the LSB Init Script format for this
purpose, but since each OS has a different standard for scripts to manage background processes, PM2 has a command to generate the correct script for each.
Let’s start with svc-userauth, and run these commands:
ubuntu@svc-userauth:/opt/pm2$ sudo npm run save
> pm2-userauth@1.0.0 save /opt/pm2
> pm2 save
[PM2] Saving current process list…
[PM2] Successfully saved in /home/ubuntu/.pm2/dump.pm2 ubuntu@svc-userauth:/opt/pm2$ sudo npm run startup
> pm2-userauth@1.0.0 startup /opt/pm2
> pm2 startup
[PM2] Init System found: systemd Platform systemd
… much output Target path
/etc/systemd/system/pm2-root.service Command list
[ ‘systemctl enable pm2-root’ ]
… much output
With npm run save, we run the pm2 save command. This command saves the current configuration into a file in your home directory.
With npm run startup, we run the pm2 startup command. This converts the saved current configuration into a script for the current OS that will manage the PM2 system. PM2, in turn, manages the set of processes you’ve configured with PM2.
In this case, it identified the presence of the systemd init system, which is the standard for Ubuntu. It generated a file, /etc/systemd/system/pm2- root.service, that tells Ubuntu about PM2. In amongst the output, it tells us how to use systemctl to start and stop the PM2 service.
Do the same on svc-notes to implement the background service there as well. And now we can test restarting the two servers with the following commands:
$ multipass restart svc-userauth
$ multipass restart svc-notes
$ multipass list
Name State IPv4 Image
svc-notes Running 192.168.64.9 Ubuntu 18.04 LTS
svc-userauth Running 192.168.64.8 Ubuntu 18.04 LTS
The machines should restart correctly, and with no intervention on our part, the services will be running. You should be able to put the Notes application through its paces and see that it works. The Twitter login functionality will not work at this time because we did not supply Twitter tokens.
It is especially informative to run this on each server:
ubuntu@svc-notes:~$ cd /opt/pm2
ubuntu@svc-notes:/opt/pm2$ sudo ./node_modules/.bin/pm2 monit
The monit command starts a monitoring console showing some statistics including CPU and memory use, as well as logging output.
When done, run the following command:
$ multipass stop svc-userauth
$ multipass stop svc-notes
This, of course, shuts down the service instances. Because of the work we’ve done, you’ll be able to start them back up at any time.
We’ve learned a lot in this section about configuring the Notes application as a managed background process. With a collection of shell scripts and configuration files, we put together a system to manage these services as background processes using PM2. By writing our own scripts, we got a clearer idea of how the underlying plumbing works.
With that, we are ready to wrap up the chapter.
Source: Herron David (2020), Node.js Web Development: Server-side web development made easy with Node 14 using practical examples, Packt Publishing.