In the previous sections, we learned how to set up an AWS infrastructure using Terraform. We’ve designed a VPC that will house the Notes application stack, we experimented with a single-node Docker Swarm cluster built on a single EC2 instance, and we set up a procedure to push the Docker images to the ECR.
Our next task is to prepare a Docker stack file for deployment to the swarm. A stack file is nearly identical to the Docker compose file we used in Chapter 11, Deploying Node.js Microservices with Docker. Compose files are used with normal Docker hosts, but stack files are used with swarms. To make it a stack file, we add some new tags and change a few things, including the networking implementation.
Earlier, we kicked the tires of Docker Swarm with the docker service create command to launch a service on a swarm. While that was easy, it does not constitute code that can be committed to a source repository, nor is it an automated process.
In swarm mode, a service is a definition of the tasks to execute on swarm nodes. Each service consists of a number of tasks, with this number depending on the replica settings. Each task is a container that has been deployed to a node in the swarm.
There are, of course, other configuration parameters, such as network ports, volume connections, and environment variables.
The Docker platform allows the use of the compose file for deploying services to a swarm. When used this way, the compose file is referred to as a stack file. There is a set of docker stack commands for handling the stack file, as follows:
- On a regular Docker host, the docker-compose.yml file is called a compose file. We use the docker-compose command on a compose file.
- On a Docker swarm, the docker-compose.yml file is called a stack file. We use the docker stack command on a stack file.
Remember that a compose file has a services tag, and each entry in that tag is a container configuration to deploy. When used as a stack file, each services tag entry is, of course, a service in the sense just described. This means that just as there was a lot of similarity between the docker run command and container definitions in the compose file, there is a degree of similarity between the docker service create command and the service entries in the stack file.
One important consideration is a policy that builds must not happen on Swarm host machines. Instead, these machines must be used solely for deploying and executing containers. This means that any build tag in a service listed in a stack file is ignored. Instead, there is a deploy tag that has parameters for the deployment in the swarm, and the deploy tag is ignored when the file is used with Compose. Put more simply, we can have the same file serve both as a compose file (with the docker compose command) and as a stack file (with the docker stack command), with the following conditions:
- When used as a compose file, the build tag is used and the deploy tag is ignored.
- When used as a stack file, the build tag is ignored and the deploy tag is used.
Another consequence of this policy is the necessity of switching the Docker context as appropriate. We have already discussed this issue—that we use the default Docker context to build images on our laptop and we use the EC2 context when interacting with the swarm on the AWS EC2 instances.
To get started, create a directory named compose-stack that’s a sibling to compose- local, notes, terraform-swarm, and the other directories. Then, copy compose- local/docker-compose.yml into compose-stack. This way, we can start from something we know is working well.
This means that we’ll create a Docker stack file from our compose file. There are several steps involved, which we’ll cover over the next several sections. This includes adding deploy tags, configuring networking for the swarm, controlling the placement of services in the swarm, storing secrets in the swarm, and other tasks.
1. Creating a Docker stack file from the Notes Docker compose file
With that theory under our belts, let’s now take a look at the existing Docker compose file and see how to make it useful for deployment to a swarm.
Since we will require some advanced docker-compose.yml features, update the version number to the following:
version: ‘3.8’
For the Compose file we started with, version ‘3’ was adequate, but to accomplish the tasks in this chapter the higher version number is required, to enable newer features.
Fortunately, most of this is straightforward and will require very little code.
For the deployment parameters, simply add a deploy tag to each service. Most of the options for this tag have perfectly reasonable defaults. To start with, let’s add this to every service, as follows:
deploy:
replicas: 1
This tells Docker that we want one instance of each service. Later, we will experiment with adding more service instances. We will add other parameters later, such as placement constraints. Later, we will want to experiment with multiple replicas for both svc-notes and svc-userauth. It is tempting to put CPU and memory limits on the service, but this isn’t necessary.
It is nice to learn that with swarm mode, we can simply change the replicas setting to change the number of instances.
The next thing to take care of is the image name. While the build tag is present, remember that it is ignored. For the Redis and database containers, we are already using images from Docker Hub, but for svc-notes and svc-userauth, we are building our own containers. This is why, earlier in this chapter, we set up a procedure for pushing the images to ECR repositories. We can now reference those images from the stack file. This means that we must make the following change:
services:
…
svc-userauth:
build: ../users
image: 098E0X9AMPLE.dkr.ecr.us-REGION-2.amazonaws.com/svc-userauth
…
svc-notes:
build: ../notes
image: 098E0X9AMPLE.dkr.ecr.us-REGION-2.amazonaws.com/svc-notes
…
If we use this with docker-compose, it will perform the build in the named directories, and then tag the resulting image with the tag in the image field. In this case, the deploy tag will be ignored as well. However, if we use this with docker stack deploy, the build tag will be ignored, and the images will be downloaded from the repositories listed in the image tag. In this case, the deploy tag will be used.
When running the compose file on our laptop, we used bridge networking. This works fine for a single host, but with swarm mode, we need another network mode that handles multi-host deployments. The Docker documentation clearly says to use the overlay driver in swarm mode, and the bridge driver for a single-host deployment.
To use overlay networking, change the networks tag to the following:
networks: frontnet:
# driver: bridge
driver: overlay
authnet:
# driver: bridge
driver: overlay
svcnet:
# driver: bridge
driver: overlay
To support switching between using this for a swarm, or for a single-host deployment, we can leave the bridge network setting available but commented out. We would then change whether overlay or bridge networking is active by changing which is commented, depending on the context.
The overlay network driver sets up a virtual network across the swarm nodes. This network supports communication between the containers and also facilitates access to the externally published ports.
The overlay network configures the containers in a swarm to have a domain name automatically assigned that matches the service name. As with the bridge network we used before, containers find each other via the domain name. For a service deployed with multiple instances, the overlay network ensures that requests to that container can be routed to any of its instances. If a connection is made to a container but there is no instance of that container on the same host, the overlay network routes the request to an instance on another host. This is a simple approach to service discovery, by using domain names, but extending it across multiple hosts in a swarm.
That took care of the easy tasks for converting the compose file to a stack file. There are a few other tasks that will require more attention, however.
1.1. Placing containers across the swarm
We haven’t done it yet, but we will add multiple EC2 instances to the swarm. By default, swarm mode distributes tasks (containers) evenly across the swarm nodes. However, we have two considerations that should force some containers to be deployed on specific Docker hosts—namely, the following:
- We have two database containers and need to arrange persistent storage for the data files. This means that the databases must be deployed to the same instance every time so that it can use the same data directory.
- The public EC2 instance, named notes-public, will be part of the To maintain the security model, most of the services should not be deployed on this instance but on the instances that will be attached to the private subnet. Therefore, we should strictly control which containers deploy to notes-public.
Swarm mode lets us declare the placement requirements for any service. There are several ways to implement this, such as matching against the hostname, or against labels that can be assigned to each node.
Add this deploy tag to the db-userauth service declaration:
services:
…
db-userauth:
..
deploy:
replicas: 1
placement:
constraints:
# – “node.hostname==notes-private-db1”
– “node.labels.type==db”
…
The placement tag governs where the containers are deployed. Rather than Docker evenly distributing the containers, we can influence the placement with the fields in this tag. In this case, we have two examples, such as deploying a container to a specific node based on the hostname or selecting a node based on the labels attached to the node.
To set a label on a Docker swarm node, we run the following command:
$ docker node update –label-add type=public notes-public
This command attaches a label named type, with the value public, to the node named notes-public. We use this to set labels, and, as you can see, the label can have any name and any value. The labels can then be used, along with other attributes, as influence over the placement of containers on swarm nodes.
For the rest of the stack file, add the following placement constraints:
services:
…
svc-userauth:
…
deploy:
replicas: 1
placement:
constraints:
– “node.labels.type==svc”
…
db-notes:
…
deploy:
replicas: 1
placement:
constraints:
– “node.labels.type==db”
…
svc-notes:
…
deploy:
replicas: 1
placement:
constraints:
– “node.labels.type==public”
…
redis:
…
deploy:
replicas: 1
placement:
constraints:
– “node.labels.type!=public”
…
This gives us three labels to assign to our EC2 instances: db, svc, and public.
These constraints will cause the databases to be placed on nodes where the type label is db, the user authentication service is on the node of type svc, the Notes service is on the public node, and the Redis service is on any node that is not the public node.
The reasoning stems from the security model we designed. The containers deployed on the private network should be more secure behind more layers of protection. This placement leaves the Notes container as the only one on the public EC2 instance. The other containers are split between the db and svc nodes. We’ll see later how these labels will be assigned to the EC2 instances we’ll create.
1.2. Configuring secrets in Docker Swarm
With Notes, as is true for many kinds of applications, there are some secrets we must protect. Primarily, this is the Twitter authentication tokens, and we’ve claimed it could be a company-ending event if those tokens were to leak to the public. Maybe that’s overstating the danger, but leaked credentials could be bad. Therefore, we must take measures to ensure that those secrets do not get committed to a source repository as part of any source code, nor should they be recorded in any other file.
For example, the Terraform state file records all information about the infrastructure, and the Terraform team makes no effort to detect any secrets and suppress recording them. It’s up to us to make sure the Terraform state file does not get committed to source code control as a result.
Docker Swarm supports a very interesting method for securely storing secrets and for making them available in a secure manner in containers.
The process starts with the following command:
$ printf ‘vuTghgEXAMPLE…’ | docker secret create
TWITTER_CONSUMER_KEY –
$ printf ‘tOtJqaEXAMPLE…’ | docker secret create
TWITTER_CONSUMER_SECRET –
This is how we store a secret in a Docker swarm. The docker secret create command first takes the name of the secret, and then a specifier for a file containing the text for the secret. This means we can either store the data for the secret in a file or—as in this case—we use – to specify that the data comes from the standard input. In this case, we are using the printf command, which is available for macOS and Linux, to send the value into the standard input.
Docker Swarm securely records the secrets as encrypted data. Once you’ve given a secret to Docker, you cannot inspect the value of that secret.
In compose-stack/docker-compose.yml, add this declaration at the end:
secrets:
TWITTER_CONSUMER_KEY:
external: true
TWITTER_CONSUMER_SECRET:
external: true
This lets Docker know that this stack requires the value of those two secrets. The declaration for svc-notes also needs the following command:
services:
…
svc-notes:
…
secrets:
– TWITTER_CONSUMER_KEY
– TWITTER_CONSUMER_SECRET
..
environment:
…
TWITTER_CONSUMER_KEY_FILE: /var/run/secrets/TWITTER_CONSUMER_KEY
TWITTER_CONSUMER_SECRET_FILE:
/var/run/secrets/TWITTER_CONSUMER_SECRET
…
This notifies the swarm that the Notes service requires the two secrets. In response, the swarm will make the data for the secrets available in the filesystem of the container as /var/run/secrets/TWITTER_CONSUMER_KEY and /var/run/secrets/TWITTE R_CONSUMER_SECRET. They are stored as in-memory files and are relatively secure.
To summarize, the steps required are as follows:
- Use docker secret create to register the secret data with the swarm.
- In the stack file, declare secrets in a top-level secrets tag.
- In services that require the secrets, declare a secrets tag that lists the secrets required by this service.
- In the environments tag for the service, create an environment variable pointing to the secrets file.
The Docker team has a suggested convention for configuration of environment variables. You could supply the configuration setting directly in an environment variable, such as TWITTER_CONSUMER_KEY. However, if the configuration setting is in a file, then the filename should be given in a different environment variable whose name has _FILE appended. For example, we would use TWITTER_CONSUMER_KEY or TWITTER_CONSUMER_KEY_FILE, depending on whether the value is directly supplied or in a file.
This then means that we must rewrite Notes to support reading these values from the files, in addition to the existing environment variables.
To support reading from files, add this import to the top of notes/routes/users.mjs:
import fs from ‘fs-extra’;
Then, we’ll find the code corresponding to these environment variables further down the file. We should rewrite that section as follows:
const twittercallback = process.env.TWITTER_CALLBACK_HOST
? process.env.TWITTER_CALLBACK_HOST
: “http://localhost:3000”;
export var twitterLogin = false;
let consumer_key;
let consumer_secret;
if (typeof process.env.TWITTER_CONSUMER_KEY !== ‘undefined’
&& process.env.TWITTER_CONSUMER_KEY !== ”
&& typeof process.env.TWITTER_CONSUMER_SECRET !== ‘undefined’
&& process.env.TWITTER_CONSUMER_SECRET !== ”) {
consumer_key = process.env.TWITTER_CONSUMER_KEY;
consumer_secret = process.env.TWITTER_CONSUMER_SECRET;
twitterLogin = true;
} else if (typeof process.env.TWITTER_CONSUMER_KEY_FILE !==
‘undefined’
&& process.env.TWITTER_CONSUMER_KEY_FILE !== ”
&& typeof process.env.TWITTER_CONSUMER_SECRET_FILE !== ‘undefined’
&& process.env.TWITTER_CONSUMER_SECRET_FILE !== ”) {
consumer_key = fs.readFileSync(process.env.TWITTER_CONSUMER_KEY_FILE, ‘utf8’);
consumer_secret = fs.readFileSync(process.env.TWITTER_CONSUMER_SECRET_FILE, ‘utf8’);
twitterLogin = true;
}
if (twitterLogin) {
passport.use(new TwitterStrategy({
consumerKey: consumer_key, consumerSecret: consumer_secret,
callbackURL: ‘${twittercallback}/users/auth/twitter/callback’
},
async function(token, tokenSecret, profile, done) {
try {
done(null, await usersModel.findOrCreate({
id: profile.username, username: profile.username, password: “”,
provider: profile.provider, familyName: profile.displayName,
givenName: “”, middleName: “”,
photos: profile.photos, emails: profile.emails
}));
} catch(err) { done(err); }
}));
}
This is similar to the code we’ve already used but organized a little differently. It first tries to read the Twitter tokens from the environment. Failing that, it tries to read them from the named files. Because this code is executing in the global context, we must read the files using readFileSync.
If the tokens are available from either source, the twitterLogin variable is set, and then we enable the support for TwitterStrategy. Otherwise, Twitter support is disabled. We had already organized the views templates so that if twitterLogin is false, the Twitter login buttons do not appear.
All of this is what we did in Chapter 8, Authenticating Users with a Microservice, but with the addition of reading the tokens from a file.
1.3. Persisting data in a Docker swarm
The data persistence strategy we used in Chapter 11, Deploying Node.js Microservices with Docker, required the database files to be stored in a volume. The directory for the volume lives outside the container and survives when we destroy and recreate the container.
That strategy relied on there being a single Docker host for running containers. The volume data is stored in a directory in the host filesystem. But in swarm mode, volumes do not work in a compatible fashion.
With Docker Swarm, unless we use placement criteria, containers can deploy to any swarm node. The default behavior for a named volume in Docker is that the data is stored on the current Docker host. If the container is redeployed, then the volume is destroyed on the one host and a new one is created on the new host. Clearly, that means that the data in that volume is not persistent.
What’s recommended in the documentation is to use placement criteria to force such containers to deploy to specific hosts. For example, the criteria we discussed earlier deploy the databases to a node with the type label equal to db.
In the next section, we will make sure that there is exactly one such node in the swarm. To ensure that the database data directories are at a known location, let’s change the declarations for the db-userauth and db-notes containers, as follows:
services:
..
db-userauth:
…
volumes:
# – db-userauth-data:/var/lib/mysql
– type: bind
source: /data/users
target: /var/lib/mysql
…
db-notes:
…
volumes:
# – db-notes-data:/var/lib/mysql
– type: bind
source: /data/notes
target: /var/lib/mysql
…
# volumes:
# db-userauth-data:
# db-notes-data:
…
In docker-local/docker-compose.yml, we used the named volumes, db- userauth-data and db-notes-data. The top-level volumes tag is required when doing this. In docker-swarm/docker-compose.yml, we’ve commented all of that out. Instead, we are using a bind mount, to mount specific host directories in the /var/lib/mysql directory of each database.
Therefore, the database data directories will be in /data/users and /data/notes, respectively.
This result is fairly good, in that we can destroy and recreate the database containers at will and the data directories will persist. However, this is only as persistent as the EC2 instance this is deployed to. The data directories will vaporize as soon as we execute terraform destroy.
That’s obviously not good enough for a production deployment, but it is good enough for a test deployment such as this.
It is preferable to use a volume instead of the bind mount we just implemented. Docker volumes have a number of advantages, but to make good use of a volume requires finding the right volume driver for your needs. Two examples are as follows:
- In the Docker documentation, at https://docker.com/storage/volumes/, there is an example of mounting a Network File System (NFS) volume in a Docker container. AWS offers an NFS service—the Elastic Filesystem (EFS) service—that could be used, but this may not be the best choice for a database container.
- The REX-Ray project (https://com/rexray/rexray) aims to advance the state of the art for persistent data storage in various containerization systems, including Docker.
Another option is to completely skip running our own database containers and instead use the Relational Database Service (RDS). RDS is an AWS service offering several Structured Query Language (SQL) database solutions, including MySQL. It offers a lot of flexibility and scalability, at a price. To use this, you would eliminate the db-notes and db-userauth containers, provision RDS instances, and then update the SEQUELIZE_CONNECT configuration in svc-notes and svc-userauth to use the database host, username, and password you configured in the RDS instances.
For our current requirements, this setup, with a bind mount to a directory on the EC2 host, will suffice. These other options are here for your further exploration.
In this section, we converted our Docker compose file to be useful as a stack file. While doing this, we discussed the need to influence which swarm host has which containers. The most critical thing is ensuring that the database containers are deployed to a host where we can easily persist the data—for example, by running a database backup every so often to external storage. We also discussed storing secrets in a secure manner so that they may be used safely by the containers.
At this point, we cannot test the stack file that we’ve created because we do not have a suitable swarm to deploy to. Our next step is writing the Terraform configuration to provision the EC2 instances. That will give us the Docker swarm that lets us test the stack file.
Source: Herron David (2020), Node.js Web Development: Server-side web development made easy with Node 14 using practical examples, Packt Publishing.