Filesystems are an often-overlooked database engine. While filesystems don’t have the sort of query features supported by database engines, they are still a reliable place to store files. The Notes schema is simple enough, so the filesystem can easily serve as its data storage layer.
Let’s start by adding two functions to the Note class in models/Notes.mjs:
export default class Note {
…
get JSON() {
return JSON.stringify({
key: this.key, title: this.title, body: this.body
});
}
static fromJSON(json) {
const data = JSON.parse(json);
if (typeof data !== ‘object’
|| !data.hasOwnProperty(‘key’)
|| typeof data.key !== ‘string’
|| !data.hasOwnProperty(‘title’)
|| typeof data.title !== ‘string’
|| !data.hasOwnProperty(‘body’)
|| typeof data.body !== ‘string’) {
throw new Error(‘Not a Note: ${json}’);
}
const note = new Note(data.key, data.title, data.body);
return note;
}
}
We’ll use this to convert the Note objects into and from JSON-formatted text.
The JSON method is a getter, which means it retrieves the value of the object. In this case, the note.JSON attribute/getter (with no parentheses) will simply give us the JSON representation of the note. We’ll use this later to write to JSON files.
fromJSON is a static function, or factory method, to aid in constructing the Note objects if we have a JSON string. Since we could be given anything, we need to test the input carefully. First, if the string is not in JSON format, JSON.parse will fail and throw an exception. Secondly, we have what the TypeScript community calls a type guard, or an if statement, to test whether the object matches what is required of a Note object. This checks whether it is an object with the key, title, and body fields, all of which must be strings. If the object passes these tests, we use the data to construct a Note instance.
These two functions can be used as follows:
const note = new Note(“key”, “title”, “body”);
const json = note.JSON; // produces JSON text
const newnote = Note.fromJSON(json); // produces new Note instance
This example code snippet produces a simple Note instance and then generates the JSON version of the note. Then, a new note is instantiated from that JSON string using from JSON().
Now, let’s create a new module, models/notes-fs.mjs, to implement the filesystem datastore:
import fs from ‘fs-extra’; import path from ‘path’;
import util from ‘util’;
import { approotdir } from ‘../approotdir.mjs’;
import { Note, AbstractNotesStore } from ‘./Notes.mjs’;
import { default as DBG } from ‘debug’;
const debug = DBG(‘notes:notes-fs’);
const error = DBG(‘notes:error-fs’);
This imports the required modules; one addition is the use of the fs-extra module. This module is used because it implements the same API as the core fs module while adding a few useful additional functions. In our case, we are interested in fs.ensureDir, which verifies whether the named directory structure exists and if not, a directory path is created. If we did not need fs.ensureDir, we would simply use fs.promises since it, too, supplies filesystem functions that are useful in async functions.
Now, add the following to models/notes-fs.mjs:
export default class FSNotesStore extends AbstractNotesStore {
async close() { }
async update(key, title, body) {
return crupdate(key, title, body);
}
async create(key, title, body) {
return crupdate(key, title, body);
}
async read(key) {
const notesdir = await notesDir();
const thenote = await readJSON(notesdir, key);
return thenote;
}
async destroy(key) {
const notesdir = await notesDir();
await fs.unlink(filePath(notesdir, key));
}
async keylist() {
const notesdir = await notesDir();
let filez = await fs.readdir(notesdir);
if (!filez || typeof filez === ‘undefined’) filez = [];
const thenotes = filez.map(async fname => {
const key = path.basename(fname, ‘.json’);
const thenote = await readJSON(notesdir, key);
return thenote.key;
});
return Promise.all(thenotes);
}
async count() {
const notesdir = await notesDir();
const filez = await fs.readdir(notesdir);
return filez.length;
}
}
The FSNotesStore class is an implementation of AbstractNotesStore, with a focus on storing the Note instances as JSON in a directory. These methods implement the API that we defined in Chapter 5, Your First Express Application. This implementation is incomplete since a couple of helper functions still need to be written, but you can see that it relies on files in the filesystem. For example, the destroy method simply uses fs.unlink to delete the note from the disk. In keylist, we use fs.readdir to read each Note object and construct an array of keys for the notes.
Let’s add the helper functions:
async function notesDir() {
const dir = process.env.NOTES_FS_DIR
|| path.join(approotdir, ‘notes-fs-data’);
await fs.ensureDir(dir);
return dir;
}
const filePath = (notesdir, key) => path.join(notesdir, ‘${key}.json’);
async function readJSON(notesdir, key) {
const readFrom = filePath(notesdir, key);
const data = await fs.readFile(readFrom, ‘utf8’);
return Note.fromJSON(data);
}
async function crupdate(key, title, body) {
const notesdir = await notesDir();
if (key.indexOf(‘/’) >= 0) {
throw new Error(‘key ${key} cannot contain ‘/”);
}
const note = new Note(key, title, body);
const writeTo = filePath(notesdir, key);
const writeJSON = note.JSON;
await fs.writeFile(writeTo, writeJSON, ‘utf8’);
return note;
}
The crupdate function is used to support both the update and create methods. For this Notes store, both of these methods are the same and write the content to the disk as a JSON file.
As the code is written, the notes are stored in a directory determined by the notesDir function. This directory is either specified in the NOTES_FS_DIR environment variable or in notes-fs-data within the Notes root directory (as learned from the approotdir variable). Either way, fs.ensureDir is used to make sure that the directory exists.
The pathname for Notes is calculated by the filePath function.
Because the pathname is ${notesDir}/${key}.json, the key cannot use characters that cannot be used in filenames. For that reason, crupdate throws an error if the key contains a / character.
The readJSON function does what its name suggests—it reads a Note object as a JSON file from the disk.
We’re also adding another dependency:
$ npm install fs-extra –save
We’re now almost ready to run the Notes application, but there’s an issue that first needs to be resolved with the import() function.
1. Dynamically importing ES6 modules
Before we start modifying the router functions, we have to consider how to account for multiple AbstractNotesStore implementations. By the end of this chapter, we will have several of them, and we want an easy way to configure Notes to use any of them. For example, an environment variable, NOTES_MODEL, could be used to specify the Notes data model to use, and the Notes application would dynamically load the correct module.
In Notes, we refer to the Notes datastore module from several places. To change from one datastore to another requires changing the source in each of these places. It would be better to locate that selection in one place, and further, to make it dynamically configurable at runtime.
There are several possible ways to do this. For example, in a CommonJS module, it’s possible to compute the pathname to the module for a require statement. It would consult the environment variable, NOTES_MODEL, to calculate the pathname for the datastore module, as follows:
const notesStore = require(‘../models/notes-${process.env.NOTES_MODEL}.js’);
However, our intent is to use ES6 modules, and so let’s see how this works within that context. Because in the regular import statement the module name cannot be an expression like this, we need to load modules using dynamic import. The dynamic import feature—the import() function, in other words—does allow us to dynamically compute a module name to load.
To implement this idea, let’s create a new file, models/notes-store.mjs, containing the following:
import { default as DBG } from ‘debug’;
const debug = DBG(‘notes:notes-store’);
const error = DBG(‘notes:error-store’);
var _NotesStore;
export async function useModel(model) {
try {
let NotesStoreModule = await import(‘./notes-${model}.mjs’);
let NotesStoreClass = NotesStoreModule.default;
_NotesStore = new NotesStoreClass();
return _NotesStore;
} catch (err) {
throw new Error(‘No recognized NotesStore in ${model} because
${err};);
}
}
export { _NotesStore as NotesStore };
This is what we might call a factory function. It uses import() to load a module whose filename is calculated from the model parameter. We saw in notes-fs.mjs that the FSNotesStore class is the default export. Therefore, the NotesStoreClass variable gets that class, then we call the constructor to create an instance, and then we stash that instance in a global scope variable. That global scope variable is then exported as NotesStore.
We need to make one small change in models/notes-memory.mjs:
export default class InMemoryNotesStore extends AbstractNotesStore {
… }
Any module implementing AbstractNotesStore will export the defined class as the default export.
In app.mjs, we need to make another change to call this useModel function. In Chapter 5, Your First Express Application, we had app.mjs import models/notes- memory.mjs and then set up NotesStore to contain an instance of InMemoryNotesStore. Specifically, we had the following:
import { InMemoryNotesStore } from ‘./models/notes-memory.mjs’;
export const NotesStore = new InMemoryNotesStore();
We need to remove these two lines of code from app.mjs and then add the following:
import { useModel as useNotesModel } from ‘./models/notes-store.mjs’;
useNotesModel(process.env.NOTES_MODEL ? process.env.NOTES_MODEL :
“memory”)
.then(store => { })
.catch(error => { onError({ code: ‘ENOTESSTORE’, error }); });
We are importing useModel, renaming it useNotesModel, and then calling it by passing in the NOTES_MODEL environment variable. In case the NOTES_MODEL variable is not set, we’ll default to the “memory” NotesStore. Since useNotesModel is an async function, we need to handle the resulting Promise. .then handles the success case, but there is nothing to do, so we supply an empty function. What’s important is that any errors will shut down the application, so we have added .catch, which calls onError to do so.
To support this error indicator, we need to add the following to the onError function in appsupport.mjs:
case ‘ENOTESSTORE’:
console.error(‘Notes data store initialization failure because’, error.error);
process.exit(1);
break;
This added error handler will also cause the application to exit.
These changes also require us to make another change. The NotesStore variable is no longer in app.mjs, but is instead in models/notes-store.mjs. This means we need to go to routes/index.mjs and routes/notes.mjs, where we make the following change to the imports:
import { default as express } from ‘express’;
import { NotesStore as notes } from ‘../models/notes-store.mjs’;
export const router = express.Router();
We are importing the NotesStore export from notes-store.mjs, renaming it notes. Therefore, in both of the router modules, we will make calls such as notes.keylist() to access the dynamically selected AbstractNotesStore instance.
This layer of abstraction gives the desired result—setting an environment variable that lets us decide at runtime which datastore to use.
Now that we have all the pieces, let’s run the Notes application and see how it behaves.
2. Running the Notes application with filesystem storage
In package.json, add the following to the scripts section:
“fs-start”: “cross-env DEBUG=notes:* PORT=3000 NOTES_MODEL=fs node
./app.mjs”,
“fs-server1”: “cross-env NOTES_MODEL=fs PORT=3001 node ./app.mjs”,
“fs-server2”: “cross-env NOTES_MODEL=fs PORT=3002 node ./app.mjs”,
With this code in place, we can now run the Notes application, as follows:
$ DEBUG=notes:* npm run fs-start
> notes@0.0.0 fs-start /Users/david/chap07/notes
> cross-env DEBUG=notes:* PORT=3000 NOTES_MODEL=fs
node ./app.mjs notes:debug Listening on port 3000 +0ms
notes:notes-fs keylist dir /home/david/Chapter07/notes/notes-fs-data files=[] +0ms
We can use the application at http://localhost:3000 as before. Because we did not change any template or CSS files, the application will look exactly as you left it at the end of Chapter 6, Implementing the Mobile-First Paradigm.
Because debugging is turned on for notes:*, we’ll see a log of whatever the Notes application is doing. It’s easy to turn this off by simply not setting the DEBUG variable.
You can now kill and restart the Notes application and see the exact same notes. You can also edit the notes in the command line using regular text editors such as vi. You can now start multiple servers on different ports, using the fs-server1 and fs- server2 scripts, and see exactly the same notes.
As we did at the end of Chapter 5, Your First Express Application, we can start the two servers’ separate command windows. This runs two instances of the application, each on different ports. Then, visit the two servers in separate browser windows, and you will see that both browser windows show the same notes.
Another thing to try is specifying NOTES_FS_DIR to define a different directory to store notes.
The final check is to create a note where the key has a / character. Remember that the key is used to generate the filename where we store the note, and so the key cannot contain a / character. With the browser open, click on ADD Note and enter a note, ensuring that you use a / character in the key field. On clicking the Submit button, you’ll see an error saying that this isn’t allowed.
We have now demonstrated adding persistent data storage to Notes. However, this storage mechanism isn’t the best, and there are several other database types to explore. The next database service on our list is LevelDB.
Source: Herron David (2020), Node.js Web Development: Server-side web development made easy with Node 14 using practical examples, Packt Publishing.