Have you ever been building an application and had the need for long-term data persistence, but didnt think you needed the full weight of a SQL or NoSQL data persistence solution? While building an embedded system with limited-to-no internet connection I ran into this situation first hand, and decided to look for a solution. After an exhaustive search through npm and github, i quickly realized all the existing Node JS implementations were insufficient, for one reason or another, so I determined that my best path forward was to create a solution myself.
The application i was building runs on an embedded system and needed to periodically scan the file system and gather metadata, based on file extensions. This metadata query process proved to be slow, especially when scanning thousands of files, so i wanted to persist this information somewhere. Unfortunately, these embedded systems have limited-to-no internet access, and can potentially be installed and provisioned by non-technical users. The amount of metadata I needed to store was also rather small, in terms of storage, so my initial thought was "why not just persist this data to a file"? The idea of having this information in-memory was intriguing, because querying a database, even a local one, thousands of times would definitely be expensive. Instead, if the data persistence solution could exist in memory, and store the data to a Json file for long-term persistence, this would suit my needs perfectly.
The json-repo library provides a standardized "Repository" interface where you can perform traditional CRUD operations on your data models, as well as a "RepositoryContext" class to manage Json data persistence. The repository context contains a collection of repositories, all of which are stored (by default) on the same Json file. To load the repository models into memory, you simply call "initialize()" on the repository context implementation; to save the repository data to the Json file, call the "saveChanges()" method on the repository context.
npm install json-repo --save
You can create a repository context by extending the "RepositoryContext" class exposed by the json-repo npm package.
First, create an a model to provide types for the data we are going to store: For example:
//file: foo.ts
export class Foo {
bar: string;
}
Next, create an implementation that extends the base "RepositoryContext" class:
//file: sample-repository.ts
import { RepositoryContext, Repository } from 'json-repo';
import { Foo } from './foo';
export class SampleRepository extends RepositoryContext {
constructor(dataPath: string) { super(dataPath); }
models = {
baz: new Repository<Foo>()
}
}
Finally, we need to initialize the repository in our application:
import * as path from 'path';
import { SampleRepository } from './sample-repository';
import { Foo } from './foo';
const jsonFilePath = path.join(__dirname, 'db.json');
let repository = new SampleRepository(jsonFilePath);
repository.initialize().then(_ => {
//you can now use the repository
let foo = new Foo(); foo.bar = 'baz';
repository.models.baz.add('1', foo);
return repository.saveChanges();
});
By default, the "RepositoryContext" class is setup to use the built-in "JsonFileService" class to store Json data on the host machine. However, there are a couple features that allow developers to customize this behavior.
If you just want the benefit of the repository pattern, without actually persisting to a Json file, simply do no provide (or use null) the "dataPath" property when constructing your repository context implementation. This is especially useful for unit testing. For example:
//file: sample-repository.ts
import { RepositoryContext, Repository } from 'json-repo';
import { Foo } from './foo';
export class SampleRepository extends RepositoryContext {
//notice the constructor calls "super" with zero arguments
constructor() { super(); }
models = {
baz: new Repository<Foo>()
}
}
If you want to use the features of json-repo, but have unique data persistence requirements, there is a way to provide a custom service to handle data persistence.
The "RepoContext" class takes an optional parameter called "nodeEntityService", which is an interface with the following definition:
export interface IEntityNodeService {
getEntityNodes(path: string): Promise<{[model: string]: EntityNode[];}>;
persistEntityNodes<T>(path: string, data: T): Promise<void>;
}
This interface could be implemented to, for example, write Json files to a network storage device, or a MongoDb collection. Once you have defined your own implementation for "IEntityNodeService", simply provide a concrete implementation to the "RepositoryContext" class when constructing. For example:
//file: sample-repository.ts
import { RepositoryContext, Repository, IEntityNodeService } from 'json-repo';
import { Foo } from './foo';
export class NetworkJsonFileService implements IEntityNodeService {
//this is your custom implementation
}
export class SampleRepository extends RepositoryContext {
constructor(dataPath: string, nodeEntityService: NetworkJsonFileService) { super(dataPath, nodeEntityService); }
models = {
baz: new Repository<Foo>()
}
}