Backend Code Guide

Backend Code Guide

Goal

The goal of this document is to explain the organisation and development guidelines of the backend component of An Scéalaí.

Current Setup

The backend can be found at AnScealai/api. This folder contains many files and folders necessary for running the backend server and connecting to the external database. As An Scéalaí is a MEAN¹ stack application, the backend is run using Node.js² with Express³ middleware and a MongoDB⁴ database. To start the backend, use Node to run the server.ts file⁵. This file creates the server that sets up all the backend functionality and is what ties in all the files described in this document.

This document serves mainly as an organisation reference and quick development guide, so it contains many links to external docs that go into greater detail on different parts of the code. These are either links to external docs for imported services, or they are links to other docs written particularly for An Scéalaí.

Node.js - Modules

Many of the backend functions and data are created using modules in order to modularise and pass data back and forth between files. Modules⁶ are self-contained units of functionality that can be exported and reused in other JavaScript files. There are many different ways to define modules, but for the most part in An Scéalaí we use the CommonJS format, as this is the standard in Node.js. We can use modules in our code by importing them with the require keyword. Node.js already has a few built-in modules, but we also import many modules we define ourselves.

Creating a module:
A module can be created and exported by using the exports keyword. For example, after defining a particular block of code (such as a function), this code can be exported by assigning it to module.exports. Below is an example of a module created and exported in the api/endpoints_functions/story/creates.js file:

This module can then be used in the Story route (api/routes/story.route.js) by importing it into the file:

Most of the JavaScript files in the backend of An Scéalaí import these types of modules or export their own.

Backend File System

This section goes into detail about the different folders and files in the backend of An Scéalaí and gives a brief explanation about how they are used within the context of the entire backend. Any links to documentation for specific functionalities are provided as needed.

Config

This folder contains a file called passport.js. This file is used by the server and in the user route file for verifying user passwords when they log in. The documentation for this file can be accessed here.

Controllers

This folder contains two control files: authentication.js and profile.js. The authentication.js file is used by the user route for any functionality regarding user authentication (logging in, registering, etc.) The documentation can be accessed here. The profile.js file is used by the user route for accessing all users and teachers from the DB. (Could possibly be refactored into a router folder, doesn’t seem like the best place for these endpoints)

Coverage

This folder stores coverage report information for any tests run on the system.

Curls

This folder contains a few bash scripts that can be run for testing certain functionalities. Not currently used by the server, just here for convenience.

Endpoint_functions

This folder contains folders of files for different routing endpoints. This is the refactored version of the original endpoints stored in AnScealai/api/routes. See this documentation that explains endpoint functions and how to create them.

Keys

This folder contains a load.ts file that stores certain private/public key values to the session environment. These keys are required in the servers.ts file in order to use JSON web tokens (JWT) and to connect to the DB. There are currently two versions for the keys, one for production and one for development, but at the moment these keys are the same.

Logs

This folder contains a couple files that contain any error logs that are written by the logger. error.log contains only error messages and combined.log contains all logged messages, success or error. These files can be referenced for debugging purposes. See this documentation for creating and using the logger.

Models

This folder contains the different mongoose schemas for the collections and objects stored in MongoDB. Data retrieved from the DB gets mapped to the schemas defined in these files so that it can be processed and sent to the frontend. See this documentation for creating and using mongoose Models.

Mongo_scripts

This folder contains a few JavaScript files for accessing the DB using Node.js. This is good for testing purposes as we can make requests to the database without having to write code in the frontend to access a particular endpoint. These scripts are only used for convenience; they are not necessary for running the server. The folder contains the following files:

• dumpStoriesFromEngagement.js: This file takes a username from a command line argument and uses it to fetch engagement objects for stories written by that author. It prints these stories to the command line.
• getAllStoriesByUser.js: This file takes a username from a command line argument and uses it to fetch all the stories written by that author. It prints these stories to the command line.
• getAllStoriesFromEngagementByStudentIdAndGiveToStudent.js: This file takes a student’s username and ID from command line arguments and uses them to find engagement objects that contain story data written by the given student. This script was used when there was a bug with student’s stories disappearing. The data from the engagement story objects was then used to create new story objects in order to restore the user’s stories.
• getAllUniqueEmails.js: This file loops through the users in the DB and prints a record of any duplicate emails (users with the same email address).
• gramadoirViewExample.js: This script calls a function in ../utils/gramadoirView to create a database view on the grammar collection in the DB.
• setEmail.js: This file takes a username and email from command line arguments and uses them to update the user’s email in the DB.
• setOwnerId.js: This file finds (old) stories in the DB who don’t have the owner property. It loops through the users in the DB and for each user loops through their stories. If a story doesn’t have the owner property, one is created and set to the user’s ID.
• setPassword.js: This file takes a username and password from command line arguments and uses them to update a user’s password.
• setStatus.js: This file takes a username and status from command line arguments and uses them to update a user's status.

Node_modules

This folder contains all the node modules used for backend development. It is necessary to reinstall these modules as needed if new dependencies are added or versions are updated. This folder is Git Ignored⁷, so they are only saved locally; therefore, they need to be initialised when cloning An Scéalaí for the first time or when changing to a development branch that uses different package versions.

Routes

This folder contains all the different routes that the server directs requests to based on the request URL. These routes contain endpoint functions that read/write to the database as requested. For example, if a request contains ‘/user’ in the URL, it is directed to the user route. If the URL contains ‘/story’, it is directed to the story route. These configurations are set up in the server.ts file. (See server.ts documentation)

This folder also contains a folder called storiesForDownload. This folder mostly remains empty, as it serves as a temporary folder to store story files that the user can download. When a user wants to download a story, the backend converts the story text data from the DB into the desired format (word, pdf, etc.) Node.js then has to create this file so it can send it back to the user for downloading. The file is created in this folder, and then it is deleted after it is sent back to the user.

Utils

This folder contains helpful JavaScript files that are called throughout the backend to perform particular tasks. These are used either for serving purposes or testing purposes. The details of each file are explained below:

• PIError.js

  • This file defines classes for particular HTTP/API errors that could occur when the user makes requests to the server. These errors are imported in certain route files so if any errors occur we can return custom messages and statuses.

• dbUrl.js

  • This file defines which URL and config values the server should use when connecting to the MongoDB database. It checks the current system session environment for these variables, and if they aren’t set then it gets the data from the ../DB.js file. Different databases are set up for development, production, and testing, so they all have different configs.

• errorHandler.js

  • This file handles the custom errors as defined in APIErrors.js. It is configured as an Express middleware when setting up the server.

• gramadoirView.js

  • This file creates a function for rendering a particular view of the gramadoir.story.history collection in the DB. (It is called from the gramadoirViewExample.js file in ../mongo_scripts)

• grammar.js

  • This file manipulates grammar data in the GramadoirCache and GramadoirStoryHistory mongo collections. (Possibly no longer used?)

• grammar.test.js

  • This file contains tests for the grammar.js file.

• jwtAuthMw.ts

  • This file configures the express-jwt module in order to use JWT tokens on the server.

• jwtAuthMw.test.js

  • This file contains tests for the jwtAuthMw.ts file.

• makeEndpoints.js

  • This file contains the makeEndpoints() function necessary for generating endpoints in the route folders using the functions defined in the endpoints_functions folder. See endpoint documentation for more details.

• makeFakeRes.js

  • This file contains a function that creates a mock Express Response object to be used for testing endpoints.

• mongoClient.js

  • This file contains the code for setting up a Mongo client on the server. (Possibly deprecated and replaced by dbUrl.js?)

• runSynth.js

  • This file runs test requests to the APIv2 synthesiser.

• test-setup-per-file.js

  • This file contains tests for setting up a Mongoose connection.

• test-setup.js

  • This file defines a Mongo URL for a test database. It requires the keys defined in ../keys/dev/load.

• test-utils.js

  • This file removes all the collections in the test database.

Views

This folder contains any HTML code that needs to be served. Currently it just contains the one file account_verification.html. This file contains the HTML for displaying the successful verification page in the browser that the user is redirected to after clicking on the verification link in their email upon successful registration. The file contains < script > tags to get data from the URL search parameters. See more information in the verify module of the authentication controller

Miscellaneous Files

The remaining files in the backend do not belong to particular folders, but they still contain particular functions. Their functionality is explained below:

• .dockerignore

  • File that tells Docker to ignore the node_modules folder when building an image (Docker integration still in progress)

• .eslintrc.js

  • Config file for ESLint⁸, which analyses the code for bugs and raises errors in the IDE accordingly during development

• .npmrc

  • Gets config settings from the command line, in this case the database environment variable DB, which is set to an-scealai

• abair_base_url.js

  • Defines the base production URL (https://www.abair.ie) that endpoints should build from

• DB.js

  • Defines the variables necessary for connecting to the test database. Used in the utils folder

• Dockerfile

  • The start of a Docker image used for running the backend as a container (Docker integration still in progress)

• jest

  • Imports Jest⁹, which is used for testing the backend

• load

  • Copy of the load.ts file stored in the keys folder. (Do we still need this?)

• logger.js

  • File that contains the functions for creating the logger that is used to log any backend errors as they occur. See logger.js documentation

• mail.js

  • Contains the functions for sending emails to users, such as for verifying accounts and resetting passwords. See mail.js documentation

• Makefile

  • A Makefile that runs configs to set up the backend on Docker (still in progress)

• package-lock.json

  • An auto-generated file that contains all the node dependency version information. It is created when the node_modules are initialised.

• package.json

  • Included in all Node projects¹⁰. This file gives metadata about the system and defines which versions of packages are necessary to install in order to run the application (such as when running npm install).

• sendinblue.json

  • This file contains the username and password data necessary for connecting to the Sendinblue mail service. It is Git Ignored since it contains sensitive information. See mail.js documentation .

• server.ts

  • The server file that is run when starting the backend. See server.ts documentation

• temp

  • A temporary TypeScript file for checking grammar tags (I don’t think this is being used)

• tsconfig.json

  • This file defines the TypeScript configuration¹¹ for the backend

---

Footnotes

1. MEAN stack: https://en.wikipedia.org/wiki/MEAN_(solution_stack) ↩

2. Node.js: https://nodejs.org/en/ ↩

3. Express: https://expressjs.com/ ↩

4. MongoDB: https://www.mongodb.com/ ↩

5. Server.ts documentation ↩

6. Modules in NodeJS: https://www.sitepoint.com/understanding-module-exports-exports-node-js/ ↩

7. Git Ignore: https://www.w3schools.com/git/git_ignore.asp?remote=github ↩

8. ESLint: https://eslint.org/ ↩

9. Jest: https://jestjs.io/ ↩

10. Package.json: https://docs.npmjs.com/cli/v9/configuring-npm/package-json ↩

11. TS config: https://www.typescriptlang.org/docs/handbook/tsconfig-json.html ↩