New Chapter: Discourse Forum
Want to setup your own Discourse forum like Devforum?
New Chapters: Monolith and Microservices (Netflix)
Every been interested in how businesses like Netflix build their application system and handle it’s massive services?
Introduction to Web Development
A tutorial for Roblox users to enter web development!
This tutorial is expected to be expanded up with topics deemed important, if you believe something should be added please post a comment.
The chapter on REST API for scripting with Nodejs is incomplete but enough to get started.
Guides
How to Program a Website (Node.js Beginners Guide)
Guide to Scripting Bots | Javascript Tutorial | FunCaptcha and New Host Info
Contact
Discord: TechSpectrum#2620
Email: CourtSmith1101@gmail.com
Updates
2019-06-30T05:00:00Z
- Discourse Forum chapter.
- Monolith chapter.
- Microservices chapter.
- Netflix and Uber (Monolith to Microservices)
2019-03-16T05:00:00Z
- Bootable USB chapter for Kali Linux and Fedora.
2019-03-04T06:00:00Z
- Minecraft Server chapter.
2019-02-13T06:00:00Z
- Career and Education chapter.
- Double Authentication chapter.
2019-02-12T06:00:00Z
- Added Docker chapter.
- Added Kubernetes chapter.
2019-02-03T06:00:00Z
- Added Cloudflare DNS setup instructions.
2019-02-02T06:00:00Z
- Added instruction on how to setup a MySQL database on the server for a VPS.
- Added pm2 auto-restart in the Digital Ocean tutorial.
2018-09-01T05:00:00Z
- Updated Digital Ocean automated deployment section for batch script and sudo permissions.
- Added the SSL configuration file path:
sudo nano /etc/nginx/snippets/ssl-params.conf
2018-08-31T05:00:00Z
- Update roblox-js library to the latest version:
4.0.2
- Express Rate Limit no longer supports delay functionality in version 3, you can still get it in version
2.11.0
however this tutorial will not use it. - Planning OAuth2 tutorial for Discord and Google.
Table of Contents
-
Introduction
- About
- Who is This For?
- Disclaimer
- Downloads
- Helpful Links
-
Tools
- Cmder
- Nodejs
- Git
-
REST API
- What is an REST API?
- Creating a Database (500 MB Free)
- Creating an Express App
- Scripting the Database
-
SQL and NoSQL
- About
- SQL
- NonSQL
-
Web Frameworks and Libraries
- About
- Bootstrap
- Angular
- React
-
Security and Attacks
- DoS and DDoS
- Cross Site Request Forgery
- Spoofing
- Phishing
- Man in the Middle
- Privilege Escalation
- Click Jacking
- Cross Site Scripting
- Brute Force
- Social Engineering
-
SSH
- What is an SSH Key and What Does it Do?
- How to Make a SSH Key
-
Digital Ocean
- What’s Digital Ocean and Why use it?
- Create a Droplet
- Connect to a Droplet
- Create a Admin User
- Disable Root Login
- Firewall Setup
- Automated Deployment
- DNS Setup
- SSL Certificate (Free)
- Nginx Setup
-
Heroku
- What is Heroku?
- Setup
- App Deployment
-
Roblox Scripting
- HttpService
- Make a Request
-
Discord
- Create a Bot
- Connect a Bot
- Writing Commands
-
MySQL
- Setup
- Tools
-
Cloudflare
- DNS Records
- SSL Allowance
- Docker
- Kubernetes
-
Career & Education
- Job Types
- Job Searching
- Getting an Education
-
Double Authentication
- SMS
- TOTP
- Biometric
- Geographical
- Timezone
- Security Device
-
Minecraft Server
- About
- Requirements
- Setup Spigot
- Server Setup
- DNS Setup
- Startup and Connect
-
OS Bootable USB
- What is a bootable USB?
- Fedora
- Kali Linux
- Boot from USB
-
Discourse Forum
- SMTP Server (Mailgun)
- Setup SSL
- Setup Nginx
- Setup Discourse
-
Monolith Applications
- Pros
- Cons
-
Monolith Applications
- Pros
- Cons
- Netflix and Uber (Monolith to Microservices)
Other Web Topics:
- Proxies: NGINX and Apache
- OAuth2 (Google, Discord, Facebook)
- Web Scrapping
- Web Sockets
- Framworks: Vue, Django, Metor, Go
- Languages: Python, PHP
- Load Balancing
Introduction
Read Chapter
About
This tutorial will cover several topic for creating a full project for Roblox in Nodejs using MongoDB as it’s database and Express as it’s web router for HTTP request. You will learn how to build an application that can use the Roblox API, use the Discord API, and most important build your own web API that will run on your host of choice.
We’ll talk about security such the attacks currently being used today by people who want to do you harm. It is very important you develop with security in mind and staying up to date with today’s methods of defending against them.
I am TechSpectrum, I develop clan databases and currently host for over 40 clans of varying size from 100 to 300,000 users. I been a Roblox since 2009 learning lua and eventually came across technology by RAT currently owned by @Polymorphic that utilizes a web based database. It was an inspiration that eventually drove me to learn how such a feat was possible on Roblox.
Today I wish share that knowledge with those who want to help themselves and learn how to down towards the path of development on a platform they love. I hope those of you who follow this tutorial enjoy, and come to understand a little bit about this field and have fun with it as I have and continue to learn with me.
With that said, lets get started!
Who is This For
This tutorial is meant to teach people who want become web developers where to start and how to make a full fledged project from start to finish using Nodejs. It’s not recommend you start learning from this tutorial without basic knowledge of Javascript or Lua. This guide will cover how to cover the tools and the setup but not the language.
It’s difficult to find all the resources you need so this is a good quickstart to learning and understanding how most web projects work and everything involved with running websites.
Disclaimer
I do not condone any malicious development or actions that may result from this tutorial. Please be ethical and understand that what you do with this knowledge I am not responsible for.
Downloads
-
Nodejs
- The Javascript Runtime Engine
- Built using C++
- Uses the Chrome Browsers V8 Engine
- Includes npm (Node Package Manager)
-
Git
- Version Control Software
- Uses webhooks to receive updates.
- Stores our application with version history.
- Repo Sites: GitHub, GitLab, GitBucket
- Includes: GitBash Console (Linux Based)
-
Cmder
- A robust and customizable command prompt.
- Will be used to run multiple consoles under one window.
- Any script editor.
-
MongoDB Compass
- Editor for MongoDB Databases.
- Download
Community Edition Stable
-
Postman
- Used to test our API
Helpful Links
-
Discord.js Documentation
- The library for using the Discord API.
-
Roblox.js Documentation
- The library for using the Roblox API
- Note: Some API is out dated (i.e. Wall Post)
-
Discord Developers: My Apps
- Your discord page for managing your applications.
-
Discord Bot Permissions Calculator
- Used to assign your bot privileges for invitations.
Last Updated: August 26, 2018
Tools
Read Chapter
Cmder
How to Use
-
cd Projects
- Access a folder in your current directory.
- Projects may be a folder in your Desktop directory.
- If the folder has spaces use quotes:
cd "My Projects"
-
cd ..
- Go back to the parent directory.
- Program Files would be the parent directory of Adobe
-
ls
- List current directory files.
-
C:
- If you want to changes drives you can type change it like this.
- If I used a external drive I may say
E:
Here is a Windows Command List and a Linux Command List
We’ll be using a few Linux commands later for Digital Ocean which uses a Linux Machine such as Ubuntu.
Installation
- Make a new folder, name it Cmder.
- Extract or Click and Drag your cmder.zip file contents into your folder.
- Place your folder in your “Programs Files” directory in your C: Drive.
- Click and start Cmder using the launcher.
- On the bottom bar press and click the green plus button (New Console).
- Click on “Run as Administrator” and Start .
- Make sure you are currently in the directory of the Cmder Folder.
(C:\Program Files\Cmder)
- Paste and run:
.\cmder.exe /REGISTER ALL
Now you can right click anywhere and see Cmder Here
Tips
- You can right click a tab to rename the tab, restart the prompt, or close it.
- On the far right is blue button where you can access settings and go to background to add your own image or color scheme.
- Use 3 tabs.
- One for the Application
- One for installations, updates, etc.
- One for your web host console (i.e. Digital Ocean).
- In Windows 10 you can click the top box where your folder name is and get the exact path of the folder which you can copy and paste into the command prompt to instantly access it.
Nodejs
Nodemon - Automatic Restarts
Before we start using Nodejs, first we want to install nodemon.
Nodemon is a package that will handle automatically restarting your application when it detects changes in your scripts when you save a file. This is convenient for us so we don’t have to keep stop and restarting the application ourselves.
To install nodemon enter npm install -g nodemon
in the command prompt.
Without nodemon installed into our computers PATH variables we would only be able to run our application by saying node app.js
but now we can say nodemon app.js
to run it with nodemon monitoring.
Create a Project
You should be in your folder directory you want to save your project in before running any commands.
We’re going to want use a few packages, to make a package.json file run the following command:
npm init
You can fill in the details and type yes at the end to complete the setup.
Git
NOTE: Preferably come back to this segment after the “REST API” tutorial.
This is a popular version control software used by most of your large platforms and developers. There’s a lot about Git to learn but for most people you only need to know a few basics to using it that you’ll use all the time. It’s important to understand that this is not a app you can open up but a program that embeds itself into your PATH variables just like Nodejs.
In order to start using it, first we have to go to our project files directory and run the following command.
git init
It’ll tell you Initialized empty Git repository
unless you already have a git repo in the folder. What this does is make a folder that’s set to hidden which contains all the information and functionallity of git for repository work.
Now we need to “commit” these files to be a saved version in time, however we first need to stage our files so we’ll run:
git add .
This command will stage our files and what that means is now we’ll be able to store these into a commit because they are ready. If we want to only commit one file we could say: git add server.js
If you want to see the status of your repository files: git status
Once your files are finished being staged we can now commit them and add a message to our commit so we know what was changed. It’ll be our authors note that will be something we can go back to later in time.
git commit -m "Starting point of the Project"
After the commit is finished, we now have something we can use and especially rollback to in a emergency.
NOTE: This next part is only for when you have a repository you want to use such as Heroku or Digital Ocean.
As of right now in this tutorial we don’t have a Repository we can send to; but we’ll go ahead and talk about push
, pull
, and remotes
.
A remote is link that is public, for example: ```https://www.mywebsite.com/techspectrum/repo` is a url where a repo might be. We can create a remote like this:
git remote add MyRepo https://www.mywebsite.com/techspectrum/repo
We can then “push” to that repo as such:
git push MyRepo master
MyRepo is the name of my remote, and it’s pushing (uploading) our files to the websites repo through a webhook (we’ll make one later in Digital Ocean). Now the way git works is each commit is that not every commit is the entire project but the changes made and saved at that time. If we had version 7 of our project, and the server had 4, then when we push it’s going to send over commits 5, 6, and 7.
If were on a new computer with a empty git repo folder we’ll have to use:
git fetch MyRepo master
This will download all our commits, remotes, and so on. If a friend were working on a project with us, and we already had version 4 of the project, we’ll use:
git pull MyRepo master
Now lets pretend we had this situation:
Developer A: 10 → Pushes to Server
Server: 6 → 10 Updated by Developer A.
Developer B: 6
If Developer B pulls from the Server which is now version 10, he would get commits 7, 8, 9, and 10.
Now what is this “master” ?
Git has what we call branches, and that’s where other developer can branch off from others developers work to expand on a feature without disrupting the other projects. There may be a function “production” branch that gets update when the director performs a git merge
on different branches to bring them together. Unless you are working with a team you may not often use this, so for this tutorial we’ll assume you’re only going to use the main “master” branch.
There are git GUI’s out there such as Git Kraken which can make the use of Git much easier on you. You don’t want to break your project so I would say you should get comfortable with it as much as possible through practice and learning the common commands and what exactly it is they do.
Here is what I suggest for Git GUI:
GitKraken
SourceTree
REST API
Read Chapter
What is a REST API?
Representational State Transfer (REST)
Application program interface (API)
Communication and Endpoints
It’s a interface to application communication network often from client to server in web. In web programming our interface would be the endpoints. Endpoints are paths in our server that can be used to interact with the server to obtain information or perform an action.
Here is an example:
https://www.mywebsite.com/api/users/search
https://www.mywebsite.com/profiles/techspectrum
This would be a web endpoint that could be used to obtain search information from a database in our first example. It may send us back some JSON which we can convert to usable code for what language we are using (javascript).
In our second example we see I’m going to a profile, this may be a web page that sends us a display of my avatar and information so instead of JSON we’re being sent html, css, and other script files the page calls.
Methods
Methods are like action types we want to make on a endpoint when we send it a request. They act as a way to identify what we want to do.
Here’s a few:
- GET - Used to retrieve data from the server.
- POST - used to send data to the server.
- PUT - Used to update data.
- DELETE - Used to delete data.
In Roblox you used to only be able to use GET and POST but as of recently we can now modify headers to get the type of action we want. A endpoint can support multiple methods if you want to use the same one for different purposes.
Example:
https://mywebsite.com/api/profile
On this endpoint we may want to use GET to retrieve the user info. However we could also use DELETE if we want to destroy the users profile.
So where are these methods, how do we apply them?
Well they are in your http request header, that’s not something you can see in a search bar. if you want to see some of the request being made right now to your browser you can go to browsers web development settings. Then view Network and you’ll see a few 200’s and GET request if you reload the page.
Right now as I am writing this the website is actively sending request to it’s REST API to save my changes. Here is what it looks like:
As you can see it’s POST, which means my browser has script that are sending request to the server. When we browse on our computers to website we’re performing GET request. Our browser then uses a run time engine to display the website for us and interprets the scripts.
Status Codes
So what is this 200
and what does it mean exactly?
Status Codes are categorized in a series of hundreds, you can find the wiki here!
- Information (1XX)
- You won’t see these often.
- It’s used in the browser to communicate the state of the request initialized such as “please wait.”
- Success (2XX)
- Tells the browser we made a successful request that sent back just fine.
- The standard “OK” code when a request went through is 200.
- Others: 201 (Created), 202 (Accepted)
- Redirection (3XX)
- Commonly used in cases where you will be redirected such as by an OAuth sign in or moved to a new proxy.
- Client Error (4XX)
- This means you did something wrong on your end.
- The most commonly known 400 series code is 404 which means Not Found.
- Server Error (5XX)
- This means the server did something wrong.
- You might often see websites go down and Cloudflare will give you a error using their own codes (520, 521, etc.)
These are more or less guidelines, you do not actually have to follow or use this although it is preferred because most browsers and other clients will us the status code to perform the appropriate action.
Creating a Database (500 MB Free)
We’re going to be using a website called mLab which is a free database as a service provider for MongoDB databases. You will be able to make unlimited databases each with 500 MB free!
Sign up and login, and you be created with a web page that looks a bit empty, so we’re going to find and click on this button at the top of our page.
Choose the first option for Sandbox from any provider you wish.
Submit the order and locate your database back on your home page.
Click it and you’ll come across a similar layout.
- Collections - The database storage group (ex: users, logs, etc)
- Users - These are credentials used to access and connect to the database, we’re going to have to be make one in order to use our database.
- Stats - Explains variables in your database you’ll be able to see when connected.
- Backups - mLab’s backup system.
- Tools - Importing, Export, etc. It won’t be used unless you upgrade from Sandbox.
- MongoDB URI - We’ll be using this to connect to our database using Mongoose.
Lets head over to Users and add a user to our database.
Make sure you don’t make it read only or we can’t edit the database.
Creating an Express App
Express is a library for Nodejs which will handle our routing and middleware applications we want to use. We’ll be talking about some security measures you can add to take precautions against attackers.
In your command prompt (Cmder) make sure you are in your project folders directory and you have your package.json file.
To install express:
npm install express --save
You should see a folder now called node_modules
which contains all your packages used for express. Your package file should also now contain dependencies.
So now we’re going to make a new javascript file and call it server.js
and another called config.js
In our config script we’re going to write the following:
module.exports = {
database: process.env.DATABASE || "mongodb://RobloxDB:password1@ds133642.mlab.com:33642/roblox-database",
port: process.env.PORT || 5000,
secret: process.env.SECRET || "XXjwjw390Kp2k3203kwlwOOnsd33"
}
Make sure you use your databases URI from mLab and replace username and password with your user information you created. We also added a secret which will be what we include to authorize the access of our important API we don’t want others using without permission.
What’s process.env
?
We have environment variables we can create and put on our computer.
Back on our server script, we’re going to type he following.
const express = require('express');
const config = require ('./config')
// Creates the Express App
const app = express();
// Route
app.get('/', function(req, res){
console.log("GET Request.")
})
// Listen to Request
app.listen(config.port, function(){
console.log("Now listening for request.")
});
You can now save these files, we’re now going to run our application using nodemon in the console.
If we were to go to http://localhost:5000/
right now we would have an infinite wait because the server is still waiting for a response. Though you will see a message pop up in your console.
What we want to do right now is just send a short message back to the client and to do that we’re going to be using the response variable. res
app.get('/', function(req, res){
console.log("GET Request.")
res.send("Hello World!")
})
Now we want to get into the meat of things and create our own API web routes. However we want to organize a little.
First off, lets delete our current app.get() function as we don’t need it anymore.
Instead we will replace it with:
// API Web Routes
app.use('/api/users', require('./api/users'));
Now make a folder and name it api
and inside that folder we’re going to make another script called users.js
We’ll use this:
const express = require('express');
const router = express.Router();
router.get('/', function(req, res){
});
router.get('/:id', function(req, res){
});
router.post('/:id/updatecurrency/:value', function(req, res){
})
module.exports = router
Explanation: The router handles direction paths from other modules as if middleware. The :id
we added is so we can use it as a variable for finding a player later on.
http://localhost:5000/api/users/2939382
Right now your project should look like this:
Now we’re going to be making some data to start working with.
Scripting the Database
Before we start first we need some middleware.
Mongoose: Object Data Modeling (ODM) library mongoose for using the database.
Body Parser: To convert request data from JSON into a JS object.
Express Sanitize: Converts data into a string so we don’t end up using malicious data.
Express Rate Limit: Will help prevent our API from being abused.
Helmet: Adds security headers for us that by default prevent a few basic attacks.
npm install body-parser --save`
npm install mongoose --save```npm install express-sanitizer --save
npm install express-rate-limit --save`
npm install helmet --save`
I highly reccomend you read into each one and what they can do.
Here’s some documentation for express rate limit!
Now we have to edit our server.js
script to look like this:
const express = require('express');
const mongoose = require('mongoose')
const config = require ('./config')
const bodyParser = require('body-parser');
const rateLimit = require('express-rate-limit');
const helmet = require('helmet');
const expressSanitizer = require('express-sanitizer')
const cors = require('cors');
// Creates the Express App
const app = express();
// Connect to Database
mongoose.connect(config.database)
mongoose.Promise = global.Promise;
// Rate Limit Rule
var apiRateLimit = new rateLimit({
windowMs: 15*10000, // 15 minute window.
max: 500, // start blocking after 100 requests
message: "Too many request created from this IP."
});
// Middleware
app.use(cors());
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(expressSanitizer());
app.use(helmet())
// API Web Routes
app.use('/api/users', apiRateLimit, require('./api/users'));
// Error Handling
app.use(function(err, request, res, next){
console.log(err)
res.status(422).send({error: err.message});
})
// Listen to Request
app.listen(config.port, function(){
console.log("Now listening for request.")
});
So what did we do? First we added all of our packages we just installed, the we connected our database on mLab using Mongoose. Mongoose has a Promise that’s no longer supported so we had to replace it with Node’s Promise library.
A promise is asynchronous which means it can run along side other code at the same time without blocking other parts of a script. We’re using promises in place of callbacks.
To learn more about Promises check out the MDN Guide on Promises.
Next we added a rate limit, this is going to prevent the abuse of our API in the event someone learns it or your code ends up firing it off multiple times. It’ll help hold back traffic and you can change it’s settings depending on how much you feel need it to be tailored.
IMPORTANT
Without the API Rate Limit it’s possible to brute force your API to figure out your authentication token. a rate limit makes it much harder to forcefully get into.
Next we added some error handling which we can use if something goes wrong, it’ll send back a status code of 422 with an error message.
Moving on we’re going to make a folder called model
and inside it a javascript file named user.js
.
Inside of this script we will have:
const mongoose = require('mongoose')
const Schema = mongoose.Schema;
const UserSchema = new Schema({
userId: {
type: Number,
required: [true, "Requires a userId."]
},
cash: {
type: Number,
default: 100
}
})
let User = mongoose.model('user', UserSchema);
module.exports = User;
We’re using mongoose schema to create a structure for how we want to read our data. This Schema includes the properties and defaults we want. When a new user profile is made they will have 100 cash by default. If we don’t create a new user with the UserId it will fail. Lastly we took our Schema and create a new ‘user’ model.
Our collection will be automatically created and called ‘users’ because models are always singular and collections are always plural.
Last Updated: August 26, 2018
SQL and NoSQL
Read Chapter
About
When working on the backend you’re going to want to figure out where all your data is being saved and how. Earlier in the REST API tutorial we used mLab who host a MongoDB database service we can use which is NonSQL. You should choose which database you want to use based on the community resources, your teams skills, and the demand you’ll meet in terms of scale, traffic, and how you might be able to scale up if you need to upgrade your resources in a short amount of time without loss of data.
SQL
Structured Query Language (SQL) was made back in 1982 by Microsoft. This is still to this day a common standard to use. These databases use relational data, this means user profile data may have a “user id” variable that exist in one database that can be used to collect the same “user id” value in other database for product orders, action logs, etc. These data is primarily stored into tables and follow a schema which structures the information we want to store. To improve performance SQL databases are scaled vertically such as increasing the server RAM and CPU.
SQL Databases: MySql, Oracle, Sqlite, Postgres and MS-SQL
NoSQL
NoSQL databases existed back in the 1960’s but were not given the same labeled before SQL came about. Just recently within the past decade this back become popular once again to use with the rise of new databases such as MongoDB, RethinkDB, and more. Unlike SQL data is stored as a document usually with key-pair values. This data is considered to be considered schema-less however that only means the schema is dynamic for unstructured data. You can still even use a schema for NoSQL data such as with Mongoose. Unlike SQL, NoSQL databases are scaled horizontally which means increasing performance through adding more servers to reduce the load.
NoSQL Databases: MongoDB, BigTable, Redis, RavenDb, Cassandra, Hbase, Neo4j and CouchDb