Top

Getting Started

N|Solid

NodeSource has developed N|Solid to meet the needs of production enterprise environments. Built upon the experience and insights of several core contributors to the Node.js ecosystem, N|Solid provides live instrumentation of your production system's health and stability with no changes to your application code. In addition, N|Solid offers the ability to control access and define policies for your critical applications.

Overview of the N|Solid Components

"N|Solid Architecture"

N|Solid Runtime

At times, Node.js can feel like a black box. The shift to an asynchronous programming model changes how we interpret the data we do have and how we reason about it. In order to help customers gain more visibility, we provide the N|Solid Runtime. The N|Solid Runtime is a build of Node.js bundled with an advanced native C++ component, the N|Solid Agent. The N|Solid Agent runs on its own thread inside your application, with direct access to the V8 JavaScript engine and libuv, which are the core of Node.js.

The N|Solid Runtime provides access to metrics and allows you to control behavior at runtime. It also includes a powerful security policy model that allows restricting access to system resources by untrusted modules. A more secure memory allocation model is included to prevent unintended data exposure.

Node Compatibility

N|Solid 1.x is delivered bundled with a Node.js v4.x Argon LTS runtime. All N|Solid features are additive, meaning any application that runs on Node.js Argon will also work with N|Solid. To verify what version of Node.js that you have bundling N|Solid, use nsolid -v.

For more information about the Node.js API, consult the Node.js API Documentation.

N|Solid Hub

The N|Solid Hub provides a centralized service registry and proxy for all of your Node.js applications using N|Solid. It is designed to work with modern deployment models where services start or stop via auto-scaling or are simply containers of microservices in a cloud infrastructure.

N|Solid Console

The N|Solid Console provides centralized access to all of your applications and an aggregated view of each application's processes. This holistic view simplifies triage at runtime, makes it easier to look for outliers, and take advantage of advanced diagnostic tools directly from your browser.

Summary

The combination of the N|Solid Runtime, Hub and Console make N|Solid an invaluable tool for gaining insight into the behavior of your production applications at runtime.

Backed by NodeSource's 24x7 support, N|Solid adds enterprise-grade diagnostic and security features delivered on a release cycle that is aligned with Node.js Long Term Support (LTS) releases. This provides a stable, production platform for all of your Node applications.

Console Overview

The N|Solid Console provides valuable insight into clusters of N|Solid processes running in a variety of configurations. Processes are grouped by their associated application which makes it simple to find outliers in memory and/or CPU utilization and quickly identify unexpected behavior.

Navigating to an individual process shows various metrics and provides the capability for taking heap snapshots and CPU profiling.

"Flamegraphs"

Application View

The Application view provides a visual overview of N|Solid applications monitored by the Console. There are six columns in this view:

"Applications"

Column Description
Overview Grid The Overview Grid is a 5x5 matrix where each vertical dot represents 20% of available memory with the top being 100% memory utilization and each horizontal dot representing 20% of available CPU capacity with the far right being 100% of CPU capacity.

The size of the dot is a function of the number of processes occupying a particular usage slot; for example if five processes were at 22% CPU usage and 15% memory, that dot would be larger than a single process at 22% CPU Usage and 35% memory.

Colors are a reflection of what cell the dot is in; colors go from green to red as you move up and right on the grid. Red is naturally the color of severity, so if you have a lot of red dots on your application's grid, that is a call for immediate attention.

It is important to note that since an application can run on multiple hosts, you have have larger dots in the upper right; if you have 10 machines all running at 100% of CPU and memory, that upper right dot will be larger than a single process dot.

Note that this graph provides aggregate data: if your application executes in more than one process, there is no correlation between the number of enlarged dots and the number of processes running.
Application Name Each application gets its name from the NSOLID_APPNAME environment variable (NSOLID_APPNAME=\"SharkAttack\" nsolid myapp.js).

If NSOLID_APPNAME is not set, your application (along with any other unnamed applications) will appear in this list under nsolid-default. If two or more different processes are run with identical NSOLID_APPNAMEs, data from each process will be aggregated into a single application view entry.
Processes The number of processes being traced for this application
Hosts The number of host machines are being traced for this application.
CPU usage charts The CPU Usage Chart for the Hosts
RSS charts Resident Set Size (RSS) chart

Cluster View

The Cluster view is a force-directed scatter-plot that provides an overview of your application’s performance across all connected processes. The Y-axis plots the ratio of that process’ Resident Set Size (RSS) memory usage over the application’s total RSS memory usage. The X-axis plots the % CPU utilized for each process.

"Cluster of Processes"

Move your mouse over a node in the graph to see its PID and process title:

"PID"

Clicking a node will select it, displaying more information about the process in the sidebar on the right:

"PID Details"

You can use this to get an overview of the following metrics over time for the selected process:

  • Async Activity (Requests, Handles)
  • Host Data (Memory Used, Load Average)
  • Resident Set Size (MB)
  • Heap total (MB)
  • Heap used (MB)

Clicking the “Expand” button on the edge of the sidebar will open the Process Detail view.

Process Detail View

The Process Detail view shows detailed information relating to a specific process.

"Process Detail"

Section Description
Uptime This is the total time elapsed from the moment when the process was started.
Async Activity This chart displays two series that reflect asynchronous activity. One is the number of async handles, which tend to be generated by longer-lived larger-scale asynchronous operations, such as open sockets or timers. The other is the number of async requests. These tend to be made by shorter-lived and smaller-scale operations, such as writing to file handles.
Host Data Host data is a statistic based on the percentage of memory used and the CPU load average over the last minute. For details on the host system running the process, click [🔎 More] underneath the Host Data chart.
Resident Set Size This chart shows the actual memory footprint of the process over time. It does not include memory that belongs to the process that has been swapped out to disk.
Heap Total This is total size of the v8 heap, which includes everything in the heap, including free space which has been allocated by v8 but is unused by JavaScript objects.
Heap Used This is the size of of the v8 heap occupied directly by JavaScript objects created by the application, as well as v8 internal objects created on behalf of it.
Related Snapshots This is a list of heap snapshots associated with this process. When you click [+ New Snapshot], you will be taken to the heap snapshot viewer and a new row representing the heap snapshot will be added to this list.
Related Profiles This is a list of profiles associated with this process. When you click [+ New Profile], you will be taken to the profile viewer and a new row representing the profile will be added to this list.

Host Details

This view shows detailed information about the host system:

  • Hostname,
  • Length of time host system has been running
  • Name of the operating system platform
  • CPU details:
    • Manufacturer
    • Model
    • Clock speed
    • Number of cores

At the very bottom is a graph that shows the total system-wide load over time.

This screen is accessed by clicking on the [🔎 More] link to the center right of the screen.

"Host Details"

N|Solid Quick Start

This guide will help you quickly install all of the N|Solid components onto your development OS X or Linux machine. For instructions on installing the individual components for a production cluster, see the installation guide.

Installing The OS X Developer Bundle

To install the Developer Bundle for OS X:

  1. Go to downloads.nodesource.com
  2. Click on the Mac OS button to get an installable package
  3. Double click on the downloaded pkg file and follow the instructions to install the N|Solid compoments

Installing The Linux Developer Bundle

To install the Developer Bundle for Linux, you must first set up the repository appropriate for your system. Please refer to the "Set Up the NodeSource N|Solid Repository" instructions in the installation guide for instructions on how to do this.

Once the repository is set up, use the following command for an rpm based system as the root user:

$ yum -y install nsolid-developer-bundle

Or, for a deb based system:

$ apt-get -y install nsolid-developer-bundle

These commands will install all of the needed N|Solid components for you.

Setting Up the N|Solid Console

Using the developer bundle on either OS X or Linux is quite easy. To launch the console, open a terminal window and enter the command:

$ nsolid-console

to start the console. Note that you do not have to be the root user to run the console, only to install it. Logs for each process can be found under /tmp/nsolid-*.

Once the Console is running, you can see the user interface by opening a new web browser window to:

$ http://localhost:3000

You should be redirected to http://localhost:3000/welcome, where you will be asked to enter your License Key. (As of version 1.4, in order to access N|Solid you must have an enabled license key.) If you already have and know your License Key, paste it in here. Otherwise click Get License Key at the bottom of the form.

Welcome Screen

Fill out all fields on the Register or Retrieve Key form and a key will be emailed to you. If you have previously set up a key, your existing key will be emailed to you, if not, a new one will be generated and emailed to you.

Register or Retrieve Key

Once you've received your N|Solid license key, paste it into the form found at http://localhost:3000/welcome/license and click SUBMIT.

Enter License Key

Once a valid key is entered, you should be immediately redirected to the N|Solid console.

N|Solid Console Up and Running

Running Node Applications in N|Solid

The next step is to configure your shell to use the N|Solid Runtime.

On OS X, open a second Terminal window and run:

$ source /usr/local/nsolid/nsolid-env

On Linux, open a second Terminal window and run:

$ source /opt/nsolid/nsolid-env

This will activate the N|Solid shell.

From within the N|Solid shell, any Node application that you start from this terminal will then be shown in the N|Solid Console.

For example, monitor the Nodejs REPL using N|Solid by running:

(nsolid) $ NSOLID_APPNAME="REPL" node

If you have an existing Nodejs application, e.g. "index.js", you can monitor it with N|Solid by running:

(nsolid) $ NSOLID_APPNAME="My_Application" node index.js

You can customize the name of your application in the N|Solid Console by changing the NSOLID_APPNAME environmental variable. If you do not specify an app name, your application will be named 'nsolid-default.'

You are now ready to use N|Solid for your own applications!

N|Solid On-Prem in Production

N|Solid Installation Guide

This guide will help you quickly install all of the N|Solid components for a production cluster. We provide production rpm and deb packages for use in Linux distributions. This makes it significantly easier to set up the N|Solid components by using our repositories. We support the following rpm based 64-bit Linux distributions:

  • RHEL6 / CentOS6
  • RHEL7 / CentOS7
  • Fedora 23
  • Fedora 24

We also support the following deb based 64-bit Linux distributions:

  • Debian Wheezy
  • Debian Jessie
  • Debian Sid
  • Ubuntu Precise
  • Ubuntu Trusty
  • Ubuntu Xenial

If you do not wish to use these packages, manual installation instructions are provided below.

Installing N|Solid Packages

Step 1: Set Up the NodeSource N|Solid Repository

For rpm based distributions, use the following command as root:

$ curl -sL http://nsolid-rpm.nodesource.com/nsolid_setup_1.x | bash -

For deb based distributions, there is a similar command run as root:

$ curl -sL http://nsolid-deb.nodesource.com/nsolid_setup_1.x | bash -

In both cases, these scripts will attempt to determine the distribution you are using, set up the appropriate repository on your system, and import the NodeSource N|Solid GPG key used to sign the packages.

Step 2: Install the Packages

For rpm based systems, the command:

$ yum -y install nsolid nsolid-proxy nsolid-console etcd

will install all the required components. You can then turn them on using the standard system tools. Older init style systems, for example, would turn on the proxy component with:

$ /etc/init.d/nsolid-proxy start

whereas newer systemd style systems would use the command:

$ systemctl start nsolid-proxy

For deb based systems, the process is similar. To install the software:

$ apt-get -y install nsolid nsolid-proxy nsolid-console etcd

On deb based systems, the relevant daemons will start automatically on installation, but you will still turn the services on and off using the standard tools as shown above depending on if you have an init based system or a systemd based system.

These packages can be installed separately, and possibly on different machines if you wish. However, if you plan on installing all of the components onto a single machine, the process can be further simplified. Instead of installing the components individually by name as we've done here, you can simply install the nsolid-console-all meta package for your system. For rpm based distributions:

$ yum -y install nsolid-console-all

or for deb based distributions:

$ apt-get -y install nsolid-console-all

This meta package will install all of the components and turn them on if needed.

Upgrading N|Solid Packages

Once you have set up the NodeSource N|Solid package repository appropriate to your system, you will automatically get point releases via the standard system update tools. For example, running:

yum -y update

will fetch and install any new versions of the packages that are available automatically.

Manual Installation

These instructions use the Linux platform builds, but the same steps apply for OS X (darwin) platforms.

Step 1: Get N|Solid

  1. Go to downloads.nodesource.com
  2. Click on the Debian, Ubuntu or RHEL, Fedora, CentOS button for the latest version of N|Solid
  3. Scroll down to the Manual Installation section and click the Download button
  4. Extract the archive

The extracted archive contains all N|Solid components for your architecture:

  1. The N|Solid Runtime (e.g. nsolid-v1.4.0-linux-x64.tar.gz)
  2. The N|Solid Hub Proxy (e.g. nsolid-proxy-v4.0.0.tar.gz)
  3. The N|Solid Hub Registry (e.g. etcd-v2.3.7-linux-amd64.tar.gz)
  4. The N|Solid Console (e.g. nsolid-console-v1.7.3-linux-x64.tar.gz)

Step 2: Install the N|Solid Runtime

The N|Solid Runtime is a build of Node.js that is compiled to include the N|Solid Agent. The runtime should be installed on any application servers where you intend to run your Node.js application.

$ tar xf nsolid-v1.4.0-linux-x64.tar.gz
$ cd nsolid-v1.4.0-linux-x64
$ ls
bin  CHANGELOG.md  include  lib  LICENSE  LICENSE.NSOLID  README.md  share

We suggest installing into /usr/local for greatest compatibility, though most preferred installation methods used for Node.js will also work for N|Solid.

To install to /usr/local:

  1. remove existing npm and nsolid-cli installations:

    $ rm -rf /usr/local/lib/node_modules/npm /usr/local/lib/node_modules/nsolid-cli

  2. put the N|Solid components in place:

    $ cp -r lib bin share /usr/local/

To verify you are running Node compiled with the N|Solid Agent and see your version of N|Solid, use:

$ node -vv
v1.4.0

This package also contains the nsolid-cli application, which can act as a command-line interface to the N|Solid Agent via the N|Solid Hub, allowing scripted or manual interaction.

Step 3: Install the N|Solid Hub Registry Component

N|Solid uses a local etcd database as a service registry for your applications. It stores the locations of all of the running N|Solid Agents in your environment. This enables automatic discovery and monitoring of new processes in your application cluster.

The download bundle includes the latest etcd 2.x release for your platform.

$ tar xf etcd-v2.3.7-linux-amd64.tar.gz
$ cd etcd-v2.3.7-linux-amd64
$ ls
Documentation  etcd  etcdctl  README-etcdctl.md  README.md

To install to /usr/local:

$ cp etcd* /usr/local/bin/

To verify your installation, use:

$ etcd -version
etcd Version: 2.3.7
Git SHA: 2b67f52
Go Version: go1.5.3
Go OS/Arch: linux/amd64

The registry will work in single-node mode or as a cluster.

To start a simple single-node etcd instance:

$ etcd -name nsolid_registry -data-dir /var/opt/etcd \
    -listen-client-urls http://0.0.0.0:2379 \
    -advertise-client-urls http://0.0.0.0:2379

See "Clustering etcd" below for information about running etcd as a cluster.

You will want to run etcd as any other service in your infrastructure, using your process manager of choice.

Step 4: Install the N|Solid Hub Proxy Component

The N|Solid Hub uses a thin Node.js proxy to read from the registry and route communications between the N|Solid Console and the N|Solid Agents. Ideally the proxy is run on the same instance or a nearby instance to the registry.

The download bundle includes the cross-platform nsolid-proxy application.

$ tar xf nsolid-proxy-v4.0.0.tar.gz
$ ls nsolid-proxy-v4.0.0
help.txt  LICENSE  node_modules  package.json  proxy.js  README.md  router.js

Place this folder wherever you typically install your Node.js applications. For this example, we'll put it in /var/opt

# cp -r nsolid-proxy-v4.0.0 /var/opt
# cd /var/opt/nsolid-proxy-v4.0.0

Running npm start from within this directory will start the proxy with the default configuration.

For more information about configuration options for the N|Solid Proxy, see: configuration

You will want to run nsolid-proxy as any other Node.js service in your infrastructure, using your process manager of choice.

Step 5: Install the N|Solid Console

The N|Solid Console is a graphical interface for your entire cluster of applications running on N|Solid. It communicates with each N|Solid Agent via the N|Solid Hub to track metrics, collect information, or trigger events on any of your N|Solid processes.

The N|Solid Console is best experienced using one of the following browsers on a desktop/laptop system.

  • Google Chrome (21 and higher)
  • Mozilla Firefox (22 and higher)
  • Apple Safari (9 and higher)

The download bundle includes the nsolid-console built for your platform.

$ tar xf nsolid-console-v1.7.3-linux-x64.tar.gz
$ ls nsolid-console-v1.7.3-linux-x64
bin  build  LICENSE.txt  node_modules  package.json  README.md  scripts  src  test

Place this folder wherever you typically install your Node.js applications. For this example, we'll put it in /var/opt

# cp -r nsolid-console-v1.7.3 /var/opt
# cd /var/opt/nsolid-console-v1.7.3

Running npm start from within this directory will start the N|Solid Console with the default configuration.

For more information about configuration options for the N|Solid Console, see: configuration

You will want to run nsolid-console as any other Node.js service in your infrastructure, using your process manager of choice.

Upgrading From Previous Versions of N|Solid

To upgrade from previous versions of N|Solid, use the instructions above to install over your previous installation. At each step, verify that the latest version of the component is installed and working.

To ensure continuity of your data, make sure the new installation of the N|Solid Console uses the same storage location configuration as the previous version.

Advanced Topics

Network Considerations

N|Solid supports a wide array of networking options that should work with pretty much any network topology.

N|Solid Agent

The N|Solid Agent listens on a port for incoming connections from the N|Solid Hub proxy. This port can either be specified or configured to use a random port from the ephemeral port range on your operating system. For most systems this should be between 32768 - 61000, though we advise checking your operating system documentation. See configuration for more details on configuring the N|Solid Agent.

The N|Solid Agent must be able to connect to the N|Solid Hub registry port. Best practice is to limit access to the N|Solid Agent ports to only the N|Solid Hub instances.

N|Solid Hub

The N|Solid Hub listens on two ports and must connect to the N|Solid Agents running on your application servers. The listening ports are (by default):

  • Hub proxy: port 9000
  • Hub registry: port 2379

Both ports can be configured to meet your infrastructure requirements.

The N|Solid Hub instances must be able to connect to any N|Solid Agents that are configured to register with those Hub instances. The N|Solid Hub proxy must also be able to connect to the N|Solid Hub registry port.

N|Solid Console

By default, the N|Solid Console listens on port 3000. This port is to be exposed to the browser you intend to use to view the N|Solid Console.

The Console must be able to connect to the N|Solid Hub proxy port, and can be configured to connect to multiple N|Solid Hub instances for better load distribution.

Clustering the N|Solid Hub Registry

For stronger availability concerns, you can cluster the N|Solid Hub registry. The easiest way to do this is by using a discovery service to dynamically configure your cluster. As the current implementation of the N|Solid Hub registry is based on etcd, the instructions below are specific to that service.

CoreOS provides a public discovery server that can be used for this, or you can run your own discovery service. There are multiple ways to configure a clustered etcd setup so we suggest you read the etcd documentation and choose the configuration that works best for your infrastructure.

This example uses the CoreOS public discovery service to bootstrap a 3 node etcd cluster running on three different VMs. Using the CoreOS public discovery service, you first must request a discovery token for the size of cluster you want to create. To create a token for a 3-node cluster:

$ curl https://discovery.etcd.io/new?size=3
https://discovery.etcd.io/2fc76a1c54eff6ff371f89abfbe528ab

The reply is the discovery URL + token for your cluster. You will need this when your nodes start up.

Then for your nodes (three in this case)

$ export MY_IP=10.0.0.10
$ etcd --name nsolid-registry-1 --initial-advertise-peer-urls http://$MY_IP:2380 \
  --listen-peer-urls http://$MY_IP:2380 \
  --listen-client-urls http://$MY_IP:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://$MY_IP:2379 \
  --discovery https://discovery.etcd.io/2fc76a1c54eff6ff371f89abfbe528ab
$ export MY_IP=10.0.0.11
$ etcd --name nsolid-registry-2 --initial-advertise-peer-urls http://$MY_IP:2380 \
  --listen-peer-urls http://$MY_IP:2380 \
  --listen-client-urls http://$MY_IP:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://$MY_IP:2379 \
  --discovery https://discovery.etcd.io/2fc76a1c54eff6ff371f89abfbe528ab
$ export MY_IP=10.0.0.12
$ etcd --name nsolid-registry-3 --initial-advertise-peer-urls http://$MY_IP:2380 \
  --listen-peer-urls http://$MY_IP:2380 \
  --listen-client-urls http://$MY_IP:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://$MY_IP:2379 \
  --discovery https://discovery.etcd.io/2fc76a1c54eff6ff371f89abfbe528ab

To configure your N|Solid Agents to communicate with this cluster:

$ NSOLID_HUB=10.0.0.10:2379,10.0.0.11:2379,10.0.0.12:2379 nsolid app.js

To configure the N|Solid Hub proxy to connect to this cluster, edit the registry entry in the configuration file for the proxy .nsolid-proxyrc:

  "registry": "10.0.0.10:2379,10.0.0.11:2379,10.0.0.12:2379",

For more information, check out the official clustering guide for etcd: https://coreos.com/etcd/docs/latest/clustering.html

Configuring N|Solid

N|Solid Runtime

Environment Variables

Environment Variable Description
NSOLID_APPNAME This is the name of the application that the instance is running. Use this in order to create logical groupings of the processes in the Console. If omitted, the value defaults to nsolid-default.
NSOLID_HUB This is the route to your Hub’s Service Registry. It accepts host and/or port. The host can be provided via several formats:

IPv4 10.0.0.21
IPv6 [2001:cdba::3257:9652]
hostname nsolid_hub.local
NSOLID_SOCKET This is the address that your instance's agent will be listening on for queries. It accepts an address or hostname bound by an interface on this machine and an optional specified port. If no port is specified, or port 0 is specified (e.g. NSOLID_SOCKET=localhost:0) the agent will request an open port from the operating system's ephemeral port range.
NSOLID_TAGS This is the list of tags associated with your instance, which can be used to identify and filter instances in Console views. See Tags and Filters for more details.

Note, if the NSOLID_HUB environment variable is set, and the NSOLID_SOCKET environment variable is not set, the N|Solid Agent behave as if NSOLID_SOCKET was set to 0.0.0.0:0. The NSOLID_SOCKET environment variable will only need to be explicitly set in this case, to force the socket to be bound to a specific network interface or port.

To enable the N|Solid Agent, you must either specify a value for NSOLID_HUB and/or NSOLID_SOCKET in your environment for each process that runs under N|Solid.

Tip

We suggest allowing the N|Solid Agent to select the port number. This provides greater compatibility with modules such as cluster and will be automatically discovered correctly when using the N|Solid Hub.

N|Solid CLI

Configuration File

The N|Solid Command Line Interface (CLI) can be configured via a configuration file, .nsolid-clirc.

{
  "hub": "http://localhost:9000",
  "app": "my-node-application"
}

Broadcast vs Non-broadcast Query

Broadcast queries are the default operation of the CLI. You can use nsolid-cli --id in order to change to a non-broadcast query.

Any operation except for ls needs to be approved for broadcast. This is to prevent extraneous traffic and accidental load from performing large operations. In your .nsolid-proxyrc add the names of any commands you wish to be available for broadcast queries. If you create custom commands that you wish to be able to broadcast query you will need to add them to the broadcast list in your .nsolid-proxyrc file.

N|Solid Hub

The N|Solid Hub Proxy is the component that allows remote communication to be forwarded or broadcast to your processes via the N|Solid Console or N|Solid CLI.

Command Line Arguments

These are the command line arguments if you're starting the hub proxy explicitly.

Parameter Description
--port=<port> The optional host and required port the proxy will listen on for requests.
--registry=<registry list> A comma delimited list of host and/or ports for etcd servers that the proxy will query, to obtain the list of instances.

Configuration File

The N|Solid Hub Proxy can be configured via a configuration file, .nsolid-proxyrc.

{
  "registry": "localhost:4001",
  "port": 9000,
  "denied": [],
  "broadcast_approved": [
    "ping",
    "process_stats",
    "system_stats",
    "system_info",
    "info",
    "versions",
    "startup_times"
  ]
}
Property Description
registry A comma delimited list of host and/or ports for etcd servers that the proxy will query, to obtain the list of instances.
port The optional host and required port the proxy will listen on for requests.
denied An array of agent commands that will not be passed to instances.
broadcast_approved An array of agent commands that can be broadcast to multiple instances simultaneously.

The registry value may also be set in the REGISTRY environment variable.

The host and/or ports for the registry and port values should be in one of the following form: host:port. For registry elements, you can leave off the port, and it will default to the default etcd client port, 2379.

The host may be specified as a hostname, or as an ip address.

N|Solid Console

Command Line Arguments

These are the command line arguments if you're starting the console explicitly. For most usages, you would utilize npm start.

Parameter Description
--config=<path> Configuration file location
--hub Space separated list of N|Solid Hub(s) (e.g., --hub http://localhost:9000 http://domain.tld:9000)
--interface=<ip> Networking interface HTTP server will listen on (default: "0.0.0.0")
--port=<port> Specify which TCP port to listen on (default: 3000)
--interval=<milliseconds> How frequently should the collector poll the hub for metrics (default: 5000)
--storage=<path> Where things like heap snapshots and CPU profiles are stored
--duration=<minutes> The length of time to retain graph data in minutes. (default: 15)

Storage and Configuration default locations

By default the console will create a storage directory at

  • OSX ~/Library/Application\ Support/nsolid-console/
  • Linux ~/.config/nsolid-console/

If --config is not specified, a configuration file will also be created (<storage directory>/config).

Moving the Console Instance

Simply copy the directories specified by the --storage and --config command line arguments on the source host to the new host. Be sure to specify the new storage directory with --storage and the new config directory with --config on the new host.

You should restart the Console any time you modify the location of the storage or config directory.

Stats Retention

Collected metrics are stored on disk (<storage directory>/graphs.db) along with heap snapshots (<storage directory>/snapshots) and CPU profiles (<storage directory>/profiles). Ensure ample disk space as each heap snapshot may be on the order of hundreds of megabytes to gigabytes.

Historical metrics, displayed in the line-chart UI elements, are trimmed to the last 15 minutes by default. If you would like to show more historical data in these graphs, specify a larger --duration value when launching nsolid-console

Note: large --duration values will utilize more CPU and memory and may have adverse effects on the performance of the N|Solid Console.

Configuration File

The N|Solid Console can be configured via a configuration file specified by --config.

{
    "baseUrl": "http://localhost:3000/",
    "hub": ["http://localhost:9000"],
    "interval": 5000,
    "port": 3000,
    "securityinterval": 1800000,
    "storage": "/path/to/storage/directory",
    "timeout": 10000
}
Property Description
baseUrl Url used to access the installed N|Solid console. This is used for linking back to artifacts such as heap snapshots cpu profiles from threshold notification emails. (default: "http://localhost:3000/")
hub An array of N|Solid hub urls (default: ["http://localhost:9000"])
interface The networking interface on which the N|Solid console will listen on (default: 0.0.0.0)
interval How frequently should the collector poll the hub for metrics (default: 5000)
port Specifies which TCP port to listen on (default: 3000)
storage Where things like heap snapshots and CPU profiles are stored
timeout Maximum amount of time that a request to the N|Solid Hub can take in ms (default: 10000)

Note: each property in the configuration file may be overridden by command line arguments. The inverse is also true, instead of providing command line arguments a pre-baked configuration file may be used.

Using the N|Solid Console Logs

The N|Solid Console logs are output to the terminal and can be redirected to a file of your choosing. If errors are encountered while using the console, these logs can be used to help identify/fix the problem when conversing with N|Support. During N|Support correspondence, any error messages in your browser's console should also be included to provide as much context about your problem as possible.

For more granular logging, provide a DEBUG=* environment variable when launching the N|Solid Console.

DEBUG=* node bin/nsolid-console.js --interval=1000 --storage=/path/to/dir

Collecting Metrics From Multiple Networks

Each network should have its own N|Solid Runtime, Hub Proxy and Hub Service Registry processes. In order to collect metrics from multiple proxies, use an array of urls in the N|Solid Console configuration file:

{
  "hub": ["http://host.one:9000", "http://host.two:9000"]
}

or pass a space delimited list of urls as the --hub command line argument when executing nsolid-console:

DEBUG=* node bin/nsolid-console.js --hub http://host.one:9000 http://host.two:9000

License Keys

Registering Your License Key

In order to access N|Solid you must have an enabled license key. If you have not entered your license key, when you try and access the console you will be redirected to the welcome screen. If you already have your License Key, paste it in the license key field. Otherwise click Get License Key at the bottom of the form.

Welcome Screen

Fill out all of the fields on the Register or Retrieve Key form and a license key will be sent to you via email:

  • If you have previously set up a license key, you will receive your existing key
  • If you have not previously set up a license key, a new license key will be created

Register or Retrieve Key

Once you've received your N|Solid license key, paste it into the Welcome screen form and click SUBMIT.

Once a valid key is entered, you should be immediately redirected to the N|Solid console.

Updating Your License Key

If you need to update your license key from your original demo key (or for any other reason), click on the menu in the upper-left corner next to the N|Solid logo and select License Key.

Next, paste in your updated N|Solid license. Then click SAVE CHANGES.

Your updated license key should now be active.

Troubleshooting

Debug Messages

The N|Solid Runtime, Hub Proxy and CLI all can have debug messaging enabled by using the NODE_DEBUG=nsolid environment variable.

$ NODE_DEBUG=nsolid NSOLID_SOCKET=8000 node app.js
NSOLID 30599: registering default commands
NSOLID 30599: nsolid initializing
NSOLID 30599: nsolid initialized on port 8000

Network Access Considerations

The N|Solid Console and CLI use the N|Solid Hub to query for metrics. The host and port that the Hub is configured to listen on must be accessible to these clients.

Similarly, when the N|Solid Hub communicates with an N|Solid instance, the host and port for the instance must be accessible. Without this route, there will be no way to query specific data about the instance. This is a bi-directional route as the N|Solid instance actively communicates back to the Hub.

Using N|Solid with Docker

For people who need to use N|Solid in a Dockerized environment, NodeSource has developed the N|Solid Docker images.

These images are built with the enterprise customer in mind, developing a versatile set of independently scalable Docker images that work together to provide a unified N|Solid experience across all of your Dockerized infrastructure.

In addition to being friendly to enterprise operations teams, the N|Solid Docker Images have been designed to be accessible to developers. For those who are already using Docker, these images provide an easy way to get up and running with the N|Solid console. With the use of docker-compose, you can now get the full N|Solid Console on your local machine with a single command: docker-compose up.

Overview of N|Solid Docker Images

"N|Solid Hub"

We provide 5 separate Docker Images for N|Solid, allowing each component of the platform to be scaled and deployed independently.

  • docker pull nodesource/nsolid:v1.4.2
    • the base image that your Docker images should build from. It provides the N|Solid runtime and agent.
  • docker pull nodesource/nsolid-cli
    • provides an easy way to query statistics from your Node.js processes. It is meant to be run directly from the command line.
  • docker pull nodesource/nsolid-registry
    • contains the N|Solid Hub service registry. Your Node.js applications will need to be able to talk to this.
  • docker pull nodesource/nsolid-hub:v4.0.0
    • contains the N|Solid Hub proxy service, which is used by the N|Solid Console to query runtime statistics from your Node.js applications. This container will need to talk to both the registry and your Node.js applications
  • docker pull nodesource/nsolid-console:v1.7.3
    • contains the N|Solid console, a web based interface for monitoring your Node.js applications. You will need to bind the internal port 3000 to your host. This container will need to be able to talk to the hub container.

All of the images are built on the official ubuntu:trustyimage maintained by the Docker project.

Installing the N|Solid Docker Images

Setting up Docker

First, follow the steps for installing Docker for your operating system, we recommend you use at least Docker v1.9: https://docs.docker.com/engine/installation/

Although not required, it is recommended that you use docker-compose when first getting started. It simplifies managing these Docker images when running them locally. If you choose to use docker-compose, follow the appropriate installation guide for your operating system: https://docs.docker.com/compose/install/

Getting the Docker-Compose.yml file

If you plan to use docker-compose, you will want to download the docker-compose.yml file and save it in an empty directory under the name nsolid.yml

Downloading the Docker Images

If you are using docker-compose, open a terminal in the folder you downloaded nsolid.yml to, and run the following command:

docker-compose -f nsolid.yml pull

This command will download all of the images needed to run the N|Solid console locally. If you choose not to use docker-compose, execute the following commands:

docker pull nodesource/nsolid-registry
docker pull nodesource/nsolid-console
docker pull nodesource/nsolid-hub

Using the N|Solid Docker Images

Creating A Shared Network

The N|Solid Docker images require a level of service discovery to work. In Docker, this means that the containers must be running on the same network stack. This can be accomplished by way of a shared network.The process of creating a shared network differs depending on your Docker version

  • For Docker < v1.9, use the (deprecated) --link flag
  • For Docker 1.9+, create a shared network for your containers with the command:
docker network create nsolid

Starting the N|Solid Console

To bring up the N|Solid Console at this point, run the following command:

docker-compose -f nsolid.yml up

By default, this will bring up the N|Solid Console running on 127.0.0.1:3000, and the console will be monitoring itself.

If you are not using docker-compose, you will need to start 3 containers to run the console:

docker run -d --net nsolid -p 2379:2379 nodesource/nsolid-registry
docker run -d --net nsolid -e ‘REGISTRY=registry:2379’ -e ‘NODE_DEBUG=nsolid’ nodesource/nsolid-hub
docker run -d --net nsolid -p 3000:3000 -e ‘NSOLID_APPNAME=console’ -e ‘NSOLID_HUB=registry:2379’ -e ‘NSOLID_DEBUG=nsolid’ nodesource/nsolid-console --hub hub:9000

This will bring up the N|Solid console available on localhost:3000 and monitoring itself by default.

Adding Your Own Application

First, ensure your application is compatible with the LTS release of Node.js. NodeSource has created upgrade-ready to assist with this.

If you are new to Docker, follow the steps in our blog post to get your application into a Docker container.

Create a folder in the directory named after your image, and place your Dockerfile and source files in it. This documentation will refer to the service as example and the folder will be ./example. Your Dockerfile should begin with the following line:

FROM nodesource/nsolid:v1.4.2

Starting Your Application with Docker Compose

Next, create the file docker-compose.yml in the directory along-side nsolid.yml:

example:
  build: ./example
  net: “nsolid”
  environment:
    - NSOLID_APPNAME=example
    - NSOLID_HUB=registry:2379

For the complete documentation on defining a service with docker-compose.yml, refer to the Docker project’s documentation page: https://docs.docker.com/compose/overview/.

At this point, you are ready to bring up your application using docker-compose

docker-compose -f docker-compose.yml up

Starting Your Application without Docker Compose

After building your service with:

docker build -t example .

Start your service with:

docker run -it --net nsolid -e ‘NSOLID_APPNAME=example’ -e ‘NSOLID_HUB=registry:2379’ example

Congratulations, you are now up and running with N|Solid and Docker!

A Note on Multi-Host Deployments

Using Docker with N|Solid introduces some networking edge cases that can make multihost deploys challenging. These images have been developed specifically to address this. When used correctly, these images should seamlessly worked together across multiple hosts.

By default, Docker will create an isolated network on the host that the container binds to. Then, Docker configures a mapping from the port’s on the containers network to a port on the host’s network per the configuration you pass at runtime.

This introduces the case where, by default, the IP address that the N|Solid agent sees from inside the container is not the same IP address that the N|Solid hub needs to communicate with the agent.

There are a few methods to address this in Docker, and they all boil down to making sure the IP address that the hub needs, and the IP address that the agent sees, are the same.

Using the Host’s Network Stack

If you are using containers to isolate resources and not to run untrusted code, it is safe to use docker run --net=host to allow the container to bind directly to the host’s ports. This is the most direct way of getting the hub to talk to the Dockerized N|Solid agent.

Security

This allows the container to bind directly to the host’s ports, including low-numbered​ ports. If you are running untrusted code, this is a security risk and should be avoided.

Using Docker Swarm

When using Docker Swarm with Docker v1.9 or later, you are able to use the overlay driver for docker network. This allows all of the Docker containers to share the same network with DNS resolution built in. To get started with this, first provision a new machine that will host the key-value store by Docker Swarm for discovery

docker-machine create -d virtualbox nsolidkv

Machine Drivers

Note: In addition to Virtualbox, docker-machine supports drivers for various cloud providers. View the full list of cloud providers and how to use them with docker-machine here.

Next we will deploy consul to our new host:

$ eval "$(docker-machine env nsolidkv)"
$ docker run -d -p 8500:8500 -h consul progrium/consul -server -bootstrap
$ eval "$(docker-machine env -u nsolidkv)"

Troubleshooting

If you see client is newer than the server you will need to run docker-machine upgrade nsolidkv

Next, we will provision our swarm master:

$ docker-machine create -d virtualbox --swarm --swarm-master --swarm-discovery="consul://$(docker-machine ip nsolidkv):8500" --engine-opt="cluster-store=consul://$(docker-machine ip nsolidkv):8500" --engine-opt="cluster-advertise=eth1:2376" nsolid-master

We will now create one or more slave nodes for our swarm cluster. Re-run the following command as many times as you like (incrementing the node id) to generate slaves.

$ docker-machine create -d virtualbox --swarm --swarm-discovery="consul://$(docker-machine ip nsolidkv):8500" --engine-opt="cluster-store=consul://$(docker-machine ip nsolidkv):8500" -engine-opt="cluster-advertise=eth1:2376" nsolid-n1

Then, verify that the machines have been provisioned:

$ docker-machine ls
NAME            ACTIVE   DRIVER       STATE     URL                         SWARM
nsolid-master   -        virtualbox   Running   tcp://192.168.99.101:2376   nsolid-master (master)
nsolid-n1       -        virtualbox   Running   tcp://192.168.99.102:2376   nsolid-master
nsolid-n2       -        virtualbox   Running   tcp://192.168.99.103:2376   nsolid-master
nsolidkv        -        virtualbox   Running   tcp://192.168.99.100:2376

Finally, create an overlay network on the swarm.

$ eval "$(docker-machine env --swarm nsolid-master)"
$ docker network create -d overlay nsolid

At this point, any container you deploy to the swarm will support DNS resolution by container name, and the IP address that the N|Solid agent sees will be the same IP address the hub container needs.

Using N|Solid On-Prem

CPU Profiling

CPU Profiling can be used to indicate all the functions on the function call stack, when samples are taken. For instance, if a function foo() called a function bar() during its execution, and a sample was taken while bar() was running, the function call stack will show that foo() called bar(). Because multiple samples may be taken while bar() is executing, there will be an approximate start and stop time recorded for bar(), which is an indication of how long it took to run. In addition, further samples before and after the ones that captured bar() will capture foo(), and likewise to the bottom of the function call stack - typically, a callback of some sort.

This data can then be analyzed to show, for the function foo(), how much time was actually spent in foo() and not in bar() - every function has two time values - a self time and a total time. For the foo() and bar() case, if foo() only calls the bar() function, then the self time for foo() + the total time for bar() will equal the “total” time for foo().

Example

function foo() {
        … processing that takes a lot of time but calls no other functions ...
    bar()
        … processing that takes a lot of time but calls no other functions …
}

function bar() {
    …
}

foo’s total time = foo’s self time + bar’s total time.

Both values are interesting; total time shows you which functions are fastest and slowest, from start to finish, but doesn’t tell you if the time was spent in that function or other functions.

Named Functions

Named functions are easier to spot in CPU profiles. The stack frame entries available in a CPU profile include the name of function, and source code location information for detail views. For anonymous functions, the name will often be displayed as "(anonymous)". In some cases, the V8 JavaScript engine can calculating an "inferred name" for a function. When in doubt, name your functions so that you can easily spot them in a CPU profile.

For instance, let's say you have a function busyFunction(), which you like to easily track the callers for, and you have cases where it's called from an anonymous function, like this:

setInterval(function(){busyFunction()}, 1000)

In a CPU profile, you'll see that busyFunction() is being called by(anonymous) in this case.

To make this easier to easier to spot in a CPU profile, you can simply use:

setInterval(function busyFunctionCaller(){busyFunction()}, 1000)

In the CPU profile, you'll now see that busyFunction() is called by busyFunctionCaller().

For additional code cleanliness, and less chance of creating a "pyramid of doom", consider moving the entire function out into the same scope as the function usage; for instance:

setInterval(busyFunctionCaller, 1000)
//...
function busyFunctionCaller() {
  busyFunction()
}

Because JavaScript functions are "hoisted" to the top level of the scope they're defined in, you can reference busyFunctionCalled before it's actually defined.

CPU Profiling allows you to understand where opportunities exist to improve the speed and load capacity of your node process. N|Solid provides multiple ways to profile your CPU.

Analyzing Profiles Using the N|Solid Console

The N|Solid Console allows you to take and analyze profiles in one action. This is particularly useful if you need to take a series of profiles as it saves you the overhead of switching from one environment to another. The console also saves a history of profiles so you can quickly flip through profiles to identify troublesome areas.

  1. Launch the console and select the application of interest "All applications"
  2. Select the process of interest "Select process"
  3. Expand out the details panel and select ‘+ New Profile’
    "New profile"
  4. Select your profile window (5 to 60 seconds) and run profile "Run profile"
  5. Select a visualization (Sunburst, Flamegraph or Treemap). Once a visualization is selected, you can click on it to view the call stack to the right. The number of calls to that method are shown as are the time spent within it. Clicking on the method will show you the file responsible for that call. "Call stack"
  6. Profiles are saved in your console session so you can easily flip between profiles.

Visualizations Explained

All of the visualizations available for profile data show the unique set of stack traces captured during the profile, with the "area" of the stack indicating the proportional time spent in a function compared to that of its parent.

The Flame Graph visualization shows the time along the x-axis. The y-axis is used to show the function calls that make up a particular stack trace.

The Sunburst visualization is like the Flame Graph, where the x-axis is curved into a circle. Stack traces grow from the center to the outer parts of the graph.

The Treemap visualization shows time by area. The larger the rectangle, the more time a function took. The stack traces grow from the outer boxes into the inner boxes.

You can use the visualizations to find long-running functions. These can be identified by:

  • wide rectangles in the Flame Graph
  • long radial sections in the Sunburst
  • large areas in the Treemap

The stack trace height - how many functions deep a particular stack trace was - doesn't indicate a time issue, necessarily. You'll want to focus on the time values.

As you hover over the visualization itself, the method tied to the visualization will be show. If you click, you'll see a stack trace to the right of the visualization. Other areas in the visualization may "light up" with a less bright color as you hover over the visualization - these are entries in stack traces for the same function and indicates additional CPU usage for the function under the cursor.

So, the things to look for are stack trace entries that take up the most area: width in Flame Graph, circumference in Sunburst, and area in TreeMap.

Using the Chrome Profiler with N|Solid

You can also view the generated CPU profile in Chrome Dev Tools:

  • Click the Download Profile button on the left side of the screen
  • Open the Chrome web browser
  • Open Chrome Dev Tools from the menu View / Developer / Developer Tools
  • When the Developer Tools window (or pane) opens, select the tab Profiles
  • Click the Load button in the Select profiling type area
  • Find the .cpuprofile file that was downloaded, and click the Open button
  • Click on the profile that should now appear under CPU PROFILES list at the left
  • Click on Chart in the view selector under the Chrome Dev Tools menu
  • This will display a chart showing function execution times and what functions were on the stack at the time the system was sampled

Profiling Using the Command Line Interface (CLI)

N|Solid's Command Line Interface (CLI) is a great way to quickly pull profiles from remote processes for local examination. You can learn more about the profile_start and profile_stop commands in the N|Solid Command Line Interface (CLI) reference.

Tags and Filters

Tags

You can use the NSOLID_TAGS environment variable to add tags that will be applied to a process. This can aid in identification and filtering, particularly in the Cluster view of the N|Solid Console. Tags can also be retrieved using the info command in the N|Solid CLI.

Setting Tags

$ nsolid NSOLID_APPNAME="leaderboard" NSOLID_TAGS="DC, us-east-1"

Retrieving Tags

$ nsolid-cli -- socket XXXX info

Example JSON Response

{
  "meta": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "hostname": "cluster-1.oregon",
    "address": "192.168.2.27:38580"
  },
  "reply": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "nodeEnv":"production",
    "execPath": "/usr/local/bin/nsolid",
    "main": "/var/myapp/web/server.js",
    "tags": [ "DC", "us-east-1" ]
 }
}

Tag Constraints

  • Tags can contain any whitespace characters
  • Tags are trimmed, so only whitespace in the middle of a tag is kept intact
  • Tags cannot contain commas and commas cannot be escaped
  • Tags are transmitted exactly the way they are entered and therefore are case sensitive, i.e. world is different than wOrld
  • The byte length of a tag must be between 2 and 140
  • There is no minimum number of tags per process
  • There is a maximum of 20 tags per process allowed
  • Tags cannot be changed at runtime

Filtering

As many applications are comprised of tens, if not hundreds or thousands, of processes it is often necessary to be able to filter a subset based on some criteria. The N|Solid Console offers a filtering mechanism with auto-complete to help you navigate your complex applications.

"Filter Search"

Filters

Typing just the filter name will list all of the items for that filter. Subsequent characters are then used to find partial matches within that filter.

"Supported Filters"

Here are the currently supported filters in the N|Solid Console:

Filter Name Description
hub Hub(s) for the selected application's processes
host Hostname(s) for the selected application's processes
name Application name as specified by NSOLID_APPNAME
pid Process IDs (pids)
tag Custom process tags (if any) as specified by NSOLID_TAGS
title Process title(s)

You may enter multiple filters and search terms in the filter box. Multiple terms will be logically AND'ed together when they are applied to the set of processes being filtered.

Heap Snapshots

If you have a memory leak or performance issue, taking heap snapshots is a great way to help identify the underlying problem. N|Solid provides you with two ways to do this, a command-line friendly method and via our console.

Analyzing Heap Snapshots

Heap Snapshots capture a record of all live JavaScript objects available in your Node application, at the time the snapshot is captured. The objects are grouped by their constructor name, with aggregated counts of the total number of objects, their shallow size, and their retained size as columns in the Constructors view.

Shallow Size indicates the amount of space the object uses for its own properties. For instance, an object with no properties will have a very small shallow size, where an object with a number of properties will have a larger shallow size.

Retained Size indicates the amount of space the object is referencing, and keeping from being garbage collected.

From the Constructors view of the snapshot, you can sort by constructor name, number of objects, shallow size and retained size. You can also filter the list by constructor name.

You can use the number of objects value to look for the objects that are leaking, and you can use the retained size value to look for objects referencing those leaking objects.

In cases where you are leaking objects - and thus consuming more memory than you should be - sorting by number of objects and by retained size can provide a good indication of where to look for the cause of the leak. A large number of objects of a particular type may be an indication that these objects are still being referenced by other objects, when they could be garbage collected. A large retained size may be an indication that these objects are referencing more objects than they should be, keeping them from being garbage collected.

Taking Heap Snapshots with the N|Solid Console

Using N|Solid's Console, developers to take a series of snapshots and quickly switch between them to better identify problematic areas.

  1. Launch the console and select the application of interest "All applications"
  2. Select the process of interest "Select process"
  3. Expand out the details panel and select ‘+ New Snapshot’
    "New profile" This will take the snapshot and open the results in the snapshot window. To retake the snapshot, click the 'New Snapshot' button. The 'Related Snapshots' section in the previous screen provides a record of all the snapshots in the session. "Retake snapshot"
  4. Once the snapshot completes you can navigate through the various objects indexed during the snapshot process. As there can be a large number of indexed objects, you can use the Filter search box to narrow the results. Below you can see a listing of the various Array objects found in memory. To take another snapshot, simply press the New Snapshot button to the left. "Filter" Snapshots taken in the N|Solid Console can be downloaded to your local drive for exploration via a 3rd party tool (i.e. Chrome Developer Tools). If you wish to download the profile for analysis through other tools, simply click the Download Snapshot button "Download snapshot"

Taking Snapshots from the Command Line Interface (CLI)

N|Solid's Command Line Interface (CLI) is a great way to quickly take heap snapshots from remote processes for local examination. You can learn more about the snapshot command in the N|Solid Command Line Interface (CLI) reference.

Best Practices

Using constructors will show class names in heap snapshots. Since the heap snapshot groups objects by constructor name, if at all possible you should use named constructors for objects you'd like to track, as opposed to using literal objects. When using literal objects for trackable objects, those objects will be aggregated under the Object constructor name, along with all the other literal objects your program uses.

For instance, the following code snippet creates objects which will be categorized under the Object constructor:

trackableObject = {x: "some value", y: "another value"}

To be able to track these objects in a snapshot, use a specifically named class:

trackableObject = new TrackableObject()
trackableObject.x = "some value"
trackableObject.y = "another value"
// ...
function TrackableObject() {}

You'll then see these objects grouped by themselves in the heap snapshot.

If you'd like a bit more of an "inline" feel, you can enhance your constructor to take initialization values:

trackableObject = new TrackableObject({x: "some value", y: "another value"})
// ...
function TrackableObject(initialValue) {
  this.x = initialValue.x
  this.y = initialValue.y

With EcmaScript 6 parameter destructuring, you can make it a little shorter:

trackableObject = new TrackableObject({x: "some value", y: "another value"})
// ...
function TrackableObject({x, y}) {
  this.x = x
  this.y = y

You can also make the constructor open-ended regarding the properties:

trackableObject = new TrackableObject({x: "some value", y: "another value"})
// ...
function TrackableObject(initialValue) {
  for (var key in initialValue) {
    this[key] = initialValue[key]
  }
}

Threshold Alerts

Threshold Alerts allow you to instruct N|Solid to take certain actions, such as taking heap snapshots and/or CPU profiles, when one or more processes cross over performance thresholds that you set.

Often, performance problems occur in production code non-deterministically and the only information you have available to diagnose the situation is in an error log or possibly a stack trace. With N|Solid's threshold alerts, you can automatically capture more useful information when problems arise in production.

Setting Threshold Alerts with the N|Solid Console

  1. From the main application page, click on an application to enter the cluster view. Click on Threshold Settings, located near the upper-right corner of the page.

  2. You have two types of thresholds to choose: Heap Used and CPU Used. You may configure actions independently in any of the categories.

  1. To enable a threshold, click the switch immediately below the ENABLE THRESHOLD label.

  1. Depending on the threshold you selected, drag either the maximum heap percentage or maximum CPU percentage slider immediately below the switch. Set this value to the ceiling you do not wish your application's processes to exceed during normal operation. When enabled, there will be a dotted red line in the corresponding graphs that indicates where the threshold is set.

  1. Next, set the threshold delay. This setting allows you to prevent short blips of activity, which do not necessarily indicate an issue, from triggering a threshold at all.

  1. As threshold actions can themselves be resource intensive, the threshold snooze setting ensures that the performance of your application does not suffer due to excessive profiles or snapshots and that your inbox does not become clogged with repeated notifications.

Note: When a threshold is set, a draggable bar for that threshold will appear in the cluster view. To change its value, click and drag the square handle at the center of the bar. Double-click the aforementioned square handle to open up the Threshold Settings panel.

  1. Now that you have configured your threshold parameters, select one or more actions that N|Solid will perform when a threshold is triggered. Your choices are:

    1. Initiate CPU Profile: Select this option if you suspect that your application suffers from high CPU utilization. When enabled, N|Solid will start a CPU profile on your behalf and create an entry in the CPU profiles list in the process view.

  1. Take Heap Snapshot: Select this option if you suspect a memory leak. When enabled, N|Solid will take a heap snapshot on your behalf and create an entry in the memory profiles list in the process view.
  2. Send a Notification: Select this option to have alerts sent to your inbox. You may enter more than a single address at a time by separating them with a comma.

  1. Finally, click the Update button to save your settings.

Note: Be careful about setting a very high threshold, such as 99% Heap, and then selecting an action that uses the same resource, such as Take Heap Snapshot. Taking an intensive action at a very high ceiling is likely to negatively impact your application.

Security Vulnerabilities

The N|Solid Console can bet set up to perform periodic verification of all packages loaded by all N|Solid processes. All loaded packages are verified against a list of known vulnerabilities and checked for local modifications.

When new vulnerabilities are found, information about each vulnerability will be reported in the Console and optionally send notifications via email.

Detecting Security Vulnerabilities with the N|Solid Console

  1. Navigate to the Security view via the main menu.

    image

  2. This displays the Security view with a list of all the vulnerabilities across all applications in the left-hand sidebar.

    image

    1. Alternatively, you can narrow your search to a single application by going to the Applications view. Applications with identified vulnerabilities are highlighted in red.

      image

    2. Click on the View button for the selected vulnerable application.

      image

    3. This displays the Security view with a filtered list of all the vulnerabilities affecting the selected application in the left-hand sidebar.

      image

  3. Select the vulnerability you're interested in from the list on the left.

    image

  4. The applications and processes that are affected by the selected vulnerability are listed on the right-hand side.

    image

Displaying Affected Packages in a Process

  1. Navigate to the Process Detail page for a vulnerable process.

    image

  2. Click the VIEW REPORT button in top-left-hand corner.

    image

  3. You will be taken to the report page for this process where a list of the affected packages is displayed.

    image

  4. Vulnerable packages that have been modified will be highlighted with a special icon.

    image

Receiving Notifications About New Vulnerabilities

When a new vulnerability is found it is initially unread. Unread vulnerabilities are highlighted with red dots in the main menu.

image

Marking Notifications As Read

You can signal that you've read a notification, and it will no longer appear as a red dot.

  1. Navigate to the Security view.

    image

  2. To mark a notification as read for an application, click Mark As Read.

    image

    image

  3. To mark all notifications as read for a vulnerability across all currently affected applications, click Mark All As Read.

    image

  4. Once you've marked all vulnerabilities for all applications as read, the notification bubble will disappear.

    image

Receiving Email Notifications for Security Vulnerabilities

  1. To send an email notification to one or more recipients when a vulnerability is detected, use the Notification Settings pane on the Security view. Access this panel by clicking the Notification Settings icon in the top-right hand corner.

    image

  2. Enter one or more mail addresses for the people you want to be notified when new vulnerabilities are discovered. Each email recipient will be notified only once about any particular vulnerability. All existing vulnerabilities will be reported to any new email addresses added to this list.

    image

  3. Click Update to Save.

Manually Triggering A Security Update

  1. Navigate to the Security view via the main menu.

    image

  2. This displays the Security view with a list of all the vulnerabilities across all applications in the left-hand sidebar.

    image

  3. In the navigation bar, you will see the time of the last update.

    image

  4. You can manually trigger a security check for all processes across all applications by clicking on the Update button.

    image

Advanced Topics

N|Solid Command Line Interface (CLI)

The N|Solid Agent comes with a set of pre-defined endpoints for interaction and introspection and can be customized to add additional endpoints.

Some commands accept input from the command line and can be supplied via:

nsolid-cli --data '{"json":"object"}' ...

To see a list of all commands and options available with nsolid-cli, use the -h option:

nsolid-cli -h

The output of the nsolid-cli commands, unless otherwise noted, is lines of JSON, each line separated by a newline delimiter (\n), where each line is the output from a single N|Solid instance. The output shown in the examples below is shown expanded, for readability.

ls

Returns all of the process information currently stored in the service registry.

Usage

nsolid-cli ls

Example JSON Result

{
   "pid":86124,
   "hostname": "vm.local",
   "app":"nsolid-default",
   "address":"192.168.0.10:52602",
   "id":"f46ff6c3e27f30338ae0a6cc15bbd7376a8e964d"
}
{
   "pid":86120,
   "hostname":"vm.local",
   "app":"foo",
   "address":"192.168.0.10:52550",
   "id":"92357fe6fce23fc8dd732f105c10ca279596a052"
}

Usage

nsolid-cli --app nsolid-default ls

Example JSON Result

{
  "pid":3691,
  "hostname":"dc1mv",
  "app":"nsolid-default",
  "address":"192.168.1.104:35770",
  "id":"e55ceb733a01be8cb0803c732df685efff5a56ee"
}

ping

This is the agent's availability check, it will always reply with "PONG" if online.

Usage

nsolid-cli ping

Example JSON Result

"PONG"

info

Usage

nsolid-cli info

Returns information about the running process and N|Solid Agent configuration:

Property Description
id The execution id for this run instance
app The NSOLID_APPNAME
pid The process id
execPath Path of the executable running the application
main The main module used when the application started up

Example JSON Result

{
  "meta": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "hostname": "cluster-1.oregon",
    "address": "192.168.2.27:38580"
  },
  "reply": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "nodeEnv":"production",
    "execPath": "/usr/local/bin/nsolid",
    "main": "/var/myapp/web/server.js"
  }
}
{
  "meta": {
    "id": "ccd48968f3212a06c3aff62bbfa75c481ad5f672",
    "app": "api-server",
    "pid": 5447,
    "hostname": "cluster-1.oregon",
    "address": "192.168.2.27:52352"
  },
  "reply": {
    "id": "ccd48968f3212a06c3aff62bbfa75c481ad5f672",
    "app": "api-server",
    "pid": 5447,
    "nodeEnv":"production",
    "execPath": "/usr/local/bin/nsolid",
    "main": "/var/myapp/api/server.js"
  }
}

versions

Returns the versions of bundled components that make up Node.js and N|Solid.

Usage

nsolid-cli versions

Example JSON Result

{
  "http_parser": "2.5.2",
  "node": "4.4.3",
  "nsolid": "1.3.0",
  "v8": "4.5.103.35",
  "uv": "1.8.0",
  "zlib": "1.2.8",
  "ares": "1.10.1-DEV",
  "icu": "56.1",
  "modules": "46",
  "openssl": "1.0.2g",
  "nsolid_lib": {
    "cli": "v1.3.7",
    "function_origin": "nsolid-v1.2.0",
    "nan": "v2.1.0",
    "agent": "v5.5.0",
    "persistents_with_classid": "v1.0.1",
    "v8_profiler": "nsolid-v5.2.3-fix1"
  }
}

process_stats

Usage

nsolid-cli process_stats

Returns live metrics reflecting the health and resource utilization of the N|Solid process.

Property Description
uptime The time in seconds this process has been running
rss The total Resident Set Size (total memory used) by this process (bytes)
heapTotal The total allocated size of the JavaScript heap (bytes). A subset of rss
heapUsed The amount of heapTotal being used by JavaScript (bytes)Returns live metrics reflecting the health and resource utilization of the process.
active_requests The number of active requests the event loop will process. For more information see the async_activity command.
active_handles The number of active handles the event loop will process. For more information, see the async_activity command.
title The process title
user The user the process is running as
cpu % cpu being used by the process

Example JSON Result

{
  "uptime": 2580.584,
  "rss": 25083904,
  "heapTotal": 14423040,
  "heapUsed": 8272800,
  "active_requests": 1,
  "active_handles": 2,
  "title": "nsolid",
  "user": "user",
  "cpu": 0
}

system_info

Usage

nsolid-cli system_info

Returns static information about the hosting system for this N|Solid process.

Property Description
cpu_cores The number of CPU cores
cpu_model The CPU model
arch The CPU architecture
platform Name of the NSolid platform
hostname The host name
totalmem Total available memory in the system

Example JSON Result

{
 "cpu_cores": 8,
  "cpu_model": "Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz",
  "arch": "x64",
  "platform": "darwin",
  "hostname": "user.local",
  "totalmem": 17179869184
}

system_stats

Usage

nsolid-cli system_stats

Returns live metrics reflecting the health and resource utilization of the system hosting the N|Solid process.

Property Description
freemem The amount of free (unused) memory in bytes
uptime The uptime of the system, in seconds
load_1m The one-minute load average*
load_5m The five-minute load average*
load_15m The fifteen-minute load average*
cpu_speed The current speed of the CPU (averaged across all cores) in MHz

Example JSON Result

{
  "freemem": 3137511424,
  "uptime": 697783,
  "load_1m": 1.54150390625,
  "load_5m": 1.4052734375,
  "load_15m": 1.44189453125,
  "cpu_speed": 2500
}

startup_times

Usage

nsolid-cli startup_times

Lists the time from initial process execution to reach certain process lifecycle startup phases.

Property Description
initialized_node The time it took to initialize the Node internals.
initialized_v8 The time it took to initialize the V8 engine
loaded_environment The time it took to complete all initialization, which includes running some of node's internal JavaScript code, and your main module's top-level code.

The times are provided in hrtime syntax: a Tuple Array of 2 integers representing the time since the initial process execution in [seconds, nanoseconds].

Example JSON Result

{
  "initialized_node":   [ 0, 130404 ],
  "initialized_v8":     [ 0, 482651 ],
  "loaded_environment": [ 0, 620207709 ]
}

Custom startup timers

If you're looking for information on how to add custom startup and lifecyle events check out this guide.

async_activity

Usage

nsolid-cli async_activity

The N|Solid Agent can at any time list any pending Asynchronous operations and where the JavaScript callbacks will resume. This can be extremely helpful for determining how many concurrent operations and what they are a given N|Solid process is handling. It can also be useful to determine where a stuck application is hanging or if there are any left-behind callbacks, or misue of resources (e.g. reading the same file repetitively)

The output will list the inflight asynchronous activity, including the function location and line of source code to reference where the callback will resume.

Calling the async_activity endpoint will list two types of inflight async work -- handles and requests.

Async handles tend to be longer-lived larger-scale asynchronous operations, such as open sockets or timers.

Async requests tend to be shorter-lived smaller-scale operations such as writing to file handles and other file-related operations.

Example JSON Result

{
  "handles": [
    {
      "type": {
        "flag": 8,
        "name": "TCP socket connection"
      },
      "details": {
        "fd": 14,
        "transport": "TCP",
        "bytesDispatched": 0,
        "hadError": false,
        "host": null,
        "peerAddress": "::ffff:127.0.0.1",
        "peerFamily": "IPv6",
        "peerPort": 42097,
        "sockAddress": "::ffff:127.0.0.1",
        "sockFamily": "IPv6",
        "sockPort": 8080,
        "ondata": "socketOnData: _http_server.js:318:23",
        "onclose": "serverSocketCloseListener: _http_server.js:263:36",
        "onend": "socketOnEnd: _http_server.js:360:22"
      }
    },
    {
      "name": "__unknown_function_name__",
      "location": {
        "file": "[eval]",
        "line": 1,
        "column": 104,
        "inferredName": ""
      },
      "anonymous": true,
      "source": "function () {res.end(\"ok\")}",
      "type": {
        "flag": 1,
        "name": "setTimeout"
      },
      "details": {
        "msecs": 20000
      }
    },
    {
      "type": {
        "flag": 8,
        "name": "TCP socket connection"
      },
      "details": {
        "fd": 15,
        "transport": "TCP",
        "bytesDispatched": 0,
        "hadError": false,
        "host": null,
        "peerAddress": "::ffff:192.168.1.104",
        "peerFamily": "IPv6",
        "peerPort": 54416,
        "sockAddress": "::ffff:192.168.1.104",
        "sockFamily": "IPv6",
        "sockPort": 35770,
        "ondata": "socketOnData: _http_server.js:318:23",
        "onclose": "serverSocketCloseListener: _http_server.js:263:36",
        "onend": "socketOnEnd: _http_server.js:360:22"
      }
    }
  ],
  "requests": []
}

package_info

Returns the list of packages that have been loaded, and information relating to those packages.

Usage

nsolid-cli package_info

Example JSON Result

{
  "packages": [
    {
      "path": "/abs/path/to/dillinger/node_modules/express",
      "name": "express",
      "main": "index.js",
      "version": "4.13.4",
      "dependencies": [
        "node_modules/body-parser",
        "node_modules/cookie-parser",
        "node_modules/cookie-session",
        "node_modules/method-override",
        "node_modules/morgan",
        "../debug",
        "../depd"
      ],
      "modules": [
        { "path": "index.js", "shasum": "3f41478fdab31acabab8fa1d26126483a141ffb6" },
        { "path": "lib/express.js", "shasum": "9999379736d12b5efc0995f75b3379dc4760cdad" },
        { "path": "lib/application.js", "shasum": "ebfe2a8adda9d40c8229cc170c0c8c49bcd7032a" },
        { "path": "lib/router/index.js", "shasum": "788bb9fdb4d2a7b8e0e498bec7a20f4df5709960" },
        { "path": "lib/router/route.js", "shasum": "76517b925237ffdeda27f8aca823edd1acc4b728" },
        { "path": "lib/router/layer.js", "shasum": "27c8fe1d1c09a2699efdc3db1d384d536f6142fb" },
        { "path": "lib/middleware/init.js", "shasum": "dc71629d51c1734ca6cd288417daa053d4e489de" },
        { "path": "lib/middleware/query.js", "shasum": "3e98d56495f4a8828f8965e3566511910da2d20f" },
        { "path": "lib/view.js", "shasum": "7a49a794f95dadb506726719a21a1e6c19bbe4cf" },
        { "path": "lib/utils.js", "shasum": "3f209264d8c57b469ae4c38a13967f4e4e948a0f" },
        { "path": "lib/request.js", "shasum": "b7d310f310a15f08d1cf2b90d2e5b79bfa7797a4" },
        { "path": "lib/response.js", "shasum": "37c0c1c1b155a6cc435595282fa26340c7b220f0" }
      ]
    }
  ]
}

The data returned will be an object with a property packages, which is an array of objects describing the packages that have been loaded. Each package object contains the absolute path of the package loaded, as well as paths relative to that package, for dependant packages it loaded and modules it loaded.

profile_start, profile_stop

Usage

nsolid-cli --app XXXX --id YYYY profile_start
nsolid-cli --app XXXX --id YYYY profile_stop > my.cpuprofile

Collection will commence immediately after ‘profile_start’ hits the wire and will continue for up to 60 seconds. If profile_stop is not entered before 60 seconds then the process times out and the profile data is discarded. Order of execution is always profile_start, then profile_stop.

The profile_start and profile_stop commands may not be broadcast; they must be routed to a particular instance via the --app and --id options.

profile_start

Begins CPU Profile collection. It will collect for up to 60 seconds before timing out. If it times out, all profile data is discarded.

Only one CPU collection is allowed to be running at a time.

profile_stop

Must be run after profile_start -- completes the CPU Profile collection and sends the resulting cpuprofile data. In the N|Solid Console, this will be automatically loaded into our visualization system. From the CLI, the cpuprofile data can be saved to a file and then loaded into Chrome Dev Tools.

snapshot

Usage

nsolid-cli --app XXXX --id YYYY snapshot > my.heapsnapshot

Once the .heapsnapshot file has been created, it can be opened using Chrome’s Development Tool’s CPU Profile Debugging Tool It’s important to note that Chrome requires the generated file to have the extension .heapsnapshot so be sure to use that.

The snapshot command may not be broadcast; it must be routed to a particular instance via the --app and --id options.

Custom Commands

As a developer, you can trigger your own custom commands via the N|Solid Command Line Interface (CLI). Custom commands allow you to interact with your application's processes in ways specific to your business needs.

You can implement a custom command by creating a function to handle the command, and then registering the function with N|Solid.

The custom command function should be defined to take a single parameter, request:

function customCommandHandler(request) { ... }

The request parameter is an object with the following properties/functions:

Parameter Description
request.value An optional piece of data sent with the command, using the nsolid-cli parameter --data.
request.return(value) The function to call when you are ready to return the result of this command. The Agent will reply with the value passed as a parameter.
request.throw(error) A function to call if an error is encountered. The Agent will reply with the error passed as a parameter.

Your function you must call either request.return() or request.throw() to signal completion of the command.

A custom command handler is registered using the nsolid.on() method. You can get access to N|Solid's built-in nsolid module, by calling require("nsolid"). The nsolid.on() function takes the following parameters:

nsolid.on(commandName, handler)
Parameter Description
commandName The string name of the command to implement
handler The custom command function implementing the command

The structure of implementing and registering a custom command will look like the code below:

...
const nsolid = require("nsolid")
...
nsolid.on("foo", fooCommand)
...
function fooCommand(request) { ... }

Custom Commands with Hub

Custom Commands can be "broadcast" via the hub if added to the .nsolid-proxyrc file's broadcast_approved list, as explained here.

Example: Log Level Custom Command

Below is an example of how you can use custom commands to dynamically change the configuration state of the application; specifically the log level. The example assumes that a global boolean variable Verbose is used to indicate whether to log verbosely or not.

//------------------------------------------------------------------------------
// This program is a simple "server" which does nothing, but does implement
// an N|Solid custom command, named `verbose`.  Once you've started this program
// with the N|Solid agent enabled, you can send the `verbose` command as in:
//
//     nsolid-cli --socket <nsolid-agent-address> verbose
//     nsolid-cli --socket <nsolid-agent-address> verbose --data on
//     nsolid-cli --socket <nsolid-agent-address> verbose --data off
//
// All these forms get or set the "verbose" level of logging in the program.
//
// The server logs a message every second when "verbose" is off, and logs
// an additional message after that one when "verbose" is on.  The default
// setting of "verbose" is false.
//------------------------------------------------------------------------------

"use strict"

// get access to N|Solid's built-in module `nsolid`
const nsolid = require("nsolid")

// the current "verbose" level
let Verbose = false

// register the `verbose` command for nsolid-cli
nsolid.on("verbose", verboseCommand)

// our server which doesn't do much
setInterval(onInterval, 2000)

console.log("N|Solid custom command demo - log-level - started")
console.log("")
console.log("to use the verbose command with `nsolid-cli`, run:")
console.log("   nsolid-cli --socket <nsolid-agent-address> verbose")
console.log("   nsolid-cli --socket <nsolid-agent-address> verbose --data on")
console.log("   nsolid-cli --socket <nsolid-agent-address> verbose --data off")

//------------------------------------------------------------------------------
function onInterval() {
  log("interval event!")
  logVerbose("some extra logging here")
}

//------------------------------------------------------------------------------
// implements the `verbose` command  for nsolid-cli
//------------------------------------------------------------------------------
function verboseCommand(request) {

  // if "on" or "off" passed in with --data, set Verbose appropriately
  if (request.value == "on") {
    Verbose = true
  }
  else if (request.value == "off") {
    Verbose = false
  }
  else if (request.value) {
    return request.throw("expecting data of `on` or `off`, got " + request.value)
  }

  // return current value of Verbose
  return request.return({verbose: Verbose})
}

//------------------------------------------------------------------------------
function log(message) {
  console.log(message)
}

//------------------------------------------------------------------------------
function logVerbose(message) {
  if (!Verbose) return
  log(message)
}

When running your application with the N|Solid agent active, you can use the following command to return the current value of the Verbose setting:

nsolid-cli --socket 5000 verbose

To set Verbose on or off, use one of the following commands:

nsolid-cli --socket 5000 verbose --data on
nsolid-cli --socket 5000 verbose --data off

Custom Lifecycle Events

Implementing custom lifecycle events

In addition to the built-in life-cycle events, you can add your own using the function process.recordStartupTime(label) function. The label will then be used in the JSON output of the startup_times command.

You can use this to record the times at various stages of your application's startup. For instance:

  • when a database connection is requested
  • when the database connection is returned
  • when an http server is ready for incoming events

Using the CLI

To obtain the startup timing values, you can use the nsolid-cli startup_times command. For example:

$ nsolid-cli --socket 5000 startup_times

Example JSON Result:

{
  "initialized_node":   [ 0, 130404 ],
  "initialized_v8":     [ 0, 482651 ],
  "loaded_environment": [ 0, 620207709 ]
}

This indicates that node was initialized in 130,404 nanoseconds (which is 130 microseconds, 0.130 milliseconds, or 0.000130 seconds).

Time format

The timing information is provided in hrtime format, which is a two element array of [ seconds, nanoseconds]. A nanosecond is one billionth (1,000,000,000th) of a second. The time values are the elapsed time since the process was started.

Adding a custom timer

Below is an example web server which is instrumented to provide the time when the web server starts listening to connections:

const http = require("http")

const server = http.createServer(onRequest)

server.listen(3000, onListen)

function onListen() {
  console.log("server listening on http://localhost:3000")
  process.recordStartupTime("http_listen")
}

function onRequest(request, response) {
  response.end("Hello, world!")
}

To start this program with the N|Solid agent listening on port 5000, use the following command:

$ NSOLID_SOCKET=5000 node http_sample.js

To obtain the startup times, use the following command:

nsolid-cli --socket 5000 startup_times

This will return the following JSON output:

{
  "initialized_node":   [ 0, 129554 ],
  "initialized_v8":     [ 0, 460521 ],
  "loaded_environment": [ 0, 95201339 ],
  "http_listen":        [ 0, 94902772 ]
}

Three startup times are provided by N|Solid by default:

Parameter Description
initialized_node The time it took to initialize the Node internals.
initialized_v8 The time it took to initialize the V8 engine
loaded_environment The time it took to complete all initialization, which includes running some of node's internal JavaScript code, and your main module's top-level code.

Policies

N|Solid ships with the ability to enable some security-related policies to help harden your application. Currently these include the ability to automatically zero-fill buffers upon allocation and disabling core modules and bindings.

To enable policy configuration with N|Solid, you must create a policies file in JSON format, and specify that file via the policies flag when starting N|Solid.

nsolid --policies my-policies.json app.js

Zero-fill Allocations

The default Node.js memory allocator reuses memory without clearing it. In the case of bugs or malicious code, this can result in the application transmitting data that happened to be in memory that shouldn't be transmitted. This can be used as a basis for exploit, or may result in program data not being consistently initialized.

By default,** Node.js does not zero-fill memory for performance reasons**. This leaves data safety up to the application developer, however, this burden also exists for all third-party dependencies that you use.

You can configure N|Solid to zero-fill allocated memory to prevent these issues. There is a slight performance degradation for enabling this feature.

Example Policy File: zero-fill.policy.json

{
  "process": {
    "zeroFillAllocations": true
  }
}

Usage of zero-fill.policy.json

$ nsolid -p 'new Buffer(20)'
<Buffer 68 f3 88 77 bc 7f 00 00 68 f3 88 77 bc 7f 00 00 e0 05 14 02>

$ nsolid --policies zero-fill.policy.json -p 'new Buffer(20)'
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>

Disabling Core Modules

N|Solid includes the ability to disable core modules with a policy file. This can be used to restrict application access to system resources or features that should not be used by your application.

There are two types of core resources that can be disabled: modules and bindings:

  • modules are the JavaScript APIs of Node, e.g. fs or http;
  • bindings are the wrappers for native code, e.g. stream_wrap or tcp_wrap.

There are four different severity levels you can configure:

Parameter Level Description
0 ignore No notice will be provided during the loading of the module or binding.
This is the default behavior.
1 warning A warning is written to stderr during the loading of the module or binding. The application otherwise runs normally.
2 throw error A error is thrown only when the policy is _violated_​ which doesn't occur during the ​_loading_​ of the module, but when a function on it is called. You can catch this by writing a try/catch in your code around your require() calls.
3 exit The application exits during the loading of the module or binding.

Disabling a module in the policy file prevents all non-core modules from using it i.e a warning is printed when loaded, but only when it is used does the violation handler kick in). Core modules are still able to load it (e.g. the core module library can still use fs to read new files via require() )

Example Policy File: disable-dns-all.policy.json

{
  "// severity levels": "0 - ignore; 1 - warning; 2 - throw error; 3 - exit",

  "blacklist": {
    "modules": {
      "dns": 2
    },
    "bindings": {
      "cares_wrap": 2
    }
  }
}

Access to bindings can also be restricted, if so desired, by the policy writer. The example policy file above disables both the dns module and the cares_wrap module, both of which can be used to access DNS.

When a non-core module attempts to access a disabled core module, messages will be logged to stdout, and an exception thrown.

Usage of disable-dns-all.policy.json

$ nsolid --policies disable-dns-all.policy.json faux-dns.js

nsolid error Binding "cares_wrap" is requested via a process.binding call
nsolid error at Object.<anonymous> (/Users/user/nsolid-demos/demos/faux-dns.js:3:23)
nsolid error But it was disabled in the policies.
_module_wrap.js:423
    throw new Error('Binding "' + id +
    ^

Error: Binding "cares_wrap" has been disabled by the policies flag, using "GetAddrInfoReqWrap" on it is not allowed, terminating execution.
    at new throwError (_module_wrap.js:423:11)
    at Module._compile (module.js:430:26)
... more stack trace lines here

More Policy Usage

Disabling a child process

{
  "blacklist": {
    "modules": {
      "child_process": 2,  -- blacklist entire child_process module with severity level 2
    }
  }
}

Code usage

var ls = require("child_process").execSync("ls -l")
console.log(ls.toString())

Running with policy

$ nsolid --policies disable_childprocess.json child_process_example.js
nsolid error Module "child_process" is required
nsolid error at Object.<anonymous> (/home/bryce/ns/nsolid-node/foo.js:1:72)
nsolid error But it was disabled in the policies.
child_process.js:75
  throw new Error('"' + module.id + '" has been disabled ' +
  ^

Error: "child_process" has been disabled by the policies flag, terminating execution.
    at handleDisabledMethodCall (child_process.js:75:9)
    at Object.execSync (child_process.js:89:22)
    at Object.<anonymous> (/home/bryce/ns/nsolid-node/foo.js:1:96)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:475:10)
    at startup (node.js:149:18)
    at node.js:985:3

Function using the binding 'cares_wraps'

module.exports = lookup;

var cares = process.binding('cares_wrap');

function lookup(hostname, callback) {

  var req = new cares.GetAddrInfoReqWrap();
  req.callback = callback;
  req.family = 4;
  req.hostname = hostname;
  req.oncomplete = callback;

  var err = cares.getaddrinfo(req, hostname, 4, 0);

  if (err) {
    callback(new Error("OOPS"));
    return {};
  }
  return req;
}

Calling function

var look = require("./fauxdns.js");
look("nodesource.com", function (err, data) {
  console.log(data);
});

Without a policy file we're able to access the network via the DNS binding

$ nsolid lookup.js
[ '162.243.142.64' ]

With a policy file that prevents just the dns module

$ cat disable_dns.js
{
  "blacklist": {
    "modules": {
      "dns": 2
    }
}

We can still access it!!

$ nsolid --policies disable_dns.js lookup.js
[ '162.243.142.64' ]

To actually prevent this circumvention of the modulesBlacklist we need to blacklist the cares_wrap binding

$ cat disable_cares.json
{
  "blacklist": {
    "bindings": {
      "cares_wrap": 2
    }
  }
}

Now the behavior is blocked!

$  nsolid --policies disable_cares.json lookup.js
nsolid error Binding "cares_wrap" is requested via a process.binding call
nsolid error at Object.<anonymous> (/Users/kevin/develop/nodesource/policies/fauxdns.js:3:21)
nsolid error But it was disabled in the policies.
_module_wrap.js:423
    throw new Error('Binding "' + id +
    ^

Error: Binding "cares_wrap" has been disabled by the policies flag, using "GetAddrInfoReqWrap" on it is not allowed, terminating execution.
    at new throwError (_module_wrap.js:423:11)
    at lookup (/Users/kevin/develop/nodesource/policies/fauxdns.js:7:13)
    at Object.<anonymous> (/Users/kevin/develop/nodesource/policies/lookup.js:3:1)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:475:10)
    at startup (node.js:150:18)
    at node.js:986:3

A policy file that disables several modules and bindings

{
  "blacklist": {
    "modules": {
      "child_process": 2,  -- blacklist entire child_process module with severity level 2
      "fs": 3,             -- blacklist entire fs module with severity level 3
      "path": 0            -- blacklist path with severity level 0, same as omitting `path` from blacklist all together
    }
    "bindings": {
      "stream_wrap": 2     -- blacklist entire stream_wrap binding with severity level 2
    },
  },
  "process": {
    "zeroFillAllocations": true  -- zero fill all memory allocations i.e. for Buffers
  }
}

Comments

We allow properties whose keys start with "//" anywhere and they are ignored.

Invalid Policies

Anytime an unexpected property or a property with an invalid value is encountered, the policies are considered invalid and the process halted.

Further Thoughts

N|Solid's policies feature should not be used as a blind replacement for good security practices with regard to your code and the code of your application's dependencies. Disabling core modules and bindings does not provide a complete guarantee that your application is unable to use the features provided by those parts of Node.js. For example, child processes interacting with Unix utilities, if not disabled, can be used to replace the functionality of many of the core modules. Some core modules may be completely re-implemented in user-land modules, such as the dns and cares_wrap functionality. Compiled add-ons have complete freedom to insert new functionality that can replace anything that has been disabled.

N|Solid's policies features augment and support existing enterprise-wide application security hardening programs. Please contact NodeSource if you wish to discuss your Node.js application security needs and prepare a customized security hardening program that involves the use of internal and third-party code review, application monitoring and the use of operating system security tools.

Modules That Can Be Blacklisted

The following is a list of modules that can be blacklisted. It should be noted that for most of these modules it is rare to have a business need where blacklisting makes sense.

Module Description
_debug_agent Provides support for debugging node.js, mostly used internally.
_debugger Provides support for debugging node.js, mostly used internally.
_linklist Data structure used internally by node.js.
assert Used for writing unit tests for your applications, you can access it with require('assert').
buffer Raw data is stored in instances of the Buffer class.
child_process Provides a tri-directional popen(3) facility in order to create a child process.
console Console object is a special instance of Console whose output is sent to stdout or stderr, - constants.
crypto Crypto module offers a way of encapsulating secure credentials to be used as part of a secure HTTPS net or http connection.
cluster Allows you to easily create child processes that all share server ports.
dgram Provides Datagram sockets.
dns Provides name resolution.
domain Pending deprecation - provide a way to handle multiple different IO operations as a single group.
events Provides event emitters.
freelist Data structure used internally by node.js.
fs Provides file I/O via simple wrappers around standard POSIX functions.
http Provides ability to create http servers and clients.
_http_agent Provides support for http and https modules, mostly used internally.
_http_client Provides support for http and https modules, mostly used internally.
_http_common Provides support for http and https modules, mostly used internally.
_http_incoming Provides support for http and https modules, mostly used internally.
_http_outgoing Provides support for http and https modules, mostly used internally.
_http_server Provides support for http and https modules, mostly used internally.
https Provides ability to create https servers and clients.
module Provides the node.js module system, mostly used internally.
net Net module provides asynchronous network wrapper for server and clients.
os Provides basic operating-system related utility functions.
path Provides utilities for handling and transforming file paths.
process Same as the global process object.
punycode Provides Unicode utilities.
querystring Provides utilities for dealing with query strings.
readline Allows reading of a stream (such as process.stdin) on a line-by-line basis.
repl Provides a way to interactively run JavaScript and see the results.
stream Provides streaming functionality.
_stream_readable Provides the ReadableStream exposed via the stream module.
_stream_writable Provides the WritableStream exposed via the stream module.
_stream_duplex Provides the DuplexStream exposed via the stream module.
_stream_transform Provides the TransformStream exposed via the stream module.
_stream_passthrough Provides the PassThroughStream exposed via the stream module.
_stream_wrap Provides the wrapper to the libuv stream implementation, mostly used internally.
string_decoder Decodes a buffer to a string with utf8 support.
sys Alias for util.
timers Provides internal support for global Timers.
tls Uses OpenSSL to provide Transport Layer Security and/or Secure Socket Layer: encrypted stream communication.
_tls_common Provides support for tls module, mostly used internally.
_tls_legacy Provides support for tls module, mostly used internally.
_tls_wrap Provides support for tls module, mostly used internally.
tty Provides the tty.ReadStream and tty.WriteStream classes.
url Provides utilities for URL resolution and parsing.
util Primarily designed to support the needs of node.js's internal APIs, but many of these utilities are useful for user programs.
v8 Events and interfaces specific to the version of v8.
vm Provides support to compiled code and run immediately or compile, save it, and run later.
zlib Provides bindings to Gzip/Gunzip.
profiler Profiler module (explained in nsolid docs).
_function_origin Internal module used by nsolid.
_module_wrap Internal module used by nsolid.
_module_wrap_known_modules Internal module used by nsolid.
_module_wrap_known_bindings Internal module used by nsolid.
_policies_validation Internal module used by nsolid.
nsolid Internal module used by nsolid.
nsolid_versions Internal module used by nsolid.

Bindings That Can Be Blacklisted

The following is a list of bindings that can be blacklisted. It should be noted that for most of these bindings it is rare to have a business need where blacklisting makes sense.

Binding Description
async_wrap Base class for all async request modules which supports hooks into async events.
buffer Interface to libuv methods to operate on buffers.
cares_wrap Wraps the cares library for asynchronous name resolves.
contextify Interface to v8 in order to create execution contexts.
crypto Provides crypto functionality such as SSL and ciphers
fs Interface to file system methods provided by libuv.
fs_event_wrap Binding to listen to file related events in order to support watching files and directories for changes.
http_parser Binding to http_parser.
js_stream Binding that provides support for node.js streams.
os Interface to libuv in order to obtain process state information.
pipe_wrap Wraps libuv's socket implementation.
process_wrap Provides process functionality such as spawning child processes.
signal_wrap Supports sending and handling signals to processes.
spawn_sync Supports child_process.execSync functionality via libuv.
stream_wrap Wraps stream implementations exposed by libuv to communicate via file descriptors or network connections.
tcp_wrap Integrates TCP functionality exposed by libuv in order to create TCP connections.
timer_wrap Integrates with timer functionality provided by libuv so support things like setTimeout and setInterval.
tls_wrap Provides TLS termination functionality via crypto.
tty_wrap Integrates with TTY terminal functionality provided by libuv.
udp_wrap Provides UDP protocol functionality such as binding and broadcasting.
uv Provides libuv integration such as error name resolution.
v8 Limited interface to the v8 API, i.e. to set flags and get process information.
zlib Wraps the zlib library.