Top

Getting Started

N|Solid Overview

NodeSource has developed N|Solid to meet the needs of production enterprise environments. Built upon the experience and insights of several core contributors to the Node.js ecosystem, N|Solid provides live instrumentation of your production system's health and stability with no changes to your application code. In addition, N|Solid offers the ability to control access and define policies for your critical applications.

Overview of the N|Solid Components

"N|Solid Architecture"

N|Solid Runtime

At times, Node.js can feel like a black box. The shift to an asynchronous programming model changes how we interpret existing data and how we reason about it. In order to help customers gain more visibility, we provide the N|Solid Runtime. The N|Solid Runtime is a build of Node.js bundled with an advanced native C++ component, the N|Solid Agent. The N|Solid Agent runs on its own thread inside your application, with direct access to the V8 JavaScript engine and libuv, which are the core of Node.js.

The N|Solid Runtime provides access to metrics and allows you to control behavior at runtime. It also includes a powerful security policy model that allows restricting access to system resources by untrusted modules.

Read more about using the Runtime from the command line »

Node Compatibility

N|Solid 2.x is delivered bundled with either a Node.js v4.x Argon LTS or Node.js v6.x Boron LTS runtime. All N|Solid features are additive, meaning any application that runs on Node.js will also work with N|Solid. To verify the version of Node.js you have bundled in N|Solid, use nsolid -v.

For more information about the Node.js API, consult the Node.js API Documentation.

N|Solid Console

The N|Solid Console provides centralized access to all of your applications and an aggregated view of each application's processes. This holistic view simplifies triage at runtime, makes it easier to look for outliers, and takes advantage of advanced diagnostic tools directly from your browser.

Read more about the Console »

Summary

The combination of the N|Solid Runtime and Console make N|Solid an invaluable tool for gaining insight into the behavior of your production applications at runtime.

Backed by NodeSource's 24x7 support, N|Solid adds enterprise-grade diagnostic and security features delivered on a release cycle that is aligned with Node.js Long Term Support (LTS) releases. This provides a stable, production platform for all of your Node applications.

Console Overview

The N|Solid Console provides valuable insight into clusters of N|Solid processes running in a variety of configurations. Processes are grouped by their associated application which makes it simple to find outliers in memory and/or CPU utilization and quickly identify unexpected behavior.

Navigating to an individual process shows various metrics and provides the capability for taking heap snapshots and CPU profiling.

"Flamegraphs"

Application List View

The Application List view provides a visual overview of N|Solid applications monitored by the Console. There are six columns in this view:

"Application List"

Column Description
Overview Grid The Overview Grid is a 5x5 matrix where each vertical dot represents 20% of available memory with the top being 100% memory utilization and each horizontal dot representing 20% of available CPU capacity with the far right being 100% of CPU capacity.

The size of the dot is a function of the number of processes occupying a particular usage slot; for example if five processes were at 22% CPU usage and 15% memory, that dot would be larger than a single process at 22% CPU usage and 35% memory.

Colors are a reflection of what cell the dot is in; colors go from green to red as you move up and right on the grid. Red is naturally the color of severity, so if you have a lot of red dots on your application's grid, that is a call for immediate attention.

It is important to note that since an application can run on multiple hosts, you have have larger dots in the upper right; if you have 10 machines all running at 100% of CPU and memory, that upper right dot will be larger than a single process dot.

Note that this graph provides aggregate data: if your application executes in more than one process, there is no correlation between the number of enlarged dots and the number of processes running.
Application Name Each application gets its name from the NSOLID_APPNAME environment variable (NSOLID_APPNAME="SharkAttack" nsolid myapp.js).

If NSOLID_APPNAME is not set, it will look for name in your package.json. Otherwise, your application (along with any other unnamed applications) will appear in this list under untitled application. If two or more different processes are run with identical NSOLID_APPNAMEs, data from each process will be aggregated into a single application view entry.
Processes The number of processes being traced for this application.
Hosts The number of host machines being traced for this application.
Security Vulnerabilities If there are any security vulnerabilities in an application's dependencies, the number of vulnerabilities and a link to more information about the vulnerabilities is provided.
CPU usage charts The CPU Usage Chart for the Hosts.
RSS charts Resident Set Size (RSS) chart.

Cluster View

The Cluster view is a force-directed scatter-plot that provides an overview of your application’s performance across all connected processes. The Y-axis plots the ratio of the Resident Set Size (RSS) of each process' memory usage over the application’s total RSS memory usage. The X-axis plots the % CPU utilized for each process.

"Cluster of Processes"

Move your mouse over a node in the graph to see its PID and process title:

"PID"

Clicking a node will select it, displaying more information about the process in the sidebar on the right:

"PID Details"

You can use this to get an overview of the following metrics over time for the selected process:

  • Heap Total (MB)
  • Heap Used (MB)
  • Async Activity (Requests, Handles)
  • Host Data (Memory Used, Load Average)
  • Resident Set Size (MB)

Clicking the “Expand” button on the edge of the sidebar will open the Process Detail view.

Process Detail View

The Process Detail view shows detailed information relating to a specific process.

"Process Detail"

Section Description
Uptime This is the total time elapsed since the process started.
Heap Total This is total size of the v8 heap, which includes everything in the heap, including free space which has been allocated by v8 but is unused by JavaScript objects.
Heap Used This is the size of of the v8 heap occupied directly by JavaScript objects created by the application, as well as v8 internal objects created on behalf of it.
Resident Set Size This chart shows the actual memory footprint of the process over time. It does not include memory that belongs to the process that has been swapped out to disk.
Async Activity This chart displays two series that reflect asynchronous activity. One is the number of async handles, which tend to be generated by longer-lived larger-scale asynchronous operations, such as open sockets or timers. The other is the number of async requests. These tend to be made by shorter-lived and smaller-scale operations, such as writing to file handles.
Host Data Host data is a statistic based on the percentage of memory used and the CPU load average over the last minute. For details on the host system running the process, click [🔎 More] underneath the Host Data chart.
Related Snapshots This is a list of heap snapshots associated with this process. When you click [+ New Snapshot], you will be taken to the heap snapshot viewer and a new row representing the heap snapshot will be added to this list.
Related Profiles This is a list of profiles associated with this process. When you click [+ New Profile], you will be taken to the profile viewer and a new row representing the profile will be added to this list.

Host Details

This view shows detailed information about the host system:

  • Hostname
  • Length of time host system has been running
  • Name of the operating system platform
  • CPU details:
    • Manufacturer
    • Model
    • Number of cores
    • Clock speed

At the very bottom is a graph that shows the total system-wide load over time.

This screen is accessed by clicking on the [🔎 More] link to the center right of the screen.

"Host Details"

Quick Start Guide

This guide will help you quickly install all of the N|Solid components onto your development machine on OS X. For instructions on installing the individual components for a production cluster on Linux, see the installation guide.

Installing The OS X Developer Bundle

To install the Developer Bundle for OS X:

  1. Go to downloads.nodesource.com
  2. Click on the Mac OS button to get an installable package
  3. Double click on the downloaded pkg file and follow the instructions to install the N|Solid compoments

Setting Up the N|Solid Console

Using the developer bundle on OS X is quite easy. To launch the console, open a terminal window and enter the command:

$ nsolid-console

to start the console. Note that you do not have to be the root user to run the console, only to install it. Logs for each process can be found under /tmp/nsolid-*.

Once the Console is running, you can see the user interface by opening a new web browser window to http://localhost:6753

You should be redirected to http://localhost:6753/welcome, where you will be asked to enter your License Key. If you already have and know your License Key, paste it in here. Otherwise click Get License Key at the bottom of the form.

Welcome Screen

Fill out all fields on the Register or Retrieve Key form and a key will be emailed to you. If you have previously set up a key, your existing key will be emailed to you, if not, a new one will be generated and emailed to you.

Register or Retrieve Key

Once you've received your N|Solid license key, paste it into the form found at http://localhost:6753/welcome/license and click SUBMIT.

Enter License Key

Once a valid key is entered, you should be immediately redirected to the N|Solid console.

N|Solid Console Up and Running

Running Node Applications in N|Solid

The next step is to configure your shell to use the N|Solid Runtime.

Open a second Terminal window and run:

$ source /usr/local/nsolid/nsolid-env

This will activate the N|Solid shell.

From within the N|Solid shell, any Node application that you start from this terminal will then be shown in the N|Solid Console.

For example, monitor the Nodejs REPL using N|Solid by running:

(nsolid) $ NSOLID_APPNAME="REPL" node

If you have an existing Nodejs application, e.g. "index.js", you can monitor it with N|Solid by running:

(nsolid) $ NSOLID_APPNAME="My_Application" node index.js

You can customize the name of your application in the N|Solid Console by changing the NSOLID_APPNAME environmental variable. If you do not specify an app name, N|Solid will look for a 'name' property in your package.json, and if that is not found your application will be named 'untitled application'

You are now ready to use N|Solid for your own applications!

Migration Guide

If you are upgrading from N|Solid 1.x, here are the need-to-know steps for migrating to N|Solid 2.0.

  • N|Solid Proxy and Hub are no longer required, and may be removed. See Upgrading From Previous Versions of N|Solid in the Installation guide for more information.

  • N|Solid Storage is a new component, please see the Networking documentation for the configuration details for this service.

  • The 1.x versions of N|Solid Runtime and N|Solid Console are not compatible with the corresponding components in 2.0. All components must be upgraded to ensure proper functionality.

  • The environment variables used to configure N|Solid Runtime have changed

    • NSOLID_HUB and NSOLID_SOCKET are no longer used
    • NSOLID_COMMAND should point to the commandBindAddr of your N|Solid Storage instance
    • Further description of the connectivity can be found in the Networking documentation.
  • The nsolid-cli syntax and behavior has changed from 1.x, be sure to read the Command Line Interface documentation if you use this functionality.

  • The Docker Compose process has changed, and a new nsolid.yml file is required. See the Docker documentation for instructions on the new process.

It is important to note that connectivity has changed in N|Solid 2.0 to a model where the N|Solid 2.0 Runtime initiates connections to N|Solid Storage, compared to the previous mode of receiving connections from the N|Solid Console. You will need to ensure that your N|Solid Runtime instances can make TCP connections to the socket addresses in your N|Solid Storage configuration. Additionally, the N|Solid Console will need to make a TCP connection to the web-server port on the N|Solid Storage instance.

N|Solid On-Prem in Production

Installation

This guide will help you quickly install all of the N|Solid components for a production cluster. We provide production rpm and deb packages for use in Linux distributions. This makes it significantly easier to set up the N|Solid components by using our repositories. We support the following rpm based 64-bit Linux distributions:

  • RHEL6 / CentOS6
  • RHEL7 / CentOS7
  • Fedora 24
  • Fedora 25

We also support the following deb based 64-bit Linux distributions:

  • Debian Wheezy
  • Debian Jessie
  • Ubuntu Trusty
  • Ubuntu Xenial
  • Ubuntu Yakkety
  • Ubuntu Zesty

If you do not wish to use these packages, manual installation instructions are provided below.

Note: The default Node.js runtime used in the instructions below is Node.js v6 LTS (Boron). If you wish to use a version of N|Solid based on Node.js v4 LTS (Argon), replace any instances of boron with argon in the instructions below.

Installing N|Solid Packages

Step 1: Set Up the NodeSource N|Solid Repository

For rpm based distributions, use the following command as root:

$ curl -sL http://nsolid-rpm.nodesource.com/nsolid_setup_2.x | bash -

For deb based distributions, there is a similar command run as root:

$ curl -sL http://nsolid-deb.nodesource.com/nsolid_setup_2.x | bash -

In both cases, these scripts will attempt to determine the distribution you are using, set up the appropriate repository on your system, and import the NodeSource N|Solid GPG key used to sign the packages.

Step 2: Install the Packages

For rpm based systems, the command:

$ yum -y install nsolid-boron nsolid-storage nsolid-console

will install all the required components. You can then turn them on using the standard system tools. Older init style systems, for example, would turn on the storage component with:

$ /etc/init.d/nsolid-storage start

whereas newer systemd style systems would use the command:

$ systemctl start nsolid-storage

For deb based systems, the process is similar. To install the software:

$ apt-get -y install nsolid-boron nsolid-storage nsolid-console

On deb based systems, the relevant daemons will start automatically on installation, but you will still turn the services on and off using the standard tools as shown above depending on if you have an init based system or a systemd based system.

These packages can be installed separately, and possibly on different machines if you wish. However, if you plan on installing all of the components onto a single machine, the process can be further simplified. Instead of installing the components individually by name as we've done here, you can simply install the nsolid-console-all meta package for your system. For rpm based distributions:

$ yum -y install nsolid-boron-console-all

or for deb based distributions:

$ apt-get -y install nsolid-boron-console-all

This meta package will install all of the components and turn them on if needed.

Upgrading N|Solid Packages

Once you have set up the NodeSource N|Solid package repository appropriate to your system, you will automatically get point releases via the standard system update tools. For example, running:

yum -y update

will fetch and install any new versions of the packages that are available automatically.

Manual Installation

These instructions use the Linux platform builds, but the same steps apply for OS X (darwin) platforms.

Step 1: Get N|Solid

  1. Go to downloads.nodesource.com
  2. Scroll down to the Downloads section and click the Linux Bundle for the LTS version you wish to run
  3. Extract the archive

The extracted archive contains all N|Solid components for your architecture:

  1. The N|Solid Runtime (e.g. nsolid-v2.2.0-boron-linux-x64.tar.gz)
  2. The N|Solid Storage (e.g. nsolid-storage-v1.4.0.tar.gz)
  3. The N|Solid Console (e.g. nsolid-console-v2.2.0-linux-x64.tar.gz)

Step 2: Install the N|Solid Runtime

The N|Solid Runtime is a build of Node.js that is compiled to include the N|Solid Agent. The runtime should be installed on any application servers where you intend to run your Node.js application.

$ tar xf nsolid-v2.2.0-boron-linux-x64.tar.gz
$ cd nsolid-v2.2.0-boron-linux-x64
$ ls
bin  CHANGELOG.md  include  lib  LICENSE  LICENSE.NSOLID  README.md  share

We suggest installing into /usr/local for greatest compatibility, though most preferred installation methods used for Node.js will also work for N|Solid.

To install to /usr/local:

  1. remove existing npm and nsolid-cli installations:

    $ rm -rf /usr/local/lib/node_modules/npm /usr/local/lib/node_modules/nsolid-cli

  2. put the N|Solid components in place:

    $ cp -r lib bin share /usr/local/

To verify you are running Node compiled with the N|Solid Agent and see your version of N|Solid, use:

$ node -vv
v2.2.0

This package also contains the nsolid-cli application, which can act as a command-line interface to the N|Solid Agent via the N|Solid Storage, allowing scripted or manual interaction.

Step 3: Install the N|Solid Storage Component

N|Solid 2.0 provides a persistance component for orchestrating connectivity and retaining your metrics and assets.

$ tar xf nsolid-storage-v1.4.0.tar.gz
$ ls nsolid-storage-v1.4.0
etc  lib  LICENSE  node_modules  nsolid-storage.js  package.json  README.md  schemas  tsdb

Place this folder wherever you typically install your Node.js applications. For this example, we'll put it in /var/opt

# cp -r nsolid-storage-v1.4.0 /var/opt
# cd /var/opt/nsolid-storage-v1.4.0

Running npm start from within this directory will start the storage service with the default configuration.

For more information about configuration options for the N|Solid Storage, see: networking

You will want to run nsolid-storage as any other Node.js service in your infrastructure, using your process manager of choice.

Step 5: Install the N|Solid Console

The N|Solid Console is a graphical interface for your entire cluster of applications running on N|Solid. It communicates with each N|Solid Agent via N|Solid Storage to track metrics, collect information, or trigger events on any of your N|Solid processes.

The N|Solid Console is best experienced using one of the following browsers on a desktop/laptop system.

  • Google Chrome (21 and higher)
  • Mozilla Firefox (22 and higher)
  • Apple Safari (9 and higher)

The download bundle includes the nsolid-console built for your platform.

$ tar xf nsolid-console-v2.2.0-linux-x64.tar.gz
$ ls nsolid-console-v2.2.0-linux-x64

Place this folder wherever you typically install your Node.js applications. For this example, we'll put it in /var/opt

# cp -r nsolid-console-v2.2.0 /var/opt
# cd /var/opt/nsolid-console-v2.2.0

Running npm start from within this directory will start the N|Solid Console with the default configuration.

For more information about configuration options for the N|Solid Console, see: configuration

You will want to run nsolid-console as any other Node.js service in your infrastructure, using your process manager of choice.

Upgrading From Previous Versions of N|Solid

To upgrade from previous versions of N|Solid, use the instructions above to install over your previous installation. At each step, verify that the latest version of the component is installed and working.

If you have the N|Solid Hub or Proxy installed from an earlier version of N|Solid, these components can be removed. Their functionality is replaced with N|Solid Storage in 2.0.

For rpm based systems this can be done with

  • yum remove nsolid-proxy

For deb based systems this would be

  • apt-get remove nsolid-proxy

The etcd package may be in use by your system for other purposes, but if you no longer require it, it too can be removed

  • yum remove etcd
  • apt-get remove etcd

N|Solid 1.x Artifacts

Any snapshots or profiles recorded with a version of N|Solid prior to 2.0 will continue to exist in the N|Solid console storage location that was used, but these will not be visible in the console. If you require use of these, please access them directly on your storage device.

If you had not configured the storage path for your N|Solid Console, the default path for these files was previously

  • Production packages
    • /var/lib/nsolid/snapshots
    • /var/lib/nsolid/profiles
  • Developer bundle
    • OSX ~/Library/Application\ Support/nsolid-console/
    • Linux ~/.config/nsolid-console/

Configuration

N|Solid Runtime

Environment Variables

All environment variables are optional for the N|Solid Runtime, however functionality will be limited. It is recommended that you connect to the Storage service using NSOLID_COMMAND or a StatsD collector using NSOLID_STATSD and customize your process using any of the remaining environment variables.

Environment Variable Description
NSOLID_COMMAND The route to the Storage command port. It should be formatted as "host:port". The default host is localhost, and the default port is 9001. Without this environment variable, the N|Solid Agent will not attempt to connect.

The host can be provided via several formats:
   IPv4 10.0.0.21
   IPv6 [2001:cdba::3257:9652]
   hostname nsolid_hub.local.
NSOLID_APPNAME The name of the application that the instance is running. Use this in order to create logical groupings of the processes in the Console. If omitted, the value defaults to the name field of your package.json, if one exists, and then falls back to untitled application.
NSOLID_TAGS The list of tags associated with your instance, which can be used to identify and filter instances in Console views. See Tags and Filters for more details.
NSOLID_PUBKEY The ZMQ public key used with the N|Solid Storage server.
NSOLID_HOSTNAME The hostname the N|Solid process is running on (overrides system-determined hostname).
NSOLID_STATSD The route to a StatsD collector if you wish to send metrics directly to an external data collection service from the N|Solid process. Without this environment variable, the N|Solid Agent will not attempt to send StatsD data.

It should be formatted as "host:port". If unspecified, the default host is localhost and port is 8125. Hosts and IP addresses are accepted in the same format as NSOLID_COMMAND.
NSOLID_INTERVAL The interval, in milliseconds, in which the N|Solid Agent will collect and report metrics to the Storage service and/or the StatsD collector if either are connected. This value defaults to 3000 (3 seconds) and cannot be lower than 1000.
NSOLID_STATSD_BUCKET An override for the default StatsD bucket (key prefix) when using the NSOLID_STATSD functionality. See the StatsD section for more details.
NSOLID_STATSD_TAGS Optional tags to append to StatsD metric data when using the NSOLID_STATSD functionality. See the StatsD section for more details.

package.json

The N|Solid process can also be configured via an "nsolid" object in the package.json file for the current application. Environment variable values override the properties in the package.json file. The mapping of "nsolid" object properties in the package.json file to environment variables is as follows:

nsolid property in package.json Environment Variable
command NSOLID_COMMAND
pubkey NSOLID_PUBKEY
statsd NSOLID_STATSD
statsdBucket NSOLID_STATSD_BUCKET
statsdTags NSOLID_STATSD_TAGS
hostname NSOLID_HOSTNAME
env NODE_ENV
interval NSOLID_INTERVAL
tags NSOLID_TAGS - this may be an array of tag values or a comma-separated string of tag values
app NSOLID_APPNAME

Example package.json:

{
  "name": "message-service",
  "version": "1.0.0",
  "nsolid": {
    "env": "production",
    "command": "nsolid-command-host.local:9001",
    "app": "messaging",
    "tags": "aws, mq"
  }
}

N|Solid CLI

Configuration File

The N|Solid Command Line Interface (CLI) can be configured via a configuration file, .nsolid-clirc.

{
  "remote": "http://localhost:4000",
  "app": "my-node-application"
}

N|Solid Console

Command Line Arguments

These are the command line arguments if you're starting the console explicitly. For most usages, you would utilize npm start.

Parameter Description
--storageUrl N|Solid Storage URL (e.g., --storageUrl https://localhost:4000)
--consoleHostname N|Solid Console hostname (e.g., --consoleHostname localhost)
--consolePort=<port> Specify which TCP port to listen on (default: 6753)

Environment Variables

If you are using the developer bundle you can use the following to configure the console.

Environment Variable Description
NSOLID_CONSOLE_STORAGE_URL Same as the --storageUrl option above
NSOLID_CONSOLE_HOSTNAME Same as the --consoleHostname option above
NSOLID_CONSOLE_PORT Same as the --consolePort option above
PORT Same as the --consolePort option above
LOGLEVEL The level of logging produced; can be error, warn, info, debug; defaults to info

Stats Retention

Historical metrics, displayed in the line-chart UI elements, are trimmed to the last 15 minutes by default.

For more information about configuration options for the N|Solid Storage, see: networking

License Keys

Registering Your License Key

In order to access N|Solid you must have an enabled license key. If you have not entered your license key, when you try and access the console you will be redirected to the welcome screen. If you already have your License Key, paste it in the license key field. Otherwise click Get License Key at the bottom of the form.

Welcome Screen

Fill out all of the fields on the Register or Retrieve Key form and a license key will be sent to you via email:

  • If you have previously set up a license key, you will receive your existing key
  • If you have not previously set up a license key, a new license key will be created

Register or Retrieve Key

Once you've received your N|Solid license key, paste it into the Welcome screen form and click SUBMIT.

Once a valid key is entered, you should be immediately redirected to the N|Solid console.

Updating Your License Key

If you need to update your license key from your original demo key (or for any other reason), click on the menu in the upper-left corner next to the N|Solid logo and select License Key.

Next, paste in your updated N|Solid license. Then click SAVE CHANGES.

Your updated license key should now be active.

Troubleshooting

Debug Messages

The N|Solid Runtime and CLI all can have debug messaging enabled by using the NODE_DEBUG=nsolid environment variable.

$ NODE_DEBUG=nsolid NSOLID_APP=testapp NSOLID_COMMAND=localhost ~/ns/nsolid-node/nsolid
> NSOLID 16597: starting agent (js init) name: testapp id: e24a79b2aae58b63e385a4c0ce9ed91f3d0202ee tags: undefined
NSOLID 16597: nsolid agent e24a79b2aae58b63e385a4c0ce9ed91f3d0202ee starting with config
NSOLID 16597: { command: '9001',
  data: undefined,
  bulk: undefined,
  pubkey: undefined,
  statsd: undefined,
  statsdBucket: 'nsolid.${env}.${app}.${hostname}.${shortId}',
  statsdTags: undefined,
  hostname: 'rpi3-1',
  env: 'prod',
  interval: null,
  tags: undefined,
  app: 'testapp',
  appVersion: undefined }
NSOLID 16597: { zmq_command_remote: 'tcp://localhost:9001',
  zmq_data_remote: null,
  zmq_bulk_remote: null,
  statsd_addr: null,
  storage_pubkey: '^kvy<i^qI<r{=ZDrfK4K<#NtqY+zaH:ksm/YGE6I' }
NSOLID 16597: registering default commands
NSOLID 16597: nsolid initializing
NSOLID 16597: agent spawn()
NSOLID 16597: agent spawned
NSOLID 16597: e24a79b2aae58b63e385a4c0ce9ed91f3d0202ee nsolid initialized
NSOLID 16597: { command_remote: 'tcp://localhost:9001',
  data_remote: 'tcp://localhost:9002',
  bulk_remote: 'tcp://localhost:9003',
  storage_pubkey: '^kvy<i^qI<r{=ZDrfK4K<#NtqY+zaH:ksm/YGE6I' }

StatsD

The N|Solid Agent is able to periodically send a suite of metrics directly to a StatsD-compatible data collector endpoint. As this functionality is built directly into the nsolid executable, you do not need to be connected to the N|Solid Storage service in order to use it. Independent nsolid processes can be configured to talk to StatsD collectors using environment variables when starting your application.

Consult the Metrics in Detail section for a complete list of metrics that are sent to the StatsD collector for your N|Solid processes.

Supported backends

StatsD is a standardized protocol that is supported by many data collection databases and services. Supported backends include AWS CloudWatch, DataDog, ElasticSearch, Graphite, InfluxDB, MongoDB, OpenTSDB, StackDriver, Zabbix and many more. See the StatsD documentation for a more complete list.

Connection options

Supply the NSOLID_STATSD environment variable when starting an N|Solid process to have the Agent attempt to connect to an endpoint. The format of this value is "host:port". If unspecified, the default host is localhost and port is 8125. The host can be specified as a hostname string, an IPv4 or IPv6 address.

Once connected, a suite of metrics will be sent using the StatsD protocol to the collector. The NSOLID_INTERVAL environment variable can be used to adjust the default reporting interval of 3 seconds. Be aware that this will also change the reporting interval for the Storage connection if connected via the NSOLID_COMMAND environment variable.

StatsD buckets

StatsD metrics are reported using "buckets". A bucket is the full name, or key, of the entry being reported. Buckets need to be descriptive enough to clearly identify the process, host and metric type being reported. It is important that you have enough granularity to be able to inspect the data at the level you require, while also retaining the ability to group metrics for aggregate reporting in your frontend using bucket wildcard mechanisms. This will depend somewhat on your reporting frontend and may require some experimentation.

By default, N|Solid creates bucket names prefixed with "nsolid.<env>.<app>.<hostname>.<shortId>.", followed by the name of the metric. In this string:

  • env is the value of the NSOLID_ENV environment variable if it exists, or the "env" property of the "nsolid" section of package.json if it exists, otherwise "prod" is used.
  • app is the value of the NSOLID_APP environment variable if it exists, or the "app" property of the "nsolid" section of package.json if it exists, or the "name" property in package.json if it exists, or process.title if it has been set, otherwise "untitled application" is used.
  • hostname is the value of the NSOLID_HOSTNAME environment variable if it exists, or the "hostname" property of the "nsolid" section of in package.json if it exists, otherwise the value of os.hostname() is used.
  • shortId is the first 7 characters of the random alphanumeric ID created by the N|Solid Agent at startup. This can be used to uniquely identify your processes. The full 40 character string is also available as id if required.

To override the default StatsD metric bucket strings, provide a string via the NSOLID_STATSD_BUCKET environment variable to be used as the full prefix. ${key} style variables can be used to insert any of the above values. The default bucket prefix would be specified as follows: "nsolid.${env}.${app}.${hostname}.${shortId}".

Your StatsD data, by default, will be formatted like so when sent to the collector:

nsolid.prod.myapp.nodehost.803bbd5.uptime:314.4|g
nsolid.prod.myapp.nodehost.803bbd5.rss:28401664|g
nsolid.prod.myapp.nodehost.803bbd5.heapTotal:8425472|g
nsolid.prod.myapp.nodehost.803bbd5.heapUsed:5342488|g
...

StatsD tags

Some backends, such as DataDog support "tags" for metric reporting. By default, N|Solid does not append any tags to its metrics but if required, you can supply your own tags to be appended to all reported metrics from an individual process.

Using the NSOLID_STATSD_TAGS environment variable, or the "statsdTags" property of the "nsolid" section of package.json, supply a string with the same variable substitution format as for the buckets above. In addition to env, app, hostname, shortId and id variables, you can also make use of tags to insert the N|Solid tags that have been supplied for the process.

StatsD tags should be a comma-separated list of strings that your backend can decode. Refer to your backend documentation for how these values are made use of for reporting and whether their use will be suitable for your deployment.

Networking

N|Solid Storage

N|Solid Storage is the component that provides data persistence for your measurements and artifacts, such as snapshots and CPU profiles. It communicates directly with your N|Solid Agent processes and provides the API interface for both nsolid-cli and nsolid-console.

Command Line Arguments

These are the supported command line arguments if you're starting N|Solid Storage explicitly.

Parameter Description
--config=config.toml The configuration file for Storage. The default config will be used if none is specified
--relay Run Storage in Relay mode. This allows you to place an instance in a segment of your network that can aid connectivity and uptime

Configuration File

N|Solid Storage can be configured via a configuration file if the --config argument is specified. This file is formatted as TOML (similar to ini file format). A sample file is available in the file config-sample.toml in the root directory of the nsolid-storage server. The contents of the file are also listed below.

To run with a customized configuration, copy the file and edit it so that it contains your customized values. You can omit keys from any section which you don't want to override, and omit entire sections if no keys in that section are overridden.

Alternatively, or in addition, you can set environment variables for every configuration key and value. The names of the environment variables are listed with the configuration keys and values below.

For more information on the TOML language used here, please see: https://github.com/toml-lang/toml

#-------------------------------------------------------------------------------
# Properties that are file or directory names can be absolute or relative file
# names. When relative, the names are considered relative to the configuration
# file they were specified in.  When using relative file names as environment
# variables, they will be considered relative to the current working directory.
#-------------------------------------------------------------------------------

#-------------------------------------------------------------------------------
# set the logging level - "info" or "debug" - debug is more verbose
#-------------------------------------------------------------------------------

# environment variable NSOLID_STORAGE_LOG_LEVEL

logLevel = "info"

#-------------------------------------------------------------------------------
# settings for licensing
#
# key = "the-license-key"
#-------------------------------------------------------------------------------

[license]

# environment variable NSOLID_STORAGE_LICENSE_KEY

key = ""

#-------------------------------------------------------------------------------
# settings for the API HTTP server
#
# proto  - "http" or "https"
# server - "<ip addr | hostname>:<port>"
#
# If the environment variable PORT is set, that value, prefixed with ':', will
# be set to the server property. Eg, using PORT=42 will set web.server to ':42'.
#
# httpProxy and httpsProxy are used to set an explicit HTTP/S proxy for HTTP/S
# requests that storage makes to the public internet.  You can also set the
# environment variables HTTP_PROXY or http_proxy, and HTTPS_PROXY or https_proxy
# as the value to use.
#-------------------------------------------------------------------------------

[web]

# environment variable NSOLID_STORAGE_WEB_PROTO
# environment variable NSOLID_STORAGE_WEB_SERVER
# environment variable NSOLID_STORAGE_WEB_HTTP_PROXY
# environment variable NSOLID_STORAGE_WEB_HTTPS_PROXY

proto      = "https"
server     = "0.0.0.0:4000" # 0.0.0.0 allows access on all network interfaces
httpProxy  = ""
httpsProxy = ""

#-------------------------------------------------------------------------------
# settings for running the API HTTP server in https mode
#
# key and cert are file names, as from Node.js https.createServer() options
#
# when set to blank, the files inside the nsolid-storage/etc folder will be used:
#   key:  nsolid-self-signed.key
#   cert: nsolid-self-signed.crt
#
# See note above concerning relative paths.
#
# note: generate new default credentials with ./gen-https-creds.sh
# note: default credentials files shipped with nsolid-storage expire in 08/2026
#-------------------------------------------------------------------------------

[web.https]

# environment variable NSOLID_STORAGE_WEB_HTTPS_KEY
# environment variable NSOLID_STORAGE_WEB_HTTPS_CERT

key  = ""
cert = ""

#-------------------------------------------------------------------------------
# zmq sockets for access by N|Solid processes
#
# commandBindAddr, dataBindAddr, bulkBindAddr are the zmq port addresses
# that will be bound by the server.
#
# publicKey, privateKey are the keys used to encrypt the messages sent over
# zmq.  For more information see:
# https://github.com/msealand/zmq-zap.node/blob/master/examples/curve.js
#
# HWM, bulkHWM are used to set the high water mark for zmq's internal buffering.
# The value indicates the number of messages to buffer, and 0 indicates no limit.
#
# The N|Solid agent will use environment variables that correspond to these
# settings.  For the sockets, the following N|Solid agent environment variable
# should use an address that it can use to connect to the socket the server has
# bound.
#
#    N|Solid agent env vars    server configuration values
#    ----------------------    ---------------------------
#    NSOLID_COMMAND            sockets.commandBindAddr
#    NSOLID_DATA               sockets.dataBindAddr
#    NSOLID_BULK               sockets.bulkBindAddr
#    NSOLID_PUBKEY             sockets.publicKey
#
# If running the storage server in server mode AND relay mode on the same host,
# the configuration values for the sockets.xxxBindAddr above must be different
# for the server mode and relay configurations.
#-------------------------------------------------------------------------------

[sockets]

# For commandBindAddr, dataBindAddr, and bulkBindAddr, the format should be:
#    tcp://[ipAddress]:[port]
# [ipAddress] should be an ip address - host names cannot be used; use
# "*" to use addresses on any available network interface.

# environment variable NSOLID_STORAGE_SOCKETS_COMMAND_BIND_ADDR
# environment variable NSOLID_STORAGE_SOCKETS_DATA_BIND_ADDR
# environment variable NSOLID_STORAGE_SOCKETS_BULK_BIND_ADDR
# environment variable NSOLID_STORAGE_SOCKETS_PUBLIC_KEY
# environment variable NSOLID_STORAGE_SOCKETS_PRIVATE_KEY
# environment variable NSOLID_STORAGE_SOCKETS_HWM
# environment variable NSOLID_STORAGE_SOCKETS_BULK_HWM

commandBindAddr = "tcp://*:9001"
dataBindAddr    = "tcp://*:9002"
bulkBindAddr    = "tcp://*:9003"
publicKey       = "^kvy<i^qI<r{=ZDrfK4K<#NtqY+zaH:ksm/YGE6I"
privateKey      = "2).NRO5d[JbEFli7F@hdvE1(Fv?B6iIAn>NcLLDx"
HWM             = 0
bulkHWM         = 0

# The following socket address values are only relevant when running in --relay
# mode. They are analogs of their similiarly named 'xxxBindAddr' values, but
# used as the zmq sockets of the storage server to connect to.  These should
# be formatted as "tcp://host:port"
#
# If you want to use a different keypair on the remote endpoint, you may
# specify a remotePublicKey matching that endpoint.  This can be used to
# perform a rolling update to keys without losing data from agents with the old
# encryption key (by using multiple relays)

# environment variable NSOLID_STORAGE_SOCKETS_COMMAND_REMOTE_ADDR
# environment variable NSOLID_STORAGE_SOCKETS_DATA_REMOTE_ADDR
# environment variable NSOLID_STORAGE_SOCKETS_BULK_REMOTE_ADDR
# environment variable NSOLID_STORAGE_SOCKETS_REMOTE_PUBLIC_KEY

commandRemoteAddr = ""
dataRemoteAddr    = ""
bulkRemoteAddr    = ""
remotePublicKey   = ""

#-------------------------------------------------------------------------------
# Relay mode settings
#
# These settings are specific to running in relay mode.
#
# maxBufferSizeMb is the max buffering "hwm" values shared between the
# "pull/push" sockets in megabytes.
#
# cleanupSizeMb is the amount of megabytes of buffers to be dropped when limits
# are reached.
#
# logSizeInterval is the interval in seconds that will trigger a buffering
# status to be logged. Only happens when upstream (remote) sockets are
# disconnected.
#-------------------------------------------------------------------------------

[relay]

# environment variable NSOLID_STORAGE_RELAY_MAX_BUFFER_SIZE_MB
# environment variable NSOLID_STORAGE_RELAY_CLEANUP_SIZE_MB
# environment variable NSOLID_STORAGE_RELAY_LOG_SIZE_INTERVAL

maxBufferSizeMb = -1
cleanupSizeMb   = 100
logSizeInterval = 10

#-------------------------------------------------------------------------------
# data settings
#
# dir           - the directory where data will be persisted
# retentionDays - the number of days of metric data that will be retained
#
# See note above concerning relative paths.
#-------------------------------------------------------------------------------

[data]

# environment variable NSOLID_STORAGE_DATA_DIR
# environment variable NSOLID_STORAGE_DATA_RETENTION_DAYS

dir           = "~/.nsolid-storage/data"
retentionDays = 1

#-------------------------------------------------------------------------------
# notification settings
#
# consoleURL - the base URL of the console, for use in providing links
# back to the console in notifications.
#-------------------------------------------------------------------------------
[notification]

# environment variable NSOLID_STORAGE_NOTIFICATION_CONSOLE_URL

consoleURL = "http://localhost:6753"

#-------------------------------------------------------------------------------
# log settings
#
# influx - the name of the file where the influxd log will be written
#
# See note above concerning relative paths.
#-------------------------------------------------------------------------------

[logs]

# environment variable NSOLID_STORAGE_LOGS_INFLUX

influx = "~/.nsolid-storage/influxdb.log"

#-------------------------------------------------------------------------------
# vulnerability settings
#
# refreshMinutes - how often the vulnerability database will be refreshed
#-------------------------------------------------------------------------------

[vulnerabilities]

# environment variable NSOLID_STORAGE_VULNERABILITIES_REFRESH_MINUTES

refreshMinutes = 30

#-------------------------------------------------------------------------------
# assets settings - storage of heap snapshots, cpu profiles, etc
#
# cleanupMinutes  - how often the asset files are cleaned up; the oldest asset
#                   files are deleted to ensure constraints below are met.
# maxAssets       - maximum number of assets to retain
# maxAssetsMb     - maximum size of all assets to retain
#-------------------------------------------------------------------------------

[assets]

# environment variable NSOLID_STORAGE_ASSETS_CLEANUP_MINUTES
# environment variable NSOLID_STORAGE_ASSETS_MAX_ASSETS
# environment variable NSOLID_STORAGE_ASSETS_MAX_ASSETS_MB

cleanupMinutes  = 5
maxAssets       = 500
maxAssetsMb     = 100

Relay Mode

N|Solid Storage supports running in a relay mode in order to support proxying through firewalls, buffering agent data during downtime and scaling with complex network topologies.

The relay sits between agents and N|Solid Storage (in default or relay configuration). In this mode, nsolid-storage looks like an N|Solid Storage instance to agents but does not have any N|Solid Storage functionality enabled.

Relay Configuration

To use relay mode, set the --relay flag and the three [sockets] variables described below. These can be provided via the configuration file or as environment variables. If omitted, the defaults will be used.

You should change the NSOLID_COMMAND, NSOLID_DATA and NSOLID_BULK variables as those will be the ones that the relay itself listens on while it uses the NSOLID_STORAGE_SOCKETS-* variables in order to connect to N|Solid Storage.

In the simplest case you will set the NSOLID_STORAGE_SOCKETS_* variables to the same that the related NSOLID_* variables were set to and pick different addresses for the NSOLID_STORAGE_SOCKETS_* variables.

Environment Variables

Environment Variable Description
NSOLID_STORAGE_SOCKETS_COMMAND_REMOTE_ADDR This is the route to the Storage command port that the relay will use to connect to it. It should be formatted as "host:port" and follows the same conventions as NSOLID_COMMAND
NSOLID_STORAGE_SOCKETS_DATA_REMOTE_ADDR This is the route to the Storage data port that the relay will use. It should be formatted as "host:port" and follows the same conventions as NSOLID_COMMAND. Unlike NSOLID_DATA you have to specify this variable or include it in your config.
NSOLID_STORAGE_SOCKETS_BULK_REMOTE_ADDR This is the route to the Storage bulk port that the relay will use. It should be formatted as "host:port" and follows the same conventions as NSOLID_COMMAND. Unlike NSOLID_BULK you have to specify this variable or include it in your config.

Configuration File

Below is an example of configuring of the [sockets] section of a configuration file to support relay mode. Note that we also changed the related Bind addresses and specified an explicit public key for the remote Storage server.

[sockets]
commandBindAddr = "tcp://<storage-address>::8001"
dataBindAddr    = "tcp://<storage-address>:8002"
bulkBindAddr    = "tcp://<storage-address>:8003"

commandRemoteAddr = "tcp://<storage-address>:9001"
dataRemoteAddr = "tcp://<storage-address>:9002"
bulkRemoteAddr = "tcp://<storage-address>:9003"
remotePublicKey = "^kvy<i^qI<r{=ZDrfK4K<#NtqY+zaH:ksm/YGE6I"

Docker Compose File

Relay mode environment variables need to be added to the storage section. In order to match the settings in the above configuration file you would add the following.

  storage:
    environment:
      - NSOLID_STORAGE_SOCKETS_COMMAND_BIND_ADDR=storage:8001
      - NSOLID_STORAGE_SOCKETS_DATA_BIND_ADDR=storage:8002
      - NSOLID_STORAGE_SOCKETS_BULK_BIND_ADDR=storage:8003
      - NSOLID_STORAGE_SOCKETS_COMMAND_REMOTE_ADDR=storage:9001
      - NSOLID_STORAGE_SOCKETS_DATA_REMOTE_ADDR=storage:9002
      - NSOLID_STORAGE_SOCKETS_BULK_REMOTE_ADDR=storage:9003

See the Docker page for more information on setting up Docker Compose.

Advanced Environment Variables

The variables described below should only be necessary if your setup requires you to use a proxy in front of N|Solid Storage.

If this is not needed you may ignore this section as the variables will be configured for you by the N|Solid agent via the COMMAND interface.

Environment Variable Description
NSOLID_DATA This is the route to your Storage data port. It should be formatted as "host:port" and follows the same conventions as NSOLID_COMMAND. If you do not specify this argument, NSolid will attempt to discover this port from Storage via the COMMAND interface.
NSOLID_BULK This is the route to your Storage bulk port. It should be formatted as "host:port" and follows the same conventions as NSOLID_COMMAND. If you do not specify this argument, NSolid will attempt to discover this port from Storage via the COMMAND interface.
NSOLID_PUBKEY This is Storage's public key, if you have configured a non-default one. If you do not specify a key, the default keypair that ships with NSolid will be used.

Moving the N|Solid Storage server

The data maintained the N|Solid Storage server is stored in a directory determined by the dir property of the [data] section of the configuration file. The default value is ~/.nsolid-storage/data. The contents of the directory can be copied to a new machine, and then the N|Solid Storage server can be run with it's configuration pointing to the copied directory.

Docker

For people who need to use N|Solid in a Dockerized environment, NodeSource has developed the N|Solid Docker images.

These images are built with the enterprise customer in mind, developing a versatile set of independently scalable Docker images that work together to provide a unified N|Solid experience across all of your Dockerized infrastructure.

In addition to being friendly to enterprise operations teams, the N|Solid Docker Images have been designed to be accessible to developers. For those who are already using Docker, these images provide an easy way to get up and running with the N|Solid console. With the use of docker-compose, you can now get the full N|Solid Suite on your local machine with a single command.

Overview of N|Solid Docker Images

"N|Solid Architecture"

We provide 4 separate Docker Images for N|Solid, allowing each component of the platform to be scaled and deployed independently.

  • docker pull nodesource/nsolid
    • the base image that your Docker images should build from. It provides the N|Solid runtime and agent and needs to be properly configured to register its agent with the N|Solid Storage service. Releases
  • docker pull nodesource/nsolid-storage
    • contains the N|Solid Storage service, which is used by the N|Solid Console to query runtime statistics from your Node.js applications. It also provides historical data. N|Solid agents register themselves with this service and the N|Solid Console connects to it. Releases
  • docker pull nodesource/nsolid-console
    • contains the N|Solid console, a web based interface for monitoring your Node.js applications. You will need to bind the internal port 6753 to your host. This container will need to be able to talk to the N|Solid Storage container. Releases
  • docker pull nodesource/nsolid-cli
    • provides an easy way to query statistics from your Node.js processes. It is meant to be run directly from the command line and is optional. Releases

All of the images are built on the official ubuntu:trusty image maintained by the Docker project. The latest tag defaults to the latest Boron release. For more details please refer to the docker-nsolid github repository.

Installing the N|Solid Docker Images

Setting up Docker

First, follow the steps for installing Docker for your operating system, we recommend you use at least Docker v1.9: https://www.docker.com/products/docker

Although not required, it is recommended that you use docker-compose when first getting started. It simplifies managing these Docker images when running them locally.

Docker Compose

You can use the docker-compose file below to further understand the N|Solid platform configuration and/or to use it as a starting point for your docker setup.

NOTE: the following docker-compose file doesn't include any N|Solid application yet. We will explain how to add your application below. Use a valid path in the volumes configuration, or remove it if persistence is not important to you.

nsolid.yml

version: "2"
services:
  storage:
    image: nodesource/nsolid-storage
    container_name: nsolid.storage
    ports:
      - 4000:4000
      - 9001:9001
      - 9002:9002
      - 9003:9003
    volumes:
      - /path/to/persistent/storage:/var/lib/nsolid/storage/data
  console:
    image: nodesource/nsolid-console
    container_name: nsolid.console
    environment:
      - NSOLID_CONSOLE_STORAGE_URL=https://storage:4000
    links:
      - storage
    ports:
      - 6753:6753

Downloading the Docker Images

If you are using docker-compose, open a terminal in the folder you created nsolid.yml in, and run the following command:

docker-compose -f nsolid.yml pull

This command will download all of the images needed to run the N|Solid console locally. If you choose not to use docker-compose, execute the following commands:

docker pull nodesource/nsolid-storage
docker pull nodesource/nsolid-console

# Optional CLI
docker pull nodesource/nsolid-cli

Using the N|Solid Docker Images

Starting the N|Solid Console

To bring up the N|Solid Console at this point, run the following command:

docker-compose -f nsolid.yml up

By default, this will bring up the N|Solid Console running on 127.0.0.1:6753.

Even though we highly recommend starting up the N|Solid using docker-compose since its declarative nature makes it very easy to configure each piece of the N|Solid platform you can do so by just using docker as well as follows:

# start storage
docker run -d -p 9001-9003:9001-9003 -p 4000:4000 -v /path/to/persistent/storage:/var/lib/nsolid/storage/data --name storage nodesource/nsolid-storage
# start console
docker run --link storage:storage -d -p 6753:6753       \
  -e 'NSOLID_APPNAME=console'                 \
  -e 'NSOLID_CONSOLE_STORAGE_URL=https://storage:4000' \
  nodesource/nsolid-console

This will bring up the N|Solid console available on localhost:6753.

Visit the docker docks for more information on linking containers

Creating an Example Docker Application

Create a file called server.js:

var http = require('http');
var hostname = require('os').hostname();
var port = process.env.PORT || 8888;

var server = http.createServer(function handleRequest (request, response) {
  response.end('[' + hostname + '] Serving requests from myapp. Request URL:' + request.url);
});

server.listen(port, function () {
  console.log('Server listening on port', port);
});

Next create a file called Dockerfile:

FROM nodesource/nsolid

COPY server.js /server.js

EXPOSE 8888

CMD ["nsolid", "/server.js"]

Building the docker image:

docker build -t example .

Starting Example Application with Docker Compose

Next, create the file docker-compose.yml in the directory along-side nsolid.yml:

docker-compose.yml

version: "2"
example:
  image: example
  container_name: nsolid.example
  links:
    - storage
  ports:
    - 8888:8888 # Port your application exposes
  environment:
    - NSOLID_APPNAME=example
    - NSOLID_COMMAND=storage:9001
    - NSOLID_DATA=storage:9002
    - NSOLID_BULK=storage:9003

For the complete documentation on defining a service with docker-compose.yml, refer to the Docker project’s documentation page: https://docs.docker.com/compose/overview/.

At this point, you are ready to bring up your application using docker-compose

docker-compose -f nsolid.yml -f docker-compose.yml up

Starting Example Application without Docker Compose

Start your service with:

# start and connect your application to N|Solid
docker run -d --link storage:storage --name example -e 'NSOLID_APPNAME=example' -e 'NSOLID_COMMAND=storage:9001' -e 'NSOLID_DATA=storage:9002' -e 'NSOLID_BULK=storage:9003' example

Adding Your Own Application with docker-compose

First, ensure your application is compatible with the LTS release of Node.js. NodeSource has created upgrade-ready to assist with this.

If you are new to Docker, follow the steps in our blog post to get your application into a Docker container.

Your Dockerfile should begin with the following line:

FROM nodesource/nsolid

Congratulations, you are now up and running with N|Solid and Docker!

Using N|Solid

CPU Profiling

CPU Profiling can be used to indicate all the functions on the function call stack, when samples are taken. For instance, if a function foo() called a function bar() during its execution, and a sample was taken while bar() was running, the function call stack will show that foo() called bar(). Because multiple samples may be taken while bar() is executing, there will be an approximate start and stop time recorded for bar(), which is an indication of how long it took to run. In addition, further samples before and after the ones that captured bar() will capture foo(), and likewise to the bottom of the function call stack - typically, a callback of some sort.

This data can then be analyzed to show, for the function foo(), how much time was actually spent in foo() and not in bar() - every function has two time values - a self time and a total time. For the foo() and bar() case, if foo() only calls the bar() function, then the self time for foo() + the total time for bar() will equal the “total” time for foo().

Example

function foo() {
        … processing that takes a lot of time but calls no other functions ...
    bar()
        … processing that takes a lot of time but calls no other functions …
}

function bar() {
    …
}

foo’s total time = foo’s self time + bar’s total time.

Both values are interesting; total time shows you which functions are fastest and slowest, from start to finish, but doesn’t tell you if the time was spent in that function or other functions.

Named Functions

Named functions are easier to spot in CPU profiles. The stack frame entries available in a CPU profile include the name of function, and source code location information for detail views. For anonymous functions, the name will often be displayed as "(anonymous)". In some cases, the V8 JavaScript engine can calculating an "inferred name" for a function. When in doubt, name your functions so that you can easily spot them in a CPU profile.

For instance, let's say you have a function busyFunction(), which you like to easily track the callers for, and you have cases where it's called from an anonymous function, like this:

setInterval(function(){busyFunction()}, 1000)

In a CPU profile, you'll see that busyFunction() is being called by(anonymous) in this case.

To make this easier to easier to spot in a CPU profile, you can simply use:

setInterval(function busyFunctionCaller(){busyFunction()}, 1000)

In the CPU profile, you'll now see that busyFunction() is called by busyFunctionCaller().

For additional code cleanliness, and less chance of creating a "pyramid of doom", consider moving the entire function out into the same scope as the function usage; for instance:

setInterval(busyFunctionCaller, 1000)
//...
function busyFunctionCaller() {
  busyFunction()
}

Because JavaScript functions are "hoisted" to the top level of the scope they're defined in, you can reference busyFunctionCalled before it's actually defined.

CPU Profiling allows you to understand where opportunities exist to improve the speed and load capacity of your node process. N|Solid provides multiple ways to profile your CPU.

Analyzing Profiles Using the N|Solid Console

The N|Solid Console allows you to take and analyze profiles in one action. This is particularly useful if you need to take a series of profiles as it saves you the overhead of switching from one environment to another. The console also saves a history of profiles so you can quickly flip through profiles to identify troublesome areas.

  1. Launch the console and select the application of interest "All applications"
  2. Select the process of interest "Select process"
  3. Expand out the details panel and select ‘+ New Profile’
    "New profile"
  4. Select your profile window (5 to 60 seconds) and run profile "Run profile"
  5. Select a visualization (Sunburst, Flamegraph or Treemap). Once a visualization is selected, you can click on it to view the call stack to the right. The number of calls to that method are shown as are the time spent within it. Clicking on the method will show you the file responsible for that call. "Call stack"
  6. Profiles are saved in your console session so you can easily flip between profiles.

Visualizations Explained

All of the visualizations available for profile data show the unique set of stack traces captured during the profile, with the "area" of the stack indicating the proportional time spent in a function compared to that of its parent.

The Flame Graph visualization shows the time along the x-axis. The y-axis is used to show the function calls that make up a particular stack trace.

The Sunburst visualization is like the Flame Graph, where the x-axis is curved into a circle. Stack traces grow from the center to the outer parts of the graph.

The Treemap visualization shows time by area. The larger the rectangle, the more time a function took. The stack traces grow from the outer boxes into the inner boxes.

You can use the visualizations to find long-running functions. These can be identified by:

  • wide rectangles in the Flame Graph
  • long radial sections in the Sunburst
  • large areas in the Treemap

The stack trace height - how many functions deep a particular stack trace was - doesn't indicate a time issue, necessarily. You'll want to focus on the time values.

As you hover over the visualization itself, the method tied to the visualization will be show. If you click, you'll see a stack trace to the right of the visualization. Other areas in the visualization may "light up" with a less bright color as you hover over the visualization - these are entries in stack traces for the same function and indicates additional CPU usage for the function under the cursor.

So, the things to look for are stack trace entries that take up the most area: width in Flame Graph, circumference in Sunburst, and area in TreeMap.

Using the Chrome Profiler with N|Solid

You can also view the generated CPU profile in Chrome Dev Tools:

  • Click the Download Profile button on the left side of the screen
  • Open the Chrome web browser
  • Open Chrome Dev Tools from the menu View / Developer / Developer Tools
  • When the Developer Tools window (or pane) opens, select the tab Profiles
  • Click the Load button in the Select profiling type area
  • Find the .cpuprofile file that was downloaded, and click the Open button
  • Click on the profile that should now appear under CPU PROFILES list at the left
  • Click on Chart in the view selector under the Chrome Dev Tools menu
  • This will display a chart showing function execution times and what functions were on the stack at the time the system was sampled

Profiling Using the Command Line Interface (CLI)

N|Solid's Command Line Interface (CLI) is a great way to quickly pull profiles from remote processes for local examination. You can learn more about the profile command in the N|Solid Command Line Interface (CLI) reference.

Profiling Using the N|Solid Node API

If you have a desire to trigger a CPU Profile progamatically from within your application and have the resulting profile saved to your N|Solid console, you can do this using the N|Solid Node API.

const nsolid = require('nsolid')
nsolid.profile(duration_in_seconds, err => {
  if (err) {
    // The profile could not be started!
  }
})

Filtering and Tagging

Tags

You can use the NSOLID_TAGS environment variable to add tags that will be applied to a process. This can aid in identification and filtering, particularly in the Cluster view of the N|Solid Console. Tags can also be retrieved using the info command in the N|Solid CLI.

Setting Tags

$ NSOLID_APPNAME="leaderboard" NSOLID_TAGS="DC, us-east-1" nsolid app

Retrieving Tags

$ nsolid-cli info --force --app leaderboard

Example JSON Response

{
  "id": "57234c38b3f08dd8a3e9a567c82887a552c27b01",
  "app": "leaderboard",
  "tags": [ "DC", "us-east-1" ],
  "pid": 2440,
  ...
}

Tag Constraints

  • Tags can contain any whitespace characters
  • Tags are trimmed, so only whitespace in the middle of a tag is kept intact
  • Tags cannot contain commas and commas cannot be escaped
  • Tags are transmitted exactly the way they are entered and therefore are case sensitive, i.e. world is different than wOrld
  • The byte length of a tag must be between 2 and 140
  • There is no minimum number of tags per process
  • There is a maximum of 20 tags per process allowed
  • Tags cannot be changed at runtime

Filtering

As many applications are comprised of tens, if not hundreds or thousands, of processes it is often necessary to be able to filter a subset based on some criteria. The N|Solid Console offers a filtering mechanism with auto-complete to help you navigate your complex applications.

"Filter Search"

Filters

Typing just the filter name will list all of the items for that filter. Subsequent characters are then used to find partial matches within that filter.

"Supported Filters"

Here are the currently supported filters in the N|Solid Console:

Filter Name Description
host Hostname(s) for the selected application's processes
name Application name as specified by NSOLID_APPNAME
pid Process IDs (pids)
tag Custom process tags (if any) as specified by NSOLID_TAGS
title Process title(s)

You may enter multiple filters and search terms in the filter box. Multiple terms will be logically AND'ed together when they are applied to the set of processes being filtered.

Heap Snapshots

If you have a memory leak or performance issue, taking heap snapshots is a great way to help identify the underlying problem. N|Solid provides you with two ways to do this, a command-line friendly method and via our console.

Analyzing Heap Snapshots

Heap Snapshots capture a record of all live JavaScript objects available in your Node application, at the time the snapshot is captured. The objects are grouped by their constructor name, with aggregated counts of the total number of objects, their shallow size, and their retained size as columns in the Constructors view.

Shallow Size indicates the amount of space the object uses for its own properties. For instance, an object with no properties will have a very small shallow size, where an object with a number of properties will have a larger shallow size.

Retained Size indicates the amount of space the object is referencing, and keeping from being garbage collected.

From the Constructors view of the snapshot, you can sort by constructor name, number of objects, shallow size and retained size. You can also filter the list by constructor name.

You can use the number of objects value to look for the objects that are leaking, and you can use the retained size value to look for objects referencing those leaking objects.

In cases where you are leaking objects - and thus consuming more memory than you should be - sorting by number of objects and by retained size can provide a good indication of where to look for the cause of the leak. A large number of objects of a particular type may be an indication that these objects are still being referenced by other objects, when they could be garbage collected. A large retained size may be an indication that these objects are referencing more objects than they should be, keeping them from being garbage collected.

Taking Heap Snapshots with the N|Solid Console

Using N|Solid's Console, developers to take a series of snapshots and quickly switch between them to better identify problematic areas.

  1. Launch the console and select the application of interest "All applications"
  2. Select the process of interest "Select process"
  3. Expand out the details panel and select ‘+ New Snapshot’
    "New snapshot" This will take the snapshot and open the results in the snapshot window. To retake the snapshot, click the 'New Snapshot' button. The 'Related Snapshots' section in the previous screen provides a record of all the snapshots in the session. "Retake snapshot"
  4. Once the snapshot completes you can navigate through the various objects indexed during the snapshot process. As there can be a large number of indexed objects, you can use the Filter search box to narrow the results. Below you can see a listing of the various Array objects found in memory. To take another snapshot, simply press the New Snapshot button to the left. "Filter" Snapshots taken in the N|Solid Console can be downloaded to your local drive for exploration via a 3rd party tool (i.e. Chrome Developer Tools). If you wish to download the snapshot for analysis through other tools, simply click the Download Snapshot button "Download snapshot"

Taking Snapshots from the Command Line Interface (CLI)

N|Solid's Command Line Interface (CLI) is a great way to quickly take heap snapshots from remote processes for local examination. You can learn more about the snapshot command in the N|Solid Command Line Interface (CLI) reference.

Best Practices

Using constructors will show class names in heap snapshots. Since the heap snapshot groups objects by constructor name, if at all possible you should use named constructors for objects you'd like to track, as opposed to using literal objects. When using literal objects for trackable objects, those objects will be aggregated under the Object constructor name, along with all the other literal objects your program uses.

For instance, the following code snippet creates objects which will be categorized under the Object constructor:

trackableObject = {x: "some value", y: "another value"}

To be able to track these objects in a snapshot, use a specifically named class:

trackableObject = new TrackableObject()
trackableObject.x = "some value"
trackableObject.y = "another value"
// ...
function TrackableObject() {}

You'll then see these objects grouped by themselves in the heap snapshot.

If you'd like a bit more of an "inline" feel, you can enhance your constructor to take initialization values:

trackableObject = new TrackableObject({x: "some value", y: "another value"})
// ...
function TrackableObject(initialValue) {
  this.x = initialValue.x
  this.y = initialValue.y

With EcmaScript 6 parameter destructuring, you can make it a little shorter:

trackableObject = new TrackableObject({x: "some value", y: "another value"})
// ...
function TrackableObject({x, y}) {
  this.x = x
  this.y = y

You can also make the constructor open-ended regarding the properties:

trackableObject = new TrackableObject({x: "some value", y: "another value"})
// ...
function TrackableObject(initialValue) {
  for (var key in initialValue) {
    this[key] = initialValue[key]
  }
}

Snapshots Using the N|Solid Node API

If you want to create a Heap Snapshot progamatically from within your application and have the resulting snapshot saved to your N|Solid console, you can do this using the N|Solid Node API.

const nsolid = require('nsolid')
nsolid.snapshot(err => {
  if (err) {
    // The snapshot could not be created!
  }
})

Threshold Alerts

Threshold Alerts allow you to instruct N|Solid to take certain actions (such as taking heap snapshots or CPU profiles) when one or more processes cross over performance thresholds that you set.

Often, performance problems occur in production code non-deterministically and the only information you have available to diagnose the situation is in an error log or possibly a stack trace. With N|Solid's Threshold Alerts, you can automatically capture more useful information when problems arise in production.

Setting Threshold Alerts with the N|Solid Console

  1. From the main application page, click on an application to enter the cluster view. Click on Threshold Settings, located near the upper-right corner of the page.

  2. There are two types of thresholds to choose from: Heap Used and CPU Used. You may configure actions independently in both.

  1. To enable a threshold, click the switch immediately below the ENABLE THRESHOLD label.

  1. Depending on the threshold you selected, drag either the maximum heap percentage or maximum CPU percentage slider immediately below the switch. Set this value to the ceiling you do not wish your application's processes to exceed during normal operation. When enabled, there will be a dotted red line in the corresponding graphs that indicates where the threshold is set.

  1. Next, set the threshold delay. This setting allows you to prevent short blips of activity, which do not necessarily indicate an issue, from triggering a threshold.

  1. As threshold actions can themselves be resource intensive, the threshold snooze setting ensures that the performance of your application does not suffer due to excessive profiles or snapshots and that your inbox is not inundated with repeated notifications.

Note: When a threshold is set, a draggable bar for that threshold will appear in the cluster view. To change its value, click and drag the square handle at the center of the bar. To quickly open the Threshold Settings panel, double-click the square handle.

  1. Now that you have configured your threshold parameters, select one or more actions that N|Solid will perform when a threshold is triggered:

    1. Initiate CPU Profile: Select this option if you suspect that your application suffers from high CPU utilization. When enabled, N|Solid will start a CPU profile on your behalf and create an entry in the CPU profiles list in the process view.

  1. Take Heap Snapshot: Select this option if you suspect a memory leak. When enabled, N|Solid will take a heap snapshot on your behalf and create an entry in the memory profiles list in the process view.
  2. Send a Notification: Select this option to have alerts emailed to you. Separate multiple email addresses with a comma.

  1. Finally, don't forget to click click the Update button to save your settings.

Note: Be careful about setting a very high threshold, such as 99% Heap, and then selecting an action that uses the same resource, such as Take Heap Snapshot. Taking an intensive action at a very high ceiling is likely to negatively impact your application.

Security Vulnerabilities

The N|Solid Console can bet set up to perform periodic verification of all packages loaded by all N|Solid processes. All loaded packages are verified against a list of known vulnerabilities.

When new vulnerabilities are found, information about each vulnerability will be reported in the Console and optionally send notifications via email.

Detecting Security Vulnerabilities with the N|Solid Console

  1. Navigate to the Security view via the main menu.

    image

  2. This displays the Security view with a list of all the vulnerabilities across all applications in the left-hand sidebar.

    image

    1. Alternatively, you can narrow your search to a single application by going to the Applications view. Applications with identified vulnerabilities are highlighted in red.

      image

    2. Click on the View button for the selected vulnerable application.

      image

    3. This displays the Security view with a filtered list of all the vulnerabilities affecting the selected application in the left-hand sidebar.

      image

  3. Select the vulnerability you're interested in from the list on the left. If a patch is available, a link will be provided on this page to obtain more information.

    image

  4. The applications and processes that are affected by the selected vulnerability are listed on the right-hand side.

    image

Displaying Affected Packages in a Process

  1. Navigate to the Process Detail page for a vulnerable process.

    image

  2. Click the VIEW REPORT button in top-right-hand corner.

    image

  3. You will be taken to the report page for this process where a list of the affected packages is displayed.

    image

Receiving Notifications About New Vulnerabilities

The total number of vulnerabilities is displayed inside a red dot in the main menu.

image

Receiving Email Notifications for Security Vulnerabilities

  1. To send an email notification to one or more recipients when a vulnerability is detected, use the Notification Settings pane on the Security view. Access this panel by clicking the Notification Settings icon in the top-right hand corner.

    image

  2. Enter one or more mail addresses for the people you want to be notified when new vulnerabilities are discovered. Each email recipient will be notified only once about any particular vulnerability. All existing vulnerabilities will be reported to any new email addresses added to this list.

    image

  3. Click Update to Save.

N|Solid in the cloud

AWS Quick Start Guide

This guide provides step-by-step instructions for deploying N|Solid on the Amazon Web Services (AWS) cloud. Quick Starts are automated reference deployments that use AWS CloudFormation templates to launch, configure, and run the AWS compute, network, storage, and other services required to deploy a specific workload on AWS.

Installing N|Solid Console

  1. Log in to the AWS Marketplace at http://aws.amazon.com/marketplace.
  2. Open the N|Solid Console - AMI page and choose Continue.
  3. Review the settings and read the terms and conditions for software usage, and then click Launch with 1-click.
  4. Open AWS Management Console and wait for the instance to be in a running state.
  5. Once running, open the Public DNS or Public IP in a web browser. You will be prompted for a username: nsolid and a password: nodesource.

Setting up N|Solid Console

Complete the AWS activation screen.

N|Solid AWS start screen

Once completed, you should be immediately redirected to the N|Solid console.

N|Solid Console Up and Running

Running Node Applications in N|Solid

ssh into the machine as ubuntu user.

NSOLID_COMMAND=localhost:9001 \
NSOLID_DATA=localhost:9002 \
NSOLID_BULK=localhost:9003 \
NSOLID_APPNAME="My_Application" node app.js

You can customize the name of your application in the N|Solid Console by changing the NSOLID_APPNAME environmental variable. If you do not specify an app name, N|Solid will look for a name property in your package.json, and if that is not found your application will be named untitled application

You are now ready to use N|Solid for your own applications!

Creating instances to use N|Solid Console

  1. Log in to the AWS Marketplace at http://aws.amazon.com/marketplace.
  2. Open the N|Solid Runtime Argon - AMI page and choose Continue.
  3. Review the settings and read the terms and conditions for software usage, and then click Launch with 1-click.
  4. Open AWS Management Console and wait for the instance to be in a running state.
  5. Utilize the Public DNS or IP from N|Solid Console instance to start your application.
NSOLID_COMMAND=<NSOLID_CONSOLE_IP>:9001 \
NSOLID_DATA=<NSOLID_CONSOLE_IP>:9002 \
NSOLID_BULK=<NSOLID_CONSOLE_IP>:9003 \
NSOLID_APPNAME="My_Application" node app.js

Configuration of N|Solid Console install.

Process Management

  • Nginx: /etc/init.d/nginx
  • N|Solid Console: /etc/init.d/nsolid-console
  • N|Solid Storage: /etc/init.d/nsolid-storage

Nginx

The configuration for the external proxy is located at /etc/nginx/sites-enabled/nsolid-nginx.conf if you want to change the default login credentials.

N|Solid Storage

The default configuration file is located at /etc/nsolid/storage-config.toml will all artifacts stored at /var/lib/nsolid/storage.

Microsoft Azure Quickstart Guide

This guide provides step-by-step instructions for deploying N|Solid on the Microsoft Azure cloud. You can find N|Solid on the Azure Marketplace, which allows quick, easy, and customized deployments of N|Solid directly to your Azure-based cloud to help you get up and running with a secure, reliable, and extensible enterprise-grade Node.js runtime in the cloud.

Installing the N|Solid Console from the Azure Marketplace

To get started with N|Solid on Azure, you'll need to ensure you have a Microsoft account. Once you've got an account, you'll need to navigate to the Azure Marketplace.

  • Login into the Microsoft Azure Portal.
    • If you don't already have a Microsoft account you can create one for free.
  • Access Marketplace, as seen on the dashboard.
    • The Marketplace can also be accessed from the search bar at any time.

  Once you're in the Azure Marketplace, you'll be able to search for - and deploy - the N|Solid Console and Runtime directly to Azure.

  • Search the Azure Marketplace for nsolid.
  • Select the N|Solid Console from the search results.
    • Click the Create button to create a new VM with the N|Solid Console.
  • Fill in the required fields, and customize optional fields to your desired specifications.
    • Name
    • Username
    • Authentication Type
    • Resource Group
  • Select a VM Size
    • We suggest DS3_V2 or larger VMs for production usage.
  • Once “N|Solid Console” is up and running open the ip address in a browser.
    • You will be prompted to access it with Basic Auth
    • The default username is nsolid and the default password is nodesource

      Note: The configuration for the external proxy that provides the basic authentication is located at /etc/nginx/sites-enabled-nsolid-nginx.conf in the N|Solid Console instance. If you'd like to change the default login credentials (we highly suggest you do), you can do so there.

The N|Solid Console should be up and running! Now, you'll need to provide an N|Solid License key or request one through the console, which will be sent by email.

Creating Instances of the N|Solid Runtime

To start running your apps with N|Solid, you'll need to start VMs with instances of the N|Solid Runtime. To create instances of the N|Solid Runtime, you can follow the same process you did for the N|Solid Console in the Azure Marketplace:

  • Search the Azure Marketplace for nsolid.
  • Select the N|Solid Runtime from the search results.
    • Click the Create button to create a new VM with the N|Solid Runtime.
  • Fill in the required fields, and customize optional fields to your desired specifications.
    • Name
    • Username
    • Authentication Type
    • Resource Group
  • Select a VM Size
    • For the N|Solid Runtime, select whatever VM Size your application or project needs

Note: More than likely you will want to use the same resource group as the “N|Solid Console” instance.

The N|Solid Runtime instances are Linux machines that have the bare essentials for running Node.js. If there's additional tooling you need for your application, you can SSH into the machine and install it.

Connecting the N|Solid Runtime to the N|Solid Console on Azure

To connect an N|Solid Runtime to your N|Solid Console, SSH into your N|Solid Runtime instance, and run the following commands:

NSOLID_COMMAND={nsolid_console_ip_address}:9001 \
NSOLID_DATA={nsolid_console_ip_address}:9002 \
NSOLID_BULK={nsolid_console_ip_address}:9003 \
NSOLID_APPNAME="My Application Name" node app.js

In the above code block, you will need to replace {nsolid_console_ip_address} with the IP address of your N|Solid Console. You will also want to replace "My Application Name" with the name of your application, as you want it to be displayed in the N|Solid Console.

You can connect multiple N|Solid Runtimes to your N|Solid Console, enabling you to monitor all your Node.js deployments from a single dashboard.

Configuration of N|Solid Console install.

Process Management

  • Nginx: /etc/init.d/nginx
  • N|Solid Console: /etc/init.d/nsolid-console
  • N|Solid Storage: /etc/init.d/nsolid-storage

Nginx

The configuration for the external proxy is located at /etc/nginx/sites-enabled/nsolid-nginx.conf if you want to change the default login credentials.

N|Solid Storage

The default configuration file is located at /etc/nsolid/storage-config.toml will all artifacts stored at /var/lib/nsolid/storage.

How to Install N|Solid on GCP with Google Cloud Launcher

This guide provides step-by-step instructions for deploying N|Solid on the Google Cloud Platform (GCP) cloud. Cloud Launcher are automated reference deployments that use GCP Deployment Manager to configure, launch, and run the GCP compute engine, network, storage, and other services required to deploy a specific workload on GCP.

Locate N|Solid in the Google Cloud Launcher and click “Launch on Compute Engine”

N|Solid GCP locate screen

Details for your new N|Solid deployment will be displayed. Adjust settings to your desire and click “Deploy”

N|Solid GCP setup screen

Wait for setup to complete

N|Solid GCP pending screen

Click the "Log into N|Solid Console" button and provide the username / password listed.

N|Solid GCP complete screen

Setting up N|Solid Console

Complete the GCP activation screen.

N|Solid GCP start screen

Once completed, you should be immediately redirected to the N|Solid console.

N|Solid Console Up and Running

Running Node Applications with N|Solid

ssh into a runtime machine..

NSOLID_APPNAME="My_Application" node app.js

You can customize the name of your application in the N|Solid Console by changing the NSOLID_APPNAME environmental variable. If you do not specify an app name, N|Solid will look for a name property in your package.json, and if that is not found your application will be named untitled application

You are now ready to use N|Solid for your own applications!

Configuration of N|Solid Console install.

Process Management

  • Nginx: /etc/init.d/nginx
  • N|Solid Console: /etc/init.d/nsolid-console
  • N|Solid Storage: /etc/init.d/nsolid-storage

Nginx

The configuration for the external proxy is located at /etc/nginx/sites-enabled/nsolid-nginx.conf if you want to change the default login credentials. See Configuring SSL for adding security to N|Solid Console.

N|Solid Storage

The default configuration file is located at /etc/nsolid/storage-config.toml will all artifacts stored at /var/lib/nsolid/storage.

Advanced Topics

Command Line Interface (CLI)

N|Solid comes with a set of predefined endpoints for interaction and introspection that can be accessed at the command-line using the N|Solid Command Line Interface, nsolid-cli.

To see a list of all commands and options available with nsolid-cli, use the -h option:

nsolid-cli -h

The output of the nsolid-cli commands, unless otherwise noted, is line-delimited JSON (each line is separated by a newline delimiter \n). The output shown in the examples below is shown expanded for readability.

--start and --end options

For commands that take --start and --end options, you can pass the following formats for the date/time value:

format description
milliseconds since epoch value returned from Date.now()
yyyy-mm-ddThh:mm:ss ISO date
-Ns N seconds since now
-Nm N minutes since now
-Nh N hours since now
-Nd N days since now
0 now, when used with the --end option

The --end option can only be specified if the --start option is also specified. If the --end option is not specified, the command will continue returning JSON objects till terminated.

application

Returns objects which contains summary information about all the processes in the specified application. The current object will be returned on the specified interval.

Options Description
--app Application name (required)
--interval Number of seconds before returning next current object (default: 1)

Usage

nsolid-cli application --app=my-app-name

Example JSON Result

{
  "time": 1474520062141,
  "app": "my-app-name",
  "hostCount": 1,
  "processCount": 1,
  "processes": [
    {
      "id": "7e5c0f931033a6b1cd8e037b8ea099f9cf78af65",
      "app": "my-app-name",
      "hostname": "my-computer.local",
      "tags": [
        "region:north",
        "zone:A"
      ],
      "cpu": 0.007,
      "heapUsed": 19178512,
      "title": "my-app-name",
      "uptime": 4697.88,
      "pid": 23888,
      "processStart": 1474515364290,
      "packageCount": 80,
      "vulnerabilityCount": 0,
      "thresholds": {}
    }
  ],
  "vulnerabilities": [],
  "vulnerabilityCount": 0
}

Returns a stream of the following objects:

Property Description
time Millisecond since epoch time message was sent
app The NSOLID_APPNAME value
hostCount The number of hosts this application is running on
processCount The number of processes running this application
processes Array of process objects, described below
vulnerabilityCount The number of vulnerabilities found in all processes
vulnerabilities An array of vulnerabilities found

Process objects:

Property Description
id The agent id
app The NSOLID_APPNAME value
hostname The host the process is running on
tags The NSOLID_TAGS values
cpu % cpu being used by the process
heapUsed The size of the JavaScript heap used (bytes)
title The process title
uptime The time in seconds this process has been running
pid The operating system process id
processStart The time the process started
packageCount The number of packages loaded
vulnerabilityCount The number of vulnerabilities found
thresholds Threshold data

applications

Return objects which contains summary information about all the applications available. The current object will be returned on the specified interval.

Options Description
--interval Number of seconds before returning next current object (default: 1)

Usage

nsolid-cli applications

Example JSON Result

{
  "time": 1474520766732,
  "applications": [
    {
      "processCount": 1,
      "vulnerabilityCount": 0,
      "name": "my-app-name",
      "hostCount": 1
    }
  ]
}

Returns a stream of the following objects:

Property Description
time Millisecond since epoch time message was sent
applications Array of application objects, described below

Application objects:

Property Description
name The NSOLID_APPNAME value
processCount The number of processes running this application
vulnerabilityCount The number of vulnerabilities found
hostCount The number of hosts this application is running on

asset

Download an asset.

Options Description
--asset The asset id (required)

Usage

nsolid-cli asset --asset 217040c0-02d1-4956-8648-7fb84b78c65e > my.heapsnapshot

Asset id's are available from the assets command. The bytes of the asset will be written to stdout.

assets

Lists the assets - CPU profiles and heap snapshots - that are currently available for download.

Options Description
--id The agent id
--app The NSOLID_APPNAME value
--hostname The host the process is running on
--tag An NSOLID_TAGS value (may be specified multiple times)
--type either 'snapshot' or 'profile', to limit the type of asset returned
--start Return assets that were created after the specified time
--end Return assets that were created before the specified time

Usage

nsolid-cli assets --app my-app-name --type snapshot

Example JSON Result

{
  "time": 1476464230018,
  "asset": "217040c0-02d1-4956-8648-7fb84b78c65e",
  "type": "snapshot",
  "id": "a6d4acf3e713043a293b3fdbd5df9bac885c727a",
  "app": "my-app-name",
  "hostname": "my-computer.local",
  "tags": [
    "region:north",
    "zone:A"
  ],
  "size": 12206681,
  "pid": 77465,
  "title": "my-app-name"
}

Returns a stream of the following objects:

Property Description
time Millisecond since epoch time message was sent
asset an asset id to use the the asset command
type 'snapshot' for heap snapshots, 'profile' for CPU profiles
id The agent id
app The NSOLID_APPNAME value
hostname The host the process is running on
tags The NSOLID_TAGS values
size The size of the asset in bytes
pid The operating system process id
title The process title

async-activity

N|Solid can list any pending asynchronous operations and where the JavaScript callbacks will resume. This can be extremely helpful for determining how many concurrent operations a given N|Solid process is handling and what they are. It can also be useful to determine where an application is hanging, if there are any left-behind callbacks or a misue of resources (e.g. reading the same file repetitively)

The output will list the in-flight asynchronous activity, including the function location and line of source code to reference where the callback will resume.

Options Description
--id The agent id
--app The NSOLID_APPNAME value
--hostname The host the process is running on
--tag An NSOLID_TAGS value (may be specified multiple times)

Usage

nsolid-cli async-activity

Example JSON Result

{
  "handles": [
    {
      "name": "onrequest",
      "location": {
        "file": "nsolid-user/server-http.js",
        "line": 3,
        "column": 19,
        "inferredName": ""
      },
      "anonymous": false,
      "code": "function onrequest(req, res) { res.end('ok\n') }",
      "type": {
        "group": "handle",
        "flag": 4,
        "name": "TCP server connection"
      },
      "details": {
        "fd": 10,
        "transport": "TCP",
        "connectionKey": "6::::0",
        "connections": 0,
        "unrefd": false
      },
      "metadata": {
        "async_wrap_provider": "TCPWRAP",
        "async_wrap_providerKey": "15"
      }
    },
    {
      "type": {
        "group": "handle",
        "flag": 64,
        "name": "pipe socket"
      },
      "details": {
        "fd": 1,
        "transport": "Pipe",
        "bytesDispatched": 0,
        "hadError": false,
        "host": null,
        "pendingData": "<Buffer>"
      },
      "metadata": {
        "async_wrap_provider": "PIPEWRAP",
        "async_wrap_providerKey": "8"
      }
    },
    {
      "type": {
        "group": "handle",
        "flag": 64,
        "name": "pipe socket"
      },
      "details": {
        "fd": 2,
        "transport": "Pipe",
        "bytesDispatched": 57985,
        "hadError": false,
        "host": null,
        "pendingData": "<Buffer>"
      },
      "metadata": {
        "async_wrap_provider": "PIPEWRAP",
        "async_wrap_providerKey": "8"
      }
    }
  ],
  "requests": [
    {
      "name": "onreadFile",
      "location": {
        "file": "nsolid-user/server-http.js",
        "line": 14,
        "column": 20,
        "inferredName": ""
      },
      "anonymous": false,
      "code": "function onreadFile(err, src) {\n  if (err) return console.error(err)\n  readAgain()\n}",
      "type": {
        "group": "request",
        "flag": 4,
        "name": "readFile",
        "api": "fs.readFile(filename[, options], callback)"
      },
      "details": {
        "path": "nsolid-user/server-http.js",
        "flag": "r",
        "isUserFd": false,
        "buffers": [
          "<Buffer>"
        ],
        "buffer": "<Buffer>",
        "pos": 0,
        "encoding": null,
        "err": null
      },
      "metadata": {
        "async_wrap_provider": "FSREQWRAP",
        "async_wrap_providerKey": "3"
      }
    }
  ],
  "pending": [],
  "time": 1474407457414,
  "id": "8ed04c57f248624c6068945b68a5f023497498cd",
  "app": "runtime/async-activity/03-server-http_app",
  "hostname": "ebc192f5fae2"
}

Returns a stream of the following objects:

Property Description
handles tend to be longer-lived larger-scale asynchronous operations, such as open sockets or timers
requests tend to be shorter-lived smaller-scale operations such as writing to file handles and other file-related operations
pending represent async activity that is lower level and a lot of times are created by cod inside core, like DNS lookups, signal handlers and TCP connections

custom

Invoke a custom command. For more information on custom commands, see Custom Commands.

Options Description
--id The agent id (required)
--name The name of the custom command (required)
--data Data to be sent with the command

Usage

nsolid-cli custom --id=[agent id] --name=verbose --data=off

Example JSON Result

{
  "time": 1476754051460,
  "id": "81535293aea1fe8c1e2f3f7518d8db3f96cf7b39",
  "app": "nsolid2",
  "hostname": "pmuellr-MacBook-Pro.local",
  "tags": [
    "foobar"
  ],
  "result": {
    "verbose": false
  }
}

Returns a JSON object with the following properties

Property Description
time Millisecond since epoch time message was sent
id The agent id
app The NSOLID_APPNAME value
hostname The host the process is running on
tags The NSOLID_TAGS values
result The result of the custom command

generate-keypair

Return a randomly generated keypair suitable for use in the N|Solid Storage socket configuration. If your N|Solid Storage instance is running on an untrusted network, it is recommended that you generate and use new keys.

Usage

nsolid-cli generate-keypair

Example JSON Result

{
  "public": "[t&m}{EZH7=HR(IW:+Ttk:=r.Y$:CP+-Q&5L?2N!",
  "private": "4QZof={^Pman?I?mB0o!]%z/{Jlu6:mJfl[Ms@[^"
}

Returns a JSON object with the following properties

Property Description
public Public key value, publicKey in N|Solid Storage configuration and env var NSOLID_PUBKEY for N|Solid Runtime
private Private key value, privateKey in N|Solid Storage configuration

info

Returns objects which contain static information about processes and the hosts they are running on.

Options Description
--id The agent id
--app The NSOLID_APPNAME value
--hostname The host the process is running on
--tag An NSOLID_TAGS value (may be specified multiple times)
--field Output field (may be specified multiple times)
--start Return objects for processes that started after the specified time
--end Return objects for processes that started before the specified time

Usage

nsolid-cli info

Example JSON Result

{
  "id": "7e5c0f931033a6b1cd8e037b8ea099f9cf78af65",
  "tags": [
    "region:north",
    "zone:A"
  ],
  "pid": 23888,
  "processStart": 1474515364290,
  "execPath": "/Users/me/nsolid",
  "main": "/Users/me/Projects/my-app-name.js",
  "arch": "x64",
  "platform": "darwin",
  "totalMem": 17179869184,
  "versions": {
    "http_parser": "2.7.0",
    "node": "6.5.1",
    "nsolid": "2.0.0",
    "v8": "5.1.281.81",
    "uv": "1.9.1",
    "zlib": "1.2.8",
    "ares": "1.10.1-DEV",
    "icu": "57.1",
    "modules": "48",
    "openssl": "1.0.2h",
    "nsolid_lib": {
      "v8_profiler": "nsolid",
      "sodium": "nsolid",
      "cjson": "nsolid",
      "function_origin": "nsolid",
      "nan": "v2.3.5",
      "cli": "master",
      "agent": "master",
      "zmq-bindings": "nsolid",
      "zmq": "nsolid",
      "persistents_with_classid": "v1.1.0"
    }
  },
  "cpuCores": 8,
  "cpuModel": "Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz",
  "time": 1474520894162,
  "app": "my-app-name",
  "appVersion": "1.0.0",
  "hostname": "my-computer.local"
}

Returns a stream of the following objects:

Property Description
time Millisecond since epoch time message was sent
id The agent id
app The NSOLID_APPNAME value or name property from package.json
appVersion The version property from package.json
hostname The host the process is running on
tags The NSOLID_TAGS values
pid Operating system process id
processStart The time the process started
execPath Path of the executable running the application
main The main module used when the application started up
arch The CPU architecture
platform Name of the N|Solid platform
totalMem Total available memory in the system
cpuCores The number of CPU cores
cpuModel The CPU model
versions Object describing the versions of components used in the runtime

info with the --force option

Requests current information from each running agent, or specified agents if using a filter.

Options Description
--id The agent id
--app The NSOLID_APPNAME value
--hostname The host the process is running on
--tag An NSOLID_TAGS value (may be specified multiple times)
--field Output field (may be specified multiple times)

list

Returns an array of all available matching N|Solid processes, along with their most recent info and metrics data.

Options Description
--id The agent id
--app The NSOLID_APPNAME value
--hostname The host the process is running on
--tag An NSOLID_TAGS value (may be specified multiple times)

Usage

nsolid-cli list

Example JSON Result

{
  "id": "7e5c0f931033a6b1cd8e037b8ea099f9cf78af65",
  "app": "my-app-name",
  "hostname": "my-computer.local",
  "tags": [
    "region:north",
    "zone:A"
  ],
  "info": { ... output from the info command ... },
  "metrics": { ... output from the metrics command ... },
}

Returns an array of the following object shapes:

Property Description
info The object returned from the info command
metrics The object returned from the metrics command

metrics

Returns a stream of metrics records matching the query string parameters.

Options Description
--id The agent id
--app The NSOLID_APPNAME value
--hostname The host the process is running on
--tag An NSOLID_TAGS value (may be specified multiple times)
--field Output field (may be specified multiple times)
--start Return metrics created after the specified time
--end Return metrics created before the specified time

Usage

nsolid-cli metrics

Example JSON Result

{
  "time": 1474521264390,
  "uptime": 5900.431,
  "rss": 88784896,
  "heapTotal": 26275840,
  "heapUsed": 22151064,
  "activeRequests": 0,
  "activeHandles": 7,
  "title": "my-app-name",
  "user": "me",
  "cpu": 0.874302319950784,
  "freeMem": 190222336,
  "systemUptime": 1705341,
  "load1m": 2.20068359375,
  "load5m": 1.9150390625,
  "id": "7e5c0f931033a6b1cd8e037b8ea099f9cf78af65",
  "load15m": 1.81689453125,
  "cpuSpeed": 2500,
  "app": "my-app-name",
  "hostname": "my-computer.local",
  "percentCpuThreshold": -1,
  "percentHeapThreshold": -1,
  "percentHostMemoryUsedByProcess": 0.0052,
  "percentHostMemoryUsed": 0.9889,
  "totalHeapSizeExecutable": 8388608,
  "totalPhysicalSize": 28397568,
  "totalAvailableSize": 1471514720,
  "heapSizeLimit": 1501560832,
  "loopIdlePercent": 42.59350634392232,
  "loopEstimatedLag": 47.882432888888886,
  "loopsPerSecond": 11.989051138919644,
  "loopAvgTasks": 1.5833333333333333,
  "loopTotalCount": 5695,
  "blockInputOpCount": 0,
  "blockOutputOpCount": 4,
  "cpuPercent": 0.874302319950784,
  "cpuSystemPercent": 0.4728451232242015,
  "cpuUserPercent": 0.40145719672658253,
  "gcForcedCount": 0,
  "gcFullCount": 0,
  "gcCount": 233,
  "gcCpuPercent": 0.03792483595186012,
  "gcDurUs99Ptile": 3380,
  "gcDurUsMedian": 524,
  "gcMajorCount": 14,
  "pageFaultHardCount": 8,
  "pageFaultSoftCount": 106091,
  "ctxSwitchInvoluntaryCount": 16338,
  "ctxSwitchVoluntaryCount": 2061,
  "ipcReceivedCount": 10347,
  "ipcSentCount": 10383,
  "signalCount": 0,
  "swapCount": 0,
  "tags": [
    "region:north",
    "zone:A"
  ]
}

Consult the Metrics in Detail section for complete details on the metrics available.

metrics with the --force option

Requests current metrics from each running agent, or specified agents if using a filter.

Options Description
--id The agent id
--app The NSOLID_APPNAME value
--hostname The host the process is running on
--tag An NSOLID_TAGS value (may be specified multiple times)
--field Output field (may be specified multiple times)

packages

Return the list of packages and modules available in the specified process.

Options Description
--id The agent id (required)

Usage

nsolid-cli packages --id=[agent id]

Example JSON Result

{
  "packages": [
    {
      "path": "/Users/me/my-app-name",
      "name": "my-app-name",
      "main": "my-app-name",
      "version": "1.0.0",
      "dependencies": [
        "node_modules/async",
        "node_modules/bole",
        "node_modules/client-request",
        "node_modules/lodash",
        "node_modules/marked",
        "node_modules/minimist",
        "node_modules/mkdirp",
        "node_modules/semver",
        "node_modules/stringcase",
        "node_modules/toml",
        "node_modules/user-home",
        "node_modules/uuid"
      ],
      "modules": [
        {
          "path": "index.js",
          "shasum": "b76d1c3b6af5515dca0b36ed0ef87b248d9acc94"
        }
      ]
    }
  ]
}

Returns an object with the following shape:

Property Description
time Millisecond since epoch time message was sent
id The agent id
app The NSOLID_APPNAME value
hostname The host the process is running on
tags The NSOLID_TAGS values
packages A packages object, described below

Packages objects:

Property Description
path The path the package was loaded from
name The name of the package
version The version of the package
main The main module for the package
dependencies An array of relative package names this package depended on
modules An array of module objects, described below

Modules objects:

Property Description
path The package-relative path name of the module loaded
shasum The SHASUM hash of the module contents

ping

Have all matching processes send a simple response back.

Options Description
--id The agent id
--app The NSOLID_APPNAME value
--hostname The host the process is running on
--tag An NSOLID_TAGS value (may be specified multiple times)

Usage

nsolid-cli ping

Example JSON Result

{
  "pong": true,
  "id": "7e5c0f931033a6b1cd8e037b8ea099f9cf78af65",
  "time": 1474521577972
}

Returns a stream of the following objects:

Property Description
time Millisecond since epoch time message was sent
id The agent id
pong Will always have the value true

profile

Have a CPU profile taken in in the specified process.

Options Description
--id The agent id (required)
--duration Duration of profile in seconds. Default is 5 seconds

Usage

nsolid-cli profile --id=[agent id] > my.cpuprofile

Returns a V8 CPU profile.

Once the profile file has been created, it can be opened using Chrome’s Development Tool’s CPU Profile Debugging Tool. It’s important to note that Chrome requires the generated file to have the extension .cpuprofile to load the file.

snapshot

Have a heap snapshot taken in in the specified process.

Options Description
--id The agent id (required)

Usage

nsolid-cli snapshot --id=[agent id] > my.heapsnapshot

Returns a V8 heap snapshot.

Once the snapshot file has been created, it can be opened using Chrome’s Development Tool’s heap snapshot browser It’s important to note that Chrome requires the generated file to have the extension .heapsnapshot to load the file.

startup-times

Lists the time from initial process execution to reach certain process lifecycle startup phases.

Options Description
--id The agent id
--app The NSOLID_APPNAME value
--hostname The host the process is running on
--tag An NSOLID_TAGS value (may be specified multiple times)

Usage

nsolid-cli startup-times

Example JSON Result

{
  "time": 1476755309783,
  "id": "7e5c0f931033a6b1cd8e037b8ea099f9cf78af65",
  "app": "my-app-name",
  "hostname": "my-computer.local",
  "initialized_node": [
    0,
    166736
  ],
  "initialized_v8": [
    0,
    366153
  ],
  "loaded_environment": [
    0,
    80486563
  ]
}

Returns a stream of the following objects:

Property Description
time Millisecond since epoch time message was sent
id The agent id
app The NSOLID_APPNAME value
hostname The host the process is running on
initialized_node The time it took to initialize the Node internals
initialized_v8 The time it took to initialize the V8 engine
loaded_environment The time it took to complete all initialization, which includes running some of node's internal JavaScript code, and your main module's top-level code

The times are provided in hrtime syntax: an array of 2 integers representing the time since the initial process execution in [seconds, nanoseconds].

Additional timers can be added to your application with custom lifecycle events.

vulnerabilities

Returns the known security vulnerabilities of all processes.

Options Description
--interval Number of seconds before returning next current object (default: 1)

Usage

nsolid-cli vulnerabilities

Example JSON Result

{
  "time": 1476755941704,
  "vulnerabilities": [
    {
      "package": "moment",
      "title": "Regular Expression Denial of Service",
      "published": "2016-02-01T19:00:03.862Z",
      "credit": [
        "Adam Baldwin"
      ],
      "ids": {
        "CWE": [
          "CWE-400"
        ],
        "CVE": [],
        "NSP": 55,
        "ALTERNATIVE": [
          "SNYK-JS-MOMENT-10084"
        ]
      },
      "vulnerable": "<=2.11.1",
      "description": "markdown descript of vulnerability",
      "apps": [
        {
          "appname": "my-app-name",
          "processes": [
            {
              "id": "7e5c0f931033a6b1cd8e037b8ea099f9cf78af65",
              "hostname": "my-computer.local",
              "pid": 23888,
              "loaded": [
                {
                  "version": "2.11.0",
                  "path": "/Users/me/Projects/my-app-name/node_modules/moment"
                }
              ]
            }
          ]
        }
      ],
      "patches": [
        "... url to patches ..."
      ]
    }
  ]
}

Returns a stream of the following objects:

Property Description
time Millisecond since epoch time message was sent
vulnerabilities An array of vulnerability objects, described below

Vulnerability objects:

Property Description
package the package with the vulnerability
title the title of the vulnerability
published date the vulnerability was published
credit names of the people credited with finding the vulnerability
ids identifiers for the vulnerability; CVE, CWE, etc
vulnerable semver expression describing what versions of the package are vulnerable
description description of the vulnerability, in markdown
apps an array of application objects, describing which applications and processes are running the vulnerable module, along with the version loaded and path to the package
patches link to patches available for the vulnerability

Metrics in Detail

A rich set of metrics is available from N|Solid covering the Node.js process' host system, the process itself, the internal behavior of Node.js and the internal behavior of the V8 JavaScript engine. The combination of many of these data points can give you sophisticated insight into your Node.js deployment.

Many of the key metrics are displayed directly on the N|Solid Console. Most numeric metrics are also provided to your StatsD collector directly from the N|Solid process if you configure using the NSOLID_STATSD environment variable (see the StatsD section for more detail). For the full set of metrics, use the nsolid-cli utility.

Application details

Property Description
time Millisecond since epoch time message was sent (not available via StatsD)
id The agent id (not available via StatsD)
app The NSOLID_APP environment variable value, or "app" value from the "nsolid" object in package.json (not available via StatsD)
tags The NSOLID_TAGS environment variable value, or "tags" value from the "nsolid" object in package.json (not available via StatsD)

System data

Property Description
hostname The host the process is running on (not available via StatsD)
freeMem The amount of free (unused) memory in bytes
systemUptime The uptime of the system, in seconds
load1m The one-minute load average
load5m The five-minute load average
load15m The fifteen-minute load average
cpuSpeed The current speed of the CPU (averaged across all cores) in MHz
percentHostMemoryUsed Percent of memory used by all processes

General process stats

Property Description
uptime The time in seconds this process has been running
user The user this process is running as (not available via StatsD)
title The current process title (not available via StatsD)
rss The total Resident Set Size (total memory used) by this process (bytes)
cpuPercent % CPU used by the process
cpu Deprecated: refer to cpuPercent instead
cpuSystemPercent % CPU used by the process in system calls
cpuUserPercent % CPU used by the process in user code
percentHostMemoryUsedByProcess Percent of memory used by process
ctxSwitchInvoluntaryCount The number of involuntary context switches
ctxSwitchVoluntaryCount The number of voluntary context switches
ipcReceivedCount The number of IPC messages received
ipcSentCount The number of IPC messages sent
signalCount The number of signals received
pageFaultSoftCount The number of soft page faults (page reclaims)
pageFaultHardCount The number of hard page faults
swapCount The number of times the process has been swapped out of memory
blockInputOpCount The total number of block input operations
blockOutputOpCount The total number of block output operations

JavaScript memory stats

Property Description
heapTotal The total allocated size of the JavaScript heap (bytes). A subset of rss
heapUsed The amount of heapTotal being used by JavaScript (bytes)
totalHeapSizeExecutable The amount of executable memory allocated in the heap
totalPhysicalSize The amount of phyical memory currently committed for the heap
totalAvailableSize The amount of heap available without growing the heap
heapSizeLimit The the maximum amount of memory reserved for the heap

Asynchronous activity stats

Property Description
activeRequests The number of active requests the event loop will process. For more information see the async-activity command.
activeHandles The number of active handles the event loop will process. For more information, see the async-activity command.

Threshold stats

Property Description
percentCpuThreshold Percent of CPU threshold used (or -1 if no threshold set)
percentHeapThreshold Percent of heap threshold used (or -1 if no threshold set)

Node.js event loop stats

Property Description
loopIdlePercent Percent of time that Node is waiting for I/O ("idle")
loopEstimatedLag The amount of time a I/O response may have to wait before being processed
loopsPerSecond The amount of event loop "turns" elapsed in the last second
loopAvgTasks The average amount of async JavaScript entries per loop
loopTotalCount A cumulative count of all event loop "turns"

JavaScript garbage collection stats

Property Description
gcCount The total number of garbage collections
gcFullCount The number of garbage collections collecting all available garbage
gcMajorCount The number of significant garbage collections
gcForcedCount The number of forced garbage collections (for testing)
gcCpuPercent % CPU used during garbage collection
gcDurUs99Ptile The 99th quantile of garbage collection durations
gcDurUsMedian The median of garbage collection durations

Custom Commands

As a developer, you can trigger your own custom commands via the N|Solid Command Line Interface (CLI). Custom commands allow you to interact with your application's processes in ways specific to your business needs.

You can implement a custom command by creating a function to handle the command, and then registering the function with N|Solid.

The custom command function should be defined to take a single parameter, request:

function customCommandHandler(request) { ... }

The request parameter is an object with the following properties/functions:

Parameter Description
request.value An optional piece of data sent with the command, using the nsolid-cli parameter --data.
request.return(value) The function to call when you are ready to return the result of this command. The Agent will reply with the value passed as a parameter.
request.throw(error) A function to call if an error is encountered. The Agent will reply with the error passed as a parameter.

Your function you must call either request.return() or request.throw() to signal completion of the command.

A custom command handler is registered using the nsolid.on() method. You can get access to N|Solid's built-in nsolid module, by calling require("nsolid"). The nsolid.on() function takes the following parameters:

nsolid.on(commandName, handler)
Parameter Description
commandName The string name of the command to implement
handler The custom command function implementing the command

The structure of implementing and registering a custom command will look like the code below:

...
const nsolid = require("nsolid")
...
nsolid.on("foo", fooCommand)
...
function fooCommand(request) { ... }

Example: Log Level Custom Command

Below is an example of how you can use custom commands to dynamically change the configuration state of the application; specifically the log level. The example assumes that a global boolean variable Verbose is used to indicate whether to log verbosely or not.

//------------------------------------------------------------------------------
// This program is a simple "server" which does nothing, but does implement
// an N|Solid custom command, named `verbose`.  Once you've started this program
// with the N|Solid agent enabled, you can send the `verbose` command as in:
//
// nsolid-cli --id $NSOLID_AGENTID custom --name verbose
// nsolid-cli --id $NSOLID_AGENTID custom --name verbose --data on
// nsolid-cli --id $NSOLID_AGENTID custom --name verbose --data off
//
// All these forms get or set the "verbose" level of logging in the program.
//
// The server logs a message every second when "verbose" is off, and logs
// an additional message after that one when "verbose" is on.  The default
// setting of "verbose" is false.
//------------------------------------------------------------------------------

"use strict"

// get access to N|Solid's built-in module `nsolid`
const nsolid = require("nsolid")

// the current "verbose" level
let Verbose = false

// register the `verbose` command for nsolid-cli
nsolid.on("verbose", verboseCommand)

// your server which doesn't do much
setInterval(onInterval, 2000)

console.log("N|Solid custom command demo - log-level - started")
console.log("")
console.log("to use the verbose command with `nsolid-cli`, run:")
console.log("   nsolid-cli --id $NSOLID_AGENTID custom --name verbose")
console.log("   nsolid-cli --id $NSOLID_AGENTID custom --name verbose --data on ")
console.log("   nsolid-cli --id $NSOLID_AGENTID custom --name verbose --data off ")

//------------------------------------------------------------------------------
function onInterval() {
  log("interval event!")
  logVerbose("some extra logging here")
}

//------------------------------------------------------------------------------
// implements the `verbose` command  for nsolid-cli
//------------------------------------------------------------------------------
function verboseCommand(request) {

  // if "on" or "off" passed in with --data, set Verbose appropriately
  if (request.value == "on") {
    Verbose = true
  }
  else if (request.value == "off") {
    Verbose = false
  }
  else if (request.value) {
    return request.throw("expecting data of `on` or `off`, got " + request.value)
  }

  // return current value of Verbose
  return request.return({verbose: Verbose})
}

//------------------------------------------------------------------------------
function log(message) {
  console.log(message)
}

//------------------------------------------------------------------------------
function logVerbose(message) {
  if (!Verbose) return
  log(message)
}

When running your application with the N|Solid agent active, you can use the following command to return the current value of the Verbose setting:

nsolid-cli --id $NSOLID_AGENTID custom --name verbose

To set Verbose on or off, use one of the following commands:

NSOLID_AGENTID="69d916ad395061f80589e20bef9af3cb50ece9cb" # This will change
nsolid-cli --id $NSOLID_AGENTID custom --name verbose --data on
nsolid-cli --id $NSOLID_AGENTID custom --name verbose --data off

The output of these three cli commands looks like the following.

{"verbose":false,"time":1472496720972,"timeNS":"1472496720972042976","id":"69d916ad395061f80589e20bef9af3cb50ece9cb","app":"my-verbose-app","hostname":"titania"}

In this example you have directed a command at a single instance of your app. If you have multiple copies of this app running and you want to set all of their log levels to "verbose" simultaneously, omit the id parameter and use the app parameter instead.

nsolid-cli --app my-verbose-app custom --name verbose --data on
nsolid-cli --app my-verbose-app custom --name verbose --data off

This time multiple instances of the app might respond to the request.

{"verbose":false,"time":1472496720956,"timeNS":"1472496720956429760","id":"69d916ad395061f80589e20bef9af3cb50ece9cb","app":"my-verbose-app","hostname":"titania"}
{"verbose":false,"time":1472496720961,"timeNS":"1472496720961042976","id":"f1a6492b6bdf44db003a163430c7c7244ae28212","app":"my-verbose-app","hostname":"titania"}
{"verbose":false,"time":1472496720972,"timeNS":"1472496720972132564","id":"54153b774190fa24672708120b94b10fd41f75b8","app":"my-verbose-app","hostname":"titania"}

Custom Lifecycle Events

Implementing custom lifecycle events

In addition to the built-in life-cycle events, you can add your own using the function process.recordStartupTime(label) function. The label will then be used in the JSON output of the startup-times command.

You can use this to record the times at various stages of your application's startup. For instance:

  • when a database connection is requested
  • when the database connection is returned
  • when an http server is ready for incoming events

Using the CLI

To obtain the startup timing values, you can use the nsolid-cli startup-times command. For example:

$ nsolid-cli startup-times

Example JSON Result:

{
  "id": "bf24f4ed072b3bb4b220aa81fa3a73fde8038409",
  "app": "MyApp",
  "hostname": "myApp.example.com",
  "initialized_node":   [ 0, 130404 ],
  "initialized_v8":     [ 0, 482651 ],
  "loaded_environment": [ 0, 620207709 ]
}

This indicates that node was initialized in 130,404 nanoseconds (which is 130 microseconds, 0.130 milliseconds, or 0.000130 seconds).

Time format

The timing information is provided in hrtime format, which is a two element array of [ seconds, nanoseconds]. A nanosecond is one billionth (1,000,000,000th) of a second. The time values are the elapsed time since the process was started.

Adding a custom timer

Below is an example web server which is instrumented to provide the time when the web server starts listening to connections:

const http = require("http")

const server = http.createServer(onRequest)

server.listen(3000, onListen)

function onListen() {
  console.log("server listening on http://localhost:3000")
  process.recordStartupTime("http_listen")
}

function onRequest(request, response) {
  response.end("Hello, world!")
}

To start this program with the N|Solid agent listening on port 5000, use the following command:

$ NSOLID_APPNAME=http-sample nsolid http_sample.js

To obtain the startup times, use the following command:

nsolid-cli startup-times --app http-sample

This will return the following JSON output:

{
  "id": "bf24f4ed072b3bb4b220aa81fa3a73fde8038409",
  "app": "http-sample",
  "hostname": "http-sample.example.com",
  "initialized_node":   [ 0, 129554 ],
  "initialized_v8":     [ 0, 460521 ],
  "loaded_environment": [ 0, 95201339 ],
  "http_listen":        [ 0, 94902772 ]
}

Three startup times are provided by N|Solid by default:

Parameter Description
initialized_node The time it took to initialize the Node internals.
initialized_v8 The time it took to initialize the V8 engine
loaded_environment The time it took to complete all initialization, which includes running some of node's internal JavaScript code, and your main module's top-level code.

Policies

N|Solid ships with the ability to enable some security-related policies to help harden your application. Currently these include the ability to disable core modules and bindings.

To enable policy configuration with N|Solid, you must create a policies file in JSON format, and specify that file via the policies flag when starting N|Solid.

nsolid --policies my-policies.json app.js

Disabling Core Modules

N|Solid includes the ability to disable core modules with a policy file. This can be used to restrict application access to system resources or features that should not be used by your application.

There are two types of core resources that can be disabled: modules and bindings:

  • modules are the JavaScript APIs of Node, e.g. fs or http;
  • bindings are the wrappers for native code, e.g. stream_wrap or tcp_wrap.

There are four different severity levels you can configure:

Parameter Level Description
0 ignore No notice will be provided during the loading of the module or binding.
This is the default behavior.
1 warning A warning is written to stderr during the loading of the module or binding. The application otherwise runs normally.
2 throw error A error is thrown only when the policy is _violated_​ which doesn't occur during the ​_loading_​ of the module, but when a function on it is called. You can catch this by writing a try/catch in your code around your require() calls.
3 exit The application exits during the loading of the module or binding.

Disabling a module in the policy file prevents all non-core modules from using it i.e a warning is printed when loaded, but only when it is used does the violation handler kick in). Core modules are still able to load it (e.g. the core module library can still use fs to read new files via require() )

Example Policy File: disable-dns-all.policy.json

{
  "// severity levels": "0 - ignore; 1 - warning; 2 - throw error; 3 - exit",

  "blacklist": {
    "modules": {
      "dns": 2
    },
    "bindings": {
      "cares_wrap": 2
    }
  }
}

Access to bindings can also be restricted, if so desired, by the policy writer. The example policy file above disables both the dns module and the cares_wrap module, both of which can be used to access DNS.

When a non-core module attempts to access a disabled core module, messages will be logged to stdout, and an exception thrown.

Usage of disable-dns-all.policy.json

$ nsolid --policies disable-dns-all.policy.json faux-dns.js

nsolid error Binding "cares_wrap" is requested via a process.binding call
nsolid error at Object.<anonymous> (/Users/user/nsolid-demos/demos/faux-dns.js:3:23)
nsolid error But it was disabled in the policies.
_module_wrap.js:423
    throw new Error('Binding "' + id +
    ^

Error: Binding "cares_wrap" has been disabled by the policies flag, using "GetAddrInfoReqWrap" on it is not allowed, terminating execution.
    at new throwError (_module_wrap.js:423:11)
    at Module._compile (module.js:430:26)
... more stack trace lines here

More Policy Usage

Disabling a child process

{
  "blacklist": {
    "modules": {
      "child_process": 2,  -- blacklist entire child_process module with severity level 2
    }
  }
}

Code usage

var ls = require("child_process").execSync("ls -l")
console.log(ls.toString())

Running with policy

$ nsolid --policies disable_childprocess.json child_process_example.js
nsolid error Module "child_process" is required
nsolid error at Object.<anonymous> (/home/bryce/ns/nsolid-node/foo.js:1:72)
nsolid error But it was disabled in the policies.
child_process.js:75
  throw new Error('"' + module.id + '" has been disabled ' +
  ^

Error: "child_process" has been disabled by the policies flag, terminating execution.
    at handleDisabledMethodCall (child_process.js:75:9)
    at Object.execSync (child_process.js:89:22)
    at Object.<anonymous> (/home/bryce/ns/nsolid-node/foo.js:1:96)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:475:10)
    at startup (node.js:149:18)
    at node.js:985:3

Function using the binding 'cares_wraps'

module.exports = lookup;

var cares = process.binding('cares_wrap');

function lookup(hostname, callback) {

  var req = new cares.GetAddrInfoReqWrap();
  req.callback = callback;
  req.family = 4;
  req.hostname = hostname;
  req.oncomplete = callback;

  var err = cares.getaddrinfo(req, hostname, 4, 0);

  if (err) {
    callback(new Error("OOPS"));
    return {};
  }
  return req;
}

Calling function

var look = require("./fauxdns.js");
look("nodesource.com", function (err, data) {
  console.log(data);
});

Without a policy file we're able to access the network via the DNS binding

$ nsolid lookup.js
[ '162.243.142.64' ]

With a policy file that prevents just the dns module

$ cat disable_dns.js
{
  "blacklist": {
    "modules": {
      "dns": 2
    }
}

We can still access it!!

$ nsolid --policies disable_dns.js lookup.js
[ '162.243.142.64' ]

To actually prevent this circumvention of the modulesBlacklist we need to blacklist the cares_wrap binding

$ cat disable_cares.json
{
  "blacklist": {
    "bindings": {
      "cares_wrap": 2
    }
  }
}

Now the behavior is blocked!

$  nsolid --policies disable_cares.json lookup.js
nsolid error Binding "cares_wrap" is requested via a process.binding call
nsolid error at Object.<anonymous> (/Users/kevin/develop/nodesource/policies/fauxdns.js:3:21)
nsolid error But it was disabled in the policies.
_module_wrap.js:423
    throw new Error('Binding "' + id +
    ^

Error: Binding "cares_wrap" has been disabled by the policies flag, using "GetAddrInfoReqWrap" on it is not allowed, terminating execution.
    at new throwError (_module_wrap.js:423:11)
    at lookup (/Users/kevin/develop/nodesource/policies/fauxdns.js:7:13)
    at Object.<anonymous> (/Users/kevin/develop/nodesource/policies/lookup.js:3:1)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:475:10)
    at startup (node.js:150:18)
    at node.js:986:3

A policy file that disables several modules and bindings

{
  "blacklist": {
    "modules": {
      "child_process": 2,  -- blacklist entire child_process module with severity level 2
      "fs": 3,             -- blacklist entire fs module with severity level 3
      "path": 0            -- blacklist path with severity level 0, same as omitting `path` from blacklist all together
    }
    "bindings": {
      "stream_wrap": 2     -- blacklist entire stream_wrap binding with severity level 2
    },
  }
}

Comments

We allow properties whose keys start with "//" anywhere and they are ignored.

Invalid Policies

Anytime an unexpected property or a property with an invalid value is encountered, the policies are considered invalid and the process halted.

Further Thoughts

N|Solid's policies feature should not be used as a blind replacement for good security practices with regard to your code and the code of your application's dependencies. Disabling core modules and bindings does not provide a complete guarantee that your application is unable to use the features provided by those parts of Node.js. For example, child processes interacting with Unix utilities, if not disabled, can be used to replace the functionality of many of the core modules. Some core modules may be completely re-implemented in user-land modules, such as the dns and cares_wrap functionality. Compiled add-ons have complete freedom to insert new functionality that can replace anything that has been disabled.

N|Solid's policies features augment and support existing enterprise-wide application security hardening programs. Please contact NodeSource if you wish to discuss your Node.js application security needs and prepare a customized security hardening program that involves the use of internal and third-party code review, application monitoring and the use of operating system security tools.

Modules That Can Be Blacklisted

The following is a list of modules that can be blacklisted. It should be noted that for most of these modules it is rare to have a business need where blacklisting makes sense.

Module Description
_debug_agent Provides support for debugging node.js, mostly used internally.
_debugger Provides support for debugging node.js, mostly used internally.
_linklist Data structure used internally by node.js.
assert Used for writing unit tests for your applications, you can access it with require('assert').
buffer Raw data is stored in instances of the Buffer class.
child_process Provides a tri-directional popen(3) facility in order to create a child process.
console Console object is a special instance of Console whose output is sent to stdout or stderr, - constants.
crypto Crypto module offers a way of encapsulating secure credentials to be used as part of a secure HTTPS net or http connection.
cluster Allows you to easily create child processes that all share server ports.
dgram Provides Datagram sockets.
dns Provides name resolution.
domain Pending deprecation - provide a way to handle multiple different IO operations as a single group.
events Provides event emitters.
freelist Data structure used internally by node.js.
fs Provides file I/O via simple wrappers around standard POSIX functions.
http Provides ability to create http servers and clients.
_http_agent Provides support for http and https modules, mostly used internally.
_http_client Provides support for http and https modules, mostly used internally.
_http_common Provides support for http and https modules, mostly used internally.
_http_incoming Provides support for http and https modules, mostly used internally.
_http_outgoing Provides support for http and https modules, mostly used internally.
_http_server Provides support for http and https modules, mostly used internally.
https Provides ability to create https servers and clients.
module Provides the node.js module system, mostly used internally.
net Net module provides asynchronous network wrapper for server and clients.
os Provides basic operating-system related utility functions.
path Provides utilities for handling and transforming file paths.
process Same as the global process object.
punycode Provides Unicode utilities.
querystring Provides utilities for dealing with query strings.
readline Allows reading of a stream (such as process.stdin) on a line-by-line basis.
repl Provides a way to interactively run JavaScript and see the results.
stream Provides streaming functionality.
_stream_readable Provides the ReadableStream exposed via the stream module.
_stream_writable Provides the WritableStream exposed via the stream module.
_stream_duplex Provides the DuplexStream exposed via the stream module.
_stream_transform Provides the TransformStream exposed via the stream module.
_stream_passthrough Provides the PassThroughStream exposed via the stream module.
_stream_wrap Provides the wrapper to the libuv stream implementation, mostly used internally.
string_decoder Decodes a buffer to a string with utf8 support.
sys Alias for util.
timers Provides internal support for global Timers.
tls Uses OpenSSL to provide Transport Layer Security and/or Secure Socket Layer: encrypted stream communication.
_tls_common Provides support for tls module, mostly used internally.
_tls_legacy Provides support for tls module, mostly used internally.
_tls_wrap Provides support for tls module, mostly used internally.
tty Provides the tty.ReadStream and tty.WriteStream classes.
url Provides utilities for URL resolution and parsing.
util Primarily designed to support the needs of node.js's internal APIs, but many of these utilities are useful for user programs.
v8 Events and interfaces specific to the version of v8.
vm Provides support to compiled code and run immediately or compile, save it, and run later.
zlib Provides bindings to Gzip/Gunzip.
profiler Profiler module (explained in nsolid docs).
_function_origin Internal module used by nsolid.
_module_wrap Internal module used by nsolid.
_module_wrap_known_modules Internal module used by nsolid.
_module_wrap_known_bindings Internal module used by nsolid.
_policies_validation Internal module used by nsolid.
nsolid Internal module used by nsolid.
nsolid_versions Internal module used by nsolid.

Bindings That Can Be Blacklisted

The following is a list of bindings that can be blacklisted. It should be noted that for most of these bindings it is rare to have a business need where blacklisting makes sense.

Binding Description
async_wrap Base class for all async request modules which supports hooks into async events.
buffer Interface to libuv methods to operate on buffers.
cares_wrap Wraps the cares library for asynchronous name resolves.
contextify Interface to v8 in order to create execution contexts.
crypto Provides crypto functionality such as SSL and ciphers
fs Interface to file system methods provided by libuv.
fs_event_wrap Binding to listen to file related events in order to support watching files and directories for changes.
http_parser Binding to http_parser.
js_stream Binding that provides support for node.js streams.
os Interface to libuv in order to obtain process state information.
pipe_wrap Wraps libuv's socket implementation.
process_wrap Provides process functionality such as spawning child processes.
signal_wrap Supports sending and handling signals to processes.
spawn_sync Supports child_process.execSync functionality via libuv.
stream_wrap Wraps stream implementations exposed by libuv to communicate via file descriptors or network connections.
tcp_wrap Integrates TCP functionality exposed by libuv in order to create TCP connections.
timer_wrap Integrates with timer functionality provided by libuv so support things like setTimeout and setInterval.
tls_wrap Provides TLS termination functionality via crypto.
tty_wrap Integrates with TTY terminal functionality provided by libuv.
udp_wrap Provides UDP protocol functionality such as binding and broadcasting.
uv Provides libuv integration such as error name resolution.
v8 Limited interface to the v8 API, i.e. to set flags and get process information.
zlib Wraps the zlib library.