What is N|Solid?

For people that need to use Node.js in an enterprise environment, NodeSource has developed N|Solid. Backed by NodeSource's 24x7 support, N|Solid adds enterprise-grade diagnostic and security features delivered on a release cycle that is aligned with Node.js Long Term Support (LTS) releases. This provides a stable, production platform for all of your Node applications.

N|Solid benefits from the experience and insights of several core contributors to the Node.js ecosystem. Rather than trying to adapt existing tools that assume a synchronous program flow, we chose to build native tools for Node.js that provide live instrumentation of your production system's health and stability, including CPU profiling and heap snapshot analysis, with no changes to your application code.

Component Overview

N|Solid is a diagnostic tool that is designed to enhance Node.js to meet the needs of enterprise applications deployed in production. At times, Node.js can feel like a black box, especially as the shift to an asynchronous programming model changes how we interpret the data we do have and how we reason about it. Furthermore, the ability to control access and define policies for application is critical in enterprise environments.

In order to help customers gain visibility into the Node.js black box, we have the N|Solid Runtime. The N|Solid Runtime is a build of Node.js bundled with an advanced native C++ component, the N|Solid Agent. The N|Solid Agent runs on its own thread inside your application, with direct access to the V8 JavaScript engine and libuv, which are the core of Node.js. The runtime provides access to metrics, allows you to control behavior at runtime, and even interact with a completely non-responsive Node.js application.

N|Solid includes a powerful security policy model that allows restricting access to system resources by untrusted modules. A more secure memory allocation model is included to prevent unintended data exposure.

Complementing the tools provided by the N|Solid Agent, we also provide the N|Solid Hub and the N|Solid Console.

The N|Solid Hub provides a centralized service registry and proxy for all of your Node.js applications using N|Solid. It is designed to work with modern deployment models where services start or stop via auto-scaling or are simply containers of microservices in a cloud infrastructure.

The N|Solid Console provides centralized access to all of your applications and an aggregated view of each application's processes. This holistic view simplifies triage at runtime, makes it easier to look for outliers, and take advantage of advanced diagnostic tools directly from your browser.

The combination of the N|Solid Runtime, Hub and Console make N|Solid an invaluable tool for gaining insight into the behavior of your production applications at runtime.

Node Compatibility

N|Solid 1.0 is delivered bundled with a Node.js 4.x runtime. All N|Solid features are additive, meaning any application that runs on Node.js 4.x will also work with N|Solid 1.0. To verify what version of Node.js that you have bundling N|Solid, use node -v.

For more information about the Node.js API, consult the Node.js API Documentation.

Installing the N|Solid Runtime

Linux Installation Steps

1. Go to the NodeSource download site
2. Select the Linux installation type
3. Click the Linux .tar.gz link next to the N|Solid Runtime label to download
4. Extract to your preferred location.
5. Add the contents in the extracted bin folder to your $PATH or a location on your $PATH, such as /usr/local/bin. If you have another installation of node or npm on your system, the N|Solid bin location must be before it in your path.

This will make binaries available for nsolid, nsolid-cli, npm, and node (which is a link to nsolid).

OS X Installation Steps

1. Go to the NodeSource download site
2. Select the Mac OS .tar.gz installation type
3. Click the Mac OS X .tar.gz link next to the N|Solid Runtime label to download
4. Extract to your preferred location.
5. Add the contents in the extracted bin folder to your $PATH or a location on your $PATH, such as /usr/local/bin. If you have another installation of node or npm on your system, the N|Solid bin location must be before it in your path.

This will make binaries available for nsolid, nsolid-cli, npm, and node (which is a link to nsolid).

PKG Installer

We also provide a native installer for OS X.

1. Go to the NodeSource download site
2. Select the Mac OS .pkg installation type
3. Click the Mac OS X .pkg link next to the N|Solid Runtime label to download
4. Double-click the .pkg and follow the on-screen instructions

Verifying the N|Solid Installation

Once installed, to verify your node executable is N|Solid, you can run:

  $ node -p 'process.versions'

Verify that the nsolid version number is as expected.

{
  http_parser: '2.5.0',
  node: '4.1.0',
  nsolid: '1.0.0',   // This field indicates your N|Solid version
  v8: '4.5.103.33',
  uv: '1.7.4',
  zlib: '1.2.8',
  ares: '1.10.1-DEV',
  modules: '46',
  openssl: '1.0.2d'
}

Installing the N|Solid Hub

The N|Solid Hub facilitates communication with all of the processes in an application that is run under N|Solid. It is comprised of two components; the Service Registry and the Proxy.

The Service Registry is the component which N|Solid Agents register themselves to, in order to facilitate communication with N|Solid Hub clients.

The Proxy is the component that allows remote communication to be forwarded or broadcast to your N|Solid Agents via the N|Solid Console or N|Solid CLI.

N|Solid Hub - Service Registry

If not installed, download the latest stable version of etcd (2.x.x family) for your platform here: https://github.com/coreos/etcd/releases and follow instructions for installation.

N|Solid Hub - Proxy

The N|Solid Proxy is a Universal binary for Linux and OS X.

1. Go to the NodeSource download site
2. Select the N|Solid Proxy Universal download
3. Extract to your preferred location.
4. Run npm start to start a local version of the N|Solid Proxy

For a more permanent installation, install the N|Solid Proxy as you would any normal Node.js production application in your infrastructure.

Copying files via Finder

If you are using Finder to introspect the downloaded file and to copy the files to their final destination, be sure to copy the directory rather than the individual files in the directory. As Finder doesn't inherently show . files the .nsolid-proxyrc file will not be copied. If this file is not present, you will receive a throw new Error("Could not find configuration for port")Error: Could not find configuration for port message upon start."

Verifying the N|Solid Hub Installation

You can verify that the N|Solid Hub is installed correctly by following the instructions below.

First we need to run the service registry to allow our N|Solid instances to register themselves. This will start up your etcd server for use across the network

    $ etcd -name nsolid_proxy -listen-client-urls http://0.0.0.0:4001 -advertise-client-urls http://0.0.0.0:4001 -initial-cluster-state new ",

Next we need to run the proxy in order to allow our tools to query the registered instances of N|Solid.

  npm start

The start command will run the proxy.js file as defined in the package.json file. The default configuration for the N|Solid Hub is for the N|Solid Proxy to listen on port 9000 on all interfaces. This can be configured by updating the provided .nsolid-proxyrc configuration file.

Now, we can run an instance of N|Solid that we will query. You may replace server.js below with the filename for your Node.js application.

  export NSOLID_APPNAME=my-node-application\nexport NSOLID_HUB=localhost:4001\nexport NSOLID_SOCKET=1111\nexport NODE_ENV=production\nnsolid server.js

If you are using the hub you can set the NSOLID_SOCKET=0 as N|SOLID will pick a port number itself.

Finally we can run the query in order to see information about our instances of N|Solid.

  nsolid-cli ls

Resultant JSON:

{
  "pid": 32142,
  "hostname": "Macintosh.local",    
  "app": "my-node-application",
  "address": "172.31.99.213:1111\",
  "id": "3a69bda644089fbebc3f61d8f12a5dbb3d2ab741"
}

Installing the N|Solid Console

The N|Solid Console is a Web application that provides a consolidated view of all N|Solid applications that are registered with the N|Solid Hub.

Supported Server Operating Systems

• Linux
• OS X 10.7 and higher

Minimum Browser Requirements

The N|Solid Console is best experienced using an evergreen browser (Chrome 21+, Firefox 22+, Safari 6.1+, Microsoft Edge) on a desktop/laptop system.

Prerequisites

see: Installing the N|Solid Hub

Installation Instructions

Linux Installation Steps

1. Go to the NodeSource download site
2. Select the Linux installation type
3. Click the Linux .tar.gz link next to the N|Solid Console label to download
4. Create a new user account, ns
5. Extract into the ns user’s home directory
6. Setup ~ns/.nsolid-console.rc to point to the http endpoint provided by the N|Solid Hub see: ~/ns/nsolid-console/scripts/README.md
7. Setup the init scripts for your system. see: ~/ns/nsolid-console/scripts/README.md

OS X Installation Steps

1. Go to the NodeSource download site
2. Select the Mac OS .tar.gz installation type
3. Click the Mac OS X .tar.gz link next to the N|Solid Console label to download
4. Extract to your preferred location.

Starting the Console

$ cd ./nsolid-console\n$ npm start

Note: If you do not use npm start to start the console, make sure to have the NODE_ENV environment variable set to production:\n\n$ NODE_ENV=production nsolid bin/nsolid-console.js

Networking Requirements

The N|Solid Console uses a collection daemon that should be able to connect to the N|Solid Hub. This can be quickly tested by running a command on the host.

  $ nsolid-cli --hub <proxy ip>:9000 ping

In order for users to utilize the console, they will need to be able to connect to it via the port (--port) specified when launching the collection daemon. This can be easily tested by navigating to http://<console ip>:3000 in a browser or using curl from a user’s workstation

  $ curl -v `http://<console ip>:3000`

Note the console uses WebSockets to fetch metrics from the collection daemon that may affect the system / network configuration. If the console is able to load, but no metrics are coming through, check the browser’s network inspector to ensure WebSocket connections are being negotiated successfully. If they are, but there are still no applications displayed in the console, please confirm the N|Solid instances are running and have been properly registered with the N|Solid Hub.

Using N|Solid with Docker

For people who need to use N|Solid in a Dockerized environment, NodeSource has developed the N|Solid Docker images.

These images are built with the enterprise customer in mind, developing a versatile set of independently scalable Docker images that work together to provide a unified N|Solid experience across all of your Dockerized infrastructure.

In addition to being friendly to enterprise operations teams, the N|Solid Docker Images have been designed to be accessible to developers. For those who are already using Docker, these images provide an easy way to get up and running with the N|Solid console. With the use of docker-compose, you can now get the full N|Solid Console on your local machine with a single command: docker-compose up.

Overview of N|Solid Docker Images

We provide 5 separate Docker Images for N|Solid, allowing each component of the platform to be scaled and deployed independently.

• docker pull nodesource/nsolid:v1.2.2
  • the base image that your Docker images should build from. It provides the N|Solid runtime and agent.
• docker pull nodesource/nsolid-cli
  • provides an easy way to query statistics from your Node.js processes. It is meant to be run directly from the command line.
• docker pull nodesource/nsolid-registry:latest
  • contains the N|Solid Hub service registry. Your Node.js applications will need to be able to talk to this.
• docker pull nodesource/nsolid-hub:v3.4.2
  • contains the N|Solid Hub proxy service, which is used by the N|Solid Console to query runtime statistics from your Node.js applications. This container will need to talk to both the registry and your Node.js applications
• docker pull nodesource/nsolid-console:v1.4.4
  • contains the N|Solid console, a web based interface for monitoring your Node.js applications. You will need to bind the internal port 3000 to your host. This container will need to be able to talk to the hub container.

All of the images are built on the official ubuntu:trustyimage maintained by the Docker project.

Installing the N|Solid Docker Images

Setting up Docker

First, follow the steps for installing Docker for your operating system, we recommend you use at least Docker v1.9: https://docs.docker.com/engine/installation/

Although not required, it is recommended that you use docker-compose when first getting started. It simplifies managing these Docker images when running them locally. If you choose to use docker-compose, follow the appropriate installation guide for your operating system: https://docs.docker.com/compose/install/

Getting the Docker-Compose.yml file

If you plan to use docker-compose, you will want to download the docker-compose.yml file and save it in an empty directory under the name nsolid.yml

Downloading the Docker Images

If you are using docker-compose, open a terminal in the folder you downloaded nsolid.yml to, and run the following command:

docker-compose pull

This command will download all of the images needed to run the N|Solid console locally. If you choose not to use docker-compose, execute the following commands:

docker pull nodesource/nsolid-registry:latest
docker pull nodesource/nsolid-console:v1.4.4
docker pull nodesource/nsolid-hub:v3.4.2

Using the N|Solid Docker Images

Creating A Shared Network

The N|Solid Docker images require a level of service discovery to work. In Docker, this means that the containers must be running on the same network stack. This can be accomplished by way of a shared network.The process of creating a shared network differs depending on your Docker version

• For Docker < v1.9, use the (deprecated) --link flag
• For Docker 1.9+, create a shared network for your containers with the command:

docker network create nsolid

Starting the N|Solid Console

To bring up the N|Solid Console at this point, run the following command:

docker-compose -f nsolid.yml up

By default, this will bring up the N|Solid Console running on 127.0.0.1:3000, and the console will be monitoring itself.

If you are not using docker-compose, you will need to start 3 containers to run the console:

docker run -d --net nsolid -p 2379:2379 nodesource/nsolid-registry
docker run -d --net nsolid -e ‘REGISTRY=registry:2379’ -e ‘NODE_DEBUG=nsolid’ nodesource/nsolid-hub
docker run -d --net nsolid -p 3000:3000 -e ‘NSOLID_APPNAME=console’ -e ‘NSOLID_HUB=registry:2379’ -e ‘NSOLID_DEBUG=nsolid’ nodesource/nsolid-console --hub hub:9000

This will bring up the N|Solid console available on localhost:3000 and monitoring itself by default.

Adding Your Own Application

First, ensure your application is compatible with the LTS release of Node.js. NodeSource has created upgrade-ready to assist with this.

If you are new to Docker, follow the steps in our blog post to get your application into a Docker container.

Create a folder in the directory named after your image, and place your Dockerfile and source files in it. This documentation will refer to the service as example and the folder will be ./example. Your Dockerfile should begin with the following line:

FROM nodesource/nsolid:v1.2.2

Starting Your Application with Docker Compose

Next, create the file docker-compose.yml in the directory along-side nsolid.yml:

example:
  build: ./example
  net: “nsolid”
  environment:
    - NSOLID_APPNAME=example
    - NSOLID_HUB=registry:2379

For the complete documentation on defining a service with docker-compose.yml, refer to the Docker project’s documentation page: https://docs.docker.com/compose/overview/.

At this point, you are ready to bring up your application using docker-compose

docker-compose -f docker-compose.yml up

Starting Your Application without Docker Compose

After building your service with:

docker build -t example .

Start your service with:

docker run -it --net nsolid -e ‘NSOLID_APPNAME=example’ -e ‘NSOLID_HUB=registry:2379’ example

Congratulations, you are now up and running with N|Solid and Docker!

A Note on Multi-Host Deployments

Using Docker with N|Solid introduces some networking edge cases that can make multihost deploys challenging. These images have been developed specifically to address this. When used correctly, these images should seamlessly worked together across multiple hosts.

By default, Docker will create an isolated network on the host that the container binds to. Then, Docker configures a mapping from the port’s on the containers network to a port on the host’s network per the configuration you pass at runtime.

This introduces the case where, by default, the IP address that the N|Solid agent sees from inside the container is not the same IP address that the N|Solid hub needs to communicate with the agent.

There are a few methods to address this in Docker, and they all boil down to making sure the IP address that the hub needs, and the IP address that the agent sees, are the same.

Using the Host’s Network Stack

If you are using containers to isolate resources and not to run untrusted code, it is safe to use docker run --net=host to allow the container to bind directly to the host’s ports. This is the most direct way of getting the hub to talk to the Dockerized N|Solid agent.

Security

This allows the container to bind directly to the host’s ports, including low-numbered​ ports. If you are running untrusted code, this is a security risk and should be avoided.

Using Docker Swarm

When using Docker Swarm with Docker v1.9 or later, you are able to use the overlay driver for docker network. This allows all of the Docker containers to share the same network with DNS resolution built in. To get started with this, first provision a new machine that will host the key-value store by Docker Swarm for discovery

docker-machine create -d virtualbox nsolidkv

Machine Drivers

Note: In addition to Virtualbox, docker-machine supports drivers for various cloud providers. View the full list of cloud providers and how to use them with docker-machine here.

Next we will deploy consul to our new host:

$ eval "$(docker-machine env nsolidkv)"
$ docker run -d -p 8500:8500 -h consul progrium/consul -server -bootstrap
$ eval "$(docker-machine env -u nsolidkv)"

Troubleshooting

If you see client is newer than the server you will need to run docker-machine upgrade nsolidkv

Next, we will provision our swarm master:

$ docker-machine create -d virtualbox --swarm --swarm-master --swarm-discovery="consul://$(docker-machine ip nsolidkv):8500" --engine-opt="cluster-store=consul://$(docker-machine ip nsolidkv):8500" --engine-opt="cluster-advertise=eth1:2376" nsolid-master

We will now create one or more slave nodes for our swarm cluster. Re-run the following command as many times as you like (incrementing the node id) to generate slaves.

$ docker-machine create -d virtualbox --swarm --swarm-discovery="consul://$(docker-machine ip nsolidkv):8500" --engine-opt="cluster-store=consul://$(docker-machine ip nsolidkv):8500" -engine-opt="cluster-advertise=eth1:2376" nsolid-n1

Then, verify that the machines have been provisioned:

$ docker-machine ls
NAME            ACTIVE   DRIVER       STATE     URL                         SWARM
nsolid-master   -        virtualbox   Running   tcp://192.168.99.101:2376   nsolid-master (master)
nsolid-n1       -        virtualbox   Running   tcp://192.168.99.102:2376   nsolid-master
nsolid-n2       -        virtualbox   Running   tcp://192.168.99.103:2376   nsolid-master
nsolidkv        -        virtualbox   Running   tcp://192.168.99.100:2376

Finally, create an overlay network on the swarm.

$ eval "$(docker-machine env --swarm nsolid-master)"
$ docker network create -d overlay nsolid

At this point, any container you deploy to the swarm will support DNS resolution by container name, and the IP address that the N|Solid agent sees will be the same IP address the hub container needs.

Environment Configuration

This section describes how to configure aspects of N|Solid

N|Solid CLI

Specifying the N|Solid Hub location and Default Application Name

The Command Line Interface (CLI) can be used in conjunction with a configuration file (nsolid-clirc) to pre-set values such as the hub location and a default app name. The file .nsolid-clirc should be placed where nsolid-cli is installed or in your home directory.

{
    "hub": "http://localhost:9000",
    "app": "my-node-application"
}

Once this is set, running the CLI ls command will return values associated with the default app.

$ nsolid-cli ls

{
  "pid": 32142,
  "hostname": "Macintosh.local",
  "app": "my-node-application",
  "address": "172.31.99.213:1111",
  "id": "3a69bda644089fbebc3f61d8f12a5dbb3d2ab741"
}

If you have a default app name set and would like to query all applications, send an * as the --app value:

$ nsolid-cli --app '*' ls

{
    "pid": 32142,
    "hostname": "Macintosh.local",
    "app": "my-node-application",
    "address": "172.31.99.213:1111",
    "id": "3a69bda644089fbebc3f61d8f12a5dbb3d2ab741"
  },
  {
    "pid": 32297,
    "hostname": "Macintosh.local",
    "app": "frontend-server",
    "address": "172.31.99.213:1112",
    "id": "2b61816e6679504f6a429c259793e2691f83bf71"
  }

Troubleshooting

N|Solid, the N|Solid Proxy and N|Solid CLI all can have debug messaging enabled by using the NODE_DEBUG=nsolid environment variable.

$ NODE_DEBUG=nsolid NSOLID_SOCKET=8000 node app.js
NSOLID 30599: registering default commands
NSOLID 30599: nsolid initializing
NSOLID 30599: nsolid initialized on port 8000

Network Access Considerations

The N|Solid Console and CLI use the N|Solid Hub to query for metrics. The host and port that the Hub is configured to listen on must be accessible to these clients.

Similarly, when the N|Solid Hub communicates with an N|Solid instance, the host and port for the instance must be accessible. Without this route, there will be no way to query specific data about the instance. This is a bi-directional route as the N|Solid instance actively communicates back to the Hub.

N|Solid Proxy

The N|Solid Proxy (a part of the N|Solid Hub) is the component that allows remote communication to be forwarded or broadcast to your processes via the N|Solid Console or N|Solid CLI.

The N|Solid Proxy is configured via an rc file, with a default provided configuration file .nsolid-proxyrc.

{
  "registry": "localhost:4001",
  "port": 9000,
  "denied": [],
  "broadcast_approved": [
    "ping",
    "process_stats",
    "system_stats",
    "system_info",
    "info",
    "versions",
    "startup_times"
  ]
}

Broadcast vs Non-broadcast Query

Broadcast queries are the default operation of the CLI. You can use nsolid-cli --id in order to change to a non-broadcast query.

Any operation except for ls needs to be approved for broadcast. This is to prevent extraneous traffic and accidental load from performing large operations. In your .nsolid-proxyrc add the names of any commands you wish to be available for broadcast queries. If you create custom commands that you wish to be able to broadcast query you will need to add them to the broadcast list in your .nsolid-proxyrc file.

Key Environment Variables

NSOLID_APPNAME - This is the name of the application that the N|Solid instance is running. Use this in order to create logical groupings of the processes in the N|Solid Console. If omitted, the value defaults to nsolid-default.

NSOLID_HUB - This is the route to your N|Solid Hub’s Service Registry. It accepts host and/or port. The host can be provided via several formats: IPv4 10.0.0.21 IPv6 [2001:cdba::3257:9652] hostname nsolid_hub.local

NSOLID_SOCKET - This is the address that your instance's N|Solid Agent will be listening on for queries. It accepts an address or hostname bound by an interface on this machine and an optional specified port. If no port is specified, or port 0 is specified (e.g. NSOLID_SOCKET=localhost:0) the N|Solid Agent will request an open port from the operating system's ephemeral port range.

Tip

We suggest allowing the N|Solid Agent to select the port number. This provides greater compatibility with modules such as cluster and will be automatically discovered correctly when using the N|Solid Hub.

Refresh Rate

The rate at which the N|Solid Console sends queries can be adjusted when you start the console via the --interval which accepts a number of milliseconds to wait between polling for new information.

$ nsolid-console --interval 5000

Performance Consideration

Be aware that changing the interval to lower amounts may cause excessive network traffic. By default this value is set to 5 seconds. Setting this to larger values will result in lower traffic but delays of information given to the management console.

Policies

N|Solid ships with the ability to enable some security-related policies to help harden your application. Currently these include the ability to automatically zerofill Buffers upon allocation and disabling core modules.

To enable policy configuration with N|Solid, you must create a policies JSON file and specify that file using the policies flag when starting the N|Solid runtime.

For more information on Policies read here.

Running Node Applications with N|Solid

The default configuration when running N|Solid is to behave exactly like Node.js. The agent is disabled and the default policy set is the default Node.js behavior.

Example: Without configuration, the N|Solid Agent is disabled and inactive:

$ node app.js

N|Solid Agent

The N|Solid Agent is a small service running inside each N|Solid process. It provides a control point and access mechanism for all of the instrumentation included in N|Solid.

1. Handling command requests
2. Registering with the N|Solid Hub

All of the instrumentation included in N|Solid is controlled via the Agent. Each of these are exposed as a command that can be called via the Agent's listening socket by the N|Solid CLI or N|Solid Console. Internally it exposes an API that can be used to expose additional custom behavior that can be served by the Agent.

To enable the N|Solid Agent, you must either specify a value for NSOLID_HUB and/or NSOLID_SOCKET in your environment for each process that runs under N|Solid.

Running in Standalone Mode

Running in Standalone mode is typically for installations without N|Solid Hub, such as in a development environment. To run in Standalone mode, one simply needs to specify the NSOLID_SOCKET address for the N|Solid Agent to listen on.

An N|Solid process running in Standalone mode can be contacted directly via the N|Solid CLI.

Example: Standalone mode, with the N|Solid Agent listening on port 8000

$ NSOLID_SOCKET=8000 node app.js

This allows the nsolid-cli to send commands to the N|Solid Agent bundled with your application and print the replies:

$ nsolid-cli --socket 8000 ping
> "PONG"

Read more about the N|Solid Agent Commands here.

Running in Cluster mode

For most non-development environments, the N|Solid Hub is designed to provide a mechanism for auto-configuration and discovery of N|Solid processes. The N|Solid Agent Configuration section contains information about configuring the Agent to register with the Hub.

Example: A registered N|Solid process with a N|Solid Hub on the same machine

$ NSOLID_HUB=localhost:4001 nsolid app.js

N|Solid CLI

The N|Solid CLI provides a simple command-line interface communication to your N|Solid processes, either directly or via the N|Solid Hub.

The N|Solid CLI is bundled with N|Solid and available in your path if N|Solid is installed as described.

Commands

All Agent commands are supported as CLI commands.

Additionally, the Proxy exposes a ls command which will list N|Solid processes as registered in the N|Solid Hub.

Usage

To send a command to a single process via the Proxy:

nsolid-cli --app $app --id $id $command

ls

List all registered N|Solid processes:

$ nsolid-cli ls

{
  "pid":3691,
  "hostname":"dc1vm1",
  "app":"nsolid-default",
  "address":"192.168.1.104:35770",
  "id":"e55ceb733a01be8cb0803c732df685efff5a56ee"
}
{ "pid":7452,
  "hostname":"dc1vm2",
  "app":"foo",
  "address":"192.168.1.104:47805",
  "id":"5b79d82b911e246d533a000e9bb8c8f631482c79"
}

List all registered N|Solid processes for a given NSOLID_APPNAME:

$ nsolid-cli --app nsolid-default ls

{
  "pid":3691,
  "hostname":"dc1mv",
  "app":"nsolid-default",
  "address":"192.168.1.104:35770",
  "id":"e55ceb733a01be8cb0803c732df685efff5a56ee"
}

Direct to a single N|Solid process

To specify a single N|Solid Agent, bypassing the N|Solid Proxy:

$ nsolid-cli --socket $socket $command

E.g.

$ nsolid-cli --socket 8000 ping
"PONG"

Via the N|Solid Hub

To specify a non-default N|Solid Hub location, you can override the configuration file with --hub

$ nsolid-cli --hub 9000 ls

{
   "pid":86124,
   "hostname": "vm.local",
   "app":"nsolid-default",
   "address":"192.168.0.10:52602",
   "id":"f46ff6c3e27f30338ae0a6cc15bbd7376a8e964d"
}
{
   "pid":86120,
   "hostname":"vm.local",
   "app":"foo",
   "address":"192.168.0.10:52550",
   "id":"92357fe6fce23fc8dd732f105c10ca279596a052"
}

The default behavior is to attempt to broadcast the commands to all known N|Solid processes:

$ nsolid-cli info

Resultant JSON

{
  "meta": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "hostname": "cluster-1.oregon",
    "address": "192.168.2.27:38580"
  },
  "reply": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "nodeEnv":"production",
    "execPath": "/usr/local/bin/nsolid",
    "main": "/var/myapp/web/server.js"
  }
}
{
  "meta": {
    "id": "ccd48968f3212a06c3aff62bbfa75c481ad5f672",
    "app": "api-server",
    "pid": 5447,
    "hostname": "cluster-1.oregon",
    "address": "192.168.2.27:52352"
  },
  "reply": {
    "id": "ccd48968f3212a06c3aff62bbfa75c481ad5f672",
    "app": "api-server",
    "pid": 5447,
    "nodeEnv":"production",
    "execPath": "/usr/local/bin/nsolid",
    "main": "/var/myapp/api/server.js"
  }
}

This can be filtered at the NOSLID_APPNAME level via --app:

$ nsolid-cli --app web-server info

{
  "meta": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "hostname": "cluster-1.oregon",
    "address": "192.168.2.27:38580"
  },
  "reply": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "nodeEnv":"production",
    "execPath": "/usr/local/bin/nsolid",
    "main": "/var/myapp/web/server.js"
  }
}

When proxying commands through the N|Solid Hub, certain commands cannot be broadcast:

$ nsolid-cli profile_start
Error trying to connect to N|Solid: Error: "Command 'profile_start' is not broadcast approved, id is required."

Any command will be considered a broadcast if it does not have the app and id specified, independent of the number of processes that will be queried.

To specify a single process, you must both specify the id and app name:

$ nsolid-cli --app web-server --id 9f6e7ab8f323a33a31f56bafed32ebc172777dd8 info

{
  "meta": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "hostname": "cluster-1.oregon",
    "address": "192.168.2.27:38580"
  },
  "reply": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "nodeEnv":"production",
    "execPath": "/usr/local/bin/nsolid",
    "main": "/var/myapp/web/server.js"
  }
}

For the full list of available N|Solid CLI commands, see N|Solid Command Line Interface (CLI)

Policies

N|Solid ships with the ability to enable some security-related policies to help harden your application. Currently these include the ability to automatically zero-fill buffers upon allocation and disabling core modules and bindings.

To enable policy configuration with N|Solid, you must create a policies file in JSON format, and specify that file via the policies flag when starting N|Solid.

nsolid **--policies **my-policies.json app.js

Zero-fill Allocations

The default Node.js memory allocator reuses memory without clearing it. In the case of bugs or malicious code, this can result in the application transmitting data that happened to be in memory that shouldn't be transmitted. This can be used as a basis for exploit, or may result in program data not being consistently initialized.

By default,** Node.js does not zero-fill memory for performance reasons**. This leaves data safety up to the application developer, however, this burden also exists for all third-party dependencies that you use.

You can configure N|Solid to zero-fill allocated memory to prevent these issues. There is a slight performance degradation for enabling this feature.

Example Policy File zero-fill.policy.json

{
  "process": {
    "zeroFillAllocations": true
  }
}

** Usage of zero-fill.policy.json **

$ nsolid -p 'new Buffer(20)'
<Buffer 68 f3 88 77 bc 7f 00 00 68 f3 88 77 bc 7f 00 00 e0 05 14 02>

$ nsolid --policies zero-fill.policy.json -p 'new Buffer(20)'
<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>

Disabling Core Modules

N|Solid includes the ability to disable core modules with a policy file. This can be used to restrict application access to system resources or features that should not be used by your application.

There are two types of core resources that can be disabled: modules and bindings:

• modules are the JavaScript APIs of Node, e.g. fs or http;
• bindings are the wrappers for native code, e.g. stream_wrap or tcp_wrap.

There are four different severity levels you can configure:

Disabling a module in the policy file prevents all non-core modules from using it i.e a warning is printed when loaded, but only when it is used does the violation handler kick in). Core modules are still able to load it (e.g. the core module library can still use fs to read new files via require() )

Example Policy File disable-dns-all.policy.json

{
  "// severity levels": "0 - ignore; 1 - warning; 2 - throw error; 3 - exit",

  "blacklist": {
    "modules": {
      "dns": 2
    },
    "bindings": {
      "cares_wrap": 2
    }
  }
}

Access to bindings can also be restricted, if so desired, by the policy writer. The example policy file above disables both the dns module and the cares_wrap module, both of which can be used to access DNS.

When a non-core module attempts to access a disabled core module, messages will be logged to stdout, and an exception thrown.

Usage of disable-dns-all.policy.json

$ nsolid --policies disable-dns-all.policy.json faux-dns.js

nsolid error Binding "cares_wrap" is requested via a process.binding call
nsolid error at Object.<anonymous> (/Users/user/nsolid-demos/demos/faux-dns.js:3:23)
nsolid error But it was disabled in the policies.
_module_wrap.js:423
    throw new Error('Binding "' + id +
    ^

Error: Binding "cares_wrap" has been disabled by the policies flag, using "GetAddrInfoReqWrap" on it is not allowed, terminating execution.
    at new throwError (_module_wrap.js:423:11)
    at Module._compile (module.js:430:26)
... more stack trace lines here

More Policy Usage

Disabling a child process

{
  "blacklist": {
    "modules": {
      "child_process": 2,  -- blacklist entire child_process module with severity level 2
    }
  }
}

Code usage

var ls = require("child_process").execSync("ls -l")
console.log(ls.toString())

Running with policy

$ node --policies disable_childprocess.json child_process_example.js
nsolid error Module "child_process" is required
nsolid error at Object.<anonymous> (/home/bryce/ns/nsolid-node/foo.js:1:72)
nsolid error But it was disabled in the policies.
child_process.js:75
  throw new Error('"' + module.id + '" has been disabled ' +
  ^

Error: "child_process" has been disabled by the policies flag, terminating execution.
    at handleDisabledMethodCall (child_process.js:75:9)
    at Object.execSync (child_process.js:89:22)
    at Object.<anonymous> (/home/bryce/ns/nsolid-node/foo.js:1:96)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:475:10)
    at startup (node.js:149:18)
    at node.js:985:3

Function using the binding 'cares_wraps'

module.exports = lookup;

var cares = process.binding('cares_wrap');

function lookup(hostname, callback) {

  var req = new cares.GetAddrInfoReqWrap();
  req.callback = callback;
  req.family = 4;
  req.hostname = hostname;
  req.oncomplete = callback;

  var err = cares.getaddrinfo(req, hostname, 4, 0);

  if (err) {
    callback(new Error("OOPS"));
    return {};
  }
  return req;
}

Calling function

var look = require("./fauxdns.js");
look("nodesource.com", function (err, data) {
  console.log(data);
});

Without a policy file we're able to access the network via the DNS binding:

$ node lookup.js
[ '162.243.142.64' ]

With a policy file that prevents just the dns module:

$ cat disable_dns.js
{
  "blacklist": {
    "modules": {
      "dns": 2
    }
}

We can still access it!!

$ node --policies disable_dns.js lookup.js
[ '162.243.142.64' ]

To actually prevent this circumvention of the modulesBlacklist we need to blacklist the cares_wrap binding:

$ cat disable_cares.json
{
  "blacklist": {
    "bindings": {
      "cares_wrap": 2
    }
  }
}

Now the behavior is blocked!

$  node --policies disable_cares.json lookup.js
nsolid error Binding "cares_wrap" is requested via a process.binding call
nsolid error at Object.<anonymous> (/tmp/fauxdns.js:3:21)
nsolid error But it was disabled in the policies!
_module_wrap.js:250
      throw new Error('Binding "' + id +
            ^
Error: Binding "cares_wrap" is black listed, but no replacement was found.
    at process.bindingWrap [as binding] (_module_wrap.js:250:13)
    at Object.<anonymous> (/tmp/fauxdns.js:3:21)
    at Module._compile (module.js:431:26)
    at Object.Module._extensions..js (module.js:449:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:311:12)
    at Module.require (module.js:366:17)
    at require (module.js:385:17)
    at Object.<anonymous> (/tmp/lookup.js:1:74)
    at Module._compile (module.js:431:26)

A policy file that disables several modules and bindings

{
  "blacklist": {
    "modules": {
      "child_process": 2,  -- blacklist entire child_process module with severity level 2
      "fs": 3,             -- blacklist entire fs module with severity level 3
      "path": 0            -- blacklist path with severity level 0, same as omitting `path` from blacklist all together
    }
    "bindings": {
      "stream_wrap": 2     -- blacklist entire stream_wrap binding with severity level 2
    },
  },
  "process": {
    "zeroFillAllocations": true  -- zero fill all memory allocations i.e. for Buffers
  }
}

Comments

We allow properties whose keys start with"//" anywhere and they are ignored.

Invalid Policies

Anytime an unexpected property or a property with an invalid value is encountered, the policies are considered invalid and the process halted.

Further Thoughts

N|Solid's policies feature should not be used as a blind replacement for good security practices with regard to your code and the code of your application's dependencies. Disabling core modules and bindings does not provide a complete guarantee that your application is unable to use the features provided by those parts of Node.js. For example, child processes interacting with Unix utilities, if not disabled, can be used to replace the functionality of many of the core modules. Some core modules may be completely re-implemented in user-land modules, such as the dns and cares_wrap functionality. Compiled add-ons have complete freedom to insert new functionality that can replace anything that has been disabled.\n\nN|Solid's policies features augment and support existing enterprise-wide application security hardening programs. Please contact NodeSource if you wish to discuss your Node.js application security needs and prepare a customized security hardening program that involves the use of internal and third-party code review, application monitoring and the use of operating system security tools.

Modules that can be Blacklisted

The following is a list of modules that can be blacklisted. It should be noted that for most of these modules it is rare to have a business need where blacklisting makes sense.

Bindings that can be blacklisted

The following is a list of bindings that can be blacklisted. It should be noted that for most of these bindings it is rare to have a business need where blacklisting makes sense.

Console Overview

The N|Solid Console provides valuable insight into clusters of N|Solid processes running in a variety of configurations. Processes are grouped by their associated application which makes it simple to find outliers in memory/CPU utilization and quickly identify unexpected behavior.

Navigating to an individual process shows various metrics and provides the capability for taking heap snapshots and CPU profiling.

Stats Retention

Collected metrics are stored in memory and restarting ncollect will cause these to be lost. Heap snapshots and CPU profiles are stored in the directory specified with the --storage command line argument specified below.

Historical metrics are currently trimmed at 500 records, which is roughly 8.5 minutes worth of historical data when polling once per second. If you would like to increase the historical range, specify a higher --interval value when launching ncollect

Moving the Console Instance

Simply copy the directory specified by the --storage command line argument on the source host to the new host. Be sure to specify the new storage directory with --storage on the new host.

Command Line Arguments

These are the command line arguments if you're starting the console explicitly. For most usages, you would utilize npm start.

Common Issues

Utilizing Console Logs

The N|Solid Management Console logs are stored at /var/log/nsolid-console.log or output to the terminal, depending on the system configuration (see Installing the console: Step 5 - Setup ~ns/.nsolid-clirc.....). If errors are encountered while using the console, these logs can be used to help identify/fix the problem when conversing with N|Support. During N|Support correspondence, any error messages in your browser's console should also be included to provide as much context about your problem as possible.

DEBUG=* node bin/ncollect.js --interval=1000 --storage=/path/to/dir

Networking Issues

Please refer to the Installing N|Solid Console section for troubleshooting.

Application View

The Applications view provides a visual overview of N|Solid applications monitored by the Console. There are six columns in this view:

Cluster View

The Cluster view is a force-directed scatter-plot that provides an overview of your application’s performance across all connected processes. The Y axis plots the ratio of that process’ Resident Set Size (RSS) memory usage over the application’s total RSS memory usage. The X axis plots the % CPU utilized for each process.

Move your mouse over a node in the graph to see its PID:

Clicking a node will select it, displaying more information about the process in the sidebar on the right:

You can use this to get an overview of the following metrics over time for the selected process:

• Async Activity (Requests, Handles)
• Host Data (Memory Used, Load Average)
• Resident Set Size (MB)
• Heap total (MB)
• Heap used (MB)

Clicking the “Expand” button on the edge of the sidebar will open the Process Detail View.

Process View

The Process view shows information relating to a specific process. To select another process, click [Collapse] to close the process view. Then, select the dot in the Cluster view that represents the process of interest. Finally, click [Expand] on the sidebar to return to the process view.

Uptime: This is the total time elapsed from the moment when the process was started.

Async Activity: This chart displays two series that reflect asynchronous activity. One is the number of async handles, which tend to be generated by longer-lived larger-scale asynchronous operations, such as open sockets or timers. The other is the number of async requests. These tend to be made by shorter-lived and smaller-scale operations, such as writing to file handles. (For more information, read this.)

Host Data: Host data is a statistic based on the percentage of memory used and the CPU load average over the last minute. For details on the host system running the process, click [🔎 More] underneath the Host Data chart.

Resident Set Size: This chart shows the actual memory footprint of the process over time. It does not include memory that belongs to the process that has been swapped out to disk.

Heap Total: This is total size of the v8 heap, which includes everything in the heap, including free space which has been allocated by v8 but is unused by JavaScript objects.

Heap Used: This is the size of of the v8 heap occupied directly by JavaScript objects created by the application, as well as v8 internal objects created on behalf of it.

Related Snapshots: This is a list of heap snapshots associated with this process. When you click [+ New Snapshot], you will be taken to the heap snapshot viewer and a new row representing the heap snapshot will be added to this list.

Related Profiles: This is a list of profiles associated with this process. When you click [+ New Profile], you will be taken to the profile viewer and a new row representing the profile will be added to this list.

Host details

This view shows information about the host system, including the hostname, how long the host system has been running, the name of the operating system platform, as well as information about the CPU including the manufacturer, model, clock speed and number of cores. At the very bottom is a graph that shows the total system-wide load over time. This screen is accessed by clicking on the 'More' link with the magnifying glass to the center right of the screen.

Process And System Statistics

If you’re looking for information regarding process and system statistics you’ve come to the right place. N|Solid gives you instant access to this valuable information both via our command line interface (CLI) and the console.

N|Solid provides useful command line tools for process and system statistics.

process_stats

Usage

nsolid-cli -- socket XXXX process_stats

Returns live metrics reflecting the health and resource utilization of the N|Solid process.

Example JSON Result:

{
  "uptime": 2580.584,
  "rss": 25083904,
  "heapTotal": 14423040,
  "heapUsed": 8272800,
  "active_requests": 1,
  "active_handles": 2,
  "title": "nsolid",
  "user": "user",
  "cpu": 0
}

system_stats

Usage

nsolid-cli -- socket XXXX system_stats

Returns live metrics reflecting the health and resource utilization of the system hosting the N|Solid process.

Example JSON Result

{
  "freemem": 3137511424,
  "uptime": 697783,
  "load_1m": 1.54150390625,
  "load_5m": 1.4052734375,
  "load_15m": 1.44189453125,
  "cpu_speed": 2500
}

Read more about memory load averages here.

system_info

Usage

nsolid-cli -- socket XXXX system_info

Returns static information about the hosting system for this N|Solid process.

Example JSON Result:

{
 "cpu_cores": 8,
  "cpu_model": "Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz",
  "arch": "x64",
  "platform": "darwin",
  "hostname": "user.local",
  "totalmem": 17179869184
}

system_stats for a N|Solid Hub

The above example shows usage for connecting to a single local process; this command can also be used against a hub that is aggregating data from several N|Solid instances and/or a single process on a remote machine by including the full address

$ nsolid-cli --socket 192.168.0.111:60414 system_stats

The --socket option should be the hostname/port number of a particular N|Solid process, or the hostname/port number of an N|Solid hub that is connected to a number of N|Solid processes. The output will either be a single line of JSON, for a single N|Solid process, or multiple lines of JSON for an N|Solid hub. The json command can be used display this data in an easier to use format, and the json option --group can be used to handle each separate line of JSON from the N|Solid hub as an array of items:

$ nsolid-cli --socket 9000 system_stats | json --group # processes in hub

Example JSON Result

[
  {
    "meta": {
      ... meta information about the process
    },
    "reply": {
      "freemem": 387833856,
      "uptime": 1199433,
      "load_1m": 1.8251953125,
      "load_5m": 2.478515625,
      "load_15m": 2.58056640625,
      "cpu_speed": 2500
    }
  },
  {
    "meta": {
      ... meta information about the next process
    },
    "reply": {
     ...
    }
  }
]

info

Usage

nsolid-cli --socket XXXX info

Example JSON Result

{
  "meta": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "hostname": "cluster-1.oregon",
    "address": "192.168.2.27:38580"
  },
  "reply": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "nodeEnv":"production",
    "execPath": "/usr/local/bin/nsolid",
    "main": "/var/myapp/web/server.js"
  }
}
{
  "meta": {
    "id": "ccd48968f3212a06c3aff62bbfa75c481ad5f672",
    "app": "api-server",
    "pid": 5447,
    "hostname": "cluster-1.oregon",
    "address": "192.168.2.27:52352"
  },
  "reply": {
    "id": "ccd48968f3212a06c3aff62bbfa75c481ad5f672",
    "app": "api-server",
    "pid": 5447,
    "nodeEnv":"production",
    "execPath": "/usr/local/bin/nsolid",
    "main": "/var/myapp/api/server.js"
  }
}

versions

Usage

nsolid-cli -- socket XXXX versions

Returns the versions of bundled components that make up the N|Solid runtime

Example JSON Result:

{
  "http_parser": "2.5.0",
  "node": "4.2.2",
  "nsolid": "1.1.0",
  "v8": "4.5.103.35",
  "uv": "1.7.5",
  "zlib": "1.2.8",
  "ares": "1.10.1-DEV",
  "modules": "46",
  "openssl": "1.0.2d",
  "nsolid_lib": {
    "function_origin": "nsolid-v1.2.0",
    "nan": "v2.1.0",
    "v8_profiler": "nsolid-v5.2.3-fix1",
    "agent": "v4.2.2",
    "cli": "v1.3.3"
  }
}

System and Process Statistics via the Console

Critical information about the process and server is available within the console via a few simple clicks.

Select the application of interest

Expand the details window and statistics detailed above via the command line interface are neatly displayed for viewing.

Process and system information for this process can be seen in the next window.

Additional information is available to the right by clicking the 'More' link with the magnifying glass.

Socket Flag Usage

The --socket flag is necessary if you are connecting directly to the N|Solid Agent without using a N|Solid Hub; the common starting point of those evaluating N|Solid. If a full installation, we suggest you configure your nsolid-cli to use the N|Solid Hub.

Viewing Async Activity

A hanging or bogged down system is always a nightmare to troubleshoot. Fortunately N|Solid can report on any pending Asynchronous operations and where the JavaScript callbacks will resume. This can be extremely helpful for determining how many concurrent operations are pending and what a given N|Solid process is handling. This feature is also useful for determining where a stuck application is hanging or if there are any left-behind callbacks, or misuse of resources. (e.g. reading the same file repetitively)

For those unclear regarding the difference between the two:

• Async handles tend to be longer-lived larger-scale asynchronous operations, such as open sockets or timers.
• Async requests tend to be shorter-lived smaller-scale operations such as writing to file handles and other file-related operations.

Viewing Async Activity via command 'async_activity'

Usage:

nsolid-cli -- socket XXXX aysnc_activity

Example JSON Result:

{
  "handles": [
    {
      "name": "onInterval",
      "location": {
        "file": "/Users/user/nsolid-demos/demos/basic.js",
        "line": 19,
        "column": 19,
        "inferredName": ""
      },
      "anonymous": false,
      "source": "function onInterval() {}",
      "type": {
        "flag": 2,
        "name": "setInterval"
      },
      "details": {
        "msecs": 1000
      }
    }
  ],
  "requests": [
    {
      "name": "onRead",
      "location": {
        "file": "/Users/user/nsolid-demos/demos/basic.js",
        "line": 22,
        "column": 15,
        "inferredName": ""
      },
      "anonymous": false,
      "source": "function onRead(err, bytesRead, buffer) {...}",
      "type": {
        "flag": 32,
        "name": "read",
        "api": "fs.read(fd, buffer, offset, length, position, callback)"
      },
      "details": {
        "fd": 0,
        "buffer": {
          "type": "Buffer",
          "data": [
            0
          ]
        },
        "offset": 0,
        "length": 1,
        "position": null
      }
    }
  ]
}

Async Activity via the Console

Async activity can also be tracked on a process by process basis via the console. Detailed information about this functionality can be read here

Socket Flag Usage

The --socket flag is necessary if you are connecting directly to the N|Solid Agent without using a N|Solid Hub; the common starting point of those evaluating N|Solid. If a full installation, we suggest you configure your nsolid-cli to use the N|Solid Hub.

Profiling Startup Times

If you’re looking to see how long it takes to startup your application, check out our startup_times command. It will report to the nanosecond how long it took to bring core application elements such as Node.js and v8 online. You can also register your own custom life-cycle events by using process.recordStartupTime().

startup_times

Usage:

nsolid-cli -- socket XXXX startup_times

Lists the time from initial process execution to reach certain process lifecycle startup phases.

Example JSON Result:

{
  "initialized_node":   [ 0, 130404 ],
  "initialized_v8":     [ 0, 482651 ],
  "loaded_environment": [ 0, 620207709 ]
}

This indicates that node was initialized in 130,404 nanoseconds (which is 130 microseconds, 0.130 milliseconds, or 0.000130 seconds).

Time format

The timing information is provided in hrtime format, which is a two element array of [seconds, nanoseconds]. A nanosecond is one billionth (1,000,000,000th) of a second. The time values are the elapsed time since the process was started."

Tutorial for Custom Timers

Learn how to implement a simple custom startup timer.

Custom Commands

As a developer, you can trigger your own custom commands via the N|Solid Command Line Interface (CLI). Custom commands allow you to interact with your application's processes in ways specific to your business needs.

You can implement a custom command by creating a function to handle the command, and then registering the function with N|Solid.

The custom command function should be defined to take a single parameter, request:

function customCommandHandler(request) { ... }

The request parameter is an object with the following properties/functions:

Your function you must call either request.return() or request.throw() to signal completion of the command.

A custom command handler is registered using the nsolid.on() method. You can get access to N|Solid's built-in nsolid module, by calling require("nsolid"). The nsolid.on() function takes the following parameters:

nsolid.on(commandName, handler)

The structure of implementing and registering a custom command will look like the code below:

...
const nsolid = require(\"nsolid\")
...
nsolid.on(\"foo\", fooCommand)
...
function fooCommand(request) { ... }

Custom Commands with Hub

Custom Commands can be "broadcast" via the hub if added to the .nsolid-proxyrc file's broadcast_approved list, as explained here.

Example Implementation of Custom Command 'foo'

var nsolid = require("nsolid")

nsolid.on("foo", fooCommand)

function fooCommand(request) {
  console.log("ran the N|Solid custom command 'foo'")
  return request.return({arg: request.value})
}

setInterval(function(){}, 1000) // so process never stops

console.log("this program will run until you kill it")
console.log("")
console.log("to run the foo command with `nsolid-cli`, run:")
console.log("   nsolid-cli --socket 5000 foo --data bar")

Running foo-command.js

$ NSOLID_SOCKET=5000 nsolid foo-command.js
This program will run until you kill it.

to run the foo command with `nsolid-cli`, run:
   nsolid-cli --socket 5000 foo --data bar

Example invoking the foo command with the nsolid-cli program:

$ nsolid-cli --socket 5000 foo --data bar

> {"arg": "bar"}

Tutorial

Changing Log levels with Custom Commands

CPU Profiling

CPU profiling allows you to understand where opportunities exist to improve the speed and load capacity of your node process. N|Solid provides multiple ways to profile your CPU.

Analyzing Profiles Using the N|Solid Console

The N|Solid Console allows you to take and analyze profiles in one action. This is particularly useful if you need to take a series of profiles as it saves you the overhead of switching from one environment to another. The console also saves a history of profiles so you can quickly flip through profiles to identify troublesome areas.

1. Launch the console and select the application of interest
2. Select the process of interest
3. Expand out the details panel and select ‘+ New Profile’
   
4. Select your profile window (5 to 60 seconds) and run profile
5. Select a visualization (Sunburst, Flamegraph or Treemap). Once a visualization is selected, you can click on it to view the call stack to the right. The number of calls to that method are shown as are the time spent within it. Clicking on the method will show you the file responsible for that call.
6. Profiles are saved in your console session so you can easily flip between profiles.

Profiling using the Command Line Interface (CLI)

N|Solid's Command Line Interface (CLI) is a great way to quickly pull profiles from remote processes for local examination. With NSolid installed and on your path, a pair of commands are used to accomplish this.

profile_[start, stop]

Usage:

nsolid-cli -- socket XXXX profile_start

nsolid-cli -- socket XXXX profile_stop > my.cpuprofile

Collection will commence immediately after ‘profile_start’ hits the wire and will continue for up to 60 seconds. If profile_stop is not entered before 60 seconds then the process times out and the profile data is discarded. Order of execution is always profile_start, then profile_stop.

Looking to do a timed test?

You can do this with some simple scripting!

nsolid-cli --socket 5000 profile_start\n
ab -n 1000 -c 100 http://localhost:8000\n
nsolid-cli --socket 5000 profile_stop > 1.cpuprofile

This will do a CPU profile while running a load test with ab (Apache Bench). If you wanted to do a timed one, replace the ab invocation with sleep 10 (to sleep for 10 seconds)."

Once the .cpuprofile file has been created, it can be opened using Chrome’s Development Tool’s CPU Profile Debugging Tool. It’s important to note that Chrome requires the generated file to have the extension .cpuprofile so be sure to use that.

Profiling explained

Learn how profiling can help build a better product.

Need Help Installing the Console?

Read our Console Installation Guide.

Heap Snapshots

If you have a memory leak or performance issue, taking heap snapshots is a great way to help identify the underlying problem. N|Solid provides you with two ways to do this, a command-line friendly method and via our console.

Analyzing Heap Snapshots Using the N|Solid Console

Using N|Solid's Console you can quickly identify troublesome processes and profile visually. The Console allows developers to take a series of snapshots and quickly switch between them to better identify problematic areas.

1. Launch the console and select the application of interest
2. Select the process of interest
3. Expand out the details panel and select ‘+ New Snapshot’
   This will take the snapshot and open the results in the snapshot window. To retake the snapshot, click the 'New Snapshot' button. The 'Related Snapshots' section in the previous screen provides a record of all the snapshots in the session.
4. Once the snapshot completes you can navigate through the various objects indexed during the snapshot process. As there can be a large number of indexed objects, you can use the Filter search box to narrow the results. Below you can see a listing of the various Array objects found in memory. To take another snapshot, simply press the New Snapshot button to the left. Snapshots taken in the N|Solid Console can be downloaded to your local drive for exploration via a 3rd party tool (i.e. Chrome Developer Tools). If you wish to download the profile for analysis through other tools, simply click the Download Snapshot button

Profiling using the Command Line Interface (CLI)

N|Solid's Command Line Interface (CLI) is a great way to quickly take heap snapshots from remote processes for local examination. With NSolid installed and on your path, the following command is used to do this:

snapshot

Usage:

nsolid-cli -- socket XXXX snapshot > my.heapsnapshot"

Once the .heapsnapshot file has been created, it can be opened using Chrome’s Development Tool’s CPU Profile Debugging Tool It’s important to note that Chrome requires the generated file to have the extension .heapsnapshot so be sure to use that.

Snapshots Explained

Learn how prepping your code for snapshots can lead to more actionable results.

Memory Explained

Check out this great Google document regarding Heap Profiling

Custom Lifecycle Events

Implementing custom lifecycle events

In addition to the built-in life-cycle events, you can add your own using the function process.recordStartupTime(label) function. The label will then be used in the JSON output of the startup_times command.

You can use this to record the times at various stages of your application's startup. For instance:

• when a database connection is requested
• when the database connection is returned
• when an http server is ready for incoming events

Using the CLI

To obtain the startup timing values, you can use the nsolid-cli startup_times command. For example:

$ nsolid-cli --socket 5000 startup_times

Example JSON Result:

{
  "initialized_node":   [ 0, 130404 ],
  "initialized_v8":     [ 0, 482651 ],
  "loaded_environment": [ 0, 620207709 ]
}

This indicates that node was initialized in 130,404 nanoseconds (which is 130 microseconds, 0.130 milliseconds, or 0.000130 seconds).

Time format

The timing information is provided in hrtime format, which is a two element array of [ seconds, nanoseconds]. A nanosecond is one billionth (1,000,000,000th) of a second. The time values are the elapsed time since the process was started.

Adding a custom timer

Below is an example web server which is instrumented to provide the time when the web server starts listening to connections:

const http = require("http")

const server = http.createServer(onRequest)

server.listen(3000, onListen)

function onListen() {
  console.log("server listening on http://localhost:3000")
  process.recordStartupTime("http_listen")
}

function onRequest(request, response) {
  response.end("Hello, world!")
}

To start this program with the N|Solid agent listening on port 5000, use the following command:

$ NSOLID_SOCKET=5000 node http_sample.js

To obtain the startup times, use the following command:

nsolid-cli --socket 5000 startup_times

This will return the following JSON output:

{
  "initialized_node":   [ 0, 129554 ],
  "initialized_v8":     [ 0, 460521 ],
  "loaded_environment": [ 0, 95201339 ],
  "http_listen":        [ 0, 94902772 ]
}

Three startup times are provided by N|Solid by default:

Log Level Custom Command

As a developer, you can trigger your own custom commands via the N|Solid Command Line Interface (CLI). Custom commands allow you to interact with your application's processes in ways specific to your business needs.

You can implement a custom command by creating a function to handle the command, and then registering the function with N|Solid. Detailed information about custom command usage can be reviewed here.

Below is an example of how you can use custom commands to dynamically change the configuration state of the application; specifically the log level. The example assumes that a global boolean variable Verbose is used to indicate whether to log verbosely or not.

Log Level Custom Command Code Tutorial

//------------------------------------------------------------------------------
// This program is a simple "server" which does nothing, but does implement
// an N|Solid custom command, named `verbose`.  Once you've started this program
// with the N|Solid agent enabled, you can send the `verbose` command as in:
//
//     nsolid-cli --socket <nsolid-agent-address> verbose
//     nsolid-cli --socket <nsolid-agent-address> verbose --data on
//     nsolid-cli --socket <nsolid-agent-address> verbose --data off
//
// All these forms get or set the "verbose" level of logging in the program.
//
// The server logs a message every second when "verbose" is off, and logs
// an additional message after that one when "verbose" is on.  The default
// setting of "verbose" is false.
//------------------------------------------------------------------------------

"use strict"

// get access to N|Solid's built-in module `nsolid`
const nsolid = require("nsolid")

// the current "verbose" level
let Verbose = false

// register the `verbose` command for nsolid-cli
nsolid.on("verbose", verboseCommand)

// our server which doesn't do much
setInterval(onInterval, 2000)

console.log("N|Solid custom command demo - log-level - started")
console.log("")
console.log("to use the verbose command with `nsolid-cli`, run:")
console.log("   nsolid-cli --socket <nsolid-agent-address> verbose")
console.log("   nsolid-cli --socket <nsolid-agent-address> verbose --data on")
console.log("   nsolid-cli --socket <nsolid-agent-address> verbose --data off")

//------------------------------------------------------------------------------
function onInterval() {
  log("interval event!")
  logVerbose("some extra logging here")
}

//------------------------------------------------------------------------------
// implements the `verbose` command  for nsolid-cli
//------------------------------------------------------------------------------
function verboseCommand(request) {

  // if "on" or "off" passed in with --data, set Verbose appropriately
  if (request.value == "on") {
    Verbose = true
  }
  else if (request.value == "off") {
    Verbose = false
  }
  else if (request.value) {
    return request.throw("expecting data of `on` or `off`, got " + request.value)
  }

  // return current value of Verbose
  return request.return({verbose: Verbose})
}

//------------------------------------------------------------------------------
function log(message) {
  console.log(message)
}

//------------------------------------------------------------------------------
function logVerbose(message) {
  if (!Verbose) return
  log(message)
}

When running your application with the N|Solid agent active, you can use the following command to return the current value of the Verbose setting:

nsolid-cli --socket 5000 verbose

To set Verbose on or off, use one of the following commands:

nsolid-cli --socket 5000 verbose --data on\nnsolid-cli --socket 5000 verbose --data off

Integrating StatsD

The data returned from nsolid-cli is in JSON format, so it's easy to pull data from your N|Solid runtimes, and send it to an application monitoring aggregator. For instance, you could write a program to invoke nsolid-cli to collect performance data, and send it to a statsd aggregator.

Here's a complete example which will run nsolid-cli process_stats every 5 seconds, and send the resulting data to statsd as gauge values:

'use strict'

const path = require('path')
const dgram = require('dgram')
const child_process = require('child_process')

//------------------------------------------------------------------------------
const PROGRAM = path.basename(__filename).split('.')[0]

const Socket = dgram.createSocket('udp4')

let NSolidCliOpts = ''
let StatsdHost    = 'localhost'
let StatsdPort    = '8125'

// parse parameters
parseArgs()

let MetricsCount = 0

// get stats on user-specified interval
setInterval(collectStats, 5 * 1000)

//------------------------------------------------------------------------------
function collectStats() {
  nsolidCommand('process_stats', handleProcessStats)
}

//------------------------------------------------------------------------------
function handleProcessStats(err, messages) {
  // log process stats
  if (err) return

  messages.forEach(function(message) {
    const app      = message.meta.app
    const pid      = message.meta.pid
    const instance = app + '.' + pid

    const stats = message.reply

    logGauge(instance, 'process.rss',             stats.rss)
    logGauge(instance, 'process.heapTotal',       stats.heapTotal)
    logGauge(instance, 'process.heapUsed',        stats.heapUsed)
    logGauge(instance, 'process.active_requests', stats.active_requests)
    logGauge(instance, 'process.active_handles',  stats.active_handles)
  })
}

//------------------------------------------------------------------------------
function logGauge(instance, key, val) {
  const msg = new Buffer(instance + '.' + key + ':' + val + '|g')

  MetricsCount++
  Socket.send(msg, 0, msg.length, StatsdPort, StatsdHost, onSend)

  //-----------------------------------
  function onSend(err, bytes) {
    if (!err) return

    console.log('error sending gauge:', err)
  }
}

//------------------------------------------------------------------------------
function nsolidCommand(command, cb) {
  // invoke an nsolid command, send JSON to callback

  const cmd = 'nsolid-cli ' + NSolidCliOpts + ' ' + command
  child_process.exec(cmd, execCB)

  //-----------------------------------
  function execCB(err, stdout, stderr) {
    if (err) return cb(err)

    let data = '' + stdout

    try {
      data = data.trim().split('\n').map(function(datum) {
        return JSON.parse(datum)
      })
    }
    catch (err) {
      return cb(err)
    }

    cb(null, data)
  }
}

//------------------------------------------------------------------------------
function parseArgs() {
  const argv = process.argv.slice(2)

  if (-1 != ['?', '-?', '--?', '-h', '--h', 'help'].indexOf(argv[0])) {
    return help()
  }

  if (argv[0]) NSolidCliOpts = argv[0]
  if (argv[1]) StatsdHost    = argv[1]
  if (argv[2]) StatsdPort    = argv[2]

  log('nsolid-cli options: ' + (NSolidCliOpts || '""' ))
  log('statsd server:      udp://' + StatsdHost + ':' + StatsdPort)
}

//------------------------------------------------------------------------------
function log(message) {
  console.log(PROGRAM + ': ' + message)
}

//------------------------------------------------------------------------------
function error(message) {
  log('error: ' + message)
  process.exit(1)
}

//------------------------------------------------------------------------------
function help() {
  console.log('usage: ' + PROGRAM + '[nsolid-cli-opts] [statsd-host] [statsd-port]')
  console.log('')
  console.log('Poll for nsolid statistics using nsolid-cli and post to statsd.')
  console.log('')
  console.log('Default values for parameters:')
  console.log('  nsolid-cli-opts: ' + NSolidCliOpts)
  console.log('  statsd-host:     ' + StatsdHost)
  console.log('  statsd-port:     ' + StatsdPort)
  process.exit(0)
}

The program takes 3 optional command-line parameters:

• nsolid-cli options (eg, --hub 5000) (default empty string)
• host name of the statsd UDP server (default localhost)
• port of the statsd UDP server (default 8125)

The program currently assumes nsolid-cli will return aggregated results,instead of results for a single N|Solid agent. The gauge names take the following format:

<N|Solid app name> . <N|Solid process id> . "process" . <stat name>

For N|Solid app name "MyApp", the following sort of gauge values will be posted to the statsd server:

MyApp.56114.process.rss
MyApp.56114.process.heapTotal

N|Solid Command Line Interface (CLI)

The N|Solid Agent comes with a set of pre-defined endpoints for interaction and introspection and can be customized to add additional endpoints.

Some commands accept input from the command line, this can be supplied via nsolid-cli --data '{"json":"object"} ....

ping

This is the agent's availability check, it will always reply with "PONG" if online.

Usage

nsolid-cli --socket XXXX ping

Example JSON Result:

"PONG"

info

Usage

nsolid-cli --socket XXXX info

Returns information about the running process and N|Solid Agent configuration:

Example JSON Result:

{
  "meta": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "hostname": "cluster-1.oregon",
    "address": "192.168.2.27:38580"
  },
  "reply": {
    "id": "9f6e7ab8f323a33a31f56bafed32ebc172777dd8",
    "app": "web-server",
    "pid": 5416,
    "nodeEnv":"production",
    "execPath": "/usr/local/bin/nsolid",
    "main": "/var/myapp/web/server.js"
  }
}
{
  "meta": {
    "id": "ccd48968f3212a06c3aff62bbfa75c481ad5f672",
    "app": "api-server",
    "pid": 5447,
    "hostname": "cluster-1.oregon",
    "address": "192.168.2.27:52352"
  },
  "reply": {
    "id": "ccd48968f3212a06c3aff62bbfa75c481ad5f672",
    "app": "api-server",
    "pid": 5447,
    "nodeEnv":"production",
    "execPath": "/usr/local/bin/nsolid",
    "main": "/var/myapp/api/server.js"
  }
}

versions

Returns the versions of bundled components that make up Node.js and N|Solid.

Usage

nsolid-cli --socket XXXX versions

Example JSON Result:

{
  "http_parser": "2.5.0",
  "node": "4.2.2",
  "nsolid": "1.1.0",
  "v8": "4.5.103.35",
  "uv": "1.7.5",
  "zlib": "1.2.8",
  "ares": "1.10.1-DEV",
  "modules": "46",
  "openssl": "1.0.2d",
  "nsolid_lib": {
    "function_origin": "nsolid-v1.2.0",
    "nan": "v2.1.0",
    "v8_profiler": "nsolid-v5.2.3-fix1",
    "agent": "v4.2.2",
    "cli": "v1.3.3"
  }
}

process_stats

Usage

nsolid-cli --socket XXXX process_stats

Returns live metrics reflecting the health and resource utilization of the N|Solid process.

Example JSON Result:

{
  "uptime": 2580.584,
  "rss": 25083904,
  "heapTotal": 14423040,
  "heapUsed": 8272800,
  "active_requests": 1,
  "active_handles": 2,
  "title": "nsolid",
  "user": "user",
  "cpu": 0
}

## system_info

> **Usage**
>
> nsolid-cli -- socket XXXX **system_info**

Returns static information about the hosting system for this N|Solid process.

| Parameter | Description |
|----------------|----------------------------------------|
| cpu_cores | The number of CPU cores |
| cpu_model | The CPU model |
| arch | The CPU architecture |
| platform | Name of the NSolid platform |
| hostname | The host name |
| totalmem | Total available memory in the system |

*Example JSON Result:*

```json
{
 "cpu_cores": 8,
  "cpu_model": "Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz",
  "arch": "x64",
  "platform": "darwin",
  "hostname": "user.local",
  "totalmem": 17179869184
}

system_stats

Usage

nsolid-cli -- socket XXXX system_stats

Returns live metrics reflecting the health and resource utilization of the system hosting the N|Solid process.

Example JSON Result:

{
  "freemem": 3137511424,
  "uptime": 697783,
  "load_1m": 1.54150390625,
  "load_5m": 1.4052734375,
  "load_15m": 1.44189453125,
  "cpu_speed": 2500
}

startup_times

Usage

nsolid-cli -- socket XXXX startup_times

Lists the time from initial process execution to reach certain process lifecycle startup phases.

The times are provided in hrtime syntax: a Tuple Array of 2 integers representing the time since the initial process execution in [seconds, nanoseconds]. Example JSON Result

{
  "initialized_node":   [ 0, 130404 ],
  "initialized_v8":     [ 0, 482651 ],
  "loaded_environment": [ 0, 620207709 ]
}

Custom startup timers

If you're looking for information on how to add custom startup and lifecyle events check out this tutorial.

async_activity

Usage

nsolid-cli -- socket XXXX aysnc_activity

The N|Solid Agent can at any time list any pending Asynchronous operations and where the JavaScript callbacks will resume. This can be extremely helpful for determining how many concurrent operations and what they are a given N|Solid process is handling. It can also be useful to determine where a stuck application is hanging or if there are any left-behind callbacks, or misue of resources (e.g. reading the same file repetitively)

The output will list the inflight asynchronous activity, including the function location and line of source code to reference where the callback will resume.

Calling the async_activity endpoint will list two types of inflight async work -- handles and requests.

Async handles tend to be longer-lived larger-scale asynchronous operations, such as open sockets or timers.

Async requests tend to be shorter-lived smaller-scale operations such as writing to file handles and other file-related operations.

Example JSON Result:

{
  "handles": [
    {
      "type": {
        "flag": 8,
        "name": "TCP socket connection"
      },
      "details": {
        "fd": 14,
        "transport": "TCP",
        "bytesDispatched": 0,
        "hadError": false,
        "host": null,
        "peerAddress": "::ffff:127.0.0.1",
        "peerFamily": "IPv6",
        "peerPort": 42097,
        "sockAddress": "::ffff:127.0.0.1",
        "sockFamily": "IPv6",
        "sockPort": 8080,
        "ondata": "socketOnData: _http_server.js:318:23",
        "onclose": "serverSocketCloseListener: _http_server.js:263:36",
        "onend": "socketOnEnd: _http_server.js:360:22"
      }
    },
    {
      "name": "__unknown_function_name__",
      "location": {
        "file": "[eval]",
        "line": 1,
        "column": 104,
        "inferredName": ""
      },
      "anonymous": true,
      "source": "function () {res.end(\"ok\")}",
      "type": {
        "flag": 1,
        "name": "setTimeout"
      },
      "details": {
        "msecs": 20000
      }
    },
    {
      "type": {
        "flag": 8,
        "name": "TCP socket connection"
      },
      "details": {
        "fd": 15,
        "transport": "TCP",
        "bytesDispatched": 0,
        "hadError": false,
        "host": null,
        "peerAddress": "::ffff:192.168.1.104",
        "peerFamily": "IPv6",
        "peerPort": 54416,
        "sockAddress": "::ffff:192.168.1.104",
        "sockFamily": "IPv6",
        "sockPort": 35770,
        "ondata": "socketOnData: _http_server.js:318:23",
        "onclose": "serverSocketCloseListener: _http_server.js:263:36",
        "onend": "socketOnEnd: _http_server.js:360:22"
      }
    }
  ],
  "requests": []
}

profile_start, profile_stop

Usage

nsolid-cli -- socket XXXX profile_start nsolid-cli -- socket XXXX profile_stop > my.cpuprofile

Collection will commence immediately after ‘profile_start’ hits the wire and will continue for up to 60 seconds. If profile_stop is not entered before 60 seconds then the process times out and the profile data is discarded. Order of execution is always profile_start, then profile_stop.

profile_start

Begins CPU Profile collection. It will collect for up to 60 seconds before timing out. If it times out, all profile data is discarded.

Only one CPU collection is allowed to be running at a time.

profile_stop

Must be run after profile_start -- completes the CPU Profile collection and sends the resulting cpuprofile data. In the N|Solid Console, this will be automatically loaded into our visualization system. From the CLI, the cpuprofile data can be saved to a file and then loaded into Chrome Dev Tools.

snapshot

Usage

nsolid-cli -- socket XXXX snapshot > my.heapsnapshot

Once the .heapsnapshot file has been created, it can be opened using Chrome’s Development Tool’s CPU Profile Debugging Tool It’s important to note that Chrome requires the generated file to have the extension .heapsnapshot so be sure to use that.

custom commands

The N|Solid Agent provides a JavaScript interface for implementing custom commands that can be triggered externally. For more information read about implementing custom commands here.

Socket Flag Usage

The socket flag is necessary if you are running everything on one machine; the common starting point of those evaluating N|Solid. If you are connecting to a HUB, you would not use --socket.

Using the Chrome Profiler with N|Solid

The V8 JavaScript engine used by Node includes a sample-based profiler. The profiler collects the state of the function call stack over a fixed time length, at a particular sampling rate, to return an approximation of the execution profile of your program. Functions that take a long time to execute (and those functions’ callers) tend to stick out in the resulting profile data. Using profiling data, you can:

• Find areas in your program that are running slowly
• Look for possible optimizations
• Improve the overall speed of your program.

Typically, the V8 profiler isn’t available for access in Node distributions, but it’s built into N|Solid and can be controlled through the N|Solid agent. With the N|Solid runtime, you can direct the agent to start and stop profiling, and then return the results back as a JSON data structure. From the N|Solid Console, you can also request a profile and have the results displayed immediately in the console, with several different views available to display the profiling data.

This following section is predicated on a .cpuprofile file already being generated. To do so you can generate it either from the CLI or Console.

Loading a .cpuprofile file into Chrome Dev Tools

• Open the Chrome web browser
• Open Chrome Dev Tools from the menu View / Developer / Developer Tools
• When the Developer Tools window (or pane) opens, select the tab Profiles
• Click the Load button in the Select profiling type area
• Find the profiling output you've saved to a file, and click the Open button
• Click on the profile that should now appear under CPU PROFILES list at the left
• Click on Chart in the view selector under the Chrome Dev Tools menu
• This will display a chart showing function execution times and what functions were on the stack at the time the system was sampled

For more information about using Chrome Dev Tools CPU profiling capability, see the Chrome help topic Profiling JavaScript Performance.

Profiling Basics

CPU profiling will indicate all the functions on the function call stack, when samples are taken. For instance, if a function foo() called a function bar() during its execution, and a sample was taken while bar() was running, the function call stack will show that foo() called bar(). Because multiple samples may be taken while bar() is executing, there will be an approximate start and stop time recorded for bar(), which is an indication of how long it took to run. In addition, further samples before and after the ones that captured bar() will capture foo(), and likewise to the bottom of the function call stack - typically, a callback of some sort.

This data can then be analyzed to show, for the function foo(), how much time was actually spent in foo() and not in bar() - every function has two time values - a “self” time and a “total” time. For the foo() and bar() case, if foo() only calls the bar() function, then the “self” time for foo() + the “total” time for bar() will equal the “total” time for foo().

Eg

function foo() {
        … processing that takes a lot of time but calls no other functions ...
    bar()
        … processing that takes a lot of time but calls no other functions …
}

function bar() {
    …
}

foo’s total time = foo’s self time + bar’s total time.

Both values are interesting; total time shows you which functions are fastest and slowest, from start to finish, but doesn’t tell you if the time was spent in that function or other functions.

Visualizations Explained

All the visualization available for profile data show the unique set of stack traces captured during the profile, with the "area" of the stack indicating the proportional time spent in a function compared to that of it's parent.

The Flame Graph visualization shows the time along the x axis. The y-axis is used to show show the function calls that make up a particular stack trace.

The Sunburst visualization is like the Flame Graph, where the x-axis is curved into a circle. Stack traces grow from the center to the outer parts of the graph.

The Treemap visualization shows time by area. The larger the rectangle, the more time a function took. The stack traces grow from the outer boxes into the inner boxes.

Generally when using the profiling views, you're looking for long-running functions. This means you're looking for wide rectangles in the Frame Graph, long radial sections in the Sunburst, and large areas in the Treemap.

The stack trace height - how many functions deep a particular stack trace was - doesn't indicate a time issue, necessarily. You'll want to focus on the time values.

As you hover over the visualization itself, the method tied to the visualization will be show. If you click, you'll see a stack trace to the right of the visualization. Other areas in the visualization may "light up" with a less bright color as you hover over the visualization - these are entries in stack traces for the same function and indicates additional CPU usage for the function under the cursor.

So, the things to look for are stack trace entries with that take up the most area: width in Flame Graph, circumference in Sunburst, and area in TreeMap.

Named Functions

Named functions are easier to spot in CPU profiles. The stack frame entries available in a CPU Profile include the name of function, and source code location information for detail views. For anonymous functions, the name will often be displayed as "(anonymous)". In some cases, the V8 JavaScript engine can calculating an "inferred name" for a function. When in doubt, named your functions so that you can easily spot them in a CPU profile.

For instance, let's say you have a function busyFunction(), which you like to easily track the callers for, and you have cases where it's called from an anonymous function, like this:

setInterval(function(){busyFunction()}, 1000)

In a CPU profile, you'll see that busyFunction() is being called by(anonymous) in this case.

To make this easier to easier to spot in a CPU profile, you can simply use:

setInterval(function busyFunctionCaller(){busyFunction()}, 1000)

In the CPU profile, you'll now see that busyFunction() is called by busyFunctionCaller().

For additional code cleanliness, and less chance of creating a "pyramid of doom", consider moving the entire function out into the same scope as the function usage; for instance:

setInterval(busyFunctionCaller, 1000)
//...
function busyFunctionCaller() {
  busyFunction()
}

Because JavaScript functions are "hoisted" to the top level of the scope they're defined in, you can reference busyFunctionCalled before it's actually defined.

Snapshot Basics and Best Practices

What can analyzing heap snapshots reveal?

Heap Snapshots capture a record of all live JavaScript objects available in your node application, at the time the Snapshot is captured. The objects are grouped by their constructor name, with aggregated counts of the total number of objects, their shallow size, and their retained size as columns in the Constructor view.

Shallow size indicates the amount of space the object uses for its own properties. For instance, an object with no properties will have a very small shallow size, where an object with a number of properties will have a larger shallow size.

Retained size indicates the amount of space the object is referencing, and keeping from being garbage collected.

From the Constructors view of the Snapshot, you can sort by Constructor name, number of objects, shallow size and retained size. You can also filter the list by Constructor name.

You can use the number of objects value to look for the objects that are leaking, and you can use the retained size value to look for objects referencing those leaking objects.

In cases where you are leaking objects - and thus consuming more memory than you should be - sorting by number of objects and by retained size can provide a good indication of where to look for the cause of the leak. A large number of objects of a particular type may be an indication that these objects are still being referenced by other objects, when they could be garbage collected. A large retained size may be an indication that these objects are referencing more objects than they should be, keeping them from being garbage collected.

Best practices

Using constructors will show class names in heap snapshots Because the heap Snapshot groups objects by constructor name, if at all possible you should use named constructors for objects you'd like to track, as opposed to using literal objects. When using literal objects for trackable objects, those objects will be aggregated under the "Object" constructor name, along with all the other literal objects your program uses.

For instance, the following code snippet creates objects which will be categorized under the "Object" constructor:

trackableObject = {x: "some value", y: "another value"}

To be able to track these objects in a Snapshot, use a specifically named class:

trackableObject = new TrackableObject()
trackableObject.x = "some value"
trackableObject.y = "another value"
// ...
function TrackableObject() {}

You'll then see these objects grouped by themselves in the heap Snapshot.

If you'd like a bit more of an "inline" feel, you can enhance your constructor to take initialization values:

trackableObject = new TrackableObject({x: "some value", y: "another value"})
// ...
function TrackableObject(initialValue) {
  this.x = initialValue.x
  this.y = initialValue.y

With EcmaScript 6 parameter destructuring, you can make it a little shorter:

trackableObject = new TrackableObject({x: "some value", y: "another value"})
// ...
function TrackableObject({x, y}) {
  this.x = x
  this.y = y

You can also make the constructor open-ended regarding the properties:

trackableObject = new TrackableObject({x: "some value", y: "another value"})
// ...
function TrackableObject(initialValue) {
  for (var key in initialValue) {
    this[key] = initialValue[key]
  }
}

N|Solid Security - Thoughts and Tips

Network Access Considerations

In order to control access to your application's N|Solid tooling, it is important that you use network tooling to limit exposure outside of your network.

Following general network security best practices, you should consider all N|Solid components to be for internal use only, and not to be exposed to public networks. Use industry standard firewall tools to limit access to only trusted networks or hosts.

N|Solid Console Access

The N|Solid Console needs to access the N|Solid Hub in order to operate, but does not need to directly access any of the N|Solid Agents. Specifically it must be able to access the N|Solid Hub Proxy, but does not need to access the Registry.

Tip

We suggest using a VPN to use the N|Solid Console from remote networks.

The N|Solid CLI tool has the same considerations as the N|Solid Console.

N|Solid Hub Access

The central position of the N|Solid Hub means it can be used to provide limited access to the running N|Solid Agents in your system. With appropriate access control configurations, only the N|Solid Proxy should need to communicate with your N|Solid Agents. This means that the N|Solid Proxy should be accessible by the network running the N|Solid Console, or anywhere you want to run the N|Solid CLI tool.

The N|Solid Hub Proxy must be accessible by the N|Solid Console and N|Solid CLI tool.
The N|Solid Hub Registry must be accessible by the N|Solid Agents in your network.
The N|Solid Hub Proxy must be able to access your running N|Solid Agents.

Tip

We suggest using the N|Solid Hub as a bridge between your corporate VPN and your production VPN N|Solid components.

N|Solid Runtime Considerations

The N|Solid Agent needs to be directly accessibly by the N|Solid Hub Proxy and needs to be able to connect to the the N|Solid Hub Registry. It can optionally be directly used via the N|Solid CLI tool, though all direct operations can be also done via the N|Solid Hub.

Tip

We suggest limiting exposure of your production N|Solid Agents to only the N|Solid Hub.

N|Solid Policy Usage

N|Solid's policies feature should not be used as a blind replacement for good security practices with regard to your code and the code of your application's dependencies. Disabling core modules and bindings does not provide a complete guarantee that your application is unable to use the features provided by those parts of Node.js. For example, child processes interacting with Unix utilities, if not disabled, can be used to replace the functionality of many of the core modules. Some core modules may be completely re-implemented in user-land modules, such as the dns and cares_wrap functionality. Compiled add-ons have complete freedom to insert new functionality that can replace anything that has been disabled.

N|Solid's policies features augment and support existing enterprise-wide application security hardening programs. Please contact NodeSource if you wish to discuss your Node.js application security needs and prepare a customized security hardening program that involves the use of internal and third-party code review, application monitoring and the use of operating system security tools.