Find a memory leak

Learn how memory leaks happen in Node.js, and how to diagnose and fix them with N|Solid metrics, CPU profiles, heap snapshots, and heap profiles.

What is a memory leak?

note

A memory leak is when software keeps memory it will never need again. Memory use typically grows over time until the service degrades or fails.

JavaScript memory is managed by the garbage collector (GC), but the GC needs help to know what it can free. JavaScript developers don't have to manage most memory lifetimes, but you can't assume the garbage collector knows what you intend, or what your code will do in the future. For example, if you keep appending to a list, and the list itself is still reachable, the GC can't know which entries are no longer needed. To avoid leaks, choose GC-friendly data structures and think about how the memory graph determines what can still be reached.

The JavaScript heap is a graph of containers (nodes) and links between them (edges). Each container, such as an Object or a string, is a node that can reference other nodes or be referenced by them. A reference can imply either an ownership relationship, such as an object property or a list element, or a context relationship, such as a closure that references an object from an outer scope. This is a simplified view, but it uses the same concepts as .heapsnapshot files, which are a serialized representation of the JavaScript heap.

In this simplified model, the garbage collector can walk the graph from its roots and free entire branches that are no longer reachable, if those branches are well encapsulated. If a long-lived branch creates context links to objects that should be short-lived, those objects stay alive, and they can keep other branches alive that could otherwise be freed.

At best, a memory leak makes your application use resources it doesn't need, and normal over-provisioning may keep a minor leak from ever causing problems. Larger or unattended leaks cause cascading issues, and many teams schedule restarts to avoid crashes. Garbage collection work also grows with heap size and graph complexity, so the longer a leak runs, the more time the GC takes away from your application.

The following examples show some simplified leaks, followed by an analysis workflow.

Naive links

A quick way to create a leak is to link data together for convenience in places where it shouldn't be linked.

const cache = new Map()
app.get('/data/:id', async (req, res) => {
  const id = req.params.id

  if (cache.has(id)) {
    res.send(cache.get(id).data)
  } else {
    const result = await queryDatabase(id)

    // This permanently retains the original web request for each cache entry
    // because it is now linked to the cache entry
    cache.set(id, { ...result, request: req })

    res.send(result.data)
  }
})

In this example, a poorly designed results cache keeps incoming web requests alive indefinitely, because each cache entry holds a reference to the request that created it. Many memory leaks come from this pattern: keeping something on purpose without considering what else it keeps alive.

Unreachable data and code

In JavaScript, you can enclose functionality in a scope. This is one way to create private methods or variables. It also makes leaks hard to debug, because members of the same scope can reference each other and keep each other alive.

const leakyBucket = {}
function store (key) {
  const someMessage = 'hi'
  const entry = leakyBucket[key]
  // `unused` is created in the scope of `store`
  const unused = function () {
    // and it references another variable in the scope of `store`: `entry`
    if (entry) console.log('never called')
  }
  leakyBucket[key] = {
    longStr: new Array(1000000).join('*'),
    someMethod: function () {
      console.log(someMessage)
    }
  }
}
// Each call to `store(key)` with the same key creates a new entry whose
// `someMethod` closure shares a scope with `unused`, which references the
// previous `entry`. Calling `store()` repeatedly with the same key chains
// the entries together, so none of them can be collected.

Because JavaScript closures are so flexible, you can create islands of scope that the garbage collector can't untangle, because of how they relate to each other or how they might be accessed in the future. This is what many people picture when they hear "memory leak", but it's only one of many patterns.

Missing cleanup

Often a leak is as simple as missing cleanup on an error path.

const cache = new Map()

function start (id, data) {
  console.log(`Work started for ${id}`)
  cache.set(id, data)
}
function finish (id) {
  console.log(`Finished ${id}`)
  cache.delete(id)
}
function error (id) {
  // If this doesn't delete from the cache, it will accumulate (leak) cache
  // entries related to errors forever
  console.error(`Error while doing work on ${id}`)
}

work.forEach(item => {
  start(item.id, item.data)
  try {
    // ... some work
    finish(item.id)
  } catch (e) {
    error(item.id)
  }
})

These are usually errors of omission. The happy path is handled well, but rare paths or error paths don't clean up their resources, so objects with similar characteristics accumulate. In the example above, those are the entries for work that failed.

Memory spaces in Node.js

Node.js has two primary memory spaces:

• The V8 JavaScript heap.
• The unmanaged C++ memory of Node.js, where the contents of containers such as Buffer and TypedArray are stored. This is sometimes called "RSS", but it's more accurately "RSS minus heap".

Leaks in the heap are usually application or logic errors. Tools such as N|Solid use V8 APIs to extract and analyze the heap. Leaks in unmanaged memory are often caused by native libraries, misuse of Node.js internals, or application code, and far fewer tools exist to analyze them. Application metrics over time show which memory space is growing: one of these two, or one of the rarer spaces. This guide covers the JavaScript heap and "RSS minus heap".

The memory spaces fail differently under memory pressure. The heap size limit is set when the application starts. If you allocate beyond the maximum heap size and garbage collection can't free enough space, V8 terminates the process. RSS can grow without limit, which can push the system into swap and degrade performance, or cause the operating system or container orchestrator to kill the process when it tries to allocate more than is available.

The JavaScript heap is further divided into separate spaces for different purposes, such as the application's own instructions (executable space) and the data it works with. This guide focuses on leaks in the data space, and points out where to check that assumption.

Find memory leaks

Memory leaks can be very hard to locate in large applications. Some tools can point to the function that leaks, but often you narrow things down to a few suspects and then read the source code.

A simplified workflow:

1. Is it a leak at all?
2. RSS or heap?
3. Where in the code? (CPU profiles, metrics, heap snapshots and profiles, load testing)
4. How do you fix it?

Step 1: Is it a leak at all?

It can be hard to tell a memory leak from expected behavior. Sometimes it's clear, such as servers that routinely crash with out-of-memory (OOM) messages in their logs. Other times, what looks like a leak is a cache that is still filling its planned headroom.

Crashes when the process was near its memory limit, or logs that mention "OOM" or "out of memory", are strong evidence. But teams often restart servers proactively when they reach a memory limit, which prevents the OOM while the leak continues.

First, isolate the leak: find processes that are actively leaking, or a way to reproduce it. You may be able to reproduce it locally or in a load test, but often you can only see it in production. N|Solid can collect the information you need from production with minimal impact.

Start by looking at metrics and thinking about how the application uses memory. Is there simply no memory pressure, so a major GC hasn't run yet? Does the growth follow daily business activity? Keep these questions in mind when you look at the memory metrics below.

Step 2: RSS or JavaScript heap?

Check this early, because the tools for each memory space are very different. Most of this guide, and most of the tooling, is for leaks in the JavaScript heap. Leaks in RSS follow the same workflow, but the heap-specific tools are less helpful.

Step 3: Where in the code is it leaking?

This is where you use the rest of the toolbox.

Helpful memory metrics

First, find out whether the leak is related to usage patterns or grows steadily over time. If a particular activity triggers the leak, load testing, especially targeted load testing, can help narrow down the location and gives you a way to evaluate the fix. If the growth only correlates with time, it probably comes from a timer or interval in the application or its libraries, or from an external caller with its own timer, such as a health check. Start with application metrics over time, then match the growth to application behavior.

tip

Metrics in detail lists all N|Solid Runtime metrics.

Suggested primary metrics:

• rss: a fleet of processes normally settles into a steady range of RSS values. Unbounded growth over a long period suggests a leak.
• heapUsed: a fleet of processes normally settles into a similar steady range. If this value only increases over long periods, even while garbage collection runs, you probably have a leak.

Both metrics respond to memory pressure, or the lack of it. They can grow for a long time before anything triggers memory to be released. For RSS, that is usually other applications allocating large blocks of memory. For heapUsed, it takes a major GC to free memory.

warning

Taking a heap snapshot forces a major GC, so it can show whether GC pressure fixes what looks like a leak. But it temporarily increases the process's RSS significantly, which can be a problem if you are near hardware limits. The time the process is paused for garbage collection also grows with the size and complexity of the heap.

Potential alert metrics:

• totalAvailableSize below a safety threshold.
• rss nearing hardware limits.

Metrics that help diagnose, locate, or characterize leaks:

• totalHeapSizeExecutable: if a heapUsed leak shows up in this space, the leak is in code generation (eval or similar), misuse of the module loader, JIT issues, or other growth in code instructions.
• externalMem: helps narrow down an rss leak. Growth here points to ArrayBuffer or Buffer data.
• numberOfNativeContexts: look for (mis)use of the vm module.
• I/O primitives can narrow a leak down to code that handles a specific type of I/O. See Metrics in detail for the full list. For example:
  • activeHandles: long-term I/O primitives, so this should rarely change. Unbounded growth is a clear anti-pattern.
  • activeRequests: short-term I/O primitives, so this can change often. Unbounded growth is a clear anti-pattern.
  • promiseCreatedCount: shows whether promises are a source of the leak. Promise tracking must be enabled, with NSOLID_PROMISE_TRACKING=1 or the Track Promises toggle in the process's Dynamic Configuration dialog.
• Garbage collection statistics:
  • gcCount: a steeper slope without a matching drop in used heap may mean the garbage collector is struggling with a more complex graph because of a leak.
  • gcMajorCount: compare processes with a suspected leak to those without, and look for much more GC activity. If GC behavior is similar, the cause may be a lack of memory pressure rather than a leak.

CPU profiling

The CPU profiler is a V8 API that N|Solid uses to sample the call stack at a fixed interval (about once per millisecond by default) and record which function is running. You can analyze the resulting .cpuprofile file in the N|Solid Console, or download it and analyze it in Chrome DevTools or other tools such as Visual Studio Code. CPU profiling is mainly a performance tool, but it also records which code is actually active, which helps you narrow your focus. It doesn't record memory statistics, only code execution time. Long profiles can show which parts of a large codebase are active while the leak is triggered.

The main cost of a CPU profile is a slight slowdown while the profiler samples execution. The overhead is small enough to be considered safe in production, but avoid profiling constantly or profiling every instance at once.

You can analyze CPU profiles in the N|Solid Console, and the N|Solid Copilot can summarize them. See Optimize performance with CPU profiles.

Heap snapshots

A heap snapshot is a full graph of your application's V8 JavaScript heap, with unreachable nodes already removed. You can download the resulting .heapsnapshot file from N|Solid and open it in Chrome DevTools. Snapshots can be large: usually at least the size of the heap, because they contain most of its contents plus the graph structure. See Taking and analyzing heap snapshots for how to capture them.

Taking a heap snapshot has several effects that matter in production:

• It starts with a major "stop-the-world" garbage collection, which pauses execution, then records the freshly collected heap. The longer the collection and recording take, the longer your application is paused. On an active server, in-flight asynchronous work waits, which can mean a poor user experience.
• The snapshot's size in memory is hard to predict from the heap size, because it depends on both the contents and the complexity of the graph. It can at least double the process's RSS for a while.
• If the heap is near its maximum size, the snapshot can cause an out-of-memory crash. If the container or operating system limits how much memory the process can allocate, a large snapshot can cause the process to be killed.

A crash is the most disruptive outcome for users with in-flight requests, so take care.

Because it represents the entire heap, a snapshot can be very large on disk, especially when you are dealing with a leak. All data, the objects that contain it, and how they relate are captured in a JSON file that usually ends in .heapsnapshot.

Several tools can analyze heap snapshots, including Chrome DevTools, which is built into every Chrome browser. Open the Memory panel to load .heapsnapshot files and browse the graph.

warning

Test the impact of your tools in staging and QA before you use them in production. For example, taking a heap snapshot without enough free memory to serialize it might cause Docker to kill the process.

Allocation profiling

Heap allocation profiling (Track Allocations in the N|Solid Console) records allocations over time and adds code context to heap snapshot data. It helps you link constructors and allocation patterns to the code that created them. The resulting files use the same .heapsnapshot structure, with extra information. The performance overhead is at least as high as for heap snapshots.

You can analyze heap profiles in Chrome DevTools: open the Memory panel and load the file.

Sampling heap profiles

Heap sampling (Heap Sample in the N|Solid Console) has negligible overhead, because it doesn't require garbage collection or try to capture the whole memory graph. Instead, it samples allocations and deallocations together with their stack traces. The result is similar to a CPU profile, but shows memory growth instead of execution time, so you can see which functions allocated the most net memory during the sampling period.

You can analyze heap samples in Chrome DevTools: open the Memory panel and load the .heapprofile file.

Load testing

You may already have tests that exercise the code paths that trigger a leak. If you find one, the tools above become much more effective, and you can use them without the risk of degrading production. In general, if you can reproduce the problem outside production, analyze it there first, and run lighter confirmation tests in production.

Refactoring

Named constructors make all of the assets above much more useful. In a heap snapshot, generic objects are listed as object or Object, so everything created with {} or new Object() is grouped together. Named constructors put specific object types into their own categories, which helps when the generic buckets contain hundreds of millions of entries.

Step 4: Fix the leak

Fixing a leak usually means refactoring the code to be friendlier to the garbage collector: simpler code, simpler objects, cleaner encapsulation, and proper cleanup of resources that need manual cleanup.

Ideal solution: refactor

Leaks often come from patterns you can refactor:

• Misuse of global variables
• Missing timer or interval cleanup
• Missing event listener cleanup
• Closures with unnecessary references
• Improper cache cleanup
• Missing error handling
• Transient objects retained in long-lived objects or arrays
• Promises that never settle

Potential solution: WeakMap

Sometimes a leak that is otherwise hard to avoid can be solved with JavaScript's weak-reference types, such as WeakMap. A WeakMap lets you attach data to an object without keeping that object alive. For example, you can store results for a Request in a WeakMap, and they don't affect the lifetime of the Request:

const metadataMap = new WeakMap()

app.post('/start_work/:id', (req, res) => {
  const id = req.params.id
  metadataMap.set(req, { id, start: Date.now() })
  // ... other potentially asynchronous work
})

Because metadataMap is a WeakMap instead of a Map, its entry doesn't keep req alive. Once req is otherwise unreachable, the entry and its value can be collected. This makes it less likely that entries are left behind after requests complete. Attaching data or methods directly to Request objects has a similar effect, but it's much more likely to create circular references or to deoptimize code that handles Request objects.