This is the first post in a series on Node.js streams. In this post, we’ll cover the basics of streams, including what they are, why they’re important, and how to use them. In the next posts, we’ll dive into the different types of streams and how to use them in your applications.

Streams are one of the fundamental concepts that power Node.js applications. They are robust, powerful, and versatile tools that help to efficiently manage data in a Node.js application. Yet, they are often overlooked, misunderstood, or not optimally utilized by many developers.

Why are streams so essential? They allow you to handle data piece by piece as it becomes available, instead of needing to wait for the entire data load to complete. This approach is not only memory-efficient, but it also results in faster processing times, making your applications more scalable and responsive.

Streams, much like the rivers they’re metaphorically named after, flow with data. They can be readable (data can be read from them), writable (data can be written into them), or both (duplex and transform streams). These characteristics make them suitable for a broad range of tasks, from reading and writing files to handling HTTP requests and responses.

However, to truly leverage the potential of streams, it’s essential to understand how to use them properly. It’s about understanding when to use pipe and when to use events, how to handle errors, how to control flow and how to deal with back-pressure - the scenario where data is written faster than it can be consumed.

Let’s take a look at a simple example of reading and writing some data:

const fs = require('fs');

fs.readFile('input.txt', 'utf8', (err, data) => {
    if (err) {
        console.error('An error occurred while reading the file.', err);
        return;
    }
    
    fs.writeFile('output.txt', data, (err) => {
        if (err) {
            console.error('An error occurred while writing the file.', err);
        }
    });
});

In this example, the entire content of input.txt is read into memory before being written to output.txt. While this may work fine for small files, it poses several problems when dealing with larger files:

  • Memory Consumption: As the entire file needs to be loaded into memory, this can lead to excessive memory usage when dealing with large files. In the worst case, if the file is larger than the amount of available memory, the operation will fail.

  • Blocking I/O: While the file is being read or written, the event loop is blocked. This can degrade the performance of your application, especially when dealing with large files, as it stops other operations from being processed.

  • Lack of Responsiveness: Since the write operation doesn’t start until the entire file is read, the user may experience a lack of responsiveness if they’re waiting for the operation to complete.

The situation gets even worse when you take into account that real world use-cases are almost always processing data in transit:

const fs = require('fs');

fs.readFile('input.txt', 'utf8', (err, data) => {
    if (err) {
        console.error('An error occurred while reading the file.', err);
        return;
    }

    // Perform several processing steps on the data
    let processedData = processData(data);
    
    fs.writeFile('output.txt', processedData, (err) => {
        if (err) {
            console.error('An error occurred while writing the file.', err);
        }
    });
});

function processData(data) {
    // Expensive computational operations
    // 1. Replace every occurrence of 'apple' with 'orange' in the data
    // 2. Convert all the characters to uppercase
    // 3. Replace all spaces with underscores
    let result = data.replace(/apple/g, 'orange');
    result = result.toUpperCase();
    result = result.replace(/\s/g, '_');
    return result;
}

This additional processing further amplifies the issues we discussed earlier. Not only does each additional processing step increase the computational load and memory usage, but it also increases the time it takes to complete the operation.

Each processing step needs to be performed on the entire data before the next step can start, and the write operation cannot start until all processing steps have completed. This approach can quickly become unscalable as the size of the data or the number of processing steps increases.

Now, let’s take a look at using streams instead:

const fs = require('fs');
const { Transform } = require('stream');

const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');

const transformStream = new Transform({
    transform(chunk, encoding, callback) {
        let transformedChunk = chunk.toString();

        // Perform several processing steps on the chunk
        // 1. Replace every occurrence of 'apple' with 'orange' in the chunk
        // 2. Convert all the characters to uppercase
        // 3. Replace all spaces with underscores
        transformedChunk = transformedChunk.replace(/apple/g, 'orange');
        transformedChunk = transformedChunk.toUpperCase();
        transformedChunk = transformedChunk.replace(/\s/g, '_');

        callback(null, transformedChunk);
    }
});

readStream
    .pipe(transformStream)
    .pipe(writeStream)
    .on('error', (error) => {
        console.error('An error occurred:', error);
    });

In this example, we create a transform stream that processes the data as it’s being read and written. Rather than waiting for the entire file to be read into memory, we process the data in chunks as soon as they become available. Each chunk is first read from the source file, then processed by the transform stream, and finally written to the destination file.

This approach significantly reduces memory consumption, as we only need to hold a single chunk in memory at a time. It also improves performance and responsiveness, as we start processing and writing data as soon as the first chunk is available, instead of waiting for the entire file to be read.

Moreover, this approach allows us to efficiently pipeline multiple processing steps. Each step is performed on a chunk of data as soon as it’s available, and the processed chunk is immediately passed to the next step. This not only reduces memory consumption and improves performance, but it also makes the code easier to read and maintain by separating each processing step into its own function.

We’ll dive into each type of stream in the following subchapters, including some real world examples of utilizing native streams that Node gives to us (such as the read and write streams from the filesystem module), as well as writing our own (more to come on that later).