Incremental data processing
Process large files or network responses incrementally without buffering the complete payload.
Node.js · Module
node:stream
The node:stream module provides abstractions and utilities for processing data incrementally. It includes Readable, Writable, Duplex, and Transform stream types plus callback- and Promise-based composition APIs.
How it works
The node:stream module supplies abstractions for incremental data flow. Readable streams provide data, writable streams consume it, duplex streams support both directions, and transforms modify data while it passes through; streams also emit events and can be connected with pipeline utilities.
Syntax
import { Readable, Writable, Duplex, Transform, pipeline } from 'node:stream';
const stream = require('node:stream');Key points
- Streams represent an abstraction for processing data incrementally rather than requiring the whole data set at once.
- Stream consumers can use EventEmitter behavior to react to lifecycle and data-related events.
- Writable streams model destinations that receive chunks from an application or another stream.
- Readable streams model sources that expose chunks for application code or downstream consumers.
- Duplex streams provide independent readable and writable sides in one object.
- Transform streams are duplex streams intended to process input and produce changed output.
Common mistakes
- Use this module mainly when creating or customizing stream instances; ordinary stream consumption often needs only the concrete stream API.
- Do not assume every stream supports both operations; choose APIs according to whether the instance is readable, writable, duplex, or transform.
Example: Transform data with a stream pipeline
import { Readable, Transform, pipeline } from "node:stream";
const upper = new Transform({
transform(chunk, _encoding, callback) {
callback(null, chunk.toString().toUpperCase());
},
});
await pipeline(
Readable.from(["hello ", "streams"]),
upper,
process.stdout,
);Creates a readable stream from iterable data, transforms each chunk to uppercase, and pipelines the result to standard output.
When to use it
Composable pipelines
Connect a data source, transformation, and destination while propagating completion and errors.
Custom stream implementations
Define custom sources, destinations, bidirectional channels, or data converters.
Interview questions
QWhat are the fundamental Node.js stream types?
The four fundamental kinds are Writable, Readable, Duplex, and Transform. Writable accepts data, Readable produces data, Duplex supports both directions, and Transform is a Duplex that changes data as it passes through.
Follow-up: How would you implement a read/write stream? Implement the _read() method for the readable side and _write(), _writev(), and _final() as needed for the writable side.
QHow do readable and writable streams differ in use?
Streams are EventEmitters, so readable streams can notify application code when data is available. Writable streams expose operations such as write() and end() for sending data to their destination.
Follow-up: What API can replace callback-based stream utilities? Use the stream/promises API, available through node:stream/promises or node:stream.promises, which returns Promises.
QHow is the node:stream module accessed and what is it for?
Import the module with require('node:stream') in CommonJS or with named imports from node:stream in ES modules. The module is primarily useful for creating new stream types and accessing stream classes and utilities.
Follow-up: Which methods are needed for a read-only custom stream? Implement _read() on a Readable; writable-only implementations use _write(), _writev(), and _final() as applicable.
Related
ReadableWritableDuplexTransformpipelinefinishedstream/promises