How to work with large files without running out of memory

You wrote a script that reads a CSV, transforms each row, and writes the result. It worked perfectly with the 10MB sample file on your laptop. Then the client sent the real 2GB file, the process climbed in RAM until it froze the machine and died with a JavaScript heap out of memory (or a silent MemoryError, in Python). The bug isn't in your transformation logic. It's in a single line: readFile (or read()) that loads the whole file into memory before you even start processing.

This article is about avoiding that structurally: processing files in a stream, with constant memory, regardless of file size. It covers Node.js and Python, the concept of backpressure, and when loading everything at once is still acceptable.

Why readFile kills the process

The function that looks simplest is the most dangerous. fs.readFileSync(path) in Node and open(path).read() in Python do the same thing: they allocate a buffer the exact size of the file and dump all the content into it before returning. A 2GB file requires at least 2GB of contiguous RAM just to exist as a variable — and that's before you do any parsing, which usually doubles or triples that number (a 2GB string turning into an array of objects can easily exceed 6GB).

Node has a default heap limit in the 2GB to 4GB range depending on the version. Once it overflows, v8 aborts. Python has no such fixed ceiling, but the operating system does: when RAM runs out, the OOM killer takes down the process, or the machine starts swapping to disk and everything gets a thousand times slower. In either case, the result is the same — a job that worked in the test fails in production.

The point that matters: the file size isn't yours to control. If the source is a user upload, a database export, an application log, or a dump from another system, it can grow without warning. What fits in memory today won't fit next month.

The solution: read in chunks, process, discard

The core idea of streaming is to never have the whole file in memory at the same time. You read a piece (a chunk, or a line), process it, discard it, read the next one. The memory used is that of one chunk, not the whole file. Processing 10MB or 10GB consumes practically the same RAM — only the time changes.

In Node, the tool is the stream. Instead of readFile, you use createReadStream and consume the data as it arrives:

import { createReadStream, createWriteStream } from 'node:fs';
import { createInterface } from 'node:readline';

const input = createReadStream('large.csv');
const output = createWriteStream('out.csv');
const rl = createInterface({ input, crlfDelay: Infinity });

for await (const line of rl) {
  const transformed = process(line); // your logic, one line at a time
  output.write(transformed + '\n');
}
output.end();

The for await walks line by line. At no point does the complete array of lines exist — readline holds at most one chunk of the stream's internal buffer.

In Python the equivalent is even more direct, because iterating over a file object already reads line by line by default:

with open("large.csv") as src, open("out.csv", "w") as dst:
    for line in src:          # never loads the whole file
        dst.write(process(line))

The classic Python trap is src.readlines() or list(src) — both materialize all lines into a list at once, defeating the advantage. Iterate directly over the file object, or write a generator that does yield of one line (or record) at a time. The generator is Python's version of the stream: it produces values on demand, without accumulating.

Backpressure: don't read faster than you write

Streaming solves the read-side memory consumption, but introduces a second, subtle problem. Imagine you read from a fast file (SSD) and write to something slow (network, or a database). If you read at 100MB/s and can only write at 10MB/s, the difference accumulates in a buffer in memory — and you've reinvented the OOM problem by another path.

The answer is backpressure: the consumer signals the producer to pause when it can't keep up. In Node, pipe and pipeline do this automatically — that's why you should almost always use pipeline instead of chaining write by hand:

import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';

await pipeline(
  createReadStream('large.csv'),
  createGzip(),                  // transform along the way
  createWriteStream('large.csv.gz')
);

pipeline propagates backpressure end to end, closes the streams correctly at the end, and propagates errors — three things manual write gets wrong easily. When the destination gets slow, the read pauses on its own. Memory stays stable even with source and destination at very different speeds.

Common cases in practice

Giant CSV. Process line by line as in the examples above. Never run JSON.parse on a whole CSV nor load everything into an array to then do .map() — .map() requires the complete list in memory. If you only need to understand the format before writing the parser (which columns exist, how the separator behaves, whether there are quotes), inspect a sample: take the first few lines and, to visualize the structure of a large CSV by converting a sample to JSON, you see the shape of the data without opening the 2GB file. Then you write the stream with confidence.

Copying or transforming a file. Use pipeline with a Transform in the middle (compression, encoding swap, rewriting each line). Constant memory, and backpressure handles the speed difference between source and destination.

Huge JSON. This is the trickiest one, because JSON.parse is all-or-nothing: it needs the whole document to return. For a JSON array of hundreds of MB, use a streaming parser (in Node, libraries like stream-json; in Python, ijson) that emits each element as it reads it. If you control the source, prefer JSON Lines (one JSON object per line) — then the problem becomes "read line by line," which you already know how to solve. Encoding problems also show up a lot here; if the file came from another system, it's worth understanding the topic first, as I describe in importing CSV dealing with encoding.

When loading everything is still acceptable

Streaming isn't free: the code gets more verbose and harder to reason about. If the file is small, known, and generated by you — a 4KB config file, a 200KB translation JSON that's part of the build — readFile is the right choice. Optimizing that is a waste of time and readability.

The criterion isn't the current size of the file. It's whether the source can grow. Config versioned in your repository? Load everything. Anything coming from outside — upload, database, API, log — treat the size as untrusted and use a stream by default. The cost of writing as a stream from the start is trivial compared to the cost of discovering, with an angry client, that your import script dies on the real file.

Frequently asked questions

Is streaming slower than loading everything at once?

For small files, marginally yes, because of the overhead of managing the stream. For large files, it's the opposite: loading everything forces the system to swap to disk when RAM runs out, and then performance collapses by orders of magnitude. Streaming keeps the working set small and cache-friendly. In the range that matters — large files — the stream is faster, besides simply not crashing.

What's the ideal chunk size?

For line-by-line reading you don't choose — the abstraction handles it. When you control the buffer (binary streams), Node's default of 64KB is a good starting point. Chunks that are too small increase per-iteration overhead; too large wastes memory with no real gain. Only tune it with a measured number in front of you, not by intuition.

How do I know backpressure is working?

Monitor the process's memory usage during a run with a large file (process.memoryUsage() in Node, or watch the RSS in the task manager). If memory rises and stays stable, backpressure is doing its job. If it rises without stopping until the end, somewhere you're accumulating — probably an array.push inside the loop, or a write not respecting the return value that signals a full buffer. Using pipeline eliminates most of those mistakes upfront.

Do generators in Python serve the same purpose as streams in Node?

For reading, yes: a generator that does yield of one line at a time has the same constant-memory effect. The difference is that Python doesn't have the concept of backpressure built in like Node's pipeline — if you're writing to a slow destination, you need to control the pace by hand or use asyncio with libraries that support it. For the common case of read-transform-write to disk, iterating over the file object solves it.

What to take from here

Treat file size as untrusted input. What works with 10MB on your laptop dies with 10GB in production, and the discovery always happens at the worst possible moment. The practical rule is simple: if the source of the file can grow, use a stream by default — read in chunks, process, discard, and let backpressure handle the pace. Loading everything into memory is the exception reserved for small files you control yourself, not the default path. The cost of writing as a stream from the start is small; the cost of rewriting under pressure, after the OOM in production, is not.