A script reads a 40 GB trade file, builds a list of parsed rows, sorts by timestamp and writes a summary. It dies with a memory error. Say which parts a generator fixes, which part it cannot fix, and what you do about that part.

A script reads a 40 GB trade file, builds a list of parsed rows, sorts by timestamp and writes a summary. It dies with a memory error. Say which parts a generator fixes, which part it cannot fix, and what you do about that part.

Approach: Trace where the whole file has to exist at once, then ask which operations are streaming and which need every element before they can emit the first result.

A generator fixes the read and the parse, which become one row at a time in constant memory, and it cannot fix the sort, because a sort must see every element before emitting the first one, so that part needs an external merge sort or an exploitation of the fact that the file is already close to timestamp order. Replacing the list comprehension with a generator expression and iterating the file object directly removes the 40 GB list and makes any per-row aggregation streaming. The built-in sorted materialises the whole iterable, so it fails in exactly the same way. Write sorted chunks of a few million rows to temporary files and combine them with a streaming k-way merge over the chunk iterators, or bucket by a coarse key first and sort each bucket. If the summary is only counts and sums per symbol, drop the sort and aggregate into a dictionary in one pass.

Follow-up: How much memory does the k-way merge itself hold, and how would you pick the chunk size given 8 GB of RAM?

Key concepts: generator, streaming, external merge sort, constant memory.