Read a large file line-by-line in Python
The fileinput
module's input
method reads in files one line at a time:
import fileinput
for line in fileinput.input(["somefile.txt"]):
print(line)
BONUS: How to read a large CSV in chunks:
import pandas as pd
# `chunksize` specifies the number of rows in each chunk
chunksize = 10000
with pd.read_csv("big.csv", chunksize=chunksize) as reader:
for chunk in reader:
print(chunk)