[Riԁdle]

When I add utf-8 encoding it uses about 3000mb of memory (and freezes my PC) the exmaple below works fine,

Code

pool = ThreadPoolExecutor(max_workers=10)
with open("data.txt") as f:
        for line in f:
            pool.submit(thread, line)
pool.shutdown(wait=True)

but when I add utf-8 encoding it basically crashes my PC

Code

pool = ThreadPoolExecutor(max_workers=10)
with open("data.txt", encoding='utf8') as f:
        for line in f:
            pool.submit(thread, line)
pool.shutdown(wait=True)

[HFDone]

python 3 is unicode by default maybe its the reason

[Riԁdle]

python 3 is unicode by default maybe its the reason

well I get this error when I dont use encoding='utf8'

Code

UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 854: character maps to <undefined>

[HFDone]

yea as i said its unicode problem try str(line, 'utf-8')

[Riԁdle]

yea as i said its unicode problem try str(line, 'utf-8')

like so?

Code

with open("data.txt") as f:
        for line in f:
            line = str(line, 'utf-8')
            pool.submit(thread, line)
pool.shutdown(wait=True)

[Pillus]

Is that file large? Because it will load the file into memory, so you will have to read the file and iterate over the chunks you want to keep the memory usage low.

Look into codecs: http://stackoverflow.com/questions/16130...-different

[Riԁdle]

Is that file large? Because it will load the file into memory, so you will have to read the file and iterate over the chunks you want to keep the memory usage low.

Look into codecs: http://stackoverflow.com/questions/16130...-different

yeah it's a very large file, I'll look at this

[Pillus]

yeah it's a very large file, I'll look at this

Yeah then that is the problem. A normal file would load the entire thing into memory, so the larger the file, the more memory usage :)

[Riԁdle]

Yeah then that is the problem. A normal file would load the entire thing into memory, so the larger the file, the more memory usage :)

thank you <3

[Lambasoft]

You're loading the file 10 times in memory, in your case it seems the file size is around 300mb, loading it 10 times ( one time in every worker ThreadPoolExecutor(max_workers=10) ) requires 3000mb of RAM usage.

I'd suggest either lowering your max_workers, or splitting the file to 10 parts.

Good luck :)