+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118)
+---- Thread: [C] C++ with large files. (/showthread.php?tid=6159982)
C++ with large files. - severos - Jun 21, 2021
I have to deal with files of very large size, 100GB+, and the main problem is in seeking around those files since the normal fseek takes 32bit offset which doesn't cover the whole file.
That's when I found the functions: fopen64, fseeko64, ftello64 that supposedly allow you to use 64bit offset...
But for the last 2 days I can't get them to work, they do work as in, file opens, seek, read/write as long as the offset is within 32bit range, but soon as I try to seek beyond that fseeko64 fails and the offset is always at 0.
I did:
#define _FILE_OFFSET_BITS 64
#define __USE_FILE_OFFSET64 1
#define _GNU_SOURCE 1
But it still doesn't work, anything beyond the 32bits offset places the offset at 0.
Any idea what could be the problem?
RE: C++ with large files. - HFDone - Jun 22, 2021
you can
reading chunks
using file map api
using istream seekg
RE: C++ with large files. - severos - Jun 22, 2021
(Jun 22, 2021 02:43 AM)HFDone Wrote: you can
reading chunks
using file map api
using istream seekg
Reading chunks: worst solution ever, it'll require reading data that I won't use, performance will be super slow. Never suggest that.
File map: might be useful in other scenarios, not in my case, I'm extending the file as I read/write to it, mmf require constant file size.
istream seekg: according to my original post: "main problem is in seeking around those files since the normal fseek takes 32bit offset which doesn't cover the whole file", so seekg is actually not possible, maximum you can reach with that is 4GB.
Google: that's how I got to what I mentioned in my original post "fopen64, fseeko64, ftello64", however my google skills has failed in debugging why they keep failing. Which is why I posted here.
RE: C++ with large files. - Sajuuk - Jun 22, 2021
https://www.cplusplus.com/reference/string/string/getline/
The margin of efficiency difference in parsing massive chunks of data is negligible. Some methods may hold out better in specific cases, but in terms of massive data parsing of some generic data (generic as in specified as "100+ gB of Data") speed will be matched based on system specs unless some specific data type is specified.
RE: C++ with large files. - severos - Jun 22, 2021
(Jun 22, 2021 11:05 AM)Sajuuk Wrote: https://www.cplusplus.com/reference/string/string/getline/
The margin of efficiency difference in parsing massive chunks of data is negligible. Some methods may hold out better in specific cases, but in terms of massive data parsing of some generic data (generic as in specified as "100+ gB of Data") speed will be matched based on system specs unless some specific data type is specified.
Working with binary files, and I don't want to read the whole file. Again, my problem is in seeking after the 4GB barrier.
RE: C++ with large files. - HFDone - Jun 23, 2021
did you check SetFilePointerEx https://docs.microsoft.com/he-il/windows/win32/api/fileapi/nf-fileapi-setfilepointerex
you can use it with CreateFile
RE: C++ with large files. - severos - Jun 23, 2021
(Jun 23, 2021 12:52 PM)HFDone Wrote: did you check SetFilePointerEx https://docs.microsoft.com/he-il/windows/win32/api/fileapi/nf-fileapi-setfilepointerex
you can use it with CreateFile
My bad, forgot to mention that I'm coding on linux not windows. But thanks for the contribution.
RE: C++ with large files. - int main() - Jun 24, 2021
ulimit -n 9999999 i think would help
RE: C++ with large files. - tulamba - Jun 26, 2021
@serveros
Have you looked at multithreading or multiprocessing options to split the task into smaller subtasks?
Can you share the code?
RE: C++ with large files. - severos - Jun 26, 2021
(Jun 24, 2021 05:17 PM)int main() Wrote: ulimit -n 9999999 i think would helpI can't even think of how that would even be remotely related to my question?
(Jun 26, 2021 01:37 PM)tulamba Wrote: @serveros
Have you looked at multithreading or multiprocessing options to split the task into smaller subtasks?
Unrelated solution, I still need to seek through the file regardless of how I process it.
RE: C++ with large files. - tulamba - Jun 26, 2021
https://stackoverflow.com/questions/34751873/how-to-read-huge-file-in-c