Huge file _fseeki64 _ftelli64 in Visual C++
Writing cross-platform huge file I/O code is tricky because the 64-bit offset versions of
ftell are not standard across compilers and platforms. This article documents the CMarkup experience with this, related discoveries made along the way, and it may be useful to anyone dealing with this issue.
32-bit and 64-bit file offsets
The limit of signed 32-bit integer offsets in
fseek is 2^31-1 = 2147483647 which is around 2GB. For example, a common cross-platform way of getting file size is
SEEK_END and then
ftell cannot return a number higher than the limit of its return type.
The 64-bit versions of these functions deal in offsets of billions of gigabytes (8 exabytes) which should be enough for single file sizes in the next decade or two ;).
There are no 64-bit offset versions of
fwrite functions. This is because they do not deal explicitly with file offsets, though they do operate relative to the current file pointer. The underlying file pointer can generally handle the huge file, so in theory you can read multiple 2GB blocks using
fread even if you don't have a 64-bit
ftell function available to query the current offset.
So even as huge file support was added to the file I/O functions in C/C++ libraries by the early 90s, there was no graceful way to promote the integer types of offset variables used in existing programs. The
fwrite functions were upgraded under the covers, while new versions of
fseek were added because they dealt explicitly with the offset.
#ifdef WIN64 used in conjunction with
_fseeki64. I am not sure why they were doing that, but don't be confused. A 64-bit operating system is not required for 64-bit file offsets; you can usually use 64-bit offsets in a 32-bit operating system.
Huge files and CMarkup
The 64-bit offset versions of
ftell are used to support huge files with the CMarkup release 11.0 file mode methods. The file mode methods give read and write access to files without loading the entire document into memory. File mode does not require 64-bit offsets, but 64-bit offsets are needed if you are dealing with files over 2GB.
Most UNIX flavor compilers like gcc are standardized on the
fseeko which use the
off_t integer type that depends on compiler setup. The functions resolve to the old
ftell or huge
ftello64 in concert with the
_LARGEFILE_SOURCE macro defines.
Unfortunately VC++ doesn't use this system.
CMarkup in Visual Studio
Visual C++ provides its huge versions,
_ftelli64 inconsistently. There is no clean way to always know whether the 64-bit versions are available and to automatically compile accordingly.
In CMarkup release 11.0 the 64-bit functions are used by default in Visual Studio 2005. This is not correct for target platforms where the 64-bit functions are deliberately excluded such as for Windows CE.
Since the 64-bit functions are covertly distributed with Visual Studio 6.0 (see below), in CMarkup release 11.1 I declared prototypes for the 64-bit functions by default in Visual Studio 6.0. The caused more problems because the functions are not available in some build configurations of VC++ 6.0.
In CMarkup release 11.2 with any version of Visual Studio you will need to define
MARKUP_HUGEFILE in the project definitions if you want huge file access. This will eliminate these linker errors for first time users, while putting the burden on those who need huge file access to specify the define.
One alternative would be to use the Win32 File APIs and the
SetFilePointerEx available on Win2K+, but this would greatly increase the code differences between the different platforms.
CMarkup started using fseeki64 and ftelli64 in release 11, but they are optional. In the next release I will not use them unless you set a define. To turn them off do something like this:
Old line 223:
#elif _MSC_VER >= 1000 // VC++
New line 223:
#elif _MSC_VER > 4000 // never
In CMarkup release 11.1 I implemented a workaround to offer huge file (>2GB) access by default in Visual C++ 6.0 and Visual Studio .NET versions before VC++ 2005. Huge file access is only useful for the developer version file mode (see Open) methods.
However, I did not realize that it depended on a project setting for these functions to be available in the VC++ libraries you link with. If in your project settings you have the default Microsoft Foundation Classes setting "Use MFC is a Shared DLL" you will get the following problem when linking CMarkup release 11.1:
Linking... Markup.obj : error LNK2001: unresolved external symbol __ftelli64 Markup.obj : error LNK2001: unresolved external symbol __fseeki64
If you are able to change your Microsoft Foundation Classes project setting to "Use MFC in a Static Library" that is one way to alleviate the linker error and keep huge file access. Otherwise, you can change the compiler version directive as shown above and you cannot use file mode with files over 2GB.
Microsoft Visual Studio\VC98\CRT\SRC\FSEEKI64.C Microsoft Visual Studio\VC98\CRT\SRC\FTELLI64.C
Microsoft Visual Studio\VC98\Lib\LIBCMT.LIB
dumpbin /exports libcmt.lib
does not show it but the following does:
dumpbin /all libcmt.lib
Thank you for the excellent feedback and research. Release 11.2 is fixed according to your suggestion.