Writing cross-platform huge file I/O code is tricky because the 64-bit offset versions of fseek and ftell are not standard across compilers and platforms. This article documents the CMarkup experience with this, related discoveries made along the way, and it may be useful to anyone dealing with this issue.

32-bit and 64-bit file offsets

The limit of signed 32-bit integer offsets in ftell and fseek is 2^31-1 = 2147483647 which is around 2GB. For example, a common cross-platform way of getting file size is fseek with SEEK_END and then ftell, but ftell cannot return a number higher than the limit of its return type.

The 64-bit versions of these functions deal in offsets of billions of gigabytes (8 exabytes) which should be enough for single file sizes in the next decade or two ;).

There are no 64-bit offset versions of fread and fwrite functions. This is because they do not deal explicitly with file offsets, though they do operate relative to the current file pointer. The underlying file pointer can generally handle the huge file, so in theory you can read multiple 2GB blocks using fread even if you don't have a 64-bit ftell function available to query the current offset.

So even as huge file support was added to the file I/O functions in C/C++ libraries by the early 90s, there was no graceful way to promote the integer types of offset variables used in existing programs. The fread and fwrite functions were upgraded under the covers, while new versions of ftell and fseek were added because they dealt explicitly with the offset.

I've seen #ifdef WIN64 used in conjunction with _fseeki64. I am not sure why they were doing that, but don't be confused. A 64-bit operating system is not required for 64-bit file offsets; you can usually use 64-bit offsets in a 32-bit operating system.

Huge files and CMarkup

The 64-bit offset versions of fseek and ftell are used to support huge files with the CMarkup release 11.0 file mode methods. The file mode methods give read and write access to files without loading the entire document into memory. File mode does not require 64-bit offsets, but 64-bit offsets are needed if you are dealing with files over 2GB.

off_t

Most UNIX flavor compilers like gcc are standardized on the ftello and fseeko which use the off_t integer type that depends on compiler setup. The functions resolve to the old fseek/ftell or huge fseeko64/ftello64 in concert with the _FILE_OFFSET_BITS and _LARGEFILE_SOURCE macro defines.

Unfortunately VC++ doesn't use this system.

CMarkup in Visual Studio

Visual C++ provides its huge versions, _fseeki64 and _ftelli64 inconsistently. There is no clean way to always know whether the 64-bit versions are available and to automatically compile accordingly.

In CMarkup release 11.0 the 64-bit functions are used by default in Visual Studio 2005. This is not correct for target platforms where the 64-bit functions are deliberately excluded such as for Windows CE.

Since the 64-bit functions are covertly distributed with Visual Studio 6.0 (see below), in CMarkup release 11.1 I declared prototypes for the 64-bit functions by default in Visual Studio 6.0. The caused more problems because the functions are not available in some build configurations of VC++ 6.0.

In CMarkup release 11.2 with any version of Visual Studio you will need to define MARKUP_HUGEFILE in the project definitions if you want huge file access. This will eliminate these linker errors for first time users, while putting the burden on those who need huge file access to specify the define.

One alternative would be to use the Win32 File APIs and the SetFilePointerEx available on Win2K+, but this would greatly increase the code differences between the different platforms.

...
WINVER not defined. Defaulting to 0x0400,
  which is appropriate for all supported Windows CE versions
...
.\Markup.cpp(1498) : error C3861: '_fseeki64': identifier not found
.\Markup.cpp(1499) : error C3861: '_ftelli64': identifier not found
...

CMarkup started using fseeki64 and ftelli64 in release 11, but they are optional. In the next release I will not use them unless you set a define. To turn them off do something like this:

Old line 223:

#elif _MSC_VER >= 1000 // VC++

New line 223:

#elif _MSC_VER > 4000 // never

In CMarkup release 11.1 I implemented a workaround to offer huge file (>2GB) access by default in Visual C++ 6.0 and Visual Studio .NET versions before VC++ 2005. Huge file access is only useful for the developer version file mode (see Open) methods.

However, I did not realize that it depended on a project setting for these functions to be available in the VC++ libraries you link with. If in your project settings you have the default Microsoft Foundation Classes setting "Use MFC is a Shared DLL" you will get the following problem when linking CMarkup release 11.1:

Linking...
Markup.obj : error LNK2001: unresolved external symbol __ftelli64
Markup.obj : error LNK2001: unresolved external symbol __fseeki64

If you are able to change your Microsoft Foundation Classes project setting to "Use MFC in a Static Library" that is one way to alleviate the linker error and keep huge file access. Otherwise, you can change the compiler version directive as shown above and you cannot use file mode with files over 2GB.

Microsoft Visual Studio\VC98\CRT\SRC\FSEEKI64.C
Microsoft Visual Studio\VC98\CRT\SRC\FTELLI64.C

Microsoft Visual Studio\VC98\Lib\LIBCMT.LIB

dumpbin /exports libcmt.lib

does not show it but the following does:

dumpbin /all libcmt.lib

Thank you for the excellent feedback and research. Release 11.2 is fixed according to your suggestion.