Large XML file in C++

CMarkup is a simple cross-platform C++ XML API that allows you to parse large XML files, generate large XML files, and even append to large XML files, using a read or write file mode that has very low memory requirements. CMarkup also supports loading and generating large XML documents rapidly in memory with a light footprint. See what you can do with big XML:

Parse large XML file

A large XML file doesn't have to be loaded entirely into memory at once to get some or all of the information out of it. Extract information from any file regardless of size (even huge XML files >4GB) in a forward-only read-only way. The memory usage is only a small in-memory buffer of about 16KB. See the C++ XML reader available in CMarkup Developer.

CMarkup xml;
xml.Open( "7GB.xml", MDF_READFILE );
while ( xml.FindElem( "//record[@type='A']" )
{
  ...

Generate large XML file

Write information to a file of any size (even huge >4GB) in a forward-only write-only way. The memory usage is only a small in-memory buffer of about 16KB. See the C++ XML writer available in CMarkup Developer.

CMarkup xml;
xml.Open( "huge.xml", MDF_WRITEFILE );
while ( recordset.FetchNext() )
{
  xml.AddElem( "record" );
  ...

Append to large XML file

Write information to the end of a file of any size in a forward-only write-only way. The memory usage is only a small in-memory buffer of about 16KB. Append can be achieved by keeping the file open in write mode and calling Flush after writing each set of records, or by opening with the MDF_APPENDFILE flag and closing each time. See the C++ XML writer.

Large XML in-memory

You might be surprised that you can have even a 100MB+ document all in memory and manipulate it for extracting information, modifying and creating in a few seconds (sub-second speeds for 10MB+). Load, parse XML file, and save to file. The C++ XML API is designed to use the same simple methods whether going to file or keeping in memory. The memory footprint is typically 1.5 to 2.5 times the size of the document with additional temporary needs periodically when the document grows in size significantly.

CMarkup xml;
xml.Load( "100MB.xml" );
while ( xml.FindElem( "//record[@type='A']" )
{
  ...

Edit a large XML file

Edit 100MB+ large XML files in the free firstobject XML editor for Windows (foxe). A slow machine will load and display a 3MB UTF-8 file in a tenth of a second, along with the instantaneous tree view. The file size is limited by available system memory to hold the document in memory and to convert it if it is not UTF-8. Editing a file size over 500MB is possible on a machine with 4GB of RAM. You can also script CMarkup huge XML file access as shown above with FOAL.

The Advanced CMarkup Developer License comes with the Visual C++ source code of the firstobject XML Editor, which means you get the CDataEdit class to edit very large text documents, the instantaneous CMarkupTreeCtrl class, and the CFoalProgram compiler/virtual machine/debugger class.

See also:

File mode in CMarkup 11.0
Parse huge XML file in C++
C++ XML reader parses a very large XML file
XML reader models: SAX versus XML pull parser
C++ XML writer creates a very large XML file
CMarkup Open Method
CMarkup Close Method