See also:

video of XML splitter script
Format XML, indent align beautify clean up XML
Tree customization in the firstobject XML editor
firstobject Access Language
Counting XML tag names and values with foal
Convert ANSI file to Unicode

<?xml-stylesheet href="file.xsl" type="text/xsl"?>

If the XSL file is not specified, you must have both the XML and XSL documents open in the editor. When you invoke the Transform, you will be prompted with a list of open documents to select the other document. So if you press F9 from the XML document it will prompt you for the stylesheet, and if you press F9 from the stylesheet, it will prompt you for the XML to transform.

Like MSXML Validate Alt+F7, the Transform function uses the Windows MSXML component on your machine.

Those are just what has changed since intermediate release 2.3.5. A lot has happened since the last full release 2.3. Here are the interim releases:

MFC component improvements

Release 2.4 has many improvements since 2.3 in these source code classes:

See also:

Archived firstobject XML Editor 2.3 Release Notes
Archived firstobject XML Editor Release Notes

It became clear soon after CMarkup release 11.1 that another release would be needed to resolve a compiler issue affecting Visual C++, the _fseeki64 and_ftelli64 issue. From now on, some Visual C++ developers using file mode on files over 2GB will need to define MARKUP_HUGEFILE.

Here's the list of 11.2 enhancements:

• Cleaned up Visual C++ compiler _fseeki64 and_ftelli64 issue
• CMarkupMSXML: Added Transform method, just supply an XSL document "style sheet" (see MSXML Wrapper CMarkupMSXML)
• CMarkupMSXML: Implemented UTF-8 conversion for when MBCS (ANSI) strings are NOT being used

All the 11.2 developer version enhancements only affect file mode (see write mode and read mode).

• GetSubDoc in file read mode now uses smarter reallocations when extracting very large subdocuments. As CMarkup concatenates pieces of a subdocument spanning n read blocks, it won't realloc n times, more like Log(n)
• HUGE file support >2GB is no longer automatic in Visual Studio. For huge files in Visual C++ define MARKUP_HUGEFILE in your Project Settings Preprocessor definitions. To test whether you have huge file support make sure sizeof(MCD_INTFILEOFFSET) == 8 (see _fseeki64 and_ftelli64)
• fix: GetElemPath and GetParentElemPath in file mode
• fix: SetData in file write mode
• fix: OutOfElem in file write mode

See also previous CMarkup release notes: 11.1, 11.0, 10.1, 10.0, Archived CMarkup Release Notes

define wc_print
echo "
set $c = (wchar_t*)$arg0
while ( *$c )
  if ( *$c > 0x7f )
    printf "[%x]", *$c
  else
    printf "%c", *$c
  end
  set $c++
end
echo "\n
end

Then you can just type "wc <wide_string_variable_name>" (the wc shortform for wc_print should work as long as there are no other gdb commands starting with wc). The non-ASCII characters will be displayed as hex in square brackets.

This works for a wchar_t pointer, and even an STL wstring. I am guessing you don't need to call the c_str() member of the wstring because the first (and only? see below) data member of the wstring (._M_dataplus._M_p) is the wchar_t pointer.

printf %ls not reliable

At first I was using call printf with %ls but it sometimes exhibited a strange failure where nothing at all was output. It is also unhelpful for non-ASCII in my experience and will have undetermined behavior when the underlying wcstomb conversion fails. But FWIW, this is what I used (I had to cast the printf to (void) or gdb would complain "no return type information available"):

define wc_print
call (void)printf("\"%ls\"\n",$arg0)
end

Some googling uncovered a developer who described similar symptoms of a bug in printf %ls on cygwin (I was using OS X 10.5.7 with gdb 6.3.50). Maybe there are platforms where this is reliable, but it wasn't for me.

What happens with normal gdb print

When you normally print a std::wstring or wchar_t string you get output like this:

$3 = {
  static npos = 4294967295,
  _M_dataplus = {
    > = {
	  <__gnu_cxx::new_allocator> = {},  },
	members of std::basic_string,std::allocator >::_Alloc_hider:
    _M_p = 0x808e0c
  }
}

$4 = (const wchar_t *) 0x44920

It doesn't show the text content like it does with a char* like this:

$2 = 0x3e090 "Hello"

So instead of "p <wide_string_variable_name>" you can now just use "wc <wide_string_variable_name>".

Defining a gdb command

By defining the command, it is very convenient to display any wide string. You can put the define in a file and source the file when you need it (using the gdb source command) or put it in your .gdbinit file. If your .gdbinit file is in your work folder, you will have the command every time you start gdb there.

You can also add the following after your command definition to document it:

document wc_print
wc_print <wide_string_variable_name>
Display <wide_string_variable_name> which is a wchar_t* or wstring.
end

32-bit and 64-bit file offsets

The limit of signed 32-bit integer offsets in ftell and fseek is 2^31-1 = 2147483647 which is around 2GB. For example, a common cross-platform way of getting file size is fseek with SEEK_END and then ftell, but ftell cannot return a number higher than the limit of its return type.

The 64-bit versions of these functions deal in offsets of billions of gigabytes (8 exabytes) which should be enough for single file sizes in the next decade or two ;).

There are no 64-bit offset versions of fread and fwrite functions. This is because they do not deal explicitly with file offsets, though they do operate relative to the current file pointer. The underlying file pointer can generally handle the huge file, so in theory you can read multiple 2GB blocks using fread even if you don't have a 64-bit ftell function available to query the current offset.

So even as huge file support was added to the file I/O functions in C/C++ libraries by the early 90s, there was no graceful way to promote the integer types of offset variables used in existing programs. The fread and fwrite functions were upgraded under the covers, while new versions of ftell and fseek were added because they dealt explicitly with the offset.

I've seen #ifdef WIN64 used in conjunction with _fseeki64. I am not sure why they were doing that, but don't be confused. A 64-bit operating system is not required for 64-bit file offsets; you can usually use 64-bit offsets in a 32-bit operating system.

Huge files and CMarkup

The 64-bit offset versions of fseek and ftell are used to support huge files with the CMarkup release 11.0 file mode methods. The file mode methods give read and write access to files without loading the entire document into memory. File mode does not require 64-bit offsets, but 64-bit offsets are needed if you are dealing with files over 2GB.

off_t

Most UNIX flavor compilers like gcc are standardized on the ftello and fseeko which use the off_t integer type that depends on compiler setup. The functions resolve to the old fseek/ftell or huge fseeko64/ftello64 in concert with the _FILE_OFFSET_BITS and _LARGEFILE_SOURCE macro defines.

Unfortunately VC++ doesn't use this system.

CMarkup in Visual Studio

Visual C++ provides its huge versions, _fseeki64 and _ftelli64 inconsistently. There is no clean way to always know whether the 64-bit versions are available and to automatically compile accordingly.

In CMarkup release 11.0 the 64-bit functions are used by default in Visual Studio 2005. This is not correct for target platforms where the 64-bit functions are deliberately excluded such as for Windows CE.

Since the 64-bit functions are covertly distributed with Visual Studio 6.0 (see below), in CMarkup release 11.1 I declared prototypes for the 64-bit functions by default in Visual Studio 6.0. The caused more problems because the functions are not available in some build configurations of VC++ 6.0.

In CMarkup release 11.2 with any version of Visual Studio you will need to define MARKUP_HUGEFILE in the project definitions if you want huge file access. This will eliminate these linker errors for first time users, while putting the burden on those who need huge file access to specify the define.

One alternative would be to use the Win32 File APIs and the SetFilePointerEx available on Win2K+, but this would greatly increase the code differences between the different platforms.

...
WINVER not defined. Defaulting to 0x0400,
  which is appropriate for all supported Windows CE versions
...
.\Markup.cpp(1498) : error C3861: '_fseeki64': identifier not found
.\Markup.cpp(1499) : error C3861: '_ftelli64': identifier not found
...

CMarkup started using fseeki64 and ftelli64 in release 11, but they are optional. In the next release I will not use them unless you set a define. To turn them off do something like this:

Old line 223:

#elif _MSC_VER >= 1000 // VC++

New line 223:

#elif _MSC_VER > 4000 // never

In CMarkup release 11.1 I implemented a workaround to offer huge file (>2GB) access by default in Visual C++ 6.0 and Visual Studio .NET versions before VC++ 2005. Huge file access is only useful for the developer version file mode (see Open) methods.

However, I did not realize that it depended on a project setting for these functions to be available in the VC++ libraries you link with. If in your project settings you have the default Microsoft Foundation Classes setting "Use MFC is a Shared DLL" you will get the following problem when linking CMarkup release 11.1:

Linking...
Markup.obj : error LNK2001: unresolved external symbol __ftelli64
Markup.obj : error LNK2001: unresolved external symbol __fseeki64

If you are able to change your Microsoft Foundation Classes project setting to "Use MFC in a Static Library" that is one way to alleviate the linker error and keep huge file access. Otherwise, you can change the compiler version directive as shown above and you cannot use file mode with files over 2GB.

Microsoft Visual Studio\VC98\CRT\SRC\FSEEKI64.C
Microsoft Visual Studio\VC98\CRT\SRC\FTELLI64.C

Microsoft Visual Studio\VC98\Lib\LIBCMT.LIB

dumpbin /exports libcmt.lib

does not show it but the following does:

dumpbin /all libcmt.lib

Thank you for the excellent feedback and research. Release 11.2 is fixed according to your suggestion.

split_XML()
{
  CMarkup input;
  input.Load( "input.xml" );
  while ( input.FindElem("//npc") )
    WriteTextFile( "npc"+input.GetAttrib("id")+".xml", input.GetSubDoc() );
}

This script will read input.xml and create an npc[ID].xml file for each npc subdocument. This was the question:

<root>
<npc id="1">
  <p pid="1"/>
  <p pid="2"/>
  <p pid="3"/>
</npc>
<npc id="2">
  <p pid="3"/>
  <p pid="4"/>
  <p pid="5"/>
</npc>
<npc id="3">
  <p pid="4"/>
  <p pid="5"/>
  <p pid="6"/>
</npc>
</root>

<npc id="1">
  <p pid="1"/>
  <p pid="2"/>
  <p pid="3"/>
</npc>

<npc id="2">
  <p pid="3"/>
  <p pid="4"/>
  <p pid="5"/>
</npc>

<npc id="3">
  <p pid="4"/>
  <p pid="5"/>
  <p pid="6"/>
</npc>

The simplicity of a short bit of CMarkup code is yet another reason to avoid XSLT. Here is a similar question asked here:

<REFERRAL_DISCHARGE>
  <FILE_VERSION>1.0</FILE_VERSION>
  <REFERRAL_ID>9999</REFERRAL_ID>
  <ORGANISATION_ID>A12345-6</ORGANISATION_ID>
  <ORGANISATION_TYPE>100</ORGANISATION_TYPE>
  <EXTRACT_FROM_DATE_TIME>2009-06-01T00:00:00</EXTRACT_FROM_DATE_TIME>
  <EXTRACTED_DATE_TIME>2009-06-30T00:30:00</EXTRACTED_DATE_TIME>
  <TEAM_CODE>1111</TEAM_CODE>
  <EVENT_HCU_ID>AAA1234</EVENT_HCU_ID>
  <SEX>M</SEX>
  <DATE_OF_BIRTH>1900-05-05</DATE_OF_BIRTH>
  <REFERRAL_FROM>UN</REFERRAL_FROM>
  <START_DATE_TIME>2008-12-24T00:00:00</START_DATE_TIME>
</REFERRAL_DISCHARGE>

Since the input file is under 10MB it makes sense to Load it all at once rather than using the Open method for read mode. Also, it is easier to pick out the data for naming the output file using an in-memory XML document than being restricted by forward-only file read mode. Here's the script to divide the XML into files with one REFERRAL_DISCHARGE each:

split()
{
  CMarkup xmlInput;
  xmlInput.Load( "split.xml" );
  int nFileCount = 0;
  while ( xmlInput.FindElem("//REFERRAL_DISCHARGE") )
  {
    ++nFileCount;
    xmlInput.FindChildElem( "REFERRAL_ID" );
    str sID = xmlInput.GetChildData();
    str sFilename = "test" + "_" + sID + "_"+ nFileCount + ".xml";
    WriteTextFile( sFilename, xmlInput.GetSubDoc() );
  }
  return nFileCount;
}

In this simple REFERRAL_DISCHARGE subdocument the REFERRAL_ID is a child element and we can find it using the FindChildElem method while keeping the main position at the REFERRAL_DISCHARGE element. This allows us to still retrieve the whole REFERRAL_DISCHARGE subdocument with GetSubDoc after building the filename.

If the source is a large XML file, and loading it all into memory is not feasable, it is still pretty simple:

split()
{
  CMarkup xmlInput, xmlReferralDischarge;
  xmlInput.Open( "big_split.xml", MDF_READFILE );
  int nFileCount = 0;
  while ( xmlInput.FindElem("//REFERRAL_DISCHARGE") )
  {
    ++nFileCount;
    xmlReferralDischarge.SetDoc( xmlInput.GetSubDoc() );
    xmlReferralDischarge.FindChildElem( "REFERRAL_ID" );
    str sID = xmlReferralDischarge.GetChildData();
    str sFilename = "test" + "_" + sID + "_"+ nFileCount + ".xml";
    xmlReferralDischarge.Save( sFilename );
  }
  xmlInput.Close();
  return nFileCount;
}

Since each individual REFERRAL_DISCHARGE subdocument is small, we populate xmlReferralDischarge in-memory using SetDoc, grab the ID out of it, and then use the Save method to write the file.

And it is not too hard to put multiple subdocuments into each output file and to split XML into very large files from an extremely large file. In Split XML file into smaller pieces and the video of an XML splitter script I put multiple subdocuments into each output file and use "file mode" to keep a low footprint.

Here is the script shown in the video. I've highlighted the following things you will need to change to fit your own purposes:

split()
{
  CMarkup xmlInput, xmlOutput;
  xmlInput.Open( "50MB.xml", MDF_READFILE );
  int nObjectCount = 0, nFileCount = 0;
  while ( xmlInput.FindElem("//ACT") )
  {
    if ( nObjectCount == 0 )
    {
      ++nFileCount;
      xmlOutput.Open( "piece" + nFileCount + ".xml", MDF_WRITEFILE );
      xmlOutput.AddElem( "root" );
      xmlOutput.IntoElem();
    }
    xmlOutput.AddSubDoc( xmlInput.GetSubDoc() );
    ++nObjectCount;
    if ( nObjectCount == 5 )
    {
      xmlOutput.Close();
      nObjectCount = 0;
    }
  }
  if ( nObjectCount )
    xmlOutput.Close();
  xmlInput.Close();
  return nFileCount;
}

There is also another article about how to split XML with the editor and naming the output files based on information contained in the XML.

See also:

Format XML, indent align beautify clean up XML
Tree customization in the firstobject XML editor
firstobject Access Language
Counting XML tag names and values with foal
Convert ANSI file to Unicode

The question when splitting XML is where do you want to split it? There could be a logical place to divide the XML like into the subdocuments immediately under the root. Or you might simply have a size limit and want to divide your large XML file with ten million objects into files with one million each.

Below is C++ XML splitter code to split an XML file containing N million objects into N files containing 1 million objects. Here is the idea:

// Split XML
CMarkup xmlInput, xmlOutput;
xmlInput.Open( "please_split.xml", MDF_READFILE );
int nObjectCount = 0, nFileCount = 0;
while ( xmlInput.FindElem("//object") )
{
  if ( nObjectCount == 0 )
  {
    ++nFileCount;
    xmlOutput.Open( "piece" + StrFromInt(nFileCount) + ".xml", MDF_WRITEFILE );
    xmlOutput.AddElem( "root" );
    xmlOutput.IntoElem();
  }
  xmlOutput.AddSubDoc( xmlInput.GetSubDoc() );
  ++nObjectCount;
  if ( nObjectCount == 1000000 )
  {
    xmlOutput.Close();
    nObjectCount = 0;
  }
}
if ( nObjectCount )
  xmlOutput.Close();
xmlInput.Close();

You could also use size rather than object count as the basis of where to split the XML document. To do this, keep a tally of the subdocument sizes until a threshhold is reached. The subdocument transfer shown above occurs in xmlOutput.AddSubDoc( xmlInput.GetSubDoc() ). You can instead do it in two steps and track the size like this:

MCD_STR sObject = xmlInput.GetSubDoc();
nOutputLength += MCD_STRLENGTH(sObject);
xmlOutput.AddSubDoc( sObject );

Note though that the nOutputLength is not the same as the output file byte size if your in-memory encoding is different from your file encoding or your encoding is 2-byte based UTF-16.

]]> http://www.firstobject.com/cmarkup-11.1-release-notes.htm cmarkup-11.1-release-notes.htm Sun, 07 Jun 2009 08:00:00 GMT CMarkup Articles Release 11.1 Date: June 7, 2009, download

After "file mode" was introduced in release 11.0 (developer version) it became clear we needed a convenient way to dynamically handle subdocuments in memory and handle subdocuments or pieces of XML in memory. This release fills out those essential features with new support for file mode in AddSubDoc and GetSubDoc. To learn more about file mode in CMarkup see the C++ XML writer and C++ XML reader.

Another great developer version feature is the new attribute support in FindGetData and FindSetData extending those powerful functions to set and get attributes in one step. See Dynamic Structure Documents.

Here's the list of 11.1 enhancements:

• To deflate a CMarkup object's memory usage, SetDoc(NULL) is now different from SetDoc("") and SetDoc(str); passing NULL frees up most of the index memory and ensures minimum allocation in the string object. See SetDoc

• fix: 11.0 Bug: file read and write modes *thanks Dave
• HUGE file support >2GB is now automatic in Visual C++ before Visual C++ 2005. To test whether you have huge file support make sure sizeof(MCD_INTFILEOFFSET) == 8
• GetSubDoc is now supported in read file mode and AddSubDoc is now supported in write file mode; these support important in-memory subdocument manipulation
• FindGetData and FindSetData now support getting and setting attributes and creating any elements necessary (specified by simple absolute path) to set an attribute
• GetEncodingCodePage, GetOpenFileSize, and GetOpenFileOffset accessors added
• fix: Linux __w64 Markup.cpp line 752 pointer arithmetic warning fixed *thanks Mohan

See also:

CMarkup 11.0 Release Notes
CMarkup 10.1 Release Notes
CMarkup 10.0 Release Notes
Archived CMarkup Release Notes

In one of its features for a large XML file in C++, CMarkup's XML pull functionality was designed to provide the low overhead benefit of an XMLReader without the complexity of a SAX event based parser. An XML pull parser lets you pull from a large XML file a little bit at a time forward-only and read-only, processing the data you want, but keeping only a small block of the file in memory at a time. The overhead is very small and the speed is almost as fast as the I/O read operation.

XML pull parser functionality (file read mode) is in the developer version of CMarkup.

You can quickly write an application to process a file a little bit at a time starting with the Open method. The following code loops through all of the object elements to process the properties of each object.

CMarkup xmlpullparser;
xmlpullparser.Open( "hugexmlfile.xml", CMarkup::MDF_READFILE );
while ( xmlpullparser.FindElem("//object") )
{
  // process object properties...

If you were loading the entire file into memory, you could use methods like ResetMainPos, RestorePos and GotoElemIndex to go back and scan for each property from the beginning of the object. But in file read mode you are forward-only. So don't do this:

xmlpullparser.IntoElem();
xmlpullparser.FindElem( "property1" );
str sProp1 = xmlpullparser.GetData();
xmlpullparser.FindElem( "property2" );
str sProp2 = xmlpullparser.GetData();
xmlpullparser.OutOfElem();

If any of those properties is not found, the parser will scan to the end of the object element bypassing the remaining properties. Or if any of those properties is not in the expected order, again you could bypass some of the properties. Instead, you must handle each property in the order of occurence like this:

xmlpullparser.IntoElem();
while ( xmlpullparser.FindElem() )
{
  str sPropName = xmlpullparser.GetTagName();
  str sPropValue = xmlpullparser.GetData();
  if ( sPropName == "property1" )
    ; // do something with property 1
  else if ( sPropName == "property2" )
    ; // do something with property 2
  // etc
}
xmlpullparser.OutOfElem();

Parse large XML file

A large XML file doesn't have to be loaded entirely into memory at once to get some or all of the information out of it. Extract information from any file regardless of size (even huge XML files >4GB) in a forward-only read-only way. The memory usage is only a small in-memory buffer of about 16KB. See the C++ XML reader available in CMarkup Developer.

CMarkup xml;
xml.Open( "7GB.xml", MDF_READFILE );
while ( xml.FindElem( "//record[@type='A']" )
{
  ...

Generate large XML file

Write information to a file of any size (even huge >4GB) in a forward-only write-only way. The memory usage is only a small in-memory buffer of about 16KB. See the C++ XML writer available in CMarkup Developer.

CMarkup xml;
xml.Open( "huge.xml", MDF_WRITEFILE );
while ( recordset.FetchNext() )
{
  xml.AddElem( "record" );
  ...

Append to large XML file

Write information to the end of a file of any size in a forward-only write-only way. The memory usage is only a small in-memory buffer of about 16KB. Append can be achieved by keeping the file open in write mode and calling Flush after writing each set of records, or by opening with the MDF_APPENDFILE flag and closing each time. See the C++ XML writer.

Large XML in-memory

You might be surprised that you can have even a 100MB+ document all in memory and manipulate it for extracting information, modifying and creating in a few seconds (sub-second speeds for 10MB+). Load, parse XML file, and save to file. The C++ XML API is designed to use the same simple methods whether going to file or keeping in memory. The memory footprint is typically 1.5 to 2.5 times the size of the document with additional temporary needs periodically when the document grows in size significantly.

CMarkup xml;
xml.Load( "100MB.xml" );
while ( xml.FindElem( "//record[@type='A']" )
{
  ...

Edit a large XML file

Edit 100MB+ large XML files in the free firstobject XML editor for Windows (foxe). A slow machine will load and display a 3MB UTF-8 file in a tenth of a second, along with the instantaneous tree view. The file size is limited by available system memory to hold the document in memory and to convert it if it is not UTF-8. Editing a file size over 500MB is possible on a machine with 4GB of RAM. You can also script CMarkup huge XML file access as shown above with FOAL.

The Advanced CMarkup Developer License comes with the Visual C++ source code of the firstobject XML Editor, which means you get the CDataEdit class to edit very large text documents, the instantaneous CMarkupTreeCtrl class, and the CFoalProgram compiler/virtual machine/debugger class.

See also:

File mode in CMarkup 11.0
Parse huge XML file in C++
C++ XML reader parses a very large XML file
XML reader models: SAX versus XML pull parser
C++ XML writer creates a very large XML file
CMarkup Open Method
CMarkup Close Method

File read and write modes (see C++ XML writer too) are in the developer version of CMarkup.

Even though CMarkup is surprisingly light and quick to load and modify multi-megabyte files in memory, this XML reader mode is for cases where that is not good enough. Read mode provides read-only forward-only pull parser access, even in huge multi-gigabyte XML files.

How to use CMarkup file read mode

Here's some code that scans the elements in a large XML file fast, without trying to load the entire file at once. The main difference from regular CMarkup usage is that rather than calling Load, you Open the file for read. Notice the Open and Close calls:

CMarkup xmlreader;
xmlreader.Open( "largeXMLfile.xml", MDF_READFILE );
xmlreader.FindElem(); // root
MCD_STR sType = xmlreader.getAttrib( "infotype" );
xmlreader.IntoElem();
while ( xmlreader.FindElem() )
{
  MCD_STR sID = xmlreader.GetAttrib( "id" );
  xmlreader.IntoElem();
  MCD_STR sName = xmlreader.FindGetData( "name" );
  MCD_STR sRef = xmlreader.FindGetData( "ref" );
  xmlreader.OutOfElem();
}
xmlreader.Close();

Pull parser design

The CMarkup C++ XML reader requires no callbacks, no events, and no setup. Just open the file and pull what you want from it using the same CMarkup methods you would use if you were navigating it in memory. Rather than developing an entirely new interface for the XML reader (and XML writer) functionailty of CMarkup, much remains the same except that you open the file instead of accessing the document in memory. See XML reader models: SAX versus XML pull parser for a discussion of the major XML reader design options.

Here is an example of a query lookup based on an id. Open the file, query the information, and close it. This example will do a sequential read through the file until it finds the matching information, or until it reaches the end of the file.

CMarkup xmlreader;
xmlreader.Open( "largeXMLfile.xml", MDF_READFILE );
if ( xmlreader.FindElem("//data[@id='5632av']") )
{
  xmlreader.IntoElem();
  MCD_STR sName = xmlreader.FindGetData( "name" );
}
xmlreader.Close();

C++ XML reader methods

CMarkup's file read mode limits the methods you can use and the ways you can use those methods. The key thing to remember is that it is forward-only pull parsing from file so you can only navigate forward in the document you are reading once-through. And since you can only read in a single position, you cannot use child element methods.

Here are the CMarkup methods that can be used, and a brief explanation of how they work in file read mode:

You can also use any CMarkup static utility function because these do not involve the CMarkup object state or data members.

A window into the document

File read mode provides on-the-fly charset conversion to the in-memory charset of a MARKUP_FILEBLOCKSIZE-based size "read block" at a time. The m_strDoc document string member is used as a partial document buffer letting you view the current block of the document in the debugger variables the same way you do when not in file mode, giving great visibility into the document and the behind the scenes positioning in the actual document text.

Copying a CMarkup object in file mode

The copy constructor and assignment operator = do not work when copying a CMarkup object in either read or write file mode. This is because the CMarkup object encapsulates an open file pointer, a system handle which can only be managed by one CMarkup object at a time.

Poorly formed markup containment hierarchy

This note is only for developers dealing with HTML or loosely formed markup. CMarkup is designed to deal with ill-formed XML and things such as non-ended <br> line break tags in HTML. See Generic Markup In CMarkup.

So in that same spirit, file read mode is able to keep going despite non-hierarchically formed markup. However, the recovery algorithm works differently in read mode because CMarkup has not parsed the whole file and does not know what is to come in the rest of the file.

In the case of a non-ended element tag, file read mode has to assume the end tag will be found later in the file until it encounters the end tag of the enclosing element. This is different than the in-memory policy for dealing with non-hierarchical markup described in the CMarkup Containment Hierarchy.

File read and write modes (see C++ XML reader too) are in the developer version of CMarkup.

Even though CMarkup can lightly create multi-megabyte documents in memory and then write them to disk, file write mode lets you write to disk as you create the document to conserve memory. File write mode provides low footprint write-only forward-only push to file to create huge XML files.

How to use CMarkup file write mode

Instead of immediately adding elements to the newly instantiated CMarkup object, just use the Open method to open a file in write mode before adding elements. Here's some C++ source code to show you how it works; the only new methods you need are Open and Close:

CMarkup xmlwriter;
xmlwriter.Open( "inventorydata.xml", MDF_WRITEFILE );
xmlwriter.AddElem( "root" );
xmlwriter.SetAttrib( "infotype", "inventorydata" );
xmlwriter.IntoElem();
std::string strID, strName, strRef;
while ( GetInventoryData(strID,strName,strRef) )
{
  xmlwriter.AddElem( "data" );
  xmlwriter.SetAttrib( "id", strID );
  xmlwriter.IntoElem();
  xmlwriter.AddElem( "name", strName );
  xmlwriter.AddElem( "ref", strRef );
  xmlwriter.OutOfElem();
}
xmlwriter.Close();

C++ XML writer methods

CMarkup's file write mode limits the methods you can use and the ways you can use those methods. The key thing to remember is that it is forward-only push to file so you cannot navigate in the document you are creating; you can only add elements and nodes, and set attributes. And since you can only write in a single position, you cannot use child element methods.

Here are the CMarkup methods that can be used, and a brief explanation of how they work in file write mode:

You can also use any CMarkup static utility function because these do not involve the CMarkup object state or data members.

Appending

If you are generating a log file without a root element, you can open the file with the MDF_APPENDFILE flag to add data starting at the end of the file if it exists. The Flush method can be used to ensure data has been written to disk when you want to reduce chances of failure in the middle of a logical section of the data; however it will reduce performance if used too often.

A window into the document

In file write mode, the m_strDoc document string member is used as a partial document buffer letting you view the current segment or "write block" of the document in the debugger variables the same way you do when not in file mode, giving great visibility into the document and the behind the scenes positioning in the actual document text. The partial document text holds the markup you create until a MARKUP_FILEBLOCKSIZE-based size is reached and it is converted to the file charset as it is written to file.

Copying a CMarkup object in file mode

The copy constructor and assignment operator = do not work when copying a CMarkup object in either read or write file mode. This is because the CMarkup object encapsulates an open file pointer, a system handle which can only be managed by one CMarkup object at a time.

A developer at NASA Jet Propulsion Laboratory contacted me years ago evaluating the use of XML to store metadata for 6 to 12 GB of raw telemetry data collected each observation night. The metadata file would not be as large as the raw data file, but it needed to be one file that would grow through the night. He did't want to re-save the whole file with each update, he needed to append (to write and flush) only the additional records to get them to disk as quickly as possible to minimize data loss in the case of a computer crash.

The new "file mode" in CMarkup essentially lets you use your familiar CMarkup methods directly on files, rather than having the entire document in memory. This opens the way for more efficient large XML file-based solutions, and even really huge XML files. To learn more about file mode in CMarkup see the C++ XML writer and C++ XML reader.

Here's the list of 11.0 enhancements:

• New GetResult method returns detailed result markup which should be used instead of GetError for structured result information. All result and error reporting has been overhauled in this release, and the "empty document" message has been removed
• Debugging information improved to show current node and current parent in a single m_pCurrent pointer. See Debugging with CMarkup
• Updated charset names, aliases and code pages in CMarkup and added support for GB18030 code page 54936
• Refactored CMarkup to remove all nested structs and greatly simplify the header file while better encapsulating struct functionality in the cpp file, making the code cleaner, more readable, easier to export from a library, and compatible with more compilers

• New C++ XML writer and C++ XML reader handle huge/large XML file-based requirements. The new methods are Open, Close, and Flush
• New HasAttrib method to compliment GetAttrib in distinuishing between an empty attribute value and not having the attribute at all. Previously, you needed to iterate with GetAttribName to determine this. New also is the corresponding HasChildAttrib method

See also:

CMarkup 10.1 Release Notes
CMarkup 10.0 Release Notes
Archived CMarkup Release Notes