External Interfaces

Large File I/O

MATLAB supports the use of 64-bit file I/O operations in your MEX-file programs. This enables you to read and write data to files that are up to and greater than 2 GB (2^31-1 bytes) in size. Note that some operating systems or compilers might not support files larger than 2 GB.

This section covers the following topics on large file I/O:

• Prerequisites to Using 64-Bit I/O
• Specifying Constant Literal Values
• Opening a File
• Printing Formatted Messages
• Replacing fseek and ftell with 64-Bit Functions
• Determining the Size of an Open File
• Determining the Size of a Closed File

Prerequisites to Using 64-Bit I/O

This section describes the components you will need to use 64-bit file I/O in your MEX-file programs:

• Header File
• Type Declarations
• Functions

Header File.   Header file io64.h defines many of the types and functions required for 64-bit file I/O. The statement to include this file must be the first #include statement in your source file, and must also precede any system header include statements:

• #include "io64.h"
  #include "mex.h"
          .
          .
          .
  

Type Declarations.   Use the following types to declare variables used in 64-bit file I/O.

MEX Type	Description	POSIX
fpos_T	Declares a 64-bit int type for setFilePos() and getFilePos(). Defined in io64.h.	fpos_t
int64_T, uint64_T	Declares 64-bit signed and unsigned integer types. Defined in tmwtypes.h.	long long
structStat	Declares a structure to hold the size of a file. Defined in io64.h.	struct stat
FMT64	Used in mexPrintf to specify length within a format specifier such as %d. See example in the section Printing Formatted Messages. FMT64 is defined in tmwtypes.h.	%lld
LL, LLU	Suffixes for literal int constant 64-bit values (C Standard ISO/IEC 9899:1999(E) Section 6.4.4.1). Used only on UNIX.	LL, LLU

Functions.   Here are the functions you will need for 64-bit file I/O. All are defined in the header file io64.h.

Function	Description	POSIX
fileno()	Gets a file descriptor from a file pointer	fileno()
fopen()	Opens the file and obtains the file pointer	fopen()
getFileFstat()	Gets the file size of a given file pointer	fstat()
getFilePos()	Gets the file position for the next I/O	fgetpos()
getFileStat()	Gets the file size of a given filename	stat()
setFilePos()	Sets the file position for the next I/O	fsetpos()

Specifying Constant Literal Values

To assign signed and unsigned 64-bit integer literal values, use type definitions int64_T and uint64_T.

On UNIX, to assign a literal value to an integer variable where the value to be assigned is greater than 2^31-1 signed, you must suffix the value with LL. If the value is greater than 2^32-1 unsigned, then use LLU as the suffix. These suffixes apply only to UNIX systems and are considered invalid on Windows systems.

Note    The LL and LLU suffixes are not required for hardcoded (literal) values less than 2G (2^31-1), even if they are assigned to a 64-bit int type.

The following example declares a 64-bit integer variable initialized with a large literal int value, and two 64-bit integer variables:

• void mexFunction(int nlhs, mxArray *plhs[], int nrhs,
                    const mxArray *prhs[])
  {
  #if defined(_MSC_VER) || defined(__BORLANDC__)     /* Windows */
     int64_T large_offset_example = 9000222000;
  #else                                              /* UNIX    */
     int64_T large_offset_example = 9000222000LL;
  #endif
  
  int64_T offset   = 0;
  int64_T position = 0;
  

Opening a File

To open a file for reading or writing, use the C fopen function as you normally would. As long as you have included io64.h at the start of your program, fopen will work correctly for large files. No changes at all are required for fread, fwrite, fprintf, fscanf, and fclose.

To open an existing file for read and update in binary mode,

• fp = fopen(filename, "r+b");
  if (NULL == fp)
     {
     /* File does not exist. Create new file for writing
      * in binary mode.
      */
     fp = fopen(filename, "wb");
     if (NULL == fp)
        {
        sprintf(str, "Failed to open/create test file '%s'",
                filename);
        mexErrMsgTxt(str);
        return;
        }
     else
        {
        mexPrintf("New test file '%s' created\n",filename);
        }
     }
  else mexPrintf("Existing test file '%s' opened\n",filename);
  

Printing Formatted Messages

You cannot print 64-bit integers using the %d conversion specifier. Instead, use FMT64 to specify the appropriate format for your platform. FMT64 is defined in the header file tmwtypes.h. The following example shows how to print a message showing the size of a large file:

• int64_T large_offset_example = 9000222000LL;
  
  mexPrintf("Example large file size: %" FMT64 "d bytes.\n",
             large_offset_example);
  

Replacing fseek and ftell with 64-Bit Functions

The ANSI C fseek and ftell functions are not 64-bit file I/O capable on most platforms. The functions setFilePos and getFilePos, however, are defined as the corresponding POSIX fsetpos and fgetpos, (or fsetpos64 and fgetpos64), as required by your platform/OS. These functions are 64-bit file I/O capable on all platforms.

The following example shows how to use setFilePos instead of fseek, and getFilePos instead of ftell. It uses getFileFstat to find the size of the file, and then uses setFilePos to seek to the end of the file to prepare for adding data at the end of the file.

Note    Although the offset parameter to setFilePos and getFilePos is really a pointer to a signed 64-bit integer, int64_T, it must be cast to an fpos_T*. The fpos_T type is defined in io64.h as the appropriate fpos64_t or fpos_t, as required by your platform/OS.

• getFileFstat(fileno(fp), &statbuf);
  fileSize = statbuf.st_size;
  offset = fileSize;
  
  setFilePos(fp, (fpos_T*) &offset);
  getFilePos(fp, (fpos_T*) &position );
  

Unlike fseek, setFilePos supports only absolute seeking relative to the beginning of the file. If you want to do a relative seek, first call getFileFstat to obtain the file size, then convert the relative offset to an absolute offset that you can pass to setFilePos.

Determining the Size of an Open File

Getting the size of an open file involves two steps:

1. Refresh the record of the file size stored in memory using getFilePos and setFilePos.
2. Retrieve the size of the file using getFileFstat.

Refreshing the File Size Record.   Before attempting to retrieve the size of an open file, you should first refresh the record of the file size residing in memory. If you skip this step on a file that is opened for writing, the file size returned might be incorrect or 0.

To refresh the file size record, seek to any offset in the file using setFilePos. If you do not want to change the position of the file pointer, you can seek to the current position in the file. This example obtains the current offset from the start of the file and then seeks to the current position to update the file size without moving the file pointer:

• getFilePos( fp, (fpos_T*) &position);
  setFilePos( fp, (fpos_T*) &position);
  

Getting the File Size.   The getFileFstat function takes a file descriptor input argument (that you can obtain from the file pointer of the open file using fileno), and returns the size of that file in bytes in the st_size field of a structStat structure:

• structStat statbuf;
  int64_T fileSize = 0;
  
  if (0 == getFileFstat(fileno(fp), &statbuf))
     {
     fileSize = statbuf.st_size;
     mexPrintf("File size is %" FMT64 "d bytes\n", fileSize);
     }
  

Determining the Size of a Closed File

The getFileStat function takes the filename of a closed file as an input argument, and returns the size of the file in bytes in the st_size field of a structStat structure:

• structStat statbuf;
  int64_T fileSize = 0;
  
  if (0 == getFileStat(filename, &statbuf))
     {
     fileSize = statbuf.st_size;
     mexPrintf("File size is %" FMT64 "d bytes\n", fileSize);
     }
  

Memory Management

Using LAPACK and BLAS Functions

© 1994-2005 The MathWorks, Inc.