.. _2d-readerswriters:

Writing Image Readers & Writers
===============================

Readers and Writers are an intrinsic part of NUKE. These 2D operators allow us to overcome the problem of reading and writing to and from the ever growing existing file/image formats.

Knowing about the readers and writers architecture is therefore an important part of third-party NUKE development, which many times involves custom file formats.

This section goes through the in's and out's of creating, understanding, and using Readers and Writers.

Architecture
------------

Before we delve into the coding part of creating your own readers and writers, it is important to understand how NUKE handles the reading and writing of files.

To identify which reader or writer version is needed, NUKE relies on parsing the file extension and finding a matching reader. For example, NUKE always tries to match up a file with a *.mov* extension to a *movReader.<OS specific extension>* reader.

There is still however a problem that can arise from a single format having multiple extensions. An example of this is JPG, which can be represented as both *.jpg* and *.jpeg*. 
With just the previous system in place, it would mean creating or duplicating the jpgReader extension to satisfy both *jpg* and *jpeg*.
NUKE provides two systems to help you get around this problem: one is through a TCL script file that it tries to invoke just before it attempts to find the reader in question and can be used to redirect a reader.
This TCL script must have the same name as the source reader. In the case of *.jpg* to *.jpeg*, the script would be called *jpegReader.tcl*, as for its contents it simply needs to redirect to the correct reader as follows:

.. code-block:: tcl

  # jpegReader.tcl
  load jpgReader

This script is invoked instead of *jpegReader.<OS specific extension>* and ensures the correctly named *jpgReader.<OS specific extension>* is loaded for the Read.

The other system is by extending the supported extension types for the reader itself (this is explained below).

Readers
-------

.. _reader-base-reference:

The Reader Base Class
*********************

.. cpp:class:: DD::Image::Reader

A Reader in the context of NUKE third-party development provides a base class and interfaces to which NUKE can plug into in order to read a custom piece of data from disk.
So in order to create a custom Reader, the first step is to create a new Reader whose base class is *DD::Image::Reader*. There is a specialization of this class that can be used in its stead: *DD::Image::FileRead*. This class is a specialization of *DD::Image::Reader* and provides extended file management functionality. This will be explained in the next sub-section: :ref:`reader-filereader-reference`.

In order to create a custom reader using the *DD::Image::Reader* base class, the derived class for the new reader needs to follow some specific steps and implement the basic interfaces.

The ``DD::Image::Reader::Description`` interface must be populated with the relevant information. This allows NUKE to identify and manage the readers and their supported extensions. This static member can be declared as follows::

  class MyReader : DD::Image::Reader 
  {
    static const Description d;
    ...
  };
  
  static Reader* build( Read* r, int fd, const unsigned char* b, int n ) 
  {
      return new MyReader( r, fd, b, n );
  }
  
  static bool test( int fd, const unsigned char* block, int n ) 
  {
    return ( block[0] == MY_MAGIC_BYTE );
  }

  const Reader::Description MyReader::d( "myFile\0", "my file format", build, test );

In this example, a class *MyReader* is derived from *DD::Image::Reader*. The derived class then declares a static *Description* member. This description is what will control the construction and evaluate whether or not the *MyReader* is suitable to manipulate the current file, through the declaration of *build* and *test* in this case.

The previous implementation looks for files with the extension *.myFile*. If this specific format is found, a test is performed to ensure the format matches up to the extension. In this case, the *MY_MAGIC_BYTE* is identified as the first element of the buffer. In the case of a misnamed format, NUKE still tests against the tester function. It falls then on the user to ensure the testing is sufficient to determine the file type is readable and to avoid collisions with similar file formats.

As well as using the TCL solution suggested in the previous section, multiple extensions can be specified on a reader's description by constructing it with a *NULL* separated list for its first argument::

  const Reader::Description MyReader::d( "myFile\0someOtherFile", "my file format", build, test );

In this example, not only is the reader used for files with a *.myFile* extension but also for *.someOtherFile*.

The constructor should use the ``DD::Image::Reader::set_info()`` to set up the area to be accessed from the file IO. This should (if possible) be done by a quick read operation. Complex reading, decoding, and buffering can be done via ``DD::Image::Reader::open()``. 

The last function that needs to be overriden in order to create a basic working custom reader is ``DD::Image::Reader::engine()``. This function implements the actual reading of the file data and performs the user conversion into the NUKE native image format::

  void MyReader::engine( int y, int x, int r, ChannelMask mask, Row& row )
  {
    row.range( 0, width() );

    for( int z = 0; z < 4; z++ ) {
      // A user implemented read function
      custom_read_channel( y, z, row.writable( z ) );
    }
  }

This code represents a basic ``DD::Image::Reader::engine()`` implementation. It starts by setting the range of the row to the full width and then reads all of the channels into the data row.

Other Interesting Methods
^^^^^^^^^^^^^^^^^^^^^^^^^

These are optional methods that may be implemented to provide extra functionality and information to your Reader.

============================================================================================= ==============================================================
Function                                                                                      Description
.. cpp:function:: void DD::Image::Reader::prefetchMetaData()                                  This should be overriden to have frame-specific metadata.
.. cpp:function:: const MetaData::Bundle& DD::Image::Reader::fetchMetaData( const char* key ) A function that can be used to fetch metadata.
.. cpp:function:: bool DD::Image::Reader::supports_stereo() const                             Flag indicating whether it supports stereo or not.
.. cpp:function:: bool DD::Image::Reader::fileStereo() const                                  Flag indicating whether this file is stereo or not.
.. cpp:function:: bool DD::Image::Reader::videosequence() const                               Flag indicating whether or not this file is a video sequence.
============================================================================================= ==============================================================

.. _reader-filereader-reference:

The FileReader Base Class
*************************

.. cpp:class:: DD::Image::FileReader

FileReader is a class derived from Reader. It performs very much like its base class, and a basic *FileReader* implementation must also implement the same basic interfaces (see :ref:`reader-base-reference`). What makes *DD::Image::FileReader* interesting, however, is the added functionality it can provide. While creating your reader directly from *DD::Image::Reader* can entail having to also implement file manipulation functions (this can ofcourse be desirable for specific systems), the *FileReader* is a suitable solution for most implementations by providing functions to access and manipulate file data directly.

So in order to use this additional functionality, we simply need to inherit from our custom reader from *DD::Image::FileReader*::

  class MyReader : public DD::Image::FileReader
  {
    ...
  };

Now, our reader can use the file handling helper functions associated with FileReader.


FileReader Extended Functionality Methods
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The extended methods that are provided by the *FileReader* for extra functionality:

==================================================================================================== ===============================================
Function                                                                                             Description
.. cpp:function:: int DD::Image::FileReader::lock( FILE_OFFSET offset, int min_length, int length )  Locks the data between the given parameters.
.. cpp:function:: int DD::Image::FileReader::lock( FILE_OFFSET offset, int l )                       Locks the data between the given parameters.
.. cpp:function:: int DD::Image::FileReader::lock( FILE_OFFSET offset, unsigned l )                  Locks the data between the given parameters.
.. cpp:function:: const unsigned char& DD::Image::FileReader::byte( FILE_OFFSET n ) const            Accesses the byte at the given offset.
.. cpp:function:: const unsigned char* DD::Image::FileReader::at( FILE_OFFSET n ) const              Accesses the byte at the given offset.
.. cpp:function:: void DD::Image::FileReader::unlock()                                               Unlocks a previous locked buffer.
.. cpp:function:: int DD::Image::FileReader::read( void* p, FILE_OFFSET offset, int min_, int max_)  Reads directly from the file into memory.
.. cpp:function:: int DD::Image::FileReader::read( void* p, FILE_OFFSET offset, int l )              Reads directly from the file into memory.
.. cpp:function:: int DD::Image::FileReader::read( void* p, File_OFFSET offset, unsigned l )         Reads directly from the file into memory.
==================================================================================================== ===============================================


Knobs on Readers
****************

Adding knobs to readers requires an extra step because Readers do not behave like normal Ops. There is a special object that can be initialized and registered with the Reader to give it the same capabilities as a regular Op to create knobs. The two main steps to achieve this are:

1. Derive a custom object of type ``DD::Image::ReaderFormat``. The *ReaderFormat* is then capable of implementing the *knobs()* and *knob_changed()* interfaces.
2. Create a new object of the right type within your custom reader and make sure the correct reader format is initialized.

The following code does just that::

  class MyFileFormat : public DD::Image::ReaderFormat
  {
    // implementation of the knobs call
    void knobs(Knob_Callback c)
  }

  class MyReader : public DD::Image::FileReader
  {
    static const Description d;
  }

  static Reader* build( Read* r, int fd, const unsigned char* b, int n ) 
  {
    return new MyReader( r, fd, b, n );
  }
  
  static bool test( int fd, const unsigned char* block, int n ) 
  {
    return ( block[0] == MY_MAGIC_BYTE );
  }

  static ReaderFormat* buildformat(Read* iop)
  {
    return new MyFileFormat();
  }
  
  const Reader::Description MyReader::d( "myFile\0", "my file format", build, test, buildFormat );

The previous code now extends our reader to provide a ReaderFormat that can implement and extend the knobs collection for the *MyReader* object.

Reading MetaData
****************

Metadata is accessed using the same pattern as the decoding. The ``DD::Image::fetchMetaData( const char* key )`` can be implemented to return an object of type ``DD::Image::MetaData::Bundle``.

::

  class MyReader : public DD::Image::FileReader
  {
    ...
    DD::Image::MetaData::Bundle& fetchMetaData( const char* key )
    {
      return _metaData;
    }
    
    DD::Image::MetaData::Bundle _metaData;
  }

In this example, the _metaData object is returned when requested. This metadata is already pre-built somewhere else during object/reader construction.

Testing with Tester
*******************

As it was previously mentioned (see :ref:`reader-base-reference`), when a file is read, NUKE tries to identify which reader to use based on the registered valid extensions for each individual reader. A second mechanism also exists to safeguard against misnamed files - the ``DD::Image::Reader::(Tester*)( int fd, const unsigned char* block, int n )``::

  #define MY_MAGIC_BYTE 0x000000001
  static bool test( int fd, const unsigned char* block, int n ) 
  {
    return ( block[0] == MY_MAGIC_BYTE );
  }

The previous code is a simple test which checks the first byte of that and ensures it matches the defined *MY_MAGIC_BYTE*. This is of course a very simple example of how to set it up. A truly usable implementation would check for a more complex bit pattern in order to ensure the format matches.


Working Metadata
****************

Metadata in NUKE is handled through the ``DD::Image::MetaData::Bundle`` object. This abstract object provides all of the functionality needed to write and read data of integral types (int, float, string...).

There are two important things that need to be taken into account when handling MetaData, as metadata can be split into two categories:

1. NUKE recognized and defined metadata. This is the case for time codes, where a specific key (``DD::Image::MetaData::TIMECODE``) should be used to store this value on a specific format. So when NUKE wants to read or write a time code, this is the key it uses.
2. Custom metadata. These are recognized internally as simple key-value pairs and should be handled on a case-by-case basis by the user.

For example, if you decide to use a key named "somenamespace/timecode" instead of ``DD::Image::MetaData::TIMECODE`` to store your time codes, then you are responsible for maintaining and ensuring that data is accessed and written to the file where applicable.

Whenever possible it is always advisable to use the internal key descriptions so that the metadata can take advantage of any specific NUKE systems that use it.

Writers
-------


.. _writer-base-reference:

The Writer Base Class
*********************

.. cpp:class:: DD::Image::Writer

Writing a custom writer is a homologous part of writing a Reader. In most cases when you spend the time to write a custom Reader, you probably also want to have a Writer capable of bringing your new/modified work back onto your custom file format. To achieve this, in the same way that we have developed a Reader (see :ref:`reader-base-reference`), we can develop a writer by inheriting from the *DD::Image::Writer*. This of course does not mean that you need to have a writer for your custom reader or vice-versa.

In order to have a basic *Writer*, much like the *Reader* there are a set of interfaces that need to be implemented that provide the basic functionality and description of this new writer so that it can be managed by NUKE.

The first step to achieve this is to inherit the class from our ``DD::Image::Writer`` and make sure we implement a ``DD::Image::Description`` for the Writer::

  class MyWriter : public DD::Image::Writer
  {
    static const Description d;
    ...
  };
  
  static Writer* build()
  {
    return new MyWriter();
  }
  const Writer::Description MyWriter::d( "myFile\0", "my file type", build );

This code sets up a basic writer. We have created a new writer named *MyWriter* that derives from *DD::Image::Writer*. Within this writer, we added a description. In our definition, we can then declare and define a build function. This is the factory method that actually creates the *MyWriter*  object. The description is then populated with all of this relevant information. The writer is triggered for files of extension *.myFile*. Unlike the reader, there is no test function needed since a writer should be able to write from NUKE's internal format to any custom output data format.

In the same way as the Reader, we can extend the file extensions that are associated with this writer by simply extending the file types, ensuring each element is separated by a *NULL* terminator. The writer does not suffer from misnaming problems since we can just choose another writer regardless of the extension we set for the file. The auto detect is simply for convenience.

In order to actually do any writing, there are a few more intricate interfaces that we must implement. The main one is ``DD::Image::Writer::execute()``. This function should request the data from *input0()* into the chosen file. The file name can be retrieved using ``DD::Image::Writer::filename()`` and the frame using ``DD::Image::Writer::frame()``::

  void MyWriter::execute()
  {
    // example implementation of execute
  }



This code requests the data from *Input0()* and attempts to write the data into the *my file type* format into *filename()*. Any errors can be reported on to ``DD::Image::Writer::error()``.

Other Interesting Methods 
^^^^^^^^^^^^^^^^^^^^^^^^^

This is a list of more advanced interfaces that you may want to implement for your custom writer and their description:

======================================================================================================= ===============================================================================
Function                                                                                                Description
.. cpp:function:: bool DD::Image::Writer::movie() const                                                 Flag indicating whether it is a movie or not.
.. cpp:function:: void DD::Image::Writer::finish()                                                      Should be overriden if the writer needs to perform any operations at finish.
.. cpp:function:: void DD::Image::Writer::knobs( Knob_Callback cb )                                     Should be overriden to create writer knobs.
.. cpp:function:: int  DD::Image::Writer::knob_changed( Knob* knob )                                    Should be overriden to handle knob changed events.
.. cpp:function:: LUT* DD::Image::Writer::defaultLUT() const                                            Specifies the default LUT for this writer.
.. cpp:function:: bool DD::Image::Writer::isDefaultLUTKnob( Knob* knob ) const                          Flag indicating whether the LUT is default or not.
.. cpp:function:: int  DD::Image::Writer::split_input( int i ) const                                    Flag indicating whether split inputs are present.
.. cpp:function:: const OutputContext& DD::Image::Writer::inputContext( int n, OutputContext& c ) const Lets the writer override the output context.
======================================================================================================= ===============================================================================

.. _writer-filewriter-referece:

The FileWriter Base Class
*************************

.. cpp:class:: DD::Image::FileWriter

The *FileWriter* is a writer base class that extends the current *DD::Image::Writer* functionality by providing basic file handling functionality. In any case where the basic file handling functionality is needed, this should be the class to derive for the implementation of the new writer as follows::

  class MyWriter : public DD::Image::FileWriter
  {
    static const Description d;
    ...
  };

By implementing the writer this way we are able to take advantage of the extended interface and file manipulation functions. Unless there is a compeling reason or you are using some third-party file manipulation, the FileWriter should be used as the base class.

FileWriter Extended Interface
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

================================================================================================================ ====================================================
Function                                                                                                         Description
.. cpp:function:: bool DD::Image::FileWriter::open()                                                             Opens the output file.
.. cpp:function:: bool DD::Image::FileWriter::close()                                                            Closes the output file.
.. cpp:function:: bool DD::Image::FileWriter::write( const void* p, FILE_OFFSET n )                              Writes n bytes to the file.
.. cpp:function:: bool DD::Image::FileWriter::write( FILE_OFFSET n, const void* p, FILE_OFFSET o )               Writes n bytes given the offset.
.. cpp:function:: bool DD::Image::FileWriter::seek( FILE_OFFSET o )                                              Seeks the file.
.. cpp:function:: FILE_OFFSET DD::Image::FileWriter::tell() const                                                Returns where the pointer currently is on the file.
.. cpp:function:: std::string DD::Image::FileWriter::getTempFileName( const char* pActualFileName = NULL ) const Returns the current temporary file name.
================================================================================================================ ====================================================

Writing Metadata
****************

In order to access metadata to be written into your file, the Writer itself must evaluate the metadata from an input Iop and call the corresponding *fetchMetaData()* . This data can then be used to modify or append to the file itself.

Colorspace Handling & LUTs
---------------------------

In order to effectively create Readers and Writers for use with NUKE, it is also essential to have an understanding of how NUKE handles colorspace. This section describes how NUKE handles colorspacing and LUTs in the context of Reads, Writes, and during compositing.

Viewing Data
************

While the intent of this section is not to explain the functionality associated with the Viewer and compositing, it is essential to understand NUKE's handling of colorspace. NUKE assumes that any images to be displayed in the Viewer are in a linear colorspace. This not only facilitates handling and calculations but also ensures a standard write format accross all supported file types. The Viewer itself applies a desired LUT to ensure that a preview is reliabily similar to the real output.


Reading Data
************

As mentioned previously, NUKE expects all of its formats to be read into linear space. What that means is that the user is responsible for ensuring any colorspace conversions happen at the time of reading. Once the reading is finished, NUKE assumes the colorspace is linear and applies any operations under that assumption::

  void MyReader::engine( int y, int x, int r, ChannelMask mask, Row& row )
  {
    row.range( 0, width() );
    for( int z = 0; z < 4; z++ ) {
      // A user-implemented read function
      custom_read_channel( y, z, row.writable( z ) );
    }
  }

In this example, the *custom_read_channel* when writing the channel *z* would have to also perform the appropriate calculations to ensure the data is in a linear colorspace.

Writing Data
************

Writing data as a homologous operation to a read also has to make certain assumptions. All data is assumed to be in a linear colorspace when the reader is called, and any conversions need to be made on write. The difference here is that the colorspace output can be implicit or explicit. To achieve this, there is a particular interface (``DD::Image::Writer::defaultLUT()``) that can be implemented for the *Write* that allows each write to specify a default (implicit) colorspace. It is important to note that all other registered colorspaces are also available through the NUKE UI.

::

  LUT* MyWriter::defaultLUT() const
  {
    return LUT::getLut(LUT::FLOAT);
  }

This code sets the default LUT for the *MyWriter* writer to be *FLOAT*. When a file with a supported extension is detected and the *default* colorspace setting is selected, this is the output colorspace for the write.

 

