.. _2d-readerswriters:

Writing Image Readers & Writers
===============================

Readers and Writers are an intrinsic part of NUKE. These 2D operators allow us to overcome the problem of reading and writing to and from the ever growing existing file/image formats.

Knowing about the readers and writers architecture is therefore an important part of thirdparty NUKE development, which many times may involve custom file formats.

This section will go through the in's and out's of creating, understanding and using Readers and Writers.

Architecture
------------

Before we delve into the coding part of creating your own readers and writers it is important to understand how NUKE handles the reading and writing of files:

To identify which reader or writer version is needed, NUKE relies on parsing the file extension and finding a matching reader, so NUKE will always try to match up a file with a *.mov* extension to a *movReader.<OS specific extension>* reader.

There is still however a problem that can arise from a single format having multiple extensions, an example of this is JPG which can be both represented as *.jpg* and *.jpeg*, 
with just the previous system in place it would mean creating or duplicating the jpgReader extension to satisfy both *jpg* and *jpeg*.
NUKE provides two systems to help you get around this problem one is through a TCL script file that it tries to invoke just before it attempts to find the reader in question and can be used to redirect a reader.
This TCL script must have the same name as the source reader, in the case of *.jpg* to *.jpeg* the script would be called *jpegReader.tcl* , as for its contents it simply needs to redirect to the correct reader as following:

.. code-block:: tcl

  # jpegReader.tcl
  load jpgReader

This script is invoked instead of *jpegReader.<OS specific extension>* and ensures the correctly named *jpgReader.<OS specific extension>* is loaded for the read.

The other system, is by extending the supported extension types for the reader itself ( will be explained below ).

Readers
-------

.. _reader-base-reference:

The Reader Base Class
*********************

.. cpp:class:: DD::Image::Reader

A Reader in the context of NUKE Thirdparty development provides a base class and interfaces to which NUKE can plug into in order to read a custom piece of data from disk.
So in order to create a custom Reader the first step will be to create a new Reader which the base class is a *DD::Image::Reader*, there is a specialization of this class that can be used in its stead *DD::Image::FileRead* this class is a specialization of *DD::Image::Reader* and provides extend file management functionality, this will be explained in the next sub-section *section link*.

In order to create a custom reader using the *DD::Image::Reader* base class the derived class for the new reader needs to follow some specific steps and implement the basic interfaces.

The ``DD::Image::Reader::Description`` interface must be populated with the relevant information, this allows NUKE to identify and manage the readers and their supported extensions. This static member can be declared as follows::

  class MyReader : DD::Image::Reader 
  {
    static const Description d;
    ...
  };
  
  static Reader* build( Read* r, int fd, const unsigned char* b, int n ) 
  {
      return new MyReader( r, fd, b, n );
  }
  
  static bool test( int fd, const unsigned char* block, int n ) 
  {
    return ( block[0] == MY_MAGIC_BYTE );
  }

  const Reader::Description MyReader::d( "myFile\0", "my file format", build, test );

In this example a class *MyReader* is derived from *DD::Image::Reader*, the derived class then declares a static *Description* member this description is what will control the construction and evaluate whether or not the *MyReader* is suitable to manipulate the current file, through the declaration of *build* and *test* in this case.

The previous implementation will look up for files with the extension *.myFile*, if this specific format a test is performed to ensure the format matches up to the extension which in this case will be to identify the *MY_MAGIC_BYTE* as the first element of the buffer. In the case of a misnamed format, NUKE will still test against the tester function. It falls then on the user to ensure the testing is sufficient to determine the file type is readable and avoid collisions with similar file formats.

As well as using the TCL solution suggested in the previous section, multiple extensions can be specified on a reader's descriptions by constructing it with a *NULL* separated list for it's first argument.::

  const Reader::Description MyReader::d( "myFile\0someOtherFile", "my file format", build, test );

In this example not only will the reader be used for files with a *.myFile* extension but also for *.someOtherFile*.

The constructor should use the ``DD::Image::Reader::set_info()`` to setup the area to be accessed from the file IO. This should ( if possible ) be done by a quick read operation. Complex reading, decoding and buffering can be done via ``DD::Image::Reader::open()``. 

The last function that needs to be overriden in order to create a basic working custom reader is ``DD::Image::Reader::engine()``, this function will implement the actual reading of the file data and perform the user conversion into the NUKE native image format.::

  void MyReader::engine( int y, int x, int r, ChannelMask mask, Row& row )
  {
    row.range( 0, width() );

    for( int z = 0; z < 4; z++ ) {
      // A user implemented read function
      custom_read_channel( y, z, row.writable( z ) );
    }
  }

This code represents a basic ``DD::Image::Reader::engine()`` implementation, starts by setting the range of the row to the full width, and then reads all of the channels into the data row.

Other Interesting Methods
^^^^^^^^^^^^^^^^^^^^^^^^^

These are optiomal methods that may be implemented to provide extra functionality and information to your Reader.

============================================================================================= ==============================================================
Function                                                                                      Description
.. cpp:function:: void DD::Image::Reader::prefetchMetaData()                                  This should be overriden to have frame specific metadata
.. cpp:function:: const MetaData::Bundle& DD::Image::Reader::fetchMetaData( const char* key ) A function that can be used to fetch metadata
.. cpp:function:: bool DD::Image::Reader::supports_stereo() const                             Flag indicating whether it supports stereo or not
.. cpp:function:: bool DD::Image::Reader::fileStereo() const                                  Flag indicating whether this file is stereo or not
.. cpp:function:: bool DD::Image::Reader::videosequence() const                               Flag indicating whether or not this file is a video sequence
============================================================================================= ==============================================================

.. _reader-filereader-reference:

The FileReader Base Class
*************************

.. cpp:class:: DD::Image::FileReader

FileReader is a class derived from Reader, it performs very much like its base class and a basic *FileReader* implementation must also implement the same basic interfaces ( see :ref:`reader-base-reference` ). What makes *DD::Image::FileReader* interesting however is the added functionality it can provide, while creating your reader directly from *DD::Image::Reader* can entail having to also implement file manipulation functions ( this can ofcourse be desirable for specific systems ), the *FileReader* is a suitable solution for most implementation by providing functions to access and manipulate file data directly.

So in order to use this additional functionality we simply need to inherit from our custom reader from *DD::Image::FileReader*.::

  class MyReader : public DD::Image::FileReader
  {
    ...
  };

Now our reader can use the associated file handling helper functions associated with FileReader.


FileReader Extended Functionality Methods
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The extended methods that are provided by the *FileReader* for extra functionality.

==================================================================================================== ===============================================
Function                                                                                             Description
.. cpp:function:: int DD::Image::FileReader::lock( FILE_OFFSET offset, int min_length, int length )  Locks the data between the given parameters
.. cpp:function:: int DD::Image::FileReader::lock( FILE_OFFSET offset, int l )                       Locks the data between the given parameters
.. cpp:function:: int DD::Image::FileReader::lock( FILE_OFFSET offset, unsigned l )                  Locks the data between the given parameters
.. cpp:function:: const unsigned char& DD::Image::FileReader::byte( FILE_OFFSET n ) const            Accesses the byte at the given offset
.. cpp:function:: const unsigned char* DD::Image::FileReader::at( FILE_OFFSET n ) const              Accesses the byte at the given offset
.. cpp:function:: void DD::Image::FileReader::unlock()                                               Unlocks a previous locked buffer
.. cpp:function:: int DD::Image::FileReader::read( void* p, FILE_OFFSET offset, int min_, int max_)  Reads directly from the file into memory
.. cpp:function:: int DD::Image::FileReader::read( void* p, FILE_OFFSET offset, int l )              Reads directly from the file into memory
.. cpp:function:: int DD::Image::FileReader::read( void* p, File_OFFSET offset, unsigned l )         Reads directly from the file into memory
==================================================================================================== ===============================================


Knobs on Readers
****************

Adding Knobs to readers requires an extra step, because Readers do not behave like normal Ops there is a special object that can be initialized and registered with the Reader to give it the same capabilities as a regular op to create knobs. The two main steps to achieve this is to derive a custom object of type ``DD::Image::ReaderFormat`` the *ReaderFormat* is then capable of implementing the *knobs()* and *knob_changed()* interfaces, the second step is creating a new object of the right type within your custom reader and making sure the correct reader format is initialized.

The following code does just that ::

  class MyFileFormat : public DD::Image::ReaderFormat
  {
    // implementation of the knobs call
    void knobs(Knob_Callback c)
  }

  class MyReader : public DD::Image::FileReader
  {
    static const Description d;
  }

  static Reader* build( Read* r, int fd, const unsigned char* b, int n ) 
  {
    return new MyReader( r, fd, b, n );
  }
  
  static bool test( int fd, const unsigned char* block, int n ) 
  {
    return ( block[0] == MY_MAGIC_BYTE );
  }

  static ReaderFormat* buildformat(Read* iop)
  {
    return new MyFileFormat();
  }
  
  const Reader::Description MyReader::d( "myFile\0", "my file format", build, test, buildFormat );

The previous code now extends our reader to provide a ReaderFormat that can implement and extend the knobs collection for the *MyReader* object.

Reading MetaData
****************

Metadata is accessed using the same pattern as the decoding, the ``DD::Image::fetchMetaData( const char* key )`` can be implemented to return an object of type ``DD::Image::MetaData::Bundle``.

::

  class MyReader : public DD::Image::FileReader
  {
    ...
    DD::Image::MetaData::Bundle& fetchMetaData( const char* key )
    {
      return _metaData;
    }
    
    DD::Image::MetaData::Bundle _metaData;
  }

In this example the _metaData object is returned when requested, this meta data is already pre-built somewhere else during object/reader construction.

Testing with Tester
*******************

As it was previously mentioned ( see :ref:`reader-base-reference` ) when a file is read NUKE tries to identify which reader to use based on the registered valid extensions for each individual reader, a second mechanism also exists to safeguard against misnamed files, the ``DD::Image::Reader::(Tester*)( int fd, const unsigned char* block, int n )``::

  #define MY_MAGIC_BYTE 0x000000001
  static bool test( int fd, const unsigned char* block, int n ) 
  {
    return ( block[0] == MY_MAGIC_BYTE );
  }

The previous code is a simple test which will check the first byte of that and ensure it matches the defined *MY_MAGIC_BYTE*, this is ofcourse a very simple example of how to set it up, a truly usable implementation would check for a more complex bit pattern in order to ensure the format matches.


Working Metadata
****************

Metadata in NUKE is handled throught the ``DD::Image::MetaData::Bundle`` object, this abstract object provides all of the functionality needed to write and read data of integral types ( int, float, string... ).

There are two important things that need to be taken into account when handling MetaData, as these can be split into two categories, one is the NUKE recognized and defined metadata this is the case for time codes, where a specific key ( ``DD::Image::MetaData::TIMECODE`` ) should be used to store this value on a specific format ( you can learn more about keys *here* ). So when NUKE wants to read or write a time code this will be the key it will use.

The second type of metadata is the custom type, these will be recognized internally as simple key-value pairs and should be handled by on a case by case basis by the user.

For example if you decide to use a key named "somenamespace/timecode" instead of ``DD::Image::MetaData::TIMECODE`` to store your time codes, then you are responsible for maintaining and ensuring that data is accessed and written to the file where applicable.

Whenever possible it is always advisable to use the internal key descriptions so that the metadata can take advantage of any specific NUKE systems that use it.

Writers
-------


.. _writer-base-reference:

The Writer Base Class
*********************

.. cpp:class:: DD::Image::Writer

A writing a custom writer is an omologous part of writing a Reader, most of the time when you spend the time to write a custom Reader you will probably also want to have a Writer capable and bringing your new/modified work back onto your custom file format. To achieve this in the same way that we have developed a Reader ( see :ref:`reader-base-reference` ) we can develop a writer by inheriting from the *DD::Image::Writer* , this ofcourse does not mean that you need to have a writer for your custom reader and vice-versa.

In order to have a basic *Writer* much like the *Reader* there are a set of interfaces that need to be implemented that provide the basic functionality and description of this new writer so that it can be managed by NUKE.

The first step to achieve this is to inherit the class from our ``DD::Image::Writer`` and make sure we implement a ``DD::Image::Description`` for the Writer::

  class MyWriter : public DD::Image::Writer
  {
    static const Description d;
    ...
  };
  
  static Writer* build()
  {
    return new MyWriter();
  }
  const Writer::Description MyWriter::d( "myFile\0", "my file type", build );

This code will setup a basic writer, we have created a new writer named *MyWriter* that derives from *DD::Image::Writer* within this writer we added a description, in our definition we can then declare and define a build function, this will be the factory method that will actually create the *MyWriter*  object. The description is then populated with all of this relevant information, the writer will be triggered for files of extension *.myFile* unlike the reader there is no test function needed, since a writer should be able to write from NUKE's internal format to any custom output data format.

In the same way as the Reader, we can extend the file extensions that will be associated with this writer by simply extending the file types ensure each element is separated by a *NULL* terminator. The writer does not suffer from misnaming problems since we can just choose another writer regardless of the extension we set for the file, the auto detect is simply for convenience.

In order to actually do any writing there are a few more intricate interfaces that we must implement, the main one being ``DD::Image::Writer::execute()`` this function should request the data from *input0()* into the chosen file, the file name can be retrieved using ``DD::Image::Writer::filename()`` and the frame using ``DD::Image::Writer::frame()``::

  void MyWriter::execute()
  {
    // example implementation of execute
  }



This code will request the data from *Input0()* and attempt to write the data into the *my file type* format into *filename()*. Any errors can be reported on to ``DD::Image::Writer::error()``.

Other Interesting Methods 
^^^^^^^^^^^^^^^^^^^^^^^^^

This is a list of more advanced interfaces that you may want to implement for your custom writer and their description.

======================================================================================================= ===============================================================================
Function                                                                                                Description
.. cpp:function:: bool DD::Image::Writer::movie() const                                                 Flag indicating whether it is a movie or not
.. cpp:function:: void DD::Image::Writer::finish()                                                      Should be overriden if the writer needs to perform any operations at finish
.. cpp:function:: void DD::Image::Writer::knobs( Knob_Callback cb )                                     Should be overriden to create writer knobs
.. cpp:function:: int  DD::Image::Writer::knob_changed( Knob* knob )                                    Should be overriden to handle knob changed events
.. cpp:function:: LUT* DD::Image::Writer::defaultLUT() const                                            Specifies the default LUT for this writer
.. cpp:function:: bool DD::Image::Writer::isDefaultLUTKnob( Knob* knob ) const                          Flag indicating whether the LUT is default or not
.. cpp:function:: int  DD::Image::Writer::split_input( int i ) const                                    Flag indicating whether split inputs are present
.. cpp:function:: const OutputContext& DD::Image::Writer::inputContext( int n, OutputContext& c ) const Lets the writer override the output context
======================================================================================================= ===============================================================================

.. _writer-filewriter-referece:

The FileWriter Base Class
*************************

.. cpp:class:: DD::Image::FileWriter

The *FileWriter* is a writer base class which extends the current *DD::Image::Writer* functionality by providing basic file handling functionality, in any case where the basic file handling functionality is needed this should be the class to derive for the implementation of the new writer as follows::

  class MyWriter : public DD::Image::FileWriter
  {
    static const Description d;
    ...
  };

By implementing the writer this way we are able to take advantage of the extended interface and file manipulation functions, unless there is a compeling reason or you are using some third party file manipulation, the FileWriter should be used as the base class.

FileWriter Extended Interface
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

================================================================================================================ ====================================================
Function                                                                                                         Description
.. cpp:function:: bool DD::Image::FileWriter::open()                                                             Opens the output file
.. cpp:function:: bool DD::Image::FileWriter::close()                                                            Closes the output file
.. cpp:function:: bool DD::Image::FileWriter::write( const void* p, FILE_OFFSET n )                              Writes n bytes to the file.
.. cpp:function:: bool DD::Image::FileWriter::write( FILE_OFFSET n, const void* p, FILE_OFFSET o )               Writes n bytes given the offset.
.. cpp:function:: bool DD::Image::FileWriter::seek( FILE_OFFSET o )                                              Seek the file
.. cpp:function:: FILE_OFFSET DD::Image::FileWriter::tell() const                                                Returns where the pointer currently is on the file
.. cpp:function:: std::string DD::Image::FileWriter::getTempFileName( const char* pActualFileName = NULL ) const Returns the current temporary file name
================================================================================================================ ====================================================

Writing Metadata
****************

In order to access metadata to be written into your file, the Writer itself must evaluate the metadata from an input Iop, and call the corresponding *fetchMetaData()* , this data can then be used to modify or append to the file itself.

Colourspace Handling & LUTs
---------------------------

In order to effectively create Readers and Writers for use with NUKE, it is also essential to have an understanding of how NUKE handles colourspace, this section will describe how NUKE handles colourspacing and LUTs in the context of Reads, Writes and during compositing.

Viewing Data
************

While the intent of this section is not to explain the functionality associated with the viewer and compositing, it is essential to understanding NUKE's handling of colourspace. NUKE assumes that any images to be displayed in the viewer are linearly colourspaced, this not only facilitates handling and calculations but also ensures a standard write format accross all supported file types, the viewer itself will apply a desired LUT to ensure that a preview is reliabily similar to the real output.


Reading Data
************

As mentioned previously, NUKE expects all of its formats to be read into linear space, what that means is that the user is responsible for ensuring any colourspace conversions happen at the time of reading, once the reading is finished, NUKE will assume the colourspace is linear and will apply any operations under that assumption::

  void MyReader::engine( int y, int x, int r, ChannelMask mask, Row& row )
  {
    row.range( 0, width() );
    for( int z = 0; z < 4; z++ ) {
      // A user implemented read function
      custom_read_channel( y, z, row.writable( z ) );
    }
  }

In this example the *custom_read_channel* when writing the channel *z* would have to also perform the appropriate calculations to ensure the data is in linear colourspace.

Writing Data
************

Writting data as an omologous operation to a read also has to make certain assumptions, all data is assumed to be in linear space when the reader is called, and any conversions need to be made on write, the difference here is that the colourspace output can be implicit or explicit, to achieve this there is a particular interface ( ``DD::Image::Writer::defaultLUT()`` ) that can be implemented for the *Write* that allows each write to specify a default ( implicit ) colour space. It is important to note that all other registered colourspaces are also available through the NUKE UI.

::

  LUT* MyWriter::defaultLUT() const
  {
    return LUT::getLut(LUT::FLOAT);
  }

This code will set the default LUT for the *MyWriter* writer to be *FLOAT*, as to when a file with a supported extension is detected and the *default* colourspace setting is selected this will be the output colourspace for the write.

 

