.. _dataformat:

***********
Data Format
***********

The ``ctapipe`` output files are using the HDF5 format files,
the content of which are described below per data level.

The following describes the contents of output files
generated by ctapipe (e.g. the ``ctapipe-process`` tool which uses the
:py:class:`~ctapipe.io.DataWriter` component to generate output data).

The tables are written with `pytables <https://www.pytables.org>`_ (not ``h5py``).

The in-memory data format is defined by the `~ctapipe.containers` structure.

To read this data, there are two high-level APIs available:

- Bulk-reading into astropy `~astropy.table.Table` objects using `~ctapipe.io.TableLoader`
- Event-wise using the `~ctapipe.io.HDF5EventSource`


DL1 Data Format
===============

DL1/Event Data format
---------------------

This describes data that change per-event.
The following datasets will be written to the group ``/dl1/event/`` in the  output file:

.. list-table::
    :header-rows: 1

    * - Group/Dataset
      - Description
      - Contents
    * - ``subarray``
      - event-wise data pertaining to a subarray
      - (group)
    * - ``subarray/trigger``
      - subarray trigger information
      - :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.TriggerContainer`
    * - ``telescope/``
      - Per-telescope Per-event information
      - (group)
    * - ``telescope/images/tel_{TEL_ID:03d}``
      - tables of telescope images (one per telescope)
      - :py:class:`~ctapipe.containers.TelEventIndexContainer`, :py:class:`~ctapipe.containers.DL1CameraContainer`
    * - ``telescope/parameters/tel_{TEL_ID:03d}``
      - tables of image parameters (one per telescope)
      - :py:class:`~ctapipe.containers.TelEventIndexContainer`, :py:class:`~ctapipe.containers.ImageParametersContainer`


DL1/Monitoring Data Format
--------------------------

This describes data that change per-telescope, but not per regular physics event.
This is used to store calibration and monitoring information, such as calibration events pixel statistics,
pedestal values, and other per-telescope monitoring and calibration information.
The following datasets will be written to the group ``/dl1/monitoring/`` in the output file:

.. list-table::
    :header-rows: 1

    * - Group/Dataset
      - Description
      - Contents
    * - ``telescope/``
      - Per-telescope monitoring information
      - (group)
    * - | ``telescope/calibration/camera/``
        | ``coefficients/tel_{TEL_ID:03d}``
      - tables of camera calibration coefficients (one per telescope)
      - (TBU)
    * - | ``telescope/calibration/camera/``
        | ``pixel_statistics/{pixel_feature}/``
        | ``tel_{TEL_ID:03d}``
      - | feature statistics for each pixel in the camera (one per telescope).
        | ``{pixel_feature}`` is defined as ``{event_type}_{feature_name}``.
      - | :py:class:`~ctapipe.containers.EventType`,
        | :py:class:`~ctapipe.containers.StatisticsContainer`,
        | :py:class:`~ctapipe.containers.ImageStatisticsContainer`,
        | :py:class:`~ctapipe.containers.IntensityStatisticsContainer`,
        | :py:class:`~ctapipe.containers.PeakTimeStatisticsContainer`

DL2 Data Format
===============

This describes data that change per-event. The following datasets will be
written to the group ``/dl2/event/subarray/<algorithm>/`` and or
``/dl2/event/telescope/<algorithm>/``, one for each reconstruction algorithm in the
output file, where ``<algorithm>`` is the identifier of the algorithm
(e.g. "HillasReconstructor"):

.. list-table::
    :widths: 25 50 25
    :header-rows: 1

    * - Group/Dataset
      - Description
      - Contents
    * - /geometry
      - shower geometry reconstruction
      - :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.ReconstructedGeometryContainer`
    * - /energy
      - shower energy reconstruction
      - :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.ReconstructedEnergyContainer`
    * - /classification
      - shower classification parameters
      - :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.ParticleClassificationContainer`


Simulation Data Format
======================

.. list-table::
    :header-rows: 1

    * - Group/Dataset
      - Description
      - Contents
    * - ``/simulation/event/subarray/shower``
      - true shower parameters from Monte-Carlo simulation
      - :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.SimulatedShowerContainer`
    * - ``/simulation/event/telescope/images/tel_{TEL_ID:03d}``
      - simulated camera images
      - :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.SimulatedCameraContainer`
    * - ``/simulation/event/telescope/parameters/tel_{TEL_ID:03d}``
      - Parameters derived form the simulated camera images
      - :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.ImageParametersContainer`
    * - ``/simulation/service/shower_distribution``
      - simulated shower distribution histograms
      - :py:class:`~ctapipe.containers.SimulatedShowerDistribution`


Configuration Data Format
=========================

The output file should also contain serializations of the instrument, observation (if
applicable), simulation (if applicable) configuration information, written to the
``/configuration`` group:

.. list-table::
    :widths: 25 50 25
    :header-rows: 1

    * - Group/Dataset
      - Description
      - Contents
    * - ``/instrument``
      - Serialized :py:class:`ctapipe.instrument.SubarrayDescription`, using :py:meth:`ctapipe.instrument.SubarrayDescription.to_hdf`
      - (group)
    * - ``/instrument/subarray/layout``
      - Subarray layout info
      - result of ``SubarrayDescription.to_table()`` output as HDF5 using ``astropy.table`` functionality
    * - ``/instrument/telescope/optics``
      - telescope optics information
      - result of ``SubarrayDescription.to_table(kind='optics')`` output as HDF5 using
        ``astropy.table`` functionality
    * - ``/instrument/telescope/camera/{CAMERA_ID}``
      - camera geometry information
      - result of ``CameraGeometry.to_table()`` output as HDF5 using ``astropy.table``
        functionality
    * - ``/simulation``
      - Monte-Carlo simulation configuration information
      - (group)
    * - ``/simulation/run``
      - Monte-Carlo simulation run information
      - :py:class:`~ctapipe.containers.SimulationConfigContainer`


Core Provenance
===============

The root group of the file shall contain all of the "CTA Core Provenance Metadata"
headers as user attributes, with the hierarchy flattened and separated by spaces
(e.g.  ``"CTA ACTIVITY NAME" = "ctapipe-process"``).