.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/tutorials/tableloader_and_eventsources.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_tutorials_tableloader_and_eventsources.py: IO: TableLoader and EventSources ================================ This hands-on was presented in October 2023 at the DPPS Meeting at CEA Paris-Saclay (J. Hackfeld). .. GENERATED FROM PYTHON SOURCE LINES 10-59 Introduction ------------ ``ctapipe`` provides basically two different ways of accessing its data products: - event-wise (**EventSource**) - column-wise (**TableLoader**) **EventSource(s):** EventSources read input files and generate ``ctapipe.containers.ArrayEventContainer`` instances when iterated over. A new EventSource should be created for each type of event file read into ctapipe, e.g. simtel files are read by the ``ctapipe.io.SimTelEventSource`` . EventSource provides a common high-level interface for accessing event information from different data sources. Creating an EventSource for a new file format or other event source ensures that data can be accessed in a common way, regardless of the file format or data origin. EventSource itself is an abstract class, but will create an appropriate subclass if a compatible source is found for the given ``input_url`` . **TableLoader**: Loads telescope-event or subarray-event data from ctapipe HDF5 files and returns ``astropy.table`` . See `Astropy docs `__ or `this video from A.Donath at the ESCAPE Summer School 2021 `__. This class provides high-level access to data stored in ctapipe HDF5 files, such as created by the ctapipe-process tool ( ``ctapipe.tools.process.ProcessorTool`` ). There are multiple ``TableLoader`` methods loading data from all relevant tables (depending on the options) and **joins** them into single tables: - ``TableLoader.read_subarray_events`` - ``TableLoader.read_telescope_events`` - ``TableLoader.read_telescope_events_by_id`` - ``TableLoader.read_telescope_events_by_type`` The last one returns a dict with a table per telescope type, which is needed for e.g. DL1 image data that might have different shapes for each of the telescope types as tables do not support variable length columns. **It is recommended to use the** ``TableLoader`` **when loading data into Tables, because its much faster than EventSources!** .. GENERATED FROM PYTHON SOURCE LINES 61-64 Code Examples ------------- First import some classes/modules and get the example dataset path: .. GENERATED FROM PYTHON SOURCE LINES 64-73 .. code-block:: Python :lineno-start: 65 from traitlets.config import Config from ctapipe import utils from ctapipe.calib import CameraCalibrator from ctapipe.io import DataWriter, EventSource, TableLoader simtel_path = utils.get_dataset_path("gamma_prod5.simtel.zst") .. GENERATED FROM PYTHON SOURCE LINES 74-77 EventSource(s) -------------- The already implemented EventSources are: .. GENERATED FROM PYTHON SOURCE LINES 77-83 .. code-block:: Python :lineno-start: 78 sources = EventSource.non_abstract_subclasses() maxlen = max(len(source) for source in sources) for source in sources.values(): print(f"{source.__name__:>{maxlen}s} -- {source.__module__}.{source.__qualname__}") .. rst-class:: sphx-glr-script-out .. code-block:: none HDF5EventSource -- ctapipe.io.hdf5eventsource.HDF5EventSource SimTelEventSource -- ctapipe.io.simteleventsource.SimTelEventSource .. GENERATED FROM PYTHON SOURCE LINES 84-86 ``EventSource`` will create an appropriate subclass if a compatible source is found for the given ``input_url`` : .. GENERATED FROM PYTHON SOURCE LINES 86-90 .. code-block:: Python :lineno-start: 87 source = EventSource(input_url=simtel_path, max_events=5) print(source) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 91-93 You can now loop over the ``ctapipe.containers.ArrayEventContainer`` generated by the source. .. GENERATED FROM PYTHON SOURCE LINES 93-97 .. code-block:: Python :lineno-start: 94 for event in source: print(event.count) .. rst-class:: sphx-glr-script-out .. code-block:: none 0 1 2 3 4 .. GENERATED FROM PYTHON SOURCE LINES 98-101 .. code-block:: Python :lineno-start: 99 print(repr(event)) .. rst-class:: sphx-glr-script-out .. code-block:: none ctapipe.containers.ArrayEventContainer: index.*: event indexing information with default None r0.*: Raw Data with default None r1.*: R1 Calibrated Data with default None dl0.*: DL0 Data Volume Reduced Data with default None dl1.*: DL1 Calibrated image with default None dl2.*: DL2 reconstruction info with default None simulation.*: Simulated Event Information with default None with type trigger.*: central trigger information with default None count: number of events processed with default 0 monitoring.*: container for event-wise monitoring data (MON) with default None muon.*: Container for muon analysis results with default None .. GENERATED FROM PYTHON SOURCE LINES 102-106 Every time a new loop is started through the source, it tries to restart from the first event, which might not be supported by the event source. It is encouraged to use ``EventSource`` in a **context manager** to ensure the correct cleanups are performed when you are finished with the source: .. GENERATED FROM PYTHON SOURCE LINES 106-114 .. code-block:: Python :lineno-start: 107 with EventSource(input_url=simtel_path) as source: for event in source: print( f"Event Count: {event.count}," f"Tels with trigger: {event.trigger.tels_with_trigger}" ) .. rst-class:: sphx-glr-script-out .. code-block:: none Event Count: 0,Tels with trigger: [ 9 14 104] Event Count: 1,Tels with trigger: [ 26 27 29 47 53 59 69 73 121 122 124 148 162 166] Event Count: 2,Tels with trigger: [ 82 92 175 179] Event Count: 3,Tels with trigger: [ 17 26 27 29 43 47 53 57 69 112 121 122 124 125 128 140 144 148 162 166] Event Count: 4,Tels with trigger: [ 3 4 8 9 14 15 23 24 25 29 32 36 42 46 52 56 103 104 109 110 118 119 120 126 127 129 130 137 139 143 161] Event Count: 5,Tels with trigger: [ 87 98 151] Event Count: 6,Tels with trigger: [ 68 72 76 94 98 151 165 177] .. GENERATED FROM PYTHON SOURCE LINES 115-118 You can hand in the ID's of the telescopes to be included in the data with the ``allowed_tels`` attribute. If given, only this subset of telescopes will be present in the generated events. If None, all available telescopes are used. .. GENERATED FROM PYTHON SOURCE LINES 118-130 .. code-block:: Python :lineno-start: 119 eventsource_config = Config( {"EventSource": {"max_events": 5, "allowed_tels": [3, 4, 9]}} ) with EventSource(input_url=simtel_path, config=eventsource_config) as source: for event in source: print( f"Event Count: {event.count}," f"Tels with trigger: {event.trigger.tels_with_trigger}" ) .. rst-class:: sphx-glr-script-out .. code-block:: none Event Count: 0,Tels with trigger: [9] Event Count: 1,Tels with trigger: [3 4 9] .. GENERATED FROM PYTHON SOURCE LINES 131-134 If you want to calibrate your data in the event loop and write it to an .h5 file with the ``DataWriter`` : .. GENERATED FROM PYTHON SOURCE LINES 134-149 .. code-block:: Python :lineno-start: 135 source = EventSource(input_url=simtel_path, max_events=50) calibrate = CameraCalibrator(subarray=source.subarray) with DataWriter( event_source=source, output_path="events.dl1.h5", write_dl1_parameters=False, overwrite=True, write_dl1_images=True, ) as write_data: for event in source: calibrate(event) write_data(event) .. GENERATED FROM PYTHON SOURCE LINES 150-156 Alternatively doing it with ``ctapipe-process`` would look like this: .. code-block:: bash ! ctapipe-process -i {simtel_path} -o events.dl1.h5 --overwrite --progress .. GENERATED FROM PYTHON SOURCE LINES 158-162 TableLoader ----------- Create a TableLoader instance with the above created dl1 file: .. GENERATED FROM PYTHON SOURCE LINES 162-165 .. code-block:: Python :lineno-start: 163 loader = TableLoader(input_url="events.dl1.h5") .. GENERATED FROM PYTHON SOURCE LINES 166-167 Alternatively using a config file: .. GENERATED FROM PYTHON SOURCE LINES 167-178 .. code-block:: Python :lineno-start: 168 tableloader_config = Config( { "TableLoader": { "input_url": "events.dl1.h5", } } ) loader = TableLoader(config=tableloader_config) .. GENERATED FROM PYTHON SOURCE LINES 179-180 Reading subarray-based event information: .. GENERATED FROM PYTHON SOURCE LINES 180-191 .. code-block:: Python :lineno-start: 181 subarray_events = loader.read_subarray_events( start=None, stop=None, dl2=False, simulated=True, observation_info=False, ) subarray_events .. raw:: html
Table length=7
obs_idevent_idtimetels_with_triggerevent_typetrue_energytrue_alttrue_aztrue_core_xtrue_core_ytrue_h_first_inttrue_x_maxtrue_starting_grammagetrue_shower_primary_id
TeVdegdegmmmg / cm2g / cm2
int32int64Timebool[180]int64float64float64float64float64float64float64float64float64int64
140091604409937.84765False .. False320.0728797242045402569.99999967119774359.999982697156-146.42298889160156341.643585205078132396.98828125159.411758422851560.00
151011604409953.115399False .. False321.745471358299255469.99999967119774359.999982697156-737.0097045898438-266.600952148437520947.525390625286.75674438476560.00
151031604409960.710867False .. False321.745471358299255469.99999967119774359.999982697156791.504150390625914.066223144531220947.525390625286.75674438476560.00
151041604409965.387411False .. False321.745471358299255469.99999967119774359.999982697156-534.9799194335938-443.4038696289062520947.525390625286.75674438476560.00
151051604409973.646828False .. False321.745471358299255469.99999967119774359.999982697156-324.859130859375398.549102783203120947.525390625286.75674438476560.00
151081604409990.315764False .. False321.745471358299255469.99999967119774359.999982697156-1299.1031494140625217.6831207275390620947.525390625286.75674438476560.00
151091604409992.675785False .. False321.745471358299255469.99999967119774359.999982697156-994.0153198242188485.24194335937520947.525390625286.75674438476560.00


.. GENERATED FROM PYTHON SOURCE LINES 192-193 Reading subarray-based event information in chunks: .. GENERATED FROM PYTHON SOURCE LINES 193-204 .. code-block:: Python :lineno-start: 194 subarray_events_chunked = loader.read_subarray_events_chunked( chunk_size=3, dl2=False, simulated=True, observation_info=False, ) for chunk in subarray_events_chunked: print(" \n", chunk) .. rst-class:: sphx-glr-script-out .. code-block:: none Chunk(start=0, stop=3, data= obs_id event_id ... true_starting_grammage true_shower_primary_id ... g / cm2 int32 int64 ... float64 int64 ------ -------- ... ---------------------- ---------------------- 1 4009 ... 0.0 0 1 5101 ... 0.0 0 1 5103 ... 0.0 0) Chunk(start=3, stop=6, data=
obs_id event_id ... true_starting_grammage true_shower_primary_id ... g / cm2 int32 int64 ... float64 int64 ------ -------- ... ---------------------- ---------------------- 1 5104 ... 0.0 0 1 5105 ... 0.0 0 1 5108 ... 0.0 0) Chunk(start=6, stop=7, data=
obs_id event_id ... true_starting_grammage true_shower_primary_id ... g / cm2 int32 int64 ... float64 int64 ------ -------- ... ---------------------- ---------------------- 1 5109 ... 0.0 0) .. GENERATED FROM PYTHON SOURCE LINES 205-206 Reading just LST events: .. GENERATED FROM PYTHON SOURCE LINES 206-224 .. code-block:: Python :lineno-start: 207 lst_events = loader.read_telescope_events( telescopes=[1, 2, 3, 4], start=None, stop=None, dl1_images=True, dl1_parameters=False, dl1_muons=False, dl2=False, simulated=False, true_images=True, true_parameters=False, instrument=True, observation_info=False, ) lst_events .. raw:: html
Table length=2
obs_idevent_idtel_idtime_monoevent_type_monon_trigger_pixelsimagepeak_timeis_validtrue_image_sumtrue_imagenametypepos_xpos_ypos_zcamera_nameoptics_name_1tel_descriptionoptics_name_2size_typereflector_shapemirror_arean_mirrorsn_mirror_tilesequivalent_focal_lengtheffective_focal_lengthtelescope_pointing_azimuthtelescope_pointing_altitudetimetels_with_triggerevent_type
mmmm2mmradrad
int32int64int32Timeint64int64float32[1855]float32[1855]boolint32int32[1855]str5str3float32float32float32str9str5str17str5str3str20float64int64int64float64float64float32float32Timebool[180]int64
1510531604409973.64682832-1-1.5701737 .. -0.914564627.381046 .. 29.717525True940 .. 0LSTLST-19.39665.231.0LSTCamLSTLST_LST_LSTCamLSTLSTPARABOLIC386.7332458496094119828.029.305650711059570.01.22173051604409973.646828False .. False32
1510541604409973.64682832-11.0622615 .. 0.98535552.7207987 .. 37.355812True1750 .. 0LSTLST-120.0331.15133.1LSTCamLSTLST_LST_LSTCamLSTLSTPARABOLIC386.7332458496094119828.029.305650711059570.01.22173051604409973.646828False .. False32


.. GENERATED FROM PYTHON SOURCE LINES 225-226 Loading telescope events by type returns a dict with the different telescope types: .. GENERATED FROM PYTHON SOURCE LINES 226-245 .. code-block:: Python :lineno-start: 227 telescope_events_by_type = loader.read_telescope_events_by_type( telescopes=["LST_LST_LSTCam", "MST_MST_FlashCam"], start=None, stop=None, dl1_images=True, dl1_parameters=False, dl1_muons=False, dl2=False, simulated=False, true_images=True, true_parameters=False, instrument=True, observation_info=False, ) for tel_type, table in telescope_events_by_type.items(): print(f"Telescope Type: {tel_type} \n", table, "\n") .. rst-class:: sphx-glr-script-out .. code-block:: none Telescope Type: LST_LST_LSTCam obs_id event_id tel_id ... time tels_with_trigger event_type ... ------ -------- ------ ... ----------------- ----------------- ---------- 1 5105 3 ... 1604409973.646828 False .. False 32 1 5105 4 ... 1604409973.646828 False .. False 32 Telescope Type: MST_MST_FlashCam obs_id event_id tel_id ... time tels_with_trigger event_type ... ------ -------- ------ ... ----------------- ----------------- ---------- 1 4009 9 ... 1604409937.84765 False .. False 32 1 4009 14 ... 1604409937.84765 False .. False 32 1 5101 26 ... 1604409953.115399 False .. False 32 1 5101 27 ... 1604409953.115399 False .. False 32 1 5101 29 ... 1604409953.115399 False .. False 32 1 5104 17 ... 1604409965.387411 False .. False 32 1 5104 26 ... 1604409965.387411 False .. False 32 1 5104 27 ... 1604409965.387411 False .. False 32 1 5104 29 ... 1604409965.387411 False .. False 32 1 5104 125 ... 1604409965.387411 False .. False 32 1 5105 8 ... 1604409973.646828 False .. False 32 1 5105 9 ... 1604409973.646828 False .. False 32 1 5105 14 ... 1604409973.646828 False .. False 32 1 5105 15 ... 1604409973.646828 False .. False 32 1 5105 23 ... 1604409973.646828 False .. False 32 1 5105 24 ... 1604409973.646828 False .. False 32 1 5105 25 ... 1604409973.646828 False .. False 32 1 5105 29 ... 1604409973.646828 False .. False 32 1 5105 126 ... 1604409973.646828 False .. False 32 1 5105 127 ... 1604409973.646828 False .. False 32 .. GENERATED FROM PYTHON SOURCE LINES 246-247 Loading telescope events by ID returns a dict with the different telescope IDs: .. GENERATED FROM PYTHON SOURCE LINES 247-265 .. code-block:: Python :lineno-start: 248 telescope_events_by_id = loader.read_telescope_events_by_id( telescopes=[3, 14], start=None, stop=None, dl1_images=True, dl1_parameters=False, dl1_muons=False, dl2=False, simulated=False, true_images=True, true_parameters=False, instrument=True, observation_info=False, ) for tel_id, table in telescope_events_by_id.items(): print(f"Telescope ID: {tel_id} \n", table, "\n") .. rst-class:: sphx-glr-script-out .. code-block:: none Telescope ID: 3 obs_id event_id tel_id ... time tels_with_trigger event_type ... ------ -------- ------ ... ----------------- ----------------- ---------- 1 5105 3 ... 1604409973.646828 False .. False 32 Telescope ID: 14 obs_id event_id tel_id ... time tels_with_trigger event_type ... ------ -------- ------ ... ----------------- ----------------- ---------- 1 4009 14 ... 1604409937.84765 False .. False 32 1 5105 14 ... 1604409973.646828 False .. False 32 .. GENERATED FROM PYTHON SOURCE LINES 266-271 - ``read_telescope_events_chunked`` - ``read_telescope_events_by_type_chunked`` - ``read_telescope_events_by_id_chunked`` are also available. .. GENERATED FROM PYTHON SOURCE LINES 273-274 Reading e.g. simulation- or observation-information: .. GENERATED FROM PYTHON SOURCE LINES 274-280 .. code-block:: Python :lineno-start: 275 simulation_configuration = loader.read_simulation_configuration() observation_information = loader.read_observation_information() simulation_configuration .. raw:: html
Table length=1
obs_idrun_numbercorsika_versionsimtel_versionenergy_range_minenergy_range_maxprod_site_B_totalprod_site_B_declinationprod_site_B_inclinationprod_site_altspectral_indexshower_prog_startshower_prog_iddetector_prog_startdetector_prog_idn_showersshower_reusemax_altmin_altmax_azmin_azdiffusemax_viewcone_radiusmin_viewcone_radiusmax_scatter_rangemin_scatter_rangecore_pos_modeatmospherecorsika_iact_optionscorsika_low_E_modelcorsika_high_E_modelcorsika_bunchsizecorsika_wlen_mincorsika_wlen_maxcorsika_low_E_detailcorsika_high_E_detail
TeVTeVuTradradmradradradraddegdegmmnmnm
int32int32int64int64float32float32float64float64float64float64float64int64int64int64int64int64int64float32float32float32float32int64float32float32float32float32int64int64int64int64int64float64float64float64int64int64
11771015933496430.003330.022.585954666137695-0.07911577820777893-0.427642464637756352147.0-2.01604404800116044098831100101.22173051.22173057.987575e-087.987575e-0800.00.02000.00.0199187235.0240.01000.00303


.. GENERATED FROM PYTHON SOURCE LINES 281-284 Now you have ``astropy.table`` s including all the relevant data you need for your analyses. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 6.766 seconds) .. _sphx_glr_download_auto_examples_tutorials_tableloader_and_eventsources.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: tableloader_and_eventsources.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: tableloader_and_eventsources.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: tableloader_and_eventsources.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_