.. Astrophysix documentation master file, created by sphinx-quickstart on Mon Jun 3 10:02:05 2019. This file is part of the 'astrophysix' Python package. ----------------------------------------------------------------------------------- Copyright © Commissariat a l'Energie Atomique et aux Energies Alternatives (CEA) ----------------------------------------------------------------------------------- ----------------------- FREE SOFTWARE LICENCING ----------------------- This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL: "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. ----------------------- COMMERCIAL SOFTWARE LICENCING ----------------------------- You can obtain this software from CEA under other licencing terms for commercial purposes. For this you will need to negotiate a specific contract with a legal representative of CEA. .. _usage_catalogs: Target objects and object catalogs ================================== *New in version 0.5.0* Introduction ------------ To any `Result` (:class:`~astrophysix.simdm.results.generic.GenericResult` or :class:`~astrophysix.simdm.results.snapshot.Snapshot`), you can attach catalogs of objects identified into your simulation data. The object types identified in any :class:`~astrophysix.simdm.catalogs.catalog.Catalog` are described as :class:`~astrophysix.simdm.catalogs.targobj.TargetObject` : * Each :class:`~astrophysix.simdm.catalogs.targobj.TargetObject` is qualified with a set of properties (:class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` instances) that are optionally grouped in property groups (:class:`~astrophysix.simdm.catalogs.targobj.ObjectPropertyGroup` instances) , * a :class:`~astrophysix.simdm.catalogs.catalog.Catalog` gather a list of fields (:class:`~astrophysix.simdm.catalogs.field.CatalogField` objects), each one referring to an :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` of the associated :class:`~astrophysix.simdm.catalogs.targobj.TargetObject`. .. image:: ../_static/img/catalog_inheritance.png :align: center A strict binding is enforced between a :class:`~astrophysix.simdm.catalogs.catalog.Catalog`'s :class:`~astrophysix.simdm.catalogs.field.CatalogField` and its :class:`~astrophysix.simdm.catalogs.targobj.TargetObject`'s :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty`, see :ref:`strict_protocol_cat_bindings` for more details. The content of a :class:`~astrophysix.simdm.catalogs.catalog.Catalog` can be represented as a table where : * the **rows** correspond to the collection of :class:`~astrophysix.simdm.catalogs.targobj.TargetObject` instances, * the **columns** correspond to the different :class:`~astrophysix.simdm.catalogs.field.CatalogField` objects. As a consequence, a :class:`~astrophysix.simdm.catalogs.catalog.Catalog` can be quite naturally converted into a `pandas.DataFrame`_, using the :meth:`Catalog.to_pandas() ` method. .. _pandas.DataFrame: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html Example study structure ^^^^^^^^^^^^^^^^^^^^^^^ .. image:: ../_static/img/simdm_hierarchy_catalog.png Target object definition ------------------------ Target objects define the type of items you will be able to characterize into a :class:`~astrophysix.simdm.catalogs.catalog.Catalog`. To initialize a :class:`~astrophysix.simdm.catalogs.targobj.TargetObject`, you only need to set its :attr:`~astrophysix.simdm.catalogs.targobj.TargetObject.name` property. Here is a more complete example of a :class:`~astrophysix.simdm.targobj.TargetObject` initialization with all optional attributes : .. code-block:: python >>> from astrophysix.simdm.catalogs import TargetObject >>> cluster = TargetObject(name="Galaxy cluster", description="My cluster object full description") .. seealso:: - :class:`~astrophysix.simdm.catalogs.targobj.TargetObject` API reference. Target object properties ^^^^^^^^^^^^^^^^^^^^^^^^ You can insert :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` into a :class:`~astrophysix.simdm.catalogs.targobj.TargetObject` using its :class:`~astrophysix.simdm.catalogs.targobj.TargetObject.object_properties` attribute : .. code-block:: python >>> from astrophysix.simdm.catalogs import ObjectProperty >>> # One-liner >>> pos_x = cluster.object_properties.add(ObjectProperty(property_name="pos_x", ... description="x-axis position of the cluster center of mass")) # Or >>> pos_x = ObjectProperty(property_name="pos_x", ... description="x-axis position of the cluster center of mass") >>> cluster.object_properties.add(pos_x) To initialize an :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty`, only the :attr:`ObjectProperty.property_name ` property must be set : .. code-block:: python >>> # Object properties should be initialised with at least a 'property_name' attribute. >>> unknown = ObjectProperty() AttributeError : ObjectProperty 'property_name' attribute is not defined (mandatory). Other optional :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` attributes are : .. code-block:: python >>> from astrophysix.simdm.catalogs import PropertyFilterFlag, PropertySortFlag >>> from astrophysix.simdm.utils import DataType >>> from astrophysix import units as U >>> >>> mass = ObjectProperty(property_name="$M_{500}$", description="Cluster mass", ... filter_flag=PropertyFilterFlag.BASIC_FILTER, ... sort_flag=PropertySortFlag.BASIC_SORT, ... dtype=DataType.REAL, unit=1.0e9*U.Msun) Available property filter flag enum values are : * :attr:`PropertyFilterFlag.NO_FILTER `, * :attr:`PropertyFilterFlag.BASIC_FILTER `, * :attr:`PropertyFilterFlag.ADVANCED_FILTER ` and available property sort flag enum values are : * :attr:`PropertySortFlag.NO_SORT `, * :attr:`PropertySortFlag.BASIC_SORT `, * :attr:`PropertySortFlag.ADVANCED_SORT ` .. warning:: Only :class:`CatalogFields ` associated to :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` instances with : * :attr:`~astrophysix.simdm.catalogs.targobj.ObjectProperty.filter_flag` set to :attr:`PropertyFilterFlag.BASIC_FILTER ` or :attr:`PropertyFilterFlag.ADVANCED_FILTER ` will appear as *filter fields* in the online :galactica:`Galactica object catalog search form <>`. * :attr:`~astrophysix.simdm.catalogs.targobj.ObjectProperty.sort_flag` set to :attr:`PropertyFilterFlag.BASIC_SORT ` or :attr:`PropertyFilterFlag.ADVANCED_SORT ` will appear as *sortable fields* in the online :galactica:`Galactica object catalog search form <>`. .. seealso:: * :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` API reference, * :class:`~astrophysix.simdm.catalogs.targobj.PropertyFilterFlag`, :class:`~astrophysix.simdm.catalogs.targobj.PropertySortFlag` and :class:`~astrophysix.simdm.utils.DataType` API references. Object property groups ^^^^^^^^^^^^^^^^^^^^^^ Optionally, you can group :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` instances in groups for better clarity in your workflow or enhanced display on the :galactica:`Galactica web application <>`. You can insert :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` objects into a :class:`~astrophysix.simdm.catalogs.targobj.ObjectPropertyGroup` with the :class:`ObjectPropertyGroup.group_properties ` attribute and include the :class:`~astrophysix.simdm.catalogs.targobj.ObjectPropertyGroup` into your :class:`~astrophysix.simdm.catalogs.targobj.TargetObject` using its :class:`TargetObject.property_groups ` attribute : .. code-block:: python >>> from astrophysix.simdm.catalogs import ObjectPropertyGroup >>> >>> # Additional object properties (positions along y/z axes) >>> pos_y = cluster.object_properties.add(ObjectProperty(property_name="pos_y")) >>> pos_z = cluster.object_properties.add(ObjectProperty(property_name="pos_z")) >>> >>> # Velocity properties along x/y/z axes >>> v_x = cluster.object_properties.add(ObjectProperty(property_name="v_x")) >>> v_y = cluster.object_properties.add(ObjectProperty(property_name="v_y")) >>> v_z = cluster.object_properties.add(ObjectProperty(property_name="v_z")) >>> >>> # Position property group >>> pos = ObjectPropertyGroup(group_name="position", description="Cluster position") >>> pos.group_properties.add(pos_x) >>> pos.group_properties.add(pos_y) >>> pos.group_properties.add(pos_z) >>> cluster.property_groups.add(pos) >>> >>> # Veloicity property group >>> vel = ObjectPropertyGroup(group_name="velocity", description="Cluster velocity") >>> vel.group_properties.add(v_x) >>> vel.group_properties.add(v_y) >>> vel.group_properties.add(v_z) >>> cluster.property_groups.add(vel) To initialize an :class:`~astrophysix.simdm.catalogs.targobj.ObjectPropertyGroup`, only the :attr:`ObjectProperty.property_name ` property must be set : .. code-block:: python >>> # Object property groups should be initialised with at least a 'group_name' attribute. >>> unknown = ObjectPropertyGroup() AttributeError : ObjectPropertyGroup 'group_name' attribute is not defined (mandatory). .. seealso:: :class:`~astrophysix.simdm.catalogs.targobj.ObjectPropertyGroup` API reference. Catalog construction -------------------- To define a :class:`~astrophysix.simdm.catalogs.catalog.Catalog`, only two attributes are mandatory : - **name** : a non-empty string value, - **target_object** : a :class:`~astrophysix.simdm.catalogs.targobj.TargetObject` instance. Here is a more complete example of a :class:`~astrophysix.simdm.catalogs.catalog.Catalog` initialization with all optional attributes : .. code-block:: python >>> from astrophysix.simdm.catalogs import TargetObject, Catalog >>> >>> cat = Catalog(target_object=cluster, name="Galaxy cluster catalog at $Z \simeq 2$", ... description="This is the catalog of all galaxy clusters identified in the " ... "simulation at redshift $Z\simeq 2$") Setting catalog fields ^^^^^^^^^^^^^^^^^^^^^^ Once the :class:`~astrophysix.simdm.catalogs.catalog.Catalog` has been created, you can insert :class:`~astrophysix.simdm.catalogs.field.CatalogField` objects in it, using the :attr:`Catalog.catalog_fields ` property. To define a :class:`~astrophysix.simdm.catalogs.field.CatalogField`, you must set : - **obj_prop** : catalog's :class:`~astrophysix.simdm.catalogs.targobj.TargetObject`'s :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` instance quantified in the field, - **values** : 1D :class:`numpy.ndarray` (numeric type) containing the values of the field for each object. .. code-block:: python >>> from astrophysix.simdm.catalogs import CatalogField >>> >>> cat.catalog_fields.add(CatalogField(obj_prop=pos_x, values=N.random.uniform(size=200)) >>> cat.catalog_fields.add(CatalogField(obj_prop=pos_y, values=N.random.uniform(size=200)) >>> cat.catalog_fields.add(CatalogField(obj_prop=pos_z, values=N.random.uniform(size=200)) >>> cat.catalog_fields.add(CatalogField(obj_prop=mass, values=N.array([...])) .. warning:: Once the first :class:`~astrophysix.simdm.catalogs.field.CatalogField` is inserted into a :class:`~astrophysix.simdm.catalogs.catalog.Catalog`, it sets the number of objects contained in the :class:`~astrophysix.simdm.catalogs.catalog.Catalog` (*values* 1D ``numpy.ndarray`` size). Every subsequent :class:`~astrophysix.simdm.catalogs.field.CatalogField` addition into that :class:`~astrophysix.simdm.catalogs.catalog.Catalog` **MUST** contain the same number of objects : .. code-block:: python >>> cat.catalog_fields.add(CatalogField(obj_prop=vel_x, values=N.random.uniform(size=300)) AttributeError: Cannot add a catalog field with 300 item values in a catalog containing 200 items. .. seealso:: * :class:`~astrophysix.simdm.catalogs.catalog.Catalog` and :class:`~astrophysix.simdm.catalogs.field.CatalogField` API references, * :ref:`catalog_size_modification` .. _strict_protocol_cat_bindings: Strict target object/catalog bindings ------------------------------------- When you manipulate :class:`~astrophysix.simdm.catalogs.targobj.TargetObject` and their :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` and :class:`~astrophysix.simdm.catalogs.targobj.ObjectPropertyGroup`, and you link them to a :class:`~astrophysix.simdm.catalogs.catalog.Catalog` and its list of :class:`~astrophysix.simdm.catalogs.field.CatalogField` objects), ``astrophysix`` makes sure for you that your entire study hierarchical structure remains consistent at all times : - upon object creation, - upon object addition into another object, - upon object deletion from another object. Upon object creation ^^^^^^^^^^^^^^^^^^^^ You are free to create any kind of ``astrophysix`` object, even those `linked` to another object : .. code-block:: python >>> from astrophysix.simdm.catalogs import ObjectProperty >>> from astrophysix.simdm.catalogs import CatalogField >>> >>> posy = ObjectProperty(property_name="pos_y") >>> field_posy = CatalogField(obj_prop=posy, values=N.array([0.5, 0.6, 0.48, 0.65, 0.49])) There is no risk of breaking your study hierarchical structure consistency. Upon object addition ^^^^^^^^^^^^^^^^^^^^ To avoid `dangling` references into the study hierarchical structure, ``astrophysix`` will not let you : * add a :class:`~astrophysix.simdm.catalogs.field.CatalogField` object into the :attr:`~astrophysix.simdm.catalogs.catalog.Catalog.catalog_fields` list of a :class:`~astrophysix.simdm.catalogs.catalog.Catalog` if its associated :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` does not **already** belong to the :class:`~astrophysix.simdm.catalogs.catalog.Catalog`'s :class:`~astrophysix.simdm.catalogs.targobj.TargetObject`'s :attr:`~astrophysix.simdm.catalogs.targobj.TargetObject.object_properties` list, * add an :class:`~astrophysix.simdm.catalogs.targobj.ObjectPropertyGroup` into the :attr:`~astrophysix.simdm.catalogs.targobj.TargetObject.property_groups` list of a :class:`~astrophysix.simdm.catalogs.targobj.TargetObject` if all the :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` it contains does not **already** belong to the :class:`~astrophysix.simdm.catalogs.targobj.TargetObject`'s :attr:`~astrophysix.simdm.catalogs.targobj.TargetObject.object_properties` list. * add a :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` object into a :class:`~astrophysix.simdm.catalogs.targobj.ObjectPropertyGroup` contained in the :attr:`~astrophysix.simdm.catalogs.targobj.TargetObject.property_groups` list of a :class:`~astrophysix.simdm.catalogs.targobj.TargetObject` if the :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` does not **already** belong to the :class:`~astrophysix.simdm.catalogs.targobj.TargetObject`'s :attr:`~astrophysix.simdm.catalogs.targobj.TargetObject.object_properties` list. Here are the examples : .. code-block:: python >>> from astrophysix.simdm.catalogs import TargetObject, ObjectProperty, CatalogField, \ Catalog >>> # -------------------------------------------------------------------------------- # >>> tobj = TargetObject(name="Spiral galaxy") >>> x = tobj.object_properties.add(ObjectProperty(property_name="pos_x")) >>> y = tobj.object_properties.add(ObjectProperty(property_name="pos_y")) >>> z = ObjectProperty(property_name="pos_z") # z not added in target object >>> cat = Catalog(target_object=tobj, name="Spiral galaxy catalog") >>> # Tries to add a catalog field with z property => error >>> cat.catalog_fields.add(CatalogField(z, values=N.random.uniform(size=5))) # => Error AttributeError: Cannot add catalog field. 'pos_z' target object property is not a property of 'Spiral galaxy' target object. >>> >>> # Add first the 'pos_z' property into the 'Spiral galaxy' target object, >>> tobj.object_properties.add(z) >>> # THEN add the catalog field into the catalog >>> cat.catalog_fields.add(CatalogField(z, values=N.random.uniform(size=5))) # => Ok >>> # -------------------------------------------------------------------------------- # >>> >>> # -------------------------------------------------------------------------------- # >>> alpha = ObjectProperty(property_name='alpha') >>> delta = ObjectProperty(property_name='delta') >>> beta = ObjectProperty(property_name='beta') >>> g = ObjectPropertyGroup(group_name="Group1") >>> # Tries to add properties to a group and then insert the group before the property >>> # have been added to the TargetObject property list => raises an error >>> g.group_properties.add(alpha) >>> g.group_properties.add(delta) >>> tobj.property_groups.add(g) AttributeError: 'alpha' target object property does not belong to this TargetObject object property list. >>> # Add the properties in the TargetObject first >>> tobj.object_properties.add(alpha) >>> tobj.object_properties.add(delta) >>> # THEN add the group >>> tobj.property_groups.add(g) # => Ok >>> # -------------------------------------------------------------------------------- # >>> >>> # -------------------------------------------------------------------------------- # >>> # Tries to add a property to an already registered group in the TargetObject >>> # property group list while the property has not added yet to the TargetObject >>> # property list => raises an error >>> g.group_properties.add(beta) AttributeError: 'beta' target object property does not belong to this TargetObject object property list. >>> # Add the property in the TargetObject first >>> tobj.object_properties.add(gamma) >>> # THEN insert it into the group >>> g.group_properties.add(gamma) # Ok >>> # -------------------------------------------------------------------------------- # Upon object deletion ^^^^^^^^^^^^^^^^^^^^ To avoid missing references into the study hierarchical structure, ``astrophysix`` will also prevent you from deleting an :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` from a :attr:`TargetObject.object_properties ` list if : - the :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` is included in one of the :class:`~astrophysix.simdm.catalogs.targobj.ObjectPropertyGroup` of the :attr:`TargetObject.property_groups `, - any :class:`~astrophysix.simdm.catalogs.catalog.Catalog` associated to that :class:`~astrophysix.simdm.catalogs.targobj.TargetObject` contains a :class:`~astrophysix.simdm.catalogs.field.CatalogField` that refers to the :class:`~astrophysix.simdm.catalogs.targobj.ObjectProperty` to be deleted. Here are the examples : .. code-block:: python >>> from astrophysix.simdm.protocol import SimulationCode, InputParameter, Algorithm, \ PhysicalProcess, AlgoType, Physics >>> from astrophysix.simdm.experiment import Simulation, ParameterSetting, \ AppliedAlgorithm, ResolvedPhysicalProcess >>> tobj = TargetObject(name="Pre-stellar core") >>> x = tobj.object_properties.add(ObjectProperty(property_name="pos_x")) >>> y = tobj.object_properties.add(ObjectProperty(property_name="pos_y")) >>> z = tobj.object_properties.add(ObjectProperty(property_name="pos_z")) >>> pos = tobj.property_groups.add(ObjectPropertyGroup(group_name="position")) >>> pos.group_properties.add(x) >>> pos.group_properties.add(y) >>> pos.group_properties.add(z) >>> >>> # ---------------------------------------------------------------------------------- # >>> # Tries to delete x, while it is in 'position' group >>> del tobj.object_properties[x.property_name] AttributeError: ''pos_x' target object property' cannot be deleted, the following items depend on it (try to delete them first) : ['Pre-stellar core' target object - 'position' property group - 'pos_x' target object property]. >>> >>> # Delete property from 'position' group first >>> del pos.group_properties[x.property_name] >>> # THEN delete 'pos_x' property from TargetObject => Ok >>> del tobj.object_properties[x.property_name] >>> >>> # ---------------------------------------------------------------------------------- # >>> cat = Catalog(target_object=tobj, name="Core catalog") >>> fy = cat.catalog_fields.add(CatalogField(y, values=N.random.uniform(size=10))) >>> # Tries to delete 'pos_y' property, while it is linked to fy catalog field >>> del tobj.object_properties[y.property_name] AttributeError: ''pos_y' target object property' cannot be deleted, the following items depend on it (try to delete them first) : ['Core catalog' catalog 'pos_y' catalog field]. >>> >>> # Delete CatalogField from Catalog first >>> del cat.catalog_fields[y.property_name] # THEN delete Property from TargetObject => Ok del tobj.object_properties[y.property_name]