[python-hdf5storage] 126/152: Updated documentation with regards to reading/writing structured numpy.ndarrays and fixed errors.
Ghislain Vaillant
ghisvail-guest at moszumanska.debian.org
Mon Feb 29 08:24:41 UTC 2016
This is an automated email from the git hooks/post-receive script.
ghisvail-guest pushed a commit to annotated tag 0.1
in repository python-hdf5storage.
commit cf753c774c3f232edee359e97e0a64523f74b9f2
Author: Freja Nordsiek <fnordsie at gmail.com>
Date: Fri Feb 14 18:29:14 2014 -0500
Updated documentation with regards to reading/writing structured numpy.ndarrays and fixed errors.
---
README.rst | 47 +++++++-------
doc/source/hdf5storage.Marshallers.rst | 14 ++---
doc/source/hdf5storage.utilities.rst | 7 +++
doc/source/storage_format.rst | 112 +++++++++++++++++++++------------
4 files changed, 112 insertions(+), 68 deletions(-)
diff --git a/README.rst b/README.rst
index 6d59776..2df4783 100644
--- a/README.rst
+++ b/README.rst
@@ -106,7 +106,7 @@ np.complex128 0.1 double 0.1
np.str\_ 0.1 np.uint32/16 char/uint32 0.1 [2]_
np.bytes\_ 0.1 np.bytes\_ or np.uint16 char 0.1 [3]_
np.object\_ 0.1 cell 0.1
-np.ndarray 0.1 [5]_ [5]_ 0.1 [5]_
+np.ndarray 0.1 [5]_ [6]_ [5]_ [6]_ 0.1 [5]_
np.matrix 0.1 [5]_ [5]_ 0.1 [5]_
np.chararray 0.1 [5]_ [5]_ 0.1 [5]_
============= ======= ========================== =========== ==========
@@ -128,28 +128,33 @@ np.chararray 0.1 [5]_ [5]_ 0.1 [5]_
.. [4] All keys must be ``str``.
.. [5] Container types are only supported if their underlying dtype is
supported. Data conversions are done based on its dtype.
+.. [6] Structured ``np.ndarray``s (have fields in their dtypes) can be
+ written as an HDF5 COMPOUND type or as an HDF5 Group with Datasets
+ holding its fields (either the values directly, or as an HDF5
+ Reference array to the values for the different elements of the
+ data).
This table gives the MATLAB classes that can be read from a MAT file,
the first version of this package that can read them, and the Python
type they are read as.
-============ ======= ================================
-MATLAB Class Version Python Type
-============ ======= ================================
-logical 0.1 np.bool\_
-single 0.1 np.float32 or np.complex64 [6]_
-double 0.1 np.float64 or np.complex128 [6]_
-uint8 0.1 np.uint8
-uint16 0.1 np.uint16
-uint32 0.1 np.uint32
-uint64 0.1 np.uint64
-int8 0.1 np.int8
-int16 0.1 np.int16
-int32 0.1 np.int32
-int64 0.1 np.int64
-struct 0.1 dict [7]_
-cell 0.1 np.object\_
-============ ======= ================================
-
-.. [6] Depends on whether there is a complex part or not.
-.. [7] Structure arrays are not supported.
+=============== ======= =================================
+MATLAB Class Version Python Type
+=============== ======= =================================
+logical 0.1 np.bool\_
+single 0.1 np.float32 or np.complex64 [7]_
+double 0.1 np.float64 or np.complex128 [7]_
+uint8 0.1 np.uint8
+uint16 0.1 np.uint16
+uint32 0.1 np.uint32
+uint64 0.1 np.uint64
+int8 0.1 np.int8
+int16 0.1 np.int16
+int32 0.1 np.int32
+int64 0.1 np.int64
+struct 0.1 structured np.ndarray
+cell 0.1 np.object\_
+canonical empty 0.1 ``np.float64([])``
+=============== ======= =================================
+
+.. [7] Depends on whether there is a complex part or not.
diff --git a/doc/source/hdf5storage.Marshallers.rst b/doc/source/hdf5storage.Marshallers.rst
index b52c44a..5cfb483 100644
--- a/doc/source/hdf5storage.Marshallers.rst
+++ b/doc/source/hdf5storage.Marshallers.rst
@@ -64,7 +64,7 @@ NumpyScalarArrayMarshaller
.. autoinstanceattribute:: NumpyScalarArrayMarshaller.python_attributes
:annotation: = {'Python.Type', 'Python.Shape', 'Python.Empty',
'Python.numpy.UnderlyingType',
- 'Python.numpy.Container'}
+ 'Python.numpy.Container', 'Python.numpy.Fields'}
.. autoinstanceattribute:: NumpyScalarArrayMarshaller.matlab_attributes
:annotation: = {'H5PATH', 'MATLAB_class', 'MATLAB_empty',
@@ -107,7 +107,7 @@ PythonScalarMarshaller
.. autoinstanceattribute:: PythonScalarMarshaller.python_attributes
:annotation: = {'Python.Type', 'Python.Shape', 'Python.Empty',
'Python.numpy.UnderlyingType',
- 'Python.numpy.Container'}
+ 'Python.numpy.Container', 'Python.numpy.Fields'}
.. autoinstanceattribute:: PythonScalarMarshaller.matlab_attributes
:annotation: = {'H5PATH', 'MATLAB_class', 'MATLAB_empty',
@@ -133,7 +133,7 @@ PythonStringMarshaller
.. autoinstanceattribute:: PythonStringMarshaller.python_attributes
:annotation: = {'Python.Type', 'Python.Shape', 'Python.Empty',
'Python.numpy.UnderlyingType',
- 'Python.numpy.Container'}
+ 'Python.numpy.Container', 'Python.numpy.Fields'}
.. autoinstanceattribute:: PythonStringMarshaller.matlab_attributes
:annotation: = {'H5PATH', 'MATLAB_class', 'MATLAB_empty',
@@ -159,7 +159,7 @@ PythonNoneMarshaller
.. autoinstanceattribute:: PythonNoneMarshaller.python_attributes
:annotation: = {'Python.Type', 'Python.Shape', 'Python.Empty',
'Python.numpy.UnderlyingType',
- 'Python.numpy.Container'}
+ 'Python.numpy.Container', 'Python.numpy.Fields'}
.. autoinstanceattribute:: PythonNoneMarshaller.matlab_attributes
:annotation: = {'H5PATH', 'MATLAB_class', 'MATLAB_empty',
@@ -195,7 +195,7 @@ PythonDictMarshaller
:annotation: = ['dict']
.. autoinstanceattribute:: PythonDictMarshaller.matlab_classes
- :annotation: = ['struct']
+ :annotation: = []
PythonListMarshaller
@@ -208,7 +208,7 @@ PythonListMarshaller
.. autoinstanceattribute:: PythonListMarshaller.python_attributes
:annotation: = {'Python.Type', 'Python.Shape', 'Python.Empty',
'Python.numpy.UnderlyingType',
- 'Python.numpy.Container'}
+ 'Python.numpy.Container', 'Python.numpy.Fields'}
.. autoinstanceattribute:: PythonListMarshaller.matlab_attributes
:annotation: = {'H5PATH', 'MATLAB_class', 'MATLAB_empty',
@@ -234,7 +234,7 @@ PythonTupleSetDequeMarshaller
.. autoinstanceattribute:: PythonTupleSetDequeMarshaller.python_attributes
:annotation: = {'Python.Type', 'Python.Shape', 'Python.Empty',
'Python.numpy.UnderlyingType',
- 'Python.numpy.Container'}
+ 'Python.numpy.Container', 'Python.numpy.Fields'}
.. autoinstanceattribute:: PythonTupleSetDequeMarshaller.matlab_attributes
:annotation: = {'H5PATH', 'MATLAB_class', 'MATLAB_empty',
diff --git a/doc/source/hdf5storage.utilities.rst b/doc/source/hdf5storage.utilities.rst
index 9a4ce55..91ee266 100644
--- a/doc/source/hdf5storage.utilities.rst
+++ b/doc/source/hdf5storage.utilities.rst
@@ -17,6 +17,7 @@ hdf5storage.utilities
encode_complex
get_attribute
get_attribute_string
+ get_attribute_string_array
set_attribute
set_attribute_string
set_attribute_string_array
@@ -83,6 +84,12 @@ get_attribute_string
.. autofunction:: get_attribute_string
+get_attribute_string_array
+--------------------------
+
+.. autofunction:: get_attribute_string_array
+
+
set_attribute
-------------
diff --git a/doc/source/storage_format.rst b/doc/source/storage_format.rst
index 6c1668d..6c9d0ce 100644
--- a/doc/source/storage_format.rst
+++ b/doc/source/storage_format.rst
@@ -105,7 +105,7 @@ np.chararray 0.1 np.bytes\_ or np.uint16/32 [2]_ [3]_ Dataset
as ``np.bytes_``.
.. [4] All keys must be ``str``.
.. [5] If it doesn't have any fields in its dtype or if
- :py:attr:`Options.fielded_numpy_ndarray_as_struct` is not set, it
+ :py:attr:`Options.structured_numpy_ndarray_as_struct` is not set, it
is not converted and is written as is as a Dataset. Otherwise, it
is written as a Group with its the contents of its individual
fields written as Datasets within the Group having the fields as
@@ -127,7 +127,8 @@ stored. Then, the other attributes are detailed individually.
.. note
'Python.Type', 'Python.numpy.UnderlyingType', and 'MATLAB_class' are
- all ``np.str_``. 'MATLAB_int_decode' is a ``np.int64``.
+ all ``np.bytes_``. 'MATLAB_int_decode' is a ``np.int64``.
+ 'Python.numpy.Fields' is a ``np.ndarray`` of ``np.bytes_``.
============= =================== =========================== ================== =================
Python Attributes MATLAB Attributes
@@ -178,9 +179,9 @@ np.chararray 'numpy.chararray' [8]_ 'char' [8]_
.. [7] ``2`` if it is stored as ``np.uint16`` or ``4`` if ``np.uint32``.
.. [8] The value that would be put in for a scalar of the same dtype is
used.
-.. [9] If its dtype has fields and
- :py:attr:`Options.fielded_numpy_ndarray_as_struct` is set, it is
- set to 'cell' overriding anything else.
+.. [9] If it is structured (its dtype has fields) and
+ :py:attr:`Options.structured_numpy_ndarray_as_struct` is set, it is
+ set to 'struct' overriding anything else.
Python.Shape
@@ -207,6 +208,17 @@ scalar (its type is something such as ``np.uint16``, ``np.str_``, etc.),
some form of array (its type is ``np.ndarray``), a matrix (type
is ``np.matrix``), or is a ``np.chararray`` is stored in this Attribute.
+Python.numpy.Fields
+-------------------
+
+Python Attribute
+
+``np.ndarray(dtype='bytes')``
+
+For structured ``np.ndarray`` types (and those converted to them), an
+array of the field names of the array is stored in this Attribute in
+the proper order. In the HDF5 file, they are variable length strings.
+
Python.Empty and MATLAB_empty
-----------------------------
@@ -251,6 +263,9 @@ imported, so this package does not set this Attribute at all.
Storage of Special Types
========================
+Complex Numbers
+---------------
+
Complex numbers and ``np.object_`` arrays (and things converted to them)
have to be stored in a special fashion.
@@ -267,6 +282,9 @@ automatically to ``('real', 'imag')`` when
automatically checks numeric types for many combinations of reasonably
expected field names to find complex types.
+np.object\_
+-----------
+
When storing ``np.object_`` arrays, the individual elements are stored
elsewhere and then an array of HDF5 Object References to their storage
locations is written as the data object. The elements are all written to
@@ -283,6 +301,20 @@ empty has the same format as in MATLAB and is a Dataset named 'a' of
'canonical empty' and the Attribute 'MATLAB_empty' set to
``np.uint8(1)``.
+Structure Like Data
+-------------------
+
+When storing data that is MATLAB struct like (``dict`` or structured
+``np.ndarray`` when
+:py:attr:`Options.structured_numpy_ndarray_as_struct` is set), it is
+stored as an HDF5 Group with its contents of its fields written inside
+of the Group. For single element data (``dict`` or structured
+``np.ndarray`` with only a single element), the fields are written to
+Datasets inside the Group. For multi-element data, the elements for
+each field are written in :py:attr:`Options.group_for_references` and
+an HDF5 Reference array to all of those elements is written as a Dataset
+under the field name in the Groups.
+
Optional Data Transformations
=============================
@@ -295,20 +327,20 @@ transfomations are listed below by their option name, other than
`complex_names` and `group_for_references` which were explained in the
previous section.
-=============================== ====================
-attribute value
-=============================== ====================
-delete_unused_variables ``True``
-fielded_numpy_ndarray_as_struct ``True``
-make_atleast_2d ``True``
-convert_numpy_bytes_to_utf16 ``True``
-convert_numpy_str_to_utf16 ``True``
-convert_bools_to_uint8 ``True``
-reverse_dimension_order ``True``
-store_shape_for_empty ``True``
-complex_names ``('real', 'imag')``
-group_for_references ``'/#refs#'``
-=============================== ====================
+================================== ====================
+attribute value
+================================== ====================
+delete_unused_variables ``True``
+structured_numpy_ndarray_as_struct ``True``
+make_atleast_2d ``True``
+convert_numpy_bytes_to_utf16 ``True``
+convert_numpy_str_to_utf16 ``True``
+convert_bools_to_uint8 ``True``
+reverse_dimension_order ``True``
+store_shape_for_empty ``True``
+complex_names ``('real', 'imag')``
+group_for_references ``'/#refs#'``
+================================== ====================
delete_unused_variables
@@ -321,8 +353,8 @@ Group (would end up a struct in MATLAB) that currently exist in the file
but are not in the object being stored should be deleted on the file or
not.
-fielded_numpy_ndarray_as_struct
--------------------------------
+structured_numpy_ndarray_as_struct
+----------------------------------
``bool``
@@ -386,7 +418,7 @@ needs to be set if one wants an array to end up the same shape when
imported into MATLAB. This option is necessary because Numpy and MATLAB
use opposite dimension ordering schemes, which are C and Fortan schemes
respectively. 2D arrays are stored by row in the C scheme and column in
-the Fortan scheme.
+the Fortran scheme.
store_shape_for_empty
---------------------
@@ -405,23 +437,23 @@ This table gives the MATLAB classes that can be read from a MAT file,
the first version of this package that can read them, and the Python
type they are read as if there is no Python metadata attached to them.
-============ ======= =================================
-MATLAB Class Version Python Type
-============ ======= =================================
-logical 0.1 np.bool\_
-single 0.1 np.float32 or np.complex64 [10]_
-double 0.1 np.float64 or np.complex128 [10]_
-uint8 0.1 np.uint8
-uint16 0.1 np.uint16
-uint32 0.1 np.uint32
-uint64 0.1 np.uint64
-int8 0.1 np.int8
-int16 0.1 np.int16
-int32 0.1 np.int32
-int64 0.1 np.int64
-struct 0.1 dict [11]_
-cell 0.1 np.object\_
-============ ======= =================================
+=============== ======= =================================
+MATLAB Class Version Python Type
+=============== ======= =================================
+logical 0.1 np.bool\_
+single 0.1 np.float32 or np.complex64 [10]_
+double 0.1 np.float64 or np.complex128 [10]_
+uint8 0.1 np.uint8
+uint16 0.1 np.uint16
+uint32 0.1 np.uint32
+uint64 0.1 np.uint64
+int8 0.1 np.int8
+int16 0.1 np.int16
+int32 0.1 np.int32
+int64 0.1 np.int64
+struct 0.1 structured np.ndarray
+cell 0.1 np.object\_
+canonical empty 0.1 ``np.float64([])``
+=============== ======= =================================
.. [10] Depends on whether there is a complex part or not.
-.. [11] Structure arrays are not supported.
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/debian-science/packages/python-hdf5storage.git
More information about the debian-science-commits
mailing list