How to read data

Obviously, the data to be analyzed must come from a source. They can come from files on disk, but also from python data arrays in memory. For example, the Base can be built by the user with Zone, Instant, and finally numpy arrays. If your simulation workflow is driven by python, then you may be able to retrieve data from the CFD solver in-memory and therefore avoid disk I/O operations.

Most of the time, all that users want is to have their data loaded. They do not want to bother with data loading. They want to concentrate on the analysis. Antares Readers are made to build the database for you from files.

The Antares library is developed in a pragmatic way. It supports as many (useful) file formats as possible to attract users.

The main supported ascii and binary formats are:

  • Tecplot

  • HDF5 (AVBP, Antares, CGNS)

  • Column

  • Plot3d

  • Fieldview

  • V3d (elsA)

  • VTK

  • matlab

  • fluent

  • netcdf4

https://cerfacs.fr/antares/src/io/reader.html

Some formats can described structured or unstructured grids, with one block or multi-blocks, multi-instants, etc.

The reading of data in files is handled by instances of the class Reader.

Like the majority of the classes of antares, the class Reader also derives from a Python ordered dict. So, its instances are ordered dictionaries.

Readers behave like a dictionary. However, only a given set of keys can be used. All keys are not valid for all Readers. Please refer to the documentation to know all the keys that you can set.

We are going to look at some of them in the following.

The main features of Reader will be explained with the TECPLOT format.

You can download data at http://cerfacs.fr/antares/downloads/reader_tutorial_data.tgz

untar the archive in your working directory

tar xzf reader_tutorial_data.tgz

Reading one file

When you want to read a file with antares, then you have to create an instance of the class Reader.

Your file should respect a format you must know and give as the parameter of type string of the Reader constructor as,

Reader(< format >).

The antares library handles many formats, and the parameter must take a value among the following ones. The file formats available can be obtained dynamically using:

import antares as ant
print(ant.Reader())

In this example, the file format is the binary tecplot 112 file format, which is named ‘bin_tp’.

reader = ant.Reader('bin_tp')

Most of the time, when the Reader is created, then you have to give the key ‘filename’ as a string.

import os
reader['filename'] = os.path.join('folder_1', 'flow.dat')

When all keys have been set, use the method read(). It returns a Base object created from the data stored in the files.

base = reader.read()

print(base)
print(base[0])
print(base[0][0])
print(type(base[0][0][0]))

base.show()

Reading multiple files at once with tags

You can put the tag <zone> in the value of the key ‘filename’. All characters concerned by the tag will serve as zone names in the output Base. So in one line you can read many files. This is very practical for multiblock mesh, split domains. Of course, the filenames should be structured beforehand to ease the forthcoming data processing.

! ls folder_2

reader = ant.Reader('bin_tp')
reader['filename'] = os.path.join('folder_2', 'flow_<zone>.dat')
base = reader.read()

print(base)
print(base[0])
print(base[0][0])  # or base['0000']

You can put the tag <instant> in the value of the key ‘filename’. All characters concerned by the tag will serve as instant names in the output Base. This is very practical for unsteady flows. Of course, the filenames should be structured beforehand to ease the forthcoming data processing.

The key ‘filename’ may contain both tags <zone> and <instant>.

! ls folder_3

reader = ant.Reader('bin_tp')
reader['filename'] = os.path.join('folder_3', 'flow_<zone>_<instant>.dat')
base = reader.read()

print(base)
print(base[0])
print(base[0][0])

Tags <zone> and <instant> are available with the following file formats: + fieldview, HDF5 - antares, labs, avbp -, plot3d, tecplot, v3d, vtk, CSV, column, matlab

Reading shared data

If you need to read shared data from files, then you can use the key ‘shared’. This way, all the variables in the file will go in the shared instant. This is very often used when you need the mesh coordinates in the context of unsteady flows.

Therefore, you can not use the tag <instant> with the key ‘shared’.

! ls folder_5/*

import os
r = ant.Reader('bin_tp')
r['filename'] = os.path.join('folder_5', 'mesh', 'mesh_<zone>')
r['shared'] = True
base = r.read()

print(base)
print(base[0])
print(base[0].shared)
# remove the comment to check that there is no instant
# print(base[0][0])

Reading of shared information is also available ### directly in the reader + the mesh coordinates are put in the shared Instant for the following file formats: fluent and HDF5 - cgns

with the option ‘shared’ = True

with the option ‘shared_mesh’ = True

  • This applies to mesh coordinates for the following file formats: ensight, HDF5 - labs, tecplot binary

Reading and appending data

If you need to read data from files and append the data in an already existing Base, then you can set the Base object to the key ‘base’. This is often used to complement the mesh coordinates with the unsteady flow variables.

r = ant.Reader('bin_tp')
r['base'] = base  # add to the previous base
r['filename'] = os.path.join('folder_5', '<instant>', 'flow_<zone>.dat')
base = r.read()

print(base)
print(base[0])
print(base[0][0])

A very important point when dealing with large database is that some readers are implemented with the lazy loading pattern for the variables.

Lazy loading is a design pattern commonly used in computer programming to defer initialization of an object until the point at which it is needed. The opposite of lazy loading is eager loading.

It can contribute to efficiency in the program’s operation if properly and appropriately used.

This way, variables are only read when they are used in the python script. If a variable is not used, then it will not be read from the files. Some file formats are not compatible with this pattern. Other file formats may not be implemented with this pattern. The following file formats support the lazy loading pattern: fieldview, HDF5 - antares, avbp, cgns -, netcdf4, plot3d, tecplot binary, V3D

Some file formats store topological information: fieldview, HDF5 - avbp, cgns -, netcdf4

This information is read by antares and distributed to Boundary and Family objects.

To show an example, download the file https://cgns.github.io/CGNSFiles/Poinot/sqnz_s.hdf.cgns.gz and untar the archive in your working directory.

r = ant.Reader('hdf_cgns')
r['filename'] = os.path.join('.', 'sqnz_s.hdf.cgns')
base = r.read()

print(base)
print(base[0])
print(base[0][0])
print(base.families)
print(base[0].boundaries)

The previous examples were essentially dedicated to the Tecplot file format. In general, each reader has a different behavior. They do not all provide all options. Some can even read topological information (families, boundaries). Please refer to the documentation https://cerfacs.fr/antares/src/io/reader.html.