#
# stub function definition file for docstring parsing
#
[docs]def importasdm(asdm, vis='', createmms=False, separationaxis='auto', numsubms='auto', corr_mode='all', srt='all', time_sampling='all', ocorr_mode='ca', compression=False, lazy=False, asis='', wvr_corrected_data='no', scans='', ignore_time=False, process_syspower=True, process_caldevice=True, process_pointing=True, process_flags=True, tbuff=0.0, applyflags=False, savecmds=False, outfile='', flagbackup=True, verbose=False, overwrite=False, showversion=False, useversion='v3', bdfflags=False, with_pointing_correction=False, convert_ephem2geo=True, polyephem_tabtimestep=0.):
r"""
Convert an ALMA Science Data Model observation into a CASA visibility file (MS)
[`Description`_] [`Examples`_] [`Development`_] [`Details`_]
Parameters
- asdm_ (string) - Name of input asdm directory (on disk)
- vis_ (string='') - Root name of the ms to be created. Note the .ms is NOT added
- createmms_ (bool=False) - Create a Multi-MS output
.. raw:: html
<details><summary><i> createmms = True </i></summary>
- separationaxis_ (string='auto') - Axis to do parallelization across (scan, spw, baseline, auto)
- numsubms_ ({string, int}='auto') - The number of SubMSs to create (auto or any number)
.. raw:: html
</details>
- corr_mode_ (string='all') - Specifies the correlation mode to be considered on input. A quoted string containing a sequence of ao, co, ac,or all separated by whitespaces is expected
- srt_ (string='all') - Specifies the spectral resolution type to be considered on input. A quoted string containing a sequence of fr, ca, bw, or all separated by whitespaces is expected
- time_sampling_ (string='all') - Specifies the time sampling (INTEGRATION and/or SUBINTEGRATION) to be considered on input. A quoted string containing a sequence of i, si, or all separated by whitespaces is expected
- ocorr_mode_ (string='ca') - Output data for correlation mode AUTO_ONLY (ao) or CROSS_ONLY (co) or CROSS_AND_AUTO (ca)
- compression_ (bool=False) - Flag for turning on data compression
- lazy_ (bool=False) - Make the MS DATA column read the ASDM Binary data directly (faster import, smaller MS)
- asis_ (string='') - Creates verbatim copies of the ASDMtables in the ouput measurement set. Value given must be a string of table names separated by spaces; A * wildcard is allowed.
- wvr_corrected_data_ (string='no') - Specifies which values are considerd in the SDM binary data to fill the DATA column in the MAIN table of the MS; yes for corrected, no for uncorrected, both for corrected and uncorrected (resulting in two MSs)
- scans_ (string='') - Processes only the specified scans. A scan specification consists in an exec bock index followed by the : character, followed by a comma separated list of scan indexes or scan index ranges. (e.g. 0:1;1:2~6,8;2:,3:24~30)
- ignore_time_ (bool=False) - All the rows of the tables Feed, History, Pointing, Source, SysCal, CalDevice, SysPower, and Weather are processed independently of the time range of the selected exec block / scan.
- process_syspower_ (bool=True) - Process the SysPower table?
- process_caldevice_ (bool=True) - Process the CalDevice table?
- process_pointing_ (bool=True) - Process the Pointing table?
- process_flags_ (bool=True) - Create online flags in the FLAG_CMD sub-table?
.. raw:: html
<details><summary><i> process_flags = True </i></summary>
- tbuff_ (double=0.0) - Time padding buffer (seconds)
- applyflags_ (bool=False) - Apply the flags to the MS.
- savecmds_ (bool=False) - Save flag commands to an ASCII file
- outfile_ ({string, stringArray}='') - Name of ASCII file to save flag commands
.. raw:: html
</details>
- flagbackup_ (bool=True) - Back up flag column before applying flags.
- verbose_ (bool=False) - Output lots of information while the filler is working
- overwrite_ (bool=False) - Over write an existing MS(s)
- showversion_ (bool=False) - Report the version of asdm2MS being used
- useversion_ (string='v3') - Version of asdm2MS to be used (v3 default, should work for all data)
- bdfflags_ (bool=False) - Set the MS FLAG column according to the ASDM _binary_ flags
- with_pointing_correction_ (bool=False) - Add (ASDM::Pointing::encoder - ASDM::Pointing::pointingDirection) to the value to be written in MS::Pointing::direction
- convert_ephem2geo_ (bool=True) - if True, convert any attached ephemerides to the GEO reference frame (time-spacing not changed)
- polyephem_tabtimestep_ (double=0.) - Timestep (days) for the tabulation of polynomial ephemerides. A value <= 0 disables tabulation.
.. _Description:
Description
.. note:: NOTE: **importevla** has now been deprecated, and
**importasdm** combined with **flagdata** should be used to
import JVLA data, as explained in a dedicated section below.
The **importasdm** task will fill SDM1.2 and SDM1.3 format data
into a CASA visibility data set (MS). The importasdm task supports
all changes in the ALMA ASDM made since the previous CASA release
(an up-to-date description of the most recent tables of the SDM
can be found
`here <../../notebooks/casa-fundamentals.ipynb#Science-Data-Model>`__).
ALMA data were recorded in SDM1.2 format from October 2009 until
May 2011. Since May 2011, ALMA is using the SDM1.3 format. In
particular, all science data from Cycle 0 onward will be in
SDM1.3. The JVLA started using SDM1.2 in October 2009, but
currently also uses SDM1.3. The **importasdm** task can read all
of the above formats. For the default parameter settings for
**importasdm**, see the Examples tab.
The basic modes of importing data utilizing **importasdm** can be
set by using various parameters. For example,
- If the parameter *scans* is set, then **importasdm** processes
only the scans specified in the option’s value. This value is a
semicolon-separated list of scan specifications. A scan
specification consists in an execution (exec) block index
followed by the character ’:’ followed by a comma-separated
list of scan indices or scan index ranges. A scan index is
relative to the exec block it belongs to. Scan indices are
1-based while exec blocks are 0-based. The expressions:
- "0:1"
- "2:2~6"
- "0:1;1:2~6,8;2:,3:24~30"
- "1,2"
- "3:"
- are all valid values for the selection. The "3:" selector will
be interpreted as ’all the scans of the exec block 3’. A scan
index or a scan index range not preceded by an exec block index
will be interpreted as ’all the scans with such indexes in all
the exec blocks’. By default, all the scans are considered.
- When *process_flags=True*, the task will create online flags
based on the Flag.xml, Antenna.xml and SpectralWindow.xml files
and copy them to the *FLAG_CMD* sub-table of the MS. The flags
will NOT be applied unless the parameter *applyflags* is set to
True. Optionally, the flags can also be saved to an external
ASCII file if *savecmds* is set to True. The flags can later be
applied to the MS using task **flagdata** in list mode.
- When *bdfflags=True*, the task will apply online flags
contained in the ASDM BDF data by calling the executable
bdflags2MS which the user can also do from the OS prompt.
Setting *bdfflags=True* is recommended for ALMA data.
- When *createmms=True*, the task will create a Multi-MS
partitioned according to the given separation axis. For more
detailed documentation on partition, Multi-MS, and the MPI use
in CASA, please see the global task list pages describing
**partition** and **mstransform**. `The
Multi-MS <../../notebooks/parallel-processing.ipynb#The-Multi-MS>`__ also
contains more information on Multi-MS creation.
- When setting the values for *wvr_corrected_data*, the task will
read the SDM binary data and fill the DATA column in the MAIN
table of the MS for those ALMA data that have either corrected
data, uncorrected data or both. Expected values for this option
are 'no' for the uncorrected data (this is the default), 'yes'
for the corrected data, and 'both' for corrected and
uncorrected data. In the latter case, two MeasurementSets are
created, one containing the uncorrected data and the other
containing the corrected data; the name of the corrected
MS will be given a suffix of '-wvr-corrected'. See the relevent
documentation on the ALMA Science Portal regarding those ALMA
cycles which provided both corrected and uncorrected data
streams.
.. rubric:: Import of JVLA data with importasdm
As of CASA 5.4, the task importevla is no longer available to
import JVLA data. The functionality is replaced by importasdm,
which is also being used by the VLA pipeline. However, several
additional steps are required to duplicate the behaviour of
importevla when using importasdm, involving a difference in
default parameters and the fact that some of the on-the-go
flagging cannot be performed by importasdm.
To mimic the behaviour of importevla, change the following
parameters in **importasdm** from their default settings:
- *ocorr_mode = 'co'* to import cross-correlations only
(discarding auto-correlations) *
*
- *with_pointing_correction = True* to add pointing
corrections *
*
- *process_flags = True* (default) to read in the online flags,
then *applyflags = True* to apply the online flags and/or
*savecmd = True* to save flag commands to an ascii table **.**
- For ephemeris objects: convert_ephem2geo = False
While online flags can thus be created by leaving the parameter
*process_flags = True* by default, additional flagging steps need
to be performed after **importasdm** to flag zero values and
shadowing of antennas:
- **Shadow flags:** use task **flagdata**, with *mode = 'shadow'*
(and optionally *reason = 'shadow'*). The parameters
*tolerance* and *addantenna* can be specified in flagdata in
the same way they were used in importevla. *
*
- **Zero clipping flags:** use task **flagdata**, with *mode =
'clip',* *correlation = 'ABS_ALL',* and *clipzeros = True* (and
optionally *reason = 'clip'*) *.* Note that the non-default
case in importevla where *flagpol = False c* an be replicated
by setting *correlation="ABS_RR, ABS_LL".*
Like **importasdm**, the task **flagdata** can also save the
flagging commands to an ascii table by setting *savepars = True.*
To NOT apply the flags (*applyflags=False* in importevla) add
*action='calculate'* to flagdata. You may also chose to add a
reason using the cmdreason argument, e.g.
*cmdreason="CLIP_ZERO_ALL".*
.. warning:: **WARNING**: The task **flagdata** can only write out the
flag commands for that invocation of flagdata. The default
*overwrite=True* must be used to overwrite an existing file. In
order to save the commands from all 3 possible flagging steps
(importasdm, zero, and shadow) each step must be saved to a
separate file, which must then be concatenated into a single
file to be used to flag the data.
.. rubric:: Import of ASDM data with option *lazy=True*
For the parameter *lazy*, if the default value False is chosen,
**importasdm** will fill the visibilities into a newly created
DATA column (FLOAT_DATA for total power data) of the MS converting
them from their binary format in the ASDM to the CASA Table
format. If *lazy* is set to True, the task will create the
DATA/FLOAT_DATA column with an ASDM-specific storage manager, the
(asdmstman), which enables CASA to directly read the binary data
from the ASDM with on-the-fly conversion. No redundant copy of the
raw data is created.
This procedure has the advantage that it saves more than 60% disk
space and at least in some cases makes the access to the DATA
column ≥ 10% faster because the data I/O volume is decreased. For
the same reason, it also accelerates the import itself by ca. a
factor 2. The acceleration is particularly large in the
**applycal** task and here particularly on standard SATA disks.
E.g., if your ASDM has a size of 36 GB, the import with default
parameters will turn this into an MS of 73 GB size (total disk
space consumption = 36 GB + 73 GB = 109 GB). With *lazy=True*, the
imported MS has a size of only 2 GB (total disk space consumption
= 36 GB + 2 GB = 38 GB). I.e. your total disk space savings are
ca. 65%. Even when you compare to the case where you delete the
ASDM after normal import, the solution with lazy import and
keeping the ASDM will save you ca. 48% disk space (in the example
above 38 GB compared to 73 GB). The only caveats are the
following:
#. You must not delete your ASDM. You can, however, move it but
you have to update the reference stored in the MS. Symbolic
links will work. See below on how to use the tool method
**ms.asdmref** to manipulate the ASDM reference.
#. The lazily imported DATA/FLOAT_DATA column is read-only. But in
any normal data reduction, the DATA/FLOAT_DATA column (as
opposed to CORRECTED DATA) is treated as read-only anyway.
The lazily imported MS is numerically identical with the
traditionally imported MS and so are all results derived from the
MSs. The setting *lazy=True* might be made the default setting in
future CASA releases. An important additional tool to manipulate
lazily imported MSs is the method **ms.asdmref** in the MS tool.
If the MS is imported from an ASDM with option *lazy=True*, the
DATA/FLOAT_DATA column of the MS is virtual and directly reads the
visibilities from the ASDM. A reference to the original ASDM is
stored with the MS. If the ASDM needs to be moved to a different
path, the reference to it in the MS needs to be updated. This can
be achieved with **ms.asdmref**. The method takes one argument:
*abspath*. When called with *abspath* equal to an empty string
(default), the method just reports the currently set ASDM path or
an empty string if the ASDM path was not set, i.e. the MS was not
lazily imported. If you want to move the referenced ASDM to a
different path, you can set the new absolute path by providing it
as the value of *abspath* to the method.
::
ms.open(’uid___A12345_X678_X910.ms’,False)
ms.asdmref(’/home/alma/myanalysis/uid___A12345_X678_X910’)
ms.close()
will set the new location of the referenced ASDM to
/home/alma/myanalysis/uid___A12345_X678_X910.
.. note:: **NOTE**: The lazily imported MS can be moved without any
restrictions independently from the referenced ASDM as long as
the absolute path to the ASDM remains accessible, even across
file systems.
.. _Examples:
Examples
In the simplest form, setting bdfflags=True and verbose=True:
::
importasdm(asdm='uid___A002_Xbbadbe_X88ec.asdm.sdm',
vis='uid___A002_Xbbadbe_X88ec.ms', bdfflags=True, verbose=True)
Import both the corrected and uncorrected WVR data from an ALMA
dataset with wvr_corrected_data='both' and setting bdfflags=True
and verbose=True:
::
importasdm(asdm='uid___A002_Xbbadbe_X88ec.asdm.sdm',
vis='uid___A002_Xbbadbe_X88ec.ms', wvr_corrected_data='both',
bdfflags=True, verbose=True)
In this case, two MeasurementSets are created, one with
WVR-uncorrected data filled in the MAIN table and the other with
WVR-corrected data filled in the MAIN table.
To import data from the VLA (and replicate the behaviour of the
deprecated task **importevla**):
::
importasdm(asdm='19A-119.sb123243.58235.79924266203',
vis='19A-119.sb123243.58235.79924266203.ms', ocorr_mode='co',
with_pointing_correction=True, process_flags=True)
flagdata(vis='19A-119.sb123243.58235.79924266203.ms',
mode='shadow')
flagdata(vis='19A-119.sb123243.58235.79924266203.ms',
mode='clip', correlation='ABS_ALL', and clipzeros=True)
Note that while online flags can thus be created by leaving the
parameter *process_flags = True* by default, the additional
flagging steps need to be performed after **importasdm** to flag
zero values and shadowing of antennas, in order to replicate the
behavior of the deprecated task **importevla**. See the CASA Docs
pages on `importing
(u,v)-data <../../notebooks/visibilities_import_export.ipynb#UV-Data-Import>`__
for details.
.. _Development:
Development
No additional development details
.. _Details:
Parameter Details
Detailed descriptions of each function parameter
.. _asdm:
| ``asdm (string)`` - Name of input ASDM file (directory)
| Default: none
| Example: asdm='ExecBlock3'
.. _vis:
| ``vis (string='')`` - Root ms name.
| Default: none
| Note that a prefix (.ms) is NOT appended to this
| name.
.. _createmms:
| ``createmms (bool=False)`` - Create a Multi-MS partitioned according to the given
| separation axis.
| Default: False
| Options: False|True
| For more detailed documentation on partition,
| Multi-MS and the MPI use in CASA, please see CASA
| Docs (https://casa.nrao.edu/casadocs/).
.. _separationaxis:
| ``separationaxis (string='auto')`` - Axis to do parallelization across
| Default: 'auto'
| Options: 'scan', 'spw', 'baseline', 'auto'
| * auto: will partition per scan/spw to obtain
| optimal load balancing with the following
| criteria:
| 1 - Maximize the scan/spw/field distribution
| across sub-MSs
| 2 - Generate sub-MSs with similar size
| * 'scan' or 'spw': will partition the MS into
| scan or spw. The individual sub-MSs may not be
| balanced with respect to the number of rows.
| * 'baseline': mostly useful for Single-Dish
| data. This axis will partition the MS based on
| the available baselines. If the user wants only
| auto-correlations, use the
| ocorr_mode='ao'. Note that if numsubms='auto',
| partition will try to create as many subMSs as
| the number of available servers in the
| cluster. If the user wants to have one subMS
| for each baseline, set the numsubms parameter
| to a number higher than the number of baselines
| to achieve this.
.. _numsubms:
| ``numsubms ({string, int}='auto')`` - The number of sub-MSs to create in the Multi-Ms.
| Default: 'auto'
| Options: any integer number (example: numsubms=4)
| The default 'auto' is to partition using the
| number of available servers given when launching
| CASA. If the task is unable to determine the
| number of running servers, or the user did not
| start CASA using mpicasa, numsubms will use 8 as
| the default.
| Example: Launch CASA with 5 engines, where 4
| of them will be used to create the MMS (the
| first engine is used as the MPIClient):
| mpicasa -n 5 casa --nogui --log2term
| CASA> importasdm('uid__A1', createmms=True)
.. _corr_mode:
| ``corr_mode (string='all')`` - Correlation mode to be considered on input.
| Default: 'all'
| Options: ao, co, ac, or all
.. _srt:
| ``srt (string='all')`` - Spectral resolution type.
| Default: 'all'
| Options: fr, ca, bw, or all
.. _time_sampling:
| ``time_sampling (string='all')`` - Specifies the time sampling (INTEGRATION and/or
| SUBINTEGRATION) to be considered on input.
| Default: 'all'
| Options: i, si, or all
| A quoted string containing a sequence of i, si,
| or all separated by whitespaces is expected
.. _ocorr_mode:
| ``ocorr_mode (string='ca')`` - Output data for correlation mode AUTO_ONLY (ao) or
| CROSS_ONLY (co) or CROSS_AND_AUTO (ca)
| Default: 'ca'
| Options: ao, co, ca
.. _compression:
| ``compression (bool=False)`` - Produce compressed columns in the resulting measurement
| set.
| Default: False
| Options: False|True
.. _lazy:
| ``lazy (bool=False)`` - Make the MS DATA column read the ASDM Binary data
| directly (faster import, smaller MS).
| Default: False
| Options: False|True
| Instead of writing a copy of the visibilities
| into a standard DATA column, lazy=True will make
| importasdm only write a lookup-table such that
| later access to the DATA column will read the
| ASDM binary visibility data directly. This
| requires that the ASDM not be removed from its
| location as long the the DATA column is
| needed. Use method ms.asdmref() to query and
| manipulate the reference to the ASDM.
| lazy=True will save ca. 50% disk space and
| accelerate the DATA column access by
| ca. 10%. lazy=True will only work when there is
| visibility data in the ASDM, not with pure
| radiometer data.
.. _asis:
| ``asis (string='')`` - Creates verbatim copies of the ASDM tables in the output
| measurement set.
| Default: none
| The value given to this option must be a list of
| table names separated by space characters; the
| wildcard character '*' is allowed in table
| names.
.. _wvr_corrected_data:
| ``wvr_corrected_data (string='no')`` - Specifies which values are considerd in the ASDM binary
| data to fill the DATA column in the MAIN table of the MS.
| Default: no
| Options: no|yes|both
| * no: uncorrected data
| * yes: corrected data
| * both: for corrected and uncorrected data. Note
| if both is selected, two measurement sets are
| created, one with uncorrected data and the
| other with corrected data (which name is
| suffixed by '-wvr-corrected')
.. _scans:
| ``scans (string='')`` - Processes only the scans specified in the option's value.
| Default: none (all scans)
| This value is a semicolon separated list of scan
| specifications. A scan specification consists in
| an exec bock index followed by the character ':'
| followed by a comma separated list of scan
| indexes or scan index ranges. A scan index is
| relative to the exec block it belongs to. Scan
| indexes are 1-based while exec blocks's are
| 0-based.
| Examples:
| '0:1'
| '2:2~6'
| '0:1;1:2~6,8;2:,3:24~30'
| '1,2'
| '3:' alone will be interpreted as 'all the
| scans of the exec block#3'. An scan index or a
| scan index range not preceded by an exec block
| index will be interpreted as 'all the scans
| with such indexes in all the exec blocks'.
.. _ignore_time:
| ``ignore_time (bool=False)`` - All the rows of the tables Feed, History, Pointing,
| Source, SysCal, CalDevice, SysPower, and Weather are processed
| independently of the time range of the selected exec block / scan.
| Default: False
| Options: False|True
.. _process_syspower:
| ``process_syspower (bool=True)`` - The SysPower table is processed if and only if this
| parameter is set to true.
| Default: True
| Options: True|False
.. _process_caldevice:
| ``process_caldevice (bool=True)`` - The CalDevice table is processed if and only if this
| parameter is set to true.
| Default: True
| Options: True|False
.. _process_pointing:
| ``process_pointing (bool=True)`` - The Pointing table is processed if and only if this
| parameter is set to true.
| Default: True
| Options: True|False
| If set to False, the POINTING table is empty in
| the resulting MS
.. _process_flags:
| ``process_flags (bool=True)`` - Create online flags based on the Flag.xml, Antenna.xml
| and SpectralWindow.xml files and copy them to the FLAG_CMD sub-table
| of the MS.
| Default: True
| Options: True|False
| The flags will NOT be applied unless the
| parameter applyflags is set to True. Optionally,
| the flags can also be saved to an external ASCII
| file if savecmds is set to True.
.. _tbuff:
| ``tbuff (double=0.0)`` - Time padding buffer (seconds)
| Subparameter of process_flags=True
| Default: 0.0
| NOTE: this time is in seconds. You should
| currently set the value of tbuff to be 1.5x the
| correlator integration time if greater than 1
| second. For example, if the SDM has integrations
| of 3 seconds, set tbuff=4.5. Likewise, set
| tbuff=15.0 for 10-sec integrations.
.. _applyflags:
| ``applyflags (bool=False)`` - Apply the online flags to the MS.
| Subparameter of process_flags=True
| Default: False
| Options: False|True
.. _savecmds:
| ``savecmds (bool=False)`` - Save the flag commands to an ASCII file given by the
| parameter outfile.
| Subparameter of process_flags=True
| Default: False
| Options: False|True
.. _outfile:
| ``outfile ({string, stringArray}='')`` - Filename or list of filenames where to save the online
| flag commands.
| Subparameter of process_flags=True
| Default: '' (it will save on a filename composed
| from the MS name(s).) E.g., for vis='uid_A02.ms',
| the outfile will be 'uid_A02_cmd.txt'.
.. _flagbackup:
| ``flagbackup (bool=True)`` - Back up flag column before applying flags.
| Default: True
| Options: True|False
.. _verbose:
| ``verbose (bool=False)`` - Produce log output as asdm2MS is being run
| Default: False
| Options: False|True
.. _overwrite:
| ``overwrite (bool=False)`` - Over write an existing MS(s) or MS(s), if the option
| wvr_corrected_data='both'
| Default: False (do not overwrite)
| Options: False|True
| NOTE: the overwrite parameter affects all the
| output of the task. If any of the following
| exist, it will not overwrite them. MS(s),
| .flagversions, online flag files. When set to
| True, it will overwrite the MS, .flagversions and
| online flag file.
.. _showversion:
| ``showversion (bool=False)`` - Report the version of asdm2MS being used
| Default: False
| Options: False|True
.. _useversion:
| ``useversion (string='v3')`` - Version of asdm2MS to be used
| Default: 'v3' (should work for all data)
.. _bdfflags:
| ``bdfflags (bool=False)`` - Set the MS FLAG column according to the ASDM _binary_
| flags
| Default: False
| Options: False|True
.. _with_pointing_correction:
| ``with_pointing_correction (bool=False)`` - Add (ASDM::Pointing::encoder -
| ASDM::Pointing::pointingDirection) to the value to be written in
| MS::Pointing::direction
| Default: False
| Options: False|True
.. _convert_ephem2geo:
| ``convert_ephem2geo (bool=True)`` - if True, convert any attached ephemerides to the GEO
| reference frame (time-spacing not changed)
| Default: True
| Options: True|False
| ALMA uses ephemerides with observer location
| equal to the ALMA site. For later processing of
| the radial velocity information in, e.g. cvel, a
| geocentric ephemeris is needed. Setting this
| option to True will perform the conversion of
| positions and velocities on all attached
| ephemerides in the imported MS. This will neither
| change the time-spacing nor the duration of the
| ephemeris. No interpolation in time is done.
.. _polyephem_tabtimestep:
| ``polyephem_tabtimestep (double=0.)`` - Timestep (days) for the tabulation of polynomial
| ephemerides. A value less than or equal to 0 disables tabulation.
| Default: 0
| Presently, VLA data can contain polynomial
| ephemerides. ALMA data uses tabulated values.
"""
pass