importasdm(asdm, vis='', createmms=False, separationaxis='auto', numsubms='auto', corr_mode='all', srt='all', time_sampling='all', ocorr_mode='ca', compression=False, lazy=False, asis='', wvr_corrected_data='no', scans='', ignore_time=False, process_syspower=True, process_caldevice=True, process_pointing=True, process_flags=True, tbuff=0.0, applyflags=False, savecmds=False, outfile='', flagbackup=True, verbose=False, overwrite=False, bdfflags=False, with_pointing_correction=False, convert_ephem2geo=True, polyephem_tabtimestep=0.0)¶
Convert an ALMA Science Data Model observation into a CASA visibility file (MS)
asdm (path) - Name of input asdm directory (on disk)
vis (string=’’) - Root name of the ms to be created. Note the .ms is NOT added
createmms (bool=False) - Create a Multi-MS output
corr_mode (string=’all’) - Specifies the correlation mode to be considered on input. A quoted string containing a sequence of ao, co, ac,or all separated by whitespaces is expected
srt (string=’all’) - Specifies the spectral resolution type to be considered on input. A quoted string containing a sequence of fr, ca, bw, or all separated by whitespaces is expected
time_sampling (string=’all’) - Specifies the time sampling (INTEGRATION and/or SUBINTEGRATION) to be considered on input. A quoted string containing a sequence of i, si, or all separated by whitespaces is expected
ocorr_mode (string=’ca’) - Output data for correlation mode AUTO_ONLY (ao) or CROSS_ONLY (co) or CROSS_AND_AUTO (ca)
compression (bool=False) - Flag for turning on data compression
lazy (bool=False) - Make the MS DATA column read the ASDM Binary data directly (faster import, smaller MS)
asis (string=’’) - Creates verbatim copies of the ASDMtables in the ouput measurement set. Value given must be a string of table names separated by spaces; A * wildcard is allowed.
wvr_corrected_data (string=’no’) - Specifies which values are considerd in the SDM binary data to fill the DATA column in the MAIN table of the MS; yes for corrected, no for uncorrected, both for corrected and uncorrected (resulting in two MSs)
scans (string=’’) - Processes only the specified scans. A scan specification consists of an exec block index followed by the : character, followed by a comma separated list of scan indexes or scan index ranges. (e.g. 0:1;1:2~6,8;2:,3:24~30)
ignore_time (bool=False) - All the rows of the tables Feed, History, Pointing, Source, SysCal, CalDevice, SysPower, and Weather are processed independently of the time range of the selected exec block / scan.
process_syspower (bool=True) - Process the SysPower table?
process_caldevice (bool=True) - Process the CalDevice table?
process_pointing (bool=True) - Process the Pointing table?
process_flags (bool=True) - Create online flags in the FLAG_CMD sub-table?
flagbackup (bool=True) - Back up flag column before applying flags.
verbose (bool=False) - Output lots of information while the filler is working
overwrite (bool=False) - Over write an existing MS(s)
bdfflags (bool=False) - Set the MS FLAG column according to the ASDM _binary_ flags
with_pointing_correction (bool=False) - Add (ASDM::Pointing::encoder - ASDM::Pointing::pointingDirection) to the value to be written in MS::Pointing::direction
convert_ephem2geo (bool=True) - if True, convert any attached ephemerides to the GEO reference frame (time-spacing not changed)
polyephem_tabtimestep (double=0.) - Timestep (days) for the tabulation of polynomial ephemerides. A value <= 0 disables tabulation.
NOTE: importevla has now been deprecated, and importasdm combined with flagdata should be used to import JVLA data, as explained in a dedicated section below.
The importasdm task will fill SDM1.2 and SDM1.3 format data into a CASA visibility data set (MS). The importasdm task supports all changes in the ALMA ASDM made since the previous CASA release (an up-to-date description of the most recent tables of the SDM can be found here). ALMA data were recorded in SDM1.2 format from October 2009 until May 2011. Since May 2011, ALMA is using the SDM1.3 format. In particular, all science data from Cycle 0 onward will be in SDM1.3. The JVLA started using SDM1.2 in October 2009, but currently also uses SDM1.3. The importasdm task can read all of the above formats. For the default parameter settings for importasdm, see the Examples tab.
The basic modes of importing data utilizing importasdm can be set by using various parameters. For example,
If the parameter scans is set, then importasdm processes only the scans specified in the option’s value. This value is a semicolon-separated list of scan specifications. A scan specification consists in an execution (exec) block index followed by the character ’:’ followed by a comma-separated list of scan indices or scan index ranges. A scan index is relative to the exec block it belongs to. Scan indices are 1-based while exec blocks are 0-based. The expressions:
are all valid values for the selection. The “3:” selector will be interpreted as ’all the scans of the exec block 3’. A scan index or a scan index range not preceded by an exec block index will be interpreted as ’all the scans with such indexes in all the exec blocks’. By default, all the scans are considered.
When process_flags=True, the task will create online flags based on the Flag.xml, Antenna.xml and SpectralWindow.xml files and copy them to the FLAG_CMD sub-table of the MS. The flags will NOT be applied unless the parameter applyflags is set to True. Optionally, the flags can also be saved to an external ASCII file if savecmds is set to True. The flags can later be applied to the MS using task flagdata in list mode.
When bdfflags=True, the task will apply online flags contained in the ASDM BDF data by calling the executable bdflags2MS which the user can also do from the OS prompt. Setting bdfflags=True is recommended for ALMA data.
When createmms=True, the task will create a Multi-MS partitioned according to the given separation axis. For more detailed documentation on partition, Multi-MS, and the MPI use in CASA, please see the global task list pages describing partition and mstransform. The Multi-MS also contains more information on Multi-MS creation.
When setting the values for wvr_corrected_data, the task will read the SDM binary data and fill the DATA column in the MAIN table of the MS for those ALMA data that have either corrected data, uncorrected data or both. Expected values for this option are ‘no’ for the uncorrected data (this is the default), ‘yes’ for the corrected data, and ‘both’ for corrected and uncorrected data. In the latter case, two MeasurementSets are created, one containing the uncorrected data and the other containing the corrected data; the name of the corrected MS will be given a suffix of ‘-wvr-corrected’. See the relevent documentation on the ALMA Science Portal regarding those ALMA cycles which provided both corrected and uncorrected data streams.
Import of JVLA data with importasdm
As of CASA 5.4, the task importevla is no longer available to import JVLA data. The functionality is replaced by importasdm, which is also being used by the VLA pipeline. However, several additional steps are required to duplicate the behaviour of importevla when using importasdm, involving a difference in default parameters and the fact that some of the on-the-go flagging cannot be performed by importasdm.
To mimic the behaviour of importevla, change the following parameters in importasdm from their default settings:
ocorr_mode = ‘co’ to import cross-correlations only (discarding auto-correlations) * *
with_pointing_correction = True to add pointing corrections * *
process_flags = True (default) to read in the online flags, then applyflags = True to apply the online flags and/or savecmd = True to save flag commands to an ascii table .
For ephemeris objects: convert_ephem2geo = False
While online flags can thus be created by leaving the parameter process_flags = True by default, additional flagging steps need to be performed after importasdm to flag zero values and shadowing of antennas:
Shadow flags: use task flagdata, with mode = ‘shadow’ (and optionally reason = ‘shadow’). The parameters tolerance and addantenna can be specified in flagdata in the same way they were used in importevla. * *
Zero clipping flags: use task flagdata, with mode = ‘clip’, correlation = ‘ABS_ALL’, and clipzeros = True (and optionally reason = ‘clip’) . Note that the non-default case in importevla where flagpol = False c an be replicated by setting correlation=”ABS_RR, ABS_LL”.
Like importasdm, the task flagdata can also save the flagging commands to an ascii table by setting savepars = True. To NOT apply the flags (applyflags=False in importevla) add action=’calculate’ to flagdata. You may also chose to add a reason using the cmdreason argument, e.g. cmdreason=”CLIP_ZERO_ALL”.
WARNING: The task flagdata can only write out the flag commands for that invocation of flagdata. The default overwrite=True must be used to overwrite an existing file. In order to save the commands from all 3 possible flagging steps (importasdm, zero, and shadow) each step must be saved to a separate file, which must then be concatenated into a single file to be used to flag the data.
Import of ASDM data with option lazy=True
For the parameter lazy, if the default value False is chosen, importasdm will fill the visibilities into a newly created DATA column (FLOAT_DATA for total power data) of the MS converting them from their binary format in the ASDM to the CASA Table format. If lazy is set to True, the task will create the DATA/FLOAT_DATA column with an ASDM-specific storage manager, the (asdmstman), which enables CASA to directly read the binary data from the ASDM with on-the-fly conversion. No redundant copy of the raw data is created.
This procedure has the advantage that it saves more than 60% disk space and at least in some cases makes the access to the DATA column ≥ 10% faster because the data I/O volume is decreased. For the same reason, it also accelerates the import itself by ca. a factor 2. The acceleration is particularly large in the applycal task and here particularly on standard SATA disks. E.g., if your ASDM has a size of 36 GB, the import with default parameters will turn this into an MS of 73 GB size (total disk space consumption = 36 GB + 73 GB = 109 GB). With lazy=True, the imported MS has a size of only 2 GB (total disk space consumption = 36 GB + 2 GB = 38 GB). I.e. your total disk space savings are ca. 65%. Even when you compare to the case where you delete the ASDM after normal import, the solution with lazy import and keeping the ASDM will save you ca. 48% disk space (in the example above 38 GB compared to 73 GB). The only caveats are the following:
You must not delete your ASDM. You can, however, move it but you have to update the reference stored in the MS. Symbolic links will work. See below on how to use the tool method ms.asdmref to manipulate the ASDM reference.
The lazily imported DATA/FLOAT_DATA column is read-only. But in any normal data reduction, the DATA/FLOAT_DATA column (as opposed to CORRECTED DATA) is treated as read-only anyway.
The lazily imported MS is numerically identical with the traditionally imported MS and so are all results derived from the MSs. The setting lazy=True might be made the default setting in future CASA releases. An important additional tool to manipulate lazily imported MSs is the method ms.asdmref in the MS tool. If the MS is imported from an ASDM with option lazy=True, the DATA/FLOAT_DATA column of the MS is virtual and directly reads the visibilities from the ASDM. A reference to the original ASDM is stored with the MS. If the ASDM needs to be moved to a different path, the reference to it in the MS needs to be updated. This can be achieved with ms.asdmref. The method takes one argument: abspath. When called with abspath equal to an empty string (default), the method just reports the currently set ASDM path or an empty string if the ASDM path was not set, i.e. the MS was not lazily imported. If you want to move the referenced ASDM to a different path, you can set the new absolute path by providing it as the value of abspath to the method.
ms.open(’uid___A12345_X678_X910.ms’,False) ms.asdmref(’/home/alma/myanalysis/uid___A12345_X678_X910’) ms.close()
will set the new location of the referenced ASDM to /home/alma/myanalysis/uid___A12345_X678_X910.
NOTE: The lazily imported MS can be moved without any restrictions independently from the referenced ASDM as long as the absolute path to the ASDM remains accessible, even across file systems.
In the simplest form, setting bdfflags=True and verbose=True:
importasdm(asdm='uid___A002_Xbbadbe_X88ec.asdm.sdm', vis='uid___A002_Xbbadbe_X88ec.ms', bdfflags=True, verbose=True)
Import both the corrected and uncorrected WVR data from an ALMA dataset with wvr_corrected_data=’both’ and setting bdfflags=True and verbose=True:
importasdm(asdm='uid___A002_Xbbadbe_X88ec.asdm.sdm', vis='uid___A002_Xbbadbe_X88ec.ms', wvr_corrected_data='both', bdfflags=True, verbose=True)
In this case, two MeasurementSets are created, one with WVR-uncorrected data filled in the MAIN table and the other with WVR-corrected data filled in the MAIN table.
To import data from the VLA (and replicate the behaviour of the deprecated task importevla):
importasdm(asdm='19A-119.sb123243.58235.79924266203', vis='19A-119.sb123243.58235.79924266203.ms', ocorr_mode='co', with_pointing_correction=True, process_flags=True) flagdata(vis='19A-119.sb123243.58235.79924266203.ms', mode='shadow') flagdata(vis='19A-119.sb123243.58235.79924266203.ms', mode='clip', correlation='ABS_ALL', and clipzeros=True)
Note that while online flags can thus be created by leaving the parameter process_flags = True by default, the additional flagging steps need to be performed after importasdm to flag zero values and shadowing of antennas, in order to replicate the behavior of the deprecated task importevla. See the CASA Docs pages on importing (u,v)-data for details.
No additional development details
- Parameter Details
Detailed descriptions of each function parameter
asdm (path)- Name of input ASDM file (directory)Default: noneExample: asdm=’ExecBlock3’
vis (string='')- Root ms name.Default: noneNote that a prefix (.ms) is NOT appended to thisname.
createmms (bool=False)- Create a Multi-MS partitioned according to the givenseparation axis.Default: FalseOptions: False|TrueFor more detailed documentation on partition,Multi-MS and the MPI use in CASA, please see CASADocs (https://casa.nrao.edu/casadocs/).
separationaxis (string='auto')- Axis to do parallelization acrossDefault: ‘auto’Options: ‘scan’, ‘spw’, ‘baseline’, ‘auto’* auto: will partition per scan/spw to obtainoptimal load balancing with the followingcriteria:1 - Maximize the scan/spw/field distributionacross sub-MSs2 - Generate sub-MSs with similar size* ‘scan’ or ‘spw’: will partition the MS intoscan or spw. The individual sub-MSs may not bebalanced with respect to the number of rows.* ‘baseline’: mostly useful for Single-Dishdata. This axis will partition the MS based onthe available baselines. If the user wants onlyauto-correlations, use theocorr_mode=’ao’. Note that if numsubms=’auto’,partition will try to create as many subMSs asthe number of available servers in thecluster. If the user wants to have one subMSfor each baseline, set the numsubms parameterto a number higher than the number of baselinesto achieve this.
numsubms (string='auto')- The number of sub-MSs to create in the Multi-Ms.Default: ‘auto’Options: any integer number (example: numsubms=4)The default ‘auto’ is to partition using thenumber of available servers given when launchingCASA. If the task is unable to determine thenumber of running servers, or the user did notstart CASA using mpicasa, numsubms will use 8 asthe default.Example: Launch CASA with 5 engines, where 4of them will be used to create the MMS (thefirst engine is used as the MPIClient):mpicasa -n 5 casa –nogui –log2termCASA> importasdm(‘uid__A1’, createmms=True)
corr_mode (string='all')- Correlation mode to be considered on input.Default: ‘all’Options: ao, co, ac, or all
srt (string='all')- Spectral resolution type.Default: ‘all’Options: fr, ca, bw, or all
time_sampling (string='all')- Specifies the time sampling (INTEGRATION and/orSUBINTEGRATION) to be considered on input.Default: ‘all’Options: i, si, or allA quoted string containing a sequence of i, si,or all separated by whitespaces is expected
ocorr_mode (string='ca')- Output data for correlation mode AUTO_ONLY (ao) orCROSS_ONLY (co) or CROSS_AND_AUTO (ca)Default: ‘ca’Options: ao, co, ca
compression (bool=False)- Produce compressed columns in the resulting measurementset.Default: FalseOptions: False|True
lazy (bool=False)- Make the MS DATA column read the ASDM Binary datadirectly (faster import, smaller MS).Default: FalseOptions: False|TrueInstead of writing a copy of the visibilitiesinto a standard DATA column, lazy=True will makeimportasdm only write a lookup-table such thatlater access to the DATA column will read theASDM binary visibility data directly. Thisrequires that the ASDM not be removed from itslocation as long the the DATA column isneeded. Use method ms.asdmref() to query andmanipulate the reference to the ASDM.lazy=True will save ca. 50% disk space andaccelerate the DATA column access byca. 10%. lazy=True will only work when there isvisibility data in the ASDM, not with pureradiometer data.
asis (string='')- Creates verbatim copies of the ASDM tables in the outputmeasurement set.Default: noneThe value given to this option must be a list oftable names separated by space characters; thewildcard character ‘*’ is allowed in tablenames.
wvr_corrected_data (string='no')- Specifies which values are considerd in the ASDM binarydata to fill the DATA column in the MAIN table of the MS.Default: noOptions: no|yes|both* no: uncorrected data* yes: corrected data* both: for corrected and uncorrected data. Noteif both is selected, two measurement sets arecreated, one with uncorrected data and theother with corrected data (which name issuffixed by ‘-wvr-corrected’)
scans (string='')- Processes only the scans specified in the option’s value.Default: none (all scans)This value is a semicolon separated list of scanspecifications. A scan specification consists inan exec bock index followed by the character ‘:’followed by a comma separated list of scanindexes or scan index ranges. A scan index isrelative to the exec block it belongs to. Scanindexes are 1-based while exec blocks’s are0-based.Examples:‘0:1’‘2:2~6’‘0:1;1:2~6,8;2:,3:24~30’‘1,2’‘3:’ alone will be interpreted as ‘all thescans of the exec block#3’. An scan index or ascan index range not preceded by an exec blockindex will be interpreted as ‘all the scanswith such indexes in all the exec blocks’.
ignore_time (bool=False)- All the rows of the tables Feed, History, Pointing,Source, SysCal, CalDevice, SysPower, and Weather are processedindependently of the time range of the selected exec block / scan.Default: FalseOptions: False|True
process_syspower (bool=True)- The SysPower table is processed if and only if thisparameter is set to true.Default: TrueOptions: True|False
process_caldevice (bool=True)- The CalDevice table is processed if and only if thisparameter is set to true.Default: TrueOptions: True|False
process_pointing (bool=True)- The Pointing table is processed if and only if thisparameter is set to true.Default: TrueOptions: True|FalseIf set to False, the POINTING table is empty inthe resulting MS
process_flags (bool=True)- Create online flags based on the Flag.xml, Antenna.xmland SpectralWindow.xml files and copy them to the FLAG_CMD sub-tableof the MS.Default: TrueOptions: True|FalseThe flags will NOT be applied unless theparameter applyflags is set to True. Optionally,the flags can also be saved to an external ASCIIfile if savecmds is set to True.
tbuff (double=0.0)- Time padding buffer (seconds)Subparameter of process_flags=TrueDefault: 0.0NOTE: this time is in seconds. You shouldcurrently set the value of tbuff to be 1.5x thecorrelator integration time if greater than 1second. For example, if the SDM has integrationsof 3 seconds, set tbuff=4.5. Likewise, settbuff=15.0 for 10-sec integrations.
applyflags (bool=False)- Apply the online flags to the MS.Subparameter of process_flags=TrueDefault: FalseOptions: False|True
savecmds (bool=False)- Save the flag commands to an ASCII file given by theparameter outfile.Subparameter of process_flags=TrueDefault: FalseOptions: False|True
outfile (string='')- Filename or list of filenames where to save the onlineflag commands.Subparameter of process_flags=TrueDefault: ‘’ (it will save on a filename composedfrom the MS name(s).) E.g., for vis=’uid_A02.ms’,the outfile will be ‘uid_A02_cmd.txt’.
flagbackup (bool=True)- Back up flag column before applying flags.Default: TrueOptions: True|False
verbose (bool=False)- Produce log output as asdm2MS is being runDefault: FalseOptions: False|True
overwrite (bool=False)- Over write an existing MS(s) or MS(s), if the optionwvr_corrected_data=’both’Default: False (do not overwrite)Options: False|TrueNOTE: the overwrite parameter affects all theoutput of the task. If any of the followingexist, it will not overwrite them. MS(s),.flagversions, online flag files. When set toTrue, it will overwrite the MS, .flagversions andonline flag file.
bdfflags (bool=False)- Set the MS FLAG column according to the ASDM _binary_flagsDefault: FalseOptions: False|True
with_pointing_correction (bool=False)- Add (ASDM::Pointing::encoder -ASDM::Pointing::pointingDirection) to the value to be written inMS::Pointing::directionDefault: FalseOptions: False|True
convert_ephem2geo (bool=True)- if True, convert any attached ephemerides to the GEOreference frame (time-spacing not changed)Default: TrueOptions: True|FalseALMA uses ephemerides with observer locationequal to the ALMA site. For later processing ofthe radial velocity information in, e.g. cvel, ageocentric ephemeris is needed. Setting thisoption to True will perform the conversion ofpositions and velocities on all attachedephemerides in the imported MS. This will neitherchange the time-spacing nor the duration of theephemeris. No interpolation in time is done.
polyephem_tabtimestep (double=0.)- Timestep (days) for the tabulation of polynomialephemerides. A value less than or equal to 0 disables tabulation.Default: 0Presently, VLA data can contain polynomialephemerides. ALMA data uses tabulated values.