uvcontsub

uvcontsub(vis, outputvis='', field='', spw='', scan='', intent='', array='', observation='', datacolumn='data', fitspec='', fitmethod='gsl', fitorder=0, writemodel=False)[source]

continuum subtraction in the uv domain

[Description] [Examples] [Development] [Details]

Parameters

vis (path) - Name of input visibility file (MeasurementSet)
outputvis (string=’’) - Name of output MeasurementSet (visibility file)
field (string=’’) - Select field using field id(s) or field name(s)
spw (string=’’) - Select spectral window/channels
scan (string=’’) - Select scans by scan numbers
intent (string=’’) - Select observing intent
array (string=’’) - Select (sub)array(s) by array ID number
observation (string=’’) - Select by observation ID(s)
datacolumn (string=’data’) - Which data column to process
fitspec ({string, record}=’’) - Specification of polynomial order and spectral window:channel for fitting
fitmethod (string=’gsl’) - Choose fitting method
fitorder (int=0) - Polynomial order for the fits
writemodel (bool=False) - Write fitted model into the MODEL column of the output MS

Returns: gof (dict) - goodness of fit statistics grouped by field, scan, SPW, polarization and real/imaginary part

Description

Note

The task uvcontsub has been updated. It does not yet support combining spws for fitting. If you need this functionality please use uvcontsub_old.

Note

Strictly speaking, the continuum fitted produced by this task is only a good representation of the continuum at the phase center. Residuals may be visible for sources far away and one may wish to try the following for improved results: a) shifting the phase center of the data to the location of the continuum source, uv continuum subtracting, and then shifting the data back or b) try imcontsub in the image domain.

This task can perform continuum fitting and subtraction in the uv domain.

The function estimates the continuum emission by fitting polynomials to the real and imaginary parts of the data. The spectral windows and channels on which the fitting is calculated can be specified using the parameter fitspec. The resulting fit represents a model of the continuum in all channels. The fitted continuum spectrum is subtracted from all channels selected in spw, and the result (intended as line emission only) is produced in an output MeasurementSet. Polynomial models are fitted and subtracted per integration, per baseline, per correlation. Note that because the real and imaginary parts are fitted separately, the fitted model amplitude has the functional form of sqrt(<polynomial of order fitorder * 2>) which, in general, is not a polynomial.

The function returns a dictionary with goodness of fit metrics, grouped by field, scan, SPW, polarization, and real and imaginary part. The goodness of fit metric included in the dictionary is the chi-squared values as calculated and minimized by the fitting algorithms. The fields count found in the dictionary give the number of polynomials fitted for every group (one individual polynomial fit for every data row and polarization, and real and imaginary parts). Simple statistics (average, minimum, and maximum) of the goodness of fit values for every field/scan/SPW/polarization group are given in the dictionary, see examples section.

The input MeasurementSet is not modified (is only read). The data column from the input MeasurementSet that is read can be selected using the datacolumn parameter. The continuum subtracted data are written into the DATA column of the output MS. Optionally, and to support inspection of results and debugging, the fitted model data can be written into the MODEL column of the output MS, using the parameter writemodel. When selection parameters (field, spw, scan, etc.) are used, the output MS includes only the data selected via those selection parameters. Note that uvcontsub takes as input the DATA column by default unless the user sets a different value for datacolumn. This behavior is different from older uvcontsub tasks where the CORRECTED data column is taken as input if available, otherwise falling back to DATA.

The fitting method and polynomial order are chosen via the parameters fitmethod and fitorder. It is also possible to use different polynomial orders for different fields and SPWs when fitspec takes the form of a dictionary. The line-free channels are given in the fitspec parameter. In it simplest form, it is a single line-free channel specification string that applies to all fields. It can also be a dictionary, with different line-free channel specifications and polynomial order for different fields and SPWs. This is to be able to produce an output MeasurementSet with multiple sources and/or fields in one single call. The per-SPW line-free channels are specified using the spw:channel notation, see the MS selection syntax.

In addition to the fitspec parameter, the channelized data flags and data weights also influence how the channels will be used for the purpose of fitting the continuum. Channels that are flagged are effectively excluded for the purpose of fitting (equivalent to excluding them from fitspec). The channel weights, adjusted for example using the statwt task, also influence how relevant different channels will be for the fitting, in a more gradual way.

For more details on how continuum subtraction is performed in the uv domain, use of flags and weights, fitting methods, goodness of fit, etc. see UV Continuum Subtraction.

Examples

Example 1:

Subtract the continuum of channels 10~100 and 300~350 in spw 0 (assuming that the line is in channels 101~299). Note that we also exclude edge channels, e.g. the first 9 channels. We use a fitorder of 0 (default):

result = uvcontsub(vis='input_ms.ms', outputvis='vis_line.ms', fitspec='0:10~100;300~350')
# result has contents as in the following example (excerpt):
result
{'description': 'summary of data fitting results in uv-continuum subtraction',
 'goodness_of_fit': {'field': {'0': {'scan': {'1': {'spw': {'0': {'polarization':
 {'0': {'chi_squared': {'average': {'imag': 1.05,
                                     'real': 0.964},
                        'max': {'imag': 1.11,
                                'real': 1.08},
                         'min': {'imag': 1.02,
                                 'real': 0.74}},
        'count': 40},
  '1': {'chi_squared': {'average': {'imag': 1.05,
                                    'real': 1.04},
                        'max': {'imag': 1.06,
                                'real': 1.15},
                        'min': {'imag': 0.992,
                                'real': 0.954}},
        'count': 40}}}}}}}}}
}

Example 2:

Subtract the continuum of channels 10~100 and 300~350 in spw 0, with fit order of 1. This example is very similar to the previous one but we just increase the order of the polynomials fitted:

uvcontsub(vis='input_ms.ms', outputvis='vis_line.ms', fitorder=1, fitspec='0:10~100;300~350')

Example 3:

Our input MS has two fields. We use one call to uvcontsub to make a single field MS for the first field, and then a second call to make a single field MS for the second field of the input MS:

uvcontsub(vis='input_ms.ms', outputvis='field0_line.ms', field=0, fitspec='0:10~100;300~350', fitorder=0)
uvcontsub(vis='input_ms.ms', outputvis='field1_line.ms', field=1, fitspec='0:20~90;200~350', fitorder=1)

Example 4:

Alternative to previous example, give fitspec as dictionary and produce an output MS with 2 fields:

spec = {'0': {'19': {'chan': '10~100;300~350', 'fitorder': 0}}}, {'1': {'19': {'chan': '20~90;200~350', 'fitorder': 1}}}
uvcontsub(vis='input_ms.ms', outputvis='vis_line.ms', fitspec=spec)

Example 5:

To ease analysis of the fitted model, we produce an output MS with the continuum subtracted data in the DATA column, and the fitted model in the MODEL column. In contrast to the examples above, fitspec is not specified which implies that all channels are used for fitting purposes in all the SPWs:

uvcontsub(vis='input_ms.ms', outputvis='vis_line.ms', writemodel=True)

Example notebook

Examples of simulated MeasurementSets that can be used to evaluate uvcontsub are included in this Jupyter notebook with demo scripts. The notebook is available on Google Colab, where the demo scripts can be run from the browser (or retrieved to be run locally). The verification tests of this task include numerical tests based on simulated data from the notebook. These numerical verification tests check the accuracy of the continuum fits produced by uvcontsub. For example, for polynomials of known coefficients, for order 0, 1, and 2, added artificially to the visibilities, the accuracy of the fitted polynomials is better than 10e-5 (measured as relative residual values through all channels and rows). For a polynomial of order 0, with added Gaussian random noise and continuum SNR of ~3.5, the relative residuals are of the order of 1% (~3% for the 75th percentile). The scripts included in the notebook can also be used to further characterize the behavior of the task for different data properties, and to run other experiments with the task.

Development

This version of uvcontsub is defined to satisfy the operational requirements of the pipelines and SRDP, as best as they could be identified throughout 2021. Additional features (or use modes) can be considered:

Channel specifications in fitspec are supported in the native frame of the input MeasurementSet. The suggestion is that frame conversions, when needed, be handled in separate (helper) functions rather than embedded in the task.
SPW combination, related to the combine parameter of the task uvcontsub_old.
Phase shifting related features and parameters (see task phaseshift).
Some CASA tasks have a parameter excludechans that inverts the channel specification of fitspec (the channels given in fitswp are excluded from the fitting instead of included). This functionality would be provided separately in a helper function.
It is expected that additional tests and support will be needed as future work once higher order polynomials and robust fitting are tried out, as well as phasecenter shifts, etc. New simulations might be needed.

Parameter Details: Detailed descriptions of each function parameter

vis (path) - Name of input visibility file.

Default: ‘’, must be specified

Example: vis=’ngc5921.ms’

outputvis (string='') - Name of output visibility file

Default: ‘’, must be specified
Example: outputvis=’ngc5921_contsub.ms’
If an MS with the output name already exists, it
will not be overwritten and the task stops with
an error.

field (string='') - Select field using field id(s) or field name(s)

Default: ‘’ (all fields)
Use ‘listobs’ to obtain the list of id’s or
names. If field string is a non-negative integer,
it is assumed a field index,  otherwise, it is
assumed a field name.
Examples:
field=’0~2’; field ids 0,1,2
field=’0,4,5~7’; field ids 0,4,5,6,7
field=’3C286,3C295’; field named 3C286 and
3C295
field = ‘3,4C*’; field id 3, all names
starting with 4C

spw (string='') - Select spectral window/channels

Default: ‘’ (meaning all spectral windows and channels)

Examples:
spw=’0~2,4’; spectral windows 0,1,2,4 (all channels)
spw=’<2’;  spectral windows less than 2 (i.e. 0,1)
spw=’0:5~61’; spw 0, channels 5 to 61
spw=’0,10,3:3~45’; spw 0,10 all channels, spw
3 - chans 3 to 45.
spw=’0~2:2~6’; spw 0,1,2 with channels 2
through 6 in each.
spw = ‘*:3~64’  channels 3 through 64 for all sp id’s
spw = ‘ :3~64’ will NOT work.

scan (string='') - Select scans by scan numbers

Default: ‘’ = all
Examples:
scan=’3’
scan=’2~23’
scan=’’4,7,9,12’

intent (string='') - Select observing intent

Default: ‘’ (no selection by intent)

Example: intent=’BANDPASS’ (selects data

labelled with BANDPASS intent)

array (string='') - Select (sub)array(s) by array ID number

Default: ‘’ (all)

observation (string='') - Select by observation ID(s)

Default: ‘’ = all

Example: observation=’0~2,4’

datacolumn (string='data') - Which data column to use for processing

(case-insensitive).
Default: ‘data’
Options: ‘data’, ‘model’, ‘corrected’,
‘float_data’, ‘lag_data’,
Example: datacolumn=’corrected’

fitspec ({string, record}='') - Specification of fit, which can be general or

specific for every field and SPW. The continuum will be fitted
using a polynomial shape and using as input data points the
channels given, which can be considered the line-free
channels. fitspec can be a simple string or a dictionary. If
it is a simple string, it gives an spw:channel specification in
MSSelection syntax, and that same spw:channel specification is
used for all fields. In that case, the polynomial order for
all fields and SPWs is defined by the fitorder parameter.
If it is a dictionary, its entries can define a different
specification of fit order and channels for every field and
for every SPW. The dictionary can have up to as many entries
as fields are present in the MeasurementSet, and within each
field entry, as many entries as SPWs are present. Each SPW
entry specifies the polynomial order (in a ‘fitorder’ entry)
and the line-free channels (in a ‘chan’ entry).
If a field or SPW is not included, the default is assumed (all
channels, and polynomial order given in the global fitorder
parameter).
Default: ‘’ (all channels for all SPWs for all fields)
Examples:
‘17:100~500;600~910;1215~1678;1810~1903,19:7~100’
(The ranges of channels given for SPW 17 and 19
are used for all the fields present in the
MeasurementSet.)
Dictionary to use different channels and polynomial
orders for different fields and SPWs, assuming:
4 SPWs, with IDs 17, 19, 21, 23
6 fields, with IDs 0, 1, 4, 5, 6, 7
{‘0’: {‘17,19,21,23’: {‘chan’: ‘’, ‘fitorder’: 1}},
‘1’: {‘17’: {‘chan’: ‘8~120’, ‘fitorder’: 0},
‘19’: {‘chan’: ‘7~100’, ‘fitorder’: 1},
‘21’: {‘chan’: ‘21:2~119’, ‘fitorder’: 1},
‘23’: {‘chan’: ‘’, ‘fitorder’: 0}},
‘4,5,6’: {‘19’: {‘chan’: ‘7~115’, ‘fitorder’: 1},
‘21’: {‘chan’: ‘’, ‘fitorder’: 1},
‘23’: {‘chan’: ‘’, ‘fitorder’: 2}},
‘7’: ‘NONE’}
(Field 0: fit all SPWs and channels (same as not giving it),
but using polynomial order 1 for all SPWs.
Field 1: fit only some channel ranges (all for SPW 23), with
different polynomial orders.
Fields 4,5,6: fit only some channel ranges in SPW 19, all
channels for SPWs 21 and 23, and don’t fit SPWs 17),
also using different polynomial orders.
Field 7: do not subtract continuum)
(The dictionary fields ‘chan’ can be given in channel
numbers, as in the example above, or in frequencies, using
the MSSelection syntax)
In the example, the empty string given for field 0 and all its
SPWs means that all the channels are fitted (and subtracted)
in all those SPWs. For the corner case where for a given field
no channels should be fitted in any SPW, the field entry can
be given as a string set to ‘NONE’.
This specification of SPWs and channels uses a syntax similar
to the spw parameter. But in contrast to the spw parameter,
the SPWs and channels given in fitspec are not used to select
data but as a mask applied on the input data only for fitting
purposes. The channels that are not included in the
specification are masked out and not used as input by the
polynomial fitting algorithm. This is different from the spw
parameter, which selects the data that will be included in the
output. All channels present in the output are
continuum-subtracted, regardless of whether they were
specified as inputs to the fitting in fitspec

fitmethod (string='gsl') - Choose fitting methods

Default: ‘gsl’
Options: ‘gsl’, ‘casacore’
Example: fitmethod=’gsl’

fitorder (int=0) - Polynomial order for the fits

Default: 0
Values of fitorder > 1 should be used with
care. Higher order polynomials are more flexible,
and may overfit and absorb line emission. They
also tend to go wild at the edges of the range of
channels used for fitting.

writemodel (bool=False) - Write fitted model into the MODEL column of the

output MeasurementSet.

Default: False
Options: True|False
By default the task creates an MS with the
continuum subtracted data in the DATA column. If
this option is enabled, the task will also write
the fitted model data into the MODEL column of the
output MS.