Source code for casatasks.manipulation.uvcontsub

#
# stub function definition file for docstring parsing
#

[docs]def uvcontsub(vis, outputvis='', field='', spw='', scan='', intent='', array='', observation='', datacolumn='data', fitspec='', fitmethod='gsl', fitorder=0, writemodel=False): r""" continuum subtraction in the uv domain [`Description`_] [`Examples`_] [`Development`_] [`Details`_] Parameters - vis_ (path) - Name of input visibility file (MeasurementSet) - outputvis_ (string='') - Name of output MeasurementSet (visibility file) - field_ (string='') - Select field using field id(s) or field name(s) - spw_ (string='') - Select spectral window/channels - scan_ (string='') - Select scans by scan numbers - intent_ (string='') - Select observing intent - array_ (string='') - Select (sub)array(s) by array ID number - observation_ (string='') - Select by observation ID(s) - datacolumn_ (string='data') - Which data column to process - fitspec_ ({string, record}='') - Specification of polynomial order and spectral window:channel for fitting - fitmethod_ (string='gsl') - Choose fitting method - fitorder_ (int=0) - Polynomial order for the fits - writemodel_ (bool=False) - Write fitted model into the MODEL column of the output MS .. _Returns: Returns gof (dict) - goodness of fit statistics grouped by field, scan, SPW, polarization and real/imaginary part .. _Description: Description .. note:: The task **uvcontsub** has been updated. It does not yet support combining spws for fitting. If you need this functionality please use uvcontsub_old. .. note:: Strictly speaking, the continuum fitted produced by this task is only a good representation of the continuum at the phase center. Residuals may be visible for sources far away and one may wish to try the following for improved results: a) shifting the phase center of the data to the location of the continuum source, uv continuum subtracting, and then shifting the data back or b) try **imcontsub** in the image domain. This task can perform continuum fitting and subtraction in the uv domain. The function estimates the continuum emission by fitting polynomials to the real and imaginary parts of the data. The spectral windows and channels on which the fitting is calculated can be specified using the parameter *fitspec*. The resulting fit represents a model of the continuum in all channels. The fitted continuum spectrum is subtracted from all channels selected in *spw*, and the result (intended as line emission only) is produced in an output MeasurementSet. Polynomial models are fitted and subtracted per integration, per baseline, per correlation. Note that because the real and imaginary parts are fitted separately, the fitted model amplitude has the functional form of sqrt(<*polynomial of order fitorder* * 2>) which, in general, is not a polynomial. The function returns a dictionary with goodness of fit metrics, grouped by field, scan, SPW, polarization, and real and imaginary part. The goodness of fit metric included in the dictionary is the chi-squared values as calculated and minimized by the fitting algorithms. The fields *count* found in the dictionary give the number of polynomials fitted for every group (one individual polynomial fit for every data row and polarization, and real and imaginary parts). Simple statistics (average, minimum, and maximum) of the goodness of fit values for every field/scan/SPW/polarization group are given in the dictionary, see examples section. The input MeasurementSet is not modified (is only read). The data column from the input MeasurementSet that is read can be selected using the *datacolumn* parameter. The continuum subtracted data are written into the DATA column of the output MS. Optionally, and to support inspection of results and debugging, the fitted model data can be written into the MODEL column of the output MS, using the parameter *writemodel*. When selection parameters (field, spw, scan, etc.) are used, the output MS includes only the data selected via those selection parameters. Note that uvcontsub takes as input the DATA column by default unless the user sets a different value for *datacolumn*. This behavior is different from older uvcontsub tasks where the CORRECTED data column is taken as input if available, otherwise falling back to DATA. The fitting method and polynomial order are chosen via the parameters *fitmethod* and *fitorder*. It is also possible to use different polynomial orders for different fields and SPWs when *fitspec* takes the form of a dictionary. The line-free channels are given in the *fitspec* parameter. In it simplest form, it is a single line-free channel specification string that applies to all fields. It can also be a dictionary, with different line-free channel specifications and polynomial order for different fields and SPWs. This is to be able to produce an output MeasurementSet with multiple sources and/or fields in one single call. The per-SPW line-free channels are specified using the spw:channel notation, see the `MS selection syntax <../../notebooks/visibility_data_selection.ipynb>`__. In addition to the *fitspec* parameter, the channelized data flags and data weights also influence how the channels will be used for the purpose of fitting the continuum. Channels that are flagged are effectively excluded for the purpose of fitting (equivalent to excluding them from fitspec). The channel weights, adjusted for example using the statwt task, also influence how relevant different channels will be for the fitting, in a more gradual way. For more details on how continuum subtraction is performed in the uv domain, use of flags and weights, fitting methods, goodness of fit, etc. see `UV Continuum Subtraction <../../notebooks/uv_manipulation.ipynb#UV-Continuum-Subtraction>`__. .. _Examples: Examples **Example 1:** Subtract the continuum of channels 10~100 and 300~350 in spw 0 (assuming that the line is in channels 101~299). Note that we also exclude edge channels, e.g. the first 9 channels. We use a fitorder of 0 (default): .. code-block:: python result = uvcontsub(vis='input_ms.ms', outputvis='vis_line.ms', fitspec='0:10~100;300~350') # result has contents as in the following example (excerpt): result {'description': 'summary of data fitting results in uv-continuum subtraction', 'goodness_of_fit': {'field': {'0': {'scan': {'1': {'spw': {'0': {'polarization': {'0': {'chi_squared': {'average': {'imag': 1.05, 'real': 0.964}, 'max': {'imag': 1.11, 'real': 1.08}, 'min': {'imag': 1.02, 'real': 0.74}}, 'count': 40}, '1': {'chi_squared': {'average': {'imag': 1.05, 'real': 1.04}, 'max': {'imag': 1.06, 'real': 1.15}, 'min': {'imag': 0.992, 'real': 0.954}}, 'count': 40}}}}}}}}} } **Example 2:** Subtract the continuum of channels 10~100 and 300~350 in spw 0, with fit order of 1. This example is very similar to the previous one but we just increase the order of the polynomials fitted: .. code-block:: python uvcontsub(vis='input_ms.ms', outputvis='vis_line.ms', fitorder=1, fitspec='0:10~100;300~350') **Example 3:** Our input MS has two fields. We use one call to uvcontsub to make a single field MS for the first field, and then a second call to make a single field MS for the second field of the input MS: .. code-block:: python uvcontsub(vis='input_ms.ms', outputvis='field0_line.ms', field=0, fitspec='0:10~100;300~350', fitorder=0) uvcontsub(vis='input_ms.ms', outputvis='field1_line.ms', field=1, fitspec='0:20~90;200~350', fitorder=1) **Example 4:** Alternative to previous example, give fitspec as dictionary and produce an output MS with 2 fields: .. code-block:: python spec = {'0': {'19': {'chan': '10~100;300~350', 'fitorder': 0}}}, {'1': {'19': {'chan': '20~90;200~350', 'fitorder': 1}}} uvcontsub(vis='input_ms.ms', outputvis='vis_line.ms', fitspec=spec) **Example 5:** To ease analysis of the fitted model, we produce an output MS with the continuum subtracted data in the DATA column, and the fitted model in the MODEL column. In contrast to the examples above, *fitspec* is not specified which implies that all channels are used for fitting purposes in all the SPWs: .. code-block:: python uvcontsub(vis='input_ms.ms', outputvis='vis_line.ms', writemodel=True) **Example notebook** Examples of simulated MeasurementSets that can be used to evaluate uvcontsub are included in this `Jupyter notebook <../../notebooks/simulations_uvcontsub_ALMA.ipynb>`__ with demo scripts. The notebook is available on `Google Colab <https://colab.research.google.com/github/casangi/casadocs/blob/master/docs/notebooks/simulations_uvcontsub_ALMA.ipynb>`_, where the demo scripts can be run from the browser (or retrieved to be run locally). The verification tests of this task include numerical tests based on simulated data from the notebook. These numerical verification tests check the accuracy of the continuum fits produced by uvcontsub. For example, for polynomials of known coefficients, for order 0, 1, and 2, added artificially to the visibilities, the accuracy of the fitted polynomials is better than 10e-5 (measured as relative residual values through all channels and rows). For a polynomial of order 0, with added Gaussian random noise and continuum SNR of ~3.5, the relative residuals are of the order of 1% (~3% for the 75th percentile). The scripts included in the notebook can also be used to further characterize the behavior of the task for different data properties, and to run other experiments with the task. .. _Development: Development This version of uvcontsub is defined to satisfy the operational requirements of the pipelines and SRDP, as best as they could be identified throughout 2021. Additional features (or use modes) can be considered: - Channel specifications in *fitspec* are supported in the native frame of the input MeasurementSet. The suggestion is that frame conversions, when needed, be handled in separate (helper) functions rather than embedded in the task. - SPW combination, related to the *combine* parameter of the task **uvcontsub_old**. - Phase shifting related features and parameters (see task **phaseshift**). - Some CASA tasks have a parameter **excludechans** that inverts the channel specification of fitspec (the channels given in fitswp are excluded from the fitting instead of included). This functionality would be provided separately in a helper function. - It is expected that additional tests and support will be needed as future work once higher order polynomials and robust fitting are tried out, as well as phasecenter shifts, etc. New simulations might be needed. .. _Details: Parameter Details Detailed descriptions of each function parameter .. _vis: | ``vis (path)`` - Name of input visibility file. | Default: '', must be specified | Example: vis='ngc5921.ms' .. _outputvis: | ``outputvis (string='')`` - Name of output visibility file | Default: '', must be specified | Example: outputvis='ngc5921_contsub.ms' | If an MS with the output name already exists, it | will not be overwritten and the task stops with | an error. .. _field: | ``field (string='')`` - Select field using field id(s) or field name(s) | Default: '' (all fields) | Use 'listobs' to obtain the list of id's or | names. If field string is a non-negative integer, | it is assumed a field index, otherwise, it is | assumed a field name. | Examples: | field='0~2'; field ids 0,1,2 | field='0,4,5~7'; field ids 0,4,5,6,7 | field='3C286,3C295'; field named 3C286 and | 3C295 | field = '3,4C*'; field id 3, all names | starting with 4C .. _spw: | ``spw (string='')`` - Select spectral window/channels | Default: '' (meaning all spectral windows and channels) | | Examples: | spw='0~2,4'; spectral windows 0,1,2,4 (all channels) | spw='<2'; spectral windows less than 2 (i.e. 0,1) | spw='0:5~61'; spw 0, channels 5 to 61 | spw='0,10,3:3~45'; spw 0,10 all channels, spw | 3 - chans 3 to 45. | spw='0~2:2~6'; spw 0,1,2 with channels 2 | through 6 in each. | spw = '*:3~64' channels 3 through 64 for all sp id's | spw = ' :3~64' will NOT work. .. _scan: | ``scan (string='')`` - Select scans by scan numbers | Default: '' = all | Examples: | scan='3' | scan='2~23' | scan=''4,7,9,12' .. _intent: | ``intent (string='')`` - Select observing intent | Default: '' (no selection by intent) | Example: intent='*BANDPASS*' (selects data | labelled with BANDPASS intent) .. _array: | ``array (string='')`` - Select (sub)array(s) by array ID number | Default: '' (all) .. _observation: | ``observation (string='')`` - Select by observation ID(s) | Default: '' = all | Example: observation='0~2,4' .. _datacolumn: | ``datacolumn (string='data')`` - Which data column to use for processing | (case-insensitive). | Default: 'data' | Options: 'data', 'model', 'corrected', | 'float_data', 'lag_data', | Example: datacolumn='corrected' .. _fitspec: | ``fitspec ({string, record}='')`` - Specification of fit, which can be general or | specific for every field and SPW. The continuum will be fitted | using a polynomial shape and using as input data points the | channels given, which can be considered the line-free | channels. fitspec can be a simple string or a dictionary. If | it is a simple string, it gives an spw:channel specification in | MSSelection syntax, and that same spw:channel specification is | used for all fields. In that case, the polynomial order for | all fields and SPWs is defined by the fitorder parameter. | If it is a dictionary, its entries can define a different | specification of fit order and channels for every field and | for every SPW. The dictionary can have up to as many entries | as fields are present in the MeasurementSet, and within each | field entry, as many entries as SPWs are present. Each SPW | entry specifies the polynomial order (in a 'fitorder' entry) | and the line-free channels (in a 'chan' entry). | If a field or SPW is not included, the default is assumed (all | channels, and polynomial order given in the global fitorder | parameter). | Default: '' (all channels for all SPWs for all fields) | Examples: | '17:100~500;600~910;1215~1678;1810~1903,19:7~100' | (The ranges of channels given for SPW 17 and 19 | are used for all the fields present in the | MeasurementSet.) | Dictionary to use different channels and polynomial | orders for different fields and SPWs, assuming: | 4 SPWs, with IDs 17, 19, 21, 23 | 6 fields, with IDs 0, 1, 4, 5, 6, 7 | {'0': {'17,19,21,23': {'chan': '', 'fitorder': 1}}, | '1': {'17': {'chan': '8~120', 'fitorder': 0}, | '19': {'chan': '7~100', 'fitorder': 1}, | '21': {'chan': '21:2~119', 'fitorder': 1}, | '23': {'chan': '', 'fitorder': 0}}, | '4,5,6': {'19': {'chan': '7~115', 'fitorder': 1}, | '21': {'chan': '', 'fitorder': 1}, | '23': {'chan': '', 'fitorder': 2}}, | '7': 'NONE'} | (Field 0: fit all SPWs and channels (same as not giving it), | but using polynomial order 1 for all SPWs. | Field 1: fit only some channel ranges (all for SPW 23), with | different polynomial orders. | Fields 4,5,6: fit only some channel ranges in SPW 19, all | channels for SPWs 21 and 23, and don't fit SPWs 17), | also using different polynomial orders. | Field 7: do not subtract continuum) | (The dictionary fields 'chan' can be given in channel | numbers, as in the example above, or in frequencies, using | the MSSelection syntax) | In the example, the empty string given for field 0 and all its | SPWs means that all the channels are fitted (and subtracted) | in all those SPWs. For the corner case where for a given field | no channels should be fitted in any SPW, the field entry can | be given as a string set to 'NONE'. | This specification of SPWs and channels uses a syntax similar | to the spw parameter. But in contrast to the spw parameter, | the SPWs and channels given in fitspec are not used to select | data but as a mask applied on the input data only for fitting | purposes. The channels that are not included in the | specification are masked out and not used as input by the | polynomial fitting algorithm. This is different from the spw | parameter, which selects the data that will be included in the | output. All channels present in the output are | continuum-subtracted, regardless of whether they were | specified as inputs to the fitting in fitspec .. _fitmethod: | ``fitmethod (string='gsl')`` - Choose fitting methods | Default: 'gsl' | Options: 'gsl', 'casacore' | Example: fitmethod='gsl' .. _fitorder: | ``fitorder (int=0)`` - Polynomial order for the fits | Default: 0 | Values of fitorder > 1 should be used with | care. Higher order polynomials are more flexible, | and may overfit and absorb line emission. They | also tend to go wild at the edges of the range of | channels used for fitting. .. _writemodel: | ``writemodel (bool=False)`` - Write fitted model into the MODEL column of the | output MeasurementSet. | Default: False | Options: True|False | By default the task creates an MS with the | continuum subtracted data in the DATA column. If | this option is enabled, the task will also write | the fitted model data into the MODEL column of the | output MS. """ pass