flagdata¶
-
flagdata
(vis, mode='manual', autocorr=False, inpfile='', reason='any', tbuff=0.0, spw='', field='', antenna='', uvrange='', timerange='', correlation='', scan='', intent='', array='', observation='', feed='', clipminmax='', datacolumn='DATA', clipoutside=True, channelavg=False, chanbin=1, timeavg=False, timebin='0s', clipzeros=False, quackinterval=1.0, quackmode='beg', quackincrement=False, tolerance=0.0, addantenna='', lowerlimit=0.0, upperlimit=90.0, ntime='scan', combinescans=False, timecutoff=4.0, freqcutoff=3.0, timefit='line', freqfit='poly', maxnpieces=7, flagdimension='freqtime', usewindowstats='none', halfwin=1, extendflags=True, winsize=3, timedev='', freqdev='', timedevscale=5.0, freqdevscale=5.0, spectralmax=1000000.0, spectralmin=0.0, antint_ref_antenna='', minchanfrac=0.6, verbose=False, extendpols=True, growtime=50.0, growfreq=50.0, growaround=False, flagneartime=False, flagnearfreq=False, minrel=0.0, maxrel=1.0, minabs=0, maxabs=- 1, spwchan=False, spwcorr=False, basecnt=False, fieldcnt=False, name='Summary', action='apply', display='', flagbackup=True, savepars=False, cmdreason='', outfile='', overwrite=True, writeflags=True)[source]¶ All-purpose flagging task based on data-selections and flagging modes/algorithms.
[Description] [Examples] [Development] [Details]
- Parameters
vis (path) - Name of input visibility file
mode (string=’manual’) - Flagging mode (list/manual/clip/quack/shadow/elevation/tfcrop/rflag/antint/extent/unflag/summary)
mode = manual
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
timerange ({string, stringVec}=’’) - Select data based on time range
correlation ({string, stringVec}=’’) - Select data based on correlation
scan ({string, stringVec}=’’) - Scan number range
intent ({string, stringVec}=’’) - Select observing intent
array ({string, stringVec}=’’) - (Sub)array numbers
uvrange ({string, stringVec}=’’) - Select data by baseline length.
observation ({string, int}=’’) - Select by observation ID(s)
feed ({string, stringVec}=’’) - Multi-feed numbers: Not yet implemented
autocorr (bool=False) - Flag only the auto-correlations?
mode = manualflag
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
timerange ({string, stringVec}=’’) - Select data based on time range
correlation ({string, stringVec}=’’) - Select data based on correlation
scan ({string, stringVec}=’’) - Scan number range
intent ({string, stringVec}=’’) - Select observing intent
array ({string, stringVec}=’’) - (Sub)array numbers
uvrange ({string, stringVec}=’’) - Select data by baseline length.
observation ({string, int}=’’) - Select by observation ID(s)
feed ({string, stringVec}=’’) - Multi-feed numbers: Not yet implemented
autocorr (bool=False) - Flag only the auto-correlations?
mode = list
mode = clip
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
timerange ({string, stringVec}=’’) - Select data based on time range
correlation ({string, stringVec}=’’) - Select data based on correlation
scan ({string, stringVec}=’’) - Scan number range
intent ({string, stringVec}=’’) - Select observing intent
array ({string, stringVec}=’’) - (Sub)array numbers
uvrange ({string, stringVec}=’’) - Select data by baseline length.
observation ({string, int}=’’) - Select by observation ID(s)
feed ({string, stringVec}=’’) - Multi-feed numbers: Not yet implemented
datacolumn ({string, stringVec}=’DATA’) - Data column on which to operate
clipminmax (doubleVec=’’) - Range to use for clipping
clipoutside ({bool, boolVec}=True) - Clip outside the range, or within it
channelavg ({bool, boolVec}=False) - Pre-average data across channels before analyzing visibilities for flagging
chanbin ({int, intVec}=1) - Bin width for channel average in number of input channels
timeavg ({bool, boolVec}=False) - Pre-average data across time before analyzing visibilities for flagging.
timebin (string=’0s’) - Bin width for time average in seconds
clipzeros (bool=False) - Clip zero-value data
mode = quack
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
timerange ({string, stringVec}=’’) - Select data based on time range
correlation ({string, stringVec}=’’) - Select data based on correlation
scan ({string, stringVec}=’’) - Scan number range
intent ({string, stringVec}=’’) - Select observing intent
array ({string, stringVec}=’’) - (Sub)array numbers
uvrange ({string, stringVec}=’’) - Select data by baseline length.
observation ({string, int}=’’) - Select by observation ID(s)
feed ({string, stringVec}=’’) - Multi-feed numbers: Not yet implemented
quackinterval ({double, doubleVec, int, intVec}=1.0) - Quack n seconds from scan beginning or end
quackmode ({string, stringVec}=’beg’) - Quack mode. Flag intervals of the scan according to given mode.
quackincrement ({bool, boolVec}=False) - Increment quack flagging in time taking into account flagged data or not.
mode = shadow
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
timerange ({string, stringVec}=’’) - Select data based on time range
correlation ({string, stringVec}=’’) - Select data based on correlation
scan ({string, stringVec}=’’) - Scan number range
intent ({string, stringVec}=’’) - Select observing intent
array ({string, stringVec}=’’) - (Sub)array numbers
uvrange ({string, stringVec}=’’) - Select data by baseline length.
observation ({string, int}=’’) - Select by observation ID(s)
feed ({string, stringVec}=’’) - Multi-feed numbers: Not yet implemented
tolerance (double=0.0) - Amount of shadow allowed (in meters)
addantenna ({string, record}=’’) - File name or dictionary with additional antenna names, positions and diameters
mode = elevation
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
timerange ({string, stringVec}=’’) - Select data based on time range
correlation ({string, stringVec}=’’) - Select data based on correlation
scan ({string, stringVec}=’’) - Scan number range
intent ({string, stringVec}=’’) - Select observing intent
array ({string, stringVec}=’’) - (Sub)array numbers
uvrange ({string, stringVec}=’’) - Select data by baseline length.
observation ({string, int}=’’) - Select by observation ID(s)
feed ({string, stringVec}=’’) - Multi-feed numbers: Not yet implemented
lowerlimit (double=0.0) - Lower limiting elevation (in degrees)
upperlimit (double=90.0) - Upper limiting elevation (in degrees)
mode = tfcrop
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
timerange ({string, stringVec}=’’) - Select data based on time range
correlation ({string, stringVec}=’’) - Select data based on correlation
scan ({string, stringVec}=’’) - Scan number range
intent ({string, stringVec}=’’) - Select observing intent
array ({string, stringVec}=’’) - (Sub)array numbers
uvrange ({string, stringVec}=’’) - Select data by baseline length.
observation ({string, int}=’’) - Select by observation ID(s)
feed ({string, stringVec}=’’) - Multi-feed numbers: Not yet implemented
ntime ({double, string}=’scan’) - Time-range to use for each chunk (in seconds or minutes)
combinescans (bool=False) - Accumulate data across scans depending on the value of ntime.
datacolumn ({string, stringVec}=’DATA’) - Data column on which to operate
timecutoff (double=4.0) - Flagging thresholds in units of deviation from the fit
freqcutoff (double=3.0) - Flagging thresholds in units of deviation from the fit
timefit (string=’line’) - Fitting function for the time direction (poly/line)
freqfit (string=’poly’) - Fitting function for the frequency direction (poly/line)
maxnpieces (int=7) - Number of pieces in the polynomial-fits (for freqfit or timefit poly)
flagdimension (string=’freqtime’) - Dimensions along which to calculate fits (freq, time, freqtime, timefreq)
usewindowstats (string=’none’) - Calculate additional flags using sliding window statistics (none, sum, std, both)
halfwin (int=1) - Half-width of sliding window to use with usewindowstats (1,2,3).
extendflags (bool=True) - Extend flags along time, frequency and correlation.
channelavg ({bool, boolVec}=False) - Pre-average data across channels before analyzing visibilities for flagging
chanbin ({int, intVec}=1) - Bin width for channel average in number of input channels
timeavg ({bool, boolVec}=False) - Pre-average data across time before analyzing visibilities for flagging.
timebin (string=’0s’) - Bin width for time average in seconds
mode = rflag
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
timerange ({string, stringVec}=’’) - Select data based on time range
correlation ({string, stringVec}=’’) - Select data based on correlation
scan ({string, stringVec}=’’) - Scan number range
intent ({string, stringVec}=’’) - Select observing intent
array ({string, stringVec}=’’) - (Sub)array numbers
uvrange ({string, stringVec}=’’) - Select data by baseline length.
observation ({string, int}=’’) - Select by observation ID(s)
feed ({string, stringVec}=’’) - Multi-feed numbers: Not yet implemented
ntime ({double, string}=’scan’) - Time-range to use for each chunk (in seconds or minutes)
combinescans (bool=False) - Accumulate data across scans depending on the value of ntime.
datacolumn ({string, stringVec}=’DATA’) - Data column on which to operate
winsize (int=3) - Number of timesteps in the sliding time window
timedev (variant=’’) - Time-series noise estimate
freqdev (variant=’’) - Spectral noise estimate
timedevscale (double=5.0) - Threshold scaling for timedev
freqdevscale (double=5.0) - Threshold scaling for freqdev.
spectralmax (double=1E6) - Flag whole spectrum if freqdev is greater than spectralmax
spectralmin (double=0.0) - Flag whole spectrum if freqdev is less than spectralmin
extendflags (bool=True) - Extend flags along time, frequency and correlation.
channelavg ({bool, boolVec}=False) - Pre-average data across channels before analyzing visibilities for flagging
chanbin ({int, intVec}=1) - Bin width for channel average in number of input channels
timeavg ({bool, boolVec}=False) - Pre-average data across time before analyzing visibilities for flagging.
timebin (string=’0s’) - Bin width for time average in seconds
mode = antint
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
datacolumn ({string, stringVec}=’DATA’) - Data column on which to operate
antint_ref_antenna (string=’’) - Antenna of interest. Baselines with this antenna will be checked for flagged channels.
minchanfrac (double=0.6) - Minimum fraction of flagged channels required for a baseline to be deemed as flagged
verbose (bool=False) - Print timestamps of flagged integrations to the log
mode = extend
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
timerange ({string, stringVec}=’’) - Select data based on time range
correlation ({string, stringVec}=’’) - Select data based on correlation
scan ({string, stringVec}=’’) - Scan number range
intent ({string, stringVec}=’’) - Select observing intent
array ({string, stringVec}=’’) - (Sub)array numbers
uvrange ({string, stringVec}=’’) - Select data by baseline length.
observation ({string, int}=’’) - Select by observation ID(s)
feed ({string, stringVec}=’’) - Multi-feed numbers: Not yet implemented
ntime ({double, string}=’scan’) - Time-range to use for each chunk (in seconds or minutes)
combinescans (bool=False) - Accumulate data across scans depending on the value of ntime.
extendpols (bool=True) - If any correlation is flagged, flag all correlations
growtime (double=50.0) - Flag all ntime integrations if more than X percent of the timerange is flagged (0-100)
growfreq (double=50.0) - Flag all selected channels if more than X percent of the frequency range is flagged (0-100)
growaround (bool=False) - Flag data based on surrounding flags
flagneartime (bool=False) - Flag one timestep before and after a flagged one (True/False)
flagnearfreq (bool=False) - Flag one channel before and after a flagged one (True/False)
mode = unflag
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
timerange ({string, stringVec}=’’) - Select data based on time range
correlation ({string, stringVec}=’’) - Select data based on correlation
scan ({string, stringVec}=’’) - Scan number range
intent ({string, stringVec}=’’) - Select observing intent
array ({string, stringVec}=’’) - (Sub)array numbers
uvrange ({string, stringVec}=’’) - Select data by baseline length.
observation ({string, int}=’’) - Select by observation ID(s)
feed ({string, stringVec}=’’) - Multi-feed numbers: Not yet implemented
mode = summary
field ({string, stringVec}=’’) - Select field using field id(s) or field name(s)
spw ({string, stringVec}=’’) - Select spectral window/channels
antenna ({string, stringVec}=’’) - Select data based on antenna/baseline
timerange ({string, stringVec}=’’) - Select data based on time range
correlation ({string, stringVec}=’’) - Select data based on correlation
scan ({string, stringVec}=’’) - Scan number range
intent ({string, stringVec}=’’) - Select observing intent
array ({string, stringVec}=’’) - (Sub)array numbers
uvrange ({string, stringVec}=’’) - Select data by baseline length.
observation ({string, int}=’’) - Select by observation ID(s)
feed ({string, stringVec}=’’) - Multi-feed numbers: Not yet implemented
minrel (double=0.0) - Minimum number of flags (relative)
maxrel (double=1.0) - Maximum number of flags (relative)
minabs (int=0) - Minimum number of flags (absolute)
maxabs (int=-1) - Maximum number of flags (absolute). Use a negative value to indicate infinity.
spwchan (bool=False) - Print summary of channels per spw
spwcorr (bool=False) - Print summary of correlation per spw
basecnt (bool=False) - Print summary counts per baseline
fieldcnt (bool=False) - Produce a separated breakdown for each field
name (string=’Summary’) - Name of this summary report (key in summary dictionary)
action (string=’apply’) - Action to perform in MS and/or in inpfile (none/apply/calculate)
action = apply
display (string=’’) - Display data and/or end-of-MS reports at runtime (data/report/both).
flagbackup (bool=True) - Back up the state of flags before the run
action = calculate
display (string=’’) - Display data and/or end-of-MS reports at runtime (data/report/both).
savepars (bool=False) - Save the current parameters to the FLAG_CMD table or to a file
- Returns
stats (dict) - flag counts, grouped by multiple axes, for mode=’summary’. RFlag frequency and time thresholds for mode=’rflag’ and action=’calculate’
- Description
This task can flag a MeasurementSet or a calibration table. It has two main types of operation. One type will read the parameters from the interface and flag using any of the various available modes. The other type will read the commands from a text file, a list of files or a Python list of strings, containing a list of flag commands (each line containing data selection parameters and any parameter specific for the mode being requested). Please see examples at the end of this help.
It is also possible to only save the parameters set in the interface without flagging. The parameters can be saved in the FLAG_CMD sub-table or in a text file. Note that when saving to an external file, the parameters will be appended to the given file.
The available flagging modes are: ‘manual’, ‘clip’, ‘shadow’, ‘quack’, ‘elevation’, ‘tfcrop’, ‘rflag’, ‘extend’, ‘unflag’ and ‘summary’. For automatic flagging, it is recommended to combine auto-flag modes with ‘extend’, via the list mode.
The current flags can be automatically backed up before applying new flags if the parameter flagbackup is set. Previous flag versions can be recovered using the flagmanager task.
Note
on flagging calibration tables:
flagdata can flag many types of calibration tables using mode=’manual’. It can only flag using the auto-flagging algorithms (‘clip’, ‘tfcrop’, or ‘rflag’), the cal tables that have the following data columns: CPARAM, FPARAM or SNR. The solution elements of the data columns are given in the correlation parameter using the names ‘Sol1’, ‘Sol2’, ‘Sol3’, or ‘Sol4’. See examples at the end of this help on how to flag different cal tables.
When the input is a calibration table, the modes ‘elevation’ and ‘shadow’ will be disabled. Data selection for calibration tables is limited to field, scan, timerange, antenna, spw and observation. It is only possible to save the parameters to an external file. If the calibration table was created before CASA 4.1, this task will create a dummy OBSERVATION column and OBSERVATION sub-table in the input calibration table to adapt it to the new cal table format.
Selecting antennas in some calibration tables have a different meaning compared to selecting the MS. Some calibration tables such as the antenna-based ones, created with some modes of gencal or polcal, have the ANTENNA2 column set to -1. This means that when selecting antenna=’ANT’, will select the whole ANT and not the cross-correlations between ANT and the other antennas. Similarly, the baseline syntax do not apply to this type of calibration tables. Those values with ampersand do not have any meaning when selecting antenna/baselines in antenna-based cal tables.
The task will flag a subset of data based on the following modes of operation:
‘list’ = list of flagging commands to apply to MS/cal table
‘manual’ = flagging based on specific selection parameters
‘clip’ = clip data according to values
‘quack’ = remove/keep specific time range at scan beginning/end
‘shadow’ = remove antenna-shadowed data
‘elevation’ = remove data below/above given elevations
‘tfcrop’ = automatic identification of outliers on the time-freq plane
‘rflag’ = automatic detection of outliers based on sliding-window RMS filters
‘antint’ = flag integrations if all baselines to a specified antenna are flagged
‘extend’ = extend and/or grow flags beyond what the basic algorithms detect
‘summary’ = report the amount of flagged data
‘unflag’ = unflag the specified data
A progress report line and a partial flagging summary is produced in the CASA logger every approximate 10% of the input data (since CASA 6.2), when the logger priority level is INFO (default). Additional messages are visible when setting a more detailed level of logging.
Tip
The partial or on-the-fly summaries are calculated from the flags as they are processed, and, as a consequence, they can report misleading counts under certain circumstances, for example when using the extend mode (see explanations below). Also, when using channel or time averaging the on-the-fly summaries can be difficult to interpret as they report flag counts for the on-the-fly binned data (see Flags and data averaging for an explanation of how this can distort the counts or percentages of flags). Accurate flag counts for the flags once written back to the MeasurementSet can be obtained afterwards via the ‘summary’ mode.
Parameter descriptions
vis
Name of input visibility file or calibration table. Default: ‘’ (none). Examples: vis=’uid___A002_X2a5c2f_X54.ms’, vis=’cal-X54.B1’
mode
Mode of operation. Options: ‘list’, ‘manual’, ‘clip’, ‘quack’, ‘shadow’, ‘elevation’, ‘tfcrop’, ‘extend’, ‘unflag’, ‘summary’. Default: ‘manual’
mode expandable parameters (except mode=’list’)
field
Select fields in mosaic. Use field id(s) or field name(s). [go listobs to obtain the list id’s or names] Default: ‘’ = all fields. If field string is a non-negative integer, it is assumed to be a field index otherwise, it is assumed to be a field name. Examples: field=’0~2’, field ids 0,1,2; field=’0,4,5~7’, field ids 0,4,5,6,7; field=’3C286,3C295’, field named 3C286 and 3C295; field = ‘3,4C*’, field id 3 and all names starting with 4C.
spw
Select data based on spectral window and channels. Default: ‘’ => all spectral windows and channels. Examples: spw=’0~2,4’, spectral windows 0,1,2,4 (all channels); spw=’0:5~61’, spw 0, channels 5 to 61; spw=’<2’, spectral windows less than 2 (i.e. 0,1); spw=’0,10,3:3~45’, spw 0,10 all channels, spw 3, channels 3 to 45; spw=’0~2:2~6’; spw 0,1,2 with channels 2 through 6 in each; spw=’0:0~10;15~60’; spectral window 0 with channels 0-10,15-60; spw=’0:0~10,1:20~30,2:1;2;3’; spw 0, channels 0-10, spw 1, channels 20-30, and spw 2, channels, 1,2 and 3.
Note
For modes ‘clip’, ‘tfcrop’, and ‘rflag’, channel-ranges can be excluded from flagging by leaving them out of the selection range. This is a way to protect known spectral-lines from being flagged by the autoflag algorithms.
antenna
Select data based on baseline. Default: ‘’ (all). Examples: antenna=’DV04&DV06’ baseline DV04-DV06; antenna=’DV04&DV06;DV07&DV10’ baselines DV04-DV06 and DV07-DV10; antenna=’DV06’ all cross-correlation baselines between antenna DV06 and all other available antennas; antenna=’DV04,DV06’ all baselines with antennas DV04 and DV06; antenna=’DV06&&DV06’ only the auto-correlation baselines for antenna DV06; antenna=’DV04&&*’ cross and auto-correlation baselines between antenna DV04 and all other available antennas; antenna=’0~2&&&’ only the auto-correlation baselines for antennas in range 0~2
Note
For some antenna-based calibration tables, selecting baselines with the & syntax do not apply.
timerange
Select data based on time range. Default: ‘’ (all). Examples: timerange = ‘YYYY/MM/DD/hh:mm:ss~YYYY/MM/DD/hh:mm:ss’;
Note
if YYYY/MM/DD is missing date defaults to first day in data set.
timerange=’09:14:0~09:54:0’ picks 40 min on first day; timerange=’25:00:00~27:30:00’ picks 1 hr to 3 hr 30min on NEXT day; timerange=’09:44:00’ pick data within one integration of time; timerange=’>10:24:00’ data after this time.
correlation
Correlation types or expression. Default: ‘’ (all correlations). For modes clip, tfcrop or rflag, the default means ABS_ALL. If the input is cal table that does not contain a complex data column, the default will fall back to REAL_ALL. Examples: correlation=’XX,YY’ or options: Any of ‘ABS’, ‘ARG’, ‘REAL’, ‘IMAG’, ‘NORM’ followed by any of ‘ALL’, ‘I’, ‘XX’, ‘YY’, ‘RR’, ‘LL’, ‘WVR’. ‘WVR’ refers to the water vapour radiometer of ALMA data. For calibration tables, the solutions are: ‘Sol1’, ‘Sol2’, Sol3, Sol4. Correlation selection is not supported for modes other than ‘clip’, ‘tfcrop’, or ‘rflag’ in cal tables.
Note
The operators ABS, ARG, REAL, etc. are written only once as the first value. If more than one correlation is given, the operator will be applied to all of them. The expression is used only in modes ‘clip’, ‘tfcrop’, and ‘rflag’.
scan
Scan number range. Default: ‘’ (all). Examples: scan=’1~5’. Check ‘go listobs’ to insure the scan numbers are in order.
intent
Select data based on scan intent. Intent selection is not supported for cal tables. Default: ‘’ (all). Examples: intent=’*CAL,*BAND*’*
array
Selection based on the antenna array. Array selection is not supported for cal tables. Default: ‘’ (all).
uvrange
Select data within uvrange (default units meters). Default: ‘’ (all). Examples: uvrange=’0~1000klambda’, uvrange from 0-1000 kilo-lambda; uvrange=’>4klambda’, uvranges greater than 4 kilo lambda. uvrange selection is not supported for cal tables.
observation
Selection based on the observation ID. Default: ‘’ (all). Examples: observation=’1’ or observation=1
feed
Selection based on the feed - NOT IMPLEMENTED YET
mode=’manual’ expandable parameters
Flag according to the data selection specified. This is the default mode (used when the mode is not specified).
autocorr
Flag only the auto-correlations. Note that this parameter is only active when set to True. If set to False it does NOT mean “do not flag auto-correlations”. When set to True, it will only flag data from a processor of type CORRELATOR. Default: False. Otions: True, False
mode=’list’ expandable parameters
Flag according to the data selection and flag commands specified in the input list. The input list may come from a text file, a list of text files or from a Python list of strings. Each input line may contain data selection parameters and any parameter specific to the mode given in the line. Default values will be used for the parameters that are not present in the line. Each line will be taken as a command to the task. If data is pre-selected using any of the selection parameters, then flagging will apply only to that subset of the MS.
For optimization and whenever possible, the task will create a union of the data selection parameters present in the list and select only that portion of the MS.
Note
The flag commands will be applied only when action=’apply’. If action=’calculate’ the flags will be calculated, but not applied. This is useful if display is set to something other than ‘none’. If action=’’ or ‘none’, the flag commands will not be applied either. An empty action is useful only to save the parameters of the list to a file or to the FLAG_CMD sub-table.
NOTE2: quackincrement = True works based on the state of prior flagging, and unless it is the first item in the list the agent doing the quacking in list mode doesn’t know about the state of prior flags. In this case, the command with quackincrement=True will be ignored and the task will issue a WARNING.
inpfile
Input ASCII file, list of files or a Python list of command strings. Default: ‘’. Options: [ ] with flag commands or [ ] with filenames or ‘ ‘ with a filename.
Warning
IMPORTANT: From CASA 4.3 onwards, the parser will be strict and accept only valid flagdata parameters in the list. It will check each parameter name and type and exit with an error if any of them is wrong. String values must contain quotes around them or the parser will not work. The parser evaluates the commands in the list and considers only existing Python types.
Note
There should be no whitespace between KEY=VALUE since the parser first breaks command lines on whitespace, then on “=”. Use only one whitespace to separate the parameters (no commas). Scroll down to the bottom to see a detailed description of the input list syntax..
Example1: The following commands can be saved to a file or group of files and given to the task (e.g., save it to ‘flags.txt’):
scan='1~3' mode='manual' mode='clip' clipminmax=[0,2] correlation='ABS_XX' clipoutside=False spw='9' mode='tfcrop' correlation='ABS_YY' ntime=51.0 mode='extend' extendpols=True
flagdata(vis, mode='list', inpfile='flags.txt')
or
flagdata(vis, mode='list', inpfile=['onlineflags.txt','otherflags.txt'])
Example2: The same commands can be given in a Python list on the command line to the task.
cmd=["scan='1~3' mode='manual'", "mode='clip' clipminmax=[0,2] correlation='ABS_XX' clipoutside=False", "spw='9' mode='tfcrop' correlation='ABS_YY' ntime=51.0", "mode='extend' extendpols=True"] flagdata(vis,mode='list',inpfile=cmd)
reason
Select flag commands based on REASON(s). Can be a string, or list of strings. If inpfile is a list of files, the reasons given in this parameter will apply to all the files. Default: ‘any’ (all flags regardless of reason). Examples: reason=’FOCUS_ERROR’; reason=[‘FOCUS_ERROR’, ‘SUBREFLECTOR_ERROR’]
Note
NOTE: what is within the string is literally matched, e.g. reason=’’ matches only blank reasons, and reason = ‘FOCUS_ERROR, SUBREFLECTOR_ERROR’ matches this compound reason string only. See the syntax for writing flag commands at the end of this help.
tbuff
A time buffer or list of time buffers to pad the timerange parameters in flag commands. When a list of 2 time buffers is given, it will subtract the first value from the lower time and the second value will be added to the upper time in the range. The 2 time buffer values can be different, allowing to have an irregular time buffer padding to time ranges. If the list contains only one time buffer, it will use it to subtract from t0 and add to t1. If more than one list of input files is given, tbuff will apply to all of the flag commands that have timerange parameters in the files.
Each tbuff value should be a float number given in seconds. Default: 0.0 (it will not apply any time padding). Example: tbuff=[0.5, 0.8] inpfile=[‘online.txt’,’userflags.txt’]. The timerange parameters in the ‘online.txt’ file are first converted to seconds. Then, 0.5 is subtracted from t0 and 0.8 is added to t1, where t0 and t1 are the two intervals given in timerange. Similarly, tbuff will be applied to any timerange in ‘userflags.txt’.
Warning
IMPORTANT: This parameter assumes that timerange = t0 ~ t1, therefore it will not work if only t0 or t1 is given.
Note
The most common use-case for tbuff is to apply the online flags that are created by importasdm when savecmds=True. The value of a regular time buffer should be tbuff=0.5*max (integration time).
mode=’clip’ expandable parameters
Clip data according to values of the following subparameters. The polarization expression is given by the correlation parameter. For calibration tables, the solutions are also given by the correlation parameter.
clipminmax
Range of data (Jy) that will NOT be flagged. It will always flag the NaN/Inf data, even when a range is specified. Default: [ ]. Example: clipminmax=[0.0,1.5]
clipoutside
Clip OUTSIDE the range. Default: True. Example: clipoutside=False, flag data WITHIN the clipminmax range.
clipzeros
Clip zero-value data. Default: False.
mode=’clip’, ‘tfcrop’, or ‘rflag’ expandable parameters
datacolumn
Column to use for clipping. Default: ‘DATA’. Options: MS columns: ‘DATA’, ‘CORRECTED’, ‘MODEL’, ‘RESIDUAL’, ‘RESIDUAL_DATA’, ‘WEIGHT_SPECTRUM’, ‘WEIGHT’, ‘FLOAT_DATA’. Cal table columns: ‘FPARAM’, ‘CPARAM’, ‘SNR’, ‘WEIGHT’.
Note
RESIDUAL = CORRECTED - MODEL
RESIDUAL_DATA = DATA - MODEL
When datacolumn is WEIGHT, the task will internally use WEIGHT_SPECTRUM. If WEIGHT_SPECTRUM does not exist, it will create one on-the-fly based on the values of WEIGHT.
channelavg
Pre-average data across channels before analyzing visibilities for flagging. Partially flagged data is not be included in the average unless all data contributing to a given output channel is flagged. If present, WEIGHT_SPECTRUM/ SIGMA_SPECTRUM are used to compute a weighted average (WEIGHT_SPECTRUM for CORRECTED_DATA and SIGMA_SPECTRUM for DATA). Default: False. Options: True/False
Note
NOTE1: Pre-average across channels is meant to be used with the auto-flagging methods (clip, tfcrop, rflag) only. In list mode, if channelavg is enabled and any other method than clip, tfcrop, rflag is used, that is forbidden and flagdata will produce an error message and stop. The same applies to timeavg.
NOTE2: Pre-average across channels is not supported for calibration tables.
chanbin
Bin width for channel average in number of input channels. If a list is given, each bin applies to one of the selected SPWs. When chanbin is set to 1 all input channels are used for the average to produce a single output channel, this behaviour aims to preserve backwards compatibility with the previous pre-averaging feature of clip mode. Default: 1
timeavg
Pre-average data across time before analyzing visibilities for flagging. Partially flagged data is not be included in the average unless all data contributing to a given output channel is flagged. If present, WEIGHT_SPECTRUM/ SIGMA_SPECTRUM are used to compute a weighted average (WEIGHT_SPECTRUM for CORRECTED_DATA and SIGMA_SPECTRUM for DATA). Otherwise WEIGHT/ SIGMA are used to average together data from different integrations. Default: False. Options: True/False
Note
NOTE1: Pre-average across time is meant to be used with the auto-flagging methods (clip, tfcrop, rflag) only. In list mode, if timeavg is enabled and any other method than clip, tfcrop, rflag is used, that is forbidden and flagdata will produce an error message and stop. The same applies to channelavg.
NOTE2: Pre-average across time is not supported for calibration tables
timebin
Bin width for time average in seconds. Default: ‘0s’
Note
The auto-flagging methods (clip, tfcrop, rflag) can be used together with timeavg and channelavg, and other methods. But when one of the auto-flagging methods are employed and timeavg, channelavg (or both) are enabled the set of other methods or agents that can be used simultaneously is limited to the following ones: extendflags, antint, and the display=’data’ GUI.
display=’data’ can be added as a parameter in the flagdata call. extendflags can be added either in the flagdata call (as a subparameter of tfcrop or rflag) or in the list of commands in list mode. antint can only be added in the list of commands in list mode, as there is no subparameter of clip, rflag, or tfcrop for this.
mode=’quack’ expandable parameters
Option to remove specified part of scan beginning/end.
quackinterval
Time in seconds from scan beginning or end to flag. Make time slightly smaller than the desired time. Default: 0.0. Type: int or float.
quackmode
Quack mode. Default: ‘beg’. Options:
‘beg’ ==> flag an interval at the beginning of scan
‘endb’ ==> flag an interval at the end of scan
‘tail’ ==> flag all but an interval at the beginning of scan
‘end’ ==> flag all but an interval at end of scan
Visual representation of quack mode flagging one scan with 1s duration. The following diagram shows what is flagged for each quack mode when quackinterval is set to 0.25s. The flagged part is represented by crosses (+++++++++):
scan with 1s duration -------------------------------------------- beg +++++++++++--------------------------------- endb ---------------------------------+++++++++++ tail -----------+++++++++++++++++++++++++++++++++ end +++++++++++++++++++++++++++++++++-----------
quackincrement
Increment quack flagging in time taking into account flagged data or not. Default: False. Type: bool
False ==> the quack interval is counted from the scan boundaries, as determined by the quackmode parameter, regardless if data has been flagged or not.
True ==> the quack interval is counted from the first unflagged data in the scan.
Warning
quackincrement = True works based on the state of prior flagging, and unless it is the first item in the list the agent doing the quacking in list mode doesn’t know about the state of prior flags. In this case, the command with quackincrement=True will be ignored and the task will issue a WARNING.
mode=’shadow’ expandable parameters
Option to flag data of shadowed antennas. This mode is not available for cal tables.
All antennas in the ANTENNA subtable of the MS (and the corresponding diameters) will be considered for shadow-flag calculations. For a given timestep, an antenna is flagged if any of its baselines (projected onto the uv-plane) is shorter than radius \(_{1}\) \(+\) radius \(_{2}\) \(-\) tolerance. The value of ‘w’ is used to determine which antenna is behind the other.
The uvw values of the baselines are defined as follows:
For the baselines present in the data, the uvw values are taken from the UVW column of the corresponding data rows.
If one or more baselines are not present in the data for a given timestep, the baseline uvw are additionally calculated using the phase-reference center for antenna-pointing direction. In this second case, both the antenna positions (expected in ITRF spatial coordinate frame) and the phase-reference center are converted to J2000 coordinate frame, and the UVW are calculated using that frame.
Shadow mode does not flag rows where the two antennas of the baseline are the same, or in other words, antennas do not shadow themselves.
tolerance
Amount of shadowing allowed (or tolerated), in meters. A positive number allows antennas to overlap in projection. A negative number forces antennas apart in projection. Zero implies a distance of radius \(_{1}\) \(+\) radius \(_{2}\) between antenna centers. Default: 0.0
addantenna
It can be either a file name with additional antenna names, positions and diameters, or a Python dictionary with the same information. You can use the flaghelper functions to create the dictionary from a file. Default: ‘’. Type: string or {} (dictionary). To create a dictionary inside CASA:
import flaghelper as fh antdic = fh.readAntennaList(antfile)
Where antfile is a text file in disk that contains information such as:
name=VLA01 diameter=25.0 position=[-1601144.96146691, -5041998.01971858, 3554864.76811967] name=VLA02 diameter=25.0 position=[-1601105.7664601889, -5042022.3917835914, 3554847.245159178]
mode=’elevation’ expandable parameters
Option to flag based on antenna elevation. This mode is not available for cal tables.
lowerlimit
Lower limiting elevation in degrees. Data coming from a baseline where one or both antennas were pointing at a strictly lower elevation (as function of time), will be flagged. Default: 0.0
upperlimit
Upper limiting elevation in degrees. Data coming from a baseline where one or both antennas were pointing at a strictly higher elevation (as function of time), will be flagged. Default: 90.0
mode=’tfcrop’, ‘rflag’,* or *’extend’ expandable parameters
ntime
Time range (in seconds or minutes) over which to buffer data before running the algorithm. Options: ‘scan’ or any other float value or string containing the units. Default: ‘scan’. Examples: ntime=’1.5min’; ntime=1.2 (taken in seconds). The dataset will be iterated through in time-chunks defined here.
Warning
WARNING: If ntime=’scan’ and combinescans=True, all the scans will be loaded at once, thus requesting a lot of memory depending on the available spws.
combinescans
Accumulate data across scans depending on the value of ntime. Default: False. This parameter should be set to True only when ntime is specified as a time-interval (not ‘scan’). When set to True, it will remove SCAN from the sorting columns, therefore it will only accumulate across scans if ntime is not set to ‘scan’.
mode=’tfcrop’ expandable parameters
Flag using the TFCrop autoflag algorithm. For each field, spw, timerange (specified by ntime), and baseline:
Average visibility amplitudes along time dimension to form an average spectrum
Calculate a robust piece-wise polynomial fit for the band-shape at the base of RFI spikes. Calculate ‘stddev’ of (data - fit).
Flag points deviating from the fit by more than N-stddev
Repeat (1-3) along the other dimension.
This algorithm is designed to operate on un-calibrated data (step (2)), as well as calibrated data. It is recommended to extend the flags after running this algorithm. See the sub-parameter extendflags below.
timecutoff
Flag threshold in time. Flag all data-points further than N-stddev from the fit. This threshold catches time-varying RFI spikes (narrow and broad-band), but will not catch RFI that is persistent in time. Default: 4.0.
Flagging is done in up to 5 iterations. The stddev calculation is adaptive and converges to a value that reflects only the data and no RFI. At each iteration, the same relative threshold is applied to detect flags. (Step (3) of the algorithm).
freqcutoff
Flag threshold in frequency. Flag all data-points further than N-stddev from the fit. Same as timecutoff, but along the frequency-dimension. This threshold catches narrow-band RFI that may or may not be persistent in time. Default: 3.0
timefit
Fitting function for the time direction. Default: ‘line’. Options: ‘line’, ‘poly’
A ‘line’ fit is a robust straight-line fit across the entire timerange (defined by ntime). A ‘poly’ fit is a robust piece-wise polynomial fit across the timerange.
Note
A robust fit is computed in upto 5 iterations. At each iteration, the stddev between the data and the fit is computed, values beyond N-stddev are flagged, and the fit and stddev are re-calculated with the remaining points. This stddev calculation is adaptive, and converges to a value that reflects only the data and no RFI. It also provides a varying set of flagging thresholds, that allows deep flagging only when the fit best represents the true data. Choose ‘poly’ only if the visibilities are expected to vary significantly over the timerange selected by ntime, or if there is a lot of strong but intermittent RFI.
freqfit
Fitting function for the frequency direction. Same as for the timefit parameter. Default: ‘poly’. Options: ‘line’, ‘poly’. Choose ‘line’ only if you are operating on bandpass-corrected data, or residuals, and expect that the bandshape is linear. The ‘poly’ option works better on uncalibrated bandpasses with narrow-band RFI spikes.
maxnpieces
Maxinum number of pieces to allow in the piecewise-polynomial fits. Default: 7. Options: 1 - 9. This parameter is used only if timefit or freqfit are chosen as ‘poly’. If there is significant broad-band RFI, reduce this number. Using too many pieces could result in the RFI being fitted in the clean bandpass. In later stages of the fit, a third-order polynomial is fit per piece, so for best results, please ensure that nchan/maxnpieces is at-least 10.
flagdimension
Choose the directions along which to perform flagging. Default: ‘freqtime’; first flag along frequency, and then along time. Options: ‘time’, ‘freq’, ‘timefreq’, ‘freqtime’. For most cases, ‘freqtime’ or ‘timefreq’ are appropriate, and differences between these choices are apparant only if RFI in one dimension is significantly stronger than the other. The goal is to flag the dominant RFI first. If there are very few (less than 5) channels of data, then choose ‘time’. Similarly for ‘freq’.
usewindowstats
Use sliding-window statistics to find additional flags. Default: ‘none’. Options: ‘none’, ‘sum’, ‘std’, ‘both’
Warning
This parameter is experimental!
The ‘sum’ option chooses to flag a point, if the mean-value in a window centered on that point deviates from the fit by more than N-stddev \(/ 2.0\).
Note
stddev is calculated between the data and fit as explained in Step (2). This option is an attempt to catch broad-band or time-persistent RFI that the above polynomial fits will mistakenly fit as the clean band. It is an approximation to the sumThreshold method found to be effective by Offringa et.al (2010) for LOFAR data.
The ‘std’ option chooses to flag a point, if the ‘local’ stddev calculated in a window centered on that point is larger than N-stddev \(/2.0\). This option is an attempt to catch noisy RFI that is not excluded in the polynomial fits, and which increases the global stddev, and results in fewer flags (based on the N-stddev threshold).
halfwin
Half width of sliding window to use with usewindowstats. Default: 1 (a 3-point window size). Options: 1,2,3
Warning
This is experimental!
mode=’tfcrop’ or ‘rflag’ expandable parameters
extendflags
Extend flags along time, frequency and correlation. Default: True
Note
It is usually helpful to extend the flags along time, frequency, and correlation using this parameter, which will run the ‘extend’ mode after ‘tfcrop’ and extend the flags if more than 50% of the timeranges are already flagged, and if more than 80% of the channels are already flagged. It will also extend the flags to the other polarizations. The user may also set extendflags to False and run the ‘extend’ mode in a second step within the same flagging run. See the example below.
mode=’rflag’ expandable parameters
Detect outliers based on the RFlag algorithm [1]. The polarization expression is given by the correlation parameter. Iterate through the data in chunks of time. For each chunk, calculate local statistics, and apply flags based on user supplied (or auto-calculated) thresholds.
Time analysis (for each channel):
calculate local RMS of real and imaginary visibilities within a sliding time window
calculate the median RMS across time windows, deviations of local RMS from this median, and the median deviation
flag if local RMS is larger than timedevscale \(x\) (medianRMS \(+\) medianDev)
Spectral analysis (for each time):
calculate avg of real and imaginary visibilities and their RMS across channels
calculate the deviation of each channel from this avg, and the median-deviation
flag if deviation is larger than freqdevscale \(x\) medianDev
It is recommended to extend the flags after running this algorithm. See the sub-parameter extendflags below.
Notice that by default the flag implementation in CASA is able to calculate the thresholds and apply them on-the-fly (OTF). There is a significant performance gain with this approach, as the visibilities don’t have to be read twice, and therefore is highly recommended (see example 1). Otherwise it is possible to reproduce the AIPS usage pattern by doing a first run with action=’calculate’ and a second run with action=’apply’. The advantage of this approach is that the thresholds are calculated using the data from all scans, instead of calculating them for one scan only (see example 3).
Example usage :
Calculate thresholds automatically per scan, and use them to find flags. Specify scale-factor for time-analysis thresholds, use default for frequency.
flagdata('my.ms', mode='rflag', spw='9', timedevscale=4.0)
Supply noise-estimates to be used with default scale-factors.
flagdata(vis='my.ms', mode='rflag', spw='9', timedev=0.1, freqdev=0.5, action='calculate')
Two-passes. This replicates the usage pattern in AIPS.
The first pass saves commands in output text files, with auto-calculated thresholds. Thresholds are returned from ‘rflag’ only when action=’calculate’. The user can edit this file before doing the second pass, but the python-dictionary structure must be preserved. The parameters timedevscale and freqdevscale are not used in this first pass.
The second pass applies these commands (action=’apply’).
flagdata(vis='my.ms', mode='rflag', spw='9,10', timedev='tdevfile.txt', freqdev='fdevfile.txt', action='calculate')
flagdata(vis='my.ms', mode='rflag', spw='9,10', timedev='tdevfile.txt', freqdev='fdevfile.txt', action='apply')
With action=’calculate’, display=’report’ will produce diagnostic plots showing data-statistics and thresholds (the same thresholds as those written out to ‘tdevfile.txt’ and ‘fdevfile.txt’). In this second pass, with action=’apply’, the parameters freqdevscale and timedevscale can be used to re-scale the thresholds calculated in the first pass.
Note
NOTE1: The RFlag algorithm was originally developed by Eric Greisen in AIPS [1] .
NOTE2: Since this algorithm operates with two passes through each chunk of data (time and freq axes), some data points get flagged twice. This can affect the flag-percentage estimate printed in the logger at runtime. An accurate estimate can be obtained via the ‘summary’ mode.
NOTE3: RFlag calculates statistics across all selected correlations. Therefore, if there is a significant amplitude difference between parallel-hand and cross-hand correlations, or between different solutions in a gain table, it is advisable to pre-select subsets of correlations (or sols) on which to run one instance of RFlag. For example, correlation=’RR,LL’ or correlation=’ABS sol1,sol2’.
Note
Dictionaries returned by action=’calculate’. Rflag with action=’calculate’ (the first pass of the two-passes usage) can return a dictionary. The dictionary holds the freqdev and timedev thresholds calculated in that first pass. For example:
thresholds = flagdata(vis=’my.ms’, mode=’rflag’, action=’calculate’)
print(thresholds)
{‘type’: ‘list’, ‘report0’: {‘type’: ‘rflag’, ‘freqdev’: array([[ 1.0e+00, 0.0e+00, 3.13e-02], … , ‘name’: ‘Rflag’, ‘timedev’: array([[ 1.0e+00, 0.0e+00, 6.8e-03], … ])}, ‘nreport’: 1}
The timedev and freqdev items from this dictionary can be used in the second pass call to flagdata, but their respective values need to be passed as separate parameters. For example:
flagdata(vis=ms, mode=’rflag’, action=’apply’, timedev=thresholds[‘report0’][‘timedev’], freqdev=thresholds[‘report0’][‘freqdev’])
This is an alternative approach (and fully equivalent) to using two files to save and reuse the timedev and freqdev values.
winsize
Number of timesteps in the sliding time window (fparm(1) in AIPS). Default: 3
timedev
Time-series noise estimate (noise in AIPS). Default: [ ]. Examples: timedev = 0.5: Use this noise-estimate to calculate flags. Do not recalculate; timedev = [[1,9,0.2], [1,10,0.5]]: Use noise-estimate of 0.2 for field 1, spw 9, and noise-estimate of 0.5 for field 1, spw 10; timedev = [ ]: Auto-calculate noise estimates; timedev=’timedevfile’: Auto-calculate noise estimates and write them into a file with the name given (any string will be interpreted as a file name which will be checked for existence).
freqdev
Spectral noise estimate (scutoff in AIPS). This step depends on having a relatively-flat bandshape. Same parameter-options as timedev. Default: [ ]
timedevscale
For Step 1 (time analysis), flag a point if local RMS around it is larger than timedevscale \(x\) timedev (fparm(0) in AIPS). This scale parameter is only applied when flagging (action=’apply’) and displaying reports (display option). It is not used when the thresholds are simply calculated and saved into files (action=’calculate’, as in the two-passes usage pattern of AIPS). Default: 5.0
freqdevscale
For Step 2 (spectral analysis), flag a point if local rms around it is larger than freqdevscale \(x\) freqdev (fparm(10) in AIPS). Similarly as with timedevscale, freqdevscale is not used when the thresholds are simply calculated and saved into files (action=’calculate’, as in the two-passes usage pattern of AIPS). Default: 5.0
spectralmax
Flag whole spectrum if freqdev is greater than spectralmax (fparm(6) in AIPS). Default: 1E6
spectralmin
Flag whole spectrum if freqdev is less than spectralmin (fparm(5) in AIPS). Default: 0.0
mode=’extend’ expandable parameters
Extend and/or grow flags beyond what the basic algorithms detect. This mode will extend the accumulated flags available in the MS, regardless of which algorithm created them. It is recommended that any autoflag (tfcrop, rflag) algorithm be followed up by a flag extension. Extensions will apply only within the selected data, according to the settings of extendpols, growtime, growfreq, growaround, flagneartime, and flagnearfreq.
Note
Runtime summary counts in the logger can sometimes report larger flag percentages than what is actually flagged. This is because extensions onto already-flagged data-points are counted as new flags. An accurate flag count can be obtained via the ‘summary’ mode.
extendpols
Extend flags to all selected correlations. Default: True. Options: True/False. For example, to extend flags from RR to only RL and LR, a data-selection of correlation=’RR,LR,RL’ is required along with extendpols=True.
growtime
For any channel, flag the entire timerange in the current 2D chunk (set by ntime) if more than X% of the timerange is already flagged. Default: 50.0. Options: 0.0 - 100.0. This option catches the low-intensity parts of time-persistent RFI.
growfreq
For any timestep, flag all channels in the current 2D chunk (set by data-selection) if more than X% of the channels are already flagged. Default: 50.0. Options: 0.0 - 100.0. This option catches broad-band RFI that is partially identified by earlier steps.
growaround
Flag a point based on the number of flagged points around it. Default: False. Options: True/False. For every un-flagged point on the 2D time/freq plane, if more than four surrounding points are already flagged, flag that point. This option catches some wings of strong RFI spikes.
flagneartime
Flag points before and after every flagged one, in the time-direction. Default: False. Options: True/False. Note that this can result in excessive flagging.
flagnearfreq
Flag points before and after every flagged one, in the frequency-direction. Default: False. Options: True/False. This option allows flagging of wings in the spectral response of strong RFI. Note that this can result in excessive flagging.
mode=’antint’ expandable parameters
This mode flag all integrations in which a specified antenna is flagged. This mode operates for an spectral window. It flags any integration in which all baselines to a specified antenna are flagged, but only if this condition is satisfied in a fraction of channels within the spectral window of interest greater than a nominated fraction. For simplicity, it assumes that all polarization products must be unflagged for a baseline to be deemed unflagged. The antint mode implements the flagging approach introduced in ‘antintflag’ (https://doi.org/10.5281/zenodo.163546)
The motivating application for introducing this mode is removal of data that will otherwise lead to changes in reference antenna during gain calibration, which will in turn lead to corrupted polarization calibration.
antint_ref_antenna
Check the baselines to this antenna. Note that this is not the same as the general ‘antenna’ parameter of flagdata. The parameter antint_ref_antenna is mandatory with the ‘antint’ mode and chooses the antenna for which the fraction of channels flagged will be checked.
minchanfrac
Minimum fraction of flagged channels required for a baseline to be deemed as flagged. Takes values between 0-1 (float). In this mode flagdata does the following for every point in time. It checks the fraction of channels flagged for any of the polarization products and for every baseline to the antenna of interest. If the fraction is higher than this ‘minchanfrac’ threshold then the data are flagged for this pont in time (this includes all the rows selected with the flagdata command that have that timestamp). This parameter will be ignored if spw specifies a channel.
verbose
Print timestamps of flagged integrations to the log.
mode=’unflag’ expandable parameters
Unflag according to the data selection specified.
mode=’summary’ expandable parameters
List the number of rows and flagged data points for the MS’s meta-data. The resulting summary will be returned as a Python dictionary.
In ‘summary’ mode, the task returns a dictionary of flagging statistics.
Example1:
s = flagdata(..., mode='summary')
s will be a dictionary which contains:
s[‘total’]: total number of data
s[‘flagged’]: amount of flagged data
Example2: two summary commands in ‘list’ mode, intercalating a manual flagging command.
s = flagdata(..., mode='list', inpfile=["mode='summary' name='InitFlags'", "mode='manual' autocorr=True", "mode='summary' name='Autocorr'"])
The dictionary returned in s will contain two dictionaries, one for each of the two summary modes.
s[‘report0’][‘name’]: ‘InitFlags’
s[‘report1’][‘name’]: ‘Autocorr’
minrel
Minimum number of flags (relative) to include in histogram. Default: 0.0
maxrel
Maximum number of flags (relative) to include in histogram. Default: 1.0
minabs
Minimum number of flags (absolute, inclusive) to include in histogram. Default: 0
maxabs
Maximum number of flags (absolute, inclusive) to include in histogram. To indicate infinity, use any negative number. Default: -1
spwchan
List the number of flags per spw and per channel. Default: False
spwcorr
Llist the number of flags per spw and per correlation. Default: False
basecnt
List the number of flags per baseline. Default: False
fieldcnt
Produce a separated breakdown per field. Default: False
name
Name for this summary, to be used as a key in the returned Python dictionary. It is possible to call the ‘summary’ mode multiple times in ‘list’ mode. When calling the ‘summary’ mode as a command in a list, one can give different names to each one of them so that they can be easily pulled out of the summary’s dictionary. Default: ‘Summary’
action
Action to perform in MS/cal table or in the input list of parameters. Options: ‘none’, ‘apply’, ‘calculate’. Default: ‘apply’
action=’apply’ or ‘calculate’ expandable parameters
action=’apply’ applies the flags to the MS. action=’calculate’ only calculates the flags but does not write them to the MS. This is useful if used together with the display to analyze the results before writing to the MS.
display
Display data and/or end-of-MS reports at run-time. It needs to read a datacolumn for the plotting. The default for an MS is DATA, but the task will use FLOAT_DATA for a Single-dish MS. Default: ‘none’. Options: ‘none’, ‘data’, ‘report’, ‘both’
‘none’ –> It will not display anything. ‘data’ –> display data and flags per-chunk at run-time, within an interactive GUI.
This option opens a GUI to show the 2D time-freq planes of the data with old and new flags, for all correlations per baseline.
The GUI allows stepping through all baselines (prev/next) in the current chunk (set by ntime), and stepping to the next-chunk.
The flagdata task can be quit from the GUI, in case it becomes obvious that the current set of parameters is just wrong.
There is an option to stop the display but continue flagging.
‘report’ –> displays end-of-MS reports on the screen. ‘both’ –> displays data per chunk and end-of-MS reports on the screen
action=’apply’ expandable parameters
flagbackup
Automatically backup flags before running the tool. Flagversion names are chosen automatically, and are based on the mode being used. Default: True. Options: True/False
action=’’ or ‘none’ description
When set to empty or ‘none’, the underlying tool will not be executed and no flags will be produced. No data selection will be done either. This is useful when used together with the parameter savepars to only save the current parameters (or list of parameters) to the FLAG_CMD sub-table or to an external file.
savepars
Save the current parameters to the FLAG_CMD table of the MS or to an output text file.
Note that when display is set to anything other than ‘none’, savepars will be disabled. This is done because in an interactive mode, the user may skip data which may invalidate the initial input parameters and there is no way to save the interactive commands. When the input is a calibration table it is only possible to save the parameters to a file.
Default: False. Options: True/False
savepars=True expandable parameters
cmdreason
A string containing a reason to save to the FLAG_CMD table or to an output text file given by the outfile sub-parameter. If the input contains any reason, they will be replaced with this one. At the moment it is not possible to add more than one reason. Default: ‘ ‘, no reason will be added to output. Examples: cmdreason=’CLIP_ZEROS’
outfile
Name of output file to save the current parameters. Default: ‘ ‘, will save the parameters to the FLAG_CMD table of the MS. Examples: outfile=’flags.txt’ will save the parameters in a text file.
overwrite
Overwrite the existing file given in outfile. Options: True/False. Default: True, it will remove the existing file given in outfile and save the current flag commands to a new file with the same name. When set to False, the task will exit with an error message if the file exist.
SYNTAX FOR COMMANDS GIVEN IN A FILE or LIST OF STRINGS
Basic Syntax Rules
Commands are strings (which may contain internal “strings”) consisting of KEY=VALUE pairs separated by one whitespace only.
Note
There should be no whitespace between KEY=VALUE.The parser first breaks command lines on whitespace, then on “=”.
Use only ONE white space to separate the parameters (no commas). Each key should only appear once on a given command line/string.
There is an implicit mode for each command, with the default being ‘manual’ if not given.
Comment lines can start with ‘#’ and will be ignored. The parser used in flagdata will check each parameter name and type and exit with an error if the parameter is not a valid flagdata parameter or of a wrong type.
Example:
scan='1~3' mode='manual' # this line will be ignored spw='9' mode='tfcrop' correlation='ABS_XX,YY' ntime=51.0 mode='extend' extendpols=True scan='1~3,10~12' mode='quack' quackinterval=1.0
Bibliography
1. Greisen, Eric, Dec 31, 2011. AIPS documentation: Section E.5 of the AIPS cookbook (Appendix E: Special Considerations for EVLA data calibration and imaging in AIPS, http://www.aips.nrao.edu/cook.html#CEE ) #ref-cit1
- Examples
Examples of flagging a MeasurementSet
Note
NOTE: The vector mode of the flagdata task (pre-dating CASA 3.4) can be achieved with this task by using it with mode=’list’ and the commands given in a list in inpfile=[].
Flag using the ‘list’ mode and flag commands
flagdata('my.ms', inpmode='list', inpfile=["mode='clip' clipzeros=True","mode='shadow'])
Manually flag scans 1~3 and save the parameters to the FLAG_CMD sub-table.
flagdata('my.ms', scan='1~3, mode='manual', savepars=True)
Save the parameters to a file that is open in append mode.
flagdata('my.ms', scan='1~3, mode='manual', savepars=True, outfile='flags.txt')
Flag all the commands given in the Python list of strings.
cmd = ["scan='1~3' mode='manual'", "spw='9' mode='tfcrop' correlation='ABS_RR,LL' ntime=51.0", "mode='extend' extendpols=True"] flagdata('my.ms', mode='list', inpfile=cmd)
Flag all the commands given in the file called ‘flags.txt’.
cat flags.txt scan='1~3' mode='manual' spw='9' mode='tfcrop' correlation='ABS_RR,LL' ntime=51.0 mode='extend' extendpols=True flagdata('my.ms', mode='list', inpfile='flags.txt')
Display the data and flags per-chunk and do not write flags to the MS.
flagdata('my.ms', mode='list', inpfile='flags.txt', action='calculate', display='data')
Flag all the antennas except antenna=5.
flagdata(vis='my.ms', antenna='!5', mode='manual)
Clip the NaN in the data. An empty clipminmax will flag only NaN.
flagdata('my.ms', mode='clip')
Clip only the water vapor radiometer data.
flagdata('my.ms',mode='clip',clipminmax=[0,50], correlation='ABS_WVR')
Clip only zero-value data.
flagdata('my.ms',mode='clip',clipzeros=True)
Flag only auto-correlations of non-radiometer data using the autocorr parameter.
flagdata('my.ms', autocorr=True)
Flag only auto-correlations using the antenna selection.
flagdata('my.ms', mode='manual', antenna='\*&&&')
Flag based on selected reasons from a file.
This box is intended for CASA Inputs. Insert your text here.> cat flags.txt scan='1~3' mode='manual' reason='MYREASON' spw='9' mode='clip' clipzeros=True reason='CLIPZEROS' mode='manual' scan='4' reason='MYREASON' flagdata('my.ms', mode='list', inpfile='flags.txt', reason='MYREASON').
The same result of 10a can be achieved using the task flagcmd.
flagcmd('my.ms', inpmode='file', inpfile='flags.txt', action='apply', reason='MYREASON')
Automatic flagging using ‘rflag’, using auto-thresholds, and specifying a threshold scale-factor to use for flagging.
flagdata('my.ms', mode='rflag', spw='9', timedevscale=4.0, action='apply')
Save the interface parameters to the FLAG_CMD sub-table of the MS. Add a reason to the flag command. This cmdreason will be added to the REASON column of the FLAG_CMD sub-table. Apply flags in flagcmd.
flagdata('my.ms', mode='clip', channelavg=False, clipminmax=[30., 60.], spw='0:0~10', correlation='ABS_XX,XY', action='', savepars=True, cmdreason='CLIPXX_XY') #Select based on the reason. flagcmd('my.ms', action='apply', reason='CLIPXX_XY')
Flag antennas that are shadowed by antennas not present in the MS.
> Create a text file with information about the antennas. > cat ant.txt name=VLA01 diameter=25.0 position=[-1601144.96146691, -5041998.01971858, 3554864.76811967] name=VLA02 diameter=25.0 position=[-1601105.7664601889, -5042022.3917835914, 3554847.245159178] name=VLA09 diameter=25.0 position=[-1601197.2182404203, -5041974.3604805721, 3554875.1995636248] name=VLA10 diameter=25.0 position=[-1601227.3367843349,-5041975.7011900628,3554859.1642644769] flagdata('my.vis', mode='shadow', tolerance=10.0, addantenna='ant.txt')
The antenna information can also be given as a Python dictionary. To create the dictionary using the flaghelper functions, do the following inside casapy:
> import flaghelper as fh > antdic = fh.readAntennaList(antfile) flagdata('my.vis', mode='shadow', tolerance=10.0, addantenna=antdic)
Apply the online flags that come from importasdm.
> In importasdm, save the online flags to a file. importasdm('myasdm', 'asdm.ms', process_flags=True, savecmds=True, outfile='online_flags.txt') > You can edit the online_flags.txt to add other flagging commands or apply it directly. flagdata('asdm.ms', mode='list', inpfile='online_flags.txt') > The same result can be achieved using the task flagcmd. flagcmd('asdm.ms', inpmode='file', inpfile='online_flags.txt', action='apply')
Clip mode pre-averaging data across channels and across time.
flagdata(vis='Four_ants_3C286.ms', flagbackup=False, mode='clip', datacolumn='DATA', timeavg=True, timebin='2s', channelavg=True, chanbin=2)
Reduce the fraction of channels that are required to be flagged, and print information for every integration that is flagged.
flagdata(vis, ..., mode='antint', spw='9', antint_ref_antenna='ea01', minchanfrac=0.3, verbose=True)
Examples of flagging a calibration table
Clip zero data from a bandpass calibration table.
flagdata('cal-X54.B1', mode='clip', clipzeros=True, datacolumn='CPARAM')
Clip data from a cal table with SNR <4.0.
flagdata('cal-X54.B1', mode='clip', clipminmax=[0.0,4.0], clipoutside=False, datacolumn='SNR')
Clip the g values of a switched power caltable created using the gencal task. The g values are usually < 1.0.
flagdata('cal.12A.syspower', mode='clip', clipminmax=[0.1,0.3], correlation='Sol1,Sol3', datacolumn='FPARAM')
Now, clip the Tsys values of the same table from above. The Tsys solutions have values between 10 – 100s.
flagdata('cal.12A.syspower', mode='clip', clipminmax=[10.0,95.0], correlation='Sol2,Sol4', datacolumn='FPARAM')
- Development
No additional development details
- Parameter Details
Detailed descriptions of each function parameter
vis (path)
- Name of input visibility fileDefault: noneExample: vis=’ngc5921.ms’mode (string='manual')
- Flagging modeDefault: ‘manual’Options: ‘list’, ‘manual’, ‘clip’, ‘quack’,‘shadow’, ‘elevation’, ‘tfcrop’, ‘rflag’,‘antint’, ‘extend’, ‘unflag’, ‘summary’* ‘list’: Flag according to the data selectionand flag commands specified in the inputlist. The input list may come from a text file,a list of text files or from a Python list ofstrings. Each input line may contain dataselection parameters and any parameterspecific to the mode given in the line. Defaultvalues will be used for the parameters that arenot present in the line. Each line will betaken as a command to the task. If data ispre-selected using any of the selectionparameters, then flagging will apply only tothat subset of the MS.For optimization and whenever possible, thetask will create a union of the data selectionparameters present in the list and select onlythat portion of the MS.NOTE1: the flag commands will be applied onlywhen action=’apply’. If action=’calculate’ theflags will be calculated, but not applied. Thisis useful if display is set to something otherthan ‘none’. If action=’’ or ‘none’, the flagcommands will not be applied either. An emptyaction is useful only to save the parameters ofthe list to a file or to the FLAG_CMDsub-table.NOTE2: In list mode the parameterquackincrement=True is not supported as part ofany quack flag command, unless it is the firstcommand of the list. See more about this in thequack mode section of this help.* ‘manual’: Flag according to the data selectionspecified. This is the default mode* ‘clip’: Clip data according to values of thefollowing subparameters. The polarizationexpression is given by the correlationparameter. For calibration tables, thesolutions are also given by the correlationparameter.* ‘quack’: Option to remove specified part ofscan beginning/end.* ‘shadow’: Option to flag data of shadowedantennas. This mode is not available for caltables.All antennas in the antenna-subtable of the MS(and the corresponding diameters) will beconsidered for shadow-flag calculations. For agiven timestep, an antenna is flagged if any ofits baselines (projected onto the uv-plane) isshorter than radius_1 + radius_2 -tolerance. The value of ‘w’ is used todetermine which antenna is behind theother. The phase-reference center is used forantenna-pointing direction.* ‘elevation’: Option to flag based on antennaelevation. This mode is not available for caltables.* ‘tfcrop’: Flag using the TFCrop autoflagalgorithm.For each field, spw, timerange (specified byntime), and baseline,(1) Average visibility amplitudes along timedimension to form an average spectrum(2) Calculate a robust piece-wise polynomialfit for the band-shape at the base of RFIspikes. Calculate ‘stddev’ of (data - fit).(3) Flag points deviating from the fit by morethan N-stddev(4) Repeat (1-3) along the other dimension.This algorithm is designed to operate onun-calibrated data (step (2)), as well ascalibrated data. It is recommended to extendthe flags after running this algorithm. See thesub-parameter extendflags below.* ‘rflag’: Detect outliers based on the RFlagalgorithm (ref. E.Greisen, AIPS, 2011). Thepolarization expression is given by thecorrelation parameter.Iterate through the data in chunks of time.For each chunk, calculate local statistics, andapply flags based on user supplied (orauto-calculated) thresholds.Step 1 : Time analysis (for each channel)– calculate local rms of real and imagvisibilities, within a sliding time window– calculate the median rms across timewindows, deviations of local rms from thismedian, and the median deviation– flag if local rms is larger thantimedevscale x (medianRMS + medianDev)Step 2 : Spectral analysis (for each time)– calculate avg of real and imagvisibilities and their rms across channels– calculate the deviation of each channelfrom this avg, and the median-deviation– flag if deviation is larger thanfreqdevscale x medianDevIt is recommended to extend the flags afterrunning this algorithm. See the sub-parameterextendflags below.Note that by default the flag implementation inCASA is able to calculate the thresholds andapply them on-the-fly (OTF). There is asignificant performancegain with this approach,as the visibilities don’t have to be readtwice,and therefore is highlyrecommended. Otherwise it is possibletoreproduce the AIPS usage pattern by doing afirst run with ‘calculate’ mode and a secondrun with ‘apply’ mode. The advantage of thisapproach is that the thresholdsare calculatedusing the data from all scans, instead ofcalculating them for one scan only.For more information and examples of ‘rflag’,see the task pages of rflag in CASA Docs* ‘antint’: Flag integrations if all baselines toa specified antenna are flaggedThis mode flag all integrations in which aspecified antenna is flagged. This modeoperates for an spectral window. It flags anyintegration in which all baselines to aspecified antenna are flagged, but only if thiscondition is satisfied in a fraction ofchannels within the spectral window of interestgreater than a nominated fraction. Forsimplicity, it assumes that all polarizationproducts must be unflagged for a baseline to bedeemed unflagged. The antint mode implementsthe flagging approach introduced in‘antintflag’The motivating application for introducing thismode is removal of data that will otherwiselead to changes in reference antenna duringgain calibration, which will in turn lead tocorrupted polarization.* ‘extend’: Extend and/or grow flags beyond whatthe basic algorithms detect. This mode willextend the accumulated flags available in theMS, regardless of which algorithm created them.It is recommended that any autoflag (tfcrop,rflag) algorithm be followed up by a flagextension.Extensions will apply only within the selecteddata, according to the settings ofextendpols,growtime,growfreq,growaround,flagneartime,flagnearfreq.Note : Runtime summary counts in the logger cansometimes report larger flag percentages thanwhat is actually flagged. This is becauseextensions onto already-flagged data-points arecounted as new flags. An accurate flag countcan be obtained via the summary mode.* ‘unflag’: Unflag according to the dataselection specified.* ‘summary’: List the number of rows and flaggeddata points for the MS’s meta-data. Theresulting summary will be returned as a Pythondictionary.autocorr (bool=False)
- Flag only the auto-correlations?Subparameter of mode=’manual’Default: FalseOptions: False|TrueNOTE: this parameter is only active when set toTrue. If set to False it does NOT mean “do notflag auto-correlations”. When set to True, itwill only flag data from a processor of typeCORRELATOR.inpfile ({string, stringVec}='')
- Input ASCII file, list of files or Python list of stringswith flag commands.Subparameter of mode=’list’Default: ‘’Options: [] with flag commands or[] with filenames or‘’ with a filename.The parser will be strict and accept only validflagdata parameters in the list. The parserevaluates the commands in the list and considersonly existing Python types.It will check eachparameter name and type and exit with an error ifany of them is wrong.NOTE: There should be no whitespace betweenKEY=VALUE since the parser first breaks commandlines on whitespace, then on “=”. Use only onewhitespace to separate the parameters (nocommas).reason ({string, stringVec}='any')
- Select flag commands based on REASON(s)Subparameter of mode=’list’Default: ‘any’ (all flags regardless of reason)Can be a string, or list of stringsExamples:reason=’FOCUS_ERROR’reason=[‘FOCUS_ERROR’,’SUBREFLECTOR_ERROR’]If inpfile is a list of files, the reasons givenin this parameter will apply to all the files.NOTE: what is within the string is literallymatched, e.g. reason=’’ matches only blankreasons, and reason =‘FOCUS_ERROR,SUBREFLECTOR_ERROR’ matches thiscompound reason string only.tbuff ({double, doubleVec}=0.0)
- A time buffer or list of time buffers to pad thetimerange parameters in flag commands.Subparameter of mode=’list’Default: 0.0 (it will not apply any time padding)When a list of 2 time buffers is given, it willsubtract the first value from the lower time andthe second value will be added to the upper timein the range. The 2 time buffer values can bedifferent, allowing to have an irregular timebuffer padding to time ranges. If the listcontains only one time buffer, it will use it tosubtract from t0 and add to t1. If more than onelist of input files is given, tbuff will apply toall of the flag commands that have timerangeparameters in the files. Each tbuff value shouldbe a Float number given in seconds.Examples:tbuff=[0.5, 0.8]inpfile=[‘online.txt’,’userflags.txt’]The timeranges in the online.txt file arefirst converted to seconds. Then, 0.5 issubtracted from t0 and 0.8 is added to t1,where t0 and t1 are the two intervals given intimerange. Similarly, tbuff will be applied toany timerange in userflags.txt.IMPORTANT: This parameter assumes that timerange= t0 ~ t1, therefore it will not work if only t0or t1 is given.NOTE: The most common use-case for tbuff is toapply the online flags that are created byimportasdm when savecmds=True. The value of aregular time buffer should betbuff=0.5*max(integration time).spw ({string, stringVec}='')
- Select spectral window/channelsDefault: ‘’ (all spectral windows and channels)Examples:spw=’0~2,4’; spectral windows 0,1,2,4 (all channels)spw=’<2’; spectral windows less than 2 (i.e. 0,1)spw=’0:5~61’; spw 0, channels 5 to 61spw=’0,10,3:3~45’; spw 0,10 all channels, spw3 - chans 3 to 45.spw=’0~2:2~6’; spw 0,1,2 with channels 2through 6 in each.spw = ‘*:3~64’ channels 3 through 64 for all sp id’sspw = ‘ :3~64’ will NOT work.NOTE: For modes clip, tfcrop and rflag,channel-ranges can be excluded from flagging byleaving them out of the selection range. This isa way to protect known spectral-lines from beingflagged by the autoflag algorithms. For example,if spectral-lines fall in channels 6~9, set theselection range to spw=’0:0~5;10~63’.field ({string, stringVec}='')
- Select field using field id(s) or field name(s)Default: ‘’ (all fields)Use ‘go listobs’ to obtain the list id’s ornames. If field string is a non-negative integer,it is assumed a field index, otherwise, it isassumed a field name.Examples:field=’0~2’; field ids 0,1,2field=’0,4,5~7’; field ids 0,4,5,6,7field=’3C286,3C295’; field named 3C286 and3C295field = ‘3,4C*’; field id 3, all namesstarting with 4Cantenna ({string, stringVec}='')
- Select data based on antenna/baselineSubparameter of selectdata=TrueDefault: ‘’ (all)If antenna string is a non-negative integer, itis assumed an antenna index, otherwise, it isassumed as an antenna nameExamples:antenna=’5&6’; baseline between antennaindex 5 and index 6.antenna=’VA05&VA06’; baseline between VLAantenna 5 and 6.antenna=’5&6;7&8’; baselines withindices 5-6 and 7-8antenna=’5’; all baselines with antenna index5antenna=’05’; all baselines with antennanumber 05 (VLA old name)antenna=’5,6,10’; all baselines with antennas5,6,10 index numbersNOTE: for some antenna-based calibration tables,selecting baselines with the & syntax do notapply.uvrange ({string, stringVec}='')
- Select data by baseline length.Default = ‘’ (all)Examples:uvrange=’0~1000klambda’; uvrange from 0-1000 kilo-lambdauvrange=’>4klambda’;uvranges greater than 4 kilo-lambdauvrange=’0~1000km’; uvrange in kilometersNOTE: uvrange selection is not supported for cal tables.timerange ({string, stringVec}='')
- Select data based on time rangeSubparameter of selectdata=TrueDefault = ‘’ (all)Examples:timerange =‘YYYY/MM/DD/hh:mm:ss~YYYY/MM/DD/hh:mm:ss’(Note: if YYYY/MM/DD is missing date defaultsto first day in data set.)timerange=’09:14:0~09:54:0’ picks 40 min onfirst daytimerange= ‘25:00:00~27:30:00’ picks 1 hr to 3hr 30min on NEXT daytimerange=’09:44:00’ pick data within oneintegration of timetimerange=’>10:24:00’ data after this timecorrelation ({string, stringVec}='')
- Select data based on correlationDefault: ‘’ ==> allOptions: Any of ‘ABS’, ‘ARG’, ‘REAL’, ‘IMAG’,‘NORM’ followed by any of ‘ALL’, ‘I’, ‘XX’, ‘YY’,‘RR’, ‘LL’, ‘WVR’ (‘WVR’ = water vapourradiometer of ALMA data).Example: correlation=”XX,YY”.For modes clip, tfcrop or rflag, the defaultmeans ABS_ALL. If the input is cal table thatdoes not contain a complex data column, thedefault will fall back to REAL_ALL.For calibration tables, the solutions are:‘Sol1’, ‘Sol2’, Sol3, Sol4.NOTE: correlation selection is not supported formodes other than clip, tfcrop or rflag in caltables.scan ({string, stringVec}='')
- Scan number rangeSubparameter of selectdata=TrueDefault: ‘’ = allintent ({string, stringVec}='')
- Select observing intentDefault: ‘’ (no selection by intent)Example: intent=’*BANDPASS*’ (selects datalabelled with BANDPASS intent)NOTE: intent selection is not supported for caltables.array ({string, stringVec}='')
- Selection based on the antenna arrayDefault: ‘’ (all)NOTE: array selection is not supported for caltables.observation ({string, int}='')
- Select by observation ID(s)Subparameter of selectdata=TrueDefault: ‘’ = allExample: observation=’0~2,4’feed ({string, stringVec}='')
- Selection based on the feed: Not yet implementedclipminmax (doubleVec='')
- Range to use for clippingSubparameter of mode=’clip’Default: [] (it will flag only NaN and Infs)It will always flag the NaN/Inf data, even when a range is specified.Example: [0.0,1.5]datacolumn ({string, stringVec}='DATA')
- Data column to image (data or observed, corrected)Subparameter of mode=’clip|tfcrop|rflag’Default:’corrected’Options: data, corrected, model, weight, etc.If ‘corrected’ does not exist, it will use ‘data’insteadclipoutside ({bool, boolVec}=True)
- Clip outside the range?Subparameter of mode=’clip’Default: TrueOptions: True|Falsechannelavg ({bool, boolVec}=False)
- Pre-average data across channels before analyzingvisibilities for flaggingSubparameter of mode=’clip|tfcrop|rflag’Default: FalseOptions: False|TruePre-average data across channels before analyzingvisibilities for flagging. Partially flagged datais not be included in the average unless alldata contributing to a given output channel isflagged. If present, WEIGHT_SPECTRUM /SIGMA_SPECTRUM are used to compute a weightedaverage (WEIGHT_SPECTRUM for CORRECTED_DATA andSIGMA_SPECTRUM for DATA).NOTE: Pre-average across channels is notsupported for calibration tableschanbin ({int, intVec}=1)
- Bin width for channel average in number of input channelsSubparameter of mode=’clip|tfcrop|rflag’Default: 1Bin width for channel average in number of inputchannels. If a list is given, each bin applies toone of the selected SPWs. When chanbin is set to1 all input channels are used for the averageto produce a single output channel, thisbehaviour aims to preserve backwardscompatibility with the previous pre-averagingfeature of clip mode.timeavg ({bool, boolVec}=False)
- Pre-average data across time before analyzingvisibilities for flagging.Subparameter of mode=’clip|tfcrop|rflag’Default: FalseOptions: False|TruePre-average data across time before analyzingvisibilities for flagging. Partially flagged datais not be included in the average unless alldata contributing to a given output channel isflagged. If present, WEIGHT_SPECTRUM /SIGMA_SPECTRUM are used to compute a weightedaverage (WEIGHT_SPECTRUM for CORRECTED_DATA andSIGMA_SPECTRUM for DATA). Otherwise WEIGHT/SIGMAare used to average together data from differentintegrations.NOTE: Pre-average across time is notsupported for calibration tablestimebin (string='0s')
- Bin width for time average in secondsSubparameter of mode=’clip|tfcrop|rflag’Default: ‘0s’clipzeros (bool=False)
- Clip zero-value dataSubparameter of mode=’clip’Default: FalseOptions: False|Truequackinterval ({double, doubleVec, int, intVec}=1.0)
- Time in seconds from scan beginning or end to flag.Subparameter of mode=’quack’Default: 0.0Note: Make time slightly smaller than the desiredtime.quackmode ({string, stringVec}='beg')
- Quack mode flags the region of the scan given by one of theoptions below using the time set at quackinterval.Subparameter of mode=’quack’Default: ‘beg’Options:‘beg’ : flag an interval at the beginning of scan‘endb’: flag an interval at the end of scan‘tail’: flag all but an interval at the beginning of scan‘end’ : flag all but an interval at end of scanVisual representation of quack mode flagging onescan with 1s duration. The following diagramshows what is flagged for each quack mode whenquackinterval is set to 0.25s. The flagged partis represented by crosses (+++++++++)scan with 1s duration——————————————–beg+++++++++++———————————endb———————————+++++++++++tail———–+++++++++++++++++++++++++++++++++end+++++++++++++++++++++++++++++++++———–quackincrement ({bool, boolVec}=False)
- Increment quack flagging in time taking into accountflagged data or not.Subparameter of mode=’quack’Default: FalseOptions: False|TrueFalse: the quack interval is counted from thescan boundaries, as determined by the quackmodeparameter, regardless of if data has been flaggedor not.True: the quack interval is counted from thefirst unflagged data in the scan.NOTE: on adding quack to a command in ‘list’mode: quackincrement = True works based on thestate of prior flagging, and unless it is thefirst item in the list the agent doing thequacking in list mode doesn’t know about thestate of prior flags. In this case, the commandwith quackincrement=True will be ignored and thetask will issue a WARNING.tolerance (double=0.0)
- Amount of shadowing allowed (or tolerated), in meters.Subparameter of mode=’shadow’Default: 0.0A positive number allows antennas to overlap inprojection. A negative number forces antennasapart in projection. Zero implies a distance ofradius_1+radius_2 between antenna centers.addantenna ({string, record}='')
- File name or dictionary with additional antenna names,positions and diametersSubparameter of mode=’shadow’Default: ‘’It can be either a file name with additionalantenna names, positions and diameters, or aPython dictionary with the same information. Youcan use the flaghelper functions to create thedictionary from a file.To create a dictionary inside casapy.> import flaghelper as fh> antdic = fh.readAntennaList(antfile)Where antfile is a text file in disk thatcontains information such as:name=VLA01diameter=25.0position=[-1601144.96146691, -5041998.01971858,3554864.76811967]name=VLA02diameter=25.0position=[-1601105.7664601889,-5042022.3917835914, 3554847.245159178]lowerlimit (double=0.0)
- Lower limiting elevation (in degrees)Subparameter of mode=’elevation’Default: 0.0Lower limiting elevation in degrees. Data comingfrom a baseline where one or both antennas werepointing at a strictly lower elevation (asfunction of time), will be flagged.upperlimit (double=90.0)
- Upper limiting elevation (in degrees)Subparameter of mode=’elevation’Default: 90.0Upper limiting elevation in degrees. Data comingfrom a baseline where one or both antennas werepointing at a strictly higher elevation (asfunction of time), will be flagged.ntime ({double, string}='scan')
- Timerange (in seconds or minutes) over which to bufferdata before running the algorithm.Subparameter of mode=’tfcrop|rflag|extend’Default: ‘scan’Options: ‘scan’ or any other float value orstring containing the units.The dataset will be iterated through intime-chunks defined here.Example: ntime=’1.5min’; 1.2 (taken inseconds)WARNING: if ntime=’scan’ and combinescans=True,all the scans will be loaded at once, thusrequesting a lot of memory depending on theavailable spws.combinescans (bool=False)
- Accumulate data across scans depending on the value ofntime.Subparameter of mode=’tfcrop|rflag|extend’Default: FalseOptions: False|TrueThis parameter should be set to True only whenntime is specified as a time-interval (not‘scan’). When set to True, it will remove SCANfrom the sorting columns, therefore it will onlyaccumulate across scans if ntime is not set to‘scan’.timecutoff (double=4.0)
- Flagging thresholds in units of deviation from the fitSubparameter of mode=’tfcrop’Default: 4.0Flag all data-points further than N-stddev fromthe fit. This threshold catches time-varying RFIspikes (narrow and broad-band), but will notcatch RFI that is persistent in time.Flagging is done in upto 5 iterations. The stddevcalculation is adaptive and converges to a valuethat reflects only the data and no RFI. At eachiteration, the same relative threshold is appliedto detect flags. (Step (3) of the algorithm).freqcutoff (double=3.0)
- Flag threshold in frequency.Subparameter of mode=’tfcrop’Default: 3.0Flag all data-points further than N-stddev fromthe fit. Same as timecutoff, but along thefrequency-dimension. This threshold catchesnarrow-band RFI that may or may not be persistentin time.timefit (string='line')
- Fitting function for the time direction (poly/line)Subparameter of mode=’tfcrop’Default: ‘line’Options: line|poly‘line’: fit is a robust straight-line fit acrossthe entire timerange (defined by ‘ntime’).‘poly’: fit is a robust piece-wise polynomial fitacross the timerange.NOTE: A robust fit is computed in upto 5iterations. At each iteration, the stddev betweenthe data and the fit is computed, values beyondN-stddev are flagged, and the fit and stddev arere-calculated with the remaining points. Thisstddev calculation is adaptive, and converges toa value that reflects only the data and no RFI.It also provides a varying set of flaggingthresholds, that allows deep flagging only whenthe fit best represents the true data.Choose ‘poly’ only if the visibilities areexpected to vary significantly over the timerangeselected by ‘ntime’, or if there is a lot ofstrong but intermittent RFI.freqfit (string='poly')
- Fitting function for the frequency direction (poly/line)Subparameter of mode=’tfcrop’Default: ‘poly’Options: line|polySame as for the ‘timefit’ parameter.Choose ‘line’ only if you are operating onbandpass-corrected data, or residuals,and expectthat the bandshape is linear. The ‘poly’ optionworks better on uncalibrated bandpasses withnarrow-band RFI spikes.maxnpieces (int=7)
- Number of pieces in the polynomial-fits (for “freqfit” or“timefit” = “poly”)Subparameter of mode=’tfcrop’Default: 7Options: 1-9This parameter is used only if ‘timefit’ or‘freqfit’ are chosen as ‘poly’. If there issignificant broad-band RFI, reduce thisnumber. Using too many pieces could result in theRFI being fitted in the ‘clean’ bandpass. Inlater stages of the fit, a third-order polynomialis fit per piece, so for best results, pleaseensure that nchan/maxnpieces is at-least 10.flagdimension (string='freqtime')
- Choose the directions along which to perform flaggingSubparameter of mode=’tfcrop’Default: ‘freqtime’ (first flag along frequency,and then along time)Options: ‘time’, ‘freq’, ‘timefreq’, ‘freqtime’For most cases, ‘freqtime’ or ‘timefreq’ areappropriate, and differences between thesechoices are apparant only if RFI in one dimensionis significantly stronger than the other. Thegoal is to flag the dominant RFI first.If there are very few (less than 5) channels ofdata, then choose ‘time’. Similarly for ‘freq’.usewindowstats (string='none')
- Use sliding-window statistics to find additional flags.Subparameter of mode=’tfcrop’Default: ‘none’Options: ‘none’, ‘sum’, ‘std’, ‘both’NOTE: This is experimental!The ‘sum’ option chooses to flag a point, if themean-value in a window centered on that pointdeviates from the fit by more than N-stddev/2.0.NOTE: stddev is calculated between the data andfit as explained in Step (2). This option is anattempt to catch broad-band or time-persistentRFI that the above polynomial fits willmistakenly fit as the clean band. It is anapproximation to the sumThreshold method found tobe effective by Offringa et.al (2010) for LOFARdata. The ‘std’ option chooses to flag a point,if the ‘local’ stddev calculated in a windowcentered on that point is larger thanN-stddev/2.0. This option is an attempt to catchnoisy RFI that is not excluded in the polynomialfits, and which increases the global stddev, andresults in fewer flags (based on the N-stddevthreshold).halfwin (int=1)
- Half-width of sliding window to use with “usewindowstats”(1,2,3).Subparameter of mode=’tfcrop’Default: 1 (a 3-point window size)Options: 1, 2, 3NOTE: This is experimental!extendflags (bool=True)
- Extend flags along time, frequency and correlation.Subparameter of mode=’tfcrop|rflag’Default: TrueOptions: True|FalseNOTE: It is usually helpful to extend the flagsalong time, frequency, and correlation using thisparameter, which will run the “extend” mode after“tfcrop” and extend the flags if more than 50% ofthe timeranges are already flagged, and if morethan 80% of the channels are already flagged. Itwill also extend the flags to the otherpolarizations. The user may also set extendflagsto False and run the “extend” mode in a secondstep within the same flagging run:Example: cmd=[“mode=’tfcrop’ freqcutoff=3.0usewindowstats=’sum’ extendflags=False”,“mode=’extend’ extendpols=True growtime=50.0growaround=True”]flagdata(vis, mode=’list’, inpfile=cmd)winsize (int=3)
- Number of timesteps in the sliding time window ( fparm(1)in AIPS )Subparameter of mode=’rflag’Default: 3timedev (variant='')
- Time-series noise estimate ( noise in AIPS )Subparameter of mode=’rflag’Default: []Examples:timedev = 0.5 : Use this noise-estimate tocalculate flags. Do not recalculate.timedev = [ [1,9,0.2], [1,10,0.5] ] : Usenoise-estimate of 0.2 for field 1, spw 9, andnoise-estimate of 0.5 for field 1, spw 10.timedev = [] : Auto-calculate noiseestimates.‘tdevfile.txt’ : Auto-calculate noiseestimates and write them into a file with thename given (any string will be interpreted asa file name which will be checked forexistence).freqdev (variant='')
- Spectral noise estimate ( scutoff in AIPS )Subparameter of mode=’rflag’Default: []This step depends on having a relatively-flatbandshape. Same parameter-options as ‘timedev’.timedevscale (double=5.0)
- Threshold scaling for timedev( fparm(9) in AIPS ).For Step 1 (time analysis), flag a point if localrms around it is larger than ‘timedevscale’ x‘timedev’ (fparm(0) in AIPS)Subparameter of mode=’rflag’Default: 5.0This scale parameter is only applied whenflagging (action=’apply’) and displaying reports(display option). It is not used when thethresholds are simply calculated and saved intofiles (action=’calculate’, as in the two-passesusage pattern of AIPS)freqdevscale (double=5.0)
- Threshold scaling for freqdev (fparm(10) in AIPS ).For Step 2 (spectral analysis), flag a point iflocal rms around it is larger than ‘freqdevscale’x ‘freqdev’ (fparm(10) in AIPS)Subparameter of mode=’rflag’Default: 5.0Similarly as with timedevscale, freqdevscale isused when applying flags and displayingreports. It is not used when the thresholds aresimply calculated and saved into files(action=’calculate’, as in the two-passes usagepattern of AIPS)spectralmax (double=1E6)
- Flag whole spectrum if ‘freqdev’ is greater thanspectralmax ( fparm(6) in AIPS )Subparameter of mode=’rflag’Default: 1E6spectralmin (double=0.0)
- Flag whole spectrum if ‘freqdev’ is less than spectralmin( fparm(5) in AIPS )Subparameter of mode=’rflag’Default: 0.0antint_ref_antenna (string='')
- Antenna of interest. Baselines with this antenna will bechecked for flagged channels.Subparameter of mode=’antint’Note that this is not the same as the general‘antenna’ parameter of flagdata. The parameterantint_ref_antenna is mandatory with the ‘antint’mode and chooses the antenna for which thefraction of channels flagged will be checked.minchanfrac (double=0.6)
- Minimum fraction of flagged channels required for abaseline to be deemed as flaggedSubparameter of mode=’antint’Takes values between 0-1 (float).In this mode flagdata does the following forevery point in time. It checks the fraction ofchannels flagged for any of the polarizationproducts and for every baseline to the antenna ofinterest. If the fraction is higher than this‘minchanfrac’ threshold then the data are flaggedfor this pont in time (this includes all the rowsselected with the flagdata command that have thattimestamp).This parameter will be ignored if spw specifies achannel.verbose (bool=False)
- Print timestamps of flagged integrations to the logSubparameter of mode=’antint’Examples:flagdata(vis, …, spw=’9’,antint_ref_antenna=’ea01’)flagdata(vis, …, spw=’9’,antint_ref_antenna=’ea01’, minchanfrac=0.3,verbose=True) ==> reduce the fraction ofchannels that are required to be flagged, andprint information for every integration thatis flagged.extendpols (bool=True)
- Extend flags to all selected correlationsSubparameter of mode=’extend’Default: TrueOptions: True|FalseFor example, to extend flags from RR to onlyRL and LR, a data-selection ofcorrelation=’RR,LR,RL’ is required along withextendpols=True.growtime (double=50.0)
- For any channel, flag the entire timerange in the current2D chunk (set by ‘ntime’) if more than X% of the timerange is alreadyflagged.Subparameter of mode=’extend’Default: 50.0Options: 0.0 - 100.0This option catches the low-intensity parts oftime-persistent RFI.growfreq (double=50.0)
- For any timestep, flag all channels in the current 2Dchunk (set by data-selection) if more than X% of the channels arealready flagged.Subparameter of mode=’extend’Default: 50.0Options: 0.0 - 100.0This option catches broad-band RFI that ispartially identified by earlier steps.growaround (bool=False)
- Flag a point based on the number of flagged points around it.Subparameter of mode=’extend’Default: FalseOptions: False|TrueFor every un-flagged point on the 2D time/freqplane, if more than four surrounding points arealready flagged, flag that point. This optioncatches some wings of strong RFI spikes.flagneartime (bool=False)
- Flag points before and after every flagged one, in thetime-direction.Subparameter of mode=’extend’Default: FalseOptions: False|TrueNOTE: This can result in excessive flagging.flagnearfreq (bool=False)
- Flag points before and after every flagged one, in thefrequency-directionSubparameter of mode=’extend’Default: FalseOptions: False|TrueNOTE: This can result in excessive flaggingminrel (double=0.0)
- Minimum number of flags (relative) to include inhistogramSubparameter of mode=’summary’Default: 0.0maxrel (double=1.0)
- Maximum number of flags (relative) to include inhistogramSubparameter of mode=’summary’Default: 1.0minabs (int=0)
- Minimum number of flags (absolute, inclusive) to includein histogramSubparameter of mode=’summary’Default: 0maxabs (int=-1)
- Maximum number of flags (absolute, inclusive) to includein histogramSubparameter of mode=’summary’Default: -1To indicate infinity, use any negative number.spwchan (bool=False)
- List the number of flags per spw and per channel.Subparameter of mode=’summary’Default: FalseOptions: False|Truespwcorr (bool=False)
- List the number of flags per spw and per correlation.Subparameter of mode=’summary’Default: FalseOptions: False|Truebasecnt (bool=False)
- List the number of flags per baselineSubparameter of mode=’summary’Default: FalseOptions: False|Truefieldcnt (bool=False)
- Produce a separated breakdown per fieldSubparameter of mode=’summary’Default: FalseOptions: False|Truename (string='Summary')
- Name for this summary, to be used as a key in thereturned Python dictionarySubparameter of mode=’summary’Default: ‘Summary’It is possible to call the summary mode multipletimes in list mode. When calling the summary modeas a command in a list, one can give differentnames to each one of them so that they can beeasily pulled out of the summary’s dictionary.In summary mode, the task returns a dictionary of flagging statistics.Example 1:s = flagdata(…, mode=’summary’)Then s will be a dictionary which containss[‘total’] : total number of datas[‘flagged’] : amount of flagged dataExample 2:Two summary commands in list mode, intercalating amanual flagging command.s = flagdata(…, mode=’list’,inpfile=[“mode=’summary’ name=’InitFlags’”,“mode=’manual’ autocorr=True”, “mode=’summary’name=’Autocorr’”])The dictionary returned in ‘s’ will contain twodictionaries, one for each of the two summarymodes.s[‘report0’][‘name’] : ‘InitFlags’s[‘report1’][‘name’] : ‘Autocorr’action (string='apply')
- Action to perform in MS/cal table or in the input list ofparameters.Default: ‘apply’Options: ‘none’, ‘apply’,’calculate’- ‘apply’: Apply the flags to the MS.- ‘calculate’: Only calculate the flags but donot write them to the MS. This is useful ifused together with the display to analyse theresults before writing to the MS.- ‘’: When set to empty, the underlying tool willnot be executed and no flags will beproduced. No data selection will be doneeither. This is useful when used together withthe parameter savepars to only save the currentparameters (or list of parameters) to theFLAG_CMD sub-table or to an external file.display (string='')
- Display data and/or end-of-MS reports at runtime.Subparameter of action=’apply|calculate’Default: ‘none’Options: ‘none’, ‘data’, ‘report’, ‘both’- ‘none’: will not display anything.- ‘data’: display data and flags per-chunk atrun-time, within an interactive GUI.This option opens a GUI to show the 2Dtime-freq planes of the data with old and newflags, for all correlations per baseline.– The GUI allows stepping through allbaselines (prev/next) in the current chunk (setby ‘ntime’), and stepping to the next-chunk.– The ‘flagdata’ task can be quit from theGUI, in case it becomes obvious that thecurrent set of parameters is just wrong.– There is an option to stop the display butcontinue flagging.- ‘report’: displays end-of-MS reports on thescreen.- ‘both’: displays data per chunk and end-of-MSreports on the screenflagbackup (bool=True)
- Automatically backup flags before the run?Default: TrueOptions: True|FalseFlagversion names are chosen automatically, andare based on the mode being used.savepars (bool=False)
- Save the current parameters to the FLAG_CMD table of theMS or to an output text file?Default: FalseOptions: False|TrueNote that when display is set to anything otherthan ‘none’, savepars will be disabled. This isdone because in an interactive mode, the user mayskip data which may invalidate the initial inputparameters and there is no way to save theinteractive commands. When the input is acalibration table it is only possible to save theparameters to a file.cmdreason (string='')
- A string containing a reason to save to the FLAG_CMDtable or to an output text file given by the outfile sub-parameter.Subparameter of savepars=TrueDefault: ‘’ (no reason will be added to output)If the input contains any reason, they will bereplaced with this one. At the moment it is notpossible to add more than one reason.Example: cmdreason=’CLIP_ZEROS’outfile (string='')
- Name of output file to save current parameters. If empty,save to FLAG_CMDSubparameter of savepars=TrueDefault: ‘’ (save the parameters to the FLAG_CMDtable of the MS)Example: outfile=’flags.txt’ will save theparameters in a text file.overwrite (bool=True)
- Overwrite the existing file given in ‘outfile’Default: TrueOptions: True|FalseThe default True will remove the existing filegiven in ‘outfile’ and save the current flagcommands to a new file with the same name. Whenset to False, the task will exit with an errormessage if the file exist.writeflags (bool=True)
- Internal hidden parameter. Do not modify.