flagdata – All-purpose flagging task based on data-selections and flagging modes/algorithms. – editing task
Description
This task can flag a Measurement Set or a calibration table. It has two main types of operation. One type will read the parameters from the interface and flag using any of the various available modes. The other type will read the commands from a text file, a list of files or a Python list of strings, containing a list of flag commands (each line containing data selection parameters and any parameter specific for the mode being requested). Please see examples at the end of this help.
It is also possible to only save the parameters set in the interface without flagging. The parameters can be saved in the FLAG_CMD sub-table or in a text file. Note that when saving to an external file, the parameters will be appended to the given file.
The available flagging modes are: manual, clip, shadow, quack, elevation, tfcrop, rflag, extend, unflag and summary. For automatic flagging, it is recommended to combine auto-flag modes with extend, via the list mode.
The current flags can be automatically backed up before applying new flags if the parameter flagbackup is set. Previous flag versions can be recovered using the flagmanager task.
Flagdata can flag many types of calibration tables using mode=’manual’. It can only flag using the auto-flagging algorithms (clip, tfcrop or rflag), the cal tables that have the following data columns: CPARAM, FPARAM or SNR. The solution elements of the data columns are given in the correlation parameter using the names ‘Sol1’, ‘Sol2’, ‘Sol3’, or ‘Sol4’. See examples at the end of this help on how to flag different cal tables.
When the input is a calibration table, the modes ‘elevation’ and ‘shadow’ will be disabled. Data selection for calibration tables is limited to field, scan, time, antenna, spw and observation. It is only possible to save the parameters to an external file. If the calibration table was created before CASA 4.1, this task will create a dummy OBSERVATION column and OBSERVATION sub-table in the input calibration table to adapt it to the new cal table format.
Selecting antennas in some calibration tables have a different meaning compared to selecting the MS. Some calibration tables such as the antenna-based ones, created with some modes of gencal or polcal, have the ANTENNA2 column set to -1. This means that when selecting antenna=’ANT’, will select the whole ANT and not the cross-correlations between ANT and the other antennas. Similarly, the baseline syntax do not apply to this type of calibration tables. Those values with ampersand do not have any meaning when selecting antenna/baselines in antenna-based cal tables.
The task will flag a subset of data based on the following modes of operation:
list = list of flagging commands to apply to MS/cal table manual = flagging based on specific selection parameters clip = clip data according to values quack = remove/keep specific time range at scan beginning/end shadow = remove antenna-shadowed data elevation = remove data below/above given elevations tfcrop = automatic identification of outliers on the time-freq plane rflag = automatic detection of outliers based on sliding-window RMS filters antint = flag integrations if all baselines to a specified antenna are flagged extend = extend and/or grow flags beyond what the basic algorithms detect summary = report the amount of flagged data unflag = unflag the specified data
Parameters
Parameter |
Default |
Description |
|---|---|---|
vis |
|
|
mode |
|
|
autocorr |
|
|
inpfile |
|
|
reason |
|
|
tbuff |
|
|
spw |
|
|
field |
|
|
antenna |
|
|
uvrange |
|
|
timerange |
|
|
correlation |
|
|
scan |
|
|
intent |
|
|
array |
|
|
observation |
|
|
feed |
|
|
clipminmax |
|
|
datacolumn |
|
|
clipoutside |
|
|
channelavg |
|
|
chanbin |
|
|
timeavg |
|
|
timebin |
|
|
clipzeros |
|
|
quackinterval |
|
|
quackmode |
|
|
quackincrement |
|
|
tolerance |
|
|
addantenna |
|
|
lowerlimit |
|
|
upperlimit |
|
|
ntime |
|
|
combinescans |
|
|
timecutoff |
|
|
freqcutoff |
|
|
timefit |
|
|
freqfit |
|
|
maxnpieces |
|
|
flagdimension |
|
|
usewindowstats |
|
|
halfwin |
|
|
extendflags |
|
|
winsize |
|
|
timedev |
|
|
freqdev |
|
|
timedevscale |
|
|
freqdevscale |
|
|
spectralmax |
|
|
spectralmin |
|
|
antint_ref_antenna |
|
|
minchanfrac |
|
|
verbose |
|
|
extendpols |
|
|
growtime |
|
|
growfreq |
|
|
growaround |
|
|
flagneartime |
|
|
flagnearfreq |
|
|
minrel |
|
|
maxrel |
|
|
minabs |
|
|
maxabs |
|
|
spwchan |
|
|
spwcorr |
|
|
basecnt |
|
|
fieldcnt |
|
|
name |
|
|
action |
|
|
display |
|
|
flagbackup |
|
|
savepars |
|
|
cmdreason |
|
|
outfile |
|
|
overwrite |
|
|
writeflags |
|
Parameter Explanations
vis
''
Name of MS file or calibration table to flag
mode
'manual'
Flagging mode
autocorr
False
Flag only the auto-correlations
inpfile
''
Input ASCII file, list of files or Python list of strings with flag commands.
reason
'any'
Select by REASON types
tbuff
float(0.0)
List of time buffers (sec) to pad timerange in flag commands
spw
''
Spectral-window/frequency/channel: ‘’ ==> all, spw=”0:17~19”
field
''
Field names or field index numbers: ‘’ ==> all, field=”0~2,3C286”
antenna
''
Antenna/baselines: ‘’ ==> all, antenna =”3,VA04”
uvrange
''
UV range: ‘’ ==> all; uvrange =”0~100klambda”, default units=meters
timerange
''
Time range: ‘’ ==> all,timerange=”09:14:0~09:54:0”
correlation
''
Correlation: ‘’ ==> all, correlation=”XX,YY”
scan
''
Scan numbers: ‘’ ==> all
intent
''
Scan intent: ‘’ ==> all, intent=”CAL*POINT*”
array
''
(Sub)array numbers: ‘’ ==> all
observation
''
Observation ID: ‘’ ==> all
feed
''
Multi-feed numbers: Not yet implemented
clipminmax
numpy.array( [ ] )
Range to use for clipping
datacolumn
'DATA'
Data column on which to operate (data,corrected,model,weight,etc.)
clipoutside
True
Clip outside the range, or within it
channelavg
False
Pre-average data across channels before analyzing visibilities for flagging
chanbin
int(1)
Bin width for channel average in number of input channels
timeavg
False
Pre-average data across time before analyzing visibilities for flagging.
timebin
'0s'
Bin width for time average in seconds
clipzeros
False
Clip zero-value data
quackinterval
float(1.0)
Quack n seconds from scan beginning or end
quackmode
'beg'
Quack mode. beg: first n seconds of scan; endb: last n seconds of scan; end: all but first n seconds of scan; tail: all but last n seconds of scan.
quackincrement
False
Flag incrementally in time?
tolerance
float(0.0)
Amount of shadow allowed (in meters)
addantenna
''
File name or dictionary with additional antenna names, positions and diameters
lowerlimit
float(0.0)
Lower limiting elevation (in degrees)
upperlimit
float(90.0)
Upper limiting elevation (in degrees)
ntime
'scan'
Time-range to use for each chunk (in seconds or minutes)
combinescans
False
Accumulate data across scans depending on the value of ntime.
timecutoff
float(4.0)
Flagging thresholds in units of deviation from the fit
freqcutoff
float(3.0)
Flagging thresholds in units of deviation from the fit
timefit
'line'
Fitting function for the time direction (poly/line)
freqfit
'poly'
Fitting function for the frequency direction (poly/line)
maxnpieces
int(7)
Number of pieces in the polynomial-fits (for “freqfit” or “timefit” = “poly”)
flagdimension
'freqtime'
Dimensions along which to calculate fits (freq/time/freqtime/timefreq)
usewindowstats
'none'
Calculate additional flags using sliding window statistics (none,sum,std,both)
halfwin
int(1)
Half-width of sliding window to use with “usewindowstats” (1,2,3).
extendflags
True
Extend flags along time, frequency and correlation.
winsize
int(3)
Number of timesteps in the sliding time window [aips:fparm(1)]
timedev
''
Time-series noise estimate [aips:noise]
freqdev
''
Spectral noise estimate [aips:scutoff]
timedevscale
float(5.0)
Threshold scaling for timedev [aips:fparm(9)]
freqdevscale
float(5.0)
Threshold scaling for freqdev [aips:fparm(10)]
spectralmax
float(1E6)
Flag whole spectrum if freqdev is greater than spectralmax [aips:fparm(6)]
spectralmin
float(0.0)
Flag whole spectrum if freqdev is less than spectralmin [aips:fparm(5)]
antint_ref_antenna
''
Antenna of interest. Baselines with this antenna will be checked for flagged channels.
minchanfrac
float(0.6)
Minimum fraction of flagged channels required for a baseline to be deemed as flagged
verbose
False
Print timestamps of flagged integrations to the log
extendpols
True
If any correlation is flagged, flag all correlations
growtime
float(50.0)
Flag all “ntime” integrations if more than X% of the timerange is flagged (0-100)
growfreq
float(50.0)
Flag all selected channels if more than X% of the frequency range is flagged(0-100)
growaround
False
Flag data based on surrounding flags
flagneartime
False
Flag one timestep before and after a flagged one (True/False)
flagnearfreq
False
Flag one channel before and after a flagged one (True/False)
minrel
float(0.0)
minimum number of flags (relative)
maxrel
float(1.0)
maximum number of flags (relative)
minabs
int(0)
minimum number of flags (absolute)
maxabs
int(-1)
maximum number of flags (absolute). Use a negative value to indicate infinity.
spwchan
False
Print summary of channels per spw
spwcorr
False
Print summary of correlation per spw
basecnt
False
Print summary counts per baseline
fieldcnt
False
Produce a separated breakdown for each field
name
'Summary'
Name of this summary report (key in summary dictionary)
action
'apply'
Action to perform in MS and/or in inpfile (none/apply/calculate)
display
''
Display data and/or end-of-MS reports at runtime (data/report/both).
flagbackup
True
Back up the state of flags before the run
savepars
False
Save the current parameters to the FLAG_CMD table or to a file
cmdreason
''
Reason to save to output file or to FLAG_CMD table.
outfile
''
Name of output file to save current parameters. If empty, save to FLAG_CMD
overwrite
True
Overwrite an existing file to save the flag commands
writeflags
True
Internal hidden parameter. Do not modify.