\n",
" COLUMN NAME DESCRIPTION\n",
" ----------- -----------\n",
" Date/Time Time stamp of data sample (YYMMDD/HH:MM:SS UT)\n",
" Intrf Interferometer baseline (antenna names)\n",
" UVDist uv-distance (units of wavelength)\n",
" Fld Field ID (if more than 1)\n",
" SpW Spectral Window ID (if more than 1)\n",
" Chn Channel number (if more than 1)\n",
" (Correlated Correlated polarizations (eg: RR, LL, XY)\n",
" polarization) Sub-columns are: Amp, Phs, Wt, F\n",
" Amp Visibility amplitude\n",
" Phs Visibility phase (deg)\n",
" Wt Weight of visibility measurement\n",
" F Flag: 'F' = flagged datum; ' ' = unflagged\n",
" UVW UVW coordinates (meters)\n",
"
\n",
"\n",
"Note that MS listings can be very large. Use selectdata=True and subselect the data to obtain the desired information as much as possible. \n",
"\n",
"\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4JLWbkat0EC9"
},
"source": [
"## SDM Summary\n",
"\n",
"The task **listsdm** summarizes the content of archival data in the format that is used by ALMA and the VLA for visibility data transport and archiving (the [Science Data Model](casa-fundamentals.ipynb#science-data-model), known as ASDM for ALMA). The output in the casalog is very similar to that of **listobs**, which reads the metadata after conversion to a MeasurementSet (see next section). **listsdm** therefore does not require the SDM to be filled into an MS. The output of **listsdm** is also returned as a dictionary.\n",
"\n",
"\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wl_gV1m80EC9"
},
"source": [
"## MS Metadata List/Change\n",
"\n",
"\n",
"The **vishead** task is provided to access keyword information in the MeasurementSet. The default inputs are:\n",
"\n",
"```\n",
"#vishead :: List, get, and put metadata in a MeasurementSet\n",
"vis = '' #Name of input visibility file\n",
"mode = 'list' #options: list, summary, get, put\n",
"listitems = [] #items to list ([] for all)\n",
"```\n",
"\n",
"The *mode = 'summary'* option just gives the same output as **listobs**.\n",
"\n",
"For *mode = 'list'* the default options are: *'telescope', 'observer', 'project', 'field', 'freq_group_name', 'spw_name', 'schedule', 'schedule_type', 'release_date'.* To obtain other options, put *mode = 'list'* and *listitems = \\[\\]*; see [vishead task pages](../api/casatasks.rst) for a description of these additional options.\n",
"\n",
"```python\n",
"CASA <29>: vishead('ngc5921.demo.ms',mode='list',listitems=[])\n",
" Out[29]:\n",
"{'cal_grp': (array([-1, -1, -1], dtype=int32), {}),\n",
" 'field': (array(['1331+30500002_0', '1445+09900002_0', 'N5921_2'],\n",
" dtype='|S16'),\n",
" {}),\n",
" 'fld_code': (array(['C', 'A', ''],\n",
" dtype='|S2'), {}),\n",
" 'freq_group_name': (array(['none'],\n",
" dtype='|S5'), {}),\n",
" 'log': ({'r1': False}, {}),\n",
" 'observer': (array(['TEST'],\n",
" dtype='|S5'), {}),\n",
" 'project': (array([''],\n",
" dtype='|S1'), {}),\n",
" 'ptcs': ({'r1': array([[[-2.74392758]],\n",
" [[ 0.53248521]]]),\n",
" 'r2': array([[[-2.42044692]],\n",
" [[ 0.17412604]]]),\n",
" 'r3': array([[[-2.26020138]],\n",
" [[ 0.08843002]]])},\n",
" {'MEASINFO': {'Ref': 'J2000', 'type': 'direction'},\n",
" 'QuantumUnits': array(['rad', 'rad'],\n",
" dtype='|S4')}),\n",
" 'release_date': (array([ 4.30444800e+09]),\n",
" {'MEASINFO': {'Ref': 'TAI', 'type': 'epoch'},\n",
" 'QuantumUnits': array(['s'],\n",
" dtype='|S2')}),\n",
" 'schedule': ({'r1': False}, {}),\n",
" 'schedule_type': (array([''],\n",
" dtype='|S1'), {}),\n",
" 'source_name': (array(['1331+30500002_0', '1445+09900002_0', 'N5921_2'],\n",
" dtype='|S16'),\n",
" {}),\n",
" 'spw_name': (array(['none'],\n",
" dtype='|S5'), {}),\n",
" 'telescope': (array(['VLA'],\n",
" dtype='|S4'), {})}\n",
"```\n",
"\n",
"You can use *mode='get'* to retrieve the values of specific keywords, and likewise *mode='put'* to change them. The inputs are:\n",
"\n",
"```\n",
"mode = 'get' #options: list, summary, get, put\n",
"hdkey = '' #keyword to get/put\n",
"hdindex = '' #keyword index to get/put, counting from zero. ==>all\n",
"```\n",
"\n",
"and\n",
"\n",
"```\n",
"#vishead :: List, summary, get, and put metadata in a MeasurementSet\n",
"mode = 'put' #options: list, summary, get, put\n",
"hdkey = '' #keyword to get/put\n",
"hdindex = '' #keyword index to get/put, counting from zero. ==>all\n",
"hdvalue = '' #value of hdkey\n",
"```\n",
"\n",
"For example, a common operation is to change the Telescope name (e.g. if it is unrecognized), e.g.\n",
"\n",
"```\n",
"CASA <36>: vishead('ngc5921.demo.ms',mode='get',hdkey='telescope')\n",
" Out[36]:\n",
" (array(['VLA'],\n",
" dtype='|S4'), {})\n",
"\n",
"CASA <37>: vishead('ngc5921.demo.ms',mode='put',hdkey='telescope',hdvalue='JVLA')\n",
"\n",
"CASA <38>: vishead('ngc5921.demo.ms',mode='get',hdkey='telescope')\n",
" Out[38]:\n",
" (array(['JVLA'],\n",
" dtype='|S5'), {})\n",
"```\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pQ82NIMq0EC-"
},
"source": [
"## Visibility Statistics\n",
"\n",
"\n",
"The **visstat** task is provided to obtain simple statistics for a MeasurementSet, useful in regression tests.\n",
"\n",
"The inputs are:\n",
"\n",
"```\n",
"#visstat :: Displays statistical information from a MeasurementSet, or from a Multi-MS\n",
"\n",
"vis = '' #Name of MeasurementSet or Multi-MS\n",
"axis = 'real' #Which values to use\n",
" datacolumn = 'data' #Which data column to use (data, corrected, model, float_data)\n",
"\n",
"useflags = False #Take flagging into account?\n",
"spw = '' #spectral-window/frequency/channel\n",
"field = '1' #Field names or field index numbers: ''==>all, field='0~2,3C286'\n",
"selectdata = True #More data selection parameters (antenna, timerange etc)\n",
" antenna = '' #antenna/baselines: ''==>all, antenna = '3,VA04'\n",
" timerange = '' #time range: ''==>all, timerange='09:14:0~09:54:0'\n",
" correlation = 'RR' #Select data based on correlation\n",
" scan = '' #scan numbers: ''==>all\n",
" array = '' #(sub)array numbers: ''==>all\n",
" observation = '' #observation ID number(s): '' = all\n",
" uvrange = '' #uv range: ''==>all; uvrange = '0~100klambda', default units=meters\n",
"\n",
"timeaverage = False #Average data in time.\n",
"intent = '' #Select data by scan intent.\n",
"reportingaxes = 'ddid' #Which reporting axis to use (ddid, field, integration)\n",
"```\n",
"\n",
"\n",
"\n",
"Running this task returns a record (Python dictionary) with the statistics, which can be captured in a Python variable. For example,\n",
"\n",
"```python\n",
"CASA <54>: mystat=visstat(vis='data/regression/unittest/setjy/ngc5921.ms', axis='amp', datacolumn='data', useflags=False, spw='', field='', selectdata=True, correlation='RR', timeaverage=False, intent='', reportingaxes='ddid')\n",
"\n",
"CASA <55>: mystat\n",
"Out[55]:\n",
"{'DATA_DESC_ID=0': {'firstquartile': 0.023732144385576248,\n",
" 'isMasked': False,\n",
" 'isWeighted': False,\n",
" 'max': 73.75,\n",
" 'maxDatasetIndex': 12,\n",
" 'maxIndex': 1204,\n",
" 'mean': 4.511831488357214,\n",
" 'medabsdevmed': 0.0432449858635664,\n",
" 'median': 0.051963627338409424,\n",
" 'min': 2.2130521756480448e-05,\n",
" 'minDatasetIndex': 54,\n",
" 'minIndex': 4346,\n",
" 'npts': 1427139.0,\n",
" 'rms': 16.42971891790897,\n",
" 'stddev': 15.798076313999745,\n",
" 'sum': 6439010.678462409,\n",
" 'sumOfWeights': 1427139.0,\n",
" 'sumsq': 385235713.187832,\n",
" 'thirdquartile': 0.3004012107849121,\n",
" 'variance': 249.57921522295976}}\n",
"\n",
"CASA <56>: mystat['DATA_DESC_ID=0']['stddev']\n",
"Out[56]: 15.798076313999745\n",
"```\n",
"\n",
"```\n",
"The options for axis are:\n",
"```\n",
"\n",
"```\n",
"axis='amplitude' #or ('amp') axis='phase' axis='imag' (or 'imaginary') axis='real'\n",
"```\n",
"\n",
"The phase of a complex number is in radians with range (−π, π).\n",
"\n",
"\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "s2y3n7xS0EC-"
},
"source": [
"## Plot Antenna Positions\n",
"\n",
"This task is a simple plotting interface to produce plots of the antenna positions (taken from the ANTENNA sub-table of the MS). The location of the antennas in the MS will be plotted with X-toward local east, Y-toward local north.\n",
"\n",
"The inputs to **plotants** are:\n",
"\n",
"```\n",
" #plotants :: Plot the antenna distribution in the local reference frame:\n",
"vis = '' #Name of input visibility file (MS)\n",
"figfile = '' #Save the plotted figure to this file\n",
"antindex = False #Label antennas with name and antenna ID\n",
"logpos = False #Whether to plot logarithmic positions\n",
"exclude = '' #Antenna name/id selection to exclude from plot\n",
"checkbaselines = False #Whether to check baselines in the main table.\n",
"title = '' #Title for the plot.\n",
"showgui = True #Show plot on gui.\n",
"\n",
"```\n",
"\n",
"For most telescopes, the default X/Y plot is in meters. For VLBA antenna plots, latitude vs. longitude (degrees) is plotted instead.\n",
"\n",
"Supported format extensions for the *figfile* include emf, eps, pdf, png, ps, raw, rgba, svg, and svgz, depending on which python modules are installed on your system. Formats currently available in a downloaded CASA package include all but emf (enhanced metafile).\n",
"\n",
"Each antenna position is labeled with the antenna name. VLBA antenna plots label the positions with \\\"name @ station\\\" format, e.g. \\\"2\\@FD\\\" for the Fort Davis, Texas, antenna. To add the antenna ID to the name, set *antindex=True* as shown in Figure 1.\n",
"\n",
"{.image-inline}\n",
"\n",
">ALMA antenna positions with *antindex=True*.\n",
" \n",
"\n",
"By default, **plotants** plots the positions of all antennas in the ANTENNA subtable. However, the user has the option to exclude certain antennas with the *exclude* parameter. Its value is a string to select which antennas to exclude, using the same syntax as the antenna parameter in [MeasurementSet selection](visibility_data_selection.ipynb#Visibility-Data-Selection). For example, *exclude=\\\"5\\~6\\\"* would exclude the PM antennas from the plot in Figure 1.\n",
"\n",
"To plot only those antennas which appear in the MAIN table (e.g. after a split, which retains the entire ANTENNA subtable in the dataset), set *checkbaselines=True*. This parameter would have automatically removed antenna 7 (DV10) from the plot in Figure 1, as it does not appear in the main table of this dataset.\n",
"\n",
"To plot logarithmic positions instead of X/Y positions, set *logpos=True* as shown in Figure 2:\n",
"\n",
"{.image-inline}\n",
"\n",
">Antenna positions with *logpos=True*\n",
" \n",
"\n",
"The default title for the plot is \\\"Antenna Positions for \\\" the MS name (*vis* argument), as shown in all figures on this page. To set a custom title, set the *title* parameter to the desired string.\n",
"\n",
"\n",
"\n",
"**The plotants GUI**\n",
"\n",
"By default, the plotants GUI will be shown when the task is used. If the GUI is not needed, as in scripting mode to produce a *figfile*, set *showgui=False*. When casa flags are set to avoid starting GUI tools or to run without the matplotlib \\'tkagg\\' backend (*\\--nogui, \\--pipeline,* or *\\--agg*), the plotants GUI will not be shown regardless of the value of the *showgui* parameter.\n",
"\n",
"The antennas will be plotted in a plotter window as shown below. Several tool buttons are available to manipulate and save the plot:\n",
"\n",
"- The \\'Home\\' button (leftmost house icon) is used to return to the first, default view after panning or zooming.\n",
"- The \\'Forward\\' and \\'Back\\' buttons (left- and right-arrow icons) are used to navigate between previous plot views after pan/zoom actions.\n",
"- The \\'Pan/Zoom\\' button (crossed blue arrows, fourth icon) is used to drag the plot to a new position by pressing and holding the mouse button.\n",
"- The \\'Zoom-to-rectangle\\' button (magnifier icon, fifth from left) is used to mark a rectangular region with the mouse in order to zoom in on the plot.\n",
"- The \\'Subplot-configuration\\' button (sixth icon) can be used to stretch or compress the left, right, top, or bottom of the plot, as well as the ability to reset the plot to the original shape after manipulation before exiting the configuration dialog.\n",
"- The \\'Save\\' button (rightmost icon) is used to export the plot. A file save dialog is launched to select a location, name, and format (default png) for the file.\n",
"\n",
"{.image-inline}\n",
"\n",
">**plotants** GUI for a VLA dataset with *antindex=True*. Note the tool buttons at the bottom of the window.\n",
" \n",
"\n",
"\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MMrWfSUL0EC_"
},
"source": [
"## VLA Weather Information\n",
"\n",
"Weather data for the VLA can be displayed with the task **plotweather**. This task will also calculate opacities based on the weather data taken at the time of the observation, or from a seasonal model.\n",
"\n",
"Inputs are:\n",
"\n",
"```\n",
"#plotweather :: Plot elements of the weather table; estimate opacity.\n",
"vis = '' #MS name\n",
"seasonal_weight = 0.5 #weight of the seasonal model\n",
"doPlot = True #set this to True to create a plot\n",
"plotName = '' #(Optional) the name of the plot file\n",
"```\n",
"\n",
"The amount of seasonal data can be set by the parameter *seasonal_weight*, where a value of 1 will only use the seasonal model and a value of 0 will only use the actual weather data to calculate opacities.\n",
"\n",
"Typical output of **plotweather** looks like below:\n",
"\n",
"{.image-inline}\n",
"\n",
">Typical output from plotweather. The panel at the top displays the following properties as a fiunction of time across the observation: elevation of the sun, wind speed and direction, temperature and dew point, and precipitable water vapor (pwv). The bottom panel shows the calculated zenith opacity as a function of frequency. The opacities calculated from the actual weather data, from a seasonal model and the specified mix of both are shown in the PWV and Tau plots.\n",
" \n",
"\n",
"The methods used in this task are described [EVLA Memo 143](http://library.nrao.edu/public/memos/evla/EVLAM_143.pdf), [VLA Test Memo 232](http://library.nrao.edu/public/memos/vla/test/VLAT_232.pdf), and [VLA Scientific Memo 176](http://library.nrao.edu/public/memos/vla/sci/VLAS_176.pdf). The wind direction aligns with the meteorological definition, i.e., north is up (0deg) with the angle increasing clockwise E, S, W (e.g., a vector pointing to the right indicates westerly winds with an angle of 270deg). \n",
"\n",
"\n",
"\n",
"Allowed output plot formats are those supported by matplotlib, currently emf, eps, pdf, png, ps, raw, rgba, svg, and svgz.\n",
"\n",
"\n",
"\n",
"\n",
"Alert: **plotweather** accesses the WEATHER table in the MS. The task may therefore also work for non-VLA data as long as such a table is present. The plots and calculations, however, have been tailored for the VLA, so non-VLA data may or may not be interpreted correctly. \n",
"
\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Zun0Bh1G0EDA"
},
"source": [
"## Browse MS/Calibration Tables\n",
"\n",
"The *browsetable* task is available for viewing data directly. It handles all CASA tables, including MeasurementSets, calibration tables, and images. This task brings up a CASA Qt table browser.\n",
"\n",
"browsetable is not required for normal data reduction but is useful for troubleshooting or for identifying table column names and formats. If you want to edit a long column or extract data for manipulation outside CASA (e.g. the uv data), see flagdata and the table tools in the [Global Tool List](../api/casatools.rst). The MeasurementSet columns and subtables are described [here](casa-fundamentals.ipynb#measurementset-v2).\n",
"\n",
"For CASA 6, inp/go no longer works with browsetable, and the application can no longer be launched from outside CASA using **casabrowser**. Browsetable should be invoked using the argument:\n",
"\n",
"```\n",
"browsetable('ngc5921_ut.ms')\n",
"```\n",
"\n",
"Available sub-parameters are given on the [browsetable task pages](../api/tt/casalith.browsetable.html).\n",
"\n",
"\n",
"\n",
"For an MS, as in this example, the table browser will display the MAIN table (Figure 1). To look at subtables, use the *table keywords* tab along the left side to bring up a panel with the subtables listed (Figure 2), then choose (double-click) a table name (Keyword) to display the subtable in a new tab (Figures 3 and 4). You can double-click on a cell in a table to view the contents (fourth figure below) then use the \\\"Close\\\" or \\\"Close All\\\" buttons at the bottom of the contents display to close one or all displayed values.\n",
"\n",
"{.image-inline}\n",
"\n",
">The browser displays the MAIN table within a frame. You can scroll through the data with the sliders at right and bottom, and step through the pages with the \\\"\\<\\<\\\" and \\\"\\>\\>\\\" buttons or using \\\"First\\\" and \\\"Last\\\" to quickly advance to the beginning or end. To go to a specific page, input the page number in the text box then click \\\"Go\\\". By default, 1000 rows of the table are loaded at a time, but you can specify this setting in the \\\"Loading \\... rows\\\" text box.\n",
"\n",
"{.image-inline}\n",
"\n",
">Use the \\\"table keywords\\\" tab to look at other tables within an MS. Double-click on a table name to view its contents in a new tab, as shown in the following figures.\n",
"\n",
"{.image-inline}\n",
"\n",
">Viewing the *ANTENNA* table of the MS.\n",
"\n",
"{.image-inline}\n",
"\n",
">The *POLARIZATION* table shows the number and types of correlations. The *CORR_TYPE* integer array indicates the Stokes type as defined in the Stokes class enumeration. Common types include RR (5), RL (6), LR (7), and LL (8) for circular polarization, and XX (9), XY (10), YX (11), and YY (12) for linear polarization.\n",
"\n",
"{.image-inline}\n",
"\n",
">Double-click a cell in the table or sub-table to see its value displayed to the right. Here, the *DATA* column cell (top right) contains a \\[2,63\\] array of complex numbers. The *WEIGHT_SPECTRUM* for this data is shown below it as a \\[2,63\\] array of float values. Use the sliders to see other values in the arrays, and click \\\"Close\\\" to close the cell contents display or \\\"Close All\\\" to close all contents displays.\n",
"\n",
"Many options are available on the browsetable toolbar and menus:\n",
"\n",
"- To open a table, click the \\\"Open Table\\\" button or use the **File \\> Open Table** menu to open a file browser dialog box.\n",
"- To close a table, click the \\\"Close Table\\\" button to close the table in the active tab, or use the options on the \\\"File\\\" menu to close the table in the active tab (\\\"Close Table\\\"), to select an open table to close (\\\"Close\\...\\\"), or to close all tables (\\\"Close All\\\"). You can also \\\"Close All and Exit\\\" the table browser.\n",
"- To edit the table and its contents, click the \\\"Edit Table\\\" button or use the **Edit \\> Edit Table** menu. You can also use the \\\"Edit**\\\"** menu to add a row to the table. Be careful with this, and make a backup copy of the table before editing!\n",
"- To view table information, click the \\\"Table Information\\\" (blue ***\\\"i\\\"*** button) or use the **View** **\\> Table Information** menu. You can also hover the mouse pointer over the table name tab to get a popup with information.\n",
"- To set a TaQL filter, click the \\\"Filter on Fields\\\" button or use the **View \\> Filter on Fields** menu. This will open a \\\"Filter Rules\\\" dialog box within the table browser in which to set the filter. Another option is to use the *taql* parameter in the browsetable() call.\n",
"- To choose which columns to display, use **View \\> Columns** to select the columns from a list, which you can select individually or toggle with \\\"Show All Columns\\\" or \\\"Hide All Columns\\\". Another option is to use the *skipcols* parameter in the browsetable() call.\n",
"- To format the contents of the column cells, use **View \\> Format Display** to select a column then choose its formatting (depending on its type). For example, for numerical values you can set the precision and choose to use scientific format, or set the font and color for negative and nonnegative values.\n",
"- To find data using filter rules, click the \\\"Find\\\" button or use **Tools \\> Find** to open a Search Rules dialog box.\n",
"- To sort the table, click the \\\"Sort\\\" button, use the **Tools \\> Sort** menu to open a Table Sorter dialog box in which you can select the sort columns,or just click on the column name. Another option is to use the *sortlist* parameter in the browsetable() call.\n",
"- To plot table data, click the \\\"Plot 2D\\\" button or use the **Tools \\> Plot 2D** menu to open a Plot Options dialog box where you can select the rows and axes to plot, along with plot display options. Click \\\"Overplot\\\" or \\\"Clear and Plot\\\" to make the plot in the Table Browser Plotter window. There is also an option to export the plot; select PNG or JPG format and click Go.\n",
"- Currently, **Export \\> VOTable** results in a Fatal IO Error and kills the table browser.\n",
"- The default display is 1000 rows, but this can be set in the input box at the lower right. To page through the table, use the PAGE NAVIGATION buttons to advance forward or backward one page, or go directly to the First or Last page. You can also enter a page number and click Go.\n",
"- To exit the table browser, use **File \\> Exit** or click the Close \\\"X\\\" button at the upper right of the window.\n",
"\n",
"\n",
"**Alert:** You are likely to find that browsetable needs to get a table lock before proceeding. Use the **clearstat** command to clear the lock status in this case. You may also be unable to use other tasks on the table while it is open in the table browser.\n",
"
\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HG36KbeS0EDB"
},
"source": [
"## Plot/Flag Visibilities\n",
"\n",
"A number of CASA tasks handle the plotting and flagging of visibilities. The following subsections describe the usage of the relevant tasks:\n",
"\n",
"- **plotms** \\-\\-- create X-Y plots of data in MS and calibration tables, flag data\n",
"- **flagdata** \\-\\-- data flagging\n",
"- **flagcmd** \\-\\-- manipulate and apply flags using FLAG_CMD table\n",
"- **flagmanager** \\-\\-- manage versions of data flags\n",
"- **msview** \\-\\-- two-dimensional viewer used for manipulating visibilities\n",
"\n",
"\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-TZcgaOs0EDB"
},
"source": [
"## Plot/Edit using plotms\n",
"\n",
"**plotms** is a GUI-style plotter, based on Qt, for creating X-Y plots of visibility data and calibration tables. It should be started as a task within CASA. This task also provides editing capability.\n",
"\n",
"**plotms** was originally intended to plot MeasurementSets (the \\\"ms\\\" in \\\"plotms\\\") but has been extended to include calibration tables. Supported cal table types include B Jones, B TSYS, BPOLY, D Jones, Df Jones, DfLLS Jones, EGainCurve, F Jones, Fringe Jones, G Jones, GlinXphf Jones, G EVLASWPOW, GSPLINE, K Jones, KAntPos Jones, Kcross Jones, T Jones, TOpac, Xf Jones, A Mueller, M Mueller, and SDSKY_PS (single-dish sky calibration). Some axis choices do not apply to calibration tables, and the calibration axes do not apply to MeasurementSets. Selection can be applied to calibration tables; where relevant, channel selection has been implemented and tested for the cal tables listed. Averaging cannot be used for BPOLY and GSPLINE tables, which use an older table format. Some options, such as certain axes and transformations, cannot be used for calibration tables.\n",
"\n",
"For simplicity, this document primarily addresses plotting MeasurementSets.\n",
"\n",
"The current inputs and default values for **plotms** include:\n",
"\n",
"```\n",
"plotms :: A plotter/interactive flagger for visibility data.\n",
"vis = '' # input MS or CalTable (blank for none)\n",
"gridrows = 1 # number of subplot rows (default 1). \n",
"gridcols = 1 # number of subplot columns (default 1).\n",
"rowindex = 0 # row location of the plot (0-based, default 0)\n",
"colindex = 0 # column location of the plot (0-based, default 0)\n",
"plotindex = 0 # index to address a subplot (0-based, default 0)\n",
"xaxis = '' # plot x-axis (blank for default/current) \n",
"yaxis = '' # plot y-axis (blank for default/current) \n",
"selectdata = True # data selection parameters \n",
" field = '' # field names or field index numbers (blank for all)\n",
" spw = '' # spectral windows:channels (blank for all)\n",
" timerange = '' # time range (blank for all)\n",
" uvrange = '' # uv range (blank for all)\n",
" antenna = '' # antenna/baselines (blank for all)\n",
" scan = '' # scan numbers (blank for all)\n",
" correlation = '' # correlations/polarizations (blank for all)\n",
" array = '' # (sub)array numbers (blank for all)\n",
" observation = '' # observation ID(s) (blank for all)\n",
" intent = '' # observing intent (blank for all)\n",
" feed = '' # feed (blank for all)\n",
" msselect = '' # MS selection (blank for all)\n",
"averagedata = True # data averaging parameters\n",
" avgchannel = '' # average over channel (blank = False, otherwise value in channels)\n",
" avgtime = '' # average over time (blank = False, other value in seconds)\n",
" avgscan = False # average over scans if time averaging is enabled\n",
" avgfield = False # average over fields if time averaging is enabled\n",
" avgbaseline = False # average over all baselines (mutually exclusive with avgantenna)\n",
" avgantenna = False # average by per-antenna (mutually exclusive with avgbaseline)\n",
" avgspw = False # average over all spectral windows\n",
" scalar = False # do scalar averaging\n",
"transform = False # transform data in various ways\n",
"extendflag = False # extend flagging to other data points\n",
"iteraxis = '' # axis over which to iterate\n",
"customsymbol = False # set a custom symbol for unflagged points\n",
"coloraxis = '' # set data axis to use for colorizing\n",
"customflaggedsymbol = False # set a custom plot symbol for flagged points\n",
"xconnector = '' # set connector for data points (blank=\"none\"; \"line\",\"step\")\n",
"plotrange = [] # plot axes ranges: [xmin,xmax,ymin,ymax]]\n",
"title = '' # title written along top of plot\n",
"titlefont = 0 # font for plot title\n",
"xlabel = '' # text for horizontal axis. Blank for default.\n",
"xaxisfont = 0 # font for plot x-axis\n",
"ylabel = '' # text for vertical axis. Blank for default.\n",
"yaxisfont = 0 # font for plot y-axis\n",
"showmajorgrid = False # show major grid lines (horiz and vert.)\n",
"showminorgrid = False # show minor grid lines (horiz and vert.)\n",
"showlegend = False # show a legend on the plot\n",
"plotfile = '' # name of plot file to save automatically\n",
"showgui = True # show GUI\n",
"clearplots = True # remove any existing plots (do not overplot)\n",
"callib = [''] # calibration library string or filename for on-the-fly calibration.\n",
"headeritems = '' # comma-separated list of pre-defined page header items\n",
"showatm = False # compute and overlay the atmospheric transmission curve\n",
"showtsky = False # compute and overlay the sky temperature curve\n",
"showimage = False # compute and overlay the image sideband curve\n",
"colorizeoverlay = False # colorize the curve overlay using coloraxis\n",
"```\n",
"\n",
"Note that when some parameters are set or are True, their subparameters are displayed by inp( ). By default, selectdata, averagedata, and showgui are True and their subparameters are shown above. Other parameters with subparameters include:\n",
"\n",
"```\n",
"xaxis = 'real' # plot x-axis (blank for default/current) \n",
" xdatacolumn = '' # data column for x-axis (blank for default/current) \n",
"yaxis = 'imag' # plot y-axis (blank for default/current)\n",
" ydatacolumn = '' # data column for y-axis (blank for default/current)\n",
" yaxislocation = 'left' # set yaxis to the left of the plot\n",
"transform = True # transform data in various ways?\n",
" freqframe = '' # frame in which to render frequency and velocity axes\n",
" restfreq = '' # rest frequency to use for velocity conversions\n",
" veldef = 'RADIO' # definition in which to render velocity\n",
" phasecenter = '' # direction coordinates of new phase center\n",
"extendflag = True # extend flagging to other data points\n",
" extcorr = False # extend flags based on correlation\n",
" extchannel = False # extend flags based on channel\n",
"iteraxis = 'baseline' # axis over which to iterate\n",
" xselfscale = False # use common x-axis range (scale) for iterated plots\n",
" yselfscale = False # use common y-axis range (scale) for iterated plots\n",
" xsharedaxis = False # enable iterated plots on a grid to share a common external x-axis per column\n",
" ysharedaxis = False # enable iterated plots on a grid to share a common external y-axis per row\n",
"customsymbol = True # set a custom symbol for unflagged points\n",
" symbolshape = 'autoscaling' # shape of plotted unflagged symbols\n",
" symbolsize = 2 # size of plotted unflagged symbols\n",
" symbolcolor = '0000ff' # color of plotted unflagged symbols\n",
" symbolfill = 'fill' # fill type of plotted unflagged symbols\n",
" symboloutline = False # select outlining plotted unflagged points\n",
"customflaggedsymbol = True # set a custom plot symbol for flagged points\n",
" flaggedsymbolshape = 'nosymbol # shape of plotted flagged symbols\n",
" flaggedsymbolsize = 2 # size of plotted flagged symbols\n",
" flaggedsymbolcolor = 'ff0000' # color of plotted flagged symbols\n",
" flaggedsymbolfill = 'fill' # fill type of plotted flagged symbols\n",
" flaggedsymboloutline = False # select outlining plotted flagged points\n",
"showmajorgrid = True # show major grid lines (horiz and vert.)\n",
" majorwidth = 0 # line width in pixels of major grid lines\n",
" majorstyle = '' # major grid line style: solid dash dot none\n",
" majorcolor = '' # color of major grid lines as name or hex code\n",
"showminorgrid = True # show minor grid lines (horiz and vert.)\n",
" minorwidth = 0 # line width in pixels of minor grid lines\n",
" minorstyle = '' # minor grid line style: solid dash dot none\n",
" minorcolor = '' # color of minor grid lines as name or hex code\n",
"plotfile = 'plot.jpg' # name of plot file to save automatically\n",
" expformat = '' # export format type (jpg, png, ps, pdf, txt), else use plotfile extension\n",
" verbose = True # include metadata in text export\n",
" exprange = '' # export all iteration plots or only the current one\n",
" highres = False # use high resolution\n",
" dpi = -1 # DPI of exported plot\n",
" width = -1 # width of exported plot\n",
" height = -1 # height of exported plot\n",
" overwrite = False # overwrite plot file if it already exists\n",
"```\n",
"\n",
"Note that if the *vis* parameter is set to the name of a MeasurementSet here, when you start **plotms** the *entire* MeasurementSet will be plotted, which can be time consuming. You may want to set selection or averaging parameters first.\n",
"\n",
"To start a \\\"blank\\\" **plotms** window then enter your selections interactively in the GUI, use these commands:\n",
"\n",
"```\n",
"default plotms\n",
"plotms\n",
"```\n",
"\n",
"Alternatively, they can be specified as task parameters in a **plotms** call, for scripting:\n",
"\n",
"```\n",
"plotms(vis1, yaxis='phase', ydatacolumn='corrected', xaxis='frequency', coloraxis='spw', antenna='1', spw='0:3~10', corr='RR', avgtime='1e8', plotfile='vis1.jpg')\n",
"\n",
"```\n",
"\n",
"Note that subsequent **plotms** calls will return any unspecified parameters in that call to their default values. See also the Examples tab in the **plotms** [task](../api/casatasks.rst) for **plotms** calls using many of the parameters.\n",
"\n",
"The **plotms** GUI will be described in the following sections, along with the corresponding parameters for the task interface or scripting. For non-interactive scripting, set *showgui=False* and export the plot into an image specified by *plotfile.*\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4DGTJNyt0EDC"
},
"source": [
"### The Plot Tab\n",
"\n",
"**Loading, Selecting, and Averaging Data: the Plot Data Tab**\n",
"\n",
"{.image-inline}\n",
"\n",
">The plotms window starts on the *Plot \\> Data* tab. No parameters have been set.\n",
" \n",
"\n",
"*File Selection*\n",
"\n",
"When **plotms** is first started, by default it will display the *Plot* tab (as chosen from the tabs at the top of the **plotms** window) and its *Data* subtab (as chosen from the tabs on the left side) as shown in Figure 1. First, a MeasurementSet or calibration table should be loaded by clicking on *Browse* in the *File* section and selecting a MeasurementSet directory (just select the directory itself; do not descend into it).\n",
"\n",
"A plot can now be made of the MeasurementSet by clicking on the *Plot* button, but you may want to set selection or averaging parameters first rather than plot the entire dataset. By default, **plotms** will plot Amplitude versus Time for a MeasurementSet; see the Axes Tab section for axis options. The default axes change for calibration tables depending on the table type. **plotms** self-scales axes and the symbol size. For a very large range, this can hide points close to zero; see the Axes Tab section for setting axis ranges and the Display Tab section for setting symbol size.\n",
"\n",
"The **plotms** task parameter for file selection is *vis.*\n",
"\n",
"*Data Selection*\n",
"\n",
"The options for data selection are:\n",
"\n",
"- field\n",
"- spw\n",
"- timerange\n",
"- uvrange\n",
"- antenna\n",
"- scan\n",
"- corr (correlated polarizations)\n",
"- array\n",
"- observation\n",
"- intent\n",
"- feed\n",
"- msselect\n",
"\n",
"Note that, unlike when setting data selection parameters from the CASA command line, no quotation marks are needed around strings in the GUI. For more information on data selection strings, see the documentation [here](visibility_data_selection.ipynb#Visibility-Data-Selection). To view information about your data in order to make your selection, use the Summary menu or the listobs task.\n",
"\n",
"Plotms correlation selection supports standard Stokes parameters (I, Q, U, V), and polarization quantities (Ptotal, Plinear, PFtotal, PFlinear, and Pangle). When the MeasurementSet does not have the requested quantities, they will be computed. The selection must be combined with a visibility axis (amp, phase, real, imag) since the derived quantities may be complex.\n",
"\n",
"Calibration table selection may differ from MeasurementSet selection:\n",
"\n",
"- *antenna* selection for a calibration table depends on its type.\n",
"1. For antenna-based cal tables without a reference antenna, only ANTENNA1 is matched and the \"&\" operators in selection expressions are ignored. Single-dish sky calibration tables use ANTENNA1 selection.\n",
"2. For antenna-based cal tables with a reference antenna, ANTENNA2 is interpreted as a reference antenna and matched against the ANT2 in \"ANT1&ANT2\" type expressions. \"ANT\" selections continue to match ANTENNA1 only.\n",
"3. For baseline-based cal tables, antenna selection uses both ANTENNA1 and ANTENNA2 as described in the MSSelection documentation.\n",
"- *corr* selection is used to select calibration table polarizations, including \\\"/\\\" for a ratio plot.\n",
"\n",
"\n",
"The **plotms** task parameter for data selection is *selectdata* (default is True, but no selection occurs unless one or more subparameters is set). Its subparameters include *field, spw, timerange. uvrange, antenna, scan, correlation, array, observation, intent, feed,* and *msselect.* These should be set to string values.\n",
"\n",
"*Averaging Data*\n",
"\n",
"**plotms** enables averaging of the data in order to increase signal-to-noise of the plotted points or to increase plotting speed.\n",
"\n",
"Averaging is currently not supported for *Ant-Ra* and *Ant-Dec* axes and will result in a warning in the log, then the unaveraged data will be plotted.\n",
"\n",
"Averaging is supported for calibration tables with the exception of BPOLY and GSPLINE tables, which use an older table format. \n",
"\n",
"The options for averaging in the *Plot \\> Data* tab include:\n",
"\n",
"- channel\n",
"- time (optionally over scans or fields)\n",
"- all baselines or per antenna\n",
"- all spectral windows\n",
"- vector (default) or scalar\n",
"\n",
"The box next to a given averaging mode needs to be checked for that averaging to take effect. The Weight and Sigma axes are not supported in some averaging modes. Note that the \\\"average weight\\\" is actually the weight *sum* accumulated when performing the average; i.e., the net weight of a weighted-averaged datum is the sum of the weights going into the average.\n",
"\n",
"When averaging, **plotms** will prefer unflagged data. If an averaging bin contains any unflagged data at all, only the average of the unflagged will be shown. For averaging bins that contain *only* unflagged data, the average of that unflagged data will be shown. When flagging on a plot of averaged data, the flags will be applied to the unaveraged data in the MS.\n",
"\n",
"The **plotms** task parameter for averaging is *averagedata* (default is True, but no averaging occurs unless one or more subparameters are set). It subparameters include *avgchannel* and *avgtime* (set to a string value in channels or seconds, default \\\"\\\"), and boolean parameters *avgscan, avgfield, avgbaseline, avgantenna, avgspw, and scalar* (True/False, default False). Invalid combinations of averaging will result in an error message (e.g. *avgbaseline=True, avgantenna=True)* or will be ignored (e.g. *avgscan=True* but *avgtime* has not been set).\n",
"\n",
"- *Channel Averaging*: to average n channels together, the user would click on the box next to *Channel* so that an \"X\" appears in it, and then type the number n in the empty box. When the user next clicks on *Plot*, every n channels will then be averaged together and plotted against the average channel numbers. The total number of channels plotted will be decreased by a factor of n.\n",
" - Channel selection may be combined with channel averaging. For MeasurementSets, each selected channel range is binned and averaged individually, but for calibration tables, the selected channels are treated as contiguous. See examples in the **plotms** [task](../api/casatasks.rst) channel averaging documentation.\n",
" - **Warning**: If a complex channel selection is made e.g. of continuum in the presense of multiple lines, channel averaging is unlikely to produce a meaningful plot.\n",
" - The averaged channel ids are reindexed starting at 0 to reflect the **bin number**, not the averaged channel number.\n",
" - The averaged frequency and velocity values are the average of the frequencies/velocities in the bin.\n",
"\n",
"- *Time Averaging*: Time averaging is controlled by three fields. If the checkbox next to *Time* is checked, a blank box with units of seconds will become active, along with two additional checkboxes: *Scan* and *Field*. If averaging is desired over a relatively short interval (say, 30 seconds, shorter than the scan length), a number can simply be entered into the blank box and, when the data are replotted, the data will be time averaged. Clicking on the *Scan* or *Field* checkbox in this case will have no impact on the time averaging. These checkboxes become relevant if averaging over a relatively long time---say the entire observation, which consists of multiple scans---is desired. Regardless of how large a number is set in the *Time* averaging box, only data within individual scans will be averaged together. In order to average data across scan boundaries, the *Scan* checkbox must be checked and the data replotted. Finally, clicking on the *Field* checkbox enables the averaging of multiple fields together in time.\n",
" - When averaging over scan, the scan number is the first scan number in the bin, independent of unflagged/flagged data.\n",
" - When averaging over field, the field number is the first field number in the bin, independent of unflagged/flagged data.\n",
"\n",
"- *Averaging All Baselines/Per Antenna*: Clicking on the *All Baselines* checkbox will average all baselines in the array together. Alternatively, the *Per Antenna* box may be checked, which will average all baselines for a given antenna together. In this case, all baselines are represented twice; baseline 3-24 will contribute to the averages for both antenna 3 and antenna 24. This can produce some rather strange-looking plots if the user also selects on antenna---say, if the user requests to plot only antenna 0 and then averages *Per Antenna*, In this case, an average of all baselines including antenna 0 will be plotted, but each individual baseline including antenna 0 will also be plotted (because the presence of baselines 0-1, 0-2, 0-3, etc. trigger *Per Antenna* averaging to compute averages for antennae 1, 2, 3, etc. Therefore, baseline 0-1 will contribute to the average for antenna 0, but it will also singlehandedly be the average for antenna 1.) These averaging modes currently do not support the Weight and Sigma axes.\n",
"\n",
"- *Averaging All Spectral Windows*: Spectral windows can be averaged together by checking the box next to *All Spectral Windows*. This will result in, for a given channel n, all channels n from the individual spectral windows being averaged together. This averaging mode currently does not support the Weight and Sigma axes.\n",
"\n",
"- *Vector/Scalar Averaging*: The default mode is vector averaging, where the complex average is formed by averaging the real and imaginary parts of the relevant visibilities. If *Scalar* is chosen, then the amplitude of the average is formed by a scalar average of the individual visibility amplitudes.\n",
"\n",
"*Brief Note Regarding plotms Memory Usage*\n",
"\n",
"In order to provide a wide range of flexible interactive plotting options while minimizing the I/O burden and speeding up the plotting, **plotms** caches the data values for the plot (along with a subset of relevant meta-info) in as efficient a manner as possible. Sometimes, however, the data changes on disk, for example when other data processing tasks are applied. To force plotms to reload the data, check the *Reload* box next to the *Plot* button or press the SHIFT key while clicking the *Plot* button.\n",
"\n",
"For plots of large numbers of points, the total memory requirement can be quite large. **plotms** attempts to predict the memory it will require (typically 5 or 6 bytes per plotted point when only one axis is a data axis, depending upon the data shapes involved), and will complain if it believes there is insufficient memory to support the requested plot. For most practical interactive purposes (plots that load and draw in less than a few or a few 10s of minutes), there is usually not a problem on typical modern workstations. Attempts to plot large datasets on small laptops might be more likely to encounter problems here.\n",
"\n",
"The absolute upper limit on the number of simultaneously plotted points is currently set by the ability to index the points in the cache. For modern 64 bit machines, this is about 4.29 billion points (requiring around 25GB of memory). Such plots are not especially useful interactively, since the I/O and draw become prohibitive.In general, it is usually most efficient to plot data in modest chunks of no more than a few hundred million points or less, either using selection or averaging. Note that all iterations are (currently) cached simultaneously for iterated plots, so iteration is not a way to manage memory use. A few hundred million points tends to be the practical limit of interactive **plotms** use with respect to information content and utility in the resulting plots, especially when you consider the number of available pixels on your screen.\n",
"\n",
"\n",
"**On-The-Fly Calibration: the Plot Calibration Tab**\n",
"\n",
"{.image-inline}\n",
"\n",
">The plotms Calibration tab. This MeasurementSet has no *CORRECTED_DATA* column. A calibration library file was selected with the file browser and applied on the fly.\n",
" \n",
"\n",
"One can apply calibration tables to the uncalibrated data on the fly, i.e. without a run of applycal beforehand, by specifying a calibration library and selecting the *corrected* Data Column for the plotted axes. See the [Cal Library Syntax](cal_library_syntax.ipynb) documentation for more information on specifying calibration in a string or file.\n",
"\n",
"The *Calibration* tab on the left hand side contains a field to specify a calibration library file, or use *Browse* to open a file selection dialog. You can also specify the calibration library commands directly in a string. There is a switch to apply the calibration library to produce the corrected data (*Calibration On*) or to show an existing *CORRECTED_DATA* column (*Calibration Off*). If the *corrected* Data Column is requested but the column is not present in the MS and the calibration library is not set or enabled, **plotms** issues a warning and plots the *DATA* column instead.\n",
"\n",
"The **plotms** task parameter *'callib'* can be used to provide a calibration library file or a string containing the cal library commands. It is enabled by default when the parameter is set.\n",
"\n",
"\n",
"**Selecting Plot Axes: The Plot Axes Tab**\n",
"\n",
"{.image-inline}\n",
"\n",
">The plotms *Plot* *\\> Axes* tab, used here to make a plot of Amp vs. Channel.\n",
"\n",
"*Selecting Axes*\n",
"\n",
"The X and Y axes of a plot are selected by clicking on the *Plot \\> Axes* tab and choosing an entry from the drop-down menus below *X Axis* and *Y Axis*. The axes are grouped by type and listed in this order:\n",
"\n",
"**Metadata:**\n",
" - **Scan** --- The scan number, as listed by listobs or the **plotms** summary. When averaging over scan, the first scan number in the bin is used.\n",
" - **Field ---** The field number, as listed by listobs or the **plotms** summary. When averaging over field, the first field id in the bin is used.\n",
" - **Time ---**The time at which the visibility was observed, given in terms of calendar year (yyyy/mm/dd/hh:mm:ss.s).\n",
" - **Interval ---** The integration time in seconds.\n",
" - **Spw** --- The spectral window number. The characteristics of each spectral window are listed in listobs or the **plotms** summary.\n",
" - **Channel** --- The spectral channel number. When channel averaging, the channel ids are reindexed starting at 0 to reflect the **bin number**, not the averaged channel number.\n",
" - **Frequency** --- Frequency in units of GHz. The frame for the frequency (e.g., topocentric, barycentric, LSRK) can be set in the Plots \\> Transform tab.\n",
" - **Velocity** --- Velocity in units of km s−1, as defined by the Frame, Velocity Defn, and Rest Freq parameters in the Plots \\> Transform tab.\n",
" - **Corr** --- Correlations which have been assigned integer IDs, including RR (5), RL (6), LR (7), LL (8), XX (9), XY (10), YX (11), and YY (12). The axis values are these IDs, as listed by listobs or the **plotms** summary. This axis may also be used to plot polarizations for calibration tables.\n",
" - **Antenna1** --- The first antenna in a baseline pair; for example, for baseline 2-4, Antenna1= 2. Antennae are numbered according to the antenna IDs listed in listobs or the **plotms** summary.\n",
" - **Antenna2** --- The second antenna in a baseline pair; for baseline 2-4, Antenna2 = 4. Antennae are numbered according to the antenna IDs listed in listobs or the **plotms** summary.\n",
" - **Antenna** --- Antenna ID for plotting antenna-based quantities. Antennae are numbered according to the antenna IDs listed in listobs or the **plotms** summary.\n",
" - **Baseline** --- The baseline number.\n",
" - **Row** --- The MS data row number. A row number corresponds to a unique timestamp, baseline, and spectral window in the MeasurementSet.\n",
" - **Observation ---** The observation ID (index)\n",
" - **Intent** --- The intent ID (index)\n",
" - **Feed1** --- The first feed number, most useful for single-dish data.\n",
" - **Feed2** --- The second feed number, most useful for single-dish data.\n",
"\n",
"**Visibility values and flags:**\n",
" - **Amp** --- Data amplitudes in units which are proportional to Jansky (for data which are fully calibrated, the units should be in Jy).\n",
" - **Phase** --- Data phases in units of degrees.\n",
" - **Real** and **Imag** --- The real and imaginary parts of the visibility in units which are proportional to Jansky (for data which are fully calibrated, the units should be Jy).\n",
" - **Wt and **Wt\\*Amp** --- the weight of the visibility and the product of the weight and the amplitude.\n",
" - **WtSp** --- WEIGHT_SPECTRUM column, i.e. a weight per channel.\n",
" - **Sigma** --- the SIGMA column of the visibilities\n",
" - **SigmaSp** --- SIGMA_SPECTRUM column, i.e. a sigma per channel\n",
" - **Flag** --- Data which are flagged have Flag = 1, whereas unflagged data are set to Flag = 0. Note that, to display flagged data, you will have to click on the Plots \\> Display tab and choose a Flagged Points Symbol.\n",
" - **FlagRow** --- In some tasks, if a whole data row is flagged, then FlagRow will be set to 1 for that row. Unflagged rows have FlagRow = 0. However, note that some tasks (like **plotms**) may flag a row, but not set FlagRow = 1. It is probably better to plot Flag than FlagRow for most applications.\n",
"\n",
"**Observational geometry:**\n",
" - **UVdist** --- Projected baseline separations in units of meters. Note that UVDist is not a function of frequency.\n",
" - **UVwave** --- Projected baseline separations in units of the observing wavelength (lambda, not kilolambda). UVwave is a function of frequency, and therefore, there will be a different data point for each frequency channel.\n",
" - **U**, **V**, and **W** --- u, v, and w in units of meters.\n",
" - **Uwave**, **Vwave**, and **Wwave** --- u, v, and w in units of wavelengths lambda.\n",
" - **Azimuth** and **Ant-Azimuth** --- Azimuth in units of degrees. Azimuth plots a fiducial value for the entire array, while Ant-Azimuth plots the azimuth for each individual antenna (their azimuths will differ depending on each antenna\\'s longitude, latitude, and elevation).\n",
" - **Elevation** and **Ant-Elevation** --- Elevation in units of degrees. Elevation is a representative value for the entire array, while Ant-Elevation is the elevation for each individual antenna (their elevations will differ depending on each antenna\\'s longitude, latitude, and elevation).\n",
" - **Ant-Ra** and **Ant-Dec** --- Longitude and latitude of the direction to which the first antenna of a baseline points at data-taking timestamps.\n",
" - **HourAngle** --- Hour angle in units of hours. This is a fiducial value for the entire array.\n",
" - **ParAngle** and **Ant-ParAng** --- Parallactic angle in units of degrees. ParAngle is the fiducial parallactic angle for all antennae in the array, while Ant-ParAng plots the parallactic angle for each individual antenna (their parallactic angles will differ depending on each antenna\\'s longitude, latitude, and elevation).\n",
"\n",
"**Calibration:**\n",
" - **GainAmp**, **GainPhase**, **GainReal**, **GainImag** --- the amplitude, phase, real and imaginary part of the calibration tables for regular complex gain tables.\n",
" - **Delay** --- The delay of a delay or fringefit (Fringe Jones) calibration table.\n",
" - **Delay Rate** --- The delay rate of a fringefit (Fringe Jones) calibration table.\n",
" - **Disp Delay** --- The dispersive delay of a fringefit (Fringe Jones) calibration table.\n",
" - **SwPower** --- Switched Power values for a VLA switched power calibration table.\n",
" - **Tsys** --- Tsys for Tsys calibration tables.\n",
" - **Opac** --- Opacity values of a Opacity calibration table.\n",
" - **SNR** --- Signal-to-Noise Ratio of a calibration table.\n",
" - **TEC** --- Total Electron Content of an ionosphere correction calibration table.\n",
" - **Antenna Positions** --- Antenna position offsets for a KAntPos Jones calibration table.\n",
"\n",
"**Ephemeris:**\n",
" - **Radial Velocity** --- for an ephemeris source, in km/s.\n",
" - **Distance (rho)** --- for an ephemeris source, in km.\n",
"\n",
"If the data axis selected from the drop-down menu is already stored in the cache (therefore implying that plotting will proceed relatively quickly), an \"X\" will appear in the checkbox next to *Cached*. To reload the data from disk, the *Reload* checkmark should be set at the bottom of this display.\n",
"\n",
"The **plotms** task parameters used to select the axes are *xaxis* and *yaxis*. Valid options include \\'scan\\', \\'field\\', \\'time\\', \\'interval\\', \\'spw\\', \\'chan\\' (or \\'channel\\'), \\'freq\\' (or \\'frequency\\'), \\'vel\\' (or \\'velocity\\'), \\'corr\\' (or \\'correlation), \\'ant1\\' (or \\'antenna1\\'), \\'ant2\\' (or \\'antenna2\\'), \\'baseline\\', \\'row\\', \\'observation\\', \\'intent\\', \\'feed1\\', \\'feed2\\', \\'amp\\' (or \\'amplitude\\'), \\'phase\\', \\'real\\', \\'imag\\', \\'wt\\' (or \\'weight\\'), \\'wtsp\\' (or \\'weightspectrum\\'), \\'flag\\', \\'flagrow\\', \\'uvdist\\', \\'uvwave\\' (or \\'uvdistl\\'), \\'u\\', \\'v\\', \\'w\\', \\'uwave\\', \\'vwave\\', \\'wwave\\', \\'azimuth\\', \\'elevation\\', \\'hourang\\' (or \\'hourangle\\'), \\'parang\\' (or \\'parangle\\'), \\'ant\\' (or \\'antenna\\'), \\'ant-azimuth\\', \\'ant-elevation\\', \\'ant-ra\\', \\'ant-dec\\', \\'ant-parang\\' (or \\'ant-parangle\\'), \\'gainamp\\' (or \\'gamp\\'), \\'gainphase\\' (or \\'gphase\\'), \\'gainreal\\' (or \\'greal\\'), \\'gainimag\\' (or \\'gimag\\'), \\'delay\\' (or \\'del\\'), \\'delayrate\\' (or \\'rate\\'), \\'dispdelay\\' (or \\'disp\\'), \\'swpower\\' (or \\'swp\\' or \\'spgain\\'), \\'tsys\\', \\'opacity\\' (or \\'opac\\'), \\'snr\\', \\'tec\\', \\'radialvelocity\\', \\'distance\\' (or \\'rho\\').\n",
"\n",
"When left as the default empty strings (\\\"\\\"), the axes for a MeasurementSet will be Amp vs. Time. The default axes for a calibration table depend on the type.\n",
"\n",
"*Setting Axes Parameters*\n",
"\n",
"*Data Columns*:\n",
"- For relevant data axes like Amp and Phase, the user will be presented with the option to plot raw data or calibrated data. This can be selected via a *Data Column* drop-down menu, located directly under the drop-down menu for *X Axis* or *Y Axis* selection. To plot raw data, select \"data\"; to plot calibrated data, select \"corrected\". Note that this choice will only have an impact on a plot if a calibration table has been applied to the MeasurementSet or a calibration library is set and enabled.\n",
"- If a data model is present in the MeasurementSet (e.g., created by setjy, clean, or ft), it can be plotted by selecting \"model\" from the *Data Column* menu. For MeasurementSets with float data instead of complex data, common in singledish datasets, select the \\\"float\\\" datacolumn.\n",
"- Residuals can be plotted via \\\"corrected-model_vector\\\", \\\"corrected-model_scalar\\\", \\\"data-model_vector\\\", data-model_scalar\\\", \\\"corrected/model_vector\\\", \\\"corrected/model_scalar\\\", \\\"data/model_vector\\\", and \\\"data/model_scalar\\\". The vector and scalar options distinguish between versions where values like amp, phase, etc. are calculated before (scalar) or after (vector) the subtraction or division.\n",
"- The **plotms** task parameters used to select the data columns are *xdatacolumn* and *ydatacolumn*. Valid options include \\'data\\', corrected\\', \\'model\\', \\'float\\', \\'corrected-model\\' (vector implied), \\'corrected-model_vector\\', \\'corrected-model_scalar\\', \\'data-model\\' (vector implied), \\'data-model_vector\\', \\'data-model_scalar\\', \\'corrected/model\\' (vector implied), \\'corrected/model_vector\\', \\'corrected/model_scalar\\', \\'data/model\\' (vector implied), \\'data/model_vector\\', and \\'data/model_scalar\\'. The implied vector residual datacolumns were kept for backwards compatibility. Default data columns for x and y are both \\'data\\'.\n",
"\n",
"*Antenna Pointing Direction Parameters*\n",
"- Ant-Ra, Ant-Dec axes are the longitude and the latitude of the direction to which the first antenna of a baseline points at data-taking timestamps. Their value is computed by:\n",
"- Interpolating with a user-supplied method the direction of that antenna at that data-taking timestamp, from the known directions pointed by that antenna at pointing-direction-recording timestamps, recorded in MeasurementSet\\'s POINTING table\n",
"- Converting the result to a user-specified output reference frame\n",
"- **plotms** task parameters to set ant-ra and ant-dec axes parameters are:\n",
" 1. xinterp: interpolation method to use when xaxis=\\'ant-ra\\' or xaxis=\\'ant-dec\\'\n",
" 2. xframe: output reference frame to use when xaxis=\\'ant-ra\\' or xaxis=\\'ant-dec\\'\n",
" 3. yinterp: interpolation method to use when yaxis=\\'ant-ra\\' or yaxis=\\'ant-dec\\'\n",
" 4. yframe: output reference frame to use when yaxis=\\'ant-ra\\' or yaxis=\\'ant-dec\\'\n",
" 5. Valid values for xframe and yframe are: \\'icrs\\', \\'j2000\\', \\'b1950\\', \\'galactic\\', \\'azelgeo\\'. The default value is \\'icrs\\'.\n",
" 6. Valid values for xinterp and yinterp are: \\'nearest\\', \\'cubic spline\\', \\'spline\\'. The default value is \\'cubic spline\\'.\n",
"\n",
"- Note:\n",
" 1. \\'spline\\' is a synonym for \\'cubic spline\\'\n",
" 2. When the interpolation method is set to \\'nearest\\', reference frame conversion is performed at the nearest pointing-recording timestamp, not at the data-taking timestamp.\n",
"\n",
"\n",
"WARNING: plotting antennas pointing directions with the Ant-Ra / Ant-Dec axes has only been implemented for ALMA, ASTE, and NRO data.\n",
"
\n",
"\n",
"\n",
"\n",
"*Axis Locations*\n",
"\n",
"The location of the x-axis and y-axis can be set using the radio buttons in the GUI, where the x-axis can be located at the *Bottom* (default) or *Top*, and the y-axis can be located at the *Left* (default) or *Right.*\n",
"\n",
"The **plotms** task parameter to set the y-axis location is *yaxislocation*. Valid values for this parameter include \\'left\\' (default) and \\'right\\'. There is no parameter to set the x-axis location.\n",
"\n",
"\n",
"*Axes Ranges*\n",
"\n",
"The X and Y ranges of the plot can be set manually or automatically. By default, the circle next to *Automatic* will be checked, and the ranges will be auto-scaled in ascending order based on the values plotted.\n",
"\n",
"To define the range, click on the circle below *Automatic* and enter a minimum and maximum value in the blank boxes. Note that if identical values are placed in the blank boxes (xmin=xmax and/or ymin=ymax), then the values will be ignored and automatically set. When xmin > xmax or ymin > ymax, the tick values will be descending (reversed).\n",
"\n",
"The **plotms** task parameter used to set the axes ranges is *plotrange*, and its value is a list of numbers in the format \\[xmin, xmax, ymin, ymax\\] (default \\[ \\], automatic range).\n",
"\n",
"*Plotting Multiple Y-Axes*\n",
"\n",
"Different values of the same dataset can be shown at the same time. To add a second y-axis, press the *Add Y Axis Data* button at the bottom of the *Axes* tab. Then select the parameters for the newly created axis by selecting from the new \"Y Axis Data\" drop-down menu. If the two y-axes have the same units, they can be displayed both on the same axis. If they are different (or their ranges are dissimilar), e.g. Amplitude and Elevation (both versus Time; see Figure 4 below), one axis should be attached to the left and the other to the right hand side of the plot. Using more than a single y-axis data is also reflected in the *Display* tab where a drop-down menu appears in order to select multiple y-axis options; here you may colorize each axis differently. See the Plot Display Tab section below to learn more about symbol properties. To remove the additional y-axis, click *Delete Y Axis Data* at the bottom of the *Axes* tab.\n",
"\n",
"{.image-inline}\n",
"\n",
">Overplotting in plotms: Two different y-axes for the same dataset have been chosen for this plot, amplitude and elevation.\n",
" \n",
"\n",
"The **plotms** task parameters used to plot multiple y-axes are the same as for a single y-axis: *yaxis* and *yaxislocation*; multiple y-axes can be specified as a list of strings if you are specifying the **plotms** command in the terminal. The values for *yaxis* and *yaxislocation* should be set to lists of the same length:\n",
"\n",
"```\n",
"plotms(vis='ngc5921.ms', yaxis=['amp','elevation'], yaxislocation=['left','right'])\n",
"```\n",
"\n",
"*Atmospheric Curve Overlays*\n",
"\n",
"The ability to compute and overlay an atmospheric transmission curve or a sky temperature curve, available in plotbandpass, has been added to **plotms**. For this feature, the x-axis must be Channel or Frequency; if another axis is chosen, a warning is issued and the plot continues without the overlay.\n",
"\n",
"**plotms** uses the dataset\\'s subtables to compute the mean weather values: pressure, humidity, temperature, and precipitable water vapor (pwv). If these subtables are not found, reasonable defaults are used instead and reported in a log message. The [atmosphere tool](../api/casatools.rst) is then used by **plotms** to calculate dry and wet opacities to produce the requested overlay curve, corrected by the airmass based on elevation.\n",
"\n",
"{.image-inline}\n",
"\n",
">Amp vs. Frequency plot with a Tsky overlay. The Tsky y-axis is automatically added on the right, and the curve is plotted in magenta. The Plot \\> Axes tab shows the radio buttons to select the Overlay: None, Atm, or Tsky.\n",
" \n",
"\n",
"The **plotms** task parameters used to plot the overlays are *showatm* and *showtsky*. These take boolean values and their defaults are False. Only one overlay can be selected; if both are set to True, only the atmospheric curve (*showatm*) will be displayed.\n",
"\n",
"```\n",
"plotms(vis=myvis, yaxis='amp', xaxis='freq', showatm=True)\n",
"```\n",
"\n",
"The image sideband curve may also be shown in **plotms** when the atmospheric transmission or sky temperature curves are plotted. In order to do this, the MS (or associated MS for a calibration table) cannot have reindexed spectral window IDs as a result of a split, and must have an ASDM_RECEIVER table in order to read the LO frequencies. If these conditions are not met, a warning is issued and only the atm/tsky curves are calculated and plotted.\n",
"\n",
"{.image-inline}\n",
"\n",
">Gain Amp vs. Frequency plot for a bandpass calibration table with the Atm Transmission (magenta) and Image Sideband (black) overlays, colorized by spw and one antenna selected. The Plot \\> Axes tab shows the checkbox to select the image sideband curve, enabled only when the Overlay is Atm or Tsky.\n",
" \n",
"\n",
"The **plotms** task parameter used to plot the image sideband curve overlay is *showimage*. This takes a boolean value and its default is False. If *showatm=False* and *showtsky=False*, a warning is issued and the curve will not be calculated and plotted.\n",
"\n",
"```\n",
"plotms(vis=mycaltable, yaxis='amp', xaxis='freq', antenna='0',\n",
" coloraxis='spw', showatm=True, showimage=True)\n",
"```\n",
"\n",
"By default, the curve overlays are plotted in magenta (Atm, Tsky) and black (Image Sideband). To colorize the curve overlays by coloraxis, use the **plotms** parameter *colorizeoverlay* (default False). If no *coloraxis* is set, this parameter is ignored. Use the Plot Display tab to set the colorization options.\n",
"\n",
"\n",
"**Iteration and Page Header : The Plot Page Tab**\n",
"\n",
"{.image-inline}\n",
"\n",
"> The plotms Plot Page Tab, used to iterate by scan with a page header added. The scan number is appended to the plot title.\n",
" \n",
"\n",
"*Iteration*\n",
"\n",
"In many cases, it is desirable to iterate through the data that were selected in the Data tab. A typical example is to display a single baseline in an amplitude vs. time plot and then proceed to the next baselines step by step. This can be done via the *Plot \\> Page* tab. A drop-down menu allows you to select the iteration axis, with options *None, Scan, Field, Spw, Baseline, Antenna, Time,* and *Corr*. Press the *Plot* button after changing your selection. Each plot will be autoscaled according to its iteration value range unless a *Range* is specified in the *Axis* tab.\n",
"\n",
"The current iteration is indicated in the plot title of the displayed plot. To proceed to the next plot use the green arrow buttons below the main panel. Use the icons to proceed panel by panel (single arrow symbols) or to jump to the first or last panel directly (double arrow symbols).\n",
"\n",
"The number of plots per page can be selected under *Options \\> Grid*, the last of the top row of tabs, as described in the Options Tab section. There are two scaling options for the iterated axes in a grid, set in this tab: *Global* and *Shared*. Global will use a common axis range based on data loaded with the selection criteria specified in the Data tab. Shared displays one set of x-axes and y-axes for the page rather than per-plot. When left unchecked, *Global* and *Shared* results in plots with axes scaling to the data for each individual panel of the iteration. An example of global shared x-axes and y-axes is in the Options Tab section.\n",
"\n",
"The **plotms** task parameter used to select an iteration axis is *iteraxis*. The options include \\'scan\\', \\'field\\', \\'spw\\', \\'baseline\\', \\'antenna\\', \\'time\\', and \\'corr\\'.\n",
"\n",
"To use a global axis range for iterated plots, set parameters *xselfscale=True* and/or *yselfscale=True*. To use a shared external x-axis per column on a grid, set *xsharedaxis=True* (must also set *xselfscale=True* and *gridrows* greater than 1). To use a shared external y-axis per row on a grid, set *ysharedaxis=True* (must also set *yselfscale=True* and *gridcols* greater than 1).\n",
"\n",
"*Page Header*\n",
"\n",
"It is sometimes useful to display above the plots a page header showing some metadata information. To do so, select in the lower list the header items you want to display, and press the antenna-shaped \\\"arrow\\\" pointing up. This will move the items you selected to the upper list showing the header contents, without updating the page header. Multiple items can be selected at once by pressing the Shift or the Control key, Control+A selects all items. To remove items from the Contents list, select in that list the items to remove and press the antenna-shaped \\\"arrow\\\" pointing down. The arrows blink red when clicked while their corresponding selection is empty, green otherwise.\n",
"\n",
"Press the Plot button to update the page header. Items included in the Contents list are laid out on 2 columns in the page header, in \\\"Z\\\" order. The contents of the header is common to all pages.\n",
"\n",
"Header items from multiple plots can be displayed in the page header. In that case items from the first plot are laid out first, items from the second plot are then laid out starting from the first empty row, and so on.\n",
"\n",
"The **plotms** task parameter used to specify header items is *headeritems*. The value is a single string whose value can be any comma-separated combination of the following pre-defined keywords:\n",
"\n",
"- \\'filename\\', \\'projid\\', \\'telescope\\', \\'observer\\',\\'obsdate\\', \\'obstime\\', \\'targname\\', \\'targdir\\', \\'ycolumn\\'\n",
"\n",
"When selected data leaves room for multiple candidates (e.g when selected data spans multiple observations or include multiple fields or sources), the first selected row in MeasurementSet\\'s Main table is used as a starting point for looking up a single \\\"first\\\" candidate in MeasurementSet\\'s auxiliary tables.\n",
"\n",
"Observation Start Date and Observation Start Time are looked up in MeasurementSet\\'s Observation table, and therefore differ from the output of listobs task.\n",
"\n",
"\n",
"**Transforming the Velocity Frame or Phase Center: The Plot Transform Tab**\n",
"\n",
"*Frequency Frame*\n",
"\n",
"If the user plans to plot frequency, the reference frame must be defined. By default, **plotms** selects the frame keyword (if any) present in the data, usually the frame observed at the telescope unless modified during previous processing. However, transformations can be made by choosing a *Frame* from the drop-down menu in the *Plot \\> Transform* tab. Frequency reference frames can be chosen to be:\n",
"\n",
"- LSRK --- local standard of rest (kinematic)\n",
"- LSRD --- local standard of rest (dynamic)\n",
"- BARY --- barycentric\n",
"- GEO --- geocentric\n",
"- TOPO --- topocentric\n",
"- GALACTO --- galactocentric\n",
"- LGROUP --- local group\n",
"- CMB --- cosmic microwave background dipole\n",
"\n",
"The **plotms** task parameter used to select frequency frame is *freqframe.* Valid options include those listed above (strings with all caps). The default empty string \\\"\\\" results in no frame transformation.\n",
"\n",
"*Velocity*\n",
"\n",
"If Velocity is selected as an axis, by default the transformation from frequency uses the parameters in the MS metadata, or, if absent, using the central frequency and TOPO frame. The user can change this by using the *Frame, Velocity Defn,* and *Rest Freq* options in the Transform tab. The velocity definition is chosen from the *Velocity Defn* drop-down menu, offering selections of *Radio, True (Relativistic),* or *Optical.*\n",
"\n",
"For more information on frequency frames and spectral coordinate systems, see the paper by Greisen et al. (A&A, 446, 747, 2006) (Also at [http://www.aoc.nrao.edu/\\~egreisen/scs.ps](http://www.aoc.nrao.edu/%7Eegreisen/scs.ps))\n",
"\n",
"Finally, the spectral line's rest frequency in units of MHz should be typed into the *Rest Freq* input box next. You can use the [slsearch](../api/casatasks.rst) task to search a spectral line table, or the Measures tool me.spectralline method to turn transition names into frequencies:\n",
"\n",
"```\n",
"CASA <16>: me.spectralline('HI')\n",
"[ Out[17]: ]\n",
"{'m0': {'unit': 'Hz', 'value': 1420405751.786},\n",
" 'refer': 'REST',\n",
" 'type': 'frequency'}\n",
"```\n",
"\n",
"For a list of known lines in the CASA measures system, use the toolkit command me.linelist(). For example:\n",
"\n",
"```\n",
"CASA <21>: me.linelist()\n",
"[ Out[21]: 'HI H186A H185A H184A H183A H182A H181A H180A H179A H178A H177A H176A H175A ]\n",
"H174A H173A H172A H171A H170A H169A H168A H167A H166A H165A H164A H163A H162A H161A H160A...\n",
"He182A He181A He180A He179A He178A He177A He176A He175A He174A He173A He172A He171A He170A\n",
"He169A He168A He167A He166A He165A He164A He163A He162A He161A He160A He159A He158A He157A...\n",
"C186A C185A C184A C183A C182A C181A C180A C179A C178A C177A C176A C175A C174A C173A C172A\n",
"C171A C170A C169A C168A C167A C166A C165A C164A C163A C162A C161A C160A C159A C158A C157A...\n",
"NH3_11 NH3_22 NH3_33 NH3_44 NH3_55 NH3_66 NH3_77 NH3_88 NH3_99 NH3_1010 NH3_1111 NH3_1212\n",
"OH1612 OH1665 OH1667 OH1720 OH4660 OH4750 OH4765 OH5523 OH6016 OH6030 OH6035 OH6049 OH13433\n",
"OH13434 OH13441 OH13442 OH23817 OH23826 CH3OH6.7 CH3OH44 H2O22 H2CO4.8 CO_1_0 CO_2_1 CO_3_2\n",
"CO_4_3 CO_5_4 CO_6_5 CO_7_6 CO_8_7 13CO_1_0 13CO_2_1 13CO_3_2 13CO_4_3 13CO_5_4 13CO_6_5\n",
"13CO_7_6 13CO_8_7 13CO_9_8 C18O_1_0 C18O_2_1 C18O_3_2 C18O_4_3 C18O_5_4 C18O_6_5 C18O_7_6\n",
"C18O_8_7 C18O_9_8 CS_1_0 CS_2_1 CS_3_2 CS_4_3 CS_5_4 CS_6_5 CS_7_6 CS_8_7 CS_9_8 CS_10_9\n",
"CS_11_10 CS_12_11 CS_13_12 CS_14_13 CS_15_14 CS_16_15 CS_17_16 CS_18_17 CS_19_18 CS_12_19\n",
"SiO_1_0 SiO_2_1 SiO_3_2 SiO_4_3 SiO_5_4 SiO_6_5 SiO_7_6 SiO_8_7 SiO_9_8 SiO_10_9 SiO_11_10\n",
"SiO_12_11 SiO_13_12 SiO_14_13 SiO_15_14 SiO_16_15 SiO_17_16 SiO_18_17 SiO_19_18 SiO_20_19\n",
"SiO_21_20 SiO_22_21 SiO_23_22'\n",
"```\n",
"\n",
"The **plotms** task parameters used to set velocity definition and rest frequency are *veldef* and *restfreq.* Valid options for *veldef* are \\'RADIO\\', \\'TRUE\\', or \\'OPTICAL\\' (default is \\'RADIO\\'). *restfreq* should be in a string in MHz, for example \\'22235.08MHz\\'.\n",
"\n",
"*Setting the Phase Center*\n",
"\n",
"The plot's phase center can be changed in the *Plot \\> Transform* tab. This will allow coherent vector averaging of visibility amplitudes far from the phase tracking center. Enter the direction coordinates of the new phase center, specified as absolute world coordinates including frame, e.g. 'J2000 19h53m50 40d06m00'. Time-dependent systems such as AZEL are not supported, nor are ephemeris objects. This will change the plot but not the MeasurementSet data. To change the phase center in the MS, use task **phaseshift**.\n",
"\n",
"The **plotms** task parameter used to set the phase center is *phasecenter*. Its value should be a string as described above.\n",
"\n",
"**Display Options for Plots: The Plot Display Tab**\n",
"\n",
"*Colorizing Your Data*\n",
"\n",
"Data points can be given informative symbol colors using the *Colorize* option in the *Plot \\> Display* tab. By checking the box next to *Colorize* and selecting a data axis from the drop-down menu, the data will be plotted with colors that vary along that axis. For example, if \"corr\" is chosen from the *Colorize* menu, \"RR\", \"LL\", \"RL\", and \"LR\" data will each be plotted with a different color. Note that *Colorize* while plotting flagged data will override the default flagged red symbol color. The legend can be shown to list the symbol colors and their values (see the Plot Canvas tab below).\n",
"\n",
"If a curve overlay is added to the plot axes (Atm, Tsky, and Image Sideband; see the PlotAxes tab above) and *Colorize* is set, an additional check box *Colorize overlay* is enabled. When checked, the colorize axis will be applied to the overlay points in a lighter tint of the color used in the plot.\n",
"\n",
"The **plotms** task parameter used to colorize data is *coloraxis*. Options include \\'scan\\', \\'field\\', \\'spw\\', \\'antenna1\\', \\'antenna2\\', \\'baseline\\', \\'channel\\', \\'corr\\', \\'time\\', \\'observation\\', and \\'intent\\'.\n",
"\n",
"The **plotms** task parameter used to colorize curve overlays is *colorizeoverlay* (True/False, default False).\n",
"\n",
"*Customizing Your Symbols*\n",
"\n",
"Unflagged and flagged plot symbols can be customized in the *Plot \\> Display* tab. Most fundamentally, the user can choose to plot unflagged data and/or flagged data. By default, unflagged data is plotted (the circle next to *Default* is selected under *Unflagged Points Symbol*), and flagged data is not plotted (the circle next to *None* is selected under *Flagged Points Symbol*). We note here that plotting flagged data on an averaged plot is undertaken at the user's own risk, as the distinction between flagged points and unflagged points becomes blurred if data are averaged over a dimension that is partially flagged. Take, for example, a plot of Amplitude vs. Time where all channels are averaged together, but some channels have been flagged due to RFI spikes. In creating the average, **plotms** will skip over the flagged channels and only use the unflagged ones. The averaged points will be considered unflagged, and the flagged data will not appear on the plot at all.\n",
"\n",
"Symbol options include:\n",
"\n",
"- None --- no data points\n",
"- Default --- data points which are small circles (blue for unflagged data and red for flagged data)\n",
"- Custom --- allows the user to define a plot symbol\n",
"\n",
"If *Custom* plot symbols are chosen, the user can determine:\n",
"\n",
"1. **Size**, by typing a number in the blank box next to *px* or by clicking on the adjacent up or down arrows.\n",
"2. **Shape**, chosen from the drop-down menu; options include *circle, square, diamond*, *pixel*, or *autoscaling.* Note that pixel only has one possible size. *autoscaling* attempts to adjust the size of the points from dots to circles of different sizes, depending on how many points are plotted.* *\n",
"3. **Color**, chosen by typing a hex color code in the *Fill* input box or by clicking on the \\... button and selecting a color from the pop-up GUI.\n",
"4. **Fill**, using the adjacent drop-down menu for how heavily the plot symbol is shaded with this color, from heaviest to lightest; options include *fill, mesh1, mesh2, mesh3*, and *no fill*.\n",
"5. **Outline**, by selecting *None* (no outline) or *Default* (outlined in black)\n",
"\n",
"Note that if \"no fill\" and *Outline: None* are selected, the plot symbols will be invisible.\n",
"\n",
"The **plotms** task parameter and subparameters used to customize unflagged symbols include:\n",
"\n",
"- *customsymbol* (True/False, default False) - must be True for subparameters to take effect \n",
"- *symbolshape* (\\'autoscaling\\', \\'circle\\', \\'square\\', \\'diamond\\', \\'pixel\\', \\'nosymbol\\', default \\'autoscaling\\')\n",
"- *symbolsize* (in number of pixels, default 2)\n",
"- *symbolcolor* (RGB hex code e.g. \\'aa55ff\\' or string color name e.g. \\'purple\\', default \\'0000ff\\' blue)\n",
"- *symbolfill* (\\'fill\\', \\'mesh1\\', \\'mesh2\\', \\'mesh3\\', \\'no fill\\', default \\'fill\\') \n",
"- *symboloutline* (True/False, default False)\n",
"\n",
"The **plotms** task parameters used to customize flagged symbols include *customflaggedsymbol* (default False) with subparameters *flaggedsymbolshape* (default \\'nosymbol\\'), flaggedsymbolsize (default 2), *flaggedsymbolcolor* (default \\'ff0000\\' red), *flaggedsymbolfill* (default \\'fill\\'), and *flaggedsymboloutline* (default False). Supported values are the same as for unflagged symbols.\n",
"\n",
"*Symbols for Multiple Y-Axes*\n",
"\n",
"If you have added an additional y-axis in the *Plot \\> Axes* tab, you may customize each y-axis individually by selecting the axis in the *Y Axis Data* pull-down menu at the top of the *Plot \\> Display* tab and then customizing the symbols for that axis.\n",
"\n",
"To set multiple symbols in the **plotms** task, set the symbol parameters as a list:\n",
"\n",
"```\n",
"plotms(vis='ngc5921.ms', yaxis=['amp','elevation'], yaxislocation=['left','right'],\n",
" customsymbol=[True,True], symbolcolor=['purple','green'])\n",
"```\n",
"\n",
"In this plot, the \\'amp\\' axis will be purple, and the \\'elevation\\' axis will be green.\n",
"\n",
"*Connecting the Points*\n",
"\n",
"Plotms has the capability to connect points for calibration tables; support for MeasurementSets will be added later. The points are colorized and connected along the x-axis or time axis by line or step. Points with the same metadata but varying values of the x-axis or time are connected. Unflagged points are not connected to flagged points, even when they are not displayed. The *Colorize* axis will override the connection colorization.\n",
"\n",
"{.image-inline}\n",
"\n",
">Plot Display tab showing the Connect Points options for a gain table. Here, points with the same spw, channel, polarization, and antenna1 are connected along the time axis.\n",
" \n",
"\n",
"The plotms task parameters used to connect points in a calibration table plot are *xconnector* (default \\\"none\\\", options \\\"line\\\" or \\\"step\\\") and *timeconnector* (default False, or True to connect along the time axis instead of x-axis).\n",
"\n",
"For an antenna position (KAntPos Jones) calibration table, the x, y, z antenna positions are located in the first axis of the data, normally the correlation axis. To distinguish these offsets, set *coloraxis='corr'* or *xconnector='line'* as shown in the figure below. To determine which points are x, y, and z, use the Locate tool.\n",
"\n",
"{.image-inline}\n",
"\n",
"> Plot Display tab showing the Connect Points options for an antenna position table.\n",
"\n",
"**Plot Labels: The Plot Canvas Tab**\n",
"\n",
"*Plot Title*\n",
"\n",
"Options to change the plot title include *None* (no title), *Default,* and a user-input string. To set the plot title, under *Title*, click on the circle next to the input box and enter the desired text. This text box shows the grayed-out default string, \\\"%%yaxis%% vs. %%xaxis%%\\\" (to substitute the axis names for \\\"yaxis\\\" and \\\"xaxis\\\"). The user can also choose the size of the title font by checking the *Title Font* checkbox and entering the font size or using the arrows to increase or decrease the value. The default is to scale the title font depending on the plot size.\n",
"\n",
"The **plotms** task parameters used to set the title and its font are *title* (default empty string \\\"\\\" for yaxis vs. xaxis) and *titlefont* (default 0 to autoscale). Set a space \\\" \\\" for no title.\n",
"\n",
"*Legend*\n",
"\n",
"A plot symbol legend can be added to the plot by clicking on the checkbox next to *Legend*. For a simple plot, a symbol legend simply echoes the plot axes (e.g. \\\"Amp vs Time\\\") but is useful when overplotting data with custom colors so that you can identify the data (e.g. \\\"Amp vs Time\\\" in blue and \\\"Phase vs Time\\\" in green on the same plot). When *Colorize* is set in the Plot Display tab, the legend shows the colorized colors and values used in the plot rather than the axes being plotted.\n",
"\n",
"When enabled, a drop-down menu next to *Legend* allows the user to select the legend location either within the plot (Upper Right, Lower Right, Upper Left, Lower Left) or outside the plot (Out Right, Out Left, Out Top, Out Bottom).\n",
"\n",
"The **plotms** task parameter used to enable the legend is *showlegend* (default is False). To select the legend location, use *showlegend=True* and set *legendposition* to \\'upperRight\\', \\'upperLeft\\', \\'lowerRight\\', \\'lowerLeft\\', \\'exteriorRight\\', \\'exteriorLeft\\', \\'exteriorTop\\', or \\'exteriorBottom\\' (default empty string \\\"\\\" == upperRight).\n",
"\n",
"*Axis Labels*\n",
"\n",
"To enable the X- and Y-axis labels, check the *Show Label* checkboxes under *X Axis* and *Y Axis* (default is checked). As with the plot title, the user may set the label to None (no label), Default (axis name with units), or type the desired text in the blank box. The font size of labels can also be customized by enabling then setting the font size for each axis.\n",
"\n",
"The **plotms** task parameters used to set the label text and font are *xlabel and ylabel* (default empty string \\\"\\\" is axis name with units); set to \\' \\' space to disable label. Set font size with *xaxisfont* and *yaxisfont* (default 0 == autoscale).\n",
"\n",
"*Grid Lines*\n",
"\n",
"A grid of lines can be superimposed on the plot using *Grid Lines* in the *Plot \\> Canvas* tab. \"Major\" grid lines are drawn at the locations of major tick marks, while \"minor\" grid lines are drawn at minor tick marks.\n",
"\n",
"Grid line colors, thicknesses, and styles are selected independently for the \"major\" and \"minor\" grid lines. Desired line thickness should be typed into the blank boxes just to the right of the *Major* and *Minor* labels. Colors are set by clicking on the \\... buttons. The blank boxes to the left of the \\... buttons will then contain the hex codes for the selected colors (e.g., \"808080\"). Line styles can also be selected from the drop-down menus to the right of \\... buttons; style options include *solid, dash, dot*, and *none*.\n",
"\n",
"The **plotms** task parameter used to add and customize major grid lines include *showmajorgrid* (default is False) with subparameters *majorwidth* (default is 1), *majorstyle* (\\'solid\\', \\'dash\\', \\'dot\\', \\'none\\'; default is \\'solid\\'), and *majorcolor* (RGB hex code or color name; default is \\'b0b0b0\\' dark gray).\n",
"\n",
"Parameters for minor grid lines include *showminorgrid* (default is False) with subparameters *minorwidth* (default is 0), *minorstyle* (default is \\'solid\\'), and *minorcolor* (default is \\'d0d0d0\\' light gray).\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4xixiWDa0EDC"
},
"source": [
"### Flag Extensions: The Flag Tab\n",
"\n",
"{.image-inline}\n",
"\n",
">The plotms *Flag* tab. Here the *Extend flags* box has been checked, enabling the *Correlation* (selected) and *Channel* options. The plot shows unflagged data in blue and flagged data in red.\n",
" \n",
"\n",
"See the section below on interactive flagging in plotms. The options in this tab allows the user to have flagging extend to other data points besides what is marked on the plot.\n",
"\n",
"When enabled with the *Extend flags* checkbox, the user may choose to extend flags based on correlation or channel by checking the corresponding checkboxes. Future options for flag extensions are planned.\n",
"\n",
"By checking the boxes next to *Extend Flags* and *Correlation*, flags will be extended beyond the correlations displayed. Currently the only option is to extend to *All* correlations as noted by the radio button, implying that all correlations will be flagged. For example, with RR displayed, the correlations RR, RL, LR, and LL will all be flagged when this option is enabled.\n",
"\n",
"By checking the boxes next to *Extend Flags* and *Channel*, flagging will be extended to other channels in the same spw as the displayed point. For example, if spw='0:0' and channel 0 is displayed, then flagging will extend to all channels in spw 0.\n",
"\n",
"The **plotms** task parameter used to extend flags is *extendflag* (True/False, default is False) with subparameters *extcorr* (True/False), and *extchannel* (True/False). These parameters will enable flag extensions when interactively flagging the plot.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vBp6yhLb0EDD"
},
"source": [
"### Interactive Tools: The Tools Tab, Annotate Tab, and Tool Icons\n",
"\n",
"{.image-inline}\n",
"\n",
">The plotms Tools tab. Here the Tracker Display tool is showing the (X,Y) coordinates of the cursor position. A previous position was saved to the text box by pressing the SPACE bar.\n",
" \n",
"\n",
"Various interactive GUI tools are selectable with the radio buttons in the *Hand Tools* section of the *Tools* tab at the top of the **plotms** window. They are also available as icon buttons at the bottom of the **plotms** window. These tools can be used to zoom, pan, annotate, flag/unflag, and locate data. Described below are the bottom icon buttons in order.\n",
"\n",
"{.image-inline}\n",
"\n",
"- **Zoom** --- The \"magnifying glass\" button (1st on left) lets you draw a box around a region of the plot (left-click on one corner of the box, and drag the mouse to the opposite corner of the desired box), and then zooms in on this box.\n",
" \n",
"- **Pan** --- The \"four-arrow\" button (2nd from left) lets you pan around a zoomed plot.\n",
" \n",
"- **Annotate** --- The 3rd button from the left is chosen from a drop-down menu to either *Annotate Text* (\"T with a green diamond\" button) or *Annotate Rectangle* (\"pencil\" button). With *Annotate Text* activated, click on a location in the plot where text is desired; a window will pop up, allowing you to type text in it. When you click the OK button, this text will appear on the plot. *Annotate Rectangle* simply lets you draw a box on the plot by left-clicking and dragging the mouse. By clicking on the *Annotate* tab near the top of the **plotms** window, different fonts, colors, line styles, etc. can be selected for annotations.\n",
" \n",
"- **Stack Base** --- The \"house\" button (5th from left) returns to the original zoom level.\n",
" \n",
"- **Stack Back** and **Stack Forward** --- The left and right arrow buttons (4th and 6th from left) step through the zoom settings you've visited.\n",
" \n",
"- **Mark Regions** --- The \"box with a green diamond\" button (7th from left) lets you mark a region for flagging, unflagging, or locating. Left-click on one corner of the desired region, and then drag the mouse to set the opposite corner of the region. You can mark multiple boxes before performing an operation on them. The selected regions will appear on the plot as shaded rectangles. \n",
" \n",
"- **Subtract Regions** --- The \"box with a minus sign\" button (8th from left) lets you de-select marked regions (draw around a marked region and the shaded area will disappear). To de-select all marked regions, use the next button. \n",
"\n",
"- **Clear Regions** --- Clicking on the \"box with a red circle\" button (9th from left) will clear all regions which have been marked using Mark Regions.\n",
"\n",
"- **Locate** --- The \"magnifying glass on a sheet of paper\" button (10th from left) will print out information about points in the marked regions. This information is printed to the shell terminal when **plotms** was started with *casaplotms*, or to the casa logger/logfile when **plotms** was started in a casa python session. The header of the output indicates the plotted X and Y axes and the range of values in the selected region. The output for each point includes scan, field, time, baseline, spw, channel, frequency, correlation, X, Y, and observation ID. By copying this list to a text file, or setting a new logfile with *casalog.setlogfile* as described in the [CASA logger documentation](usingcasa.ipynb#logging), the *Locate* information can be edited to provide input for flagdata. To list an entire column, e.g. all visibilities for a source, use the listvis task or the [table tools](../api/casatools.rst).\n",
"\n",
"- **Flag** --- Click on the \"flag\" button (11th from left) to flag all points in the marked regions. See the section below on Interactive Flagging.\n",
" \n",
"- **Unflag** --- Click on the \"crossed-out flag\" button (12th from left) to unflag any flagged points in the marked regions (even if not displayed).\n",
" \n",
"- **Flag All** --- Click on the \\\"per-grid flag/unflag\\\" button (13th from left) to enter/leave the \\\"Flag All\\\" mode. See the section below on Interactive Flagging.\n",
"\n",
"- **Iteration** --- The next four green arrow buttons (14th through 17th from left) control iteration, with the first and last \\\"double arrow\\\" buttons used to display the first and last iteration, and the center two \\\"single arrow\\\" button to display the previous or next iteration. If the plots are on a grid, these arrows navigate through the pages of plots which contain multiple iterations.\n",
"\n",
"- **Hold Drawing** --- If the *Hold Drawing* button (rightmost, or 18th from left) is clicked to activate it, when new plot axes are selected from the *Plot \\> Axes* tab, the new data will be cached but not plotted. When the button is clicked again (de-activated), it will automatically plot the data that was last requested. This can be particularly useful when changing the size of the **plotms** window.\n",
" \n",
"The *Tools* tab also contains *Tracker* tools including *Hover* and *Display.* When *Hover* is selected and the mouse is moved over the plot, the pointer\\'s position is displayed on the plot in (X, Y) format. When *Display* is selected, the (X, Y) position is displayed in the text box under the *Display* checkbox.\n",
"\n",
"To record various tracked positions, enable *Display* then click on the plot to activate it. As usual, moving the pointer displays the position in the small display text box. Pressing the *SPACE* bar will copy the displayed line into the larger white box below it. This can be repeated many times and a log of positions and values will be created. The content in the box can then be easily copied and pasted into any other application that is used for data analysis. The *Clear* button wipes out the content of the box for a fresh start into new scientific adventures.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rXZcumQi0EDD"
},
"source": [
"### Miscellaneous Options: The Options Tab\n",
"\n",
"A few miscellaneous per-page plot options and GUI options are available in the *Options* tab, the last tab at the top of the **plotms** window.\n",
"\n",
"*Plotting on a Grid*\n",
"\n",
"The layout of the page is set on the **plotms** *Options* tab. For multiple plots per page, set the grid layout, the number of rows and columns that determine the number of sub-plots. When set, click \\\"Update\\\" to activate the grid changes.\n",
"\n",
"*Plotting Iterations on a Grid*\n",
"\n",
"{.image-inline}\n",
"\n",
">The plotms Options tab. Here a 2x2 grid has been created with iteration on the \\'antenna\\' axis.\n",
"\n",
"If iteration is enabled in the *Plot \\> Page* tab, the grid will be filled automatically with each iterated plot. The *Plot \\> Page* tab is also where common axis scales and shared axes will be set; they are enabled for the plot in Figure 9. These axis options are only available for iterated plots in a grid.\n",
"\n",
"The **plotms** task parameters used to create a grid with iteration include *gridrows and gridcols* (default is 1)*.* To create the plot shown in Figure 9, the **plotms** command would be:\n",
"\n",
"```\n",
"plotms('ngc5921_ut.ms', xaxis='freq', iteraxis='antenna', gridrows=2, gridcols=2,\n",
" xsharedaxis=True, xselfscale=True, ysharedaxis=True, yselfscale=True)\n",
"```\n",
"\n",
"*Plotting Multiple Data on a Grid*\n",
"\n",
"We note here that plotting multiple datasets or axes on a grid is possible in **plotms** but covered separately in the Plotting Multiple Data section below, as this involves many settings in the GUI or multiple **plotms** task commands. Since the grid affects all of the plots, its settings are in the *Options* tab rather than the *Plot* tab.\n",
"\n",
"**Tool Button Style**\n",
"\n",
"The *Tool Button Style* drop-down menu determines the format of the tool buttons at the bottom of the **plotms** window. The options include *Icon Only, Text Only, Text Beside Icon, and Text Under* Icon. In *Icon Only* mode (default), hovering the cursor over each icon will give a text description of the icon.\n",
"\n",
"To hide the bottom icons, see the description of the *View menu*. The tools can also be accessed in the *Tools* tab.\n",
"\n",
"**Log Events**\n",
"\n",
"This drop-down menu shows a checklist of events and **plotms** functions so that you can customize how verbose **plotms** is in documenting its actions.\n",
"\n",
"**Clear Regions and Annotations**\n",
"\n",
"The *When changing plot axes, clear any existing regions or annotations* checkbox determines when regions and annotation are deleted from the plot. By default, this is enabled.\n",
"\n",
"**Cached Images**\n",
"\n",
"A useful option is the *Use* *fixed size for cached image* checkbox. It determines how large the dots in the panel are with respect to the screen resolution. The values influence how the data is redrawn on the panel. When *Screen resolution* is selected, the **plotms** window can be resized without redrawing on the canvas -- a considerable speedup for large data sets. The penalty is that the dots of the data points are the size of a pixel on the screen, which may be very small for high resolution monitors. By default, this feature is not enabled.\n",
"\n",
"**File Chooser History Limit**\n",
"\n",
"This setting allows the user to limit how many remembered filepaths are displayed in file chooser dialogs produced by clicking *Browse* in the *Plot \\> Data* tab to select a MeasurementSet or calibration table and in the *Plot \\> Calibration* tab to select a calibration library.\n",
"\n",
"------------------------------------------------------------------------\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OxJFVrtf0EDD"
},
"source": [
"### The Plotms Menus \n",
"\n",
"**File Menu: Quit**\n",
"\n",
"The *File* menu in the top menu bar allows you to *Quit* **plotms**, or you can click the *X in the upper right corner of the window.*\n",
"\n",
"**Export Menu: Saving Your Plot**\n",
"\n",
"You can save a copy of a plot to file with the *Export* menu, which produces an *Export Plots* dialog box with many settings.\n",
"\n",
"- **Filename**: Click the *Browse* button for a GUI-based selection of the directory and filename to which the plot will be saved, or click the *Insert MS Name* button to minimize typing. You may also just type in a file name. The file format can be determined in this GUI by the suffix given to the filename: .png, .jpg, .ps, .pdf, and txt.\n",
" \n",
"\n",
"If a file already exists with the given filename, it will be overwritten without warning!\n",
"
\n",
"\n",
"- **Format**: Alternatively, the file format can be selected from the *Format* drop-down menu, with these options: \\[by file extension\\], PNG, JPG, PS, PDF, and TEXT. For the first option, if your filename is \\\"test.jpg\\\" the plot will be exported in JPG format. For the other formats, **plotms** will use the filename as given and **not** add a suffix to indicate its format. See below for an example of TEXT format; the header will include the name of the visibility and other specified parameters including selection, averaging, transformations, etc.\n",
"\n",
"- **Verbose**: When a text export is selected, the output can be verbose (include metadata). When this checkbox is unchecked, the text export will include x and y values only. This parameter is ignored for other formats.\n",
"\n",
"- **Range**: When iteration is chosen, producing multiple plots, you may select to export only the *Current Page* or *All Pages*. Each saved plot will have the name of the iteration appended to the given filename before the extension. For example, with filename \\\"ngc5921_ut.jpg\\\" and iteration on antenna, the first plot will be named ngc5921_ut_Antenna1\\@VLA:N7.jpg. This is so the exported plots can be identified without viewing them. Be warned that if you are plotting iterations on a grid, the filenames will have all of the iterations on the page appended, which can lead to a very long filename. Filenames exceeding 255 characters in length will be automatically shortened upon export. One plotfile per page is produced; multipage pdf exports are not currently supported. \n",
"\n",
"- **High Resolution:** Exporting to images in screen resolution is currently not working, so plot exports are always high resolution. A notice is issued in the console/log.\n",
"\n",
"- **DPI, Size**: Use the text boxes or up/down arrows to set the output DPI or size (in pixels) of the exported plot.\n",
"\n",
"- **Export**: When settings are complete, click *Export* to create your plotfile.\n",
"\n",
"Note: The plot files produced by the PS and PDF options can be large and time-consuming to export. The JPG is the smallest.\n",
"\n",
"The TEXT format will not save an image but all of the data points themselves. This allows one to dump the current plot into a file that is used in other programs for further processing.\n",
"\n",
"\n",
"**ALERT**: The exported TEXT file can be quite large and take some time to create. Using averaging, selection, etc. is recommended to keep the file size manageable. If a region is marked as described in the Interactive Tools section, only those points are exported to the text file.\n",
"
\n",
"\n",
"The reported data is the same as when using the *Locate* button in **plotms**, with the following format (when *verbose=True*):\n",
"\n",
"```python\n",
"#vis: ngc5921_ut.ms\n",
"#scan: 4\n",
"#channel average: 63\n",
"#time average: None\n",
"#From plot 0\n",
"#x y chan scan field ant1 ant2 ant1name ant2name time freq spw corr obs\n",
"#Time Amp None None None None None None None MJD(seconds) GHz None None None\n",
"4304483429.999 62.7219 0 4 1 1 1 2@VLA:W1 2@VLA:W1 4304483429.999 1.413332233 0 RR 0\n",
"4304483429.999 59.0717 0 4 1 1 1 2@VLA:W1 2@VLA:W1 4304483429.999 1.413332233 0 LL 0\n",
"4304483429.999 59.0252 0 4 1 27 27 28@VLA:W7 28@VLA:W7 4304483429.999 1.413332233 0 RR 0\n",
"4304483429.999 60.8603 0 4 1 27 27 28@VLA:W7 28@VLA:W7 4304483429.999 1.413332233 0 LL 0\n",
"4304483429.999 57.4048 0 4 1 7 7 8@VLA:W8 8@VLA:W8 4304483429.999 1.413332233 0 RR 0\n",
"etc.\n",
"```\n",
"\n",
"where x and y are the two plotted axes and the other columns contain additional information such as the baselines and frequencies, with an additional line for units. The header for the file includes the name of the visibility and other parameters such as selection, averaging, etc.\n",
"\n",
"The **plotms** task parameter used to export plots is *plotfile.* Unlike the Export Plots dialog box in the GUI, **plotms** will issue an error if this file exists unless *overwrite=True*. Its subparameters include:\n",
"\n",
"- *expformat* (\\'jpg\\', \\'png\\', \\'pdf\\', \\'ps\\', \\'txt\\') - select the format if no extension is included in the *plotfile*. If there is no *plotfile* extension and no *expformat* set, the plot will be exported as a PNG. \n",
"- verbose (True/False, default is True) - include metadata in text export; ignored for other formats.\n",
"- *exprange* (\\'current\\', \\'all\\'; defaults to \\'current\\') for iterated plots \n",
"- *highres* (True/False, default is False) \n",
"- *dpi* - dots per inch\n",
"- *width* (in pixels)\n",
"- *height* (in pixels)\n",
"- *overwrite* (True/False, default is False)\n",
"\n",
"**Summary Menu: Information About Your Dataset**\n",
"\n",
"Information about the MeasurementSet can be obtained from within **plotms** by clicking on the *Summary* menu in the top menu bar. If *All* is chosen from the pull-down menu next to *Type*, listobs-style output about scans, correlator configurations, and antennae will be written to the console or log. Other options for a subset of the data include *Where, What, How, Main, Tables, Antenna, Feed, Field, Observation, History, Polarization, Source, Spectral Window, Spectral Window and Polarization, SysCal*, and *Weather*.\n",
"\n",
"For calibration tables, options in the *Summary* menu include *All, Where, What, How, Main, Tables, Antenna, Field, Observation, History,* and *Spectral Window*.\n",
"\n",
"For more detail, click on the *Verbose* checkbox.\n",
"\n",
"**View Menu: Hide or Display Tool Icons**\n",
"\n",
"This menu controls the display of tool icons. Use the *View \\> Toolbars* menu to de-select and hide the *Tools, Iteration* (green arrows), or *Display* (Hold Drawing) icons. By default these icons are all selected and displayed at the bottom of the **plotms** window.\n",
"\n",
"**Help Menu: About plotms**\n",
"\n",
"This menu\\'s *About* option describes **plotms** and the versions of CASA, Qt, and Qwt it uses, along with links. Qt is the software framework that **plotms** uses for its GUI, and Qwt is a library that provides plotting functionality on top of the Qt framework.\n",
"\n",
"Click *About Qt* for more detail about this software package and its licensing.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "udjqjfDp0EDE"
},
"source": [
"### Plotting Multiple Data\n",
"\n",
"**Overplotting Multiple Datasets or Axes on the Same Plot**\n",
"\n",
"It is possible to overplot two datasets on the same plot, or the same dataset with different y-axes in each plot. To do this, set up the first plot as usual. Then press the *Add Plot* button at the bottom left of the **plotms** window. This will bring up an additional data input panel in the *Plot \\> Data* tab where you can specify the plot parameters as you did for the first one, which is automatically minimized. Use the slider to scroll vertically through the panels. Use right-click options or the *Minimize, Maximize*, or *Close* buttons to keep a better overview on the individual datasets.\n",
"\n",
"When overplotting, you may want to set different custom colors for each dataset in its *Display* tab. If you are plotting different axes or the axes ranges are a significantly different range of values, you may want to set different axes locations for each plot in the *Axes* tab. When you are done, click the *Plot* button to see the overplot.\n",
"\n",
"Use the *Close* button in each data panel to close the panel and remove that plot.\n",
"\n",
"In the **plotms** task interface, you can overplot by invoking **plotms** more than once with clearplots=False. Each **plotms** command corresponds to a plot to go on top of previous ones, and each must have its own *plotindex* (0-based, default is 0). Otherwise, with the same plotindex, the second plot will overwrite the first. In the following example, we are plotting Scan vs. Time for MeasurementSet test1.ms with plotindex 0, and Field vs. Time for MeasurementSet test2.ms on the same plot with plotindex 1. The test2 data is a different color and its yaxis is on the right.\n",
"\n",
"Note that since the *plotindex* is an index into subplots, this parameter must be assigned in consecutive order. If a *plotindex* is skipped, plotms will adjust the index number and inform the user of the corrected *plotindex* value.\n",
"\n",
"```\n",
"plotms(vis='test1.ms', yaxis='scan')\n",
"plotms(vis='test2.ms', yaxis='field', plotindex=1, clearplots=False, customsymbol=True, symbolcolor='00FF00', yaxislocation='right')\n",
"```\n",
"\n",
"**Plotting Multiple Datasets or Axes on a Grid**\n",
"\n",
"**plotms** allows you to plot more than one dataset or axes on the same page by specifying a grid size then a grid location for each plot as described below. Here is an example of two plots with different datasets:\n",
"\n",
"{.image-inline}\n",
"\n",
">Plotting multiple data sets on a 2x1 grid. Here, the MS is plotted in grid location (1,1). Then the *Add Plot* button was used to select its bandpass calibration table and plot it in grid location (2,1).\n",
" \n",
"\n",
"The process is similar to the one above, except that you specify the grid and each plot\\'s location:\n",
"\n",
"1. Set up your first plot as described above.\n",
"2. Use the *Options* tab to set up a grid by incrementing the number of rows and/or columns. By default the plot you set up in step 1 will be in row 0, column 0.\n",
"3. Use the *Add Plot* button to set up the second plot\\'s parameters. Pay particular attention to the new dataset\\'s *Page* tab, where you can set the *Grid Location* (row and column number) of the new plot. This section appears only when a grid is set up.\n",
"4. Unlike iteration, you cannot share axes among the plots.\n",
"5. Add as many plots as you desire to fill your grid, then click *Plot.*\n",
"\n",
"Several **plotms** task parameters are used to create a grid and specify a plot location.\n",
"\n",
"- gridcols and gridrows define the number of plots on the screen.\n",
"- colindex and rowindex (0-based) set the location of an individual plot\n",
"- plotindex (0-based) must be incremented by 1 for each **plotms** call\n",
"- *clearplots* is set to False to keep previous plots\n",
"\n",
"Here is an example of multiple **plotms** calls to set up two plots on a grid and export the plot page; note the defaults on the first call are *rowindex=0, colindex=0, plotindex=0* so just set up the grid. On each subsequent **plotms** call set *clearplots=False* and increment the *plotindex* by 1. To save the gridded plot, set a *plotfile* on the final plot.\n",
"\n",
"```\n",
"plotms(vis='test1.ms', yaxis='field', gridrows=2, gridcols=1)\n",
"plotms(vis='test2.ms', yaxis='field', gridrows=2, gridcols=1, rowindex=1, colindex=0, plotindex=1, clearplots=False, plotfile='fields.jpg')\n",
"```\n",
"\n",
"------------------------------------------------------------------------\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "A-M9b3x10EDF"
},
"source": [
"### Interactive Flagging\n",
"\n",
"Interactive flagging, on the principle of \"see it --- flag it\", is possible on the X-Y display of the data plotted by **plotms**. Mark one or more regions as described below, then flag or unflag the data that falls in these regions of the display.\n",
"\n",
"\n",
"Do not attempt to flag data while another task is accessing the same data set.\n",
"
\n",
"\n",
" *A note about Table locks*\n",
"\n",
"For plotting, **plotms** opens the MeasurementSet read-only, so there should be no problem if another task accesses the same dataset, unless the other task locks the file. When this happens, you can wait for the lock to be released, cancel cache-loading in the **plotms** dialog box, type *go clearstat* at the prompt, or exit **plotms.** Do not attempt to flag data in **plotms** while another task is accessing the same data set, as in this case **plotms** must open the MeasurementSet with a file lock for writing.\n",
"\n",
"*Mark data for flagging*\n",
"\n",
"Using the row of icons at the bottom of the **plotms** window, click on the *Mark Regions* button, then mark a region by left-clicking and dragging the mouse. Each click and drag will mark an additional region. You can remove all of your marked regions by clicking on the *Clear Regions* button. Once regions are marked, you can then click on one of the other buttons to take action:\n",
"\n",
"1. Flag --- flag all of the points in the region(s),\n",
"2. Unflag --- unflag flagged points in the region(s),\n",
"3. Locate --- list information (X and Y value, scan, field, baseline, frequency, etc.) about the points in the region(s) to the command line or log (Warning: this could be a long list!).\n",
"\n",
"\n",
"**WARNING**: If you *Flag* a region, using *Unflag* **will not return the data to its previous state** but will unflag all of the data in the marked region.\n",
"\n",
"The following figure shows an example of marking regions and then clicking the *Flag* button. Whenever you click on a button, that action occurs without requiring an explicit disk-write. If you quit **plotms** and re-enter, you will see your previous edits because your flag changes were written to the MeasurementSet on disk.\n",
"\n",
"{.image-inline}\n",
"{.image-inline}\n",
"\n",
">Plot of amplitude versus time, before (top) and after (bottom) flagging two marked regions. Note that flagged data is not displayed so these regions are not plotted after flagging. To unflag these regions, mark the two same regions and click the *Unflag* button.\n",
" \n",
"\n",
"New interactive flagging is available in CASA 5.5 and later. The button *Flag All*, which is located next to the *Unflag* button, can turn on and off \\\"Flag all/Unflag all\\\" mode by clicking it. Instead of flag/unflag operation on selected region, the \\\"Flag all/Unflag all\\\" mode allows you to flag/unflag whole data associated with the grid. The usage of this mode is as follows:\n",
"\n",
"1. Press the *Flag All* button \\-- enter the \\\"Flag all/Unflag all\\\" mode. Background color of completely flagged grids will become yellow. \n",
"{.image-inline}\n",
"\n",
"2. Select grids to flag/unflag \\-- click each grid to select for flag/unflag when the mode is active. Unflag is selected for the grids where all data are already flagged, otherwise flag is selected. The background color of the grids selected for flag will change to yellow while the grids selected for unflag will change to the default color. \n",
"{.image-inline}\n",
"\n",
"3. Press the *Flag All* button again \\-- leave the \\\"Flag all/Unflag all\\\" mode. At this moment, flag/unflag operations are applied to the data of the currently displayed grids selected in the previous step, and each grid is updated accordingly. \n",
"{.image-inline}\n",
"\n",
"
\n",
"**WARNING:** On macOS, the \"Flag all/Unflag all\" mode does not work as expected!\n",
"
\n",
"\n",
"On macOS, background color of grids doesn\\'t yet change properly although flag/unflag operations work fine. It is not recommended to use this mode on macOS.\n",
"\n",
"
\n",
"**WARNING:** You cannot \"undo\" flagging to a previous state!\n",
"
\n",
"\n",
"**plotms** does not automatically create flag backups in the \\
.flagversions file. It is thus recommended to save the initial flags with the flagmanager task before starting **plotms** interactive flagging. Important intermediate flagging stages may also be saved during **plotms** flagging in the same fashion. Flagging can also be performed using the interactive msview task or scripted with the flagdata or flagcmd tasks.\n",
"\n",
"\n",
"WARNING: Use of flag extensions may lead to deletion of much more data than desired. Be careful!\n",
"
\n",
"\n",
"Flags can also be extended with options in the *Flag* tab; see that section for a more detailed description of these options. Flag extension enables the user to plot a subset of the data and extend the flagging to a wider set. In this release, the only functional extensions are correlation and channel.\n",
"\n",
"\n",
"WARNING: Interactive flagging doesn't support a collaboration with **Iteration** buttons!\n",
"
\n",
"\n",
"The flag/unflag operations are applied to currently displayed grids only, although you can move to other iterations in the \\\"Flag all/Unflag all\\\" mode.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "I7aQSJIC0EDG"
},
"source": [
"### Scripting With No GUI\n",
"\n",
"When scripting to produce exported plotfiles, set the **plotms** parameter *showgui=False* to suppress the GUI and pop-up dialog boxes which require a user response. The default is *showgui=True*.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KB-C-Ae70EDG"
},
"source": [
"### Exiting **plotms**\n",
"\n",
"To exit the **plotms** GUI, select *Quit* from the *File* menu at the top of the **plotms** window, or click the \"X\" on the frame.\n",
"\n",
"Alternatively, **plotms** will keep running in the background and update with each subsequent **plotms** call. If the data file changes in the background while you are using other tasks, you can force reloading the data via the *Reload* checkbox next to the *Plot* button, or press SHIFT while clicking on *Plot* for the same purpose.\n",
"\n",
"If started in a casa session, **plotms** will automatically quit when the session is ended.\n",
"\n",
"\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DD712BFD0EDH"
},
"source": [
"## Flag using flagdata\n",
"\n",
"Flagging MeasurementSets and calibration tables\n",
"\n",
"**flagdata** is the main flagging task in CASA. **flagdata** can flag MeasurementSets and calibration tables with an elaborate selection syntax. It also contains auto-flagging routines.\n",
"\n",
"\n",
"\n",
"Other tasks related to flagging are **flagmanager** to save the existing flags before adding more, and **plotms** and **msview** for interactive flagging. In addition, optionally, data for which calibration solutions have failed will be flagged when these solutions are applied.\n",
"\n",
"\n",
"\n",
"The inputs to **flagdata** are:\n",
"\n",
"```\n",
"#In CASA\n",
"CASA<1>: inp flagdata\n",
"#flagdata :: All-purpose flagging task based on data-selections and flagging modes/algorithms.\n",
"vis = '' #Name of MS file or calibration table to flag\n",
"mode = 'manual' #Flagging mode\n",
" field = '' #Field names or field index numbers: '' ==> all, field='0~2,3C286'\n",
" spw = '' #Spectral-window/frequency/channel: '' ==> all, spw='0:17~19'\n",
" antenna = '' #Antenna/baselines: '' ==> all, antenna ='3,VA04'\n",
" timerange = '' #Time range: '' ==> all,timerange='09:14:0~09:54:0'\n",
" correlation = '' #Correlation: '' ==> all, correlation='XX,YY'\n",
" scan = '' #Scan numbers: '' ==> all\n",
" intent = '' #Observation intent: '' ==> all, intent='CAL*POINT*'\n",
" array = '' #(Sub)array numbers: '' ==> all\n",
" uvrange = '' #UV range: '' ==> all; uvrange ='0~100klambda', default units=meters\n",
" observation = '' #Observation ID: '' ==> all\n",
" feed = '' #Multi-feed numbers: Not yet implemented\n",
" autocorr = False #Flag auto-correlations\n",
"\n",
"action = 'apply' #Action to perform in MS and/or in inpfile (none/apply/calculate)\n",
" display = '' #Display data and/or end-of-MS reports at runtime (data/report/both).\n",
" flagbackup = True #Back up the state of flags before the run\n",
"\n",
"savepars = False #Save the current parameters to the FLAG_CMD table or to a file\n",
"```\n",
"\n",
"vis can take a MeasurementSet or calibration table. Data selection for calibration tables is limited to field, scan, time, antenna, spw, and observation. See section at end about which parameters are applicable for calibration tables. Since calibration tables do not have a *FLAG_CMD* table, parameter settings, if requested, can only be saved in external files.\n",
"\n",
"\n",
"**The *mode* parameter**\n",
"\n",
"The mode parameter selects the flagging algorithm and the following are available:\n",
"\n",
"```\n",
"list = list of flagging commands to apply to MS\n",
"manual = flagging based on specific selection parameters\n",
"clip = clip data according to values\n",
"quack = remove/keep specific time range at scan beginning/end\n",
"shadow = remove antenna-shadowed data\n",
"elevation = remove data below/above given elevations\n",
"tfcrop = automatic identification of outliers on the time-freq plane\n",
"rflag = automatic detection of outliers based on sliding-window RMS filters\n",
"antint = flag integrations if all baselines to a specified antenna are flagged\n",
"extend = extend and/or grow flags beyond what the basic algorithms detect\n",
"summary = report the amount of flagged data\n",
"unflag = unflag the specified data\n",
"```\n",
"\n",
"these are described in detail lower down.\n",
"\n",
"Flagging will only be applied to the data selection that is performed with the usual selection parameters. The dataset is iterated-through in chunks (small pieces of data) consisting of one field, one spw, and a user-defined timerange (default is one scan). In addition to the typical antenna, spw, timerange, etc. selections, we would like to point out some addition of the correlation syntax for modes clip, tfcrop, and rflag. One can combine correlation products with simple mathematical expressions\n",
"\n",
"```\n",
"'ABS', 'ARG', 'RE', 'IM', 'NORM'\n",
"```\n",
"\n",
"where *ABS* is the absolute value, *RE*, the real and *IM* the imaginary part. *NORM* and *ARG* refer to the amplitude and phase. This parameter is followed by the polarization products (using an underscore in between \"\\_\" )\n",
"\n",
"```\n",
"'ALL', 'I', 'XX', 'YY', 'RR', 'LL', 'WVR'\n",
"```\n",
"\n",
"'WVR' refers to the water vapour radiometer of ALMA data. Note that the operators ABS,ARG,RE, etc. are written only once as the first value. if more than one correlation is given, the operator will be applied to all of them. An example would be\n",
"\n",
"```\n",
"correlation = 'RE_XX, XY'\n",
"```\n",
"\n",
"which would select all real XX and XY polarization for flagging.\n",
"\n",
"\n",
"**The *action* parameter**\n",
"\n",
"The parameter action controls whether the actual flagging commands will be applied or not and the options are the empty string \", 'apply' and 'calculate'.\n",
"\n",
"apply is likely the most popular one as it applies the flags to the MS:\n",
"\n",
"```\n",
"#In CASA:\n",
"action = 'apply' #Action to perform in MS and/or in inpfile\n",
"#(none/apply/calculate)\n",
"display = '' #Display data and/or end-of-MS reports at runtime\n",
"#(data/report/both).\n",
"flagbackup = True #Back up the state of flags before the run\n",
"```\n",
"\n",
"flagbackup specifies if a backup of the current flags should be saved in the \"\\*.flagversions\" file. display can be \", 'data', 'report', 'both' where the empty string \" will report no individual flagging statistics, whereas 'data' launches an interactive GUI to display data and flags for each chunk to browse through. The plots are time-frequency planes and both old and new flags are being overlaid for all correlations per baseline. In the GUI, one can step though all chunks for inspection and if the flagging is unsatisfactory, one can exit without applying the flags. If the flagging is acceptable, it is also possible to continue flagging without viewing all chunks (the number of chunks can be very large for typical JVLA and ALMA data sets. display='report' lists the flagging statistics at the end of the procedure on the screen and both starts the GUI and reports all statistics at the end.\n",
"\n",
"action='calculate' calculates the flags but does not write them to the MS or calibration table. This is useful if one would like to inspect the computed flags in the GUI without a straight application:\n",
"\n",
"```\n",
"action = 'calculate' #Action to perform in MS and/or in inpfile (none/apply/calculate)\n",
"display = '' #Display data and/or end-of-MS reports at runtime (data/report/both).\n",
"```\n",
"\n",
"The empty string action=\" will do nothing and is useful when the commands themselves shall only be written to the FLAG_CMD sub-table or to an external file using the savepars parameter to specify the filename.\n",
"\n",
"savepars will save the flagging commands to a file that can be later used for input in flagdata via mode='list'. It also shares the flagcmd syntax and can be used there. The file name is specified by outfile and, if empty, the FLAG_CMD table in the MS will be populated. A REASON can be given by the reason parameter which may be useful for bookkeeping as well as for unflagging data that are marked by specific REASON parameters. The overwrite parameter will control overwriting an existing file when saving the flag commands.\n",
"\n",
"\n",
"**Flagging Modes**\n",
"\n",
"*Manual Flag/Unflag*\n",
"\n",
"```\n",
"mode = 'manual' #Flagging mode (list/manual/clip/shadow/quack/el\n",
"#evation/tfcrop/rflag/extend/unflag/summary)\n",
"field = '' #Field names or field index numbers: '' ==> all,\n",
"#field='0~2,3C286'\n",
"spw = '' #Spectral-window/frequency/channel: '' ==> all,\n",
"#spw='0:17~19'\n",
"antenna = '' #Antenna/baselines: '' ==> all, antenna\n",
"#='3,VA04'\n",
"timerange = '' #Time range: '' ==>\n",
"#all,timerange='09:14:0~09:54:0'\n",
"correlation = '' #Correlation: '' ==> all, correlation='XX,YY'\n",
"scan = '' #Scan numbers: '' ==> all\n",
"intent = '' #Observation intent: '' ==> all,\n",
"#intent='CAL*POINT*'\n",
"array = '' #(Sub)array numbers: '' ==> all\n",
"uvrange = '' #UV range: '' ==> all; uvrange ='0~100klambda',\n",
"#default units=meters\n",
"observation = '' #Observation ID: '' ==> all\n",
"feed = '' #Multi-feed numbers: Not yet implemented\n",
"autocorr = False #Flag auto-correlations\n",
"```\n",
"\n",
"The 'manual' mode is the most straight-forward of all modes. All visibilities that are selected by the various data selection parameters will be flagged or unflagged, depending on the action parameter. autocorr is a shorthand for antenna='\\*&&&' to flag all auto correlations in the data.\n",
"\n",
"\n",
"\n",
"*List*\n",
"\n",
"```\n",
"mode = 'list' #Flagging mode (list/manual/clip/shadow/quack/el\n",
"#evation/tfcrop/rflag/extend/unflag/summary)\n",
"inpfile = '' #Input ASCII file, list of\n",
"#files or Python list of strings with\n",
"\n",
"#flag commands.\n",
"reason = 'any' #Select by REASON types\n",
"```\n",
"\n",
"A list of flag commands can be provided through a file or a list of files, specified by the inpfile parameter. Each input line may contain a flagging mode with data selection parameters as well as parameters that are specific to that mode. All parameters that are not set will be reset to their default values (default mode is 'manual'). Each line of this file or list of strings will be taken as a command to the flagdata task. This mode='list' is similar to the task flagcmd with the inpmode='list' option.\n",
"\n",
"An example for such a file would be:\n",
"\n",
"```\n",
"mode='shadow'\n",
"mode='clip' clipminmax=[0,5] correlation='ABS_ALL'\n",
"mode='quack' quackmode='end' quackinterval=1.0\n",
"antenna='ea01' timerange='00:00:00~01:00:00'\n",
"antenna='ea11' timerange='00:00:00~03:00:00' spw='0~4'\n",
"```\n",
"\n",
"Alternatively, this can be issued in the task directly like:\n",
"\n",
"```\n",
"#In CASA:\n",
"CASA<1>: flagdata(vis='vis',mode='list',\n",
"inpfile=[\"mode='shadow'\",\n",
"\"mode='clip' clipminmax=[0,5] correlation='ABS_ALL'\",\n",
"\"mode='quack' quackmode='end' quackinterval=1.0\"'\n",
"\"antenna='ea01' timerange='00:00:00~01:00:00'\",\n",
"\"antenna='ea11' timerange='00:00:00~03:00:00' spw='0~4'\"])\n",
"```\n",
"\n",
" or via a variable\n",
"\n",
"```\n",
"#In CASA:\n",
"CASA<1>:cmds=[\"mode='shadow',\n",
"\"mode='clip' clipminmax=[0,5] correlation='ABS_ALL'\",\n",
"\"mode='quack' quackmode='end' quackinterval=1.0\",\n",
"\"antenna='ea01' timerange='00:00:00~01:00:00'\",\n",
"\"antenna='ea11' timerange='00:00:00~03:00:00' spw='0~4'\"]\n",
"\n",
"CASA<2>: flagdata(vis='vis',mode='list', inpfile=cmds)\n",
"```\n",
"\n",
"The syntax needs to be written with quotes e.g. mode='manual' antenna='ea10'. There should be no space between key=value. Spaces are used to separate pairs of parameters, not commas.\n",
"\n",
"\n",
"\n",
"*Clip*\n",
"\n",
"```\n",
"mode = 'clip' #Flagging mode (list/manual/clip/shadow/quack/\n",
"#elevation/tfcrop/rflag/extend/unflag/summary)\n",
"...\n",
"datacolumn = 'DATA' #Data column on which to operate\n",
"#(data,corrected,model,residual)\n",
"clipminmax = [] #Range to use for clipping\n",
"clipoutside = True #Clip outside the range, or within it\n",
"channelavg = False #Average over channels (scalar average)\n",
"timeavg = False #Average over time ranges\n",
"timebin = '' #Bin width for time averaging.\n",
"clipzeros = False #Clip zero-value data\n",
"```\n",
"\n",
"in addition to the regular selection parameters, mode='clip' also has an option to select between a number of scratch columns in datacolumn. This includes the usual DATA, CORRECTED, etc., and also clipping based on data weights WEIGHT, WEIGHT_SPECTRUM as well as other MS columns. clipminmax selects the range of values to be clipped -- usually this is combined with clipoutside=True to clip everything but the values covered in clipminmax. The data can also be averaged over the selected spw channel ranges by setting channelavg=True, or time averages via timeavg=True and setting of timebin. clip will also flag 'NaN', 'inf', and '-inf' values by default and can flag exact zero values (these are sometimes produced by the JVLA correlator) using the clipzeros parameter.\n",
"\n",
"Note : For modes clip, tfcrop and rflag, channel-ranges can be excluded from flagging by selecting ranges such as spw='0:05;1063'. This is a way to protect known spectral-lines from being flagged by the autoflag algorithms.\n",
"\n",
"\n",
"\n",
"*Shadow*\n",
"\n",
"```\n",
"mode = 'shadow' #Flagging mode (list/manual/clip/shadow/quack/\n",
"#elevation/tfcrop/rflag/extend/unflag/summary)\n",
"...\n",
"tolerance = 0.0 #Amount of shadow allowed (in meters)\n",
"addantenna = '' #File name or dictionary with additional antenna names,\n",
"#positions and diameters\n",
"```\n",
"\n",
"This option flags shadowed antennas, i.e. when one antenna blocks part of the aperture of a second antenna that is behind the first one. Shadowing can be gradual and the criterion for a shadow flag is when a baseline is shorter than radius~1~ + radius~2~ − tolerance (where the radii of the antennae are taken from the MS antenna subtable); see the figure below. addantenna may be used to account for shadowing when antennas are not listed in the MS but are physically present. Please read the flagdata inline help for the syntax of this option.\n",
"\n",
"\n",
"\n",
"\n",
"> This figure shows the geometry used to compute shadowed antennas.\n",
"\n",
"\n",
"\n",
"*Quack*\n",
"\n",
"```\n",
"mode = 'quack' #Flagging mode (list/manual/clip/shadow/quack/\n",
"#elevation/tfcrop/rflag/extend/unflag/summary)\n",
"...\n",
"quackinterval = 0.0 #Quack n seconds from scan beginning or end\n",
"quackmode = 'beg' #flag an interval at the beginning of scan\n",
"'endb' #flag an interval at the end of scan\n",
"'tail' #flag all but an interval at the beginning of\n",
"scan\n",
"'end' #flag all but an interval at end of scan\n",
"\n",
"quackincrement = False #Flag incrementally in time?\n",
"```\n",
"\n",
"quack is used to remove data at scan boundaries. quackinterval specifies the time in seconds to be flagged, and quackmode can be 'beg' to flag the quackinterval at the beginning of each selected scan, 'endb' at the end of scan. 'tail' flags all but the beginning of scan and 'end' all but the end of scan. The quackincrement is either True or False, depending if one wishes to flag the quackinterval from the first unflagged data in the scan, or from the scan boundaries independent of data being already flagged or not.\n",
"\n",
"\n",
"\n",
"Because *quackincrement=True* needs to know the state of the previous flags in order to know which is the first unflagged data, it cannot be used in \\'list\\' mode, unless it is the first command of the list.\n",
"\n",
"```python\n",
"Visual representation of quack mode when flagging a scan\n",
"with 1s duration. The following diagram shows what is flagged\n",
"for each quack mode when quackinterval is set to 0.25s.\n",
"The flagged part is represented by crosses (+++++++++)\n",
"\n",
"scan with 1s duration\n",
"\n",
"--------------------------------------------\n",
"beg\n",
"+++++++++++---------------------------------\n",
"endb\n",
"---------------------------------+++++++++++\n",
"tail\n",
"-----------+++++++++++++++++++++++++++++++++\n",
"end\n",
"+++++++++++++++++++++++++++++++++-----------\n",
"```\n",
"\n",
"\n",
"*Elevation*\n",
"\n",
"```\n",
"mode = 'elevation' #Flagging mode (list/manual/clip/shadow/quack/\n",
"#elevation/tfcrop/rflag/extend/unflag/summary)\n",
"...\n",
"lowerlimit = 0.0 #Lower limiting elevation (in degrees)\n",
"upperlimit = 90.0 #Upper limiting elevation (in degrees)\n",
"```\n",
"\n",
"Flagging based on the elevation of the antennae. This may be useful to avoid data taken at very low elevations or close to transit and the lowerlimit and upperlimit parameters specify the range of good elevations.\n",
"\n",
"\n",
"*Tfcrop*\n",
"\n",
"```\n",
"mode = 'tfcrop' #Flagging mode (list/manual/clip/shadow/quack/\n",
"#elevation/tfcrop/rflag/extend/unflag/summary\n",
"#)\n",
"...\n",
"ntime = 'scan' #Time-range to use for each chunk (in seconds\n",
"#or minutes)\n",
"combinescans = False #Accumulate data across scans.\n",
"datacolumn = 'DATA' #Data column on which to operate\n",
"#(data,corrected,model,residual)\n",
"timecutoff = 4.0 #Flagging thresholds in units of deviation\n",
"#from the fit\n",
"freqcutoff = 3.0 #Flagging thresholds in units of deviation\n",
"#from the fit\n",
"timefit = 'line' #Fitting function for the time direction\n",
"#(poly/line)\n",
"freqfit = 'poly' #Fitting function for the frequency direction\n",
"#(poly/line)\n",
"maxnpieces = 7 #Number of pieces in the polynomial-fits (for\n",
"#'freqfit' or 'timefit' = 'poly')\n",
"flagdimension = 'freqtime' #Dimensions along which to calculate fits\n",
"#(freq/time/freqtime/timefreq)\n",
"usewindowstats = 'none' #Calculate additional flags using sliding\n",
"#window statistics (none,sum,std,both)\n",
"halfwin = 1 #Half-width of sliding window to use with\n",
"#'usewindowstats' (1,2,3).\n",
"extendflags = True #Extend flags along time,\n",
"#frequency and correlation.\n",
"channelavg = False #Pre-average data across channels before\n",
"#analyzing visibilities for flagging.\n",
"chanbin = False #Bin width for channel average in\n",
"#number of input channels.\n",
"timeavg = False #Pre-average data across time before\n",
"#analyzing visibilities for flagging\n",
"timebin = False #Bin width for time average in seconds\n",
"```\n",
"\n",
"TFCrop is an autoflag algorithm that detects outliers on the 2D time-frequency plane, and can operate on un-calibrated data (non bandpass-corrected). The original implementation of this algorithm is described in NCRA Technical Report 202 (Oct 2003).\n",
"\n",
"The algorithm iterates through the data in chunks of time. For each chunk, the result of user-specified visibility-expressions are organized as 2D time-frequency planes, one for each baseline and correlation-expression result, and the following steps are performed.\n",
"\n",
"As of CASA 4.6 the data can also be pre-averaged over the selected spw channel ranges by setting channelavg=True and chanbin to the desired bin (in number of channels), or time averaged over the selected time ranges by setting timeavg=True and timebin to the desired time range (in seconds). This averaging is independent from the tfcrop time/channel average, and allows to specify custom time/channel average bins, instead of averaging all data across time and/or channel direction.\n",
"\n",
"1. Calculate a bandshape template : Average the data across time, to construct an average bandpass. Construct an estimate of a clean bandpass (without RFI) via a robust piece-wise polynomial fit to the average bandpass shape.\n",
"\n",
" Note : A robust fit is computed in up to 5 iterations. It begins with a straight line fit across the full range, and gradually increases to 'maxnpieces' number of pieces with third-order polynomials in each piece. At each iteration, the stddev between the data and the fit is computed, values beyond N-stddev are flagged, and the fit and stddev are re-calculated with the remaining points. This stddev calculation is adaptive, and converges to a value that reflects only the data and no RFI. At each iteration, the same relative threshold is applied to detect flags, and this results in a varying set of flagging thresholds, that allows deep flagging only when the fit represents the true data best. Iterations stop when the stddev changes by less than 10%, or when 5 iterations are completed.\n",
"\n",
" The resulting clean bandpass is a fit across the base of RFI spikes.\n",
"\n",
"2. Divide out this clean bandpass function from all timesteps in the current chunk. Now, any data points that deviate from a mean of 1 can be considered RFI. This step helps to separate narrow-band RFI spikes from a smooth but varying bandpass, in situations where a simple range-based clipping will flag good sections of the bandpass.\n",
"\n",
"3. Perform iterative flagging (robust flagging) of points deviating from a value of 1.\n",
"\n",
" Flagging is done in up to 5 iterations. In each iteration, for every timestep, calculate the stddev of the bandpass-flattened data, flag all points further than N times stddev from the fit, and recalculate the stddev. At each iteration, the same relative threshold is applied to detect flags. Optionally, use sliding-window based statistics to calculate additional flags.\n",
"\n",
"4. Repeat steps 1 and 3, but in the other direction (i.e. average the data across frequency, calculate a piece-wise polynomial fit to the average time-series, and find flags based on deviations w.r.to this fit.)\n",
"\n",
"The default parameters of the tfcrop implementation are optimized for strong narrow-band RFI (see the example figure below). With broad-band RFI, the piece-wise polynomial can sometimes model it as part of the band-shape, and therefore not detect it as RFI. In this case, reducing the maximum number of pieces in the polynomial can help. This algorithm usually has trouble with noisy RFI that is also extended in time of frequency, and additional statistics-based flagging is recommended (via the 'usewindowstats' parameter). It is often required to set up parameters separately for each spectral-window.\n",
"\n",
"If frequency ranges of known astronomical spectral lines are known a-priori , they can be protected from automatic flagging by de-selecting those frequency-ranges via the 'spw' data-selection parameter.\n",
"\n",
"The extendflag parameter will clean up small portions of data between flagged data points along time and/or frequency when more than 50% of all timeranges or 80% of all channels are already flagged. It will also extend the flags to the other polarizations. Alternatively, mode='extend' can be used (see section below).\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"> This screenshot represents a run where 'tfcrop' was run on a spw='9' with mainly narrow-band RFI. RIGHT : An example of protecting a spectral line (in this case, demonstrated on an RFI spike) by setting the spw-selection to spw='0:0 45;53 63'. In both figures, the top row indicates the data before flagging, and the bottom row after flagging.\n",
" \n",
"\n",
"*Rflag*\n",
"\n",
"```\n",
"mode = 'rflag' #Flagging mode (list/manual/clip/shadow/quack/\n",
"#elevation/tfcrop/rflag/extend/unflag/summary\n",
"#)\n",
"...\n",
"ntime = 'scan' #Time-range to use for each chunk (in seconds\n",
"#or minutes)\n",
"combinescans = False #Accumulate data across scans.\n",
"datacolumn = 'DATA' #Data column on which to operate\n",
"#(data,corrected,model,residual)\n",
"winsize = 3 #Number of timesteps in the sliding time\n",
"#window [aips:fparm(1)]\n",
"timedev = '' #Time-series noise estimate [aips:noise]\n",
"freqdev = '' #Spectral noise estimate [aips:scutoff]\n",
"timedevscale = 5.0 #Threshold scaling for timedev [aips:fparm(9)]\n",
"freqdevscale = 5.0 #Threshold scaling for freqdev\n",
"#[aips:fparm(10)]\n",
"spectralmax = 1000000.0 #Flag whole spectrum if freqdev is greater\n",
"#than spectralmax [aips:fparm(6)]\n",
"spectralmin = 0.0 #Flag whole spectrum if freqdev is less than\n",
"#spectralmin [aips:fparm(5)]\n",
"extendflags = True #Extend flags along time, frequency and correlation.\n",
"channelavg = False #Pre-average data across channels before\n",
"#analyzing visibilities for flagging.\n",
"chanbin = False #Bin width for channel average in\n",
"#number of input channels.\n",
"timeavg = False #Pre-average data across time before\n",
"#analyzing visibilities for flagging\n",
"timebin = False #Bin width for time average in seconds\n",
"```\n",
"\n",
"RFlag is an autoflag algorithm based on a sliding window statistical filter. The RFlag algorithm was originally developed by Eric Greisen in AIPS (31DEC11). AIPS documentation : Subsection E.5 of the AIPS cookbook (Appendix E : Special Considerations for JVLA data calibration and imaging in AIPS [http://www.aips.nrao.edu/cook.html\\#CEE](http://www.aips.nrao.edu/cook.html#CEE))\n",
"\n",
"In RFlag, the data is iterated-through in chunks of time, statistics are accumulated across time-chunks, thresholds are calculated at the end, and applied during a second pass through the dataset.\n",
"\n",
"The CASA implementation also optionally allows a single-pass operation where statistics and thresholds are computed and also used for flagging, within each time-chunk (defined by 'ntime' and 'combinescans').\n",
"\n",
"For each chunk, calculate local statistics, and apply flags based on user supplied (or auto-calculated) thresholds.\n",
"\n",
"As of CASA 4.6 the data can also be pre-averaged over the selected spw channel ranges by setting channelavg=True and chanbin to the desired bin (in number of channels), or time averaged over the selected time ranges by setting timeavg=True and timebin to the desired time range (in seconds). This averaging is independent from the rflag time/channel sliding window, as it performs not only an average but also a binning operation (so there is no data overlap between adjacent bins), and allows to specify independent time/channel bins.\n",
"\n",
"1. Time analysis (for each channel)\n",
" a. Calculate local rms of real and imag visibilities, within a sliding time window\n",
" b. Calculate the median rms across time windows, deviations of local rms from this median, and the median deviation\n",
" c. Flag if local rms is larger than timedevscale x (medianRMS + medianDev)\n",
"2. Spectral analysis (for each time)\n",
" a. Calculate avg of real and imag visibilities and their rms across channels\n",
" b. Calculate the deviation of each channel from this avg, and the median-deviation\n",
" c. Flag if deviation is larger than freqdevscale x medianDev\n",
"\n",
"The extendflags parameter will clean up small portions of data between flagged data points along time and/or frequency when more than 50% of all timeranges or 80% of all channels are already flagged. It will also extend the flags to the other polarizations. Alternatively, mode='extend' can be used.\n",
"\n",
"Some examples (see figure below):\n",
"\n",
"1. Calculate thresholds automatically per scan, and use them to find flags. Specify scale-factor for time-analysis thresholds, use default for frequency.\n",
" \n",
"```\n",
"#In CASA:\n",
"CASA<1>: flagdata('my.ms', mode='rflag',spw='9',timedevscale=4.0)\n",
"```\n",
"\n",
"2. Supply noise-estimates to be used with default scale-factors.\n",
"\n",
"```\n",
"#In CASA:\n",
"CASA<1>: flagdata(vis='my.ms', mode='rflag', spw='9', timedev=0.1, freqdev=0.5);\n",
"```\n",
"\n",
"Two-passes. This replicates the usage pattern in AIPS.\n",
"\n",
" - The first pass saves commands in an output text files, with auto-calculated thresholds. Thresholds are returned from rflag only when action=\\'calculate\\' (calc-only mode). The user can edit this file before doing the second pass, but the python-dictionary structure must be preserved.\n",
" - The second pass applies these commands (action=\\'apply\\').\n",
" \n",
"```\n",
"#In CASA:\n",
" CASA<1>: flagdata(vis='my.ms', mode='rflag', spw='9,10', timedev='tdevfile.txt', freqdev='fdevfile.txt', action='calculate');\n",
" CASA<2>: flagdata(vis='my.ms', mode='rflag', spw='9,10', timedev='tdevfile.txt', freqdev='fdevfile.txt', action='apply');\n",
"```\n",
" \n",
"\n",
" \n",
"> Example of rflag on narrow-band RFI\n",
"\n",
"\n",
"*Extend*\n",
"\n",
"```\n",
"mode = 'extend' #Flagging mode (list/manual/clip/shadow/quack/el\n",
"#evation/tfcrop/rflag/extend/unflag/summary)\n",
"field = '' #Field names or field index numbers: '' ==> all,\n",
"#field='0~2,3C286'\n",
"spw = '' #Spectral-window/frequency/channel: '' ==> all,\n",
"#spw='0:17~19'\n",
"antenna = '' #Antenna/baselines: '' ==> all, antenna\n",
"#='3,VA04'\n",
"timerange = '' #Time range: '' ==>\n",
"#all,timerange='09:14:0~09:54:0'\n",
"correlation = '' #Correlation: '' ==> all, correlation='XX,YY'\n",
"scan = '' #Scan numbers: '' ==> all\n",
"intent = '' #Observation intent: '' ==> all,\n",
"#intent='CAL*POINT*'\n",
"array = '' #(Sub)array numbers: '' ==> all\n",
"uvrange = '' #UV range: '' ==> all; uvrange ='0~100klambda',\n",
"#default units=meters\n",
"observation = '' #Observation ID: '' ==> all\n",
"feed = '' #Multi-feed numbers: Not yet implemented\n",
"ntime = 'scan' #Time-range to use for each chunk (in seconds or\n",
"#minutes)\n",
"combinescans = False #Accumulate data across scans.\n",
"extendpols = True #If any correlation is flagged, flag all\n",
"#correlations\n",
"growtime = 50.0 #Flag all 'ntime' integrations if more than X%\n",
"#of the timerange is flagged (0-100)\n",
"growfreq = 50.0 #Flag all selected channels if more than X% of\n",
"#the frequency range is flagged(0-100)\n",
"growaround = False #Flag data based on surrounding flags\n",
"flagneartime = False #Flag one timestep before and after a flagged\n",
"#one (True/False)\n",
"flagnearfreq = False #Flag one channel before and after a flagged one\n",
"#(True/False)\n",
"```\n",
"\n",
"Although the modes tfcrop and rflag already have extendflags parameters, some autoflagging algorithms may still leave small islands of unflagged data behind, data that are surrounded by flagged visibilities in the time-frequency space. Although the algorithm may deem these visibilities as good ones, they are frequently affected by low-level RFI that spills from the adjacent, flagged points and one may wish to clean those up.\n",
"\n",
"ntime specifies the time ranges over which to clean up, e.g. '1.5min' or 'scan' which checks on all data within a scan. To span time ranges larger than scans, one can set combinescans to True.\n",
"\n",
"extendpols=True would extend all flags to all polarization products when at least one of them is flagged.\n",
"\n",
"growtime flags the entire time range for a flagged channel, when a certain fraction of flagged time intervals is exceeded.\n",
"\n",
"growfreq is similar but extends the flags in frequency when a given fraction of channels is already flagged.\n",
"\n",
"growaround checks for flagged data points in the time-frequency domain that neighbor a datum. The threshold is four data points. If more surrounding points are flagged, the central datum will be flagged, too.\n",
"\n",
"flagneartime flags adjacent data points along the time axis, around a flagged datum\n",
"\n",
"flagnearfreq flags neighboring channels.\n",
"\n",
"\n",
"\n",
"> This screenshot represents a run where 'tfcrop' was run only on 'ABS_RR' (top row) and followed by an extension along time and correlations (bottom row).\n",
"\n",
"\n",
"\n",
"*Unflag*\n",
"\n",
"```\n",
"mode = 'unflag' #Flagging mode (list/manual/clip/shadow/quack/\n",
"#elevation/tfcrop/rflag/extend/unflag/summary\n",
"#)\n",
"field = '' #Field names or field index numbers: ''==>all,\n",
"#field='0~2,3C286'\n",
"spw = '' #spectral-window/frequency/channel\n",
"antenna = 'ea01' #antenna/baselines: ''==>all, antenna\n",
"#='3,VA04'\n",
"timerange = '' #time range:\n",
"#''==>all,timerange='09:14:0~09:54:0'\n",
"correlation = '' #Select data based on correlation\n",
"scan = '' #scan numbers: ''==>all\n",
"intent = '' #Select data based on observation intent:\n",
"#''==>all\n",
"feed = '' #multi-feed numbers: Not yet implemented\n",
"array = '' #(sub)array numbers: ''==>all\n",
"uvrange = '' #uv range: ''==>all; uvrange ='0~100klambda',\n",
"#default units=meters\n",
"observation = '' #Select data based on observation ID: ''==>all\n",
"```\n",
"\n",
"The selection data will be unflagged.\n",
"\n",
"\n",
"*Summary*\n",
"\n",
"```\n",
"mode = 'summary' #Flagging mode (list/manual/clip/shadow/quack/\n",
"#elevation/tfcrop/rflag/extend/unflag/summary\n",
"#)\n",
"...\n",
"minrel = 0.0 #minimum number of flags (relative)\n",
"maxrel = 1.0 #maximum number of flags (relative)\n",
"minabs = 0 #minimum number of flags (absolute)\n",
"maxabs = -1 #maximum number of flags (absolute). Use a\n",
"#negative value to indicate infinity.\n",
"spwchan = False #Print summary of channels per spw\n",
"spwcorr = False #Print summary of correlation per spw\n",
"basecnt = False #Print summary counts per baseline\n",
"```\n",
"\n",
"This mode reports the number of rows and data points that are flagged. The selection of reported points can be restricted (see inline help for details).\n",
"\n",
"mode='summary' can also report back a dictionary if the task is run as\n",
"\n",
"```\n",
"#In CASA:\n",
"CASA<1>: s = flagdata(..., mode='summary')\n",
"```\n",
"\n",
"with a variable assigned, here 's'.\n",
"\n",
"***\n",
"\n",
"For advanced information about some parts of flagdata, please visit this historic memo on Flaggin in CASA:\n",
"\n",
"http://www.aoc.nrao.edu/~rurvashi/FlaggerDocs/FlaggerDocs.html\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mUJPUPYJ0EDI"
},
"source": [
"## Flag using flagcmd\n",
"\n",
"Flag the visibility data set or calibration table based on a specified list of flagging commands\n",
"\n",
"\n",
"**Info:** We recommend using task **flagdata** as the preferable and safer way for flagging based on the visitibilites inspection and for many other capabilities. The option to import XML files with online flag in **flagcmd** has largely become obsolete with the deprecation of **importevla**, because the recommended **importasdm** task cannot copy the actual XML tables from the original SDM to the newly created MS (it can only apply the online flags directly, or write them into ascii tables).\n",
"
\n",
"\n",
"The task **flagcmd** will flag the visibility data set or calibration table based on a specified set of flagging commands using a flagging syntax. These commands can be input from the *FLAG_CMD* table within the MS, from an ascii file, or from input python strings. Input can also be from an XML table within a VLA SDM, but given that importasdm does not copy XML files (and importevla is deprecated), the *Flag.xml, Antenna.xml* and *SpectralWindow.xml* tables must first be copied manually into the top-level MS directory for use by **flagcmd** (not the recommended approach). Facilities for manipulation, listing, or plotting of these flags are also provided.\n",
"\n",
"When doing any flagging with **flagcmd** it is wise to use the *flagbackup=True* parameter to save the current flags into a .flagversions file. See **flagmanager** for more details about this.\n",
"\n",
"\n",
"**Alert**: The *FLAG_CMD* sub-table is meant only for meta-data selections such as online flags. Using it to save other parameters (from modes such as *clip, quack, shadow,* etc) is possible but carries a risk that in future releases these parameters maybe renamed or changed their default values. There will be no automatic way to rename any parameter that changes in the future.\n",
"
\n",
"\n",
"\n",
"**Alert**: There is no way to guarantee that a command from the *COMMAND* column has been applied or not to the MS, even if the *APPLIED* column is set to True. If you use other ways to flag such as interactive flagging in plotms, the *FLAG_CMD* will not be updated! Use at your own risk!\n",
"
\n",
"\n",
"The inputs to **flagcmd** are:\n",
"\n",
"```\n",
"#flagcmd :: Flagging task based on batches of flag-commands\n",
"vis = '' #Name of MS file or calibration table to flag\n",
"inpmode = 'table' #Input mode for flag commands(table/list/xml)\n",
"inpfile = '' #Source of flag commands\n",
"[tablerows = [] #Rows of inpfile to read]\n",
"reason = 'any' #Select by REASON types\n",
"useapplied = False #Select commands whose rows\n",
" #have APPLIED column set to True\n",
"action = 'apply' #Action to perform in MS and/or in inpfile\n",
" #(apply/unapply/list/plot/clear/extract)\n",
"flagbackup = True #Automatically backup the\n",
" #FLAG column before execution\n",
"savepars = False #Save flag commands to the MS or to a file\n",
"```\n",
"\n",
"\n",
"The default input mode is *inpmode='table'* which directs the task to input flag commands from the *FLAG_CMD* internal MS table. Other options include *list* and *xml*, explained below.\n",
"\n",
"The default operation mode is action='apply' directing the task to apply relevant flagging commands to the MS data main table. Other options include *\\'unapply\\', \\'list\\', \\'plot\\', \\'clear\\',* and *\\'extract\\'*, explained below.\n",
"\n",
"See the Flagging Command Syntax section below for more detail.\n",
"\n",
"\n",
"**Alert:** It is possible to flag calibration tables using **flagcmd**, although we recommend using the **flagdata** task for this in most cases.\n",
"
\n",
"\n",
"When using **flagcmd** to flag calibration tables, only the apply and list actions are supported. Because calibration tables do not have a *FLAG_CMD* sub-table, the default inpmode='table' can only be used if an MS is given in the inpfile parameter so that flags from the MS are applied to the calibration table directly. Otherwise, the flag commands must be given using inpmode='list', either from a file or from a list of strings.\n",
"\n",
"\n",
"\n",
"**Input modes inpmode:**\n",
"\n",
"The inpmode parameter selects options for the input mode for the flagging commands.\n",
"\n",
"Available inpmode options are:\n",
"\n",
"- 'table' --- input from MS table\n",
"- 'list' --- input from ASCII file or from a list of strings\n",
"- 'xml' --- input from XML table (largely obsolete with the deprecation of **importevla** in CASA 5.4) \n",
"\n",
"The default input mode is inpmode='table' which directs the task to input flag commands from a *FLAG_CMD* MS table. This has the sub-parameters:\n",
"\n",
"```\n",
"inpmode = 'table' #Input mode for flag commands(table/list/xml)\n",
"inpfile = '' #Source of flag commands\n",
"[tablerows = [] #Rows of inpfile to read]\n",
"reason = 'any' #Select by REASON types\n",
"useapplied = False #Select commands whose rows\n",
" #have APPLIED column set to\n",
" #True\n",
"```\n",
"\n",
"If inpfile = \\'\\' then it will look for the *FLAG_CMD* table in the MS given by vis. You can use this sub-parameter to tell the task to read the *FLAG_CMD* table from another MS and apply then to the MS given in *vis*.\n",
"\n",
"The tablerows sub-parameter is a simple Python list of the row numbers of the table to consider in processing flags. The default is all rows. Currently it only takes fully-enumerated integer lists. Use the Python range function to generate ranges,\n",
"\n",
"```\n",
"tablerows = range(0,30) + range(50,55)\n",
"\n",
"#Do not use strings such as '0~29,50~54'\n",
"```\n",
"\n",
"The useapplied sub-parameter toggles whether only flag commands marked as not having been applied are considered (the default), or to allow (re)processing using all commands.\n",
"\n",
"The reason sub-parameter selects the *reason* type to process. The default 'any' means all commands.\n",
"\n",
"\n",
"**Info:** what is within the string is literally matched, e.g. *reason=''* matches only blank reasons, and *reason = 'FOCUS_ERROR,SUBREFLECTOR_ERROR'* matches this compound reason string only. To flag by either of these reasons alone, run **flagcmd** twice, once with *reason='FOCUS_ERROR'*, and then with the other reason.\n",
"
\n",
"\n",
"One use case is to read the flag commands from the *FLAG_CMD* of an MS and apply them to a calibration table given in the parameter vis. Example:\n",
"\n",
"```\n",
"flagcmd(vis='cal-X54.B1', inpmode='table',inpfile='uid___A002_X2a5c2f_X54.ms', action='apply')\n",
"```\n",
"\n",
"1. Input mode *inpmode='list'*\n",
"\n",
" See **flagdata** help for syntax.\n",
"\n",
" This mode allows one to give a list of strings with flagging commands, the name of a file or a list of filenames that contains these commands equivalent to the mode='list' in flagdata. E.g. a file * flags.txt* that contains\n",
"\n",
"```\n",
"scan='1~3' mode='manual'\n",
"mode='clip' clipminmax=[0,2] correlation='ABS_XX' clipoutside=False\n",
"spw='9' mode='tfcrop' correlation='ABS_YY' ntime=51.0\n",
"mode='extend' extendpols=True\n",
"```\n",
"\n",
" - can be used via\n",
"\n",
"```\n",
"flagcmd(vis='some.ms',inpmode='list',inpfile='flags.txt')\n",
"```\n",
"\n",
" - A list of input files can also be given:\n",
"\n",
"```\n",
"flagcmd(vis='some.ms',inpmode='list',inpfile=['flags.txt,'userflags.txt'])\n",
"```\n",
"\n",
" - Alternatively, the individual flagging commands can be directly provided in the call itself such as:\n",
"\n",
"```\n",
"inpfile=[\"scan='1~3' mode='manual'\",\n",
" \"mode='clip' clipminmax=[0,2] correlation='ABS_XX' clipoutside=False\",\n",
" \"spw='9' mode='tfcrop' correlation='ABS_YY' ntime=51.0\",\n",
" \"mode='extend' extendpols=True\"]\n",
"```\n",
"\n",
"\n",
"2. Input mode inpmode='xml'\n",
"\n",
" **Alert:** With the deprecation of **importevla**, XML files can no longer be imported directly from the original SDM into the newly created MS, but only by manually copying the *Flag.xml*, *Antenna.xml* and *SpectralWindow.xml* tables into the top-level MS directory (not the recommended approach). Also, the XML mode is not available for cal tables, therefore it will not work for ALMA MSs. However, task **importasdm** with *process_flags=True* will copy the flags from the XML files directly to the *FLAG_CMD* sub-table, see **importasdm** help for options. This is the recommend way of dealing with the online flags.\n",
"\n",
" The input mode inpmode='xml' tells the task to input flag commands from XML SDM files, which contain the online flags. When set this opens the sub-parameters:\n",
"\n",
"```\n",
"inpmode = 'xml' #Input mode for flag commands(table/list/xml)\n",
"tbuff = 0.0 #Time buffer (sec) to pad flags\n",
"ants = '' #Allowed flag antenna names to select by\n",
"reason = 'any' #Select by REASON types\n",
"```\n",
"\n",
" - This mode will look for files called Flag.xml, Antenna.xml and optionally SpectralWindow.xml inside the MS directory specified under vis.\n",
"\n",
"\n",
"**Info:** You can only apply the flags from an XML file. It is not possible to unapply them. For that, transfer the flags to the *FLAG_CMD* table using *action='list'*, then unapply them.\n",
"
\n",
"\n",
" - The tbuff sub-parameter sets a padding buffer (in seconds) to the begin and end times of the online flags in the XML file. The online flag time buffer tbuff is specified in seconds, but in fact should be keyed to the intrinsic online integration time to allow for events (like slewing) that occur within an integration period. This is particularly true for JVLA data, where a tbuff value of 0.5× to 1.5× the integration time is needed. For example, if data were taken with 1-second integrations, then at least tbuff=0.5 should be used, likewise tbuff=5 for 10-second integrations.\n",
"\n",
"\n",
"Info: For JVLA data you should use 1.5× (e.g. tbuff=15 for 10-second integrations) for data taken in early 2011 or before due to a timing error. We do not yet know what ALMA data will need for padding (if any).\n",
"
\n",
"\n",
" - The ants sub-parameter selects the antennas from which online flags will be selected (default is all antennas). For example, ants='ea01' is a valid choice for JVLA data.\n",
"\n",
" The reason sub-parameter selects by the REASON field in the Flag.xml file. The default 'any' means all commands. Note that reason=\" would only select flags who have a blank REASON field entry.\n",
"\n",
"\n",
"\n",
"**Operation types action**\n",
"\n",
" The action selects options for operating on the selected flags and possibly the data.\n",
"\n",
"Available action options are:\n",
"\n",
"- 'apply' --- apply flag commands to data\n",
"- 'unapply' --- unapply flags in data\n",
"- 'list' --- list and/or save flag commands\n",
"- 'plot' --- plot flag commands\n",
"- 'clear' --- clear rows from FLAG_CMD table\n",
"- 'extract' --- extract internal flag dictionary\n",
"\n",
"1. Apply flags action='apply'\n",
"\n",
" The default operation mode is action='apply' directing the task to apply relevant flagging commands to the *vis* data main table.\n",
"\n",
"```\n",
"action = 'apply' #Action to perform in MS and/or in inpfile\n",
" #(apply/unapply/list/plot/clear/extract)\n",
"flagbackup = True #Automatically backup the\n",
" #FLAG column before execution\n",
"```\n",
"\n",
" - The flagbackup parameter toggles whether a new copy of the MS FLAG column is written to the .flagversions backup directory for that MS before the requested flagging operation.\n",
"\n",
"\n",
"2. Unapply flags action='unapply'\n",
"\n",
" The unapply option allows unflagging of data based on the selected flag commands. This choice opens the sub-parameters:\n",
"\n",
"```\n",
"action = 'unapply' #Action to perform in MS and/or in inpfile\n",
" #(apply/unapply/list/plot/clear/extract)\n",
" flagbackup = True #Automatically backup the\n",
" #FLAG column before execution\n",
"```\n",
"\n",
" - As in action='apply', it is possible to make a backup to the \\*.flagversions file by using flagbackup=True.\n",
"\n",
"\n",
"\n",
"3. List and save flags action='list'\n",
"\n",
" The 'list' option will give a listing of the flagging commands. This choice opens the sub-parameters:\n",
"\n",
"```\n",
"action = 'list' #Action to perform in MS and/or in inpfile\n",
" #(apply/unapply/list/plot/clear/extract)\n",
" savepars = True #Save flag commands to the MS or to a file\n",
" outfile = '' #Name of output file to save commands\n",
" overwrite = True #Overwrite an existing file to save the flag commands\n",
"```\n",
"\n",
" - This action lists the commands on the screen without applying them. One can save the flagging script to an file specified in the outfile parameter when savepars=True. If outfile is empty, it will save the commands to a new row in the MS *FLAG_CMD* table given in vis.\n",
"\n",
" - The format of the listing output depends on the source of the flagging commands. A set of flagging commands specified through inpmode='list' will be listed directly. The flagging commands extracted through inpmode='table' will reflect the columns in the table: \\'*Row*\\', \\'*Timerange*\\', \\'*Reason*\\', \\'*Type*\\', \\'*Applied*\\', \\'*Lev*\\', \\'*Sev*\\', \\'*Command*\\' while commands from inpmode='xml' will be shown with the SDM XML table fields: \\'*Key*\\', \\'*FlagID*\\', \\'*Antenna*\\', \\'*Reason*\\', \\'*Timerange*\\'\n",
"\n",
"4. Plot flags action='plot'\n",
"\n",
" The 'plot' option will produce a graphical plot of flags of time versus antenna. This choice opens the sub-parameters:\n",
"\n",
"```\n",
"action = 'plot' #Action to perform in MS and/or in inpfile\n",
" #(apply/unapply/list/plot/clear/extract)\n",
" plotfile = '' #Name of output file to save plot\n",
"```\n",
"\n",
" - This is only useful for MS. flagcmd is most often used to plot the VLA or ALMA flags generated online using *impmode=\\'table\\'* or *\\'xml\\'* and provided in a *FLAG_CMD* or *Flags.xml* table. Using these tables, only the standard on-line REASONs are recognised. These include *\\'FOCUS\\',\\'SUBREFLECTOR\\', \\'OFF SOURCE\\', \\'NOT IN SUBARRAY\\'* for the VLA and \\'*Calibration_device\\_(ACD)\\_is_not_in_the_correct_position\\',**\\'Mount_is_off_source\\', \\'FrontEnd_LO_Amps_not_optimized\\' \\'Power_levels_are_being_optimized.\\', \\'The_WCA_is_not_locked.\\'* for ALMA\n",
"\n",
" - If the plotfile sub-parameter is non-blank, then a plotfile will be made with that name instead of appearing in a matplotlib plotter window on the users workstation. There are additional parameters that control the shape of the output file, such as dimensions, and resolution.\n",
"\n",
"\n",
"**Alert:** The plotted enumerations are currently only those known to be allowed for JVLA online flags as of 15 April 2011, and include:\n",
"\n",
" - *'FOCUS', 'SUBREFLECTOR', 'OFF SOURCE', 'NOT IN SUBARRAY'* with all others being plotted as *'Other*'.\n",
"
\n",
"\n",
"5. Clear flags action='clear'\n",
"\n",
" This is only useful for MS, using *inpmode=\\\"table\\\"*. The 'clear' action will delete selected rows from the FLAG_CMD MS table. This choice opens the sub-parameters:\n",
"\n",
"```\n",
"action = 'clear' #Action to perform in MS and/or in inpfile\n",
" #(apply/unapply/list/plot/clear/extract)\n",
" clearall = False #Delete all rows from FLAG_CMD\n",
"[ rowlist = [] #FLAG_CMD rows to clearThis box is intended for CASA Inputs. Insert your text here.]\n",
"```\n",
"\n",
" - The rowlist sub-parameter is a simple Python list of the row numbers of the table to consider in processing flags. The default is a blank list which indicates the desire to clear all rows. Currently it only takes fully-enumerated integer lists. Use the Python range function to generate ranges,\n",
"\n",
"```\n",
"tablerows = range(0,30) + range(50,55)\n",
"#Do not use strings such as '0~29,50~54'\n",
"```\n",
"\n",
" - In either case, if clearall=False then nothing will happen by default as a safeguard. If clearall=True, then a blank list will direct the deletion of the selected rows from the table.\n",
"\n",
"\n",
"Alert: Use this option with care. You can easily mess up the FLAG_CMD table.\n",
"
\n",
"\n",
"6. Extract Flag Commands action='extract'\n",
"\n",
" This will return the requested flags (depending on *inpmode*) as a Python dictionary.\n",
"\n",
"```\n",
"action = 'extract' #Action to perform in MS and/or in inpfile\n",
" #(apply/unapply/list/plot/clear/extract)\n",
"```\n",
"\n",
" - The dictionary can be saved to a variable such as shown below. If a variable is not set, only the first 1000 flags will be printed to the terminal:\n",
"\n",
"```\n",
"myflagd = flagcmd(vis=msfile,useapplied=True,action='extract')\n",
"```\n",
"\n",
" - An example of the dictionary returned by *action=\\'extract\\'* is given below:\n",
"\n",
"```python\n",
"{0: {'antenna': 'PM04&&*',\n",
" 'applied': False,\n",
" 'command': 'antenna=PM04&&* timerange=2013/04/28/04:35:58.217~2013/04/28/04:35:58.468 ',\n",
" 'id': '662',\n",
" 'interval': 0.0,\n",
" 'level': 0,\n",
" 'mode': '',\n",
" 'reason': 'ACD_motors_are_not_in_position.',\n",
" 'severity': 0,\n",
" 'time': 0.0,\n",
" 'timerange': '2013/04/28/04:35:58.217~2013/04/28/04:35:58.468',\n",
" 'type': ''},\n",
" 1: {'antenna': 'CM03&&*',\n",
" 'applied': False,\n",
" 'command': 'antenna=CM03&&* timerange=2013/04/28/04:35:58.196~2013/04/28/04:35:58.503 ',\n",
" 'id': '663',\n",
" 'interval': 0.0,\n",
" 'level': 0,\n",
" 'mode': '',\n",
" 'reason': 'ACD_motors_are_not_in_position.',\n",
" 'severity': 0,\n",
" 'time': 0.0,\n",
" 'timerange': '2013/04/28/04:35:58.196~2013/04/28/04:35:58.503',\n",
" 'type': ''}}\n",
"```\n",
"\n",
"\n",
"**Flagging command syntax**\n",
"\n",
"A flagging command syntax has been devised to populate the COMMAND column of the *FLAG_CMD* table and to direct the operation of the **flagcmd** task.\n",
"\n",
"The syntax is the same used in **flagdata**, so please check *\\\"help flagdata\\\"* for more updated info.\n",
"\n",
"Commands are a string (which may contain internal \\\"strings\\\") consisting of KEY=VALUE pairs separated by whitespace (see examples below).\n",
"\n",
"\n",
"**Alert:** There should be no whitespace between KEY=VALUE or within each KEY or VALUE, since the simple parser first breaks command lines on whitespace, then on \"=\".\n",
"
\n",
"\n",
"The key is the name of a parameter and the value is the value of that parameter. The parameter data types enforced in **flagdata** are the same used in these flag commands. As an example, the parameter *clipminmax* accepts only a list of double values in **flagdata** and should have the same type when given in a flag command list, e.g. *mode=\\'clip\\' clipminmax=\\[0.1,10.2\\]*\n",
"\n",
"Each key should only appear once on a given command line/string\n",
"\n",
"There is an implicit \\\"mode\\\" for each command, with the default being 'manual' if not given.\n",
"\n",
"Comment lines can start with '\\#' and will be ignored.\n",
"\n",
"1. Data selection parameters (used by all flagging modes) \n",
"\n",
"```\n",
" antenna=''\n",
" spw=''\n",
" correlation=''\n",
" field=''\n",
" scan=''\n",
" feed=''\n",
" array=''\n",
" timerange=''\n",
" uvrange=''\n",
" intent=''\n",
" observation=''\n",
"```\n",
"\n",
"\n",
"**Info:** a command consisting only of meta-data selection key-value pairs is a basic \"manual\" operation, i.e. flag the data meeting the selection\n",
"
\n",
"\n",
"2. Modes with default values for relevant parameters (for further details, refer to the task **flagdata**)\n",
"\n",
" a. Mode manual\n",
"\n",
" ```\n",
" autocorr=False\n",
" ```\n",
"\n",
" b. Mode clip\n",
"\n",
" ```\n",
" datacolumn='DATA'\n",
" clipminmax=[]\n",
" clipoutside=True\n",
" channelavg=False\n",
" clipzeros=False\n",
" timeavg=False\n",
" timebin=''\n",
" ```\n",
"\n",
" c. Mode shadow\n",
"\n",
" ```\n",
" tolerance=0.0\n",
" addantenna=''\n",
" ```\n",
"\n",
" d. Mode quack\n",
"\n",
" ```\n",
" quackinterval=0.0\n",
" quackmode='beg'\n",
" quackincrement=False\n",
" ```\n",
"\n",
" e. Mode elevation\n",
"\n",
" ```\n",
" lowerlimit=0.0\n",
" upperlimit=90.0\n",
" ```\n",
"\n",
" f. Mode tfcrop\n",
"\n",
" ```\n",
" ntime='scan'\n",
" combinescans=False\n",
" datacolumn='DATA'\n",
" timecutoff=4.0\n",
" freqcutoff=3.0\n",
" timefit='line'\n",
" freqfit='poly'\n",
" maxnpieces=7\n",
" flagdimension='freqtime'\n",
" usewindowstats='none'\n",
" halfwin=1\n",
" extendflags=True\n",
" channelavg=False\n",
" chanbin=1\n",
" timeavg=False\n",
" timebin='0s'\n",
" ```\n",
"\n",
" g. Mode extend\n",
"\n",
" ```\n",
" ntime='scan'\n",
" combinescans=False\n",
" extendpols=True\n",
" growtime=50.0\n",
" growfreq=50.0\n",
" growaround=False\n",
" flagneartime=False\n",
" flagnearfreq=False\n",
" ```\n",
"\n",
" h. Mode rflag\n",
"\n",
" ```\n",
" ntime='scan'\n",
" combinescans=False\n",
" datacolumn='DATA'\n",
" winsize=3\n",
" timedev=''\n",
" freqdev=''\n",
" timedevscale=5.0\n",
" freqdevscale=5.0\n",
" spectralmax=1000000.0\n",
" spectralmin=0.0\n",
" extendflags=True\n",
" channelavg=False\n",
" chanbin=1\n",
" timeavg=False\n",
" timebin='0s'\n",
" ```\n",
"\n",
" i. Mode antint\n",
"\n",
" ```\n",
" minchanfrac=0.6\n",
" verbose=False\n",
" ```\n",
"\n",
" j. Mode unflag\n",
"\n",
" This mode does not have any sub-parameters.\n",
"\n",
" \n",
"3. Typical example of a list with flag commands\n",
"\n",
" ```\n",
" spw='0,1'\n",
" antenna='ea1,ea10'\n",
" autocorr=True\n",
" mode='clip'\n",
" clipzeros=True\n",
" datacolumn='DATA'\n",
" mode='summary'\n",
" name='Initial_flags'\n",
" ```\n",
"\n",
"4. Basic elaboration options for online and interface use\n",
"\n",
"```\n",
"id='' #flag ID tag (not necessary)\n",
"reason='' #reason string for flag\n",
"flagtime='' #a timestamp for when this flag was generated (for\n",
"user history use)\n",
"```\n",
"\n",
"\n",
"**Info:** there is no flagtime column in *FLAG_CMD* at this time, but we will propose to add this as an optional column. These are currently ignored and not used\n",
"
\n",
"\n",
"5. Extended elaboration options for online and interface use Note: these are FLAG_CMD columns, but their use is not clear but included here for compatibility and future expansion\n",
"\n",
"```\n",
"level=N #flagging \"level\" for flags with same reason\n",
"severity=N #Severity code for the flag, on a scale of 0-10 in order\n",
"of increasing severity; user specific\n",
"```\n",
"\n",
"\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-OZxaVSR0EDI"
},
"source": [
"## Manage flag versions\n",
"\n",
"Managing flag versions\n",
"\n",
"The **flagmanager** task will allow you to manage different versions of flags in your data. These are stored inside a CASA flag versions table, under the name of the MS \\.flagversions. For example, for the MS jupiter6cm.usecase.ms, there will need to be jupiter6cm.usecase.ms.flagversions on disk. This is created on import (by **importasdm**, **importvla** or **importuvfits**) or when flagging is first done on an MS without a .flagversions.\n",
"\n",
"By default, when the .flagversions is created, this directory will contain a flags.Original table containing a copy of the original flags in the MAIN table of the MS so that you have a backup. It will also contain a file called FLAG_VERSION_LIST that has the information on the various flag versions there. The flag versions are cumulative, i.e. a specific version number contains all the flags from the lower version numbers, too.\n",
"\n",
"The flags are stored in an MS column of the same spectral and correlator polarization shape as the visibility data values, with Boolean 1 to indicate that a particular time, channel, polarization has been flagged or 0 for unflagged.\n",
"\n",
"\n",
"\n",
"The inputs for **flagmanager** are:\n",
"\n",
"```\n",
"vis = '' #Name of input visibility file (MS)\n",
"mode = 'list' #Flag management operation (list,save,restore,delete)\n",
"```\n",
"\n",
"The mode='list' option will list the available flag versions from the \\.flagversions file in the logger. For example:\n",
"\n",
"```\n",
"CASA <102>: default('flagmanager')\n",
"CASA <103>: vis = 'jupiter6cm.usecase.ms'\n",
"CASA <104>: mode = 'list'\n",
"CASA <105>: flagmanager()\n",
"MS : /home/imager-b/smyers/Oct07/jupiter6cm.usecase.ms\n",
"\n",
"main : working copy in main table\n",
"Original : Original flags at import into CASA\n",
"flagautocorr : flagged autocorr\n",
"flagdata_1 : Flags autosave on 2018-07-16 08:57:20]\n",
"```\n",
"\n",
"*mode=\\'list\\'* will also return a Python dictionary with the available flag versions. For example:\n",
"\n",
"```\n",
"myflags = flagmanager('jupiter6cm.usecase.ms', mode='list')\n",
"\n",
"myflags\n",
"\n",
"{0: {'comment': 'Original flags at import into CASA', 'name': 'Original'},1: {'comment': 'flagged autocorr', 'name': 'flagautocorr'},2: {'comment': 'Flags autosave on 2018-07-16 08:57:20', 'name': 'flagdata_1'},'MS': '/home/imager-b/smyers/Oct07/jupiter6cm.usecase.ms'}}\n",
"```\n",
"\n",
"The mode parameter expands the options. For example, if you wish to save the current flagging state of vis=\\:\n",
"\n",
"```\n",
"mode = 'save' #Flag management operation (list, save, restore, delete)\n",
"versionname = '' #Name of flag version (no spaces)\n",
"comment = '' #Short description of flag version\n",
"merge = 'replace' #Merge option (replace, and, or)\n",
"```\n",
"\n",
"with the output version name specified by versionname. For example, the above xyflags version was written using:\n",
"\n",
"```\n",
"default('flagmanager')\n",
"vis = 'jupiter6cm.usecase.ms'\n",
"mode = 'save'\n",
"versionname = 'xyflags'\n",
"comment = 'Plotxy flags'\n",
"flagmanager()\n",
"```\n",
"\n",
"and you can see that there is now a sub-table in the flag versions directory:\n",
"\n",
"```\n",
"CASA <106>: ls jupiter6cm.usecase.ms.flagversions/\n",
" IPython system call: ls -F jupiter6cm.usecase.ms.flagversions/\n",
" flags.flagautocorr flags.Original flags.xyflags FLAG_VERSION_LIST\n",
"```\n",
"\n",
"It is recommended that you use this task regularly to save versions during flagging.\n",
"\n",
"Note that if a flag version already exists under a name, the task will give a warning and add a suffix '.old.timestamp' to the previous version.\n",
"\n",
"You can restore a previously saved set of flags using the mode='restore' option:\n",
"\n",
"```\n",
"mode = 'restore' #Flag management operation (list,save,restore,delete)\n",
"versionname = '' #Name of flag version (no spaces)\n",
"merge = 'replace' #Merge option (replace, and, or)\n",
"```\n",
"\n",
"The merge sub-parameter will control how the flags are restored. For merge='replace', the flags in versionname will replace those in the MAIN table of the MS. For merge='and', only data that is flagged in BOTH the current MAIN table and in versionname will be flagged. For merge='or', data flagged in EITHER the MAIN or in versionname will be flagged.\n",
"\n",
"The mode='delete' option can be used to remove versionname from the flag versions:\n",
"\n",
"```\n",
"mode = 'delete' #Flag management operation (list,save,restore,delete)\n",
"versionname = '' #Name of flag version (no spaces)\n",
"```\n",
"\n",
"\n",
"\n",
"***\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6Xb4zSZF0EDJ"
},
"source": [
"## 2-D Plot/Flag: msview\n",
"\n",
"2-dimensional visualization and editing of visibilities with the msview GUI.\n",
"\n",
"\n",
"**Viewing MeasurementSets**\n",
"\n",
"\n",
"\n",
"Visibility data can also be displayed and flagged directly from the task **msview**. Task msview allows the user to select data before it is loaded into the GUI and displayed. A screenshot is shown in Data Selection in [msview.](#FigMSView) Selection parameters are *field, spectral window, time range, uv range, antenna, corr, scan, array, and ms selection expression* in the usual CASA selection syntax (see [Data Selection](visibility_data_selection.ipynb#Visibility-Data-Selection)).]\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"For MeasurementSet files the only option for display is 'Raster' (similar to AIPS task TVFLG). An example of MS display is shown in [Data Display Options](#FigDataDisplayOptions); loading of an MS is shown in [Loading a MeasurementSet](#FigLoadMS).\n",
"\n",
"\n",
"Warning: *Only one MS should be registered at a time on a Display Panel.* Only one MS can be shown in any case. You do not have to close other images/MSs, but you should at least 'unregister' them from the Display Panel used for viewing the MS. If you wish to see other images or MSs at the same time, create multiple Display Panel windows.\n",
"
\n",
"\n",
"\n",
"\n",
"**Data Display Options Panel for MeasurementSets**\n",
"\n",
"\n",
"The Data Display Options panel provides adjustments for MSs similar to those for images, and also includes flagging options. As with images, this window appears when you choose the Data\\\\:Adjust menu or use the wrench icon from the Main Toolbar. It is also shown by default when an MS is loaded. The right panel of [Data Display Options](#FigDataDisplayOptions) shows a Data Options window. It has a tab for each open MS, containing a set of categories. The options within each category can be either 'rolled up' or expanded by clicking the category label.\n",
"\n",
"For a MeasurementSet, the categories are:\n",
"\n",
"- Advanced\n",
"- MS and Visibility Selection\n",
"- Display Axes\n",
"- Flagging Options\n",
"- Basic Settings\n",
"- Axis Drawing and Labels\n",
"- Color Wedge\n",
"\n",
"\n",
"**MS Options --- Basic Settings**\n",
"\n",
"The Basic Settings roll-up is expanded by default. It contains entries similar to those for a raster image, [Data Display Options - Basic Settings](#viewerDataDisplayOptionsBasic)). Together with the brightness/contrast and colormap adjustment icons on the Mouse Toolbar of the Display Panel, they are especially important for adjusting the color display of your MS.\n",
"\n",
"The available Basic options are:\n",
"\n",
"- Data minimum/maximum\n",
"\n",
" This has the same usage as for raster images. Lowering the data maximum will help brighten weaker data values.\n",
"\n",
"- Scaling power cycles\n",
"\n",
" This has exactly the same usage as for raster images (see [Data Display Options - Basic Settings](#viewerDataDisplayOptionsBasic)). Again, lowering this value often helps make weaker data visible. If you want to view several fields with very different amplitudes simultaneously, this is typically one of the best adjustments to make early, together with the Colormap fiddling mouse tool, which is on the middle mouse button by default.\n",
"\n",
"- Colormap\n",
"\n",
" Greyscale or Hot Metal colormaps are generally good choices for MS data.\n",
"\n",
"\n",
"**MS Options--- MS and Visibility Selections**\n",
"\n",
"- Visibility Type\n",
"- Visibility Component\n",
"- Moving Average Size\n",
"\n",
"This roll-up provides choice boxes for Visibility Type (Observed, Corrected, Model, Residual) and Component (Amplitude, Phase, Real, or Imaginary).\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"Changes to Visibility Type or Component (changing from Phase to Amplitude, for example) require the data to be retrieved again from the disk into memory, which can be a lengthy process. When a large MS is first selected for viewing, the user must trigger this retrieval manually by pressing the Apply button (located below all the options), after selecting the data to be viewed (see Field IDs and Spectral Windows, below).\n",
"\n",
"Tip: Changing visibility type between 'Observed' and 'Corrected' can also be used to assure that data and flags are reloaded from disk. You should do this if you're using another flagging tool such as autoflag simultaneously, so that msview sees the other tool's new edits and doesn't overwrite them with obsolete flags. The Apply button alone won't reload unless something within msview itself requires it; in the future, a button will be provided to reload flags from the disk unconditionally.\n",
"\n",
"You can also choose to view the difference from a running mean or the local RMS deviation of either Phase or Amplitude. There is a slider for choosing the nominal number of time slots in the 'local neighborhood' for these displays.\n",
"\n",
"\n",
"(Note: Insufficient Data is shown in the tracking area during these displays when there is no other unflagged data in the local neighborhood to compare to the point in question. The moving time windows will not extend across changes in either field ID or scan number boundaries, so you may see this message if your scan numbers change with every time stamp. An option will be added later to ignore scan boundaries).\n",
"
\n",
"\n",
"- Field IDs\n",
"- Spectral Windows\n",
"\n",
"You can retrieve and edit a selected portion of the MS data by entering the desired Spectral Window and Field ID numbers into these boxes.\n",
"\n",
"\n",
"Important: Especially with large MSs, often the first thing you'll want to do is to select spectral windows which all have the same number of channels and the same polarization setup. It also makes sense to edit only a few fields at a time. Doing this will also greatly reduce data retrieval times and memory requirements.\n",
"
\n",
"\n",
"You can separate the ID numbers with spaces or commas; you do not need to enter enclosing brackets. Changes to either entry box will cause the selected MS data to be reloaded from disk.\n",
"\n",
"If you select, say, spectral windows 7, 8, 23, and 24, the animator, slice position sliders, and axis labeling will show these as 0, 1, 2, and 3 (the 'slice positions' or 'pixel coordinates' of the chosen spectral windows). Looking at the position tracking display is the best way to avoid confusion in such cases. It will show something like: Sp Win 23 (s 2) when you are viewing spectral window 23 (plane 2 of the selected spectral windows).\n",
"\n",
"Changes to MS selections will not be allowed until you have saved (or discarded) any previous edits you have made (see Flagging Options \\-- Save Edits, below). A warning is printed on the console (not the logger).\n",
"\n",
"Initially, all fields and spectral windows are selected. To revert to this 'unselected' state, choose 'Original' under the wrench icons next to the entry boxes.\n",
"\n",
"See [MeasurementSet Visibility Selections](#FigMSVisibilitySelection) for an example showing the use of the MS and Visibility Selections controls when viewing an MS.\n",
"\n",
"\n",
"\n",
"**MS Options --- Display Axes**\n",
"\n",
"This roll-up is very similar to that for images: it allows the user to choose which axes (from Time, Baseline, Polarization, Channel, and Spectral Window) are on the display and the animator. There are also sliders here for choosing positions on the remaining axes. (It's useful to note that the data is actually stored internally in memory as an array with these five axes).\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"Within the Display Axes rollup you may also select whether to order the baseline axis by antenna1-antenna2 (the default) or by (unprojected) baseline length.\n",
"\n",
"See [MeasurementSet Display Axes](#FigMSDisplayAxes)--[Changing the Axis of a MeasurementSet](#FigMSChangedAxis) showing the use of the Display Axes controls to change the axes on the animation and sliders.\n",
"\n",
"\n",
"\n",
"**MS Options --- Flagging Options**\n",
"\n",
"These options allow you to edit (flag or unflag) MS data. The Point Tool and Rectangle Region Mouse Tools are used on the display to select the area to edit. When using the Rectangle Region tool, double-click inside the selected rectangle to confirm the edit.\n",
"\n",
"The options below determine how edits will be applied.\n",
"\n",
"- Show Flagged Regions\\...\n",
"\n",
" You have the option to display flagged regions in the background color (as in TVFLG) or to highlight them with color. In the former case, flagged regions look just like regions of no data. With the (default) color option, flags are shown in shades of blue: darker blue for flags already saved to disk, lighter blue for new flags not yet saved; regions with no data will be shown in black.\n",
"\n",
"- Flag or Unflag\n",
"\n",
" This setting determines whether selected regions will be flagged or unflagged. This does not affect previous edits; it only determines the effect which later edits will have. Both flagging and unflagging edits can be accumulated and then saved in one pass through the MS.\n",
"\n",
"- Flag/Unflag All\\...\n",
"\n",
" These flagging extent checkboxes allow you to extend your edit over any of the five data axes. For example, to flag all the data in a given time range, you would check all the axes except Time, and then select the desired time range with the Rectangle Region mouse tool. Such edits will extend along the corresponding axes over the entire selected MS (whether loaded into memory or not) and optionally over unselected portions of the MS as well (Use Entire MS, below). Use care in selecting edit extents to assure that you're editing all the data you wish to edit.\n",
"\n",
"- Flag/Unflag Entire Antenna?\n",
"\n",
" This control can be used to extend subsequent edits to all baselines which include the desired antenna\\[s\\]. For example, if you set this item to 'Yes' and then click the point tool on a visibility position with baseline 3-19, the edit would extend over baselines 0-3, 1-3, 2-3, 3-3, 3-4, \\... 3-]nAntennas-1. Note that the second antenna of the selection (19) is irrelevant here -- you can click anywhere within the 'Antenna 3 block', i.e., where the *first* antenna number is 3, to select all baselines which include antenna 3.\n",
"\n",
" This item controls the edit extent only along the baseline axis. If you wish to flag all the data for a given antenna, you must still check the boxes to flag all Times, Channels, Polarizations and Spectral Windows. There would be no point, however, in activating both this item and the 'Flag All Baselines' checkbox. You can flag an antenna in a limited range of times, etc., by using the appropriate checkboxes and selecting a rectangular region of visibilities with the mouse. \n",
"\n",
"\n",
"Note: You do not need to include the entire 'antenna block' in your rectangle (and you may stray into the next antenna if you try). Anywhere within the block will work. To flag higher-numbered antennas, it often helps to zoom in.\n",
"
\n",
"\n",
"- Undo Last Edit\n",
"\n",
"- Undo All Edits\n",
"\n",
" The 'Undo' buttons do the expected thing: completely undo the effect of the last edit (or all unsaved edits). Please note, however, that only unsaved edits can be undone here; there is no ability to revert to the flagging state at the start of the session once flags have been saved to disk (unless you have previously saved a 'flag version'. The flag version tool is not available through msview directly).\n",
"\n",
"- Use Entire MS When Saving Edits?\n",
"\n",
" \\\"Yes\\\" means that saving the edits will flag/unflag over the entire MS, including fields (and possibly spectral windows) which are not currently selected for viewing. Specifically, data within time range(s) you swept out with the mouse (even for unselected fields) will be edited.\n",
"\n",
" In addition, if \\\"Flag/Unflag All\\...\\\" boxes were checked, such edits will extend throughout the MS. Note that only unselected times (fields) can be edited without checking extent boxes for the edits as well. Unselected spectral windows, e.g., will not be edited unless the edit also has \\\"Flag/Unflag All Spectral Windows\\\" checked. \n",
" \n",
"\n",
"Warning: Beware of checking \"All Spectral Windows\" unless you have also checked \"All Channels\" or turned \"Entire MS\" off; channel edits appropriate to the selected spectral windows may not be appropriate to unselected ones. Set \"Use Entire MS\" to \"No\" if your edits need to apply only to the portion of the MS you have selected for viewing. Edits can often be saved significantly faster this way as well. Also note that checkboxes apply to individual edits, and must be checked before making the edit with the mouse. \"Use Entire MS\", on the other hand, applies to all the edits saved at one time, and must be set as desired before pressing \"Save Edits\".\n",
"
\n",
"\n",
"- Save Edits\n",
"\n",
" MS editing works like a text editor in that you see all of your edits immediately, but nothing is committed to disk until you press \"Save Edits\". Feel free to experiment with all the other controls; nothing but 'Save Edits' will alter your MS on disk. As mentioned previously, however, there is no way to undo your edits once they are saved, except by manually entering the reverse edits (or restoring a previously-saved 'flag version').\n",
"\n",
" Also, you must save (or discard) your edits before changing the MS selections. If edits are pending, the selection change will not be allowed, and a warning will appear on the console.\n",
"\n",
" If you close the MS in msview, unsaved edits are simply discarded, without prior warning. It's important, therefore, to remember to save them yourself. You can distinguish unsaved flags (when using the 'Flags In Color' option), because they are in a lighter shade of blue.\n",
"\n",
" The program must make a pass through the MS on disk to save the edits. This can take a little time; progress is shown in the console window.\n",
"\n",
"\n",
"\n",
"**MS Options--- Advanced**\n",
"\n",
"These settings can help optimize your memory usage, especially for large MSs. A rule of thumb is that they can be increased until response becomes sluggish, when they should be backed down again.\n",
"\n",
"You can run the unix 'top' program and hit 'M' in it (to sort by memory usage) in order to examine the effects of these settings. Look at the amount of RSS (main memory) and SWAP used by the X server and 'casaviewer' processes. If that sounds familiar and easy, then fiddling with these settings is for you. Otherwise, the default settings should provide reasonable performance in most cases.\n",
"\n",
"- Cache size\n",
"\n",
" The value of this option specifies the maximum number of different views of the data to save so that they can be redrawn quickly. If you run an animation or scroll around zoomed data, you will notice that the data displays noticeably faster the second time through because of this feature. Often, setting this value to the number of animation frames is ideal Note, however, that on multi-panel displays, each panel counts as one cached image.\n",
"\n",
" Large images naturally take more room than small ones. The memory used for these images will show up in the X server process. If you need more Visibility Memory (below) for a really large ms, it is usually better to forgo caching a large number of views.\n",
"\n",
"- Max. Visibility Memory\n",
"\n",
" This option specifies how many megabytes of memory may be used to store visibility data from the MeasurementSet internally. Even if you do not adjust this entry, it is useful to look at it to see how many megabytes are required to store your entire (selected) MS in memory. If the slider setting is above this, the whole selected MS will fit into the memory buffer. Otherwise, some data planes will be 'grayed out' (see [MS Options - Apply Button](#viewerApplyButton)), and the selected data will have to be viewed one buffer at a time, which is somewhat less convenient. In most cases, this means you should select fewer fields or spectral windows -- see [MS Options - MS and Visibility Selections](#viewerMSVisibilitySelection). The 'casaviewer' process contains this buffer memory, which can take most of the space.\n",
"\n",
"\n",
"\n",
"**MS Options --- Apply Button**\n",
"\n",
"When viewing large MSs the display may be partially or completely grey in areas where the required data is not currently in memory, either because no data has been loaded yet, or because not all the selected data will fit into the allowed memory (see Max. Visibility Memory above). When the cursor is over such an area, the following message shows in the position tracking area:\n",
"\n",
"```\n",
" press 'Apply' on Adjust panel to load data\n",
"```\n",
"\n",
"Pressing the Apply button (which lies below all the options) will reload the memory buffer so that it includes the slice you are trying to view.\n",
"\n",
"The message No Data has a different meaning; in that case, there simply is no data in the selected MS at the indicated position.\n",
"\n",
"For large MeasurementSets, loading visibility data into memory is the most time-consuming step. Progress feedback is provided in the console window. Again, careful selection of the data to be viewed can greatly speed up retrieval.\n",
"\n",
"\n",
"\n",
"\n"
]
}
]
}