Open in Colab: https://colab.research.google.com/github/casangi/casadocs/blob/2348961/docs/notebooks/external-data.ipynb


External Data

Each CASA distribution comes with a minimal repository of binary data that is required for CASA to function properly. This is contained in the casadata repository and bundled in to a casadata package for modular CASA. The repository includes Measures Tables that deal with the Earth Orientation Parameters (EOPs), reference frames, as well as ephemeris data. In particular the EOPs include predictions for the near future which drift until they are well determined.

Casacore Measures

The casacore Measures tables are needed to perform accurate conversions of reference frames. Casacore infrastructure includes classes to handle physical quantities with a reference frame, so-called Measures. Each type of Measure has its own distinct class in casacore which is derived from the Measure base class. One of the main functionalilties provided by casacore w.r.t. Measures, is the conversion of Measures from one reference frame to another using the MeasConvert classes.

Many of the spectral, spatial, and time reference frames are time-dependent and require the knowledge of the outcome of ongoing monitoring measurements of properties of the Earth and astronomical objects by certain service observatories. This data is tabulated in a number of tables (Measures Tables) which are stored in the casadata repository in the subdirectory geodetic. A snapshot of this repository is included in each tarball distribution of CASA and in the casadata module for CASA6+.

Measures tables are updated daily based on the refinement of the geodetic information from the relevant services like the International Earth Rotation and Reference Systems Service (IERS). Strictly speaking, the Measures tables are part of the casacore infrastructure which is developed by NRAO, ESO, NAOJ, CSIRO, and ASTRON. In order to keep the repository consistent between the partners, the Measures tables are initially created at a single institution (ASTRON) and then copied into the NRAO casadata repository from which all CASA users can retrieve them. As of March 2020, the update of the NRAO CASA copy of the Measures tables in geodetic and the planetary ephemerides in directory ephemerides takes place every day between 18 h UTC and 19 h UTC via two redundant servers at ESO (Garching).

The following list describes the individual Tables in subdirectory geodetic:

  • IERSeop2000: The IERS EOP2000C04_05 Earth Orientation Parameters using the precession/nutation model “IAU2000” (files eopc04_IAU2000.xx)

  • IERSeop97: The IERS EOPC04_05 Earth Orientation Parameters using the precession/nutation model “IAU 1980” (files eopc04.xx)

  • IERSpredict: IERS Earth Orientation Data predicted from NEOS (from file ftp://ftp.iers.org/products/eop/rapid/daily/finals.daily)

  • IGRF: International Geomagnetic Reference Field Schmidt semi-normalised spherical harmonic coefficients. (Note that this still uses IGRF12. An update to IGRF13 is underway.)

  • IMF (not a Measures Table proper, access not integreated in Measures framework): Historical interplanetary magnetic field data until MJD 52618 (December 2002).

  • KpApF107 (not a Measures Table proper, access not integreated in Measures framework): Historical geomagnetic and solar activity indices until MJD 54921 (April 2009)

  • Observatories: Table of the official locations of radio observatories. Maintained by the CASA consortium.

  • SCHED_locations (not a Measures Table proper, access not integreated in Measures framework): VLBI station locations

  • TAI_UTC: TAI_UTC difference (i.e. leap second information) obtained from USNO

Measures Tables in the directory ephemerides:

Ephemeris Data

The ephemeris tables hold a selection of the solar system objects from JPL-Horizons database. The data tables are generated from the JPL Horizons system’s on-line solar system data and ephemeris computation service (https://ssd.jpl.nasa.gov/?horizons ). These are primarily used to determine flux models for the solar system objects used in the setjy task. These tables are stored as CASA tables in the casadata repository under ephemerides/JPL-Horizons. The current ephemeris tables cover ephemerides until December 31, 2030 for those objects officially supported in setjy.

Available objects, which include major planets, satellites, and asteroids, are: Mercury, Venus, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto, Io, Europa, Ganymede, Callisto, Titan, Ceres, Vesta, Pallas, Juno, Lutetia, Sun and Moon (the objects in bold are those supported in ‘Butler-JPL-Horizons 2012’ standard in setjy.).

The format of the table name of these tables is objectname_startMJD_endMJD_J2000.tab These tables required by setjy task are included in the data directory in the CASA distribution. The available tables can be listed by the following commands:

#In CASA6

CASA <1>: import glob

CASA <2>: jpldatapath=os.getenv('CASAPATH').split(' ')[0]+'/data/ephemerides/JPL-Horizons/*J2000.tab'

CASA <3>: glob.glob(jpldatapath)

The following data are retrieved from the JPL-Horizons system (the nubmer in the parentheses indicates the column number listed in the JPL-Horizons system). One should refer https://ssd.jpl.nasa.gov/?horizons_doc for the detailed descreption of each of these quantities.

Quantities

column no.

Unit/format

Descrition

column label

Date

n.a.

YYYY-MM-DD

HH:MM

Date__(UT)__HR:MN

Astrometric RA & DEC

1

degrees

Astrometric RA and Dec with respect to the observer’s location (GEOCETRIC)

R.A._(ICRF)_DEC

Observer sub-long& sub-lat

14

degrees

Apparent planetodetic (“geodetic”) longitude and latitude of the center of the target seen by the OBSERVER at print-time

ob-lon, ob-lat

Solar sub-long & sub-lat

15

degrees

Apparent planetodetic (“geodetic”) longitude and latitude of the Sun seen by the OBSERVER at print-time

Sl-lon, Sl-lat

North Pole Pos. ang. & dist.

17

degrees and arcseconds

Target’s North Pole position angle and angular distance from the “sub-observer” point

NP.ang, NP.ds

Helio range & range rate

19

AU, km/s

Heliocentric range (r) and range-rate

(rdot)

Observer range & range rate

20

AU, km/s

Range (delta) and range-rate (deldot) of the target center with respect to the observer

delta, dedot

S-T-O angle

24

degrees

Sun-Target-Observer angle

S-T-O

The script request.py (located in casatasks.private for CASA6) can be used to retrieve the ephemeris data from the JPL-Horizons system via e-mail (See also Manipulate Ephemeris Objects page). Further, the saved email text file is converted to a CASA table format using JPLephem_reader2.

#In CASA6

CASA <5>: from casatasks.private import JPLephem_reader2 as jplreader

CASA <6>: outdict = jplreader.readJPLephem('titan-jpl-horizons-ephem.eml')
opened the file=titan-jpl-horizons-ephem.eml

CASA <7>: jplreader.ephem_dict_to_table(outdict,'Titan_test_ephem.tab')
Got nrows = 3653 from MJD

The converted table contains following columns.

Column name

unit/format

description

MJD

day

modified Julian date

RA

degree

atrometric right acension in ICRF/J2000 frame

DEC

degree

astrometric declination in ICRF/J2000 frame

Rho

AU

Geocentric distance

RadVal

AU/d

Geocentric distance rate

NP_ang

degree

North pole position angle

NP_dist

degree

North pole angular distance

DiskLong

degree

Sub-observer longitude

DiskLat

degree

Sub-observer latitude

Sl_lon

degree

Sub-Solar longitude

Sl_lat

degree

Sub-Solar latitude

r

AU

heliocentric distance

rdot

km/s

heliocentric distance rate

phang

degree

phase angle

Array Configuration

Array configuration files for various telescopes are distributed with each CASA release. These configuration files can be used to define the telescope for simulator tools and tasks. Currently, configuration files for the following telescopes are available in CASA:

  • ALMA / 12m Array

  • ALMA / 7m ACA

  • VLA

  • VLBA

  • Next-Generation VLA (reference design)

  • ATCA

  • MeerKat

  • PdBI (pre-NOEMA)

  • WSRT

  • SMA

  • Carma

The full list of antenna configurations can be found in the CASA Guides on Simulations.

One can also locate the directory with the configurations in the CASA distribution and then list the configuration files, using the following commands in CASA:

CASA <1>: print os.getenv('CASAPATH').split(' ')[0] + '/data/alma/simmos/'
/home/casa/packages/RHEL7/release/casa-release-5.4.0-68/data/alma/simmos/

CASA <2>: ls /home/casa/packages/RHEL6/release/casa-release-5.4.0-68/data/alma/simmos/

If a configuration file is not distributed with CASA but retrieved elsewhere, then the configuration file can be called by explicitly writing the full path to the location of the configuration file in the antennalist paramter of the simulator tasks.

NOTE: the most recent ALMA configuration files may not always be available in the most recent CASA version. ALMA configuration files for all cycles are available for download here. For the Next-Generation VLA reference design, the latest information can be found here.

Locating the Data Tables

The casadata package with all necessary runtime data tables is included in each tarball distribution of monolithic CASA. For modular CASA6, the casadata package is pip installed as one of the modules and located alongside other python packages in the python environment. Therefore the default location of the casadata tables depends on the type of CASA installation as follows:

  • Modular CASA 6 : located inside the venv created during installation

venv/lib/python3.6/site-packages/casadata/__data__
  • Monolithic CASA 6 : located inside the tar file bundle on Linux RedHat or CASA.app on Mac OSX

casa-6.x.y-z/data (Linux)
or
CASA.app/Contents/data (Mac)

A user-specified location for the casadata tables may be set in config.py using the --datapath option. See the CASA API for more information.

Note that for when building from source code, the repository snapshot is taken from .casa/config.py using datapath=[‘/some-directory/’]. More developer information on the Data Repository for build versions of CASA can be found in this Knowledgebase Article.

Updating the Data Tables

Some data tables (such as Measures) are regularly updated from the originating sources in the casadata repository at NRAO. Other data updated less frequently is also stored in the repository, such as beam models, antenna and Jy/K correction tables, and the antenna configuration files for the CASA simulator. Many tasks such as tclean, simobserve, and calibration tasks, need an up-to-date data repository to work properly. Because of this, the casadata package within the CASA distribution must be updated occasionally as well.

For observatory use, the update period should not be longer than weekly in order to have the EOPs up-to-date for upcoming observations. The shortest reasonable update interval is daily and the recommended method is via rsync. For offline data analysis use, the update period should not be longer than monthly. Weekly update is recommended. Typically, the administrator of a CASA installation sets up a cron job to perform the update automatically.

Legacy installations processing old data do not have to be updated because the relevant contents of the Measures Tables is not changing anymore for the more distant past.

Note that currently the casadata package is only updated weekly. Users needing daily updates should use the custom location, repository copy, or rsync methods.

When using a shared site-wide installation of CASA, a system administrator may be needed

Updating the default location

The default casadata location can be updated to the latest version of the data tables using the following methods. An additional command to view the current version of the data tables is included for reference. This method is limited to weekly updates.

  • Modular CASA 6 : from the terminal with active venv

(venv) bash$ pip install --upgrade --extra-index-url https://go.nrao.edu/pypi casadata
(venv) bash$ pip list | grep casadata
  • Monolithic CASA 6 : from within a running CASA shell

CASA <1>: !update-data
CASA <2>: import importlib_metadata
CASA <3>: importlib_metadata.version('casadata')

or from the terminal:

bash$ cd <release top directory>

bash$ bin/pip3 install --upgrade --extra-index-url https://go.nrao.edu/pypi casadata
bash$ bin/pip3 list | grep casadata  (for Linux)
or
bash$ Contents/MacOS/pip3 install --upgrade --extra-index-url https://go.nrao.edu/pypi casadata
bash$ Contents/MacOS/pip3 list | grep casadata  (for Mac)

where <release top directory> is the top directory of the tar file (i.e., casa-6.x.y-z for Linux RedHat and CASA.app for Mac OSX).

Warning

Temporary scratch space is used to hold the intermediate download artifacts. If you have limited storage or a disk quota, you must ensure at least 2GB of free space. Otherwise you must specify a TMP path to another location by executing the following:

bash$ mkdir /bigdisk/tmp
bash$ export TMPDIR=/bigdisk/tmp

Updating a custom location

If users cannot update casadata within a shared site-wide installation of CASA, or wish to have different versions of casadata installed concurrently, a custom location(s) outside of the CASA distribution can be specified and updated.

Custom locations are specified in the config.py file in the users home ~/.casa directory using the rundata option. See the CASA API for more information. Alternatively, a specific location can be supplied to the update command.

  • Modular CASA 6 : from the terminal with active venv

(venv) bash$ python -m casatools --update-user-data
(venv) bash$ python -m casatools --update-user-data=/tmp/mydata
  • Monolithic CASA 6 : from the terminal

bash$ cd <release top directory>
bash$ bin/python3 -m casatools --update-user-data (Linux)
or
bash$ Contents/MacOS/python3 -m casatools --update-user-data (Mac)
bash$ bin/python3 -m casatools --update-user-data=/tmp/mydata (Linux)
or
bash$ Contents/MacOS/python3 -m casatools --update-user-data=/tmp/mydata (Mac)

This method pulls directly from the underlying casadata repository, not the package, and as such provides daily updates.

Updating via repository copy

The standard method of updating the default location of the casadata package is currently limited to weekly changes. The underlying casadata repository is updated daily, but the overhead of creating and publishing packages built from the repository currently limits the cadence to weekly.

Updating to a custom location triggers a direct copy from the casadata repository, and thus allows for more frequent daily updates. In situations where the default location needs daily updates, the custom location method may be used with a specified destination of the default location.

Note that if this is done the version number of the casadata package will no longer be useful for understanding which state of the runtime data is being used

  • Modular CASA 6 : from the terminal with active venv

(venv) bash$ cd <venv top directory>
(venv) bash$ python -m casatools --update-user-data=`find lib -name __data__`
  • Monolithic CASA 6 : from the terminal

bash$ cd <release top directory>
bash$ bin/python3 -m casatools --update-user-data=`find lib -name __data__` (Linux)
or
bash$ Contents/MacOS/python3 -m casatools --update-user-data='find lib -name __data__' (Mac)

Updating via rsync

The previous methods are intended to update the minimum set of runtime data necessary for CASA execution. The runtime data is a subset of a larger casadata repository. Users wishing to access the entire repository (or who prefer this method to the previous) may use rsync to pull update casadata from an NRAO rsync server.

This method also allows for daily updates.

Note that if this is done the version number of the casadata package will no longer be useful for understanding which state of the runtime data is being used

  • Modular and Monolithic CASA : from the terminal

rsync -avz rsync://casa-rsync.nrao.edu/casa-data <location of casadata tables>

This data repository contains runtime data, reference data and test data. However, the CASA project is currently (Fall 2020) working to better organize and partition this data so these instructions will probably stop working in the future.