Subject: Start of the FITS-IDI Convention Comment Period From: "Pence, William D. (GSFC-6601)" Date: Mon, 1 Jun 2009 13:53:07 -0500 To: FITSBITS This is to announce the start of the 30-day Public Comment Period on the FITS Interferometry Data Interchange (FITS-IDI) Convention for the interchange of data recorded by interferometric telescopes, particularly at radio frequencies and very long baselines. This is the 16th in a growing series of conventions submitted to the Registry which is maintained by the IAU FITS Working Group. Detailed information about this convention and a sample FITS file that uses it are available for public review and comment from the FITS registry web page at http://fits.gsfc.nasa.gov/fits_registry.html Comments about this convention may be posted here on the FITSBITS mail exploder or the mirrored sci.astro.fits newsgroup. Minor typographical issues may be sent directly to the authors of the convention. Bill Pence (on behalf of the IAU FITS Working Group) ================================================================= Subject: Re: [fitsbits] Start of the FITS-IDI Convention Comment Period From: Harro Verkouter Date: Tue, 2 Jun 2009 03:51:41 -0500 To: FITSBITS Hi all, After reading through the document I'd like to make a few comments. Some of those apply to the implementation of the/a reader, nonetheless they seem to indicate incompatibility between documentation and practical use-cases. This is written from the perspective of the EVN FITS-IDI data-export software, which has been implemented exactly following the Flatters(1999) document [and thereby finding discrepancies between documentation and implementation]. * I have noticed that in most cases the order in which the keywords are written (also the non-mandatory FITS keywords) may have to be identical as in the documentation or else your FITS-IDI file may be rejected. I do not know if this situation has changed over the last few years, I did not experiment with writing keywords "out-of-order" since finding out that it most likely 'breaks' your FITS-IDI file. * The "SORT" optional keyword to the UV_TABLE header (p.15) does not seem to get honoured. Our FITS-writer writes the hard-coded value "TB" (Time-Baseline) since that is the order in which the data is forcibly written, nonetheless it is never recognized by the reader, making the user have to run uvsort (or wossname of the task) to make the data recognized as being sorted in TB order. It may well be that this behaviour is triggered by our FITS-writer doing something else wrong however it is not documented - making it quite unfixable yet. * I'm glad to see the POLARX/Y situation being somewhat resolved. I beg to differ, however, from the statement in the document "The units were changed from the meters specified by the earlier documents, but seldom used in actual implementations.". Especially since it is exactly this early misdocumentation (documentation says "meters", s/w interprets them as "arcseconds") that has caused many of our users headaches (EVN data). Cheers, harro verkouter ================================================================= Subject: Re: [fitsbits] Start of the FITS-IDI Convention Comment Period From: Eric Greisen Date: Mon, 15 Jun 2009 16:00:08 -0500 To: FITSBITS Harro Verkouter wrote: > > Hi all, > > > > After reading through the document I'd like to make a few comments. > > Some of those apply to the implementation of the/a reader, nonetheless > > they seem to indicate incompatibility between documentation and > > practical use-cases. This is written from the perspective of the EVN > > FITS-IDI data-export software, which has been implemented exactly > > following the Flatters(1999) document [and thereby finding > > discrepancies between documentation and implementation]. > > > > * I have noticed that in most cases the order in which the keywords > > are written (also the non-mandatory FITS keywords) may have to be > > identical as in the documentation or else your FITS-IDI file may be > > rejected. I do not know if this situation has changed over the last > > few years, I did not experiment with writing keywords "out-of-order" > > since finding out that it most likely 'breaks' your FITS-IDI file. > > > > * The "SORT" optional keyword to the UV_TABLE header (p.15) does not > > seem to get honoured. Our FITS-writer writes the hard-coded value > > "TB" (Time-Baseline) since that is the order in which the data is > > forcibly written, nonetheless it is never recognized by the reader, > > making the user have to run uvsort (or wossname of the task) to make > > the data recognized as being sorted in TB order. It may well be that > > this behaviour is triggered by our FITS-writer doing something else > > wrong however it is not documented - making it quite unfixable yet. > > > > * I'm glad to see the POLARX/Y situation being somewhat resolved. I > > beg to differ, however, from the statement in the document "The units > > were changed from the meters specified by the earlier documents, but > > seldom used in actual implementations.". Especially since it is > > exactly this early misdocumentation (documentation says "meters", s/w > > interprets them as "arcseconds") that has caused many of our users > > headaches (EVN data). I think all of these comments apply to the implementation of the AIPS task FITLD rather than to the IDI convention itself. I hope that I have fixed the earlier versions of FITLD so that the columns may now occur in any order. I just tested it and the SORT values are honored - but note that 'T*' is changed by FITLD to 'TB' or to '**' if unsorted data are found. Leonia Kogan has studied the POLARX/Y question and determined that the software can, for data within the last 30 years, tell which units were used. AIPS has been corrected to make that test and enforce the new (and usual) convention. A question was raised outside this forum about adding PULSAR BIN as an optional UV data axis. I have not added that to the document since its support would force much labor on my part for aips and other software people as well. Until there are users who promise to use it, I am inclined to stay away from that addition. Cheers, Eric Greisen ================================================================= Subject:[fitsbits] [Fwd: FITS-IDI] From: Eric Greisen Date: Wed, 29 Jul 2009 09:01:32 -0500 To: "fitsbits@NRAO.EDU" Comments from Bill Cotton Subject: FITS-IDI From: Bill Cotton Date: Tue, 28 Jul 2009 13:26:38 -0500 To: "egreisen@nrao.edu" Eric, I was just going throught the FITS-IDI draft you have on the AIPS site (AIPS memo 113) and jotten down some notes which I though I'd pass along. - Sect 2.1. Even after a decade, this really gets me torqued up. The weight was intended to be just that - a relative weight and not a patch for the VLBA's inability to do divisions. This perversion of the definition borders on making it useless. There REALLY HAS to be some way to convey relative weights. - sect 4.1.1 6th paragraph. Shouldn't the RA and Dec be specified to be those of the standard equinox and epoch? If the coordinates are given in the UV-DATA header, shouldn't the equinox and epoch also be given?, there may not be a SOURCE table in this case. - Sect 5.2, ARRAY_GEOMETRY table, Mount type should also have a value defined for an aperature array (as opposed to a steerable device). - Sect. 6 ANTENNA table. This really should include the antenna diameter as a number of arrays are heterogenous. The polarization calibration parameterization may need to be rethought with the upcoming wide band systems. A single calibration per band may not be enough. - Sect 8.2 SOURCE table. This MUST include the epoch of the position as well as the equinox. Eq. 7 is wrong, the subtraction should be a division. "Source-specific frequency offset" paragraph: the text should say that these are the frequency offsets of the frequency of the reference pixel and should be added to the "BANDFREQ" value from the FREQUENCY table "RESTFREQ" paragraph. This should state that this is the rest frequency of the transition in that band. It should also say what to do if there are multiple transitions. "Source positions" paragraph. Does the VLBA really use 2000.0 or 1950.0 for sources with meaningful proper motions? Footnote 4.If this is what the VLBA really does then this is REALLY BAD. - Sect 16, BANDPASS table. There is no provision for a parameterized bandpass function. - Sect 17, CALIBRATION table. The behavior of AIPS is not part of the format definition. However, I do agree with the sentiments you express at the beginning. The use of TSYS_n, TANT_n, SENSITIVITY_n and PHASE_n are undefined and I can't think of a sensible use of them in this context. The text should spell out the usage of REAL_n, IMAG_n, RATE_n and DELAY_n. It's also not clear that this bilinear phase model is adequate for wide bandwidths (may need higher order in delay) and/or high frequencies (may need higher order in time). -Bill ================================================================= Subject: Review of the FITS-IDI convention From: "Pence, William D. (GSFC-6601)" Date: Mon, 22 Mar 2010 12:36:45 -0500 To: FITSBITS Somewhat belatedly on my part, here is an update on the status of the FITS Interferometry Data Interchange (FITS-IDI) Convention. As you may recall, this convention was submitted to the Registry of FITS conventions (http://fits.gsfc.nasa.gov/fits_registry.html) in May 2009, and has been under public review since then. As described below, Eric Greisen has revised the definition document of the IDI convention in response to the comments from Bill Cotton. I have placed the revised document on the FITS Registry web page, along with a new "Comments and/or Critiques" link to all the comment that have been posted so far on FITSBITS regarding this convention. The public review period will remain open for a few more weeks in case there are any new comments about these revisions. - Bill Pence ...................................................... William Pence wrote: > > Hi Eric, > > > > I'm getting back to some unfinished FITS-related business, one of which > > is to get the FITS-IDI convention (documented at > > http://fits.gsfc.nasa.gov/fits_registry.html) moved from the "under > > review" category into the "registered" category. Before doing so, it > > seems that some sort of response to the comments made by Bill Cotton > > needs to be provided. Are you planning to revise the document to > > address his concerns? I attach Bill's text with detailed comments and new PS and PDF versions. The changes during the FITS-IDI comment period are shown in blue, those between the early Flatters document and the present on in red. Cheers, Eric ............................................................ I have revised the document, using blue color to represent changes made during the IAU Committee period. Bill's comments are precede by a > and my answers follow each with no indentation. I have also corrected more typographical errors. > > - Sect 2.1. Even after a decade, this really gets me torqued up. > > The weight was intended to be just that - a relative weight and not a > > patch for the VLBA's inability to do divisions. This perversion of > > the definition borders on making it useless. There REALLY HAS to be > > some way to convey relative weights. A third weighting method was added (p 7) along with a keyword WEIGHTYP in the UV_DATA header. The keyword is optional with default the old meaning for weights (p 16) > > - sect 4.1.1 6th paragraph. Shouldn't the RA and Dec be specified to > > be those of the standard equinox and epoch? If the coordinates are > > given in the UV-DATA header, shouldn't the equinox and epoch also be > > given?, there may not be a SOURCE table in this case. The EQUINOX keyword is added as an optional keyword in the UV_DATA for single-source data (p 13, 15, 16) > > - Sect 5.2, ARRAY_GEOMETRY table, Mount type should also have a value > > defined for an aperature array (as opposed to a steerable device). Added (p 19) > > - Sect. 6 ANTENNA table. > > This really should include the antenna diameter as a number of > > arrays are heterogenous. The ARRAY_GEOMETRY table (p 18, 19) now includes an antenna diameter The ANTENNA table contains a FWHM parameter with one value per band. > > The polarization calibration parameterization may need to be > > rethought with the upcoming wide band systems. A single calibration per > > band may not be enough. Indeed but beyond the scope of this manuscript. > > > > - Sect 8.2 SOURCE table. This MUST include the epoch of the position as > > well as the equinox. The EPOCH keyword has been added (p 23, 24) > > Eq. 7 is wrong, the subtraction should be a division. Yes (p 23) > > "Source-specific frequency offset" paragraph: the text should say > > that these are the frequency offsets of the frequency of the reference > > pixel and should be added to the "BANDFREQ" value from the FREQUENCY > > table Sentence added (p 23) > > "RESTFREQ" paragraph. This should state that this is the > > rest frequency of the transition in that band. It should also say > > what to do if there are multiple transitions. Clarifications added incl that there is no way for > 1 / band (p 24) > > "Source positions" paragraph. Does the VLBA really use 2000.0 or > > 1950.0 for sources with meaningful proper motions? Worse - it uses EPOCH not EQUINOX for EQUINOXes - note added (p 24) > > Footnote 4.If this is what the VLBA really does then this is REALLY > > BAD. In AIPS I have given up on apparent position columns and on input I recompute them (not good for moving sources). The EVLA UV-FITS output has apparent positions precessed back to MJAD 0. > > - Sect 16, BANDPASS table. There is no provision for a parameterized > > bandpass function. Correct - sentences added to this effect (p 36). My experience is that it is bad to interpolate in the parametrization values and there are way too many parameterizations possible. Should BP tables be written (and only LWA had tried), they should be expanded to the simple form we can all understand and interpolate (in time) as nneded. > > - Sect 17, CALIBRATION table. The behavior of AIPS is not part of the > > format definition. However, I do agree with the sentiments you > > express at the beginning. The use of TSYS_n, TANT_n, SENSITIVITY_n > > and PHASE_n are undefined and I can't think of a sensible use of them > > in this context. The text should spell out the usage of REAL_n, > > IMAG_n, RATE_n and DELAY_n. > > It's also not clear that this bilinear phase model is adequate for > > wide bandwidths (may need higher order in delay) and/or high > > frequencies (may need higher order in time). I do not understand the remark about AIPS nor the remark about bi-linear, neither are mentioned in the text. I have left the description of this table deliberately vague since I do not believe that anyone has even considered implementing it. A number of columns in it make no sense to me in terms of how AIPS does calibration - the Tsys, Tant, Sensitivity, and Phase columns make no sense to me. The latter 2 are also covered in the real and imaginary columns of the gain. A further remark about the uncertainties in this table was added (p 38). Eric Greisen