From: Mark Calabretta @atnf.csiro.au Date: 05/17/2010 On Wed 2010/05/12 15:36:33 -0400, William Pence wrote in a message to: Bob Garwood , Mark Calabretta Hi Bill, > >meeting. From previous discussions with Bob, I understand that one > >problem with adding this convention to the Registry is that the SDFITS > >format has evolved separately at different observatories, so there is no > >single document that describes all the different varieties of SDFITS. Harvey's document is somewhat out-of-date. There really isn't that much difference between the variants. From memory, the only significant difference is that I write TSYS (a CORE keyword) as a vector, one element for each polarization, instead of a scalar, and I write SCAN as an integer. I also use variable-length arrays for multiple-IF data, but only if the IFs have differing numbers of channels/polarizations (but I think Bob later followed suit), and I don't write NMATRIX, TMATXnnn, MAXIS, or MAXISnnn (but I believe these are all defunct now anyway). I use a few additional keywords that I would consider critical, namely CYCLE, BEAM, and IF. While they do not conflict with the SDFITS convention, the data would be all but useless without them. Regards, Mark =================================================================== From Bob Garwood @nrao.edu Date: 05/17/2010 Mark Calabretta wrote: > > Harvey's document is somewhat out-of-date. > Harvey's opinion - not explicitly expressed in the paper and one I didn't learn until maybe 2003 or so - is that the frequency-like axis should be the only non-degenerate axis. So we went that route for the sdfits we write for the GBT. > There really isn't that much difference between the variants. From > memory, the only significant difference is that I write TSYS (a CORE > keyword) as a vector, one element for each polarization, instead of > a scalar, and I write SCAN as an integer. I also use variable-length > arrays for multiple-IF data, but only if the IFs have differing numbers > of channels/polarizations (but I think Bob later followed suit), and I > don't write NMATRIX, TMATXnnn, MAXIS, or MAXISnnn (but I believe > these are all defunct now anyway). > I write TDIMnnn as a column and use that instead of the NMATRIX, TMATXnnn or MAXIS, MAXISnnn ways described in the paper for variable length data arrays. I did that for the SDFITS writer I wrote for aips++. For the GBT and GBTIDL it almost never happens that you switch between difference sizes from one row to the next and so we now just create a new HDU every time we need to write out a differently sized DATA array. > I use a few additional keywords that I would consider critical, > namely CYCLE, BEAM, and IF. While they do not conflict with the > SDFITS convention, the data would be all but useless without them. > > We have some similar keywords at the GBT. My impression is that Arecibo attempted to update the convention by bringing some of the WCS usages that have been developed since the SDFITS convention was described and it would be good to document those. -Bob =================================================================== To: [fitsbits] Subject: SDFITS Convention Comment: Hardware Keywords From: Tom Kuiper kuiper at jpl.nasa.gov Date: Mon Aug 16 16:41:21 EDT 2010 William Pence wrote: > This is to announce the start of the Public Comment Period on the > SDFITS binary table convention that is used for interchange of single > disk data in radio astronomy. This is another in a series of > conventions that have been submitted to the Registry of FITS > Conventions, which is maintained by the IAU FITS Working Group. > Bill, when does this Public Comment Period end? I'm now working through the process of writing the results of a particular series of observations into SDFITS. It'll be the first step in trying to get all the DSN spectroscopic runs to produce compatible SDFITS files. There will be issues along the way. I expect it'll take a few months. Group, an issue for me right now is how many keywords are needed to describe the observing equipment fully. I like the TELESCOP keyword because it can be a key to a whole host of other information for which there are no keywords, like the diameter or collecting area, the variation of gain with frequency and elevation. Can we agree on a standard set of values so all can share the same look-up table? Failing that, BEAMEFF, APEREFF, ETAL, ETAFSS, ANTGAIN, and so on must be included for very signal path which has a sufficiently different frequency so that it matters. What I mean is that DSS-28 can simultaneously have both polarizations at, for example, 3, 6, 9 and 12 GHz. That's a lot of keywords. Easier if I just tell you that it's DSS-28. Some of our other antenas can give you LCP and RCP at both 8.4 and 32 GHz. Fortunately, they all are quite similar with respect to antenna parameters. So there should be a registry to ensure that each antenna has a unique TELESCOP keyword. From a relational database perspective, an entry in the TELSCOP table would have a key to another table which has antenna parameters according to antenna type. Certainly, TELESCOP should be defined narrowly enough so that it doesn't include the receiver chain and backend. I like the FRONTEND keyword to mean what package has been selected by whatever mechanism the antenna uses: a rotating sub-reflector like the DSN 70-m's, a turret mechanism, a rotating tertiary, whatever. However, that's not the end of the story. First of all, there may be two, separated by a dichroic, as implied above. I'm leaning towards using two values (in one SDFITS file) for the FRONTEND keyword to handle that. That is, FRONTEND will be a column. In this case, the value of FRONTEND only needs to be unique for the telescope. Two telescopes can certainly have a FRONTEND called "K". There may be multiple feeds, the FEED keyword handles that well, in principle. However, it's only a number without the FRONTEND keyword to say what it means. Each feed generally puts out two orthogonal polarizations. The POLNO keyword used in ASAP seems appropriate, but not STOKES because at that point the signal is still a voltage. STOKES doesn't become a meaningful keyword until the signal reaches the end of the chain. Indeed, it may never become a meaningful keyword if the raw voltages are recorded for later processing in software. Behind each OMT output is an LNA. So you cannot make do with one TRX. Likewise, TCAL is probably a column, since the cal signal injected can in principle be different for each channel. However, THOT is probably the same for all channels. Behind each OMT output is a down-converter. At least in our case, the two are not necessarily permanently married. There is a switch to select the down-converter. I haven't yet found a situation in which it matters how the down-conversion is done, as long as the frequency vs channel number (CRVAL1, CRPIX1, CDELT1) is correct. However, we should probably allow for such a possibility. I think that defining RECEIVER narrowly to mean the RF -> baseband converter will take care of that. Then it can be used as a key to look up such details as whether there is one baseband output, or a LSB/USB pair, or an I/Q pair. I think what I'm getting at here is that pretty much anything related to the hardware can be a column. I think there is no hope of fully describing a back-end with keywords. The keyword BACKEND should be strictly defined to be the device in which the signal ends up, which could be pretty much anything. However, it is important to know what it is. For example, something that puts out channelized data could be an autocorrelator, a hardware FFT, or a PPFB. At least in the latter two examples, there isn't a clue about their difference in other keywords. A final comment about raw voltages. The future is here. The primary data files for the observations I mentioned in the beginning are recorded that way. I haven't yet decided how I will finally process them. I think PSRFITS has an extension called SUBINT. We should adopt it as part of SDFITS. Best regards Tom ------------------------------------------------------------------------ To: [fitsbits] Subject: SDFITS Convention Comment: Hardware Keywords From: Tom Kuiper kuiper at jpl.nasa.gov Date: Tue Aug 17 15:14:12 EDT 2010 I haven't participated in one of these before and so I don't know the best way to proceed. I realize that no one might react to my previous e-mail and you wouldn't know if that indicated agreement, disagreement or disinterest. Would it be more helpful if I put my ideas in a declarative form? Then, if there is no reaction, agreement can be assumed. Here's an example: The TELESCOP keyword is required in the extension header and its associated value will be a unique string which identifies the antenna or antenna array and can be used as a key to a database of antenna parameters. NRAO will maintain a registry for TELESCOP keyword values. Regards Tom ------------------------------------------------------------------------ To: [fitsbits] Subject: SDFITS Convention Comment: Hardware Keywords From: Eric Greisen egreisen at nrao.edu Date: Tue Aug 17 15:32:17 EDT 2010 Tom Kuiper wrote: > > I haven't participated in one of these before and so I don't know the > best way to proceed. I realize that no one might react to my previous > e-mail and you wouldn't know if that indicated agreement, disagreement > or disinterest. Would it be more helpful if I put my ideas in a > declarative form? Then, if there is no reaction, agreement can be > assumed. Here's an example: > > The TELESCOP keyword is required in the extension header and its > associated value will be a unique string which identifies the antenna or > antenna array and can be used as a key to a database of antenna > parameters. NRAO will maintain a registry for TELESCOP keyword values. I actually like the idea of a registry - but who at NRAO did you have in mind? There was once a FITS office informally at NRAO but that individual retired some years ago. The AIPS project has been deprecated for many years and depends solely on me to keep all its users afloat. The CASA project has little to no interest in FITS. The VLB users maintain a registry of sorts but at a more detailed level intended to represent that multiple names and abbreviations used for the same telescopes when they are used in various VLBI configurations. The individual who has created that list is openly considering a partial retirement. Eric Greisen ------------------------------------------------------------------------ To: [fitsbits] Subject: SDFITS Convention Comment: Hardware Keywords From: Bob Garwood bgarwood at nrao.edu Date: Tue Aug 17 16:07:05 EDT 2010 I was going to say pretty much what Eric said. I think it would be useful to register telescope names somewhere. Perhaps there are other similar string values as labels that it would be useful to register the individual labels somewhere. ------------------------------------------------------------------------ To: [fitsbits] Subject: SDFITS Convention Comment: Hardware Keywords From: Bob Garwood bgarwood at nrao.edu Date: Tue Aug 17 16:15:57 EDT 2010 I wrote this yesterday but haven't sent it yet. I'm not sure whether this covers your points yesterday or just readers or what. Personally, while I agree there's plenty of room for SDFITS to be improved, I've mumbled that to various people for years without finding the time to follow through on that and so after > 10 years writing SDFITS (closer to 20 years) I think its most important to document something that summarizes how SDFITS has been used in the past without trying to improve SDFITS right now. So, here's what I wrote yesterday: This convention is primarily to document existing usage - there's plenty of room for improvement, but the history of this convention has shown that were we to wait for those discussions to happen this convention would never get documented. If there's some momentum for extending sdfits to include additional things - e.g. improving the axes descriptions in light of the WCS papers that didn't exist when this convention was written, or expanding the list of SHARED columns, I'd hope that those discussions could be put off until we've documented what's already being written [it would be useful to include an Arecibo example case here, I think. I'm not sure where else SDFITS might already be in use]. Since I wasn't at the original meeting in Green Bank where the ideas behind the SDFITS convention was first discussed, this is simply my understanding of the intention after talking to a few of the people that were there. I think the idea was that SDFITS would primarily be useful in the exchange of calibrated and reduced data between telescopes and data analysis packages and it was less useful for exchanging raw data. We use it at the GBT for raw data by liberally adding lots of telescope-specific columns. By the time the data is calibrated the CORE columns plus the data axes descriptions are what's important. So, I think in that spirit you should feel free to use whatever keywords you feel are necessary to fully describe your data, especially for your internal use and for historical archives, but it's unlikely that a package not specifically familiar with your data will know what to do with most of that. I've always felt that TELESCOP describes the whole thing collecting the photos. ALMA is a TELESCOP with multiple antennas. NRAO-GBT is a TELESCOP with one antenna. SDFITS doesn't go into useful keywords, etc, for collections of spectra from multi-antenna telescopes. It would be useful to have a registry of TELESCOP names. We (GBT sdfits writer) put polarization information into a stokes-like axis: e.g CTYPE4='STOKES', CRVAL4=-5 (which translates to "XX", ala AIPS - in that usage negative values aren't really stokes IQUV but they are recorded in the "STOKES" axis nevertheless). As per the GB convention that underlies SDFITS - the distinction between a keyword and a column is very fuzzy. A keyword is a column with a constant value. So there's no reason that FRONTEND couldn't be a keyword (in fact, that's what we write for the GBT). -Bob ------------------------------------------------------------------------ To: [fitsbits] Subject: SDFITS Convention Comment: Hardware Keywords From: Bob Garwood bgarwood at nrao.edu Date: Tue Aug 17 16:17:43 EDT 2010 Bob Garwood wrote: > I'm not sure whether > this covers your points yesterday or just readers or what. > I have no idea what I was trying to say in the second half of that "sentence". Sorry. ------------------------------------------------------------------------ To: [fitsbits] Subject: SDFITS Convention Comment: Hardware Keywords From: Norman Gray norman at astro.gla.ac.uk Date: Wed Aug 18 04:29:33 EDT 2010 Greetings. On 2010 Aug 17, at 20:32, Eric Greisen wrote: > Tom Kuiper wrote: >> The TELESCOP keyword is required in the extension header and its >> associated value will be a unique string which identifies the antenna or >> antenna array and can be used as a key to a database of antenna >> parameters. NRAO will maintain a registry for TELESCOP keyword values. > > I actually like the idea of a registry - but who at NRAO did you have in > mind? There was once a FITS office informally at NRAO but that > individual retired some years ago. The AIPS project has been deprecated > for many years and depends solely on me to keep all its users afloat. > The CASA project has little to no interest in FITS. The VLB users > maintain a registry of sorts but at a more detailed level intended to > represent that multiple names and abbreviations used for the same > telescopes when they are used in various VLBI configurations. An alternative to a registry (or, in one view, a decentralised zeroconf registry) is to require that the TELESCOP value be a URI. This URI should be one which the telescope, or its associated archive, publishes as the long-term 'name' of the instrument. When retrieved, it can return either or both of human-readable and machine-readable information about the instrument and its parameters. There are Standards-based ways of doing this. Having said that, a URI can function perfectly happily as a _name_ for a thing, without it being required to be dereferenceable. The DNS provides the namespacing and uniqueness guarantees. If it's desirable (and I think it is) to have a half-way house between this decentralised approach and something more registry-like, then there'd be a very good case for using PURLs (see http://purl.org) to add a level of (curatable) indirection. This would also help to keep the URIs within the FITS value character limit. Best wishes, Norman -- Norman Gray : http://nxg.me.uk School of Physics and Astronomy, University of Glasgow, UK