From dishfits-request Thu Mar 2 13:43:57 1995 X-VM-Message-Order: (1 2 3 5 6 7 8 4 9) X-VM-Summary-Format: "%n %*%a %-17.17F %-3.3m %2d %4l/%-5c %I\"%s\"\n" X-VM-Labels: nil X-VM-VHeader: ("Resent-" "From:" "Sender:" "To:" "Apparently-To:" "Cc:" "Subject:" "Date:") nil X-VM-Bookmark: 9 X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["5404" "Thu" "2" "March" "1995" "13:43:52" "+0500" "Bob Garwood" "bgarwood@nrao.edu" "<9503021843.AA00945@sngldsh.cv.nrao.edu>" "124" "Re: GBT use of FITS for single dish data" "^From:" nil nil "3" "1995030208:43:52" "GBT use of FITS for single dish data" (number " " mark " Bob Garwood Mar 2 124/5404 " thread-indent "\"Re: GBT use of FITS for single dish data\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-CV/1.8) id AA03416; Thu, 2 Mar 95 13:43:57 EST Return-Path: Message-Id: <9503021843.AA00945@sngldsh.cv.nrao.edu> X-Sun-Charset: US-ASCII Content-Length: 5402 From: bgarwood@NRAO.EDU (Bob Garwood) Sender: dishfits-request@fits.cv.nrao.edu To: dishfits@fits.cv.nrao.edu Subject: Re: GBT use of FITS for single dish data Date: Thu, 2 Mar 1995 13:43:52 +0500 Hi Ray, Thanks for joining the conversation. I hope that others out there also chime in from time to time. Before I get to your specific questions, let me state my philosophy on this single dish FITS convention. The purpose of the convention is to convey as much telescope independent information as possible so that the data can be correctly used in various analysis environments that may not have knowledge of the quirks of the telescope that generated the data. Ok, now on to your questions: "How do we store a series of spectra obtained with a telescope?" First, this convention deals with binary tables only. It does NOT preclude multiple binary tables in a single FITS file. In general, any FITS file may have multiple extensions, some of which may be binary tables, some of which may follow this convention. So, you are always free to divide up your data into multiple tables if that makes more sense (either by making it easier on the writter, or because it logically makes sense to divide the data, or for whatever reason). Having said that, each row of the binary table will contain one multi-dimensional matrix that holds the data (spectra for spectral-line observations). This may in fact be more than one spectra so long as we can agree on what the additional dimensions are (for example, one might put data taken on a regular grid in sky position [i.e. a spectral line cube]). Clearly we need to talk about what axes are allowed. Yes, this probably does mean that if you are writing the data in real time that you need to write to disk and not to tape. I think this is a problem for the folks doing the writing and not a concern of this convention. From your list of possible answers to your question, I clearly prefer (1). I would not write a new file for each observation (your number 2). Since this convention is strictly concerning binary tables, you could write each observation to its own table (your number 3) and append the tables to one file but I think that's just as much a data management problem as writing out several files. Your (4) is therefor the same as (3). " A common observing mode at Parkes (and I assume other instruments) is to observe two transitions simultaneously (e.g. megamaser observations at HI 1420 MHz and OH 1665 MHz). Unfortunately many of the FITS header keywords suggested (e.g. TSYS, RESTFREQ, etc) would be good for only one of these bands, and so it would not be appropriate to put them in the FITS header." A keyword in a FITS binary table should be thought of as a column name where the value for that column is the same for each row of the column. If the keyword in question may vary in the table, it should be a column. So, if TSYS varies from row to row (which it will), it should be a column, etc, etc. If it is import to know what band the spectra came from, you should have some telescope dependent keyword that indicates that information. Then the same TSYS, RESTFREQ, etc, keywords could work for each row, independent of which spectral band the data came from. Note that above I said that you could use the multi-dimensional array to hold several spectra. If, however, you want to specify a different TSYS for each spectra, you should break that multi-dimemensional array up into individual spectra. We need to make the job of reading this data as simple as we can and it should require NO telescope specific information to read the understand the basics about the data. So, there should be a single TSYS column, a single RESTFREQ column, etc. You might alternatively want to break the table up into a table for each band. But I don't think its necessary at all. "A related problem is that of multi-beam feeds (e.g. the 13-beam HI feed now being built for Parkes). Again, we need a different Tsys etc for each beam. Again, maybe a separate table for each beam is needed." Again, while you might choose separate tables, I don't think its necessary. You may choose to put some telescope specific column indicating the feed for each row. But as long as the fundamental information such as the sky-position, frequency, etc, for each row is correct, you should be able to put all of it one table. "Another related problem is the use of arrays of different lengths, which others (e.g. Bob) have already noted." I've come to the conclusion that we simply can't ignore the proposed variable length array convention and we should allow for it in this convention. Now, I have a request I'ld like to make of everyone. Last Jan. 31st Bob Payne reported that the GBT has decided to use the preliminary table coordinates proposal for describing coordinates in their binary tables. This proposal differs from the proposal adopted at Green Bank in 1989 for single dish data. The proposal is available via ftp at : Preliminary Table Coordinates proposal ftp://legacy.gsfc.nasa.gov/fits_info/ofwg_recomm/coord_keywords.ps I would like everyone to read that proposal. I have had some conversations with a couple of people here who do not like the proposal, especially as it relates to our needs. I personally think it makes life unnecessarily complicated for readers of the proposed single dish FITS convention. I think the decision on whether to follow the original Green Bank coordinates convention or the above proposal should be the first decision we should try and reach. Cheers, Bob From dishfits-request Thu Mar 16 06:07:03 1995 X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2053" "Thu" "16" "March" "1995" "21:06:32" "EST" "Ray Norris" "rnorris@atnf.csiro.au" "<9503161106.AA02534@brogar.rp.CSIRO.AU>" "46" "Re: GBT use of FITS for single dish data" "^From:" nil nil "3" "1995031702:06:32" "GBT use of FITS for single dish data" (number " " mark " Ray Norris Mar 16 46/2053 " thread-indent "\"Re: GBT use of FITS for single dish data\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-CV/1.8) id AA21019; Thu, 16 Mar 95 06:07:03 EST Return-Path: Message-Id: <9503161106.AA02534@brogar.rp.CSIRO.AU> From: rnorris@atnf.CSIRO.AU (Ray Norris) Sender: dishfits-request@fits.cv.nrao.edu To: dishfits@fits.cv.nrao.edu, bgarwood@NRAO.EDU Subject: Re: GBT use of FITS for single dish data Date: Thu, 16 Mar 95 21:06:32 EST Hi Bob Thanks for your answer. there's nothing you say with which I fundamentally disagree. Certainly I agree with your philosphy, although perhaps I would add the proviso that we should be careful to remain pragmatic - the data have to be handled by real people and real machines. So when you say > Yes, this probably does mean that if you are writing the data > in real time that you need to write to disk and not to tape. I think > this is a problem for the folks doing the writing and not a concern of this > convention. then I think I disagree in principle - this *should* be our concern. The original FITS convention said that you had to be able to write to tape in real time. I for one am happy to abandon that, as disks are now cheap and fast, and transfer to tape (or CDROM, or whatever) can take place off-line, but I think we should make such changes consciously and not merely implicitly, or by default. In this case the technology has changed so this old proviso is no longer appropriate. In this particular case, maybe I'm being a little pedantic, but I think as a matter of principle we should think through all the practical implications of wehat we agree on. > > >From your list of possible answers to your question, I clearly prefer > (1). I would not write a new file for each observation (your number 2). > Since this convention is strictly concerning binary tables, you could > write each observation to its own table (your number 3) and append the > tables to one file but I think that's just as much a data management > problem as writing out several files. Your (4) is therefor the same > as (3). Incidentally, maybe I didn't make myself clear in (3) - I meant that some groups write the whole thing (main FITS header and all) out again and again - standard FITS readers would barf at that so I don't think it's to be advocated, but some groups do it! The rest of what you say I agree with. If we treat each keyword as a column header, and allow variable-length arrays, then all my problems are solved. Cheers ray From dishfits-request Thu Mar 16 11:17:23 1995 X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3256" "Thu" "16" "March" "1995" "11:17:19" "+0500" "Bob Garwood" "bgarwood@nrao.edu" "<9503161617.AA01092@sngldsh.cv.nrao.edu>" "71" "Re: GBT use of FITS for single dish data" "^From:" nil nil "3" "1995031606:17:19" "GBT use of FITS for single dish data" (number " " mark " Bob Garwood Mar 16 71/3256 " thread-indent "\"Re: GBT use of FITS for single dish data\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-CV/1.8) id AA21457; Thu, 16 Mar 95 11:17:23 EST Return-Path: Message-Id: <9503161617.AA01092@sngldsh.cv.nrao.edu> X-Sun-Charset: US-ASCII Content-Length: 3254 From: bgarwood@NRAO.EDU (Bob Garwood) Sender: dishfits-request@fits.cv.nrao.edu To: dishfits@sngldsh.cv.nrao.edu Subject: Re: GBT use of FITS for single dish data Date: Thu, 16 Mar 1995 11:17:19 +0500 Harvey has asked that his response to Ray's comments be forwarded to the dish fits exploder. ----- Begin Included Message ----- From hliszt@polaris.cv.nrao.edu Thu Mar 16 09:22:02 1995 Subject: Re: GBT use of FITS for single dish data To: rnorris@atnf.CSIRO.AU (Ray Norris) Date: Thu, 16 Mar 1995 09:21:43 -0500 (EST) From: "Harvey Liszt,222,344,9733744" Cc: bgarwood@polaris.cv.nrao.edu (Bob Garwood), hliszt@polaris.cv.nrao.edu (Harvey Liszt) Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Ray-- I've been lurking, reading the dialog between you and Bob Garwood recently. With regard to the matter of formatting data, I second Bob's comments. I think data shouldn't be hidden; writing one row per spectrum is the best thing conceptually and practically. The rows of one table may have to have the same structure but within that constraint, there can be quite big differences from row to row. There's no barrier to mixing OH and CO spectra in the same table, array length aside. With regard to the real-time problem, I think it is appropriate to think of a "fix" to FITS which accomodates this better, although I think that solutions are possible even without a formal fix. I'm thinking of other instances where formats that looked off-line-ish and finished and writable a priori got such fixes. The one I deal with most commonly is Postscript. This is supposed to have a bounding box in the header; because it isn't known ahead of time, most writers just put the size of a whole page and then go on to cover whatever space they need; this can make it difficult to merge postscript illustrations. Another problem is speed, because resources are used and discarded when they might better be cached. To solve the problem, one can now put in an afterword at the end of the file which lists the fonts and resources and the actual space. Some readers can use this info but it isn't absolutely necessary. This works typically because one deals with disk files, but it is a response to exactly the same phenomenon you have addressed. FITS is a slightly different case because it is so highly structured. It seems to me that in many (most?) cases, the actual dimensions along all n axes don't need to be contained in the header; only n-1. Given the fixed size of each instance, the size of the file alone determines the dimension along the last axis. Or alternatively one can simply read until the final END statment is encountered or an EOF. Does this make sense? It seems to me that either a reader could purposefully ignore the supposed value in the header for the number of rows in a table or in a map, or the naxisn[naxes] could be set = * in the header to tell a reader to read exhaustively, not up to a fixed index. This applies to tape more than to a disk file; if the size of a file is known, the naxisn value along the last axis can be figured out even if it is not in the header. We've grown too accustomed to having the FITS header spell everything out for us; remember the old days of writing programs that read in a number of data points that _wasn't_ known ahead of time? regards, Harvey Liszt ----- End Included Message ----- From dishfits-request Thu Mar 16 18:38:36 1995 X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["587" "Fri" "17" "March" "1995" "09:38:12" "EST" "Ray Norris" "rnorris@atnf.csiro.au" "<9503162338.AA04850@brogar.rp.CSIRO.AU>" "17" "Re: GBT use of FITS for single dish data" "^From:" nil nil "3" "1995031714:38:12" "GBT use of FITS for single dish data" (number " " mark " Ray Norris Mar 17 17/587 " thread-indent "\"Re: GBT use of FITS for single dish data\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-CV/1.8) id AA22968; Thu, 16 Mar 95 18:38:36 EST Return-Path: Message-Id: <9503162338.AA04850@brogar.rp.CSIRO.AU> Content-Length: 585 From: rnorris@atnf.CSIRO.AU (Ray Norris) Sender: dishfits-request@fits.cv.nrao.edu To: dishfits@sngldsh.cv.nrao.edu, bgarwood@NRAO.EDU Subject: Re: GBT use of FITS for single dish data Date: Fri, 17 Mar 95 09:38:12 EST Hi harvey Yes I agree with everything you say. In fact, our local data format, RPFITS, which is a variant of FITS (and starts with SIMPLE = FALSE!), does indeed set the number of groups to -1, and the reader then just checks for invalid data to let it know the data has finished. We also have variable length arrays in it, but not in a binary table format. But if FITS evolves to include the sorts of things you are suggesting, then RPFITS would become redundant. I look forward to the day when we can throw away RPFITS and use standard FITS for our local data format! Cheers ray From dishfits-request Fri Mar 17 07:47:05 1995 X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["446" "Fri" "17" "March" "1995" "07:47:00" "EST" "Bill Cotton" "bcotton@gorilla.cv.nrao.edu" "<9503171247.AA02604@gorilla.cv.nrao.edu>" "10" "Re: GBT use of FITS for single dish data" "^From:" nil nil "3" "1995031712:47:00" "GBT use of FITS for single dish data" (number " " mark " Bill Cotton Mar 17 10/446 " thread-indent "\"Re: GBT use of FITS for single dish data\"\n") "<9503162338.AA04850@brogar.rp.CSIRO.AU>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-CV/1.8) id AA25159; Fri, 17 Mar 95 07:47:05 EST Return-Path: Message-Id: <9503171247.AA02604@gorilla.cv.nrao.edu> In-Reply-To: <9503162338.AA04850@brogar.rp.CSIRO.AU> References: <9503162338.AA04850@brogar.rp.CSIRO.AU> Content-Length: 444 From: bcotton@gorilla.cv.nrao.edu (Bill Cotton) Sender: dishfits-request@fits.cv.nrao.edu To: dishfits@sngldsh.cv.nrao.edu Subject: Re: GBT use of FITS for single dish data Date: Fri, 17 Mar 95 07:47:00 EST If I might comment on the real-time problem; the VLBA correlator format has partially addressed this problem by defining the appropriate response to a premature EOF. All descriptive tables (antenna positions, etc.) are given first and then the data table. If the data stream is terminated before reading the number of rows given in the table header this is treated as an abnormal condition but does not invalidate data already read. -Bill From dishfits-request Fri Mar 17 08:39:05 1995 X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1788" "Fri" "17" "March" "1995" "08:38:58" "-0500" "Bob Payne" "rpayne@sadira.gb.nrao.edu" "<9503171338.AA02198@vega.gb.nrao>" "46" "GBT use of FITS for single dish data" "^From:" nil nil "3" "1995031713:38:58" "GBT use of FITS for single dish data" (number " " mark " Bob Payne Mar 17 46/1788 " thread-indent "\"GBT use of FITS for single dish data\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-CV/1.8) id AA25211; Fri, 17 Mar 95 08:39:05 EST Return-Path: Message-Id: <9503171338.AA02198@vega.gb.nrao> From: rpayne@sadira.gb.nrao.edu (Bob Payne) Sender: dishfits-request@fits.cv.nrao.edu To: dishfits@fits.cv.nrao.edu Subject: GBT use of FITS for single dish data Date: Fri, 17 Mar 1995 08:38:58 -0500 We would not have decided to use FITS for GBT data collection if the multidimensional array and variable length array conventions were not available and supported. The libraries I use, "fitsio" for fortran and IDL astronomy library for PV-WAVE all support this convention. I note that Ray Norris >Another related problem is the use of arrays of different lengths, >which others (e.g. Bob) have already noted. Typically in multi-band >observations you might observe with different numbers of channels in >each band, and different bandwidths, to achieve comparable velocity >resolution. So I for one would urge that we try to incorporate >support for variable length arrays. and Phil Jewell >Also, Phil Jewell has suggested via e-mail that continuum data may be >a prime example where variable length arrays are extremely useful. and the GBT all reccommend use of the variable length array convention. Consequently I do not see the point of Bob Garwood's recommendation against variable length arrays. >My inclination is that the SD-FITS standard echo the >recommendation against variable length arrays from the FITS binary >table standard but recognize that there are situations where it will >be used. For the GBT we plan on using the simplest FITS format appropriate for the data. Most table data such as weather station data and even continuum backend data will be stored in binary tables with scaler data values. More complex backends such as the spectral processor require data values which are multidimensional. While not currently used in that mode the spectral processor is capable of generating variable length data arrays. I suggest that the SD-FITS standard recognize the use of the variable length array convention and drop all recommendations against its use. From dishfits-request Fri Mar 17 08:59:21 1995 X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["447" "Fri" "17" "March" "1995" "08:59:15" "+0500" "Bob Garwood" "bgarwood@nrao.edu" "<9503171359.AA01742@sngldsh.cv.nrao.edu>" "18" "Re: GBT use of FITS for single dish data" "^From:" nil nil "3" "1995031703:59:15" "GBT use of FITS for single dish data" (number " " mark " Bob Garwood Mar 17 18/447 " thread-indent "\"Re: GBT use of FITS for single dish data\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-CV/1.8) id AA25263; Fri, 17 Mar 95 08:59:21 EST Return-Path: Message-Id: <9503171359.AA01742@sngldsh.cv.nrao.edu> X-Sun-Charset: US-ASCII Content-Length: 445 From: bgarwood@NRAO.EDU (Bob Garwood) Sender: dishfits-request@fits.cv.nrao.edu To: dishfits@fits.cv.nrao.edu, rpayne@sadira.gb.nrao.edu Subject: Re: GBT use of FITS for single dish data Date: Fri, 17 Mar 1995 08:59:15 +0500 Bob Payne writes: > > I suggest that the SD-FITS standard recognize the use of the > variable length array convention and drop all recommendations against > its use. > In an earlier note to dishfits (on Mar 2) I wrote: "I've come to the conclusion that we simply can't ignore the proposed variable length array convention and we should allow for it in this convention." So I'm pretty sure we're all in agreement on this one already. -Bob From dishfits-request Wed Mar 22 12:21:31 1995 X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2659" "" "22" "March" "1995" "11:52" "EDT" "Barry M. Schlesinger" "bschlesinger@nssdca.gsfc.nasa.gov" "<22MAR199511522887@nssdca.gsfc.nasa.gov>" "64" "Re: GBT use of FITS for single dish data" "^From:" nil nil "3" "1995032215:52:00" "GBT use of FITS for single dish data" (number " " mark " Barry M. Schlesin Mar 22 64/2659 " thread-indent "\"Re: GBT use of FITS for single dish data\"\n") "<9503161617.AA01092@sngldsh.cv.nrao.edu>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-CV/1.8) id AA14873; Wed, 22 Mar 95 12:21:31 EST Return-Path: Message-Id: <22MAR199511522887@nssdca.gsfc.nasa.gov> Organization: NASA Goddard Space Flight Center -- Greenbelt, Maryland USA Path: solitaire.cv.nrao.edu!iraf!newsfeed.gsfc.nasa.gov!nssdca.gsfc.nasa.gov!bschlesinger References: <9503161617.AA01092@sngldsh.cv.nrao.edu> Newsgroups: adass.fits.dishfits From: bschlesinger@nssdca.gsfc.nasa.gov (Barry M. Schlesinger) Sender: dishfits-request@fits.cv.nrao.edu To: dishfits@fits.cv.nrao.edu Subject: Re: GBT use of FITS for single dish data Date: 22 Mar 1995 11:52 EDT In article <9503161617.AA01092@sngldsh.cv.nrao.edu>, bgarwood@NRAO.EDU (Bob Garwood) writes... > >Harvey has asked that his response to Ray's comments be forwarded to the >dish fits exploder. > >----- Begin Included Message ----- > >From hliszt@polaris.cv.nrao.edu Thu Mar 16 09:22:02 1995 >Subject: Re: GBT use of FITS for single dish data [...] >Ray-- > [...] >It >seems to me that in many (most?) cases, the actual dimensions along all n axes >don't need to be contained in the header; only n-1. Given the fixed size of >each instance, the size of the file alone determines the dimension along the >last axis. Or alternatively one can simply read until the final END statment >is encountered or an EOF. > >Does this make sense? It seems to me that either a reader could purposefully >ignore the supposed value in the header for the number of rows in a table or >in a map, or the naxisn[naxes] could be set = * in the header to tell a reader >to read exhaustively, not up to a fixed index. This applies to tape more than >to a disk file; if the size of a file is known, the naxisn value along the last >axis can be figured out even if it is not in the header. We've grown too >accustomed to having the FITS header spell everything out for us; remember the >old days of writing programs that read in a number of data points that _wasn't_ >known ahead of time? > >regards, > >Harvey Liszt > The problem with this proposal is that it violates a requirement of the Generalized Extensions paper: "-A program scanning a tape should be able to locate the begginning of any extension and should be able to skip over the extension, i. e., to find the start of the next one. This requirement implies that the extension header must specify in some consistent and standardized manner the total number of data bits which are associated with it." The original motivation for this requirement was to permit software to skip extensions in a FITS file that it did not know how to handle, by reading the name and size from the header. The name would identify the extension type as one the software could not handle and the size would allow the software to skip to the next extension. So, in the first instance, readers that cannot read BINTABLE should be able to skip to the next extension. This requirement also makes it possible to construct utility programs that dump all the headers in a FITS file without reading the data. After reading one header, they use the information in it to skip to the next. Such software is a very useful tool for examining newly received files. Barry Schlesinger FITS Support Office From dishfits-request Thu Mar 23 02:42:27 1995 X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["646" "Thu" "23" "March" "1995" "17:42:12" "EST" "Ray Norris" "rnorris@atnf.csiro.au" "<9503230742.AA18723@brogar.rp.CSIRO.AU>" "19" "Re: GBT use of FITS for single dish data" "^From:" nil nil "3" "1995032322:42:12" "GBT use of FITS for single dish data" (number " " mark " Ray Norris Mar 23 19/646 " thread-indent "\"Re: GBT use of FITS for single dish data\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-CV/1.8) id AA17472; Thu, 23 Mar 95 02:42:27 EST Return-Path: Message-Id: <9503230742.AA18723@brogar.rp.CSIRO.AU> From: rnorris@atnf.CSIRO.AU (Ray Norris) Sender: dishfits-request@fits.cv.nrao.edu To: dishfits@fits.cv.nrao.edu Subject: Re: GBT use of FITS for single dish data Date: Thu, 23 Mar 95 17:42:12 EST Regarding Barry's comment: > This requirement also makes it possible to construct utility programs > that dump all the headers in a FITS file without reading the data. > After reading one header, they use the information in it to skip to > the next. Such software is a very useful tool for examining newly > received files. Yes, but it's also trivial to write utility programs, and FITS readers, that just check the first few bytes of each FITS block to see if it's any sort of header (we do this in RPFITS). So although this standard does indeed violate the original FITS standard, maybe it's time to re-examine that standard. Cheers Ray