From dishfits-request@fits.CX.NRAO.EDU Wed Oct 4 13:26:17 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA02422; Wed, 4 Oct 89 13:26:15 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA12945; Wed, 4 Oct 89 13:27:18 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA02418; Wed, 4 Oct 89 13:18:59 EDT Return-Path: X-Vms-Mail-To: DISHFITS Message-Id: <891004130929.0000214A0B1@cvax.CV.NRAO.EDU> Status: RO From: HLISZT@NRAO.EDU Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Draft agenda and travel info for single-dish meeting in Green Bank Date: Wed, 4 Oct 89 13:09:29 EDT Single-Dish FITS Standards Workshop Green Bank W. Va, 31 October-1 November 1989 Dear Colleague: Thank you for the interest you have shown in establishing a standard data exchange format for single-dish radioastronomy. With interest and enthusiasm, we will create one! This message contains travel information and a reply form which you can return via post or EMail. We are also sending by conventional mail a package which is similar to this one but with more complete travel information (and a map). The last portion of this mail message contains a draft agenda for the meeting. You will notice that this agenda begins with a formal schedule of presentations concerning various aspects of FITS, but soon becomes only an outline of steps which we might take to design and implement the new standard. Any suggestions you might have regarding the format of the meeting are welcome. If you have (even preliminary) ideas concerning data exchange, please let us hear from you; the floor will be open in what we hope will be a free and spirited exchange of such ideas. If you wish to make a formal presentation, "we" would be glad to place you on the program. On the second day of the meeting, we have placed on the agenda a rather general item concerning single-dish software. If the previous discussions fill the available time, this item will be sacrificed. However, if time is available, it would be informative for the group to hear about recent developments at your telescope. ================================================================================ INFORMATION REGARDING TRAVEL TO GREEN BANK (AND/OR CHARLOTTESVILLE) Transportation will largely be the responsibility of the individual participants. The nearest major airports to Green Bank are in Washington, DC [Dulles (IAD) and National (DCA)] and Pittsburgh, PA. Direct travel to GB from either of these metropolitan areas involves a 4-5 hour drive through the picturesque and twisty mountains of West Virginia and western VA. Otherwise, one might wish to fly into Charlottesville, VA where transport between NRAO Headquarters and Green Bank will be arranged. Green Bank is about 75 miles due West of Charlottesville, or about 120 miles (some 3 hours) by road. About one hour of this trip is fairly adventurous driving. A map giving driving directions to Green Bank is included in the non-electronic version of this mailing. We regret that there can be no financial subsidies from the NRAO. However, once in Green Bank, lodging in the NRAO dormitories and on-site houses (this may be either double or single occupancy) will be provided without charge. The NRAO will also provide meals and frequent liquid refreshment. For those specifically desiring single rooms, lodging in nearby motels can be arranged (by the NRAO) at the participant's expense. Carpooling or van pickup for the sessions will be organized. The weather at the end of October will be fairly cold with crisp nights (Green Bank is at 2700 ft altitude). Visitors are advised to bring warm clothes and to be prepared for the possibility of rain or snow. ================================================================================ REGARDING YOUR TRAVEL PLANS Please return by post to H. Liszt/Single-Dish Fits Meeting NRAO Edgemont Road Charlottesville, VA USA 22903-2475, or EMail via BITNET to HLISZT@NRAO, Internet to HLISZT@NRAO.edu, SPAN to 6654::HLISZT or phone Harvey Liszt 804-296-0344 (an answering machine will pick up the phone after 3 1/2 rings) -------------------------------------------------------------------------------- Name:............................... Phone/E-Mail:............................ I will arrive in (city)....................... at (date/time)................... And will depart from.......................... at (date/time)................... Before the meeting, I will need accomodations in Charlottesville on (dates)............................... After the meeting, I will need accomodations in Charlottesville on (dates)............................... Please arrange transport for me from Charlottesville to Green Bank on 30 Oct. yes....... no....... Please arrange transport for me from Green Bank to Charlottesville on 1 Nov. yes....... no....... I will share a room with...................................... I really strongly prefer a single room yes....... no....... I will be accompanied by (non-participants only)........................ ================================================================================ Single-Dish FITS Standards Workshop--(DRAFT) AGENDA -------------------- day 1 -- TUESDAY, OCTOBER 31, 1989 ------------------------ ==>Introduction: (8:45 am) Welcome and Introductory Remarks.......................H. Liszt History, Background, and Current Status of FITS Origins and Evolution of FITS..................D. Wells Remarks from the AAS WGAS......................R. Hanisch The IAU FITS Standards Committee...............P. Grosbol The NASA FITS Standards Office.................B. Schlesinger Use of FITS in High-Energy Astrophysics................S. Murray (Coffee) ==>Steps Toward a Single-Dish FITS Format: The 1986 Format........................................R. Maddelena Alternative: A Compact Binary Table Format.............D. Wells Other alternatives?....................................(Anyone) General Discussion of Design and Methodology............EVERYONE! (Lunch) ==>Specifying A Standard For Single Dish Data Exchange: Methodology/Architecture/Design Use of tables, random groups, other FITS mechanisms Vocabulary--Variable types; Usage of keywords/table headers Semantics--Meaning of keywords, definition of units World Coordinates--describing the velocity (or frequency) axis (Coffee) ==>Charting the Future, Part 1: For the immediate future Defining the crucial details Defining a working group to settle these details Construction of first-generation FITS readers and writers (Cocktails and Dinner) ================================================================================ Single-Dish FITS Standards Workshop--(DRAFT) AGENDA ------------------- day 2 -- WEDNESDAY, NOVEMBER 1, 1989------------------------ ==>Charting the Future, Part 2: (8:45 am) For the long-term Disseminating the products of the working group General implementation of the new standard Assuring general acceptance of the standard Assuring adherence to the new standard (coffee) ==>General Concerns Affecting the Software Performance of Single Dishes Presentations from various instruments.................? ==>Wrap-up And General Summary of the Workshop ================================================================================ From dishfits-request@fits.CX.NRAO.EDU Thu Oct 5 18:02:59 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA00824; Thu, 5 Oct 89 18:02:57 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA16314; Thu, 5 Oct 89 18:03:23 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA00743; Thu, 5 Oct 89 17:55:48 EDT Return-Path: Message-Id: <8910052150.AA13865@RA.STSCI.EDU> Status: RO From: (Bob Hanisch) Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@nrao.edu Subject: single dish data and FITS format Date: Thu, 5 Oct 89 16:50:51 EST Some Comments About Single-Dish FITS Format and Related FITS Issues Bob Hanisch, ST ScI October 1989 In preparation for the meeting in Green Bank later this month, I have written down some ideas concerning how to handle the special case of single dish radio astronomy data within the FITS format. This is being distributed in advance of the meeting so that attendees can think about the issues prior to arrival. Harvey Liszt distributed an e-mail message to all of us which contained a careful analysis of the header structures proposed by Stobie and Morgan. Unfortunately I think this analysis does not really address the general problem, i.e., how to manage the keyword space unique to a certain class of instruments, detectors, or a particular project. From the FITS point of view this is the critical issue, and most of the items discussed in the memo are the details that need to be worked out once the more general problem has been addressed. What Stobie and Morgan described in their original memo is a simple hierarchical keyword space. The presence of the hierarchy is denoted by the use of the SINGLDSH keyword, followed by a secondary hierarchy (GENERAL, NRAO-12M, etc.) which further defines the scope of the keyword definitions. The AIPS system has already implemented a hierarchical scheme of sorts, in that AIPS uses the FITS COMMENT and HISTORY records for storing information unique to AIPS. Non-AIPS FITS readers have no obligation to parse the contents of these cards beyond simply recording them with the other header information, or even ignoring them if they wish. The point here is to have a keyword space which can be hidden from FITS reading programs that do not understand or have no interest in some specialized set of keywords. The advantages of such a scheme include the fact that we need not have any community-wide consensus on the definitions of ALL FITS keywords. Rather, subgroups such as the single-dish radio astronomy community can agree upon their own conventions and be largely independent of other subdisciplines. If we agree that having some sort of hierarchical keyword naming scheme is worthwhile (and I think that this is NOT a foregone conclusion -- more comments on this later), the first thing to do is determine the best syntax for its implementation. I see several difficulties with the proposal of Stobie and Margon (and implicitly with the AIPS technique of hiding similar information behind COMMENT and HISTORY records): o The proposal is limited to a two-level hierarchy, a primary domain (SINGLDSH) and a secondary domain (GENERAL, NRAO-12M, etc.). This seems unnecessarily restrictive, although it is clearly simple to generalize to several more levels. o The keyword hierarchy takes up a lot of space on the header line, leaving little space for values or comments. This situation is exacerbated if deeper hierarchies are to be permitted. o FITS readers require special code to parse the header and retrieve the data values. o Everytime a new domain is introduced, FITS readers everywhere will have to be updated to recognize (and most often ignore) the new domain name. Some time ago I proposed implementing hierarchical keywords by REQUIRING that each hierarchical declaration actually begin with the keyword HIERARCH (this was during a FITS planning meeting held in Charlottesville in Jan. 1988, as I recall). The reasoning was that this is such an ugly keyword that no one else in their right mind would use it, thus it would serve as a flag to indicate that a special parser would have to go into action in order to retrieve the actual keyword name and value buried further to the right on the card. In retrospect, this plan suffers the same disadvantages mentioned above EXCEPT for the last one; by reserving a special keyword to introduce a hierarchical keyword domain, the problem of the arbitrary introduction of new domains is circumvented. However, the other problems, especially the lack of space on the header card, are not resolved. An alternative scheme has come to mind (based on an idea proposed by Doug Tody at the same meeting) which looks much more like simple nested do-loops in Fortran, or structure definition statements in any modern computer language. It would be based on begin/end statements for various keyword domains, and could be nested arbitrarily deep, accommodating things like SINGLDSH/GENERAL/NRAO/12METER/... or whatever. I propose to use keywords of the form KHBEGIN and KHEND to signify a keyword domain (KH = keyword hierarchy; I don't really care what the particular names are so long as we can agree on something unique and representative). For example, the syntax might be something like this (paying little attention to the actual column spacing): From dishfits-request@fits.CX.NRAO.EDU Mon Oct 9 17:01:11 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA05146; Mon, 9 Oct 89 17:01:08 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA26636; Mon, 9 Oct 89 17:02:06 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA05123; Mon, 9 Oct 89 16:55:43 EDT Return-Path: X-Vms-Mail-To: DISHFITS Message-Id: <891009165133.00004A45081@cvax.CV.NRAO.EDU> Status: RO From: HLISZT@NRAO.EDU Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Comments by R. Hanisch (St Sci) Date: Mon, 9 Oct 89 16:51:33 EDT ****************************************************************** The following was sent to NRAO by R. Hanisch (hanisch@stsci.edu or hanisch@scivax.span) ****************************************************************** From: STSCIC::HANISCH 5-OCT-1989 17:50:34.74 To: STOBIE CC: Subj: current version of FITS memo Some Comments About Single-Dish FITS Format and Related FITS Issues Bob Hanisch, ST ScI October 1989 In preparation for the meeting in Green Bank later this month, I have written down some ideas concerning how to handle the special case of single dish radio astronomy data within the FITS format. This is being distributed in advance of the meeting so that attendees can think about the issues prior to arrival. Harvey Liszt distributed an e-mail message to all of us which contained a careful analysis of the header structures proposed by Stobie and Morgan. Unfortunately I think this analysis does not really address the general problem, i.e., how to manage the keyword space unique to a certain class of instruments, detectors, or a particular project. From the FITS point of view this is the critical issue, and most of the items discussed in the memo are the details that need to be worked out once the more general problem has been addressed. What Stobie and Morgan described in their original memo is a simple hierarchical keyword space. The presence of the hierarchy is denoted by the use of the SINGLDSH keyword, followed by a secondary hierarchy (GENERAL, NRAO-12M, etc.) which further defines the scope of the keyword definitions. The AIPS system has already implemented a hierarchical scheme of sorts, in that AIPS uses the FITS COMMENT and HISTORY records for storing information unique to AIPS. Non-AIPS FITS readers have no obligation to parse the contents of these cards beyond simply recording them with the other header information, or even ignoring them if they wish. The point here is to have a keyword space which can be hidden from FITS reading programs that do not understand or have no interest in some specialized set of keywords. The advantages of such a scheme include the fact that we need not have any community-wide consensus on the definitions of ALL FITS keywords. Rather, subgroups such as the single-dish radio astronomy community can agree upon their own conventions and be largely independent of other subdisciplines. If we agree that having some sort of hierarchical keyword naming scheme is worthwhile (and I think that this is NOT a foregone conclusion -- more comments on this later), the first thing to do is determine the best syntax for its implementation. I see several difficulties with the proposal of Stobie and Margon (and implicitly with the AIPS technique of hiding similar information behind COMMENT and HISTORY records): o The proposal is limited to a two-level hierarchy, a primary domain (SINGLDSH) and a secondary domain (GENERAL, NRAO-12M, etc.). This seems unnecessarily restrictive, although it is clearly simple to generalize to several more levels. o The keyword hierarchy takes up a lot of space on the header line, leaving little space for values or comments. This situation is exacerbated if deeper hierarchies are to be permitted. o FITS readers require special code to parse the header and retrieve the data values. o Everytime a new domain is introduced, FITS readers everywhere will have to be updated to recognize (and most often ignore) the new domain name. Some time ago I proposed implementing hierarchical keywords by REQUIRING that each hierarchical declaration actually begin with the keyword HIERARCH (this was during a FITS planning meeting held in Charlottesville in Jan. 1988, as I recall). The reasoning was that this is such an ugly keyword that no one else in their right mind would use it, thus it would serve as a flag to indicate that a special parser would have to go into action in order to retrieve the actual keyword name and value buried further to the right on the card. In retrospect, this plan suffers the same disadvantages mentioned above EXCEPT for the last one; by reserving a special keyword to introduce a hierarchical keyword domain, the problem of the arbitrary introduction of new domains is circumvented. However, the other problems, especially the lack of space on the header card, are not resolved. An alternative scheme has come to mind (based on an idea proposed by Doug Tody at the same meeting) which looks much more like simple nested do-loops in Fortran, or structure definition statements in any modern computer language. It would be based on begin/end statements for various keyword domains, and could be nested arbitrarily deep, accommodating things like SINGLDSH/GENERAL/NRAO/12METER/... or whatever. I propose to use keywords of the form KHBEGIN and KHEND to signify a keyword domain (KH = keyword hierarchy; I don't really care what the particular names are so long as we can agree on something unique and representative). For example, the syntax might be something like this (paying little attention to the actual column spacing): From dwells@fits.CX.NRAO.EDU Wed Oct 11 17:56:50 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA08854; Wed, 11 Oct 89 17:56:48 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA02170; Wed, 11 Oct 89 17:56:20 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA08809; Wed, 11 Oct 89 17:38:09 EDT Return-Path: Message-Id: <8910112138.AA08803@fits.cx.nrao.edu> Status: RO From: dwells@fits.CX.NRAO.EDU (Don Wells) Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Re: single dish data and FITS format Date: Wed, 11 Oct 89 17:38:05 EDT From hanisch@stsci.edu Fri Oct 6 11:24:52 1989 Subject: my memo seems to have been chopped off for some people Don, My memo on hierarchical keywords seems to have been truncated for some recipients. Bill Cotton reported to me that his copy had been chopped, and I just checked mine and it also is chopped, at the same place. Could I ask you to check on the mail exploder and see if it might be the culprit? Here's the text. Please send it out to the exploder, if you would. Have to head off to the airport in the next few minutes. Thanks, Bob. [see technical explanation at end -- DWells] Some Comments About Single-Dish FITS Format and Related FITS Issues Bob Hanisch, ST ScI October 1989 In preparation for the meeting in Green Bank later this month, I have written down some ideas concerning how to handle the special case of single dish radio astronomy data within the FITS format. This is being distributed in advance of the meeting so that attendees can think about the issues prior to arrival. Harvey Liszt distributed an e-mail message to all of us which contained a careful analysis of the header structures proposed by Stobie and Morgan. Unfortunately I think this analysis does not really address the general problem, i.e., how to manage the keyword space unique to a certain class of instruments, detectors, or a particular project. From the FITS point of view this is the critical issue, and most of the items discussed in the memo are the details that need to be worked out once the more general problem has been addressed. What Stobie and Morgan described in their original memo is a simple hierarchical keyword space. The presence of the hierarchy is denoted by the use of the SINGLDSH keyword, followed by a secondary hierarchy (GENERAL, NRAO-12M, etc.) which further defines the scope of the keyword definitions. The AIPS system has already implemented a hierarchical scheme of sorts, in that AIPS uses the FITS COMMENT and HISTORY records for storing information unique to AIPS. Non-AIPS FITS readers have no obligation to parse the contents of these cards beyond simply recording them with the other header information, or even ignoring them if they wish. The point here is to have a keyword space which can be hidden from FITS reading programs that do not understand or have no interest in some specialized set of keywords. The advantages of such a scheme include the fact that we need not have any community-wide consensus on the definitions of ALL FITS keywords. Rather, subgroups such as the single-dish radio astronomy community can agree upon their own conventions and be largely independent of other subdisciplines. If we agree that having some sort of hierarchical keyword naming scheme is worthwhile (and I think that this is NOT a foregone conclusion -- more comments on this later), the first thing to do is determine the best syntax for its implementation. I see several difficulties with the proposal of Stobie and Margon (and implicitly with the AIPS technique of hiding similar information behind COMMENT and HISTORY records): o The proposal is limited to a two-level hierarchy, a primary domain (SINGLDSH) and a secondary domain (GENERAL, NRAO-12M, etc.). This seems unnecessarily restrictive, although it is clearly simple to generalize to several more levels. o The keyword hierarchy takes up a lot of space on the header line, leaving little space for values or comments. This situation is exacerbated if deeper hierarchies are to be permitted. o FITS readers require special code to parse the header and retrieve the data values. o Everytime a new domain is introduced, FITS readers everywhere will have to be updated to recognize (and most often ignore) the new domain name. Some time ago I proposed implementing hierarchical keywords by REQUIRING that each hierarchical declaration actually begin with the keyword HIERARCH (this was during a FITS planning meeting held in Charlottesville in Jan. 1988, as I recall). The reasoning was that this is such an ugly keyword that no one else in their right mind would use it, thus it would serve as a flag to indicate that a special parser would have to go into action in order to retrieve the actual keyword name and value buried further to the right on the card. In retrospect, this plan suffers the same disadvantages mentioned above EXCEPT for the last one; by reserving a special keyword to introduce a hierarchical keyword domain, the problem of the arbitrary introduction of new domains is circumvented. However, the other problems, especially the lack of space on the header card, are not resolved. An alternative scheme has come to mind (based on an idea proposed by Doug Tody at the same meeting) which looks much more like simple nested do-loops in Fortran, or structure definition statements in any modern computer language. It would be based on begin/end statements for various keyword domains, and could be nested arbitrarily deep, accommodating things like SINGLDSH/GENERAL/NRAO/12METER/... or whatever. I propose to use keywords of the form KHBEGIN and KHEND to signify a keyword domain (KH = keyword hierarchy; I don't really care what the particular names are so long as we can agree on something unique and representative). For example, the syntax might be something like this (paying little attention to the actual column spacing): . . (standard FITS header) . . KHBEGIN = 'SINGLDSH' / begin a keyword hierarchy called SINGLDSH STATION = 'NRAO-12M' / one or more keywords unique to this UNIQKW2 = 1.2345678 / hierarachy . . KHBEGIN = 'GENERAL' / second level of hierarchy UNIQKW3 = 9.8765432 / one or more keywords unique to this . / second level of the hierarchy . . . (other KHBEGIN keywords define additional levels of the hierarchy, . as needed) KHEND = 'GENERAL' / close out the GENERAL keyword list KHEND = 'SINGLDSH' / close out the SINGLDSH keyword list . . Now, there are some things that I haven't thought through that carefully yet, like whether or not the KHBEGIN and KHEND keywords should have a numeral attached (minimizing the number of keywords with the same name and perhaps making the parsing of the hierarchy a bit more fool-proof). Also, there's the question of whether you would want to allow the full flexibility of going in and out of a particular domain. That is, should this sort of thing be permitted:? KHBEGIN = 'SINGLDSH' / begin a keyword hierarchy called SINGLDSH . . KHBEGIN = 'GENERAL' / second level of hierarchy . . KHEND = 'GENERAL' / close out the GENERAL keyword list . . KHBEGIN = 'NRAO-12M' / another second level hierarchy . . KHEND = 'NRAO-12M' . . KHBEGIN = 'GENERAL' / reopen the GENERAL list . . KHEND = 'GENERAL' KHEND = 'SINGLDSH' / close out the SINGLDSH keyword list This is not a big deal, and maybe it would simply be a matter of style whether or not this would be allowed or just discouraged. However, there are a couple of major advantages of this approach to managing the keyword space: o The keywords within the hierarchy can all be parsed with standard FITS readers. o There is no wasted space on the individual keyword cards, allowing more complete use of the comment field. If there are enough `private' keywords to warrant setting up a hierarchy, the overhead of the BEGIN and END cards is small. o The logic for skipping over the `private' keyword definitions is straightforward: upon encountering a KHBEGIN card, the reader would be free to skip parsing other cards until the corresponding KHEND card was found. In order to simplify this somewhat, we may wish to limit the depth of the hierarchy, say to 9 levels (that should be enough for just about everyone, and would allow the formation of numbered hierarchies, KHBEGIN1, KHBEGIN2, etc., as described above). There is clearly redundant information in using both the numbering scheme AND the value field of the BEGIN/END cards. o The definition of new keyword hierarchies (SINGLDSH, INTERF, AIPS, CCD, HST, etc.) does not require FITS readers to be updated. A one-time update is all that is necessary, and even this is not essential for existing FITS readers to work since they could simply ignore all of this stuff, with no harm done. Thus, if the need for private keyword definition spaces is genuine, a scheme such as that proposed here might be a more viable one for implementation than that previously proposed. However, the question remains as to whether such private keyword domains are even required. Just to review this situation a bit, the prime motivation for having a hierarchical keyword space of some sort is so that various projects, disciplines, etc., can define any set of new keywords without having to worry about name collisions with other such groups. But is this really a concern? Does it really matter if a CCD data set and a single-dish radio data set have inconsistent definitions of BIAS? or REFPT? or DELTAX? I guess it depends on what level of interoperability we expect to achieve with FITS. In a practical sense I think we all recognize that the idea of establishing some sort of global dictionary for FITS keywords is impossible. The hierarchical scheme says, ok, we can't do it so let's hide all of our private definitions. The alternative says, ok, we can't do it but it doesn't really matter because the types of data we're transferring are so heterogeneous. (By the way, I wish to point out something about the practice of defining new FITS keywords. There seems to have arisen some idea that it is forbidden to define keywords other than those mentioned in the series of FITS standards papers. Nonsense. Reread Paper I, p. 366, middle of the 2nd column: "Note that the two long keywords, INSTRUMEN and TELESCOP are names which have been truncated after 8 characters. We suggest this as a good convention for the creation of new keywords." Whether they intended it or not, Don, Eric, and Ron left the door wide open for the creation of new keywords and even recommended how it should be done.) What does matter, I think, is to have a small set of reserved keywords that all FITS readers might reasonably be expected to understand, or at least not interfere with. FITS Paper I defines the fundamental set (SIMPLE, BITPIX, NAXIS, NAXISi (i=1,NAXIS), and END) and an optional set (BSCALE, BZERO, BUNIT, CRVALi, etc.). One solution to this problem would be for the various FITS committees to review Paper I and agree upon such a set of reserved keywords, and then let various projects use the rest of the keyword space to their hearts' desires. This is, after all, the situation we have now and maybe it's not so bad. Anyway, wait until you all see HST data! In any case, I think that a clarification and standardization of some small set of keywords is desirable regardless of whether we adopt a hierarchical naming convention project or discipline dependent keywords. Well, those are my thoughts at the moment. I've gone on a bit longer than I thought, but I hope that those of you who've stuck with me to this point have got some food for thought and that we can have a lively discussion about this in Green Bank. See you all there. ======================= non-FITS technical discussion below: ============= Why did Bob's message on 05 October get truncated? The SMTP (Simple Mail Transfer Protocol) of the Internet terminates a message by a line that has *only* one period (.) on it. This rule allows the mail protocol to work the same way on top of a variety of lower protocols, character sets and OSes. So, what does an SMTP mailer do if a message contains a text line with only a period? Why, double it, of course! And, of course, a mail daemon which receives such a message will change the double period back to a single period before delivery to the customer mailbox. Messages containing single-period lines are rare. Bob's message at 1755EDT last Thursday 05 October contains the first such lines I have seen in several years. His message was truncated *exactly* at the first occurrence of such a line, and so we know that the problem may be that some mail daemon somewhere failed to double the period on transmission. At present I am unable to say with certainty which of three different computers, one at STScI and two at NRAO, was the guilty party. I have added tabs before the periods in this copy to defeat the RFC822 rule. -Don From dishfits-request@fits.CX.NRAO.EDU Fri Oct 13 20:34:52 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA02034; Fri, 13 Oct 89 20:34:47 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA02650; Fri, 13 Oct 89 20:34:18 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA02026; Fri, 13 Oct 89 20:08:14 EDT Return-Path: X-Vms-Mail-To: DISHFITS AT NRAO Message-Id: <891013195606.00000B24031@cvax.CV.NRAO.EDU> Status: RO From: "GUILLOTE@FRGAG51" Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Single-dish FITS Date: Fri, 13 Oct 89 19:56:06 EDT Date sent: 13-OCT-1989 13:52:50.84 To: DISHFITS@NRAO Single-Dish FITS Format Stephane Guilloteau (1) and Thierry Forveille (2) (1) Institut de Radio Astronomie Millimetrique (IRAM) 300 Rue de la Piscine F-38406 Saint Martin d'Here FRANCE E-Mail "GUILLOTE@FRGAG51.BITNET" (2) Groupe d'Astrophysique de Grenoble (GAG) Observatoire de Grenoble BP 53 X F-38041 Grenoble Cedex FRANCE E-Mail "FORVEILL@FRGAG51.BITNET" Dear collegues, In the previous mails delivered through the DISHFITS exploder, the goal of using FITS for single-dish data was not clearly exposed. We present here an (possibly obvious, and certainly not exhaustive) analysis of the basic needs and goals, leading to a proposal. FITS is designed for "transport" of images or other data. For single-dish data, we may consider several cases of "transport" 0) Transport between the same analysis program, running on identical computers, on different sites. FITS is not required in this case 1) Transport between the same analysis program, running on different computers. FITS may be required, since the binary data format can be different on the two computers. Using FITS instead of dedicated conversion programs avoids the need of dedicated programs on each type of computer 2) Transport between different analysis programs, probably running on different computers. 3) As a subset of 2), possible transport between dedicated single-dish analysis programs and "general" image processing systems (AIPS or others). 4) Archiving. FITS may be desirable in this case, as it guarantees the readability on any computer later on. Each case impose some constraint on the design of a single-dish (or image) FITS format. Case 1) and 4) require that ALL the information be written on FITS tape. This can be handled using dedicated keywords, or, as done for AIPS, substructures in the COMMENT and HISTORY cards. Whatever the choice of implementation, keywords are required matching EXACTLY the data format, with a simple provision for future extensions. IRAM and other observatories are currently using the CLASS program for single-dish data analysis. The data structure for this program is segmented in "Sections" describing some aspects of the data. The currently used sections is described in Appendix in this MAIL. Case 2), considered here as exchange between two single-dish data analysis programs, requires some agreement upon a reasonable subset of keywords in order to preserve most (if not all) information in the transport. Some loss of information seems unavoidable, since not all analysis programs have a one to one correspondence in header parameters. Case 3) is even more tricky. It is likely that a lot of information will be lost in this case. However, a minimal requirement should be that the data can be correctly transferred. Following the original FITS rule, this would imply putting the data in the standard data area, NOT in a table extension. The minimal subset of keywords to be defined would then be NAXIS, NAXISi and BITPIX. However, most image processing are (or should be) able to process coordinates properly, including frequency information, and it would be highly desirable to transfer in the most general way this information too. Looking a little bit closer to what is a single-dish data set, we may consider two cases : - Spectra - Continuum drifts Spectra can be considered as 1-dimensional subsets of 3-D data cubes. AIPS-style FITS headers could be readily used to described completely REDUCED single-dish spectra. As an example, consider the following FITS header, which has been produced by the CLASS to FITS conversion program : ---------------------------------------------------------------------- SIMPLE = T / BITPIX = 16 / NAXIS = 4 / NAXIS1 = 512 / NAXIS2 = 1 / NAXIS3 = 1 / NAXIS4 = 1 / BLANK = 32767 / Blanking value BSCALE = 0.1595546564204E-05 / BZERO = 0.4077937081456E-01 / DATAMIN = -0.1150350086391E-01 / DATAMAX = 0.9305904805660E-01 / BLOCKED = T / BUNIT = 'K ' / CTYPE1 = 'FREQ ' / CRVAL1 = 0.0000000000000E+00 / Offset frequency CDELT1 = 0.1220703125000E+05 / Frequency resolution CRPIX1 = 0.2565000000000E+03 / CTYPE2 = 'RA---GLS' / EQUINOX = 0.1950000000000E+04 / CRVAL2 = 0.5150368630886E+02 / CDELT2 = 0.8333333046443E-01 / CRPIX2 = 0.0000000000000E+00 / CTYPE3 = 'DEC--GLS' / CRVAL3 = 0.3108135491610E+02 / CDELT3 = 0.8333333046443E-01 / CRPIX3 = 0.0000000000000E+00 / CTYPE4 = 'STOKES ' / CRVAL4 = 1.0000000000000 / CDELT4 = 0.0000000000000 / CRPIX4 = 0.0000000000000 / TELESCOP= '100-M ' / OBJECT = 'NGC1333 ' / LINE = 'NH3(1,1) ' / Line name RESTFREQ= 0.2369449499989E+11 / Rest frequency VLSR = 0.6500000000000E+04 / Velocity of reference channel ORIGIN = 'CLASS-Grenoble ' / DATE = '13/10/89' / Date written DATE-OBS= ' 1/ 9/84' / Date observed DATE-RED= '27/ 6/89' / Date reduced END ---------------------------------------------------------------------- Note that CRVAL2 and CRVAL3 are coordinates at the point of tangency, as should be, and CDELT2 CDELT3 represent here the offsets of the observed point with respect to this reference direction. Keywords CD00i00i may be used instead. The frequency axis is defined as an offset to a rest frequency given in keyword RESTFREQ, and the corresponding source velocity is given in keyword VLSR. (It may be better to specify the frequency axis as effective observing frequency, and use a keyword VELO-LSR for the source velocity). This header is very close to a standard AIPS FITS header produced for a 3-D VLM cube for example, and no "vital" information has been lost if the data is already reduced. We are not expert in "image" processing, but presumably such a FITS header could be understood by any FITS reading program without major loss of information (the only two non standard keywords are RESTFREQ and VLSR). Continuum drifts are handled in a similar way: the frequency axis has dimension 1, and one sky coordinate axis has dimension equal to the number of data points. It may be interesting to define a very special "projection code" for drifts in horizontal coordinates, which are frequently used to check the telescope pointing, e.g. something like -HOR. Turning back to points 1) and 4), which imply in fact essentially the same constraints, we believe that a hierarchical structure as described by Bob Hanisch essentially fulfills all the requirements. As an example, here is a (quasi) FITS header part which corresponds to the data structure used in CLASS (See Appendix). Reference to notes are numbered (*i). We did not bother editing a real FITS header, but only the keywords are mentionned. KHBEGIN = 'CLASS' / (*1) VERSION = 'SEP-1989' / Date of version KHBEGIN = 'POINTER' / (*2) GENERAL = 20 / POSITION= 11 / SPECTRO = 15 / Spectroscopic data FSWITCH = 5 / for NPHASE = 1 DRIFT = 0 / Not present BEAM = 0 CALIBRAT= 19 / May be shorter SKYDIP = 0 GAUSS = 9 / for NLINE = 1 HFS = 0 SHELL = 0 CONTINUU= 0 HISTORY = 9 / Sequence = 4 PLOT = 4 BASE = 8 / For 2 windows XCOORD = 0 / (*5) KHEND COMMENT * OBS.GENERAL : General parameters, always present KHBEGIN ='GENERAL' ! (mgen=9,mgen2=11) NUMBER = ! rnum ! Observation number VERSION = ! rver ! Version number TELESCOP= ! rteles(3) ! Telescope name DATE_OBS= ! rdobs ! Date of observation DATE_RED= ! rdred ! Date of reduction SYSTEM = ! rtypec ! Type of coordinates KIND = ! rkind ! Type of data QUALITY = ! rqual ! Quality of data SCAN = ! rscan ! Scan number UT_TIME = ! rut ! UT of observation LST_TIME= ! rst ! LST of observation AZIMUTH = ! raz ! Azimuth ELEVATIO= ! rel ! Elevation TAU = ! rtau ! Opacity TSYS = ! rtsys ! System temperature INTEGRAT= ! rtime ! Integration time KHEND COMMENT * OBS.POSITION : Position information. KHBEGIN ='POSITION' ! (mpos=11) SOURCE = ! rsourc(3) ! Source name EPOCH = ! repoch ! Epoch of coordinates LAMBDA = ! rlam ! Lambda BETA = ! rbet ! Beta LAMBDA_O= ! rlamof ! Offset in Lambda BETA_OFF= ! rbetof ! Offset in Beta PROJECTI= ! rproj ! Projection system KHEND COMMENT * OBS.SPECTRO : Spectroscopic information (for spectra) KHBEGIN ='SPECTRO' ! (mspec=15) LINE = ! rline(3) ! Line name REST = ! rrestf ! Rest frequency NCHAN = ! rnchan ! Number of channels REFERENC= ! rrchan ! Reference channel FRES = ! rfres ! Frequency resolution FOFF = ! rfoff ! Frequency offset VRES = ! rvres ! Velocity resolution VOFF = ! rvoff ! Velocity at reference channel BAD = ! rbad ! Blanking value IMAGE = ! rimage ! Image frequency VELOCITY= ! rvtype ! Type of velocity KHEND COMMENT * OBS.BASE : Baseline information (for spectra of drifts) KHBEGIN ='BASE' ! (mwind=20,mbase=4+2*mwind) DEGREE = ! rdeg ! Degree of last baseline SIGMA = ! rsigfi ! Sigma AREA = ! raire ! Area under windows NWINDOW = ! rnwind ! Number of line windows WIND1(NWINDOW) ! rw1(mwind)! Lower limits of windows (*3) WIND2(NWINDOW) ! rw2(mwind)! Upper limits of windows KHEND COMMENT * OBS.HISTORY : Scan numbers of initial observations KHBEGIN ='HISTORY' ! (mseq=100,morig=2*mseq+1) SEQUENCE= ! rnseq ! Number of sequences START(SEQUENCE) ! rstart(mseq) ! Start can number of seq. (*4) REND(SEQUENCE) ! rend(mseq) ! End scan number of seq. KHEND COMMENT * OBS.PLOT : Default plotting limits. KHBEGIN = 'PLOT' ! (mplot=4) YMIN = ! ramin ! Min Y value plotted YMAX = ! ramax ! Max Y value plotted XMIN = ! rvmin ! Min X value plotted XMAX = ! rvmax ! Max X value plotted KHEND COMMENT * OBS.CALIBRATION : Calibration parameters KHBEGIN = 'CALIBRATION' ! (mcalib=19) BEAM_EFF= ! rbeeff ! Beam efficiency FORW_EFF= ! rfoeff ! Forward efficiency GAIN_IM = ! rgaini ! Image/Signal gain ratio H2OMM = ! rh2omm ! MM of water vapor PAMB = ! rpamb ! Ambient pressure (hPa) TAMB = ! rtamb ! Ambient temperature (K) TATMSIG = ! rtatms ! Atmosphere temp. signal band TCHOP = ! rtchop ! Chopper temperature TCOLD = ! rtcold ! Cold load temperature TAUSIG = ! rtaus ! Opacity signal band TAUIMA = ! rtaui ! Opacity image band TATMIMA = ! rtatmi ! Atmosphere temp. image band TREC = ! rtrec ! Receiver temperature MODE = ! rcmode ! Calibration mode FACTOR = ! ratfac ! Applied calibration factor ALTITUDE= ! ralti ! Site elevation COUNTS(3) ! rcount(3) ! Power of Atm., Chopp., Cold KHEND COMMENT * OBS.GAUSS : Gauss fit results (for spectra or drifts) KHBEGIN = 'GAUSS' ! (MXGAUS=5,mfit=3+6*MXGAUS) NLINE = ! rnline ! Number of components SIGMA_BA= ! rsigba ! Sigma on base SIGMA_RA= ! rsigra ! Sigma on line FIT(3*NLINE) ! rnfit(3*MXGAUS) ! Fit results ERR(3*NLINE) ! rnerr(3*MXGAUS) ! Errors KHEND KHEND = 'CLASS' / End CLASS hierarchy ---------------------------------------------------------------------- (*1) Define a CLASS hierarchy (*2) This structure is not required since the parameters can be derived from a sequential reading of the following ones. (*3) Here appears a rather general problem : the definition of arrays into FITS. Common practice has been to use an index as in NAXISi for example. But this is limited to small arrays in practice (size < 10 probably). In the list of original scan numbers (*4), the size may be larger. This problem is fairly general and here is an opportunity to address it. A possibility would be to use a single (non-indexed) keyword. If the count has been previously defined, this solution is unambigous: auto increment should simply be used by the reading program. The two following examples would be valid, and correspond to the following list of original scans added to produced the current one : 100 to 110, 112 to 120, 345 to 360, 363 and 364, 400. KHBEGIN ='HISTORY' SEQUENCE= 5 START = 100 REND = 110 START = 112 REND = 120 START = 345 REND = 360 START = 363 REND = 364 START = 400 REND = 400 KHEND KHBEGIN ='HISTORY' SEQUENCE= 5 START = 100 START = 112 START = 345 START = 363 START = 400 REND = 110 REND = 120 REND = 360 REND = 364 REND = 400 KHEND However this solution is very space consuming for long arrays. Should table-like data be used for those array keywords? If so, how could it be logically placed if hierarchies are implemented as suggested by Bob Hanisch ? Could we use something more compact such as KHBEGIN ='HISTORY' SEQUENCE= 5 START = 100, 112, 345, 363, 400 REND = 110, 120, 360, 364, 400 SEQUENCE= 1 START = 500 REND = 510 KHEND This is non standard FITS in terms of alignment of formatted variables, but can be easily read on any computer using a list directed I/O Fortran format. The number of values in each line is defined by keyword SEQUENCE here, so that the total number must be computed by adding all values of the SEQUENCE keyword (note that the name SEQUENCE is specific to the example. Other names would apply to other cases). (*5) XCOORD is used in CLASS for data (spectra or drifts) which are not regularly sampled. It has been used for example to store in CLASS format the IRAS LRS survey. The corresponding section contains a unit for the array coordinates (integer code), followed by the array of coordinates of the samples (here "coordinates" may be frequency of course). This problem of irregularly sampled data may be important in single-dish data as shown by the IRAS LRS example. A logical solution to handle that in FITS format is the use of standard extension tables. In so far, we have not discussed the fate of multi-backend and/or multi-beam observations. For multi-backend, the use of AIPS-style FITS header seems to imply to separate each backend data set in a different FITS file. An a priori alternative would be to declare the frequency axis as randomly sampled, but we then turn into several difficulties: 1) for reduced data, considering regularly sampled data as random ones is obviously a lack of simplicity, and probably most processing systems are not even able to do so (3_D VLM data cubes in AIPS with random velociy axis?) 2) for raw data, it is likely that the calibration parameters depend upon the backend. A specific structure would be required for each backend. A different FITS file for each backend is much simpler. The only disadvantage is the loss of storage space, but the cost (and volume) of storage is decreasing every day. For multi-beam, the problem may be different, because the relative beam positions are intrinsically random parameters, while the calibration parameters are mostly identical for all beams. A suggestion would be the use of GROUP data structure like in UVFITS. A few parameters must be precised for each beam however (beam efficiency and beam sizes, and possibly a scaling factor representing the average gain of the backend part used for this beam). Using GROUP data structure may also be possible for reduced (calibrated) data, allowing to store a complete "random" map (regularly sampled maps should use the normal axis coordinates). For raw data, this probably does not make sense, as many calibration parameters will be valid only for a few observations. ---------------------------------------------------------------------- In summary, following those guidelines, our proposal is (in priority order) 1) Always put the data in the FITS data area, NOT in an extension. 2) Put all important informations in a "standard" FITS header, based on the one used in AIPS for example. This "standard" header must be general enough to be understood by ALL FITS reading programs. 3) Adopt the Bob Hanisch proposal for hierarchy implementation and add a list of hierarchic keywords to describe completely the other informations. For simplicity reasons, this list may duplicate informations already present in the "standard" header. 4) Use table extensions for the coordinates of non regularly sampled data. 5) Use GROUP data structure for multi-beam observations, and possibly also for complete reduced "random" maps. 6) Find a solution for "array" keywords; we only suggested possible alternatives. 7) If possible, agree upon names for the hierarchical keywords, in order to ease data exchange between different single-dish reduction programs. (*) This does not preclude use of specific keywords for each analysis program, specially for data archive. Comments please... (*) This is the lowest priority for us, because our FITS reading program has the capability to define equivalent names for a keyword. This capability is easy to implement in any program, and already saved us a lot of additional programming time. ---------------------------------------------------------------------- Turning to the 6 questions asked by W.R. Burns in his original letter, our answers are : 1) CLASS (Continuum and Line Analysis Simple System) for single-dish data, CLIC (Continuum and Line Interferometer Calibration) for IRAM interferometer data, with a data format which is an extension of the CLASS format GILDAS (Grenoble Image and Line Data Analysis Software) for image processing. and AIPS as a reference in interferometry. 2) CLASS CLIC and GILDAS are developped in collaboration between IRAM and GAG, and the software is now used on many sites. 3) External support, limited to bug repairs by lack of manpower. Enhancement done in-house on a "urgent need" basis. In house software works only under VMS up to now. 4) Hardware : fast workstations, probably UNIX Software : UNIX conversion, may be X-windows, upgrade of image processing to handle more specifically IRAM Plateau de Bure data, and more homogenously 30-m and Plateau de Bure data. 5) We use AIPS, and distribute our software. Sharing code with some standardization would seem unpractical given our limited manpower (however if anybody is interested by our small code to get equivalent names, it can be done of course!). 6) Of course, single dish data could make use of other packages. This is precisely the goal of our proposal points 1 and 2. PS: Stephane Guilloteau is unable to attend the meeting, and Thierry Forveille will be the IRAM representative. ---------------------------------------------------------------------- Appendix : data structure (in VAX-VMS FORTRAN syntax) used for single-dish data in CLASS format, with corresponding standard FORTRAN variable names and physical meaning: * STRUCTURE /OBSERVATION_DESCR/ * * OBS.POINTER : Length of sections. If 0, section not present. STRUCTURE /POINTER_DESCR/ POINTER INTEGER GENERAL INTEGER POSITION INTEGER SPECTRO INTEGER FSWITCH INTEGER DRIFT INTEGER BEAM INTEGER CALIBRATION INTEGER SKYDIP INTEGER GAUSS INTEGER HFS INTEGER SHELL INTEGER CONTINUUM INTEGER HISTORY INTEGER PLOT INTEGER BASE INTEGER XCOORD END STRUCTURE * * Section -2 /CRPAR/ * OBS.GENERAL : General parameters, always present STRUCTURE /GENERAL_DESCR/ GENERAL ! (mgen=9,mgen2=11) INTEGER NUMBER ! rnum ! Observation number INTEGER VERSION ! rver ! Version number CHARACTER*12 TELESCOPE ! rteles(3) ! Telescope name INTEGER DATE_OBS ! rdobs ! Date of observatio n INTEGER DATE_RED ! rdred ! Date of reduction INTEGER SYSTEM ! rtypec ! Type of coordinate s INTEGER KIND ! rkind ! Type of data INTEGER QUALITY ! rqual ! Quality of data INTEGER SCAN ! rscan ! Scan number REAL*8 UT_TIME ! rut ! UT of observation REAL*8 LST_TIME ! rst ! LST of observation REAL*4 AZIMUTH ! raz ! Azimuth REAL*4 ELEVATION ! rel ! Elevation REAL*4 TAU ! rtau ! Opacity REAL*4 TSYS ! rtsys ! System temperature REAL*4 INTEGRATION ! rtime ! Integration time END STRUCTURE * * Section -3 * OBS.POSITION : Position information. STRUCTURE /POSITION_DESCR/ POSITION ! (mpos=11) CHARACTER*12 SOURCE ! rsourc(3) ! Source name REAL*4 EPOCH ! repoch ! Epoch of coordinat es REAL*8 LAMBDA ! rlam ! Lambda REAL*8 BETA ! rbet ! Beta REAL*4 LAMBDA_OFF ! rlamof ! Offset in Lambda REAL*4 BETA_OFF ! rbetof ! Offset in Beta INTEGER PROJECTION ! rproj ! Projection system END STRUCTURE * * Section -4 * OBS.SPECTRO : Spectroscopic information (for spectra) STRUCTURE /SPECTRO_DESCR/ SPECTRO ! (mspec=15) CHARACTER*12 LINE ! rline(3) ! Line name REAL*8 REST ! rrestf ! Rest frequency INTEGER NCHAN ! rnchan ! Number of channels REAL*4 REFERENCE ! rrchan ! Reference channels REAL*4 FRES ! rfres ! Frequency resoluti on REAL*4 FOFF ! rfoff ! Frequency offset REAL*4 VRES ! rvres ! Velocity resolutio n REAL*4 VOFF ! rvoff ! Velocity at refere nce channel REAL*4 BAD ! rbad ! Blanking value REAL*8 IMAGE ! rimage ! Image frequency INTEGER VELOCITY ! rvtype ! Type of velocity END STRUCTURE * * Section -5 * OBS.BASE : Baseline information (for spectra of drifts) STRUCTURE /BASE_DESCR/ BASE ! (mwind=20,mbase=4+2*mwind) INTEGER DEGREE ! rdeg ! Degree of last bas eline REAL*4 SIGMA ! rsigfi ! Sigma REAL*4 AREA ! raire ! Area under windows INTEGER NWINDOW ! rnwind ! Number of line win dows REAL*4 WIND1(MWIND) ! rw1(mwind)! Lower limits of wi ndows REAL*4 WIND2(MWIND) ! rw2(mwind)! Upper limits of wi ndows END STRUCTURE * * Section -6 * OBS.HISTORY : Scan numbers of initial observations STRUCTURE /HISTORY_DESCR/ HISTORY ! (mseq=100,morig=2*mseq+1) INTEGER SEQUENCE ! rnseq ! Number of sequence s INTEGER START(MSEQ) ! rstart(mseq) ! Start can numbe r of seq. INTEGER END(MSEQ) ! rend(mseq) ! End scan number of seq. END STRUCTURE * * Section -7 * OBS.PLOT : Default plotting limits. STRUCTURE /PLOT_DESCR/ PLOT ! (mplot=4) REAL*4 YMIN ! ramin ! Min Y value plotte d REAL*4 YMAX ! ramax ! Max Y value plotte d REAL*4 XMIN ! rvmin ! Min X value plotte d REAL*4 XMAX ! rvmax ! Max X value plotte d END STRUCTURE * * Section -8 * OBS.FSWITCH : Frequency switching information (for spectra) STRUCTURE /FSWITCH_DESCR/ FSWITCH ! (mxphas=5,mfsw=1+4*mxphas) INTEGER NPHASE ! rnphas ! Number of phases REAL*8 DECALAGE(MXPHAS) ! rdecal(mxphas) ! Frequency off sets REAL*4 DUREE(MXPHAS) ! rduree(mxphas) ! Time per phas e REAL*4 POIDS(MXPHAS) ! rpoids(mxphas) ! Weight of eac h phase END STRUCTURE * * Section -14 * OBS.CALIBRATION : Calibration parameters STRUCTURE /CALIBRATION_DESCR/ CALIBRATION ! (mcalib=19) REAL*4 BEAM_EFF ! rbeeff ! Beam efficiency REAL*4 FORW_EFF ! rfoeff ! Forward efficiency REAL*4 GAIN_IM ! rgaini ! Image/Signal gain ratio REAL*4 H2OMM ! rh2omm ! MM of water vapor REAL*4 PAMB ! rpamb ! Ambient pressure ( hPa) REAL*4 TAMB ! rtamb ! Ambient temperatur e (K) REAL*4 TATMSIG ! rtatms ! Atmosphere temp. s ignal band REAL*4 TCHOP ! rtchop ! Chopper temperatur e REAL*4 TCOLD ! rtcold ! Cold load temperat ure REAL*4 TAUSIG ! rtaus ! Opacity signal ban d REAL*4 TAUIMA ! rtaui ! Opacity image band REAL*4 TATMIMA ! rtatmi ! Atmosphere temp. i mage band REAL*4 TREC ! rtrec ! Receiver temperatu re INTEGER MODE ! rcmode ! Calibration mode REAL*4 FACTOR ! ratfac ! Applied calibratio n factor REAL*4 ALTITUDE ! ralti ! Site elevation REAL*4 COUNTS(3) ! rcount(3) ! Power of Atm., Cho pp., Cold END STRUCTURE * * Section -16 * OBS.SKYDIP : For Skydips observations. No associated data. STRUCTURE /SKYDIP_DESCR/ SKYDIP ! (msky=10,mskydip=10+4*msky) CHARACTER*12 LINE ! rsline(3) ! Line name REAL*8 REST ! rsrest ! Rest frequency REAL*8 IMAGE ! rsimag ! Image frequency INTEGER NSKY ! rnsky ! Number of points o n sky INTEGER NCHOP ! rnchop ! - - - - chopper INTEGER NCOLD ! rncold ! - - - - cold load REAL*4 ELEVATION(MSKY) ! relev(msky) ! Elevations REAL*4 EMISSION (MSKY) ! remiss(msky) ! Power on sky REAL*4 CHOPPER (MSKY) ! rchopp(msky) ! Power on choppe r REAL*4 COLD (MSKY) ! rcold(msky) ! Power on cold l oad END STRUCTURE * * Section -9 /CRGAUS/ * OBS.GAUSS : Gauss fit results (for spectra or drifts) STRUCTURE /GAUSS_DESCR/ GAUSS ! (MXGAUS=5,mfit=3+6*MXGAUS) INTEGER NLINE ! rnline ! Number of componen ts REAL*4 SIGMA_BASE ! rsigba ! Sigma on base REAL*4 SIGMA_RAIE ! rsigra ! Sigma on line REAL*4 FIT(3*MXGAUS) ! rnfit(3*MXGAUS) ! Fit results REAL*4 ERR(3*MXGAUS) ! rnerr(3*MXGAUS) ! Errors END STRUCTURE * * Section -12 /CRSHEL/ * OBS.SHELL : "Stellar shell" profile fit results (for spectra) STRUCTURE /SHELL_DESCR/ SHELL ! (mshell=43) INTEGER NLINE ! rlshel ! Number of componen ts REAL*4 SIGMA_BASE ! rbshel ! Sigma on base REAL*4 SIGMA_RAIE ! rrshel ! Sigma on line REAL*4 FIT(20) ! rnshel(20)! Fit results REAL*4 ERR(20) ! reshel(20)! Errors END STRUCTURE * * Section -13 /CRNH3/ * OBS.HFS : "Hyperfine Structure" profile fit results (for spectra) * (e.g. NH3, HCN) STRUCTURE /HFS_DESCR/ HFS ! (mnh3=27) INTEGER NLINE ! rlnh3 ! Number of componen ts REAL*4 SIGMA_BASE ! rbnh3 ! Sigma on base REAL*4 SIGMA_RAIE ! rrnh3 ! Sigma on line REAL*4 FIT(12) ! rnnh3(12) ! Fit results REAL*4 ERR(12) ! renh3(12) ! Errors END STRUCTURE * * Section -10 /CRCONT/ * OBS.DRIFT : Continuum drift description (for drifts) STRUCTURE /DRIFT_DESCR/ DRIFT ! (mcont=16) REAL*8 REST ! rfreq ! Rest frequency REAL*4 WIDTH ! rwidth ! Bandwidth INTEGER NPOINT ! rnpoin ! Number of data poi nts REAL*4 REFERENCE ! rrpoin ! Reference point REAL*4 TIMOFF ! rtref ! Time at reference REAL*4 ANGOFF ! raref ! Angular offset at ref. REAL*4 PA ! rapos ! Position angle of drift REAL*4 TIMRES ! rtres ! Time resolution REAL*4 ANGRES ! rares ! Angular resolution REAL*4 BAD ! rcbad ! Blanking value INTEGER SYSTEM ! rctype ! Type of offsets REAL*8 IMAGE ! rcimag ! Image frequency REAL*4 COLLA ! rcolla ! Collimation error Az REAL*4 COLLE ! rcolle ! Collimation error El END STRUCTURE * * Section -11 * OBS.BEAM : Beam-switching parameters (for spectra or drifts) STRUCTURE /BEAM_DESCR/ BEAM ! (mbeam=5) REAL*4 AZIMUTH ! rcazim ! Azimuth of observa tion REAL*4 ELEVATION ! rcelev ! Elevation of obser vation REAL*4 THROW ! rspace ! Beam spacing REAL*4 PA ! rbpos ! Position angle of beams INTEGER SYSTEM ! rbtype ! System for angle END STRUCTURE * * Section -15 /CRPOINT/ * OBS.CONTINUUM : Double gaussian and baseline fit results (for drifts) STRUCTURE /CONT_DESCR/ CONTINUUM ! (mfcont=19) INTEGER NLINE ! rlcont ! Number of componen ts REAL*4 SIGMA_BASE ! rbcont ! Sigma on base REAL*4 SIGMA_RAIE ! rrcont ! Sigma on line REAL*4 FIT(8) ! rncont(8) ! Results REAL*4 ERR(8) ! recont(8) ! Errors END STRUCTURE END STRUCTURE * From dwells@fits.CX.NRAO.EDU Mon Oct 16 10:03:50 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA05546; Mon, 16 Oct 89 10:03:48 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA09306; Mon, 16 Oct 89 10:03:40 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA05491; Mon, 16 Oct 89 09:55:57 EDT Return-Path: Message-Id: <8910161355.AA05485@fits.cx.nrao.edu> Status: RO From: dwells@fits.CX.NRAO.EDU (Don Wells) Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Cc: dwells@fits.CX.NRAO.EDU Subject: Re: Single-dish FITS Date: Mon, 16 Oct 89 09:55:54 EDT Dear friends, The Guilloteau and Forveille paper is a perceptive and sophisticated discussion of FITS issues for single-dish astronomy. It is a valuable contribution in preparation for the Green Bank workshop. Here are some comments, corrections, differences of opinion: 1. re matrices versus tables: Stephane and Thierry stress that data transfer from single-dish systems to "general" IP systems should be by means of images (n-dimensional matrices) in classical FITS form. I agree that *if* the data are an n-dimensional matrix then the Basic FITS format should be used. However, I am optimistic that we will soon have effective interchange capability for other types of data structures, particularly tables. I contend that we must make the effort to build and demonstrate this capability in many systems so that astronomical data interchange and archiving will no longer be confined to matrices. I would like to see the single-dish community take a leadership role in this effort. 2. re list-directed-READ for FITS value fields: In March 1979 Eric Greisen and I agreed that FITS keyword value fields would use the notation of FORTRAN list-directed READs. The list-directed READ design allows for lists of values, and Eric and I understood what we were proposing. My memory is that when the design went to the review panel in April-May 1979 there were strong protests that many computer systems were not sophisticated enough to cope with completely free-field headers, that our design was too sophisticated and that it ran the risk of being ignored by many groups in astronomy. We compromised. We stopped talking about multi-valued keywords and we even gave up the free-field concept --- we specified that the required keywords of a FITS header must be in a rigid format so that unsophisticated readers would have an easier job. The multiple-value problem is fundamental. I do not have a nice, neat answer for it. In headers we have another problem: Eric and I decided that all standard header lines would fit into one "card". This limits how many values a keyword can have. I believe now that the best solutions will use tables, with separate tuples for the values. 3. re irregularly sampled data: The classical answer to this problem is the random-groups format. Indeed, since 1980 the leading FITS designers have all been somewhat sad that we did not think of random-groups until after the Basic FITS Agreement had already been implemented in production systems. We wish that Basic FITS had been the random-groups format, so that simple matricies would be a special case. In 1984 we made the Generalized Extensions Agreement have this form. Jim Condon of NRAO and his collaborators have made several surveys of the northern sky with the 300-foot in a rapid scanning mode using a seven-beam feed. The randomly sampled data were formatted as random-groups FITS. AIPS is able to read such data (see relevant sections of Going AIPS for details) and there are tasks which can grid the samples into maps of the sky. Indeed, the tasks are variations on the aperture synthesis task UVMAP (the Fourier transform step is skipped). The random samples are sorted using task UVSRT. I.e., FROM THE POINT OF VIEW OF A PROGRAMMER, RANDOM SAMPLES OF FLUX ON THE SKY ARE STRUCTURALLY ALMOST IDENTICAL TO THE RANDOM SAMPLES OF THE UV PLANE ACQUIRED IN APERTURE SYNTHESIS, AND SIMILAR SOFTWARE APPROACHES ARE FEASIBLE. Note that scanning infrared satellite data (IRAS) and high-energy photon event data fall in this same class. Although the 300-foot surveys were continuum data only I believe that the AIPS tasks are more general, that they can sort and grid spectral-line surveys. Bill Cotton's "3-D Binary Tables" design, which he devised for VLBA calibration, is yet another approach to the irregularly sampled data problem. It is not as elegant mathematically, but the bit stream that it produces is essentially *identical* to that produced by random groups (so that there is no difference in efficiency) and he can transmit data types that random-groups cannot handle. 4. re hierarchical keyword syntax: Bob Hanisch has raised two issues: (1) "Do we really need a hierarchical namespace?" and (2) "What syntax should we use?". I still believe that the FITS community should endeavour to transmit *MEANING* (semantics) as well as mere *DATA* (syntax), and I believe that it will be necessary to organize our name-space in order to achieve this goal reliably. However I am grateful to Bob for having the courage to go against the common wisdom in this area, to make us think about the issue. Regarding syntax, there are two proposals. Either we devise rules for what I call the "push-down-stack" syntax (the concept discussed by Hanisch and endorsed by Guilloteau and Forveille), or we adapt the existing "domain-name-system" rules for single keyword=value header cards. I think that either approach can probably be made towork in the end, but I have more confidence in the second approach. I have a paper of my own which has been in preparation for several weeks, and I expect to broadcast it soon. Regards, Don Donald C. Wells, Associate Scientist | NSFnet: dwells@nrao.edu [192.33.115.2] National Radio Astronomy Observatory | SPAN: NRAO::DWELLS [6654::] Edgemont Road | BITnet: DWELLS@NRAO Charlottesville, VA 22903-2475 USA | UUCP: ...!uunet!nrao.edu!dwells +1-804-296-0277 (38:02.2N/78:31.1W) | TWX=510-587-5482, Fax=+1-804-296-0278 From dwells@fits.CX.NRAO.EDU Mon Oct 16 17:23:36 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA06260; Mon, 16 Oct 89 17:23:28 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA10347; Mon, 16 Oct 89 17:22:44 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA06207; Mon, 16 Oct 89 17:03:31 EDT Return-Path: Message-Id: <8910162103.AA06200@fits.cx.nrao.edu> Status: RO From: dwells@fits.CX.NRAO.EDU (Don Wells) Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Binary Table Structure for Spectra Date: Mon, 16 Oct 89 17:03:26 EDT Dear Friends-of-FITS: The paper below is a proposal to the single-dish radio community, which could adopt it even if the rest of astronomy doesn't want to do so. I, of course, hope that it may be possible to forge an alliance across all of astronomical spectroscopy. My actual proposal is only the first 9 pages. The remaining 23 pages are appendices containing the text of the Floating Point Agreement, the Grosbol/Wells Hierarchical Keyword draft proposal and a detailed description of AIPS support for single-dish random-scan data and Cotton's 3-D Binary Table extension format. Donald C. Wells, Associate Scientist | NSFnet: dwells@nrao.edu [192.33.115.2] National Radio Astronomy Observatory | SPAN: NRAO::DWELLS [6654::] Edgemont Road | BITnet: DWELLS@NRAO Charlottesville, VA 22903-2475 USA | UUCP: ...!uunet!nrao.edu!dwells +1-804-296-0277 (38:02.2N/78:31.1W) | TWX=510-587-5482, Fax=+1-804-296-0278 ------------------- LaTeX, 32 pages, cut here ------------------------- \documentstyle[11pt,twoside]{article} \pagestyle{headings} \raggedbottom % \begin{document} \title{A Binary Table Structure for Spectra} \author{Donald C. Wells\\ {\small \raisebox{0.1cm}{\rule{0cm}{0.3cm}}NRAO, Charlottesville, VA}\\ {\small 804-296-0277, {\tt dwells@nrao.edu}}} \date{{\em DRAFT} \today} \maketitle \begin{abstract} The 3-D binary table FITS extension is proposed as an efficient format to transmit groups of astronomical spectra and associated parameters. Syntactic conventions for keywords and table column labels are suggested. \end{abstract} \tableofcontents \clearpage \section{The Problem} The basic FITS (Flexible Image Transport System) Agreement of 1979 defined a syntax for transmitting N-dimensional matricies. It was (and still is) perfectly capable of transmitting spectra, for which N=1, as vectors of binary data described by ASCII headers using the keyword=value notation. However, such usage is inefficient: in most cases more bytes are needed for the header than for the binary vector! Spectroscopy observations generally need more ancillary parameters, more keyword=value pairs, than do imaging observations and this only exaggerates the disproportion between header and data for spectroscopy. The basic FITS format is simply not efficient for spectroscopy applications. Spectroscopy observations often involve large numbers of short integrations, which implies that large numbers of ASCII headers must be written, transmitted, archived and processed. Although many people have wished that the vectors as well as the headers could be transmitted in ASCII form so that {\em everything} would be human-readable, this is simply not a practical option for our new data systems in which more and more feeds are analyzed by spectrometers with higher and higher resolution. Binary notation is essential for the spectra, and we must strive to reduce the overhead of the headers. This means that even the headers must be in binary, and redundancy must be minimized. \section{The Solution} We need binary data structures which include the ancillary parameters in binary form, bound to the binary vectors. These structures must be described in FITS style in order to make them {\em flexible.} The ASCII description will require almost as many bytes as would a basic FITS header. To minimize this overhead we need to be able to transmit {\em arrays} of the structures. In this concept only one copy of the ASCII description is needed for an arbitrarily large number of binary spectra plus parameters, all grouped together in one FITS file. There are two existing FITS formats that have the desired property: the random groups format\footnote{Greisen, E.W., Harten, R.H.: 1981, Astron. Astrophys. Suppl. Ser. 44, 371, "An Extension of FITS for Groups of Small Arrays of Data".} and the 3-D binary tables format\footnote{Cotton, circa 1986, coded in AIPS; see Appendix~\ref{app:C3DBTF} and Section~\ref{sect:C3DBTF}.}. There is no question that random groups are a feasible solution, but binary 3-D tables are preferable, for two reasons. First, experience shows that many people have difficulty understanding the random groups concept but everyone claims that they understand tables intuitively. Second, the random groups format requires that the same data type be used for both spectra and parameters but the 3-D binary tables allow mixed data types, a great convenience and valuable notational power. I propose that Cotton's 3-D binary table format be adopted for astronomical spectroscopy data interchange. \section{Conventions} The basic concept that I am proposing is that a set of spectroscopic observations be regarded as a {\em table}, with the spectral vectors appearing as one of the columns of the table and the ancillary parameters appearing in other columns. The following subsections propose certain conventions for the organization of the table. \subsection{Minimizing Redundancy}\label{sect:redundant} Many of the parameters associated with a set of spectroscopic observations are constant for the whole set. If each such parameter must appear in a separate column of the table the same bytes will appear over and over. I propose that such parameters be moved to the extension header and encoded as keyword=value, and that they be regarded as a {\em virtual column.} \subsection{Multiple Feeds} Many spectroscopic instruments today observe several positions on the sky simultaneously. There are two possible encoding conventions. First, we could encode such observations as one structure containing several binary vectors, with many parameters shared between the spectra and many other parameters appearing once for each of the spectra. Alternatively, we could encode the several spectra as separate structures, with the parameters which are identical being duplicated but without the notational complexity of the multiple instances of spectra and parameters in one structure. I believe that the latter case, the separate structure scheme, is easier to design and to implement, and so I propose that it be adopted. Some spectrometers measure more than one band simultaneously. This case is analogous to multiple feeds and the separate structure approach handles it with no new notation. In some cases datasets contain spectra with different numbers of channels. I propose that the number of channels be specified in a column. If all spectra have the same length the column need not be carried with the spectra in the structures --- it can be given once in the extension header (see section \ref{sect:redundant}). \subsection{Hierarchical Table Labels} Table column labels pose the same problems of potential name conflict as do keywords. It appears that the only feasible solution will be some kind of hierarchical classification scheme. The two draft proposals available currently are the Hierarchical Keyword Agreement (the ``domain-name-system'' proposal) of Grosb{\o}l and Wells, whose text is given below in Appendix~\ref{app:HKA}, and the ``push-down-stack'' proposal of Tody and Hanisch, as discussed in Hanisch's DISHFITS-exploder message of 05~October. In the remainder of this paper I will discuss the Grosb{\o}l/Wells notation because it leads to a natural generalization for table labels. I have not yet decided whether the Tody/Hanisch proposal offers an equally suitable solution for this application. The FITS ASCII Tables standard\footnote{Harten, R.H., Grosb{\o}l, P., Greisen, E.W., Wells, D.C.: 1988, Astron. Astrophys. Suppl. Ser. 73, 365, "The FITS Tables Extension".} does not specify any limit on the length of column label strings. This is yet another example of the continuing need for interpretation of the rules of the standards. Let us assume that the string-value rule of the Basic FITS Agreement\footnote{Wells, D.C., Greisen, E.W., Harten, R.H.: 1981, Astron. Astrophys. Suppl. Ser. 44, 363, "FITS: A Flexible Image Transport System". See p.~364-365.} applies: string values may be of any length which fits on one card (80 characters), but should be unique/useful if truncated to 8 characters. I therefore propose that hierarchical classification codes be used to obtain uniqueness in the labels, {\em but that the order be reversed so that the last substring, the keyword, will be in the first 8 characters.} If the same code string is used as a keyword in the extension header, as proposed in Section~\ref{sect:redundant} above, it will appear in normal order, with the {\tt HIERARCH} code in the first 8 columns and the keyword at the end of the string, just before the equals sign. \subsection{Units} I strongly recommend that only SI units be used in data interchange in astronomy, as specified in the Basic FITS Agreement. \subsection{Spectral Format Table Labels} I propose that column names and parameter values should be the keyword and axis-type names used previously in FITS designs, wherever possible. By this I mean that in situations where in other proposals keywords are used in FITS headers that those same names be used in table column labels for tables of spectra, and that where various conventions are specified for the value fields of those keywords that those same conventions be used for the value fields in those columns. Thus, I am proposing that the parameter columns be regarded as {\em virtual header cards}. \section{Status of Relevant Draft Standards} \subsection{Generalized Extensions, ASCII Tables} The two FITS standards were published in Astron.~Astrophys.~Suppl. in 1988, and were approved by IAU Commission~5 at the Baltimore meeting in August 1988.\footnote{IAU Resolution B1, p.~10, IAU Information Bulletin 61, January 1989.} \subsection{The Floating Point Agreement} The FPA has been discussed for at least four years in committees. The prototype implementations were done circa 1985-6. The final text was adopted 28~February~1989, and is given in Appendix~\ref{app:FPA}. The FPA was endorsed by the European Working Group for Coordination of Astronomical Software in May 1989, and by the AAS Working Group for Astronomical Software in June 1989. The Agreement is now under consideration by the IAU FITS Working Group, which is awaiting the final interchange and interoperability proof for the 64-bit data type. Activation is anticipated for 1990. \subsection{Binary 3-D Table Format}\label{sect:C3DBTF} Bill Cotton designed his table format as an extension of the standard ASCII table format in order to represent certain complicated calibration table structures needed for the VLBA analysis software. The only description currently available is from section 14.5.3 of ``Going AIPS'', the AIPS programmer's reference manual. This text has been extracted from the AIPS directories and is given below in Appendix~\ref{app:C3DBTF}. This extension was implemented in AIPS circa 1986, and code supporting it has been operational on several hundred computers for several years now. The extension was implemented in MIDAS circa 1987, and AIPS-MIDAS interoperability trials have been made. As of 1989 the extension is being implemented as a format for photon lists in high energy astronomy, first for the ROSAT data analysis software. The 3-D Binary Table design is generally believed by the FITS standards committees to be the draft proposal ``most likely to succeed''. Only one or two trivial changes are anticipated when it finally passes the IAU sometime in the future. \subsection{Hierarchical Keyword Agreement} The original idea was implemented by Eric Greisen in the first FITS software for the NRAO-Charlottesville IBM synthesis mapping package in 1979. A more recent prototype circa 1986-7 is the SNGLDSH format at NRAO. The concept has been discussed in the FITS committees since 1987. A current draft proposal by Grosb{\o}l and Wells dates from 28~February~1989; it is reproduced below in Appendix~\ref{app:HKA}. Alternative concepts are still being offered. At present no clear consensus exists favoring any one proposal. \section{An Example Showing Packing Efficiency} The preceding discussion has proposed Cotton's FITS extension as a solution appropriate for all spectroscopic applications in astronomy. At present the main astronomy discipline considering this problem is single-dish radio astronomy. There has been much preliminary work involving keywords, syntax and even semantics for this application, and it need not be repeated here. Indeed, even the header syntax for the 3-D extension is well documented in an appendix to this paper, and so I choose not to give a full example. The important point to be addressed is the {\em packing efficiency} which is possible for the proposed design.. Consider a radio spectrometer with 1024 channels in floating point form. A data file will start with a basic FITS header which has no data matrix ({\tt NAXIS=0}). This will be followed by the extension header, which will specify {\tt XTENSION='A3DTABLE'} (once the IAU approves this extension the value will probably be {\tt '3DTABLE'}). Each spectrum will be encoded as a binary data structure which is described with the notational machinery of the 3-D tables. A grossly simplified data structure which has an example of each data type might be: \begin{center} \begin{tabular}{|l|l|l|l|l|l|l|} \hline label: &{\tt SCAN} &{\tt SOURCE} &{\tt BASEFREQ} &{\tt DELTFREQ} &{\tt NCHAN} &{\tt SPECTRUM} \\ \hline format: &{\tt 1J} &{\tt 20A} &{\tt 1D} &{\tt 1F} &{\tt 1I} &{\tt 1024F} \\ \hline bytes: &4 & 20 & 8 & 4 & 2 & 4096 \\ \hline \end{tabular} \end{center} The {\tt SCAN} number is a 32-bit integer, the {\tt SOURCE} name is a string, the frequency of the reference channel (in Hertz, of course) is a double precision number and the frequency increment is single precision. The spectrum itself is single precision floating point and the number of channels {\tt NCHAN} is specified by a 16-bit integer (note that the {\em maximum} number of channels is given by the format of the {\tt SPECTRUM} field; {\tt NCHAN} might be less than this number). {\em The column labels used here are not to be regarded as a part of my proposal; rather, if my 3-D table concept is accepted for astronomical spectra an ad hoc task force should negotiate these details.} The binary data types can be flexible: one site might encode {\tt SCAN} numbers as 16-bit integers while another used 32-bits. Likewise, some implementations may choose to encode spectra using 16-bit integers rather than 32-bit reals in order to save bytes in the files (the FITS machinery for integer scaling permits this to work even for inherently non-integer values). For this example there will be 2880 bytes in the basic header, 5760 bytes in the extension header, 4134 bytes per spectrum and one tapemark. It is true that a real spectral observation will probably need about 50 parameters, not 5, but even at 4 bytes per parameter this will add only 200 more bytes per spectrum. Therefore, it will still be true that for one spectrum the header overhead will dominate. Indeed, 50 more columns in the table will require several more logical records of extension header to document, making the imbalance even worse! However, additional spectra will add no more overhead. If the table contains as many as 20 spectra the efficiency will exceed 90\%. \section{Unresolved Issues, Other Ideas} \begin{itemize} \item{\bf Hierarchical Keywords:} There is no consensus yet on the syntax, or even on the necessity of a hierarchical data space. \item{\bf Vocabulary, Semantics, Units:} This proposal is only concerned with the syntax for transmitting spectra. Each discipline of astronomy must agree on the vocabulary and on the semantics and units for its special language. \item{\bf World Coordinates:} We must agree on conventions for describing the mapping between pixel number and frequency in the spectra. I recommend that we utilize notations similar to those devised by Eric Greisen for imagery. This means that we need to identify how many types of mappings occur in astronomical instruments and assign names to them. We must analyze their mathematical forms to determine how many parameters each has and attempt to devise a uniform convention for the notation. An important question will be whether all of astronomy can agree to use frequency as the independent variable for spectroscopic data interchange; if not, we will have both wavelength and frequency (wavenumbers are just frequency by another name, of course). All of these world coordinates issues are beyond the scope of this paper. \item{\bf Higher-Level Organization of Datasets:} Multiple named table extensions may be grouped together in one file --- this provides a higher level organizational tool. The Generalized Extensions standard also specifies keywords and conventions for a hierarchical structure of extensions, which is yet another higher level organizational tool. \item{\bf Auxilliary Tables:} The FITS table extensions could be used to transmit calibrations, source lists, etc., in addition to the spectra --- this could permit the elimination of even more redundancy in the sense of ``normalizing'' a relational DBMS schema. \end{itemize} \appendix % ================================================================== \clearpage \section{Floating Point Proposal for FITS}\label{app:FPA} \begin{center}Don Wells, Preben Grosb{\o}l\end{center} \begin{center}28 February 1989\footnote{This text is very slightly revised from the version that was endorsed by the European FITS Committee in April 1989 and by the AAS Working Group for Astronomical Software in June 1989. It will now be reviewed by the IAU FITS Working Group for adoption as a part of the FITS standards, and will probably be activated in 1990.}\end{center} \subsection{The Proposal} The Basic FITS, Random Groups and Generalized Extensions Agreements are revised to add IEEE-754 32- and 64-bit floating point numbers to the original set of FITS data types. BITPIX=-32 and BITPIX=-64 signify 32- and 64-bit IEEE floating point numbers; the absolute value of BITPIX is used for computing the sizes of data structures. The full IEEE set of number forms are allowed for FITS interchange, including all special values (e.g., the 'not-a-number' cases). The order of the bytes will be sign and exponent first, followed by the mantissa bytes in order of decreasing significance. The BLANK keyword will be ignored by FITS readers when BITPIX=-32 or -64. \subsection{Discussion} The dynamic ranges of astronomical data have increased steadily in the years since the original FITS Agreement (March 1979) which defined 8-, 16- and 32-bit integers as the only data formats. All integer formats are limited by an absolute error, whereas floating point formats can span a much wider numerical range with only a relative error. Floating point formats are desirable for the applications that need increased dynamic range. Many astronomical data systems now use floating point in data processing, and conversion to and from FITS integer formats sacrifice accuracy and dynamic range and consume significant computer time. Almost all of the new computers designed since about 1981 use the IEEE format (see references below). We are proposing that FITS be revised to allow transmission of 32- and 64-bit floating point data within the FITS format using the IEEE standard. This Floating Point Agreement will also apply to the Random Groups Extension and to those Generalized Extensions for which BITPIX is not explicitly restricted (e.g., BITPIX=8 for XTENSION='TABLE'). In order to transmit floating point data in the main data matrix we must add some new convention to the main FITS header. We propose to add two new allowed values for the basic keyword BITPIX. The set of values specified in the original FITS Agreement was 8, 16 and 32; we propose to add -32 and -64 to indicate IEEE single- and double-precision floating point data. Thus, the number of bits per pixel would be the absolute value of BITPIX. This implies that the number of 2880-byte logical records used by the main data matrix must be computed using the absolute value of BITPIX (this rule was specified in the Generalized Extensions Agreement (Grosb{\o}l etal. 1988)). The IEEE standard specifies a variety of special exponent and mantissa values in order to support the concepts of plus and minus infinity, plus and minus zero, 'denormalized' numbers and 'not-a-number' (NaN). We propose that all of these special cases should be fully accepted for FITS interchange. We propose that the order of the bytes should be sign and exponent first, followed by the mantissa bytes in order of decreasing significance (i.e., the standard non-byte-swapped order). The BLANK keyword of the original FITS Agreement should be ignored by FITS readers when BITPIX=-32 or -64 (the NaNs of the IEEE format will act as the blank). FITS writers should not write the BLANK keyword if BITPIX=-32 or -64. The BSCALE and BZERO values should be applied by FITS readers if they differ from 1.0 and 0.0. However, such usage be regarded as bad practice, and strongly discouraged, because of the risk of generating overflows and underflows in FITS readers. \subsection{Bibliography} "An Implementation Guide to a Proposed Standard for Floating-Point Arithmetic", COMPUTER magazine, Vol. 13, No. 1, pp.68-79, January 1980. COMPUTER, Vol. 14, No. 3, March 1981. ANSI/IEEE 754-1985, "IEEE Standard for Binary Floating Point Arithmetic". ISDN-xxxxx? Wells, D.C., Greisen, E.W., Harten, R.H.: 1981, Astron. Astrophys. Suppl. Ser. 44, 363, "FITS: A Flexible Image Transport System". Greisen, E.W., Harten, R.H.: 1981, Astron. Astrophys. Suppl. Ser. 44, 371, "An Extension of FITS for Groups of Small Arrays of Data". Grosb{\o}l, P., Harten, R.H., Greisen, E.W., Wells, D.C.: 1988, Astron. Astrophys. Suppl. Ser. 73, 359-364, "Generalized Extensions and Blocking Factors for FITS". Harten, R.H., Grosb{\o}l, P., Greisen, E.W., Wells, D.C.: 1988, Astron. Astrophys. Suppl. Ser. 73, 365, "The FITS Tables Extension". \subsection{Implementation Notes} Note-1: Many data analysis systems today use floating point internally and so conversion costs for FITS input and output will be reduced by this proposal. For machines with IEEE hardware the conversion cost will be nearly zero, especially when BZERO=0 and BSCALE=1 and the reader has two DO-loops (one with and one without scaling). Note-2: The interpretation of the IEEE sign, exponent and mantissa bit fields is precisely specified by the five rules of Section 3.2.1 of ANSI/IEEE Standard 754-1985. Algorithms for transformation between IEEE and non-IEEE formats can be derived from these rules. The FITS Committees will provide a central archive for conversion software for various architectures. FITS readers on non-IEEE systems should test the IEEE exponents for the special values 0 and 255/2047 (for 32/64-bit numbers). Exponent zero should be mapped to the local value for floating zero (exponent zero is used by IEEE for 'denormalized' numbers as well as for both plus and minus zero); exponent 255/2047 should be mapped to the local value used for blank pixels (all IEEE infinity and NaN cases have exponent 255/2047). The IEEE exponent field can be selected from the 32/64-bit values by masking with hexadecimal byte string 7F,80,00,00/7F,F0,00,00,00,00,00,00 (i.e., the constant {\tt 7F800000}/{\tt 7FF0000000000000} hexadecimal on computers with the normal word ordering; for VAXes use constant {\tt 00007F80}/{\tt 0000000000007FF0} instead). The two tests are made by comparing the result against zero and against the mask. FITS writers executing on non-IEEE systems should map their local value for blank pixels to an IEEE NaN value; we suggest that all bits set (minus one in 32- and 64-bit two-s complement integers) is a good choice. The FITS writer should also map local floating point absolute values which exceed the largest value of IEEE (approximately 10**38 for 32-bit, 10**304 for 64-bit) to IEEE-Infinity. We recommend that absolute values smaller than the smallest normalized IEEE value (approximately 10**-38 or 10**-304) be mapped to zero. These two tests are particularly important on machines which use 'hexadecimal' normalization, with an exponent range of approximately 10**75 in single-precision. The DEC VAX architecture is an important special case. VAX systems use the DEC-proprietary 'F' format for 32-bit floating data and their byte order is 'swapped'. Once the swap has been corrected the F-format differs from the IEEE 32-bit format only in that the exponent bias differs by one and the (hidden) binary point is shifted by one bit. The two differences add, and the total effect is that the two number systems differ by a factor of 4.0 (VAXF = 4.0 * IEEE32, IEEE32 = 0.25 * VAXF). This floating point multiplication operation is little, if any, more expensive than the conversion operation between integer and floating point which is required by the original FITS Agreement. VAX systems should test the exponent field on input to detect the infinity and NaN cases and map them to the local blank code using the test specified previously. The minus zero and denormalized cases may also need to be trapped. Initial prototype implementations of this Agreement have disclosed that some IEEE hardware produces -0.0 results and that reserved operand errors occur in VAXes when these values are used as operands. The 32-bit operand should be compared against 32768 to detect this case, and the result 0.0 be given rather than multiplication by 4.0. The DEC VAX architecture supports two different double precision formats, 'D' and 'G'. The 'G' format has the same relationship to IEEE 64-bit format that the DEC 'F' format has to IEEE 32-bit (VAXG = 4.0 * IEEE64). The 'D' format, however, requires a complicated transformation to convert to/from IEEE 64-bit; programmers should consult archives of FITS code to obtain suitable algorithms and implementations for this case. % =================================================================== \clearpage \section{Hierarchical Keywords in FITS Headers}\label{app:HKA} \begin{center}P.~Grosb{\o}l and D.~Wells\end{center} \begin{center}28 February 1989 (Draft-1)\end{center} \subsection{The Proposal} A new standard keyword HIERARCH\footnote{This name was derived by the traditional FITS rule that names longer than 8 characters should be truncated. It is, in fact, an English word in its own right: the {\bf Oxford English Dictionary} says (definition B.2) that a {\em hierarch} is the ``...commander of the celestial hierarchy''!} is introduced for all FITS headers, both Basic FITS headers and Generalized Extension headers. The remaining 72-characters of an 80-character HIERARCH line will contain one or more keyword classification codes followed a single keyword=value pair. Code and keyword names will be 1-8 characters (uppercase alphanumeric plus the minus sign), just as for standard keywords. Column 9 will be blank; the first code will begin in column 10 and additional codes and the keyword will be delimited by one or more blanks. Value fields will be decoded according to the rules of Basic FITS. The intent of this convention is to provide a hierarchical classification of keywords in order to reduce the possibility of conflicts of keyword names, especially 'proprietary' keywords. The set of approved top-level classification codes will be controlled by the FITS Working Group of IAU Commission 5. \subsection{Example} \begin{verbatim} 1 2 3 4 5 6 123456789012345678901234567890123456789012345678901234567890 HIERARCH topcode level2 code3 keyword = value \end{verbatim} \subsection{Discussion} The IAU FITS Working Group will need to decide whether top-level classification codes should be assigned for individual institutions and projects, or for disciplines of astronomy. Name-spaces assigned to institutions can be managed privately by those institutions. Name-spaces assigned to disciplines will require the creation of international committees. The discipline groups could assign institutional codes under their top-level codes as well as agreeing on discipline-specific generic codes and keywords. The latter approach can lead to a greater degree of standardization among special interest groups in astronomy, without burdening the IAU Working Group. This is, in principal, very desirable. However, it is unclear whether or not such interest groups can be organized in practice. \subsection{Notes} \begin{enumerate} \item The Basic-FITS rule that equal signs must be followed by one or more blanks is relaxed for HIERARCH lines because the rules for value-field syntax prevent ambiguities. \item The maximum number of classification codes is naturally limited by the finite length of a HIERARCH line; however, implementors should probably limit HIERARCH codes to no more than five levels. \item Note that Basic FITS does not specify whether keywords are limited to the set of uppercase alphanumeric characters plus the minus sign, but those are the only characters occurring in the examples given in the 1981 paper. \item Lines 2/4 and 2/5 of Figure 1 on page 370 of the Basic FITS paper (Astron. Astrophys. Suppl. Ser. 44, 363-370, 1981) show an example of a hierarchical keyword notation which is similar to the one proposed here. It is regretable that this notation was not recommended more clearly in the Basic FITS paper, as much confusion would have been prevented over the years. \end{enumerate} % =================================================================== \clearpage \section{3-D Binary Table Extension Format}\label{app:C3DBTF} % document translated from DEC RUNOFF to LaTeX format % by program RNOTOTEX version CVF02B at 13-APR-1989 10:52:05.03 % Source file: CHAP14.RNO %\chapter{FITS Tapes} {\bf Note:} The text of this appendix is taken from Chapter~14 of {\bf Going AIPS}, the AIPS programmer's reference manual. The chapter describes how AIPS uses the FITS agreements. It gives an overview of FITS from the AIPS perspective, and then proceeds to discuss each of the FITS formats used by AIPS. Section~\ref{app:C3DBTF2} below (14.5.3 in {\bf Going AIPS}) describes the 3-D binary table format, which Bill Cotton designed circa 1986 to meet the special requirements of the calibration databases for the VLBA. Subsections of the {\bf Going AIPS} chapter which are titled ``Image Files'', ``Random Group (UV data) Files'', ``Older AIPS Tables'', ``AIPS FITS INCLUDEs'', and ``AIPS FITS Parsing Routines'' have been deleted because they are not relevant to this paper. The subsection titled ``Single Dish Data'' has been retained because of its special interest for single-dish radio astronomers; it describes the support provided by AIPS for forming N-dimensional images out of signals (continuum or multi-spectral) sampled at {\em random} locations by various scanning instruments. This section shows the random-groups approach to the spectroscopic format problem. It is important to realize that any data structure which can be realized within the random groups concept can also be realized within the 3-D binary tables concept. Indeed, at the bit stream level the two formats are essentially identical. The binary tables design does support several data types which are not possible in random groups (the most important case is character strings); this is a fundamental advantage for the table approach. \subsection{Overview } The principal route for getting data and images into and out of AIPS is by FITS (Flexible Image Transport System) format tape files. FITS is an internationally adopted medium of exchange of astronomical data and allows easy interchange of data between observatories and image processing systems. FITS also has the advantages that it is a self-defining format and that the actual bit pattern on the tape is independent of the machine on which the tape was written. The purpose of this chapter is to describe the general features of FITS and the details of the AIPS implementation. This chapter is not intended to be a rigorous description of the FITS standards. See the chapter on devices for information on read and writing tapes in AIPS. The fundamental definition of the FITS system\index{FITS format} is given in Wells, Greisen, and Harten (1981), with an extension described in Greisen and Harten (1981). A proposed further extension is given in Harten, Grosb{\o}l, Tritton, Greisen and Wells, (1984). FITS has been adopted as the recommended medium of exchange of astronomical data by the IAU, the Working Group on Astronomical Software (WGAS) of the AAS, and comparable working groups in Europe. AIPS now also supports the proposal of these working groups for the writing of blocked FITS tapes.\index{blocked tapes} Because of the great flexibility of the FITS system, many of its features have been adopted for the internal data storage format in AIPS. See the chapter on the catalog header for more details on the AIPS internal storage format. There are three main portions of a FITS file (1) the main header, (2) the main data, and (3) any number of records containing auxiliary information. In addition, an extension of the original definition of the FITS structure allows storage of ungridded visibility data. Each of these is discussed in detail in the following sections. \subsection{Philosophy } FITS is a philosophy as much as a data format. The underlying philosophy is to provide a standardized, simple, and flexible means to transport data between computers or image processing systems. FITS is standardized in the sense that any FITS reader should be able to read any FITS image, at least to the degree that the array read is of the correct dimensions and the pixel values have at least the correct relative scaling. In addition, any FITS reader should be able to cope with any FITS format tape and, at least, skip over portions, or ignore keywords, that it doesn't understand. \index{FITS format} The requirement of simplicity means that the implementation of FITS reading and writing should be fairly straightforward on any computer used for astronomical image processing. Simple also implies that the structure of the file should be self-defining and, to a large degree, self-documenting. The main advantage of FITS is its flexibility. Due to the self-defining nature of the files, a large range of data transport needs are fulfilled. The introduction of new keywords gives the ability to add new pieces of information as needed, and the use of generalized extension files allows almost unlimited flexibility in the type of information to be stored. Thus, FITS can grow with the needs of the Astronomical community. The great flexibility of FITS is a potential weakness as well as a strength. There is a great temptation to proliferate keywords and new extension file types. This should be done with great caution. Since FITS is a worldwide medium of data exchange, there needs to be coordination of keywords and extension files to prevent duplication and inconsistencies in usage. The most fundamental philosophical ideal of FITS is that no change in the system should render old tapes illegal or unreadable. This philosophy is reflected in the AIPS implementation of FITS in that all obsolete implementations (e.g., old CLEAN component or antenna extension files) are trapped and processed in the most accurate manner possible. \subsection{Single Dish Data } Observations made with filled aperature instruments are frequently made at essentially random positions on the sky, possibly using a number of offset feeds or detectors. This type of data may be convienently described using the random groups (UV FITS) format. The FITS form of this data is the same as visibility data except that the number and meaning of the random parameters are different. The celestial coordinates may be either Right Ascension and declination or projected coordinates about a specified tangent point. A logical record consists of all data recorded from a given beam on the sky at a given time. A dummy AN table is optional. \subsubsection{Single dish random parameters } The single dish random parameter types (PTYPEn) are described in the following: \begin{enumerate} % list nest 1 \item 'RA' and 'DEC': These random parameters are the Right Ascension and Declination of the observation in degrees. If the coordinates have been projected onto the tangent plane then the RA and Declination types become 'RA--xxxx' and 'DEC-xxxx' where xxxx is the projection code. See the chapter on AIPS catalogue headers and/or AIPS memoes 27 and 46 for details of the projection codes. These random parameter these are required but the order is arbitrary. \item 'DATE': The time tags for the data are kept in the form of Julian date in days. This random parameter is required but the order is optional. \item 'BEAM': This random parameter gives the beam number + 256. This random parameter is optional. The beam offset makes the data look more like uv data and more of the the AIPS uv data tasks will work for this data. \item 'SCAN': This random parameter gives the scan number. This random parameter is optional. \item 'SAMPLE': This random parameter gives the sample number in the scan. This random parameter is optional. \end{enumerate} % - list nest 1 \subsubsection{Single dish regular axis coordinates } The units of the regular axis coordinates are defined by convention; the conventions used by AIPS for the regular axis types (CTYPEn) are the following: \index{uv FITS format} \begin{enumerate} % list nest 1 \item 'COMPLEX': the complex axis consists of the real, imaginary and (optional) weight. Magic value blanking is supported. The imaginary part may be used to carry any baseline values which have been subtracted. This axis is required. \item 'STOKES': this axis is used to describe which Stokes' parameters are given; the conventions are the same as used internally in AIPS. These conventions are discussed in the chapter on disk I/O. This axis is required. \item 'FREQ': the frequency axis coordinates are in Hz. This axis is required. \item 'IF': The IF axis is a construct which allows irregularly spaced groups of frequency channels. The IF number specifies an entry in the ('AIPS CH') table which must follow the data if this axis is present. This table gives the offsets from the reference frequency specified by the FREQ axis. This axis is optional but if it is present then a CH table must be present. \item 'RA' and 'DEC': the celestial coordinates are given in degrees. The values associated with these axes are irrelevant (although they should be present) for unprojected data. For data with projected coordinates the coordinate values of these axes should be the tangent point, i.e. the position on the sky at which the plane onto which the coordinates are projected is tangent to the celestial sphere and these axes should become 'RA---ccc' and 'DEC--ccc' where ccc is the projection code. These axes are required. \end{enumerate} % - list nest 1 Weights and flagging are handled the same as for visibility data. Sort order is the same as for visibility data except that the sort codes for sorting by u and v become: \begin{verbatim} U => ordered by RA V => ordered by Declination X => descending ABS (RA) Y => descending ABS (Declination) Z => ascending ABS (RA) M => ascending ABS (Declination) \end{verbatim} \subsubsection{Example Single Dish Data File Header } The following is a FITS header for a single dish data file containing 16 frequency channels and a single IF and using unprojected coordinated. The first two lines are not part of the FITS file. \index{single dish FITS format} \begin{verbatim} 000000000111111111122222222223333333333444444444455555555556666666666 123456789012345678901234567890123456789012345678901234567890123456789 SIMPLE = T / BITPIX = 16 / NAXIS = 6 / NAXIS1 = 0 /NO STANDARD IMAGE JUST GROUP NAXIS2 = 3 / NAXIS3 = 1 / NAXIS4 = 16 / NAXIS5 = 1 / NAXIS6 = 1 / EXTEND = T /TABLES FOLLOWING MAIN IMAGE BLOCKED = T /TAPE MAY BE BLOCKED OBJECT = 'ALL SKY ' /SOURCE NAME TELESCOP= '300 FT ' / INSTRUME= '6 CM ' / OBSERVER= 'PHANTOM ' / DATE-OBS= '02/12/86' /OBSERVATION START DATE DD/MM/YY DATE-MAP= '09/11/87' /DATE OF LAST PROCESSING DD/MM/YY BSCALE = 4.98494155534E-05 /REAL = TAPE * BSCALE + BZERO BZERO = 0.00000000000E+00 / BUNIT = 'JY ' /UNITS OF FLUX EPOCH = 1.950000000E+03 /EPOCH OF RA DEC BLANK = -32768 /TAPE VALUE OF BLANK PIXEL CTYPE2 = 'COMPLEX ' / CRVAL2 = 1.00000000000E+00 / CDELT2 = 1.000000000E+00 / CRPIX2 = 1.000000000E+00 / CROTA2 = 0.000000000E+00 / CTYPE3 = 'STOKES ' / CRVAL3 = -1.00000000000E+00 / CDELT3 = -1.000000000E+00 / CRPIX3 = 1.000000000E+00 / CROTA3 = 0.000000000E+00 / CTYPE4 = 'FREQ ' / CRVAL4 = 1.66499989984E+09 / CDELT4 = 5.000000000E+07 / CRPIX4 = 1.000000000E+00 / CROTA4 = 0.000000000E+00 / CTYPE5 = 'RA ' / CRVAL5 = 0.00000000000E+00 / CDELT5 = 1.000000000E+00 / CRPIX5 = 1.000000000E+00 / CROTA5 = 0.000000000E+00 / CTYPE6 = 'DEC ' / CRVAL6 = 0.00000000000E+00 / CDELT6 = 1.000000000E+00 / CRPIX6 = 1.000000000E+00 / CROTA6 = 0.000000000E+00 / GROUPS = T / GCOUNT = 14655. / PCOUNT = 5 / PTYPE1 = 'RA ' / RA in degrees PSCAL1 = 1.09890000000E-02 / PZERO1 = 0.00000000000E+00 / PTYPE2 = 'DEC ' / Declination in degrees PSCAL1 = 1.09890000000E-02 / PZERO2 = 0.00000000000E+00 / PTYPE3 = 'BEAM ' / Beam number PSCAL3 = 3.66412235782E-10 / PZERO3 = 0.00000000000E+00 / PTYPE4 = 'DATE ' / Date + time as Julian date (days) PSCAL4 = 2.50000000000E-01 / PZERO4 = 2.44603650000E+06 / PTYPE5 = 'DATE ' / PSCAL5 = 1.52587890600E-05 / PZERO5 = 0.00000000000E+00 / ORIGIN = 'AIPSNRAO CVAX 15APR87' / DATE = '13/11/87 ' / FILE WRITTEN ON DD/MM/YY HISTORY AIPS WTSCAL = 1.83759533423E+00 / CMPLX WTS=WTSCAL* (TAPE*BSCALE+BZERO) END \end{verbatim} \subsection{Extension Files } There is frequently auxiliary information associated with an image or data set which needs to be saved in the same tape file. Examples of this in AIPS are the Antenna files and CLEAN component files. There is currently a draft proposal to the IAU (Harten {\it et. al.} 1984) defining a standard format for the invention of extension files to be written after the main data records (if any) and defining a ``Tables'' type extension file. The Tables extension files will be able to carry information which can be expressed in the form of a table. AIPS also makes use of a 3-D Table extension which is similar to tables but allows arrays as table entries. The following section will describe the proposed standards which are being incorporated into AIPS. \index{FITS format} \index{FITS tables} \subsection{Standard Extension } The standard, generalized extension file is not a true tape file in the sense that it is separated by tape EOF marks, but is a number of records inside a FITS tape file which contains information of relevance to the file. Each standard extension ``file'' will have a header which is very similar to the main FITS header. This header consists of one or more 2880 8-bit byte ``logical'' records each containing 36 80-byte ``card images'' in the form: \begin{verbatim} keyword = value / comment \end{verbatim} The extension file header begins in the first record following the last record of main data (if any) or following the last record of the previous extension file. The format of the generalized extension ``file'' header is such that a given FITS reader can decide if it wants (or understands) a given extension file type and can skip over the extension file if the reader decides it doesn't. Most of the standards concerning data types and bit orders for the main FITS data records also apply to extension files. One difference is that 8-bit pixel values can be used to indicate ASCII code. The use of the generalized extension ``files'' requires the use of a single additional keyword in the main header: \begin{enumerate} % list nest 1 \item EXTEND (logical) if true (T) indicates that there {\it may} be extension files following the data records and, if there are, that they conform to the generalized extension file header standards. \end{enumerate} % - list nest 1 \index{FITS format} \index{FITS tables} The required keywords in an extension file header record are, in order: \begin{enumerate} % list nest 1 \item XTENSION (character) indicates the type of extension file, this must be the first keyword in the header. \item BITPIX (integer) gives the number of bits per ``pixel'' value. The types defined for the main data records plus 8-bit ASCII are allowed. \item NAXIS (integer) gives the number of ``axes''; a value of zero is allowed which indicates that no data records follow the header. \item NAXIS1 (integer) is the number of ``pixels'' along the first axis (if any). \item NAXISn (integer) is the number of ``pixels'' along the last axis. \item PCOUNT (integer) is the number of ``random'' parameters before each group. This is similar to the definition of random group main data records. The value may be zero. \item GCOUNT (integer) is the number of groups of data defined as for the random group main data records. If an image-like file (e.g., a table file) is being written this will be 1. \item END is always the last keyword in a header. The remainder of the record following the END keyword is blank filled. \end{enumerate} % - list nest 1 \index{FITS format} \index{FITS tables} There are three optional standard keywords for extension file header records. The order, between the required keywords and the END keyword, is arbitrary. \begin{enumerate} % list nest 1 \item EXTNAME (character) can be used to give a name to the extension file to distinguish it from other similar files. The name may have a hierarchal structure giving its relation to other files (e.g., ``map1.cleancomp'') \item EXTVER (integer) is a version number which can be used with EXTNAME to identify a file. \item EXTLEVEL (integer) specifies the level of the extension file in a hierarchal structure. The default value for EXTLEVEL should be 1. \end{enumerate} % - list nest 1 \index{FITS format} \index{FITS tables} The number of bits in an extension file (excluding the header) should be given by the formula: \begin{verbatim} NBITS = BITPIX * GCOUNT * (PCOUNT + NAXIS1 * NAXIS2 * ... * NAXISn) \end{verbatim} The number of data records following the header record are then given by: \begin{verbatim} NRECORDS = INT ((NBITS + 23039) / 23040) \end{verbatim} It is important that the above formulas accurately predict the number of data records in an extension ``file'' so that readers can skip over these ``files''. The data begins in the first record following the last record of the header. Extreme caution must be exercised when inventing new types of extension files. In particular, duplication of types, or several types with the same function, must be avoided. This means that when a new extension file type is invented, it should be as general as possible so that it may be used for other similar problems. \subsection{Tables Extension } A very common type of extension file is one containing data that can be expressed in the form of a table. That is, a number of entries which are all identical in form. A general, self-defining table extension file type is proposed by Harten {\it et. al.} (1984). The following sections describe the proposed format. \index{FITS format} \index{FITS tables} The table extension file uses ASCII records to carry the tabular information. Each table entry will contain a fixed number of entries (although the number can vary between different extension files). For each entry is given (1) a label (optional), (2) the beginning column, (3) an undefined value (optional), (4) a Fortran format to decode the entry, (5) scaling and offset information (optional), and (6) the units (optional). \subsubsection{Tables Header Record } The keywords for tables extension file headers are given in the following: \begin{enumerate} % list nest 1 \item XTENSION (character) is required to be the first keyword and has a value 'TABLE ' for table extension files. \item BITPIX (integer) is a required keyword which must have a value of 8 indicating printable ASCII characters. \item NAXIS (integer) is a required keyword which must have a value of 2 for tables extension files. \item NAXIS1 (integer) is a required keyword which gives the number of characters in a table entry (i.e., ``row''). \item NAXIS2 (integer) is a required keyword which gives the number of entries in the table (i.e., number of rows). A value of 1 is allowed. \item PCOUNT (integer) is a required keyword which must have the value of 0 for tables extension files. \item GCOUNT (integer) is a required keyword which must have the value of 1 for tables extension files. \item TFIELDS (integer) is a required keyword which must follow the GCOUNT keyword. TFIELDS gives the number of fields in each table entry. \item EXTNAME (character) is the name of the table. \item EXTVER (integer) is the version number of the table. \item EXTLEVEL (integer) is the hierarchal level number of the table, 1 is recommended. (optional) \item TBCOLnnn (integer) the pixel number of the first character in the nnn'th field. \item TFORMnnn (character) the Fortran format of field nnn (I,A,E,D) \item TTYPEnnnn (character) the label for field nnn. (optional, order arbitrary) \item TUNITnnn (character) the physical units of field nnn. (optional, order arbitrary) \item TSCALnnn (floating) the scale factor for field nnn. True\_value = tape\_value $\ast$ TSCAL + TZERO. Note: TSCALnnn and TZEROnnn are not relevant to A-format fields. Default value is 1.0 (optional, order arbitrary) \item TZEROnnn (floating) the offset for field nnn. (See TSCALnnn.) Default value is 0.0 (optional, order arbitrary) \item TNULLnnn (character) the (tape) value of an undefined value. Note: an exact left-justified match to the field width as specified by TFORMnnn is required. (optional, order arbitrary) \item AUTHOR (character) the name of the author or creator of the table. (optional, order arbitrary) \item REFERENC (character) the reference for the table. (optional, order arbitrary) \item END must always be the last keyword and the remainder of the record must be blank filled. \end{enumerate} % - list nest 1 \index{FITS format} \index{FITS tables} The TFORMnnn keywords should specify the width of the field and are of the form Iww, Aww, Eww.dd, or Dww.dd (integers, characters, single precision and double precision). If -0 is ever to be distinguished from +0 (e.g., degrees of declination), the sign field should be declared to be a separate character field. \subsubsection{Table Data Records } The table file data records begin with the next record following the last header record and each contains 2880 ASCII characters in the order defined by the header. Table entries do not necessarily begin at the beginning of a new record. The last record should be blank filled past the end of the valid data. \subsubsection{Example Table Header and Data } The first two lines of numbers are only present to show card columns and are not part of the extension file. \index{FITS format} \index{FITS tables} \begin{verbatim} 1 2 3 4 5 6 7 12345678901234567890123456789012345678901234567890123456789012345678901 XTENSION= 'TABLE ' / EXTENSION TYPE BITPIX = 8 / PRINTABLE ASCII CODES NAXIS = 2 / TABLE IS A MATRIX NAXIS1 = 60 / WIDTH OF TABLE IN CHARACTERS NAXIS2 = 449 / NUMBER OF ENTRIES IN TABLE PCOUNT = 0 / RANDOM PARAMETER COUNT GCOUNT = 1 / GROUP COUNT TFIELDS = 3 / NUMBER OF FIELDS IN EACH ROW EXTNAME = 'AIPS CC ' / AIPS CLEAN COMPONENTS EXTVER = 1 / VERSION NUMBER OF TABLE TBCOL1 = 1 / STARTING CHAR. POS. OF FIELD N TFORM1 = 'E15.6 ' / FORTRAN FORMAT OF FIELD N TTYPE1 = 'FLUX ' / TYPE (HEADING) OF FIELD N TUNIT1 = 'JY ' / PHYSICAL UNITS OF FIELD N TSCAL1 = 1.0 / SCALE FACTOR FOR FIELD N TZERO1 = 0.0 / ZERO POINT FOR FIELD N TBCOL2 = 17 / STARTING CHAR. POS. OF FIELD N TFORM2 = 'E15.6 ' / FORTRAN FORMAT OF FIELD N TTYPE2 = 'DELTAX ' / TYPE (HEADING) OF FIELD N TUNIT2 = 'DEGREES ' / PHYSICAL UNITS OF FIELD N TSCAL2 = 1.0 / SCALE FACTOR FOR FIELD N TZERO2 = 0.0 / ZERO POINT FOR FIELD N TBCOL3 = 33 / STARTING CHAR. POS. OF FIELD N TFORM3 = 'E15.6 ' / FORTRAN FORMAT OF FIELD N TTYPE3 = 'DELTAY ' / TYPE (HEADING) OF FIELD N TUNIT3 = 'DEGREES ' / PHYSICAL UNITS OF FIELD N TSCAL3 = 1.0 / SCALE FACTOR FOR FIELD N TZERO3 = 0.0 / ZERO POINT FOR FIELD N END \end{verbatim} The rest of the header block is blank filled. The data cards start on the next block boundary. \begin{verbatim} 0.183387E+00 -0.138889E-03 0.694444E-04 0.146710E+00 -0.138889E-03 0.694444E-04 0.117368E+00 -0.138889E-03 0.694444E-04 0.938941E-01 -0.138889E-03 0.694444E-04 0.183387E+00 -0.138889E-03 0.694444E-04 . . . \end{verbatim} \subsection{3-D Tables Extension }\label{app:C3DBTF2} There are several types of extension files in AIPS which do not fit in a natural way into the FITS Tables format (e.g., gain and calibration tables). These files tend to be large, and conversion to and from ASCII can be expensive. A ``3-D'' extension to the FITS tables format, in which an entry can be a 1-dimensional array, gives the needed flexibility. Use of binary data, including IEEE floating point formats, allows efficient implementation of readers and writers. \index{FITS format} \index{FITS tables} Each row in a 3-D table contains a fixed number of entries (although the number can vary between different extension files). The header is a standard FITS extension header; for each table entry is given (1) the size and data type of the entry, (2) a label (optional), (3) the units (optional), and (4) an undefined value (optional). An entry may be omitted from the table, but still defined in the header, by using a zero repeat count in the TFORMn entry. \subsubsection{3-D Tables Header Record } The keywords for 3-D tables extension file headers are given in the following: \begin{enumerate} % list nest 1 \item XTENSION (character) is required to be the first keyword and has a value 'A3DTABLE' for table extension files. \item BITPIX (integer) is a required keyword which must have a value of 8. \item NAXIS (integer) is a required keyword which must have a value of 2 for 3-D Tables extension files. \item NAXIS1 (integer) is a required keyword which gives the number of 8-bit bytes in a table row. \item NAXIS2 (integer) is a required keyword which gives the number of rows in the table. A value of 1 is allowed. \item PCOUNT (integer) is a required keyword which must have the value of 0 for tables extension files. \item GCOUNT (integer) is a required keyword which must have the value of 1 for tables extension files. \item TFIELDS (integer) is a required keyword which must follow the GCOUNT keyword. TFIELDS gives the number of fields in each table entry. \item EXTNAME (character) is the name of the table. \item EXTVER (integer) is the version number of the table. \item EXTLEVEL (integer) is the hierarchal level number of the table, 1 is recommended. (optional) \item TFORMnnn (character) is the size and data type of field nnn. If repeat count is absent, it is assumed to be 1. A value of zero is allowed. (required, order arbitrary) \item TTYPEnnnn (character) the label for field nnn. (optional, order arbitrary) \item TUNITnnn (character) the physical units of field nnn. (optional, order arbitrary) \item TNULLnnn (character) the value of an undefined value. These are only for integer data types; for floating data types, the IEEE nan (not a number) values are used. \item AUTHOR (character) the name of the author or creator of the table. (optional, order arbitrary) \item REFERENC (character) the reference for the table. (optional, order arbitrary) \item END must always be the last keyword and the remainder of the record must be blank filled. \end{enumerate} % - list nest 1 \index{FITS format} \index{FITS tables } The TFORMnnn keywords specify the width of the field and are of the form rL, rX, rI, rJ, rA, rE, or rD (logical, bit, 16-bit integers, 32-bit integers, characters, single precision and double precision, r=repeat count). If -0 is ever to be distinguished from +0 (e.g., degrees of declination), the sign field should be declared to be a separate character field. \subsubsection{Table Data Records } The 3-D table file binary data logical records begin with the next record following the last header record and each contains 2880 8-bit bytes in the order defined by the header. Table entries do not necessarily begin at the beginning of a new record. The last record should be zero filled past the end of the valid data. \index{FITS format} \index{FITS tables} The data types are defined in the following list (r is the repeat count): \begin{enumerate} % list nest 1 \item {\it rL.} A logical value consists of an 8-bit ASCII ``T'' indicating true and ``F'' indicating false. \item {\it rX.} A bit array will start in the most significant bit of the byte and the following bits in the order of decreasing significance in the byte. A bit array entry consists of an integral number of bytes with trailing bits zero. \item {\it rI.} A 16-bit integer is a two's complement integer with the bits in decreasing order of significance. \item {\it rJ.} A 32-bit integer is a two's complement integer with the bits in decreasing order of significance. \item {\it rA.} Character strings are 8-bit ASCII characters in their natural order. \item {\it rE.} Single precision floating point values are in IEEE 32-bit precision format with the bits in the following order: \begin{verbatim} 1 2 3 01234567890123456789012345678901 seeeeeeeemmmmmmmmmmmmmmmmmmmmmmm sign = -1 $\ast\ast$ s, exponent = eee..., mantissa = 1.mmmmm... . The value is given by: value = sign * 2 **(exponent-127) * mantissa \end{verbatim} Note: these values have a ``hidden'' bit and must always be normalized. The IEEE nan (not a number) values are used to indicate an invalid number; a value with all bits set is recognized as a nan. The IEEE special values (-0., +/- Infinity) are not recognized. A multiplication by a factor of 4.0 converts between VAX F and IEEE 32-bit formats. \item {\it rD.} Double precision floating point values are in IEEE 64-bit precision format with the bits in the following order: \begin{verbatim} 1 2 3 01234567890123456789012345678901 seeeeeeeeeeemmmmmmmmmmmmmmmmmmmm 3 4 5 6 23456789012345678901234567890123 mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm sign = -1 ** s, exponent = eee..., mantissa = 1.mmmmm... . The value is given by: value = sign * 2 **(exponent-1023) * mantissa \end{verbatim} Note: these values have a ``hidden'' bit and must always be normalized. The IEEE nan (not a number) values are used to indicate an invalid number; a value with all bits set is recognized as a nan. The IEEE special values (-0., +/- Infinity) are not recognized. A multiplication by a factor of 4.0 converts between VAX G and IEEE 64-bit formats. \end{enumerate} % - list nest 1 \index{FITS format} \index{FITS tables} \subsubsection{Example 3- D Table Header } The first two lines of numbers are only present to show card columns and are not part of the extension file. \index{FITS format} \index{FITS tables} \begin{verbatim} 1 2 3 4 5 6 1234567890123456789012345678901234567890123456789012345678901234 XTENSION= 'A3DTABLE' / EXTENSION TYPE BITPIX = 8 / BINARY DATA NAXIS = 2 / TABLE IS A MATRIX NAXIS1 = 12 / WIDTH OF TABLE IN BYTES NAXIS2 = 449 / NUMBER OF ENTRIES IN TABLE PCOUNT = 0 / RANDOM PARAMETER COUNT GCOUNT = 1 / GROUP COUNT TFIELDS = 3 / NUMBER OF FIELDS IN EACH ROW EXTNAME = 'AIPS CC ' / AIPS CLEAN COMPONENTS EXTVER = 1 / VERSION NUMBER OF TABLE TFORM1 = '1E ' / COUNT AND DATA TYPE OF FIELD N TTYPE1 = 'FLUX ' / TYPE (HEADING) OF FIELD N TUNIT1 = 'JY ' / PHYSICAL UNITS OF FIELD N TFORM2 = '1E ' / COUNT AND DATA TYPE OF FIELD N TTYPE2 = 'DELTAX ' / TYPE (HEADING) OF FIELD N TUNIT2 = 'DEGREES ' / PHYSICAL UNITS OF FIELD N TFORM3 = '1E ' / COUNT AND DATA TYPE OF FIELD N TTYPE3 = 'DELTAY ' / TYPE (HEADING) OF FIELD N TUNIT3 = 'DEGREES ' / PHYSICAL UNITS OF FIELD N END \end{verbatim} The rest of the header block is blank filled. The binary data starts on the next logical block boundary. The last record of table data is zero filled past the end of the valid data. \subsection{References } \hspace*{ 0.00cm } Wells, Greisen, and Harten 1981, {\it Astronomy and Astrophysics} Supplement series, vol. 44, pp 363 - 370. \smallskip \hspace*{ 0.00cm } Greisen and Harten, 1981, {\it Astronomy and Astrophysics} Supplement Series, vol. 44, pp 371 - 374. \smallskip \hspace*{ 0.00cm } Harten, Grosb{\o}l, Tritton, Greisen and Wells 1984, draft reproduced in the IAU Commission 9 {\it Astronomical Image Processing Circular.} Also appeared in {\it Mem. S. A. It.}, 56, 437 (1985). \end{document} From dwells@fits.CX.NRAO.EDU Tue Oct 17 10:41:30 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA07359; Tue, 17 Oct 89 10:41:27 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA12333; Tue, 17 Oct 89 10:41:00 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA07347; Tue, 17 Oct 89 10:09:35 EDT Return-Path: Message-Id: <8910171409.AA07341@fits.cx.nrao.edu> Status: RO From: dwells@fits.CX.NRAO.EDU (Don Wells) Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Cc: dwells@fits.CX.NRAO.EDU, hanisch@stsci.edu Subject: Re: latest proposal -- some comments and questions Date: Tue, 17 Oct 89 10:09:27 EDT Friends, Bob Hanisch sent me this last night: BH>Date: Mon, 16 Oct 89 17:51:42 EST BH>From: (Bob Hanisch) BH>Subject: latest proposal -- some comments and questions BH> BH>Hi Don. Have been reading through your proposal that came out on DISH-FITS BH>today, and am puzzled by a few things. BH> BH>First, I guess this is not really a puzzle, but just a comment. At various BH>times you equivalence columns in a binary table to keywords, and vice versa. BH>In Section 3.1, a keyword is taken as a virtual column. And in Section 3.5 BH>columns in the table are consider as virtual keywords. I don't really see BH>any difficulty here, except perhaps that this equivalence, or maybe a better BH>term would be `correspondence', be discussed more explicitly. You are right. These two sections should be merged in the next revision of the document (I did mark it as "{\em DRAFT}" and gave the date as "\today"). Yes, "correspondence" is the right word. BH>There is a BH>good analogy in SDAS format data, and for that matter, in FITS group format. BH>There are _global_keywords_, which appear in the main header, and _subset_ BH>specific_ keywords, which appear encoded in the group parameter block. The BH>STSDAS/IRAF system regards both as keywords, period, and values are accessed BH>via the name of the keyword. It's just that the latter keywords are multi- BH>valued within the same data file, whereas the former are fixed. BH> BH>I don't understand Section 3.3, specifically, "the order be reversed so that BH>the last substring, the keyword, will be in the first 8 characters." I could BH>not find any examples of what you meant, but if I take you at your word then BH>the standard header for a FITS image might have something like this BH> BH>WHATEVER= value NRAO-12M GENERAL SINGLDSH / here is some comment info BH> BH>while the same thing, written in an extension header, would be BH> BH>SINGLDSH GENERAL NRAO-12M WHATEVER= value / here is some comment info BH> BH>Is this what you mean? I'll reserve comments until I understand your intent. Sorry for the confusion -- I will have to find better words in the next draft, and add examples. What I meant was that in any header, main or extension, we would have (in the HIERARCH notation) HIERARCH SINGLDSH GENERAL NRAO-12M WHATEVER= value / here is some comment info but that if the same keyword is a *column heading* in a 3-D table, then it should appear as TTYPE13 = 'WHATEVER NRAO-12M GENERAL SINGLDSH HIERARCH' / some comment info There are two reasons for this suggestion: (1) the classic rule that string values be unique/useful in first 8 characters and (2) implementors of routines to display tables will want to be able to truncate the label strings. BH>On the more general question of using binary tables for single-dish data, BH>I don't really have any reservations that this is indeed the most efficient BH>way to go. It is logically straightforward, relatively simple to implement, BH>and clearly the most efficient thing to do within the context of FITS. As BH>has come up in other discussions, however, most notably in Ann Arbor, I think BH>we need to generalize binary tables to include multi-dimensional arrays BH>within the structure. This seems to me to be such an obvious extension, BH>I'm a bit surprised that Bill did not encounter it in dealing with VLA BH>and VLBA data. OK, OK, because I wanted to sell this idea to spectroscopists (1-D) I didn't tell them how I intended to argue in the task force session where this would be settled. I really think that, consistent with 10 years of FITS traditions, we should do an N-dimensional solution if we can. In my section 5 where I use TTYPEnn values of 'NCHAN' and 'SPECTRUM', please read 'NAXIS', 'NAXIS1' and 'MATRIX', except that they would really appear as: HIERARCH SINGLDSH GENERAL NAXIS = 1 / 1-D spectrum (virtual column) TTYPE5 = 'NAXIS1 SINGLDSH GENERAL HIERARCH' / (NCHAN) TTYPE6 = 'MATRIX SINGLDSH GENERAL HIERARCH' / (SPECTRUM) I trust that the generalizations are obvious. An interesting side-effect of this concept is that because the maximum product of the dimensions is set by TFORM6 = '1024F', if NAXIS is 2 or greater this design allows matricies of variable aspect ratio in the table! Note that value strings should be padded with blanks in the first eight columns so that the first classification code substring appears in either column 9 (as shown above) or in column 10 (if the keyword is 8 characters long). BH>The thing I was confused about in your discussion of binary tables, however, BH>was in Section 3.2 - Multiple Feeds. Aren't you overlooking being able to BH>use the prime advantage of binary tables, i.e., table columns that are arrays, BH>by saying that multiple feed data or multi-band data should go into separate BH>structures? I mean, if I have some feed- or band- dependent keywords, which BH>I wish to store in a binary table, can't I define keyword columns of depth BH>equal to the number of feeds or bands? In a simple example, suppose I have BH>four feeds, and one feed-dependent keyword. Then the binary table could look BH>like BH> BH>keyword(4I) feed1(100E) feed2(100E) feed3(100E) feed4(100E) BH> BH>Element 1 of the keyword array corresponds to feed1, element 2 to feed 2, etc. BH>I imagine this may violate some basic tenet of relational databases, right? BH>This is not an area of my expertise. Of course, another thing I'd like to do BH>is write this as BH> BH>keyword(4I) feed(4x100E) BH> BH>which seems a lot cleaner. Actually, probably want (100Ex4). Yes, what you say is the alternative concept. Because I am inexperienced in relational database design I hesitate to use RDBMS arguments one way or the other. I merely think that it will be easier to build the spectral analysis programs and easier to negotiate in the task force if we instead put the four feeds in four separate rows of the table. I.e., I think that both design and negotiation will be easier if there is only one 'MATRIX' column in a table. Anyway, if you take the "correspondence" principle seriously, you would be arguing for multi-valued keywords, which the reviewers forced us to give up in May 1979. BH>Anyway, I don't know if I'm the only one confused or not. One thing we clearly BH>agree about is that the hierarchical keyword situation needs to be resolved BH>regardless of how the data is actually structured. And by the way, don't BH>misconstrue my comments in my memo about the stack approach to hierarchies, BH>that is about whether we need hierarchies at all. I think we do; all I wanted BH>to do was get people to agree to this, and present the arguments I could think BH>of against it. I understood that and, as I said yesterday, I thought you did the community a service by raising the issue. I too hope that they will agree with us that we simply must control our namespace if we want to transmit meanings reliably. BH>Should be a fun meeting. BH> BH>Cheers, Bob Regards, Don From dishfits-request@fits.CX.NRAO.EDU Thu Oct 19 02:18:03 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA01025; Thu, 19 Oct 89 02:18:01 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA16752; Thu, 19 Oct 89 02:17:59 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA01022; Thu, 19 Oct 89 02:12:25 EDT Return-Path: X-Vms-Mail-To: FITS Message-Id: <891019013207.00002B27061@cvax.CV.NRAO.EDU> Status: RO From: "Ray Norris, Australia Telescope, CSIRO, Australia" Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Use of FITS for real-time data logging Date: Thu, 19 Oct 89 01:32:07 EDT Gday all Since nobody from the Australia Telescope will be present at the Greenbank meeting, I'll be sending a written presentation instead. Before I do so however, I'd like to invite comments on an issue which hasn't so far been raised explicitly in this particular forum. The issue is this: we would like to use FITS for logging data in real time, thus avoiding the overhead of transcribing, and minimising the proliferation of formats. But FITS generally requires the number of data to be entered in some form (e.g. GCOUNT, NAXISn) in a header prior to writing the data. However, in real life when observing you don't know in advance how much data you'll be writing. You may, for example, plan to do 100 integrations on a source, but then have to abort the observation half way through because of high wind, a hardware failure, or simply because the on-line monitoring tells you it's not worth continuing. So, you want to record the existing 50 integrations. If you are using FITS on-line, the ways of dealing with this situation are: i) You record the data on disk in an intermediate format, and only write it to tape at the end of the observation. But this defeats the point of using FITS as an on-line format. You're effectively using the intermediate format as your on-line format, and then later using valuable cpu cycles to transcribe the data into FITS. ii) You record each integration as a seperate FITS file. This works, but (a) you end up with a tape full of small files - messy!, and (b) a large fraction of the tape will contain redundant header information. iii) You argue that it's a rare event not to finish the scan, and in those rare instances you simply pad out the FITS file with blanked data so that the data match the header. But this is impractical if you've started a 6-hour scan and decide to abort after one hour. Do you throw away the 1 hour of data or do you watch your tape going round for the next few hours? This problem has been the principal reason that the AT is now using (yet another) adhoc format (RPFITS) rather than true FITS for data logging. This is a shame, as the use of standard FITS would be a much more elegant solution, and would remove the need for subsequent transcription. I would like to invite comments on the proposal that this problem could be circumvented by allowing GCOUNT etc to be set to -1, which means that the length of data is unknown. In this case, any FITS-reading program skipping through the data will simply have to read laboriously through until it sees a block of data starting with a recognisable keyword (this approach is used by RPFITS and it works). Of course for normal data interchange between sites a positve value of GCOUNT would be entered as normal. This extension of FITS can therefore sit quite happily alongside the existing FITS files, but opens up FITS to be useful for on-line logging as well as its original function of data interchange. Regards, Ray Norris From dishfits-request@fits.CX.NRAO.EDU Thu Oct 19 09:03:21 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA01411; Thu, 19 Oct 89 09:03:19 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA17509; Thu, 19 Oct 89 09:03:09 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA01357; Thu, 19 Oct 89 08:57:21 EDT Return-Path: X-Vms-Mail-To: DISHFITS AT NRAO Message-Id: <891019084303.000033B9031@cvax.CV.NRAO.EDU> Status: RO From: "GUILLOTE@FRGAG51" Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Ray Norris issue Date: Thu, 19 Oct 89 08:43:03 EDT Date sent: 19-OCT-1989 11:10:03.18 To: DISHFITS@NRAO Here are my own comments on Ray Norris issue of using FITS as an on-line data format. First general remark : FITS is a Flexible Image TRANSPORT System, NOT an all purpose data format. Use it for what it is. RN> If you are using FITS on-line, the ways of dealing with this situation are: RN> i) You record the data on disk in an intermediate format, and only write RN> it to tape at the end of the observation. But this defeats the point RN> of using FITS as an on-line format. You're effectively using the RN> intermediate format as your on-line format, and then later using RN> valuable cpu cycles to transcribe the data into FITS. Comment : Cpu cycles concern only the Header, not the data because here a mere copy is required. But you are losing many more cpu cycles in your data reduction system formatting and unformatting the FITS header later on. Are they less valuable ? RN> ii) You record each integration as a seperate FITS file. This works, but RN> (a) you end up with a tape full of small files - messy!, and (b) a large RN> fraction of the tape will contain redundant header information. Comment : This is not efficient, but it works. You could add a packing program later in your data processing stream. RN> iii) You argue that it's a rare event not to finish the scan, and in those RN> rare instances you simply pad out the FITS file with blanked data so that RN> the data match the header. But this is impractical if you've started a RN> 6-hour scan and decide to abort after one hour. Do you throw away the RN> 1 hour of data or do you watch your tape going round for the next few hours? No Comment RN> This problem has been the principal reason that the AT is now using RN> (yet another) adhoc format (RPFITS) rather than true FITS for data logging. RN> This is a shame, as the use of standard FITS would be a much more RN> elegant solution, and would remove the need for subsequent transcription. RN> I would like to invite comments on the proposal that this problem RN> could be circumvented by allowing GCOUNT etc to be set to -1, which RN> means that the length of data is unknown. In this case, any FITS-reading RN> program skipping through the data will simply RN> have to read laboriously through until it sees a block of data RN> starting with a recognisable keyword (this approach is used by RPFITS RN> and it works). Of course for normal data interchange between sites RN> a positive value of GCOUNT would be entered as normal. This extension RN> of FITS can therefore sit quite happily alongside the existing FITS files, RN> but opens up FITS to be useful for on-line logging as well as its RN> original function of data interchange. Comment : Your problem is not specifically related to FITS. It happens in any acquisition system in which the data size must be written but is not a priori predictable. I believe you missed the "classical", easy to implement, yet standard and general solution, because you reason in terms of TAPE (which is a purely sequential device) instead of FILE. If you write your data on DISK first instead of directly on tape, then it is easy to update the first 2880 bytes FITS block putting the right GCOUNT when the scan finishes. The tape copy would be done only when the file is closed. If you want to be safe against computer crashes, you could update this header each time a new information is written. I assume that your disk drives and system are reliable enough, so that real-time tape copy is not absolutely necessary. There is no need to change anything in the FITS standard to answer your request. ---------------------------------------------------------------------- A general, and personal, comment on FITS : The success of FITS is largely due to its virtue of STANDARD, and its original simplicity. Some people would argue that it is not Standard or simple enough, and propose RITS (RIGID Image Transport System) instead. I do not follow this opinion: up to now, no unnecessary feature has been added. Instead, I believe we should be carefull not to let FITS become RITS (RANDOM Image Transport System) ! Computing power is no longer a problem, but programming power is still one ! Stephane From dwells@fits.CX.NRAO.EDU Thu Oct 19 10:30:45 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA01543; Thu, 19 Oct 89 10:30:43 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA17732; Thu, 19 Oct 89 10:30:33 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA01518; Thu, 19 Oct 89 10:22:24 EDT Return-Path: Message-Id: <8910191422.AA01512@fits.cx.nrao.edu> Status: RO From: dwells@fits.CX.NRAO.EDU (Don Wells) Sender: dishfits-request@fits.CX.NRAO.EDU To: "PSI%RPEPPING::RNORRIS"@nssdca.gsfc.nasa.gov Cc: dishfits@fits.CX.NRAO.EDU Subject: Re: Use of FITS for real-time data logging Date: Thu, 19 Oct 89 10:22:17 EDT I would like to add one more comment to Stephane's response to Ray's issue. That is that in addition to using disk as an intermediate storage medium one can also use RAM. In the past this was not practical because RAM was too expensive and data acquisition machines never had enough RAM. This situation is now changed completely. RAM is *very* cheap now, and current state-of-the-art real-time CPUs can easily have large amounts of it. For example, the VLBA Correlator has Motorola MVME147 CPUs. These are two VME slots wide (the latest model of the 147 is only one slot wide) and have a 25 MHz 68030 (5 MIPS), 68882 FP coprocessor, Ether and SCSI controller, and 8 MB of RAM. NRAO is able to buy these for US$4200. It would be easy to make a competitive procurement of one or more VME cards of additional RAM if we needed it. The point is that it is now easy to have I/O buffers that are quite large, to organize data in memory and update the header just before writing to the device. The data can be in files of a few megabytes, each with a proper header. I am not necessarily saying that this approach is preferable to constructing larger files on a scratch disk, as Stephane suggested, I am merely pointing out that it is yet another option for designers. A small technical point about using disk as a buffer is that it can boost performance of a streaming tape device by avoiding stop-start overhead at intermediate data rates. One must be careful that aggregate bandwidth on the backplane can support concurrent I/O from device to RAM, RAM to disk, disk to RAM, and RAM to tape. At sustained rates of 200 kB/s this is probably not a problem (assuming independent DMA controllers), somewhere above 500 kB/s it will start to be a problem. -Don From dishfits-request@fits.CX.NRAO.EDU Fri Oct 20 07:22:46 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA03649; Fri, 20 Oct 89 07:22:44 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA20439; Fri, 20 Oct 89 07:22:38 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA03646; Fri, 20 Oct 89 07:12:39 EDT Return-Path: X-Vms-Mail-To: DISHFITS Message-Id: <891020070256.000006B3031@cvax.CV.NRAO.EDU> Status: RO From: "Ray Norris, Australia Telescope, CSIRO, Australia" Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Use of FITS for real-time data logging Date: Fri, 20 Oct 89 07:02:56 EDT Many thanks for your comments on real-time use of FITS. Before I reply, I'd like to make a general observation. FITS has been tremendously successful in easing the problems of data interchange, and for that we must be eternally grateful to its originators. My comments are not intended to point out deficiencies in FITS (in the context of its original function) but to suggest a simple extension which enables it to be used efficiently for an additional function: that of data logging. Like most observatories, we have guest observers who bring their own favourite data reduction packages, or want to take the data back with them. At present it is necessary to transcribe from one of a number of 'internal' formats to FITS or some other 'external' format. We are proposing to rationalise our internal formats, and in fact FITS with the extension I propose makes for a very convenient internal format. I agree that there is in principle no reason why you shouldn't go on generating multitudes of various different ad hoc internal formats at various observatories, except that it's very wasteful of precious manpower in inventing the formats, writing transcription programs, and transcribing data. Having said all that, my specific comments on your replies are as follows: stephane> FITS is a Flexible Image TRANSPORT System, NOT an all purpose data stephane> format. Use it for what it is. Sorry, I disagree. It could, with a small modification, be even more useful. RN> i) You record the data on disk in an intermediate format, and only write RN> it to tape at the end of the observation. But this defeats the point RN> of using FITS as an on-line format. You're effectively using the RN> intermediate format as your on-line format, and then later using RN> valuable cpu cycles to transcribe the data into FITS. stephane> Comment : stephane> Cpu cycles concern only the Header, not the data because here a mere stephane> copy is required. But you are losing many more cpu cycles in your stephane> data reduction system formatting and unformatting the FITS header stephane> later on. Are they less valuable ? I think we have to distinguish between two cases: (i) recording the data in one formnat, and then using an offline program to transcribe, in which case your comments apply, but see my comments at the start of this mail. (ii) using the observing program to transcribe, in which case - yes cpu cycles here are more valuable than offline cycles! In the real-world the real-time logging computers tend to be under-powered, and tend not to have the number-crunching capacity of off-line processors, which can always spend a little longer than real time if a particular data set is particularly voluminous. RN> ii) You record each integration as a seperate FITS file. This works, but RN> (a) you end up with a tape full of small files - messy!, and (b) a large RN> fraction of the tape will contain redundant header information. stephane>This is not efficient, but it works. You could add a packing program stephane>later in your data processing stream. I agree - but's it's still messy and inefficient! stephane>Your problem is not specifically related to FITS. It happens in stephane>any acquisition system in which the data size must be written stephane>but is not a priori predictable. I believe you missed the "classical", stephane>easy to implement, yet standard and general solution, because you stephane>reason in terms of TAPE (which is a purely sequential device) instead of FILE. stephane>If you write your data on DISK first instead of directly on tape, stephane>then it is easy to update the first 2880 bytes FITS block putting stephane>the right GCOUNT when the scan finishes. The tape copy would be stephane>done only when the file is closed. If you want to be safe against stephane>computer crashes, you could update this header each time a new stephane>information is written. I assume that your disk drives and system stephane>are reliable enough, so that real-time tape copy is not absolutely stephane>necessary. Writing to disk is fine for most present systems, but becomes impractical as data volumes get really large ( to take an extreme, non-single-dish application, the AT produces data at a max rate of about 1 Gbyte/hr. Future high-resolution multi-beam single dish systems with rapid scanning may also approach this data rate). Here you need either tape or optical disks, which are generally WORM, so you can't easily overwrite the header. And you don't can't get round it for WORM drives by leaving the writing of the header till later, because then the system becomes fragile to machine crashes. stephane>There is no need to change anything in the FITS standard to answer stephane>your request. I agree that there is no NEED to have a universal format - we can certainly continue with a proliferation of ad hoc internal formats if thats what the community wants. But the same arguments that led to the birth of FITS can be extended: the astronomical community is generally under-manned - why make life difficult when with a minor change it could be so easy! stephane>The success of FITS is largely due to its virtue of STANDARD, and stephane>its original simplicity. Some people would argue that it is not stephane>Standard or simple enough, and propose RITS (RIGID Image Transport stephane>System) instead. I do not follow this opinion: up to now, stephane>no unnecessary feature has been added. Instead, I believe we stephane>should be carefull not to let FITS become RITS (RANDOM Image stephane>Transport System) ! Yes but let's not allow it to become OITS (Obsolete Image Transport System)! In response to Don's comments about RAM etc, I guess we should look very carefully at data rates. The AT rate cited above is an extreme case, and may appear not to be relevant to the single-dish problem. However, I believe we should be looking ahead, rather than coping with problems that confront us NOW. Multi-feed, high-spectral-resolution surveys may bring the hardware to its knees (as Greenbank has already demonstrated - oops sorry - shouldn't have said that!) and any standard designed now should be designed to last for a few years. To summarise, my arguments are that (i) to use FITS for on-line logging makes sense in terms of efficient use of manpower and other resources, (ii) it can be achieved with present FITS by some form of buffering, but that is somewhat inefficient and may get impractical as data rates increase, and (iii) allowing the option of GCOUNT=-1 has no effect whatsoever on the speed, efficiency, and implementation of present FITS. You guys who don't like it can ignore it, and write out GCOUNT as at present, EXCEPT that the FITS reading programs must be extended to accomodate the possibility that they may encounter data with GCOUNT=-1. Cheers, Ray Norris From dishfits-request@fits.CX.NRAO.EDU Fri Oct 20 13:38:12 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA04196; Fri, 20 Oct 89 13:38:10 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA21167; Fri, 20 Oct 89 13:37:55 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA04193; Fri, 20 Oct 89 13:28:24 EDT Return-Path: X-Vms-Mail-To: DISHFITS AT NRAO Message-Id: <891020131623.00000CAF031@cvax.CV.NRAO.EDU> Status: RO From: "GUILLOTE@FRGAG51" Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: A comment on Don Wells reply to T.Forveille and S.Guilloteau proposal Date: Fri, 20 Oct 89 13:16:23 EDT Date sent: 20-OCT-1989 09:30:31.22 To: DISHFITS@NRAO Dear friends, I have a comment on Don Wells answer to the our (T.Forveille and myself) document. It concerns point 3) of his answer, which presumably was corresponding to item (*5) (XCOORD for not regularly sampled data). There has been here some confusion between "irregularly sampled" and "randomly sampled" data, which originate from an incomplete description in our document. In our point (*5), we were implicitely refering to FREQUENCY axis, not to mapping coordinates. The frequency axis may be irregularly sampled, (e.g. Acousto Optical Spectrometers in radio telescopes) but this is usually in a systematic (not random!) way. If one envisage to store part of a 3-D spectral mapping with such a frequency axis, the use of a Table extension to indicate Frequency Value versus Frequency Pixel is probably unavoidable (**). Using random parameter instead would imply more or less doubling the data size (more precisely adding (np-1)*nf values, where np is the number of map points, and nf the number of frequency channels). This is specially important not only for single-dish data, but also for many telescopes with spectroscopic capabilities (HST, ISO, and most optical telescopes (***). Stephane (**) In fact this is similar (NOT identical) to the extra 'IF' axis, connected to the 'AIPS CH' table, mentionned in Section C.3.2 of Don Wells proposal. (***) And here comes again the question of Frequency versus Wavelength. A regularly sampled axis in one domain is no longer regularly sampled in the other. From dishfits-request@fits.CX.NRAO.EDU Wed Oct 25 09:37:12 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA08577; Wed, 25 Oct 89 09:37:09 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA04167; Wed, 25 Oct 89 09:36:41 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA08565; Wed, 25 Oct 89 09:26:11 EDT Return-Path: X-Vms-Mail-To: DISHFITS AT NRAO Message-Id: <891025091334.000038C8031@cvax.CV.NRAO.EDU> Status: RO From: "PTW@STARLINK.RUTHERFORD.AC.UK" Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: comments on single-dish FITS proposals Date: Wed, 25 Oct 89 09:13:34 EDT Date sent: Wed, 25 Oct 89 14:12 GMT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 4118; Wed, 25 Oct 89 14:11:09 BST Received: from RL.IB by UK.AC.RL.IB (Mailer X1.25) with BSMTP id 9030; Wed, 25 Oct 89 14:11:08 BS Via: UK.AC.RL.STAR; 25 OCT 89 14:11:00 BST To: DISHFITS@NRAO Comments on single-dish FITS proposals -------------------------------------- From: P.T.Wallace Starlink Project Rutherford Appleton Lab Chilton, Didcot, Oxon OX11 0QX PTW @ STARLINK.RUTHERFORD.AC.UK 19457::PTW These are hurried comments based on one pass of the document. Apologies if I've misunderstood anything. PHILOSOPHY In our recent work on Starlink data formats, we have distinguished between two different design philosophies. The one we have adopted ourselves attempts to describe the relationships between the different data components sufficiently well for programs to process data from diverse sources correctly. The more common philosophy - in fact the only one which astronomers believe in and think they understand - is what we call the "thought of everything", or TOE, philosophy. The Single dish format seems to be an example of the latter. The TOE idea is that once you have given implementors enough pigeonholes into which to stuff the data generated by their telescope and instrument, you have thereby defined the "meaning" of everything in data processing terms. I don't believe it's that easy in practice. This is not to say that it isn't possible to use such designs, which at least mean that FITS readers can accept the contents of a tape, and that simple utilities can do things like plotting pixel value against x-coordinate. But I don't believe an astronomer can do astrophysics from a FITS tape without an accompanying letter describing how the various components have been used. To proceed rapidly to the point, I would aim for greater rigor in some parts of the single-dish proposal, and to move some of the present items out of the main format into unashamedly site-specific extensions. POINTING CORRECTIONS I think it is a mistake to try and define a standard set of telescope pointing corrections. While it is true that any two-axis mount will have just six numbers describing the purely geometrical terms involved, and many people (myself included) like to base a pointing model solidly on these six terms, it is of course possible to define a pointing model in a various completely different ways - lookup tables, polynomials or harmonics, etc. Moreover, all telescopes will have extra terms, and some may have auxiliary readings (from sensors mounted on the structure) or quantities involved in attempts to deal with hysteresis. Also, I have on some telescopes found that there is no easy way to get at the basic pointing information. On the JCMT, for example, the current pointing coefficients appear to be relative to whatever values are embedded in the system, and, when pointing tests are done, no attempt is made to log the raw encoder readings to allow a model to be fitted ab initio. (Not how I would do it.) There are also the fearful complications of subreflectors which move to correct for deforming dishes, chopping secondaries, etc. So I propose that all telescope model information be entirely site- specific. Also, I distrust the idea of user-supplied corrections, especially when it isn't made clear whether the one in azimuth (etc) is dA or dA*cos(E). I would rather leave this information in the (site-specific) telescope model. SIDEREAL TIME It doesn't say whether this is mean or apparent. I propose the latter. OMISSIONS If one of the objectives is to be able to reconstruct the pointing, consider including (i) station coordinates and (ii) UT-UTC. The station coordinates should be longitude (east of Greenwich), latitude (geodetic), and height above the current IAU reference spheroid. The longitude and latitude should, formally, be as affected by polar motion. ENVIRONMENT PARAMETERS Does "index of refraction" (REFRAC) mean the refractive index, or the tan Z refraction constant? I'm not keen on the former, and would supplement the latter with one more term (in tan**3 Z). Is it necessary to have HUMIDITY and DEWPT and MMH20? I would go for HUMIDITY alone. To calculate the radio/mm refraction properly (also important at large Z in the optical case) you need the tropospheric lapse rate, in deg K per metre. You also need the height, which I have included in my "station coordinates" suggestion. SOURCE COORDINATES There is some evidence of a confusion between the term "epoch" (meaning a moment in time) and "equinox" (shorthand for "the mean equator and equinox at a given epoch"). I suspect the keyword SINGLDSH GENERAL EPOCH means equinox for example. I recommend that the following coordinate systems are recognised: a) Pre-IAU-1976 mean RA,Dec for any given equinox and epoch. The equinox should be Besselian (and is usually 1950). The epoch is important (because the system is rotating relative to an inertial frame). E-terms are included (i.e. the quoted position embodies the E-terms just like catalogue positions). Recommended name "FK4". b) FK4 but without E-terms. I suggest the name "BN", standing for Bessel-Newcomb. c) Post-IAU-1976 mean RA,Dec for any given equinox and epoch. The equinox should be Julian (and is usually 2000). The epoch is not required unless you are going to bother with proper motions. E-terms are not involved, of course. Recommended name "FK5". d) Galactic coordinates. The document could mention that you should really calculate these from the "BN" B1950 position rather than from "FK4". Names: "L2B2", or "GAL" would be OK. e) Mean ecliptic and equinox of date, IAU 1976. Recommended name "ECL", or "MEAN_ECL" or something. f) Geocentric apparent RA,Dec, IAU 1976 + 1980. Recommended name "RADEC". Note that "FK5" to "RADEC" consists of precession, nutation, annual aberration, and the gravitational lens effect of the Sun. g) Geocentric apparent HA,Dec. Recommended name "HADEC". Computed from "RADEC" by subtracting RA from LAST. I am doubtful whether it is worth providing for topocentric HA,Dec, which differs from "HADEC" by the diurnal aberration. h) Topocentric Az,El, obtained from topocentric HA,Dec by rotating into Az,El using the long,lat as affected by polar motion. Recommended name "TOPO_AZEL". i) Observed Az,El, obtained from "TOPO_AZEL" by adding the effects of atmospheric refraction. Recommended name "OBS_AZEL". j) Raw Az,El, obtained from the telescope encoder systems. This is related to "OBS_AZEL" by the telescope pointing model. Recommended name "ENC_AZEL". The proposal needs to specify for all Az/El items whether they are raw, observed, or topocentric (and whether Az means Az or dAz*cos(El)!). ------------------------------------------------------------------------- From dishfits-request@fits.CX.NRAO.EDU Wed Oct 25 13:46:53 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA08871; Wed, 25 Oct 89 13:46:51 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA04640; Wed, 25 Oct 89 13:44:10 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA08863; Wed, 25 Oct 89 13:34:54 EDT Return-Path: >From: hoh-2!rww (R. W. Wilson) Message-Id: <8910251443.AA06406@orion.att.com> Status: RO From: rww@hoh-2.att.com Sender: dishfits-request@fits.CX.NRAO.EDU To: arpa!NRAO.edu!DISHFITS@hoh-2.att.com Date: Wed, 25 Oct 89 10:43:42 EDT Subject: Comments on a FITS Standard for Spectral Line Data >From Robert W. Wilson, AT&T Bell Laboratories Dear Single Dish FITS Conference Participants: The most important thing, of course, is to have A STANDARD. I will almost certainly write a reader and writer for what ever new standard emerges. The one thing that I think must be changed from present FITS practice is that multiple spectra should be allowed in a single file. This could be done as easily as simply writing multiple spectra one after the other in a file. It would take a minimum of cpu time to read the header of an unwanted spectrum and calculate how many (2880 byte) blocks to skip. There would be no reason to announce how many spectra are comming; the EOF would tell you when you are done. This would make exabyte tapes and disk files usable and all other efficiency considerations would be of little consequence for the amount of data that I foresee transporting in and out of Bell Labs with FITS. (Tape records blocked to a multiple [10?] of 2880 bytes would then be desirable for IBM tapes.) The other thing which is badly needed is a set of standard keywords. I think that there should be a required core set which would be adequate for reduced data. An extension set should be strongly encouraged if the information is available and applicable. As an example of what I mean by this is in the data header which we have been using at the Bell Labs 7-meter antenna. A number of the parameters correspond to the lock system which was used at the 36-foot in the early 1970's even though our lock loop is different. The only loss in keeping the old definitions is a few arithmetic operations per spectrum in the observing program (and occasionally puzzling over the code). There was considerable saving during the transition period when we had both kinds of data. I hope that I will not have to make use of very many telescope dependent parameters when reading data from other observatories. Observatories with idiosyncratic instruments should reduce their export data so that it can be described in a normal way rather than expecting all users to add special code to handle their special case. I include in this category AOS spectrometers with non-linear frequency sampling. If the original sampling is adequate, resampling on a regular grid should not change the data (although some of the original high frequency noise may be removed). Of course transfer using FITS will need to handle the raw data in whatever from it is produced for internal transfer and transfer to frequent observers. I like Bob Hanish's proposal for hierarchical keywords better than Don Wells' proposal of stacking them up on the line with the most particular word first. I worry that this extensibility will be abused. I like the idea of 3-D binary tables for efficient transfer of large amounts of data, but don't completely understand them yet. If I am forced to implement them, I might keep our raw data in that format, since, except for the initial header the data file is already a binary table. As a final comment, I think that observers should be able to refer to all of the data taken at the same time by a common observation number. I am also used to integer 'scan' numbers and don't have a good proposal for the case of multiple feeds/receivers/backends. For exchange with foreign systems, each spectrum should be a separate FITS entity with some form of observation number in it. (Actually this would argue against keeping multiple backend raw data in FITS format.) Bob Wilson rww@hoh-2.att.com 201-888-7120 From dwells@fits.CX.NRAO.EDU Thu Oct 26 15:01:52 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA11069; Thu, 26 Oct 89 15:01:50 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA07656; Thu, 26 Oct 89 15:01:40 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA11020; Thu, 26 Oct 89 14:54:19 EDT Return-Path: Message-Id: <8910261854.AA11014@fits.cx.nrao.edu> Status: RO From: dwells@fits.CX.NRAO.EDU (Don Wells) Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Hierarchical keywords Date: Thu, 26 Oct 89 14:54:15 EDT ----- Begin Included Message ----- Date: Thu, 26 Oct 89 13:50:06 EDT From: "ESOMC0::PREBEN"@cvax.CV.NRAO.EDU Subject: Hierarchical keywords To: dwells@fits.CX.NRAO.EDU X-Vms-Mail-To: NRAO::DWELLS Message-Id: <891026135006.00003FA4031@cvax.CV.NRAO.EDU> Comments on Hierarchical Keywords in FITS ------------------------------------------- Preben Grosbol 1989-Oct-26 There are several reasons to consider a hierachical concept for keywords in FITS. The main arguments for a more flexible syntax than the one offered for basic FITS keywords are: 1) A total of 8 characters is too limited to give a meaningfull description of the content. 2) The small size of keywords increrase the chance of different institutes using the same keyword with different meaning. 3) There are not clear definition of the name space for keywords which can avoid multi-defined keywords. 4) The general problem of encoding small arrays as keywords is not solved. A solution to the three first items was proposed by AIPS (using HISTORY cards). It was refined with the concept of subject/group reserved keywords like SNGLDISH for the Single Dish community. A more general definition for a HIERARCH keyword was then put foreward based on the AIPS and Single Dish experience. The main argument against the HIERARCH is number of redundent characters defining the high levels. This limits the usefull part of the 80 character card significantly. The suggestion of Bob Hanish to introduce a HKBEGIN and HKEND keywords to define the hierarchical level and use the basic keyword syntax for the lowest level would solve the first requirement and make more efficient usage of the limited space on a FITS keyword card. However, it makes the present situation even worse for 2) and 3). It would allow new low-level hierarchical keywords to be placed in column 1-8 where old FITS readers would mix them with other simple non-standard FITS keywords. It is MUST be ensured that additions to the FITS standard does not generate wrong results even for old readers. This was the case with the General Extensions, Blocking and Floating Point agreements. Thus, the proposal of Bob Hanish for HKBEGIN/HKEND is NOT acceptable due to the wrong interpretation of old FITS readers. The new concept of hierarchical keywords MUST look like comments to old readers. A number of options will provide acceptable solutions. Two main groups could be discussed: 1) The proposal already presented by Grosbol and Wells of type: HIERARCH level1 level2 level3 .. .. keyword = value / comment or 2) A mix of this and the proposal of Hanish: HIERARCH BEGIN level1 level2 level3 .. .. HIERARCH keyword = value / comment HIERARCH END where the HIERARCH keyword can be exchanged with any other reasonable combination of letters. The 'level1' identifiers must be controller and should either be allocated to subject or groups. With the choice of a special keyword (e.g. HIERARCH) one can upgrade the syntax for the remaining of the card. It would be possible to include a concept of small arrays in one of the following styles: 1) HIERARCH ARRAY(4) = 1, 2, 3, 4 / comment indicating an array with 4 elements in F77 list format of the type of the first element. 2) HIERARCH ARRAY(2..4) = 223.2, 33.3, -0.4 / comment indicating array elements 2, 3 and 4 in F77 list format of the type of the first element. 3) HIERARCH ARRAY(4I) = 1, 2, 3, 4 / comment indicating an array with 4 elements in F77 list format of integer type. To provide space for small arrays a continue card would be needed. It could be done by repeating the first keyword part (AIPS style) or as the last non-comment character use a special sign e.g. '-'. ----- End Included Message ----- From dishfits-request@fits.CX.NRAO.EDU Thu Oct 26 16:47:02 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA11211; Thu, 26 Oct 89 16:47:00 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA07936; Thu, 26 Oct 89 16:45:01 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA11208; Thu, 26 Oct 89 16:37:24 EDT Return-Path: Message-Id: <8910261304.AA13092@oso.chalmers.se> Status: RO From: Michael Olberg Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@nrao.edu Subject: Single-dish FITS Date: Thu, 26 Oct 89 14:04:35 -0100 I would like to make a last minute contribution on two issues before the workshop next week. a) Assuming that a new FITS format would impliment a hierarchy of keywords covering GENERAL, SNGLDISH GENERAL and finally SNGLDISH WHATEVER keywords or anything equivalent to that, could we produce FITS reading programs which could learn about anything that had not been general enough to be implemented right from the start? I am thinking of something like this: Allow definition of FITS keywords within FITS: 0........1.........2.........3.........4..................................8 123456789012345678901234567890123456789012345.........................67890 DEFINE = 'SEST-15M' / start definition of SEST specific keywords KEYWORD1= 'F10.1 ' / define KEYWORD1 and format how to read it KEYWORD2= 'L10 ' / example of a keyword holding a logical ... KEYWORD3= 'A10 ' / ... and character information ENDDEF / end keyword definition section This definition section could be written out as a first block on a tape integrated in a standard FITS header and should be thoroughly commented. Now, if any of the following headers contained TELESCOP= 'SEST-15M' / telescope used which is one of the allowed keywords in FITS, only then would the program be prepared within the same header to understand and correctly read any lines like SNGLDISH SEST-15M KEYWORD1= 12.3 SNGLDISH SEST-15M KEYWORD2= T SNGLDISH SEST-15M KEYWORD3= ' a string ' If the TELESCOP= line had not been present, the SNGLDISH SEST-15M lines should be ignored. This would allow slightly different implementation of identical keywords at different telescopes, i.e. everybody may use very obvious keywords without knowing how exactly they are used somewhere else. b) Along the lines of adopting IEEE floating point formats for binary encoding of data, which I very much agree with, let's also adopt a standard for an extended 8 bit character set. Maybe we could even agree on how to encode greek, japanese etc. characters. If we want others to accept a standard proposed by us, we should be open minded about standards that other commitees have already come up with.  From dishfits-request@fits.CX.NRAO.EDU Fri Oct 27 02:43:04 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA11955; Fri, 27 Oct 89 02:43:02 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA09029; Fri, 27 Oct 89 02:42:51 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA11952; Fri, 27 Oct 89 02:31:57 EDT Return-Path: X-Vms-Mail-To: DISHFITS Message-Id: <891027021112.000045A4071@cvax.CV.NRAO.EDU> Status: RO From: "Ray Norris, Australia Telescope, CSIRO, Australia" Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Comments from the AT Date: Fri, 27 Oct 89 02:11:12 EDT Comments on Single Dish FITS formats and requirements for observing Ray P. Norris Australia Telescope National Facility 1 INTRODUCTION Since the Australia Telescope (AT) can't send anyone to the Greenbank meeting, this document represents our submission to the meeting. There are three parts to this submission: (i) Comments on the structure of a future single-dish FITS standard (ii) An overview of what we have and where we're going (iii) An Appendix proposing changes to the FITS standard to make it more suitable for real-time use. 2 COMMENTS ON FUTURE SINGLE DISH FITS STANDARDS We would like to change most of our internal data formats to FITS, for all the usual reasons. To be able to do so, we require that (i) The FITS standard is extended to make it more suitable for real-time data logging (see my comments in the Appendix to this submission). In particular for spectral line use we need to be able to write many spectra into a file without knowing how many there are to come (see comments by Norris and by Wilson). (ii) We need to put in a lot of site-specific information, and so we would strongly support the concept of hierarchical keywords. The most promising format for this is option (2) in Preben Grosbol's recent Email (the mix of Grosbol, Wells, and Hanisch proposals). (iii) The idea of binary 3-D tables seems excellent, and we would support the use of IEEE standards for these. 3 THE AT: WHAT WE HAVE AND WHERE WE'RE GOING The AT consists of (a) the 6km compact array at Culgoora, which is a 6 x 22m East-West synthesis array with operation planned up to 115 GHz, (b) the Parkes 64-m antenna which operates up to 43 GHz, (c) the 22-m Mopra (Siding Spring) antenna which will operate up to 115 GHz, (d) the Long-Baseline Array which consists of the antennas above plus others used collaboratively with other Australian groups to produce baselines upto 3000 km. Here we are concerned only with single-dish observations, and so restrict attention to the single-dish use of the Parkes and Mopra antennas. Parkes is used for a broad range of single-dish experiments, including spectral line studies, polarisation (largely using the Bonn polarimeter), continuum surveys, and pulsar work. We also hope to install the NRAO multi-beam feed on Parkes in early 1990. At present most of the software differs radically from one observing mode to another, and there is little in common either in the software or in the data format. Mopra is not yet operational, but will become operational over the next few months, and will be equipped with an 8000-channel correlator for spectral-line use. 3.1 SINGLE-DISH SPECTRAL LINE OBSERVATIONS At present spectral line work at Parkes makes use of the SPECTRA observing program and the 1024-channel correlator. Data output is in a home-grown SPC format, which many would like to see replaced by FITS. Spectral-line data reduction is generally done using a 'temporary' program (S) written by Rick Forster some years ago but which has evolved into the default spectral line reduction package. We plan to replace the 1024-channel correlator shortly by a 8000-channel VLSI correlator (identical to the one for Mopra) and this would be a good opportunity to overhaul the SPECTRA observing program. It is expected that the observing software at Parkes and Mopra will be largely identical (except, of course, for the antenna control software) and so we plan to install a version of SPECTRA at Mopra. Whilst many users have come to know and love the S data reduction program, it is felt that a more sophisticated program would be useful, and so we are currently flirting with other programs such as SLAP. 3.2 RESPONSE TO QUESTIONS IN ORIGINAL CIRCULAR In the original circular about the Greenbank meeting, a number of questions were asked of each observatory. Here are our answers to those questions. Many of these answers reflect the fact that we are currently undergoing a transition from an in-house observatory/astronomical institute to a national facility, and as such we expect to provide a higher level of support than we have in the past. (i) What calibration/reduction/analysis software do you have? We have a variety of observing software, most of which is not suitable for export. Similarly, most of our data reduction software is site-specific and not suitable for export. A notable exception is AIPS which is heavily used for AT, VLA, and VLBI data. We would welcome, and probably participate in, any initiative to draw on the experience gained with AIPS and other packages to produce a real user-friendly single-dish spectral-line reduction program. (ii) How is your software effort organised in-house? While the bulk of our software effort is our Epping headquarters site, much of the on-line software has been written either by users or by staff at the Parkes and Culgoora observatories. Whilst much of this software has been immensely succesful, there exists no coherent plan for its development and this is one area that we would like to put some planning into. (iii)How do you support your users? At present we have real-time support for users, plus a limited amount of data reduction support. This user support will be increased to a cradle-to-grave philosophy in the future. (iv) What are your upgrade paths? Hardware: at present we have VAXes providing real-time computing, and that will continue for the immediate future. Off-line, our main processor is a Convex C-220 together with a number of Sun workstations, SPARCstations, etc. We expect the use of these workstations to increase, and we would like to see more use made of distributed computing. We are also likely to acquire a high-end workstation (e.g. a Sun 4/370 with TAAC board) for algorithm and visualisation development. On the other hand, use of X-windows means that we may make more use of dumb terminals like X-terminals, Macs, and PC's in the future. Software: We have grave concerns over the future of our off-line support, and believe that community-wide development is the only way in which sufficient manpower can be brought to bear on the problem of providing the level of software that we want and expect. Organising and providing this development is likely to be extremely difficult, however. Meetings such as this Greenbank meeting are a good first step. (v) How do you view shared code? As above - this is the best way forward for data reduction. (vi) Could single-dish work make better use of AIPS/IRAF/etc.? It could, but only if manpower is available to include within these packages the functions that single-dish users need. APPENDIX: USE OF FITS FOR REAL-TIME DATA LOGGING. I've already distributed my comments about the need for modifications to the FITS standard to make it suitable for real-time logging, and I won't repeat the detailed arguments here. Briefly, however, the argument runs as follows: FITS was not designed for on-line use, and isn't, at present, suitable for on-line use. However, with some small extensions, it could be made suitable for on-line use. The lack of a good on-line format has resulted in a proliferation of internal ad hoc formats around the world, with a resulting waste of manpower and resources in re-inventing various site-specific wheels. Each site then develops software for transcribing data from its internal format to FITS (more manpower!) and then observing assistants spend time transcribing tapes. This will become more of a problem as we move towards using common data reduction software in several sites. Perhaps more importantly, as multi-beam feeds and large correlators increase the volume of data (which could result in peak data rates of 100 kbyte/s in the near future), the resources needed to transcribe data approach those needed to take the data in the first place. Transcribing data then becomes a full-time task rather than a little job to be done before you back up the data, and could become a significant drain on observatory resources. I am therefore advocating the extension of FITS to make it suitable for real time use. To summarise my argument, I contend that (i) to use FITS for on-line logging makes sense in terms of efficient use of manpower and other resources, (ii) FITS doesn't let you do this efficiently at present, because FITS requires you to know in advance how much data you're going to write before you start writing the file, and in general you don't know this in real-time data logging, (iii) FITS is fragile if all the tables are put at the end of the data, (iv) the alternative of writing many small files is cumbersome and causes problems of data management. Point (ii) CAN be overcome with present FITS by some form of buffering, but that is somewhat inefficient and may get impractical as data rates increase. A better solution is to allow the option of GCOUNT=-1, meaning that you don't know how many groups you will be writing, and that any FITS reading program must in this case check each block for the start of a table, or (if scans are concatenated in a file, which I believe also to be desirable) the start of a new header. This extension of the FITS standard has no effect whatsoever on the speed, efficiency, and implementation of present FITS. Those who don't like it can ignore it, and write out GCOUNT as at present, EXCEPT that the FITS reading programs must be extended to accomodate the possibility that they may encounter data with GCOUNT=-1. Point (iii), the problem of fragility, can be overcome by writing headers and tables within the body of the file, as well as at the start and end. In the AT data format, each scan of data is preceded by a FITS-like header and tables, but all scans are contained within the same file. In this way, the damage caused by a machine crash or parity error is confined to one scan. An alternative approach to overcoming fragility has been suggested by Eric Greisen, who proposes splitting up an observation into several files - writing a fixed number of records in each tape file and then starting a new tape file. This approach is similar to mine except that I prefer to have all the data in one file. As a general point, I would propose extending the FITS standard to allow many scans to be recorded in one file - effectively allowing the concatenation of many FITS files into one file. This approach (plus the others suggested above) has been used with great effectiveness for the AT data format. From dishfits-request@fits.CX.NRAO.EDU Fri Oct 27 15:29:34 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA13116; Fri, 27 Oct 89 15:29:31 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA10595; Fri, 27 Oct 89 15:28:40 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA13111; Fri, 27 Oct 89 15:17:13 EDT Return-Path: Message-Id: <8910271919.AA11805@wells> Status: RO From: jball@wells.haystack.edu (John Ball) Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@nrao.edu Cc: jball@wells.haystack.edu Subject: Having FITS Date: Fri, 27 Oct 89 15:19:40 EDT NEROC Haystack Observatory Westford, Massachusetts 01886 USA Memorandum To: Distribution From: John Ball 1989 October 27 Subject: Having FITS At Haystack, we now have the following three programs to write FITS tapes from Haystack spectral data: 1) FITS reads stashes and writes canonical FITS tapes as defined in E.W. Greisen and R.H. Harten, "An Extension of FITS for Groups of Small Arrays of Data," Astronomy & Astrophysics Supplement Series 44:371 (June 1981). So far as I know, there is no program anywhere that can read this FITS format; this FITS program is probably useless. 2) FITS2 reads stashes and writes CLASSy FITS tapes. CLASS is a radio-astronomical data-reduction program produced in France by the Grenoble group, released to the public, and being used by Karl Menten and John Szczepanski (and others?) at CfA. CLASS now runs only on VAXes, but a version for Unix machines is being discussed. 3) FITS3 reads any of 11 spectral files (including #RTSLD, #BUSLD, #PRTDn) and writes CLASSy FITS tapes. FITS for single-dish data should be defined as a file format, not a tape format. We can reasonably expect that by the time the format is implemented, most observatories and most users will be transferring data over electronic links from one computer to another. A tape format might be just a bit-for-bit copy of the disk format or maybe an observatory-specific format for archiving. There is no particular reason why an observatory should record or archive in FITS format, although this should be possible. The point is to make FITS available to investigators over an electronic network and with luggable media. I imagine an investigator at his office anywhere calling up scans into his favorite data-reduction program and reviewing them on his screen within a few seconds after they were acquired at a remote telescope. Alternatively, an investigator at the telescope should be able to send data to his own computer, run a data- reduction program on them, view the results at the telescope, and be sure that the data are safe on his own disk before leaving the telescope. One can imagine arbitrarily complex data formats, and these may be useful for some specialized purposes. I urge, however, that a simplified version of the FITS format should be permissible within the specifications. Allow, for example, omission of parameters that can have default values. Allow omission of parameters that can be calculated from other parameters. Example: galactic latitude and longitude can be calculated from RA and Dec. Allow omission of parameters that are just unknown. Example: atmospheric attenuation. (Do you suppose that my program to calculate atmospheric attenuation from surface meteorology is better than yours?) The simplified format should contain raw data--all the raw data; other parameters can be calculated later. That is called data reduction. There is no particular reason why the reduced data should be kept in FITS format, although this should be possible. Insure that some program somewhere can read the new FITS format. How can those of us at observatories test our new FITS- writing programs if nobody can read their output? A published standard is useless and worthless if nobody can use it. Make your specifications so clear and so definite that those of us who have to read and use them, no matter how stupid or inattentive we are, cannot misunderstand your intentions. Make misinterpretation difficult, if not impossible.  From dishfits-request@fits.CX.NRAO.EDU Sat Oct 28 00:20:12 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA13853; Sat, 28 Oct 89 00:20:09 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA11666; Sat, 28 Oct 89 00:20:07 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA13849; Sat, 28 Oct 89 00:05:19 EDT Return-Path: X-Vms-Mail-To: DISHFITS AT NRAO Message-Id: <891028000222.000039BC031@cvax.CV.NRAO.EDU> Status: RO From: "GUILLOTE@FRGAG51" Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Too late may be ? Date: Sat, 28 Oct 89 00:02:22 EDT Date sent: 28-OCT-1989 04:54:46.17 To: DISHFITS@NRAO Dear FITS designers, As I won't be able to attend the meeting, I sum up here in a rather detail way my thoughts about the various propositions which came to me. I am sorry to be so late. Hope you can get it before leaving, otherwise Thierry will distribute it. Have a nice meeting ! Stephane Guilloteau Institut de Radio Astronomie Millimetrique 300 Rue de la Piscine F-38406 Saint Martin D'Heres Phone 76 82 49 43 E-Mail GUILLOTE@FRGAG51.BITNET ---------------------------------------------------------------------- 1) Reply to D.Wells proposal for 3-D tables 1.1) Comment on the general philosophy The 3-D tables idea is quite nice. It is a generalisation of the "random group" format that Thierry and I were proposing, and is essentially more flexible. This is both an advantage and a potential cause of problems. ------------------------------------------------------------------------ | The main difficulty with FITS has always be Reading FITS, not | | Writing. This was clearly realized by the original designers | | of FITS. It resulted in several compromises such as the | | fixed format and rigid alignments of keyword values. Also | | in the original papers, it was recommended that | | "transmitting institutions should write the tapes, whenever | | possible, in a way which will aid the receiving institution" | | (AA Sup p.367, middle of Section 7.) | | and | | "the user should always try to keep the requirements | | and restrictions of the reading institution in mind" | | (AA Sup p.372, end of section 4.) | ------------------------------------------------------------------------ With the 3-D tables, and with the idea of correspondence between Keywords and Columns ("virtual columns" and "virtual keywords"), the flexibility is even enhanced and these recommendations should be further enforced. One thing we MUST forbid is the use of pointers within the 3-D table. Pointers would break the correspondence mentionned above. And of course, the 3-D table should not be exclusive of more usual random groups. 1.2) Hierarchical keywords The correspondence principle is somewhat in conflict with the "HKBEGIN - HKEND" idea for hierarchical keywords. This is because the "virtual keywords" appear only as argument to keyword TTYPEnn keywords. Since the "HKBEGIN - HKEND" idea implies an ordering and (to keep it simple) a continuity also of hierarchical keywords. The simplest continuity (only) is broken for TTYPEnn arguments. Only a more subtle form of continuity is still valid. 1.3) Some misunderstandings In section 3.2, the meaning of "structure" is not clear. Does it mean "line" (by opposition to "columns") ? 1.4) Some absent issues As Thierry and I already discussed in one of our E-Mail, an important problem is that of non-regularly sampled frequency data. I suggest that the frequency (or wavelength) sampling of the various backends (if more than one) be handled in a separate 3-D table extension, placed BEFORE the data table(s). I do not believe resampling is the general solution for that (a related comment by Bob Wilson), although it should be used if possible. 1.5) Implementation constraints To keep it simple, I) the number of columns should remain relatively small, and II) the columns should concern only rather "standard" keywords. A possible way in this direction is to define two types of 3-D Tables: one for raw data and one for reduced data. For reduced data, it is possible to handle virtually any observing project in a single 3-D table, since only few parameters are actually required. For raw data, a 3-D table should contain only data for which time dependant parameters can be assumed constant (e.g. calibration, pointing corrections, etc...). Otherwise, those parameters would have to placed in columns, most likely violating either I) or II) (telescope dependent parameters). A third level might be considered : archive. In this case anything could go into the binary table, but probably only the writing program will be able to read back such complex data. 1.6) Reading constraints The FITS reading programs which will support 3-D Tables should also support the more usual "Random Groups", since it is in fact a (rigid) subset of the 3-D table capabilities. No extra code should even be necessary. 2) Preben Grosbol comment on Hierarchical keywords 2.1) I do not fully understand the point "It MUST be ensured that additions to FITS does not generate wrong results even for old readers" It seems that the blocking factor extension violate this rule? No? 2.2) Array keywords Continuation cards should be implemented using the last non-comment character. This is the only way to know a priori that the data is not complete, so that further reading is required. Simple string contatenation can then be used, with the final READ using a (now standard) list-directed format. All other schemes imply either to read keywords in advance, or to count the number of arguments. The full dimension of the array should be specified in the keyword. Sub-array notations (ARRAY(2..4) or ARRAY(2:4)) are ambiguous or require a distinct keyword to specify the array size. They should probably be prohibited. 2.3) Hanisch proposal revisited As mentionned in section 1.2, it is somewhat conflicting with the "correspondence" idea. A point to clear up... 3) Ray Norris proposal 3.1) Is the GCOUNT=-1 idea in conflict with the "It MUST be ensured that additions to FITS does not generate wrong results even for old readers" requirement ? 3.2) Turning back to our previous discussion on FITS for real-time logging, I still believe this is the true problem has not been considered. The key is Random Access Buffer Space (RABS) (either disk or memory), not only data throughput. If sufficient RABS is available, FITS file headers can be updated for the current size before writing to a sequential device. 4) Bob Wilson comments 4.1) multiple spectra : This is converging with the Ray Norris proposal. It also points out a possible difficulty in the 3-D tables. If many spectra of variable length are written in a single 3-D table, extracting just one may prove costly, since there is no way to predict the position of the spectrum in the table (other than reading part of the table). Please NEVER use pointers to circumvent that small problem: this would lead to lot of confusion (cf 1.1). 4.2) "Core" keywords for reduced data: I fully agree with this idea. It was behind the proposal 1.5 for two (or three) levels of 3-D tables. 4.3) "Scan" numbers : With the advent of multi-beam and multi-backends, this concept is obsolete. Single-dish data reduction should tend towards a descriptive approach of the data. The GAG-IRAM program CLASS is an (uncomplete) attempt in this direction. 5) Pat Wallace comments: 5.1) Pointing corrections Yes, these are really telescope specific parameters, and nothing else. They should not even be mentionned for reduced data (see 4.2 and 1.5). 5.2) Source coordinates: I have in my FITS papers a draft by Bob Hanisch and Don Wells, entitled "World Coordinate Systems Representations Within the FITS Format". It describes in a rather complete way conventions to be used for standard geometries, equinox, etc... What is the status of that draft ? I only have one very specific request. Drifts in horizontal coordinates are often used for pointing determination in radio astronomy. That king of data hardly conveys any useful astronomical content, but is quite important. Would it be possible to add a -HOR projection to the list of standard ones, implying that the coordinates are horizontal offsets from the tangent point at the time of observation? Or is that too specific? 6) Revised Stobie and Morgan memo Even though it seems that many people have diverged from this proposal, some comments might still be useful 6.1) p 22 ASCII versus Binary : Binary only please... 6.2) p 22 TABLE extension versatility for multi-beam or multi-backend The real issues are: - when will the processing systems be ready for multi-beam multi-backend? - how do they handle a multi-something: as a whole or not? - if not, where should be the separation ? - in real-time ? - in FITS writing ? - in FITS reading ? - in processing ? 6.3) p 22 Hierarchical approach : Lack of space may become a problem if HIERARCHY SINGLDSH ANALYSIS Program Structure Variable is required... Redundant keywords must be suppressed (e.g. SINGLDSH with ANALYSIS or ANALYSIS with Program). 6.4) p 22 Units Only SI units please. By the way, why are Degrees used in FITS for angles ? Radians are the only consistent choice with SI units... 6.5) p 22 Tape layout: This is against the basic FITS rule. Blocking factors are possible without specifying the layout. 6.6) P 22 Missing Keywords: That is right. Missing keywords should not be set to zero. They should be omitted when writing, and set to Not-A-Number when reading. 6.7) P 22 Keyword repetition: Not repeating the keyword imply to read and decode all previous extensions even when only one is wanted. It is a waste of time. The 3-D table extension more or less suppress this problem if the grouping of data is done properly. 6.8) p 23 Ill defined keywords: Those ones were redundant. Suppress them. See also BADCHV, redundant with the usual BLANK (p 27) 6.9) p 23 Pointing parameters: Observatory (or even telescope) dependent set. See 5.1 6.10) p 23 Observing parameters and scan numbers: See 4.3. Each observtory and each analysis program has its own definition of this number (not even consistent, see IRAM 30-m and CLASS!). It will be hard to adopt any convention. 6.11) p 23 DESORG and array keywords: We should play a leading role in array keywords. 6.12) p 24 Map parameters: Random groups or 3-D binary Tables do that even better, since no assumption is made on the map sampling. 6.13) p 24 PHASE table: See 6.11 on array keywords. 7) Three further comments 7.1) What will happen to the DD/MM/YY date format in January 2000 ? 7.2) How should we describe the various velocity reference systems and related topics in the axis definitions ? (Frequency, Wavelength, Velocity, Redshift, LSR, Heliocentric, Geocentric, Radio versus Optical definitions) 7.3) Since implementations of 3-D tables and hierarchical keywords will be a rather major issue in any case, should we break the "It MUST be ensured that additions to FITS does not generate wrong results even for old readers" rule by specifying a SIMPLE = F /No "standard" FITS FORMAT = 'HIERARCHY' /But correspond the 'HIERARCHY' revision instead and all further keywords and tape organisation using the full flexibility of Bob Hanish hierarchical proposal, array notations, full list-directed input format, 3-D tables, GCOUNT=-1 ? Wouldn't it be simpler (in terms of programming resources) for most people? From dishfits-request@fits.CX.NRAO.EDU Thu Oct 26 10:49:38 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA10643; Thu, 26 Oct 89 10:49:36 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA07135; Thu, 26 Oct 89 10:49:27 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA10628; Thu, 26 Oct 89 10:44:08 EDT Return-Path: X-Vms-Mail-To: DISHFITS Message-Id: <891026104042.00004B48081@cvax.CV.NRAO.EDU> Status: RO From: HLISZT@NRAO.EDU Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Travel plans for visitors to the NRAO Date: Thu, 26 Oct 89 10:40:42 EDT Dear People of DishFits; At the present time, I can confirm travel plans for the following visitors to the NRAO. If you are not on the list but should be, please contact H. Liszt ASAP via E-mail or by phone at 804-296-0344 (work) or 804-973-3744 (home). Both numbers have answering machines which pick up at 3 1/2 - 4 rings. W. Baan (Arecibo), R. Barvainis (Haystack), T. Forveille (Grenoble), P. Grosbol (ESO), R. Hanisch (StScI), D. Harris (CfA), P. Harrison (Jodrell) M. Olberg (Onsala), H. Matthews (JCMT), W. Peters (UA), R. Prestage (JCMT), J. Schmidt (MPI), B. Schlesinger (NASA), A. Sievers (MPI), B. Stobie (StScI), R. Warmels (ESO). Regards to all HLiszt@NRAO.EDU From dishfits-request@fits.CX.NRAO.EDU Sun Oct 29 15:59:50 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA17276; Sun, 29 Oct 89 15:59:49 EST Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA15955; Sun, 29 Oct 89 15:59:17 EST Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA17273; Sun, 29 Oct 89 15:52:15 EST Return-Path: >From: hoh-2!rww (R. W. Wilson) Message-Id: <8910292030.AA22425@hoh-2.> Status: RO From: rww@hoh-2.att.com Sender: dishfits-request@fits.CX.NRAO.EDU To: arpa!NRAO.edu!DISHFITS@hoh-2.att.com Date: Sun, 29 Oct 89 15:30:45 EST >From Bob Wilson, AT&T Bell laboratories To Dishfits community Subject: Answers to questions in July 26 letter. I will not be able to attend the conference, so let me reply to the questions. 1. What calibration/reduction/analysis software do you currently have? We use 'comb' (short for combine) for handling spectral line data from the telescope. It has all of the usual functions such as baseline removal, Gaussian fits to lines, etc. It has a well developed macro and do loop system and with a few lines of command, all of the data in a given area of the sky from raw data files can be put in a 'stacks' file. At that point, images can be made as either space-space maps of any reasonable parameter, or space-velocity maps. These are displayed as contour maps and can be written as FITS files. Images on a regular grid can be derived from irregularly sampled data. Data in the 'stacks' files are normally retrieved by position (and perhaps frequency) rather than stack numbers. We usually display the images from comb on our Sun workstations using a quick look or movie program. We use grey scale for single images or independent rgb colors when looking at three different spectral ranges. The latter works best with a 24-bit frame buffer on one of our Sun's, but can be simulated on the other 8-bit workstations using dithering. We have IRAF running and use it for most complicated image manipulating tasks since we haven't had a working version of AIPS since replacing our VAX with the Sun system. Comb is able to make scatter plots and 1-D cuts across an image. 2. How is your software effort organized in-house? Our telescope is on site and a daemon on the Sun file server transfers observations over a 9.6Kb/s line to data files on its disk as they are taken. The telescope operation is quite automatic and we often monitor progress from our office or home. We import data from other observatories (30-meter, SEST, Pom2, 12-meter, and Nobeyama come to mind). In each case a special purpose program transforms the data to one of comb's formats. I am the chief programmer of both OBS and comb. Marc Pound has done a lot with the display programs for the Sun's and John Bally has written many of the data translation programs. 3. How do you support your users? We don't have a large stream of visiting observers. We often run the observations for them as that is easier than explaining how to do it. We invite them to come and analyze the incoming data after a large enough image is built up. Comb is running at several other sites on VAX with BSD style Unix, Sun, and HP-UX. 4. What do you see as logical upgrade paths for: A. Hardware We work entirely in a UNIX environment (Suns right now) for data reduction and we are re-writing OBS for the Lynx operating system (a 'real time unix') using an AT buss 80386 machine. I think that comb could be ported to almost any Unix machine with little effort, but the data presentation programs are particular to Sunview. and may have to be re-written (for X-windows?) in the future when our hardware changes. B. Software We put multiple spectrometers in a single 'scan' in our raw data file, but treat each as a separate observation in the 'stacks' files. I haven't faced up to multiple feeds, since we don't have them yet. 5. How do you view the prospect of shared code for common use at multiple observatories? I think that the past practice of each observatory writing its own code completely is a silly waste of effort. Does anyone have an organized set of routines for pointing, etc.? I have been trying to talk the U.S.Naval Observatory out of some of their code. Peter Stumpf's 'BARVEL' of 10 years ago saved me a lot of effort. I would be happy to share any of our data reduction programs with anyone wanting to use them. When the new OBS is completed, we will be using it with AST/RO and I would be happy to share it. 6. Could single-dish work make better use of AIPS/IRAF/other packages not specifically targeted for it? Yes. We have been doing that for several years and it has been very effective. Robert W. Wilson rww@hoh-2.att.com 201-888-7120 From PTW%starlink.rutherford.ac.uk@NSFnet-Relay.AC.UK Mon Oct 30 09:59:17 1989 Return-Path: Received: from NSFnet-Relay.AC.UK ([128.86.8.6]) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA18477; Mon, 30 Oct 89 09:58:52 EST Message-Id: <8910301458.AA18477@fits.cx.nrao.edu> Received: from starlink.rutherford.ac.uk by NSFnet-Relay.AC.UK via Janet with NIFTP id aa07988; 30 Oct 89 14:17 GMT Status: RO From: "P.T.Wallace" To: DISHFITS-REQUEST <@NSFnet-Relay.AC.UK:DISHFITS-REQUEST@fits.cx.nrao.edu> Subject: reply to one of Bob Wilson's points Date: Mon, 30 Oct 89 14:19 GMT >From Pat Wallace, Starlink, UK. Bob Wilson asks whether there is any pointing software available. I can offer SLALIB and TPOINT. Both are written in ANSI Fortran 77, with a few well-defined and justified extensions. Both run on VAX, and SLALIB has been ported to other computers including SUN. SLALIB is a library of about 140 subroutines mainly concerned with positional astronomy. It supports both the old and new (pre- and post-IAU-1976) systems. TPOINT is an interactive telescope pointing analysis system. Further details of these and related products are available from me or from the Starlink Project. P.T.Wallace 19457::PTW Starlink PTW @ STARLINK.RUTHERFORD.AC.UK Rutherford Appleton Lab Chilton, Didcot, Oxon OX11 0QX, UK -------------------------------------------------------------------------