From dwells Thu Sep 3 14:10:04 1992 X-VM-VHeader: ("From:" "Sender:" "Resent-From" "To:" "Apparently-To:" "Cc:" "Subject:" "Date:" "Resent-Date:") nil X-VM-Bookmark: 5 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["893" "Thu" "3" "September" "92" "14:10:02" "EDT" "Don Wells" "dwells " nil "17" "fitsbits and anonFTP on fits.cv.nrao.edu" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA09591; Thu, 3 Sep 92 14:10:04 EDT Resent-Message-Id: <9209031810.AA09591 at fits.cv.nrao.edu> Return-Path: Message-Id: <9209031810.AA09585 at fits.cv.nrao.edu> Resent-Sender: dwells From: dwells (Don Wells) Sender: fitsbits-request at fits.CV.NRAO.EDU Resent-From: dwells (Don Wells) To: fitsbits Subject: fitsbits and anonFTP on fits.cv.nrao.edu Date: Thu, 3 Sep 92 14:10:02 EDT Resent-Date: Thu, 3 Sep 92 14:10:02 EDT On the 7th of July the Sun-3 named 'fits.cx.nrao.edu' (the workstation on my desk) was replaced by a SPARC (Sun-4) named 'fits.cv.nrao.edu'. Many facilities of my computing environment were disrupted by this change. In particular, during the past 8 weeks, (1) the 'fitsbits' exploder which is gatewayed to sci.astro.fits and (2) the anonymous FTP server of archival files of the FITS community have not been operational. I believe that 'fitsbits' and the anonFTP server are operational again. I expect to transmit copies of the July and August messages to the 'fitsbits' list within the next few days. Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From freedman at cuba.gsfc.nasa.gov (Immanuel Freedman) From: freedman at cuba.gsfc.nasa.gov (Immanuel Freedman) To: fitsbits at NRAO.EDU Cc: freedman at cuba.gsfc.nasa.gov Subject: BINTABLES or IMAGE extension ? --text follows this line-- >From freedman Thu Sep 10 12:40:15 1992 Date: Thu, 10 Sep 92 12:40:09 -0400 From: freedman (Immanuel Freedman) To: wgas at hypatia Subject: IMAGE or BINTABLES extensions ? Cc: ewing at aruba, freedman, kaltenbaugh at aruba, piper at aruba, turpie at aruba Status: RO Colleagues The COsmic Background Explorer project at NASA/Goddard Space Flight Center would like to distribute released data in FITS format from the National Space Science Data Center. Would our esteemed colleagues prefer to receive the data in BINTABLES or IMAGE format ? We have determined that IMAGE extensions are more effective than BINTABLES in terms of file size and data access time for our internal use. Thank-you for your replies in advance, Dr. Immanuel Freedman From bschlesinger at nssdca.gsfc.nasa.gov Fri Sep 11 13:08:19 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["6497" "Fri" "4" "September" "1992" "14:08:00" "GMT" "Barry Schlesinger" "bschlesinger at nssdca.gsfc.nasa.gov " nil "132" "FITS basics and information (regular posting)" "^From:" nil nil "9"]) Newsgroups: sci.astro.fits News-Software: VAX/VMS VNEWS 1.41 Nntp-Posting-Host: nssdca.gsfc.nasa.gov Organization: NASA - Goddard Space Flight Center From: bschlesinger at nssdca.gsfc.nasa.gov (Barry Schlesinger) Subject: FITS basics and information (regular posting) Date: Fri, 4 Sep 1992 14:08:00 GMT This basic FITS information is posted and updated periodically for the benefit of new readers and the reference of old readers. FITS (Flexible Image Transport System) is a data format designed to provide a means for convenient exchange of astronomical data between installations whose standard internal formats and hardware differ. A FITS file is composed of a sequence of Header Data Units (HDUs). The header consists of keyword=value statements, which describe the format and organization of the data in the HDU and may also provide additional information, for example, about the instrument or the history of the data or data set. The data follow, structured as the header specifies. The data section of the HDU may contain a digital image, but, except for the first, IT DOESN'T HAVE TO. Other possible formats include tables and multidimensional matrices that are not images. The first HDU must contain a multidimensional matrix or no data at all; the data in subsequent HDUs, called extensions, may be of any type, consistent with certain rules. The "Image" in the name comes from the original use of the format to transport digital images, but it's not just for images any more. FITS is not principally a graphics format designed for the transfer of pictures; it does not incorporate packages for decoding the data into an image. Users must develop or obtain separate software to convert the data from the FITS file into a form that can be readily displayed. As has been discussed in this newsgroup, and in alt.sci.astro.fits before it, the Extended Portable Bitmap Toolkit (pbm+) can be used for converting many FITS files to such a format. However, support is not guaranteed for all FITS files where the data are in the form of an image. In particular, there may be problems when the data matrix members are in IEEE floating point format (BITPIX<0) or the matrix has more than two dimensions (NAXIS>2). Discussion of FITS - image converters appears in this newsgroup from time to time. The fundamental references on FITS are the following four papers, often referred to collectively as the "Four FITS Papers". These papers are the formal standard for FITS, endorsed by the International Astronomical Union (IAU). Wells, D. C., Greisen, E. W., and Harten, R. H., "FITS: a flexible image transport system," Astronomy and Astrophysics Supplement Series, 44, 363-370, 1981. Greisen, E. W. and Harten, R. H., "An extension of FITS for small arrays of data," Astronomy and Astrophysics Supplement Series, 44, 371-374, 1981. (NOTE: The format described in this paper has been used almost exclusively to transport radio interferometry and is likely to be replaced by other formats in the future. Writing data other than radio interferometry data using this format is not recommended.) Grosbol, P., Harten, R. H., Greisen, E. W., and Wells, D. C., "Generalized extensions and blocking factors for FITS," Astronomy and Astrophysics Supplement Series, 73, 359-364, 1988. Harten, R. H., Grosbol. P., Greisen, E. W., and Wells, D. C., "The FITS tables extension, Astronomy and Astrophysics Supplement Series, 73, 365-372, 1988. A User's Guide for FITS, commissioned by NASA Headquarters, is maintained by the NASA/OSSA Office of Standards and Technology (NOST) FITS Support Office. This Guide is intended to be a tutorial for new FITS users. In addition to presenting the rules of FITS, it provides some of the history and reasoning behind the choice of the rules, adds recommendations on good practices, and discusses current developments in FITS. This document is available only in hard copy form. NASA is sponsoring development of a formal standard for FITS. The goal of this process is a document consistent with FITS as endorsed by the IAU, eliminating some contradictions and ambiguities in the original FITS papers, that can be endorsed by the IAU FITS Working Group as the FITS standard. The document is being developed by a Technical Panel chaired by Dr. Robert J. Hanisch (STSci), with review by the astronomical community. Only minor revisions are expected to the current draft, version 0.3b, but the form of the standard is not final, and it does not supersede the four papers and Floating Point Agreement endorsed by the IAU as the official standard for FITS. The IAU has endorsed the Floating Point Agreement, which defines how floating point numbers are to be stored in the data matrix defined in the first of the four papers. The basic agreement appears verbatim in the User's Guide, and the substance is incorporated in the Draft NASA Standard. The NOST maintains a file of FITS information available by anonymous ftp from nssdca.gsfc.nasa.gov or DECnet copy from NSSDCA, in the directory FITS. It includes copies of the current draft NASA Standard in flat ASCII, PostScript, and LaTeX. Style and index files are provided for the LaTeX form. A current list of the extension type (structure) names registered with the IAU FITS Working Group is maintained. Also available, in LaTeX form, is the text of the proposal for one of these new extension types, IMAGE. A README. file describes the contents of the directory. A SOFTWARE subdirectory, described by an included README.FIRST file, contains a program in C to read and list the headers of a FITS file and another file with information on publicly available FITS software packages. The ERRTEST subdirectory contains several versions of the same FITS file, a valid one and several with different kinds of header errors, for use in testing software to read FITS files. An included README.FIRST file contains details. Paper copies of many of the documents listed above can be obtained from the NOST Librarian. Paper copies of the User's Guide and either paper or electronic copies of the Draft NOST Standard, for those without ftp access, are available. Because of restrictions set by the copyright holder, NOST can send copies of the four FITS papers only to non-profit organizations. The NOST can be reached as follows: (Postal) NASA/OSSA Office of Standards and Technology Code 633 Goddard Space Flight Center Greenbelt MD 20771 USA (Internet) nost at nssdca.gsfc.nasa.gov (DECnet) NCF::NOST Telephone: (301)286-3575 8 a. m. - 5 p. m., U. S. Eastern Time Please mention this posting in your request. Barry M. Schlesinger Coordinator, NASA/NSSDC NOST FITS Support Office (301) 513-1634 fits at nssdca.gsfc.nasa.gov NCF::FITS From fitsbits-request Fri Sep 11 14:18:35 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["680" "" "11" "September" "92" "14:33:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " nil "14" "binary table sizes" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA02093; Fri, 11 Sep 92 14:18:35 EDT Return-Path: Message-Id: <11SEP199210330556 at stars.gsfc.nasa.gov> Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics Path: cv3.cv.nrao.edu!uvaarpa!darwin.sura.net!dtix!mimsy!nsisrv!stars.gsfc.nasa.gov!thompson From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: binary table sizes Date: 11 Sep 92 14:33:00 GMT I've been following the WGAS mail forum, and there's been some discussion lately about the size of FITS binary table files. Several people have remarked that binary tables take up more room than using the IMAGE extension. I can't understand why that would be. BINTABLE extensions use the same binary representation as every other kind of FITS data units, and there is no wasted space in packing the data together. In fact, I would have said that data put in a BINTABLE extension would take slightly less room than putting it in a series of IMAGE extensions. Is there something that I'm missing? Or is this another one of those FITS myths we need to stamp out? Bill Thompson From fitsbits-request Thu Sep 17 08:59:17 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["208" "Thu" "17" "September" "92" "08:59:17" nil "fitsbits-request" "fitsbits-request" nil "6" "" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA07132; Thu, 17 Sep 92 08:59:17 EDT Return-Path: Message-Id: <9209171259.AA13061 at cuba.gsfc.nasa.gov> From: freedman at cuba.gsfc.nasa.gov (Immanuel Freedman) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at NRAO.EDU Cc: freedman at cuba.gsfc.nasa.gov Subject: FITS formats Date: Thu, 17 Sep 92 08:59:41 -0400 From freedman Thu Sep 17 08:59:01 1992 Status: R From: freedman (Immanuel Freedman) To: wgas at hypatia Cc: freedman Subject: FITS formats Date: Thu, 17 Sep 92 08:59:00 -0400 Colleagues Thank-you for the many replies I received regarding the BINTABLES or IMAGE extension issues. It seems that the major concern should be the file size and performance and we at COBE hope to transfer the data from our internal Record Definition Language files to exactly equivalent FITS files, with format driven by the data themselves. It seems that the WGAS community is willing to accept FITS extensions and we may choose whatever is expedient. Issues of all-sky data exchange standards, the use of FITS for internal data structures (as opposed to file transfer) and data compression all seem to be hot topics. Dr. Immanuel Freedman (301) 513 7802 From fitsbits-request Thu Sep 17 15:28:14 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2632" "" "17" "September" "92" "19:09:07" "GMT" "Tom Kuchar" "kuchar at plh.af.mil " nil "63" "GreenBank CDROM" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA07296; Thu, 17 Sep 92 15:28:14 EDT Return-Path: Message-Id: <96217 at bu.edu> Organization: Phillips Laboratory, Hanscom AFB Path: cv3.cv.nrao.edu!uvaarpa!darwin.sura.net!zaphod.mps.ohio-state.edu!pacific.mps.ohio-state.edu!linac!att!bu.edu!buast7.bu.edu!kuchar From: kuchar at plh.af.mil (Tom Kuchar) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: GreenBank CDROM Date: 17 Sep 92 19:09:07 GMT I'm not certain this is the appropriate news group for this, but I suspect there are some who read this that may be able to help. I've tried reading the above mentioned CDROM on a sparcstation 2. However, when I try mounting it, I get an `invalid argument' error. The same is true with the IRAS sky survey CDs I have. Now, there's no problem reading these on a PC w/ CDROM capacity, but the workstation has a problem. The demo discs supplied by SUN have no problems mounting, just my data discs. The SUN manuals are fairly simple to follow, but give no diagnostic help for this. Does this have something to do with the CDROM format? If there's a more appropriate newsgroup to post this query, please let me know. As can be seen, I'm fairly unknowledgable about these matters. -- Tom Kuchar kuchar at buast7.bu.edu -or- kuchar at plh.af.mil Department of Astronomy Phillips Laboratory/GPOB Boston Univerity Hanscom AFB From fitsbits-request Thu Sep 17 18:17:52 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1088" "Thu" "17" "September" "1992" "22:17:29" "GMT" "Chris Flatters" "cflatter at cv3.CV.NRAO.EDU " nil "25" "Re: GreenBank CDROM" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA07347; Thu, 17 Sep 92 18:17:52 EDT Return-Path: Message-Id: <1992Sep17.221729.20357 at nrao.edu> Organization: NRAO Path: cv3.cv.nrao.edu!laphroaig!cflatter References: <96217 at bu.edu> Reply-To: cflatter at cv3.CV.NRAO.EDU From: cflatter at cv3.CV.NRAO.EDU (Chris Flatters) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: GreenBank CDROM Date: Thu, 17 Sep 1992 22:17:29 GMT In article 96217 at bu.edu, kuchar at plh.af.mil (Tom Kuchar) writes: >I've tried reading the above mentioned CDROM on a sparcstation 2. >However, when I try mounting it, I get an `invalid argument' error. >The same is true with the IRAS sky survey CDs I have. Now, there's no >problem reading these on a PC w/ CDROM capacity, but the workstation >has a problem. The demo discs supplied by SUN have no problems >mounting, just my data discs. The SUN manuals are fairly simple to >follow, but give no diagnostic help for this. The Green Bank CD uses the High Sierra format (this was designed for use with PCs running MS-Dog and is based on the MS-Dos file system). Sun CDs use the BSD 4.2 file system which Sun's mount assumes by default. You need to specify that the disk uses the High Sierra file system when you mount it. Assuming that you mount CD-ROM file systems at /cdrom you need to use the following command (as root). mount -t hsfs -r /dev/sr0 /cdrom Most CDs that do not carry UNIX specific software are likely to be in High Sierra format. Chris Flatters cflatter at nrao.edu From fitsbits-request Fri Sep 18 03:40:40 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["8291" "Fri" "18" "September" "92" "09:24:50" "ITA" "Lucio Chiappetti" "lucio at ifctr.mi.cnr.it" nil "194" "Re: GreenBank CDROM" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA07618; Fri, 18 Sep 92 03:40:40 EDT Resent-Message-Id: <9209180740.AA07618 at fits.cv.nrao.edu> Return-Path: < at vm.cnuce.cnr.it:EXOSAT at IMISIAM.BITNET> Message-Id: <9209180740.AA07612 at fits.cv.nrao.edu> Organization: Istituto di Fisica Cosmica e Tecnologie Relative In-Reply-To: Message of Thu, 17 Sep 1992 22:17:29 GMT from X-Acknowledge-To: Resent-Sender: Lucio Chiappetti From: Lucio Chiappetti Sender: fitsbits-request at fits.CV.NRAO.EDU Resent-From: fitsbits-request To: fitsbits at fits.CV.NRAO.EDU Subject: Re: GreenBank CDROM Date: Fri, 18 Sep 92 09:24:50 ITA Resent-Date: Fri, 18 Sep 92 09:24:50 ITA On Thu, 17 Sep 1992 22:17:29 GMT you said: >In article 96217 at bu.edu, kuchar at plh.af.mil (Tom Kuchar) writes: >>I've tried reading the above mentioned CDROM on a sparcstation 2. >>However, when I try mounting it, I get an `invalid argument' error. >> ..... > >The Green Bank CD uses the High Sierra format (this was designed for >use with PCs running MS-Dog and is based on the MS-Dos file system). --- Arf, arf | >Assuming that you mount CD-ROM file systems at /cdrom you need to use >the following command (as root). >mount -t hsfs -r /dev/sr0 /cdrom I do not have the CDROM referred by the previous correspondence, but I know of another possible problem with CDROMs on SUNs (this occurs for instance with the HST GSC CDROMs). Although one uses the mount statement as above (as far as I know that the only way to do it on Suns), one gets an "attribute" error and can see only directories but not files on tapes. There is a bit or byte somewhere in the definition of the format of the CD which allows or not to use filetypes longer than 3 chars. The Sun CD driver does not support this (which is used by the HST CDs) Sun calls it a "feature" (which means is not a "bug", but something they do not support and probably do not intend to support in the future) Instructions for a patch to the operating system kernel are available from STScI or from Sun (the one from STScI implies rebuiliding the kernel, the one from Sun patches two bytes in /vmunix directly and is simpler ... of course make a backup of the old kernel first |) Perhaps it would be appreciated if CDROM disributing institutions indicate on the CD whether one needs this patch or not. ----------------------------------------------------------------------------- A member of G.ASS : Group for Astronomical Software Support ----------------------------------------------------------------------------- Lucio Chiappetti - IFCTR/CNR Milano | U N I C U I Q U E via Bassini 15 - I-20133 Milano - Italy | System Manager Internet: LUCIO at IFCTR.MI.CNR.IT | FORTUNAE SUAE Decnet: IFCTR::LUCIO (39610::LUCIO) | Bitnet: EXOSAT at IMISIAM | ----------------------------------------------------------------------------- PostScriptum: It would be helpful for those like me which receive fitsbits by e-mail, if the list identifies itself as Sender: fitsbits instead of Sender: fitbsits-requests. I recall making a REPLY to a previous message and getting bounced (or am I mixing up things ?) I can do REPLY (goes to Reply-To if present or From), REPLY FROM (goes to From) and REPLY SENDER (goes to Sender) but to reply to fitsbits (which appears in the To field) I have to do REPLY ALL and then EXCLUDE all the other addressees. From fitsbits-request Fri Sep 18 20:57:45 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["896" "Sat" "19" "September" "1992" "00:19:27" "GMT" "Jim Wright" "jwright at cfht.hawaii.edu " nil "26" "logical variables" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA08448; Fri, 18 Sep 92 20:57:45 EDT Return-Path: Message-Id: Organization: University of Hawaii Path: cv3.cv.nrao.edu!uvaarpa!caen!uunet!sun-barr!ames!news.hawaii.edu!jwright From: jwright at cfht.hawaii.edu (Jim Wright) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: logical variables Date: Sat, 19 Sep 1992 00:19:27 GMT An excerpt from the NOST 100-0.2f FITS Draft Standard, dated May 14, 1991: | 5.3.2.2 Logical Variable | | If the value is a logical constant, it shall appear as a T or F in | column 30. (This is the entire section !) I would like to initialize all cards to values that show the card has not yet been filled out, and then put in the real values. For example, LAMPON = ? / lamp on during exposure And then at exposure time fill out the card with "T" or "F". This way it is very obvious if the card contains valid data or not. Is this acceptable? Yes, not putting the card in until the time it is filled out is best. But that is not an option for us. The entire FITS header is built before an exposure and individual cards are given their values during the course of the exposure. -- Jim Wright jwright at cfht.hawaii.edu Canada-France-Hawaii Telescope Corp. From fitsbits-request Sat Sep 19 01:26:43 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2375" "Sat" "19" "September" "1992" "04:29:52" "GMT" "Lee E. Brotzman" "leb at Hypatia.gsfc.nasa.gov " nil "50" "Re: logical variables" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA08470; Sat, 19 Sep 92 01:26:43 EDT Return-Path: Message-Id: Organization: Goddard Space Flight Center Path: cv3.cv.nrao.edu!uvaarpa!darwin.sura.net!spool.mu.edu!uunet!sun-barr!ames!nsisrv!Hypatia!leb References: From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: logical variables Date: Sat, 19 Sep 1992 04:29:52 GMT jwright at cfht.hawaii.edu (Jim Wright) writes: >An excerpt from the NOST 100-0.2f FITS Draft Standard, dated May 14, 1991: >| 5.3.2.2 Logical Variable >| >| If the value is a logical constant, it shall appear as a T or F in >| column 30. > >(This is the entire section !) ^^^^^^^ Actually its a sub-sub-sub-section and this is a standard, not human-readable text. :-) >I would like to initialize all cards to values that show the card has not >yet been filled out, and then put in the real values. For example, > >LAMPON = ? / lamp on during exposure > >And then at exposure time fill out the card with "T" or "F". This way it >is very obvious if the card contains valid data or not. Is this acceptable? The 'T' in FITS is for Transport. Until you try to transport the FITS file, anything goes. Your internal processing of the FITS file, and especially intermediate results, are not important. Do whatever is the most efficient and reasonable for your application. If you try to transport the FITS header with question marks instead of valid boolean 'T' or 'F', however, FITS readers the world over will throw up all over your file. I'm working on a program that extracts records from one FITS table and puts them into another FITS file. Obviously, I don't know the number of records I'm going to extract until I am done, so I put a 0 in the NAXIS2 record in the extension header. Then when I have finished the extraction I reposition the file pointer to the appropriate record position and fill in the proper number of records that were extracted. The intermediate file is in violation of the standard, and if the program aborted for some reason before it filled in the NAXIS2 record, and I tried to palm off the file as correct, I would be violating the standard. The bottom line is, use templates for filling out FITS headers, make sure you complete the template, and perhaps include a post-processing step that will strip out -- with appropriate warning messages -- unfilled records. >-- >Jim Wright >jwright at cfht.hawaii.edu >Canada-France-Hawaii Telescope Corp. -- -- Lee E. Brotzman Internet: leb at hypatia.gsfc.nasa.gov -- Hughes STX DECNET: NDADSA::BROTZMAN -- National Space Science Data Center BITNET: ZMLEB at GIBBS -- NASA Goddard Space Flight Center From freedman at cuba.gsfc.nasa.gov Sat Sep 19 12:53:39 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["843" "Thu" "17" "September" "1992" "12:59:41" "GMT" "Immanuel Freedman" "freedman at cuba.gsfc.nasa.gov " nil "28" "FITS formats" "^From:" nil nil "9"]) Newsgroups: sci.astro.fits Organization: National Radio Astronomy Observatory From: freedman at cuba.gsfc.nasa.gov (Immanuel Freedman) Subject: FITS formats Date: Thu, 17 Sep 1992 12:59:41 GMT >From freedman Thu Sep 17 08:59:01 1992 Date: Thu, 17 Sep 92 08:59:00 -0400 From: freedman (Immanuel Freedman) To: wgas at hypatia Subject: FITS formats Cc: freedman Status: R Colleagues Thank-you for the many replies I received regarding the BINTABLES or IMAGE extension issues. It seems that the major concern should be the file size and performance and we at COBE hope to transfer the data from our internal Record Definition Language files to exactly equivalent FITS files, with format driven by the data themselves. It seems that the WGAS community is willing to accept FITS extensions and we may choose whatever is expedient. Issues of all-sky data exchange standards, the use of FITS for internal data structures (as opposed to file transfer) and data compression all seem to be hot topics. Dr. Immanuel Freedman (301) 513 7802 From kuchar at plh.af.mil Sat Sep 19 12:54:26 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1017" "" "17" "September" "92" "19:09:07" "GMT" "Tom Kuchar" "kuchar at plh.af.mil " nil "23" "GreenBank CDROM" "^From:" nil nil "9"]) Newsgroups: sci.astro.fits Distribution: usa Organization: Phillips Laboratory, Hanscom AFB Originator: kuchar at buast7.bu.edu From: kuchar at plh.af.mil (Tom Kuchar) Subject: GreenBank CDROM Date: 17 Sep 92 19:09:07 GMT I'm not certain this is the appropriate news group for this, but I suspect there are some who read this that may be able to help. I've tried reading the above mentioned CDROM on a sparcstation 2. However, when I try mounting it, I get an `invalid argument' error. The same is true with the IRAS sky survey CDs I have. Now, there's no problem reading these on a PC w/ CDROM capacity, but the workstation has a problem. The demo discs supplied by SUN have no problems mounting, just my data discs. The SUN manuals are fairly simple to follow, but give no diagnostic help for this. Does this have something to do with the CDROM format? If there's a more appropriate newsgroup to post this query, please let me know. As can be seen, I'm fairly unknowledgable about these matters. -- Tom Kuchar kuchar at buast7.bu.edu -or- kuchar at plh.af.mil Department of Astronomy Phillips Laboratory/GPOB Boston Univerity Hanscom AFB From cflatter at nrao.edu Sat Sep 19 12:55:10 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1089" "Thu" "17" "September" "1992" "22:17:29" "GMT" "Chris Flatters" "cflatter at nrao.edu " nil "26" "Re: GreenBank CDROM" "^From:" nil nil "9"]) Newsgroups: sci.astro.fits Reply-To: cflatter at nrao.edu Organization: NRAO From: cflatter at nrao.edu (Chris Flatters) Subject: Re: GreenBank CDROM Date: Thu, 17 Sep 1992 22:17:29 GMT In article 96217 at bu.edu, kuchar at plh.af.mil (Tom Kuchar) writes: >I've tried reading the above mentioned CDROM on a sparcstation 2. >However, when I try mounting it, I get an `invalid argument' error. >The same is true with the IRAS sky survey CDs I have. Now, there's no >problem reading these on a PC w/ CDROM capacity, but the workstation >has a problem. The demo discs supplied by SUN have no problems >mounting, just my data discs. The SUN manuals are fairly simple to >follow, but give no diagnostic help for this. The Green Bank CD uses the High Sierra format (this was designed for use with PCs running MS-Dog and is based on the MS-Dos file system). Sun CDs use the BSD 4.2 file system which Sun's mount assumes by default. You need to specify that the disk uses the High Sierra file system when you mount it. Assuming that you mount CD-ROM file systems at /cdrom you need to use the following command (as root). mount -t hsfs -r /dev/sr0 /cdrom Most CDs that do not carry UNIX specific software are likely to be in High Sierra format. Chris Flatters cflatter at nrao.edu From jwright at cfht.hawaii.edu Sat Sep 19 12:58:45 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["896" "Sat" "19" "September" "1992" "00:19:27" "GMT" "Jim Wright" "jwright at cfht.hawaii.edu " nil "26" "logical variables" "^From:" nil nil "9"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: mercury.cfht.hawaii.edu Organization: University of Hawaii From: jwright at cfht.hawaii.edu (Jim Wright) Cc: fits at cfht.hawaii.edu, fits at dao.nrc.ca Subject: logical variables Date: Sat, 19 Sep 1992 00:19:27 GMT An excerpt from the NOST 100-0.2f FITS Draft Standard, dated May 14, 1991: | 5.3.2.2 Logical Variable | | If the value is a logical constant, it shall appear as a T or F in | column 30. (This is the entire section !) I would like to initialize all cards to values that show the card has not yet been filled out, and then put in the real values. For example, LAMPON = ? / lamp on during exposure And then at exposure time fill out the card with "T" or "F". This way it is very obvious if the card contains valid data or not. Is this acceptable? Yes, not putting the card in until the time it is filled out is best. But that is not an option for us. The entire FITS header is built before an exposure and individual cards are given their values during the course of the exposure. -- Jim Wright jwright at cfht.hawaii.edu Canada-France-Hawaii Telescope Corp. From leb at Hypatia.gsfc.nasa.gov Sat Sep 19 13:00:48 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2375" "Sat" "19" "September" "1992" "04:29:52" "GMT" "Lee E. Brotzman" "leb at Hypatia.gsfc.nasa.gov " nil "50" "Re: logical variables" "^From:" nil nil "9"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: hypatia.gsfc.nasa.gov Organization: Goddard Space Flight Center From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) Subject: Re: logical variables Date: Sat, 19 Sep 1992 04:29:52 GMT jwright at cfht.hawaii.edu (Jim Wright) writes: >An excerpt from the NOST 100-0.2f FITS Draft Standard, dated May 14, 1991: >| 5.3.2.2 Logical Variable >| >| If the value is a logical constant, it shall appear as a T or F in >| column 30. > >(This is the entire section !) ^^^^^^^ Actually its a sub-sub-sub-section and this is a standard, not human-readable text. :-) >I would like to initialize all cards to values that show the card has not >yet been filled out, and then put in the real values. For example, > >LAMPON = ? / lamp on during exposure > >And then at exposure time fill out the card with "T" or "F". This way it >is very obvious if the card contains valid data or not. Is this acceptable? The 'T' in FITS is for Transport. Until you try to transport the FITS file, anything goes. Your internal processing of the FITS file, and especially intermediate results, are not important. Do whatever is the most efficient and reasonable for your application. If you try to transport the FITS header with question marks instead of valid boolean 'T' or 'F', however, FITS readers the world over will throw up all over your file. I'm working on a program that extracts records from one FITS table and puts them into another FITS file. Obviously, I don't know the number of records I'm going to extract until I am done, so I put a 0 in the NAXIS2 record in the extension header. Then when I have finished the extraction I reposition the file pointer to the appropriate record position and fill in the proper number of records that were extracted. The intermediate file is in violation of the standard, and if the program aborted for some reason before it filled in the NAXIS2 record, and I tried to palm off the file as correct, I would be violating the standard. The bottom line is, use templates for filling out FITS headers, make sure you complete the template, and perhaps include a post-processing step that will strip out -- with appropriate warning messages -- unfilled records. >-- >Jim Wright >jwright at cfht.hawaii.edu >Canada-France-Hawaii Telescope Corp. -- -- Lee E. Brotzman Internet: leb at hypatia.gsfc.nasa.gov -- Hughes STX DECNET: NDADSA::BROTZMAN -- National Space Science Data Center BITNET: ZMLEB at GIBBS -- NASA Goddard Space Flight Center From fitsbits-request Sun Sep 20 08:02:14 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["8760" "Sun" "20" "September" "92" "13:44:54" "ITA" "Lucio Chiappetti" "lucio at ifctr.mi.cnr.it" nil "178" "Re: logical variables" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA09569; Sun, 20 Sep 92 08:02:14 EDT Resent-Message-Id: <9209201202.AA09569 at fits.cv.nrao.edu> Return-Path: < at vm.cnuce.cnr.it:EXOSAT at IMISIAM.BITNET> Message-Id: <9209201201.AA09534 at fits.cv.nrao.edu> Organization: Istituto di Fisica Cosmica e Tecnologie Relative In-Reply-To: Message of Sat, 19 Sep 1992 04:29:52 GMT from X-Acknowledge-To: Resent-Sender: Lucio Chiappetti From: Lucio Chiappetti Sender: fitsbits-request at fits.CV.NRAO.EDU Resent-From: fitsbits-request To: fitsbits at fits.CV.NRAO.EDU Subject: Re: logical variables Date: Sun, 20 Sep 92 13:44:54 ITA Resent-Date: Sun, 20 Sep 92 13:44:54 ITA On Sat, 19 Sep 1992 04:29:52 GMT said: >jwright at cfht.hawaii.edu (Jim Wright) writes: >>An excerpt from the NOST 100-0.2f FITS Draft Standard, dated May 14, 1991: >>| 5.3.2.2 Logical Variable >>| >>| If the value is a logical constant, it shall appear as a T or F in >>| column 30. >> >>I would like to initialize all cards to values that show the card has not >>yet been filled out, and then put in the real values. For example, >> >>LAMPON = ? / lamp on during exposure >> >>And then at exposure time fill out the card with "T" or "F". This way it >>is very obvious if the card contains valid data or not. Is this acceptable? > >The 'T' in FITS is for Transport. Until you try to transport the FITS file, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >anything goes. Your internal processing of the FITS file, and especially The example above indicates one specific case of a limitation with the use of FITS files other than for transport, indeed | This shows up in a more extreme case if not only the values, but the number of keywords in a file header is unknown at the moment of the creation of the file | The point is that the FITS header is a true header (i.e. comes in front of the data), and once written it cannot be extended (one could advocate reserving empty space in the header, as e.g. Bill Pence's FITSIO allows, but even the amount to be reserved may be not known a priori) There is also another case, assume I want to modify a file in place (say I want to multiply all values in a column of a BINTABLE by an amount, e.g. converting angular coordinates to microns in the focal plane multi- plying by a platescale) AND I want to record the change ADDING more keywords to the header. If my work file is FITS I cannot do it, unless I write an entirely new file, or I shift all data records (which is equivalent to rewriting all). The only way out to have an extendable and modifiable header is to have it stored either at the end of the file (the solution I prefer), or elsewhere (like IRAF .imh and .pix files). Far from diminishing the enormous value of FITS as a TRANSPORT standard, this is however a very important reason to support the traditional view (e.g. IRAF's and MIDAS's) that work files are read from FITS into a native format for manipulation and written out to FITS for export. ----------------------------------------------------------------------------- A member of G.ASS : Group for Astronomical Software Support ----------------------------------------------------------------------------- Lucio Chiappetti - IFCTR/CNR Milano | U N I C U I Q U E via Bassini 15 - I-20133 Milano - Italy | System Manager Internet: LUCIO at IFCTR.MI.CNR.IT | FORTUNAE SUAE Decnet: IFCTR::LUCIO (39610::LUCIO) | Bitnet: EXOSAT at IMISIAM | ----------------------------------------------------------------------------- From fitsbits-request Sun Sep 20 14:19:27 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1078" "" "20" "September" "1992" "10:36" "PST" "Tim Pearson" "tjp at eccles.caltech.edu " nil "30" "Re: logical variables" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA10100; Sun, 20 Sep 92 14:19:27 EDT Return-Path: Message-Id: <20SEP199210364112 at eccles.caltech.edu> Organization: California Institute of Technology Path: cv3.cv.nrao.edu!uvaarpa!concert!gatech!swrinde!cs.utexas.edu!sun-barr!ames!data.nas.nasa.gov!mustang.mst6.lanl.gov!nntp-server.caltech.edu!eccles.caltech.edu!tjp References: From: tjp at eccles.caltech.edu (Tim Pearson) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: logical variables Date: 20 Sep 1992 10:36 PST >LAMPON = ? / lamp on during exposure If a FITS keyword has a logical value, it must be either T or F. However, there is nothing in the standard that says that the keyword LAMPON must have a logical value. If you don't know whether the lamp is on or off, you could put LAMPON = 'unknown' / lamp on during exposure This would be legal FITS and should be accepted by your FITS reader. Your FITS reader might have trouble mapping a string value into a logical variable in your analysis program, but that is not a FITS problem. Personally, I prefer to avoid logical values. To record status like this, I would use LAMP = 'ON' LAMP = 'OFF' LAMP = 'UNKNOWN' for example. This allows other unthought-of possibilities to be added later, e.g., LAMP = 'GREEN' Tim Pearson, Astronomy Dept 105-24, Caltech, Pasadena, California 91125, USA Internet: TJP at Deimos.Caltech.Edu, Pearson_T at caltech.edu Telephone: +1 818 356-4980 FAX: +1 818 568-9352 From fitsbits-request Mon Sep 21 17:58:52 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3414" "Mon" "21" "September" "1992" "20:55:40" "GMT" "William Pence" "pence at heawk1.gsfc.nasa.gov " nil "63" "Re: logical variables" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA01598; Mon, 21 Sep 92 17:58:52 EDT Return-Path: Message-Id: Organization: Goddard Space Flight Center Path: cv3.cv.nrao.edu!uvaarpa!darwin.sura.net!wupost!cs.utexas.edu!sun-barr!ames!nsisrv!heawk1!pence References: <9209201201.AA09534 at fits.cv.nrao.edu> From: pence at heawk1.gsfc.nasa.gov (William Pence) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: logical variables Date: Mon, 21 Sep 1992 20:55:40 GMT Lucio Chiappetti previously wrote: >>jwright at cfht.hawaii.edu (Jim Wright) writes: >>>I would like to initialize all cards to values that show the card has not >>>yet been filled out, and then put in the real values. For example, >>> >>>LAMPON = ? / lamp on during exposure >>> >>>And then at exposure time fill out the card with "T" or "F". This way it >>>is very obvious if the card contains valid data or not. Is this acceptable? >> >>The 'T' in FITS is for Transport. Until you try to transport the FITS file, > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>anything goes. Your internal processing of the FITS file, and especially > > The example above indicates one specific case of a limitation with the > use of FITS files other than for transport, indeed > This shows up in a more extreme case if not only the values, but the > number of keywords in a file header is unknown at the moment of the > creation of the file > The point is that the FITS header is a true header (i.e. comes in front > of the data), and once written it cannot be extended (one could advocate > reserving empty space in the header, as e.g. Bill Pence's FITSIO allows, > but even the amount to be reserved may be not known a priori) (lines editted out) > Far from diminishing the enormous value of FITS as a TRANSPORT standard, > this is however a very important reason to support the traditional view > (e.g. IRAF's and MIDAS's) that work files are read from FITS into a > native format for manipulation and written out to FITS for export. One of the major recent enhancements to FITSIO, which was first introduced in version 3.20 (released in March 1992) is the ability to dynamically increase the number of keywords in the header of a previously written FITS file. Thus, users/programmers using FITSIO do not have to worry whether there is enough space in the header before writting new keywords. FITSIO will keep track of how much space is available (users may preallocate extra space when the file is created). If there is no more room, then FITSIO will shift all the following data records, including any additional extensions in the FITS file, and append a new empty 2880 byte block onto the header. This all takes place transparently to the user/programmer. This shifting of data obviously requires some overhead, but our experience in using FITS files as an internal data processing format indicates that this does not cause a significant impact. Here at the HEASARC we have spent the last year developing a comprehensive package of programs called FTOOLS for manipulating FITS format files. (The first public release of the package is planned for the end of next month). We have accumlated a lot of experience processing FITS files which shows, contrary to the conservative view that FITS is only a transport format, that FITS is an excellent analysis format in its own right which has many advantages. The self-documenting nature of the headers, the flexibility and storage efficiency of the new binary table extension format, and most importantly, the ability to simply copy any FITS files to any type of machine without having to worry about the internal machine data formats, makes FITS an ideal format for processing data in today's hetrogeneous computer processing environments. -Bill Pence pence at tetra.gsfc.nasa.gov HEASRC::PENCE From fitsbits-request Thu Sep 24 09:44:07 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["10356" "Thu" "24" "September" "92" "14:43" "GMT" "\"Clive Page, Leicester Univ, UK\"" "CGP at STARLINK.LEICESTER.AC.UK" nil "194" "Some FITS tables suggestions" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA01577; Thu, 24 Sep 92 09:44:07 EDT Return-Path: Message-Id: <9209241343.AA19253 at cv3.cv.nrao.edu> Via: uk.ac.leicester.starlink; Thu, 24 Sep 1992 14:43:09 +0100 From: "Clive Page, Leicester Univ, UK" Sender: fitsbits-request at fits.CV.NRAO.EDU To: FITSBITS Subject: Some FITS tables suggestions Date: Thu, 24 Sep 92 14:43 GMT Astronomers are using FITS tables, especially binary tables, for a wide range of applications from star catalogues to photon-event lists. As William Pence has already noted, FITS is starting to be adopted as a storage format in its own right and not just as a data interchange format. Packages that provide database functionality and can handle FITS tables already exist or are under development at several places, for example: o At GSFC the EXOSAT database system is being converted to run on Unix and use FITS tables. o At ESO the MIDAS table system is being extended with more relational database functionality, and it can read/write FITS tables. o At GSFC William Pence is developing a set of tools to work on FITS tables (sorting, selecting, etc.). o In Italy the latest version of the Astronet DIRA2 database system can read and write FITS tables. o In the UK, the Starlink Catalogue Access and Reporting (SCAR) package is highly VMS-dependent, so a little work has been done here on a portable replacement. This uses the FITS binary table as the main file format. Tests of a prototype seem quite promising. Thus FITS tables are starting to acquire relational database software packages built around them. The FITS table structure seems quite well designed for this purpose, especially as it allows one physical file to hold not only the tabular data but also column descriptors and table descriptors (epoch, equinox, etc). This makes for a nice modular system and avoids all the complications which bedevil most commercial DBMS packages, which use something like a "common data dictionary" or "system catalog" to hold all their descriptors, and therefore need a database administrator with special privileges to take charge of the whole thing. This way of working does not suit most groups of astronomers, who find it much better if their tables are entirely self-contained. The software packages noted above fulfil different sets of requirements, and data interchange between them is extremely likely. But there is a danger of minor but annoying incompatibilities creeping in unless we can agree fairly soon on a few more details of the FITS table format, or at least on some conventions for the use of FITS keywords. (1) Form of column names. The FITS Standard (section 8.1.2) suggests that names should only contain letters, digits, and underscores. I would like to propose that the first character should not be a digit, since leading digits make it hard to parse command-lines or expressions which may contain column names as well as numerical constants. (2) Column name length. There seems no length limit on a column name (except that FITS keyword strings cannot be more than 68 chars long). But section 5.3.2.1 says that "proper interpretation of the FITS file should not require decoding any more than the first eight characters of a character string". Does this limit column names to 8-chars or perhaps just require them to be unique in the first 8 characters? I note that DIRA2 does indeed have an 8-character limit, but other systems listed above have more relaxed limits, typically allowing names at least 16 characters long. (3) Sorting. Efficient access to large tables implies keyed access of some type. In practice most large FITS tables are sorted on some field, allowing a binary search for a value or range of values. For example most of the tables on the NSSDC/ADC CD-ROMs seem to be sorted on ascending RA, but there is no way of determining this for certain without reading the entire file. As some systems (such as SCAR) prefer sorting on DEC (to avoid the wrap-around problem at 24 hours/0 hours), we cannot assume sorting on any particular field even for an astronomical catalog (and some contain celestial coordinates using two or more different equinoxes, any one of which may be chosen as the principal sort field). I think we need a convention for specifying the sort field and sense in a FITS header. The simplest way would just be to name the field (or fields) and insert a minus sign for a descending sort, e.g. TSORTnnn = "-DEC" or TSORTnnn = "SP_TYPE,-VMAG" if there were minor sort fields in addition to the principal one. (4) Displaying Sexagesimal Notation. Most astronomers still use sexagesimal notation (e.g. 12:34:56, -2:34:56.7) for celestial coordinates. In fact sexagesimals already appear in many FITS tables but have to be treated either as a set of three separate columns for hours/degrees, minutes, and seconds (and sometimes a fourth one for the sign), or as a character string which may (or may not) have spaces or colons between the components. Neither of these methods is very easy to use when one has to decode the numerical value for a calculation (or for keyed access). I wonder if anyone has considered an extension to allow the TFORM keyword to specify sexagesimal formats? The obvious way of specifying this is with pseudo-fortran forms like "HMSw.d" or "DMSw.d" where the ".d" is needed only if the seconds value has a (decimal) fractional part. Obviously these formats would have to be handled by special code, but in programs written in Fortran-77 (the great majority, I would guess) this is already necessary when handling format descriptors such as "Bw.d" or "ENw.d" which are not built-in to the language. The value in the file could be assumed to be a decimal value in hours/degrees (so there would be no change to the integer part only the fractional part would be converted to minutes and seconds (or arcmins and arcsecs). Systems that could not handle the conversion could (as is permitted at present for ESw.d and ENw.d formats) just display the degrees/hours as a normal decimal number with the specified width and some sensible precision. As an enhancement, however, one could allow the TUNITnnn keyword to indicate different units (such as radians) and have an implicit conversion to hours (HMS) or degrees (DMS). But then we get into slightly tricky areas of automatic units conversion. (5) Indexing. The prototype that I am building can create an index on any column; it uses B+ trees to make indexing dynamic. MIDAS and perhaps other systems will soon include indexing. Although it would be possible to pack indexes into other data units in the same FITS file, it seems better for extensibility for them to be kept in separate files. Software that can handle indexes really needs to know which columns have been indexed when the table is opened, so that any update operations are also reflected in the indexes. Thus some keyword is needed to indicate the presence an index and perhaps its location. These can, of course, be ignored by any package that is not able to make use of indexes. We could use some standard naming scheme in which case just existence needs to be shown, for example: TINDXnnn = T Otherwise it would be sensible to indicate the name of the external file, for example: TINDXnnn = "filename" Clive Page Dept of Physics & Astronomy, University of Leicester, UK. cgp at star.le.ac.uk (or SPAN 19838::cgp). From fitsbits-request Fri Sep 25 11:21:52 1992 Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA04553; Fri, 25 Sep 92 11:21:52 EDT Return-Path: To: fitsbits at fits.CV.NRAO.EDU Date: Fri, 25 Sep 1992 15:10:00 GMT From: bschlesinger at nssdca.gsfc.nasa.gov (Barry Schlesinger) Message-Id: <25SEP199210103323 at nssdca.gsfc.nasa.gov> Organization: NASA - Goddard Space Flight Center Path: cv3.cv.nrao.edu!uvaarpa!darwin.sura.net!spool.mu.edu!agate!ames!nsisrv!nssdca.gsfc.nasa.gov!bschlesinger References: <9209241343.AA19253 at cv3.cv.nrao.edu> Subject: Re: Some FITS tables suggestions Sender: fitsbits-request at fits.CV.NRAO.EDU In article <9209241343.AA19253 at cv3.cv.nrao.edu>, "Clive Page, Leicester Univ, UK" writes... > ... >(2) Column name length. >There seems no length limit on a column name (except that FITS keyword >strings cannot be more than 68 chars long). But section 5.3.2.1 says >that "proper interpretation of the FITS file should not require decoding >any more than the first eight characters of a character string". Does >this limit column names to 8-chars or perhaps just require them to be >unique in the first 8 characters? All the information in character strings to read the accompanying data; i. e. transfer it from the FITS file to the user's system in proper form, must be in the first eight characters. Column names may be of more than eight characters. Information required only for the scientific interpretation of the data, not for reading it, may be after column 8. I recommend that the first eight characters be unique and selected to contain the most important information about the column name, but the practice is not required. In many cases, a longer name may be needed to communicate the meaning of the column heading. >(3) Sorting. >Efficient access to large tables implies keyed access of some type. >In practice most large FITS tables are sorted on some field, allowing a >binary search for a value or range of values. For example most of the >tables on the NSSDC/ADC CD-ROMs seem to be sorted on ascending RA, but >there is no way of determining this for certain without reading the >entire file. As some systems (such as SCAR) prefer sorting on DEC (to >avoid the wrap-around problem at 24 hours/0 hours), we cannot assume >sorting on any particular field even for an astronomical catalog (and >some contain celestial coordinates using two or more different >equinoxes, any one of which may be chosen as the principal sort field). >I think we need a convention for specifying the sort field and sense in a >FITS header. The simplest way would just be to name the field (or >fields) and insert a minus sign for a descending sort, e.g. >TSORTnnn = "-DEC" >or >TSORTnnn = "SP_TYPE,-VMAG" >if there were minor sort fields in addition to the principal one. > Except, as earlier discussions in this forum have noted, there are occasionally problems with the transport of hyphens (minus signs). That is why letters, numbers, and underscore are the only recommended characters for the value of TFORMn, and it might be advisable to extend the practice to characters of other keywords. > ... Barry Schlesinger NOST FITS Support Office From fitsbits-request Sun Sep 27 11:08:31 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["20141" "Sun" "27" "September" "92" "16:10:59" "ITA" "Lucio Chiappetti" "lucio at ifctr.mi.cnr.it" nil "423" "Re: Some FITS tables suggestions" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA07040; Sun, 27 Sep 92 11:08:31 EDT Resent-Message-Id: <9209271508.AA07040 at fits.cv.nrao.edu> Return-Path: < at vm.cnuce.cnr.it:EXOSAT at IMISIAM.BITNET> Message-Id: <9209271508.AA07034 at fits.cv.nrao.edu> Organization: Istituto di Fisica Cosmica e Tecnologie Relative In-Reply-To: Message of Thu, 24 Sep 92 14:43 GMT from X-Acknowledge-To: Resent-Sender: Lucio Chiappetti From: Lucio Chiappetti Sender: fitsbits-request at fits.CV.NRAO.EDU Resent-From: fitsbits-request To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Some FITS tables suggestions Date: Sun, 27 Sep 92 16:10:59 ITA Resent-Date: Sun, 27 Sep 92 16:10:59 ITA On Thu, 24 Sep 92 14:43 GMT said: >Astronomers are using FITS tables, especially binary tables, for a wide >range of applications from star catalogues to photon-event lists. As >William Pence has already noted, FITS is starting to be adopted as a >storage format in its own right and not just as a data interchange >format. Although I am personally questioning the use of FITS for storage, I believe the points raised by Clive Page below concerning use of FITS files as databases AND MORE IN GENERAL about the need of conventions about keyword usage are very relevant EVEN IF (or actually A FORTIORI if) FITS files are used for interchange. (I am personally developing a different, more compact, but system dependent storage format, but I am keeping as a basic requirement that it MUST be one-to-one mappable with FITS : for this reason for instance I am forcing all limitations in FITS kewyords, like 8-char max names, etc. into my own header kewyords - with binary values ) Among the points I feel more relevant in Clive Page's mail : >The software packages noted above fulfil different sets of requirements, >and data interchange between them is extremely likely. But there is a >danger of minor but annoying incompatibilities creeping in unless >we can agree fairly soon on a few more details of the FITS table format, >or at least on some conventions for the use of FITS keywords. YES, conventions are needed to avoid those incompatibilities, which can be very annoying. And I would say those conventions are needed beyond the area of databases, but, for instance, in the area of keywords describing the content of a photon list. At the moment I am using provisional, local conventions for the designation of columns for a photon list : for instance I call THETA and PHI the angular cooordinates of a photon (from the optical axis), X and Y the coordinates in micron on the focal plane, and XPOSITN YPOSITN the coordinates in detector pixels ...or I call ENERGY true energy in keV, but PHA energy in channels, and I ignore whether they are consistent e.g. with EXSAS or PROS - which I am told are not compatible among them) > >(1) Form of column names. >contain letters, digits, and underscores. I would like to propose that >the first character should not be a digit, since leading digits make it I tend to agree >(2) Column name length. >There seems no length limit on a column name (except that FITS keyword >strings cannot be more than 68 chars long). But section 5.3.2.1 says >that "proper interpretation of the FITS file should not require decoding >any more than the first eight characters of a character string". Does Usage of readable column names is definitely a bonus, and therefore names longer than 8 char could be useful. But in the spirit of "once FITS forever FITS" I am ready to live with 8 char. However I note that : a) FITS binary tables are NOT YET an official standard b) the names of the columns (i.e. the VALUE of a TTYPEn keyword) have a lower rank than the names of the keywords themselves | The keyword names are used by the FITS reader, but the values are used by analysis s/w Therefore the limitation of 8 chars is appropriate on names like TFORMnnn, TTYPEnnn, TUNITnnn but less on their content (and in some cases meaningless ... how do you limit to 8 char a VALUE like TUNITnnn = 'PHOTON/CM**2/S/KEV' ? >(3) Sorting. I have no comments on this since it is far from the problems I am working on at the moment. Concerning however keywords of the form Txxxxnnn, which indicate some characteristics of the columns, it would be useful to introduce some more conventional (optional) keywords. At the moment I use TMINnnn and TMAXnnn to indicate the minimum and maximum value in column nnn. >(4) Displaying Sexagesimal Notation. >Most astronomers still use sexagesimal notation (e.g. 12:34:56, >-2:34:56.7) for celestial coordinates. > I wonder if anyone has >considered an extension to allow the TFORM keyword to specify >sexagesimal formats? The obvious way of specifying this is with >pseudo-fortran forms like "HMSw.d" or "DMSw.d" where the ".d" is needed >only if the seconds value has a (decimal) fractional part. Obviously >these formats would have to be handled by special code, but in programs >written in Fortran-77 (the great majority, I would guess) this is >already necessary when handling format descriptors such as "Bw.d" or I believe here there is a mix-up of two different things. BINARY tables are concerned about the representation of the values, therefore an angular value shall be best represented as a DOUBLE PRECISION value (column of type 1D or nD) Sexagesimal notation concerns not internal representation, but displaying of the values (TDISPnnn) ... unless one is talking of ASCII tables (which I believe will be soon replaced by binary tables) Therefore if a binary table COLUMN contains angular quantities, the easiest thing is to have them in degrees as a double precision floating point value. It is up to the program interpreting them to know how to display the numbers. The story is different if we talk about header keywords. Here it might be worthwhile to define new "types" of keywords for quantities like : angles (in degrees), angles (in hours), dates and times For instance, in the software I was mentioning at the beginning, I keep header information as follows : where keyname is the FITS-like name (8 char, blank padded), type and length are two 1-byte codes, and keyvalue is one or more binary values (in general) I support basic types like characters, INTEGER*4 (also INTEGER*2 but I am going to deprecate it), REAL*4 and REAL*8, but also other derived types like ANGLE (which has the same value representation as a REAL*8), and perhaps DATE and TIME (for DATE I am inclined to use an INTEGER*4 value in seconds since 1/1/1970 which is customary in Unix, while for TIME I have not yet made clear my mind). The software which uses these files, may use the type field to decide to format an ANGLE value in sexagesimal notation, or a DATE value in a notation like YYYY-Mon-DD-hh:mm:ss.ff To convert my files to FITS I use Bill Pence's FITSIO, and in general I use the type field to know which of the FTPKYx routines to call (for the basic types) Definition of additional kewyord types (for ANGLEs, DATEs and TIMEs) within FITS would be useful (and possible, since it will be an addition). (*) Arrays of keywords I wonder also whether one should allow somehow arrays of keywords. In my binary headers the field corresponds to the number of characters in a string, and is generally 1 for numeric values, but it is possible for it to be > 1, in which case is a numeric array. (I am not and I WILL NOT be supporting arrays of strings) I was reluctant to introduce keyword arrays, but I was moved by a colleague's request motivated by cases like the following : let's assume we have an instrument with separate sub-units, like for isntance 7 PMTs (photo-multiplier tubes), of which we want to store the high voltage setting. My approach was to have 7 keywords PMTHV1 .... PMTHV7, but he insisted in having instead a single keyword PMTHV, which is a 7-element array. I told him I could support this, but regard it as private within his own software. The main reason is that I do not know of an unique convention of mapping them to FITS. For demo purposes at the moment I am proceeding as follows : I map my PMTHV keyword array to 8 (n+1) FITS keywords PMTHV is an ASCII comment string saying the keyword is an arrau from PMTHV1 to PMTHV7 PMTHV1 is the first element of the array ... PMTHV7 is the last element of the array I know for isntance that MIDAS uses array descriptors (the equivalent of header keywords) and uses a different convention when storing them to FITS. I am unsure about IRAF. I wonder whether some convention could be agreed for the representation of array quantities within FITS headers. ----------------------------------------------------------------------------- ----------------------------------------------------------------------------- Lucio Chiappetti - IFCTR/CNR Milano | U N I C U I Q U E via Bassini 15 - I-20133 Milano - Italy | System Manager Internet: LUCIO at IFCTR.MI.CNR.IT | FORTUNAE SUAE Decnet: IFCTR::LUCIO (39610::LUCIO) | Bitnet: EXOSAT at IMISIAM | ----------------------------------------------------------------------------- From fitsbits-request Mon Sep 28 05:13:19 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["443" "Mon" "28" "September" "92" "10:12" "GMT" "\"Clive Page, Leicester Univ, UK\"" "CGP at STARLINK.LEICESTER.AC.UK" nil "9" "Correction to my earlier article" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA07385; Mon, 28 Sep 92 05:13:19 EDT Return-Path: Message-Id: <9209280912.AA02045 at cv3.cv.nrao.edu> Via: uk.ac.leicester.starlink; Mon, 28 Sep 1992 10:12:12 +0100 From: "Clive Page, Leicester Univ, UK" Sender: fitsbits-request at fits.CV.NRAO.EDU To: FITSBITS Subject: Correction to my earlier article Date: Mon, 28 Sep 92 10:12 GMT I'm grateful to Lucio Chiapetti of Milan for pointing out a silly mistake in my article of last week "Some FITS tables suggestions". In the section on sexagesimal formatting I should, of course, have referred to the TDISPnnn keyword (which contains the suggested external display form) and not the TFORMnnn keyword (which describes the internal form) in binary tables. My apologies; I hope this didn't confuse too many other people. -Clive From fitsbits-request Wed Sep 30 14:00:29 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["4011" "Wed" "30" "September" "1992" "17:15:10" "GMT" "William Pence" "pence at heawk1.gsfc.nasa.gov " nil "83" "triple precision keywords" "^From:" nil nil "9"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA13326; Wed, 30 Sep 92 14:00:29 EDT Return-Path: Message-Id: Organization: Goddard Space Flight Center Path: cv3.cv.nrao.edu!uvaarpa!darwin.sura.net!jvnc.net!yale.edu!think.com!ames!nsisrv!heawk1!pence From: pence at heawk1.gsfc.nasa.gov (William Pence) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: triple precision keywords Date: Wed, 30 Sep 1992 17:15:10 GMT Here at the HEASARC we often need to express keyword values which have more significant figures than can be stored in a double precision number. I have a proposal to handle this, but it may make it difficult for other data systems to read the value, so I would like some comments from others before we implement such a system. A common example of this problem is the Modified Julian Date which has a value something like 48043.8797453703700742 Currently, we express this number as 2 separate keywords, one representing the integer value, and another with the double precision fractional value, for example: MJDI = 48043 / integer portion of MJD MJDF = 8.797453703700742E-1 / fractional portion of MJD This however has several disadvantages: - it requires 2 keywords = 160 bytes to express the value - it clutters up the header with extra keywords which makes it more difficult for people to read - it makes it more difficult to adopt clear and concise keyword names since the last character in the name must be reserved to indicate whether it is the integer or fractional part. - it requires that the reading and writing programs make 2 subroutine calls to get or put the data. In principle there is no reason that this information could not all be expressed in a single keyword value, like: MJD = 48043.8797453703700742 / MJD The only difficultly with this is that there has been no convenient way to read or write values in this format, so I have proposed to add (in fact they are already written) 2 new routines to the FITSIO subroutine library to solve this problem. The new subroutines are: ftpkyt - to write a keyword consisting of an integer + fraction ftgkyt - to read the integer an fractional parts of a keyword value Both routines have the same arguements: call ftpkyt(iunit,keyword,intval,dval,comment,status) iunit (integer) - the fortran unit number of the FITS file keyword (C*8) - the name of the keyword to read or write intval (I*4) - the integer part of the keyword value dval (R*8) - the fractional part of the keyword value comm (C*48) - the comment string status (I*4) - returned error status The intval, dval, and comm parameters are input parameters to the ftpkyt routine, and output parameters in the ftgkyt routine. The ftpkyt routine will write a new keyword in f28.16 fixed point format which consists of an integer plus a double precision fraction. The ftgkyt routine will read any numeric keyword value (it is not restricted just to the special keywords written by ftpkyt) and returns the integer and fractional parts in separate subroutine parameters. This proposal is, I believe, completely compatible with the FITS standard and any FITS reader should be able to read keywords with this format however the reader may not perserve all the precision unless is does some special processing. This does not worry me too much however since any software that needs to use the full precision of this keyword is bound to be highly specialized, hence it is not asking too much to require that the software to be able to handle this type of keyword. The bigger issue that I see is how can the content of this keyword be preserved when the FITS file is read into another data system. For example, if a FITS image containing a keyword of this type was read and converted to an IRAF format image file, then I would assume that the IRAF FITS reader would not choke on this type of keyword, but that it probably would not preserve the approximately 26 decimal places of precision in the value. Can anyone please comment whether keywords in this format would present problems for their favorite data analysis system. And if so, are these problems severe enough that we should not attempt to use this keyword format in our files? -Bill Pence pence at tetra.gsfc.nasa.gov HEASARC::PENCE