From dwells Wed Apr 7 16:36:26 1993 X-VM-VHeader: ("From:" "Sender:" "Resent-From" "To:" "Apparently-To:" "Cc:" "Subject:" "Date:" "Resent-Date:") nil X-VM-Bookmark: 8 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2280" "Wed" "7" "April" "93" "16:36:25" "EDT" "Don Wells" "dwells " nil "43" "WGAS FITS Committee voting " "^From:" nil nil "4"]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA16887; Wed, 7 Apr 93 16:36:26 EDT Return-Path: Message-Id: <9304072036.AA16881@fits.cv.nrao.edu> From: dwells (Don Wells) Sender: wfc-request@fits.CV.NRAO.EDU To: wfc Subject: WGAS FITS Committee voting Date: Wed, 7 Apr 93 16:36:25 EDT Dear WGAS FITS Committee members, We still need to hold a vote on the three pending FITS proposals. Four months ago we deferred the vote in order to consider adding arrays of strings to the BINTABLE proposal. The discussions of that proposed modification have occurred in private Email, and as of today we appear to have an agreement. Therefore, I am alerting the WFC that I intend to hold the call-for-votes during the next week (12-16 April); I have not yet decided precisely which day it will be. If any of you have some problem with this please send me private Email. The anonymous-FTP server fits.cv.nrao.edu [192.33.115.8] contains the following three files in directory FITS/Documents: -r--r--r-- 1 dwells 2183 Oct 1 1991 FITS_blocking90.txt -r--r--r-- 1 dwells 179394 Sep 23 1991 X_bintable.ps -r--r--r-- 1 dwells 94980 Jun 4 15:36 X_image.ps These are the three proposals which the European FITS Committee has already approved, and which WFC will be considering. If you have not already examined these and formed an opinion, please do so *NOW*. I am expecting that the text of an additional appendix for the X_bintable document will be provided to WFC by Bill Pence within a few days. The text of this appendix will specify a convention for substrings in a string (A) field; to do this it will append characters to the basic TFORM code. It will be of the form rAw/nnn, where r is the total length of the string field, w is the fixed, blank-padded substring length, and /nnn is the optional numeric code for a printing character to act as a delimeter for variable length strings. If the delimeter is provided the w is interpreted as the maximum substring length. This layered convention will provide the functionality which is desired as an upwardly compatible feature of the basic BINTABLE definition. This means that approval of this convention by WGAS will not be in conflict with the approval of basic BINTABLE by the European FITS Committee. -Don Donald C. Wells Associate Scientist dwells@nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From wfc-request@fits.CV.NRAO.EDU Mon Apr 12 10:27:55 1993 Status: RO X-VM-v5-Data: ([nil nil nil nil nil t nil nil nil] ["5048" "Mon" "12" "April" "93" "10:27:49" "EDT" "William Pence" "pence@tetra.gsfc.nasa.gov " nil "124" "Appendix C to BINTABLE definition document" "^From:" nil nil "4"]) Received: from cv3.cv.nrao.edu by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA01823; Mon, 12 Apr 93 10:27:54 EDT Received: from fits.cv.nrao.edu by cv3.cv.nrao.edu (4.1/DDN-DLB/1.13) id AA05453; Mon, 12 Apr 93 10:27:25 EDT Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA01820; Mon, 12 Apr 93 10:27:44 EDT Return-Path: Message-Id: <9304121427.AA24063@tetra.gsfc.nasa.gov> From: pence@tetra.gsfc.nasa.gov (William Pence) Sender: wfc-request@fits.CV.NRAO.EDU To: wfc@fits.CV.NRAO.EDU Cc: pence@tetra.gsfc.nasa.gov Subject: Appendix C to BINTABLE definition document Date: Mon, 12 Apr 93 10:27:49 EDT Dear WGAS FITS Committee members, Attached below is the proposed Appendix C to the 'Binary Table Extension to FITS' document. Since Don Wells will be calling for a vote on the 3 pending FITS proposals later this week, please circulate any comments or objections to this proposal as soon as possible. -Bill Pence ----- Latex document follows; cut here ------------------------------------------ \documentstyle{article} \begin{document} \appendix{} \begin{center} {\LARGE\bf Appendix C \par} \vskip 1em {\large\bf ``Substring Array'' Convention} \end{center} \vskip 20pt This appendix describes a layered convention for specifying that a character array field (TFORMnnn = 'rA') consists of an array of either fixed-length or variable-length substrings within the field. This convention utilizes the option described in the basic binary table definition document to have additional characters following the datatype code character in the TFORMnnn value field. The full form for the value of TFORMnnn within this convention is 'rAw/nnn' where, \begin{verse} {\it r} is an integer giving the total length (in characters) of the field, \\ {\it A} signifies that this is a character array field, \\ {\it w} is an integer $\leq $ r giving the (maximum) number of characters in an individual substring (not including the delimiter), and \\ {\it /nnn} if present, indicates that the substrings have variable-length and are delimited by an ASCII text character with decimal value nnn in the range 032 to 126 decimal, inclusive. \end{verse} To illustrate this usage: \begin{verse} '40A8' signifies that the field is 40 characters wide and consists of an array of 5 8-character fixed-length substrings. '100A8/032' signifies that the field is 100 characters wide and consists of an array of variable-length substrings where each substring has a maximum length of 8 characters and is terminated by an ASCII SPACE (decimal 32) character. \end{verse} Note that simple FITS readers that do not understand this substring convention can ignore the TFORM characters following the {\it rA} and can interpret the field simply as a single long string as described in the basic binary table definition document. The following rules complete the full definition of this convention: \begin{enumerate} \item In the case of fixed-length substrings, if {\it r} is not an integer multiple of {\it w} then the remaining odd characters are undefined and should be ignored. For example if TFORMnnn = '14A3', then the field contains 4 3-character substrings followed by 2 undefined characters. \item Fixed-length substrings must always be padded with blanks if they do not otherwise fill the fixed-length subfield. The ASCII NUL character must not be used to terminate a fixed-length substring field. \item If an array of variable-length substrings does not completely fill the binary table field, then the last substring must be terminated with an ASCII NUL character (value 000), rather than the substring delimiter character. \item The method of signifying an undefined or null substring within a fixed-length substring array is not explicitly defined by this convention (note that there is no ambiguity if the variable-length format is used). In most cases it is recommended that a completely blank substring be used for this purpose. In cases where it is necessary to make a distinction between a blank substring and an undefined substring, then any other suitable string may be used to designate the undefined substrings (e.g., 'INDEF' or 'NULL'). \item Undefined or null variable-length substrings are designated by a zero-length substring, i.e., by a delimiter character (or an ASCII NUL if it is the last substring in the table field) in the first position of the substring. An ASCII NUL in the first character of the table field indicates that the field contains no defined variable-length substrings. \item The ``Multidimensional Array''convention described in Appendix A of this document provides a syntax using the TDIMnnn keyword for describing multidimensional arrays of any datatype which can also be used to represent arrays of fixed-length substrings. To minimize confusion it is recommended that the substring convention and the TDIM convention not be used for two character columns in the same table. The substring convention should be considered the preferred method for storing arrays or lists of strings in a character field of a table record. \item This substring convention may be used in conjunction with the ``Variable Length Array'' facility described in Appendix B of this document. In this case, the full form for the value of the TFORM keyword is: \begin{verbatim} TFORMnnn= 'rPAw/nnn(maxelem)' / for variable-length substrings TFORMnnn= 'rPAw(maxelem)' / for fixed-length substrings \end{verbatim} \end{enumerate} \vskip 1em This convention is optional and will not preclude other conventions. This convention is not part of the proposed binary table definition. \end{document} From wfc-request@fits.CV.NRAO.EDU Mon Apr 12 11:10:13 1993 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1546" "Mon" "12" "April" "93" "11:09:19" "EDT" "Bob Hanisch" "hanisch@stsci.edu" nil "24" "substring proposal" "^From:" nil nil "4"]) Received: from cv3.cv.nrao.edu by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA01858; Mon, 12 Apr 93 11:10:12 EDT Received: from fits.cv.nrao.edu by cv3.cv.nrao.edu (4.1/DDN-DLB/1.13) id AA05716; Mon, 12 Apr 93 11:09:43 EDT Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA01855; Mon, 12 Apr 93 11:09:53 EDT Return-Path: Message-Id: <9304121509.AA01407@SOL.STSCI.EDU> From: Bob Hanisch Sender: wfc-request@fits.CV.NRAO.EDU To: wfc@fits.CV.NRAO.EDU Cc: hanisch@stsci.edu Subject: substring proposal Date: Mon, 12 Apr 93 11:09:19 EDT I think Bill's proposal is a good way to handle substrings _in_principal_, but object to this particular implementation. In particular, this convention is unlike any other in FITS in that it alters the list of acceptable values for a reserved keyword in a manner that could cause a FITS reader to fail. The binary tables proposal gives an explicit list of accepted values (rB, rI, rJ, rA, rE, rD, rC, rM, rP). Allowing this to be extended to rAw/nnn could easily cause problems for FITS readers that adhere to the standard for binary tables but do not support the proposed add-on convention. That is, it would not be unreasonable for a reader simply take the value of TFORMnnn _as_is_, without checking for the extension characters. This would most likely cause the reader to fail if it encountered the substring notation. Even though the TDIM convention in Appendix B is somewhat less clever, it does not have the problem identified above. If the TDIM keyword is not understood by a FITS reader, no harm is done. Thus, I would suggest that a similar technique be used to support the substring convention. A new keyword should be defined which _complements_ TFORMnnn for type rA data. The value fields for TFORMnnn should be left unchanged. Indeed, I would argue that the proposed Appendix C, should it be acceptable otherwise, forces us to revise the main part of the binary tables proposal so that it is clear that other characters might be appended to the "rA" string (and, presumably, to the other data type strings). Bob Hanisch From wfc-request@fits.CV.NRAO.EDU Mon Apr 12 11:23:31 1993 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["720" "Mon" "12" "April" "93" "08:22:43" "MST" "Doug Tody" "tody@noao.edu " nil "10" "substring proposal" "^From:" nil nil "4"]) Received: from cv3.cv.nrao.edu by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA02040; Mon, 12 Apr 93 11:23:29 EDT Received: from fits.cv.nrao.edu by cv3.cv.nrao.edu (4.1/DDN-DLB/1.13) id AA05763; Mon, 12 Apr 93 11:23:00 EDT Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA02037; Mon, 12 Apr 93 11:23:17 EDT Return-Path: Message-Id: <9304121522.AA01638@tucana.tuc.noao.edu> From: tody@noao.edu (Doug Tody) Sender: wfc-request@fits.CV.NRAO.EDU To: wfc@fits.CV.NRAO.EDU Subject: substring proposal Date: Mon, 12 Apr 93 08:22:43 MST Bob, you are missing a key point - the original BT proposal explicitly states that some "unspecified" characters may follow the rA or rP in TFORMnnn. The reason this was done was to provide a hook for just such a convention as Bill's substring proposal. BT table readers have to be able to deal with the possibilty of characters following the rA or rP, although since these are layered conventions, they can ignore these extra characters if they choose and still read the table successfully. In particular, in the case of the proposed substring convention, the array or list of strings format is layered upon the simple character vector (NUL terminated ASCII sequence) defined by the core BT proposal. - Doug From wfc-request@fits.CV.NRAO.EDU Mon Apr 12 11:39:41 1993 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["422" "Mon" "12" "April" "93" "11:39:59" "EDT" "William Pence" "pence@tetra.gsfc.nasa.gov " nil "10" "substring proposal" "^From:" nil nil "4"]) Received: from cv3.cv.nrao.edu by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA02661; Mon, 12 Apr 93 11:39:40 EDT Received: from fits.cv.nrao.edu by cv3.cv.nrao.edu (4.1/DDN-DLB/1.13) id AA05937; Mon, 12 Apr 93 11:39:11 EDT Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA02658; Mon, 12 Apr 93 11:39:33 EDT Return-Path: Message-Id: <9304121539.AA24115@tetra.gsfc.nasa.gov> From: pence@tetra.gsfc.nasa.gov (William Pence) Sender: wfc-request@fits.CV.NRAO.EDU To: wfc@fits.CV.NRAO.EDU Subject: substring proposal Date: Mon, 12 Apr 93 11:39:59 EDT I think Doug's reply pretty much answers Bob's objections. Note that this issue is specifically addressed in the second paragraph of Appendix C: "Note that simple FITS readers that do not understand this substring convention can ignore the TFORM characters following the {\it rA} and can interpret the field simply as a single long string as described in the basic binary table definition document." -Bill From wfc-request@fits.CV.NRAO.EDU Mon Apr 12 11:58:55 1993 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["123" "Mon" "12" "April" "93" "11:58:03" "EDT" "Bob Hanisch" "hanisch@stsci.edu" nil "4" "re: substring proposal" "^From:" nil nil "4"]) Received: from cv3.cv.nrao.edu by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA02747; Mon, 12 Apr 93 11:58:54 EDT Received: from fits.cv.nrao.edu by cv3.cv.nrao.edu (4.1/DDN-DLB/1.13) id AA06128; Mon, 12 Apr 93 11:58:25 EDT Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA02744; Mon, 12 Apr 93 11:58:42 EDT Return-Path: Message-Id: <9304121558.AA01510@SOL.STSCI.EDU> From: Bob Hanisch Sender: wfc-request@fits.CV.NRAO.EDU To: wfc@fits.CV.NRAO.EDU Subject: re: substring proposal Date: Mon, 12 Apr 93 11:58:03 EDT OK, Doug and Bill reminded me of the text in the bintables proposal that allows the extra text. Objection withdrawn. Bob From dwells Mon Apr 12 18:48:59 1993 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] [nil nil nil nil nil nil nil nil nil nil nil nil "^From:" nil nil nil]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA03203; Mon, 12 Apr 93 18:48:59 EDT Return-Path: Message-Id: <9304122248.AA03197@fits.cv.nrao.edu> From: dwells (Don Wells) Sender: wfc-request@fits.CV.NRAO.EDU To: wfc Subject: WFC to vote two days from now Date: Mon, 12 Apr 93 18:48:58 EDT Dear WGAS FITS Committee members, I propose to hold the WFC vote on the three FITS proposals two days from now (please don't send your votes until I call for them). If anyone objects to this schedule please tell me by Email. If anyone wants further time to discuss the proposals, or has an objection that they are reluctant to discuss publicly, please tell me by Email *now*. The WGAS BINTABLE vote will be on the basic BINTABLE document plus the Pence-Tody appendix C. It is understood, of course, that Appendices A, B and C "[are] optional and [do] not preclude other conventions [and are] not part of the proposed binary tables definition." My view of the three appendices is that (1) they are recommended practices which will guide prototyping work during the next few years and (2) they are a strong proof that the basic BINTABLE proposal is a proper foundation for future evolution. We will cast separate votes on the three proposals. As is customary in standards work, the voting will not be secret. I append the list of voting members. Report errors or changes to me. I myself expect to vote YES on all three proposals. I append some text which I sent to one of you in December, a person who said that they were new to FITS design deliberations and wanted some advice from me about how to vote. Regards, Don -*- The 20 Voting Members of WFC -*- leb@hypatia.gsfc.nasa.gov crabtree@dao.nrc.ca egreisen@nrao.edu hanisch@stsci.edu henden@mps.ohio-state.edu kibrick@lick.ucsc.edu wlupton@keck.hawaii.edu eric@cfa.harvard.edu bob@ipac.caltech.edu pence@tetra.gsfc.nasa.gov jwp@sal.wisc.edu skip@as.arizona.edu bschlesinger@nssdca.gsfc.nasa.gov teuben@astro.umd.edu rthompson@iue.gsfc.nasa.gov tody@noao.edu mev@ndadsa.gsfc.nasa.gov warnock@hypatia.gsfc.nasa.gov dwells@nrao.edu wwilliams@noao.edu -*- Extracts from a 05Dec92 message to a WFC member -*- Bob Hanisch asked you to serve in order that [your observatory/project] and related astronomy groups would be represented. People from other institutions were asked to serve for the same reason. Bob and I have no illusions that you or others who have not been involved in the earlier FITS design debates can [come] fully up to speed in [a few months]. One purpose of forming the committee and asking persons like yourself to serve on it was that, in time, the committee work would effectively increase both the number and the diversity of the people who would be knowledgeable and would be capable of participating in the esoteric design debates... Of the three proposals, the one that *really* matters is BINTABLE. I hope that you approve of this proposal. In fact, I hope that you *really* approve of it, so much so that you would use it for [your observatory/project] in appropriate places. Most people familiar with FITS regard BINTABLE as the most important proposal since the original Basic FITS agreement. I myself expect that it will eventually become the dominant data format in astronomical archives. As Doug Tody remarked, the people who negotiated BINTABLE expect to layer a variety of other agreements on top of it. If you study BINTABLE closely I expect that you will perceive the sweeping generality of this format, its power to do almost anything. The variable-length field scheme in the appendix is probably going to be immensely important in the long run. To answer your question: canvas local opinion if that makes you comfortable, or vote your own mind if that is what you want to do... The blocking proposal is cut and dried, and probably not controversial. Ditto for the IMAGE proposal. The real issue is BINTABLE. If you object, please be brave enough to say so! If you are unwilling to say it publicly, then say it to me privately, and we can discuss it. I will respect your opinion, and probably will try to change it, because I really do want [your observatory/project] to be on board on major FITS decisions. This one is major. If you think BINTABLE is OK, but you don't want to use it (at least not at present), then I recommend that you vote YES in order to demonstrate moral support for the other astronomy communities that have stated that this format is critical for their applications. -- Donald C. Wells Associate Scientist dwells@nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From dwells Wed Apr 14 09:04:08 1993 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] [nil nil nil nil nil nil nil nil nil nil nil nil "^From:" nil nil nil]) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA01579; Wed, 14 Apr 93 09:04:08 EDT Return-Path: Message-Id: <9304141304.AA01573@fits.cv.nrao.edu> From: dwells (Don Wells) Sender: wfc-request@fits.CV.NRAO.EDU To: wfc Subject: WFC vote deferred once more Date: Wed, 14 Apr 93 09:04:07 EDT Dear WFC (and IAU WG-chairman Preben Grosbol), There have been a series of private Email exchanges on the details of the string-array proposal (appendix C) during the past 36 hours. It is clear that we do not yet have a consensus, and that more discussion will be needed to obtain an agreement which everyone can support whole-heartedly. Therefore, we will wait. I exhort the people who are involved to pursue this matter diligently -- the community as a whole needs to have a final decision on BINTABLE. The people who are involved, who are experienced FITS negotiators, will need to decide whether the discussion should be public or private. The advantage of private discussion -- as always -- is speed, while the advantage of public discussion is that the FITS community at large would be able to contribute its opinions and would feel that it is a part of the process. If the decision is for public discussion, the negotiators will need to decide whether to use the wfc exploder or sci.astro.fits/fitsbits. I can think of arguments for both. -*- A major lesson of FITS history is that bi-lateral negotiations work. For ten years the astronomy community had huge success with this formula. Another lesson of the history is that tri-lateral negotiations are *much* harder than bi-lateral. Now, in the era of three regional FITS committees, one of which has 20 voting members, we have moved well beyond even tri-lateral negotiations. We are engaged in a fundamentally *political* process, unfamiliar ground for physical scientists and computing specialists. Political processes are inherently inefficient. We must always be patient, and as civil as we can be, as we work out our differences of opinion and forge a consensus so that we can accomplish something substantive. Designs done in a political context are *compromises*. This is neither good nor bad, it is just the way things are, the way they have to be. Best regards, Don Donald C. Wells Associate Scientist dwells@nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From wfc-request Wed Apr 14 12:26:29 1993 Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA02004; Wed, 14 Apr 93 12:26:29 EDT Return-Path: Date: Wed, 14 Apr 93 12:26:49 EDT From: pence@tetra.gsfc.nasa.gov (William Pence) Message-Id: <9304141626.AA24673@tetra.gsfc.nasa.gov> To: wfc@fits.CV.NRAO.EDU Subject: string convention in binary tables Cc: pence@tetra.gsfc.nasa.gov Sender: wfc-request@fits.CV.NRAO.EDU I just noticed an apparent contradiction between the core BINTABLE proposal and Appendix A, which states that each individual substring may be null terminated. But my understanding of the core proposal was that only a single null was allowed in a character array column. If we can have more than one NUL in a character column, then this neatly solves the problem of how to signify undefined fixed length substrings in the proposed new substring array convention. So, should the core proposal be modified to make it clear that the character array column can contain more than one null character? This would mean that simple FITS readers that do not understand the layered conventions cannot simply ignore any characters following a null character in a character column, but must instead preserve all the characters in the entire field. -Bill Pence