Date: Wed, 11 Jul 2007 16:14:38 -0400 From: William Pence To: FITSBITS Subject: [fitsbits] Start of the CONTINUE keyword Public Comment Period This is to announce the start of the 30-day Public Comment Period on the CONTINUE keyword convention that has been submitted for inclusion in the 'Registry of FITS Conventions' which is maintained by the IAU FITS Working Group. This is the 9th in a growing series of conventions submitted for inclusion in the Registry. Detailed information about this convention and a sample FITS file that uses it are available for public review and comment from the FITS registry web page at http://fits.gsfc.nasa.gov/fits_registry.html The CONTINUE keyword allows long character string keyword values to be continued on the next keyword(s) in the FITS header. This convention supports string keyword values that are longer than the 68-character maximum that can be represented in a single FITS keyword. A trivial example of this convention is shown below: SVALUE = 'This is a long string value that is &' / Any comments CONTINUE 'continued over three keywords& ' / may be appended CONTINUE ' in the FITS header.' / after the quoted value. Under this convention, the last character of each string that is to be continued must be the `&' character, and the continuation string is contained in a sequence of 1 or more CONTINUE keywords (which do not have a `=' character in column 9 and hence have no formal value). Comments may be posted here on the FITSBITS mail exploder or the sci.astro.fits newsgroup. Minor typographical issues may be sent directly to the authors of the convention. Bill Pence (on behalf of the IAU FITS Working Group) -- ____________________________________________________________________ Dr. William Pence pence@milkyway.gsfc.nasa.gov NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice) Greenbelt MD 20771 +1-301-286-1684 (fax) ************************************************************************ ************************************************************************ Date: Wed, 11 Jul 2007 17:37:08 -0400 From: Peter Teuben To: FITSBITS Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period perhaps it would be useful to explicitly mention that the & character is not part of the resulting keyword value. And also make the example shorter so you can easily show the equivalent string in a single-keyword example: i.e. SVALUE = 'This is a long string value &' CONTINUE 'extending& ' CONTINUE ' over 3 lines.' is the same as SVALUE = 'This is a long string value extending over 3 lines.' and also thus showing anything after the & does not count. A reminder is thus in place how the & can be made part of the SVALUE string. && ?? ************************************************************************ ************************************************************************ Date: Wed, 11 Jul 2007 16:57:41 -0400 From: Thomas McGlynn To: William Pence Cc: FITSBITS Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period It might be useful to explicitly address some corner cases, e.g., long strings that end in &. SVALUE = 'This is a long string &' CONTINUE 'really long &' COMMENT 'But is this part of it?' CONTINUE 'Or is this?' I guess that the intended value of SVALUE in this case is 'This is a long string really long &' but I could imagine some users thinking that this is an error or that the last & is dropped and I could even imagine some users (conditioned by how comment lines which can appear in the middle of statements in programs), might things that the COMMENT is ignored and the value is 'This is a long string really long Or is this?' You might also wish to explicitly note what is supposed to be done with a CONTINUE statement that doesn't follow a string ending in &. Presumably it is to be ignored. SVALUE = 'This is a short string' CONTINUE 'And this is not part of it' Tom McGlynn ************************************************************************ ************************************************************************ Date: Thu, 12 Jul 2007 11:15:06 +0200 From: Walter Jaffe To: fitsbits@nrao.edu Subject: [fitsbits] Start of the CONTINUE keyword Public Comment Period While I think this is a useful convention, I note that this introduces an implicit ordering of FITS headers, i.e. they have to be read in a certain order if you want to interpret them correctly. As far as I know, this is the first time such an implicit ordering has been introduced and it could conceivable cause some problems in copying/concatenating headers together. There are of course a few explicit ordering conventions i.e. you have to start with SIMPLE = T. I doubt this CONTINUE statement will cause much trouble in practise, but before approving it we might want to consider the more general problems of implicit ordering. Walter ************************************************************************ ************************************************************************ From: Rob Seaman Date: Thu, 12 Jul 2007 09:21:12 -0700 To: FITSbits Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period Perhaps the discussion from 1993 and 1994 could be included verbatim, or linked by thread off the web page? I'd hate to hold a debate on a topic that was already batted around a dozen years ago. Is the registry page the entire text of the proposed convention? Like Tom said, you need to cover the obvious corner cases, at least by verbiage like "it is undefined what string is conveyed if a card ending is '&' is not followed by CONTINUE". 1) Couldn't the reordering question be dealt with via appending &NNN instead of &? That could resolve some of the corner cases, too, and you wouldn't need to reserve multiple CONT_NNN keywords. We should be leery of implicitly requiring that all FITS headers be now considered strictly ordered, when there is no practical way to enforce this. In particular, this has the side effect of making the interpolation of any new keyword questionable other than immediately before END. 2) What is the reason for not strictly requiring that such a continuation string end with ampersand-quote? What advantage is conveyed by permitting semantically null trailing whitespace that merely confuses the handling? Surely it should be incumbent on the author of such keywords to trim meaningless characters, not generations of readers yet unborn to dodge around them. 3) One suspects a number of conventions will depend on valueless keywords. The standard (2.1b) is rather convoluted on how this works: 5.1.2.2 Value Indicator (bytes 9–10) If this field contains the ASCII characters “=”, it indicates the presence of a value field associated with the keyword, unless it is a commentary keyword as defined in §5.4.2.4. If the value indicator is not present or if it is a commentary keyword then columns 9–80 may contain any ASCII text. Which basically just acts to preserve the legality of COMMENT, HISTORY and keywords to have an equals sign in column 9. 4) In general, the nature of a convention versus a standard is that others are perfectly within their rights to violate a convention for either good or not-so-good reasons. The behavior here seems simple, but that's where the risk lies. Nobody would be unsurprised that deleting the INHERIT keyword turns inheritance off - it's almost a feature. Delete the CHECKSUM keyword and you simply remove an assertion about the contents of the file. One can even productively use DATASUM and CHECKSUM independently. (I'll have to check if we made that legal.) On the other end of the spectrum, conventions like tiled image compression and foreign file encapsulation involve very specific FITS usage - a file must adhere to the standard and additionally must adhere to the complex requirements of the convention. The CONTINUE convention, however, is different in providing a more general capability that could be used, for instance, within any other convention. Deleting a CONTINUE card could in principle change the meaning of any header in arbitrary ways. I wonder if perhaps we shouldn't rather be discussing how best to add this capability to the core FITS standard. Whether convention or new feature of the standard, however, the loopholes should be closed before adoption. Rob ************************************************************************ ************************************************************************ From: Rob Seaman Date: Thu, 12 Jul 2007 09:40:19 -0700 To: FITSbits Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period > The CONTINUE convention, however, is different in providing a more > general capability that could be used, for instance, within any > other convention. I don't think I emphasized this point enough. A continuation convention, whether the one on the table or any other, changes the meaning of other conventions. One could, for instance, recast a CHECKSUM keyword: CHECKSUM= 'nADTnADQnADQnADQ' / ASCII 1's complement checksum as: CHECKSUM= 'nADTnADQ& ' / ASCII 1's complement checksum CONTINUE 'nADQnADQ' This would apply to your favorite string valued keyword as well. The example above would be a legal continuation but an illegal checksum value. (More than illegal for this example since the byte ordering needs to be preserved for the zeroing feature of the checksum to work.) Perhaps we need a broader discussion of the boundaries of usage for conventions. I suspect the issue of long values for string keywords remains an issue for the trusty old standards process, not the avant garde registry. Continuation is only one possible way to handle long strings. Rob ************************************************************************ ************************************************************************ Date: Thu, 12 Jul 2007 14:41:11 -0400 From: William Pence To: FITSBITS Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period Peter Teuben wrote: > perhaps it would be useful to explicitly mention that the & character is > not part of the resulting keyword value. > > And also make the example shorter so you can easily show the equivalent string > in a single-keyword example: Agreed. I'll make these changes in the next version of the documentation. Thomas McGlynn wrote: > It might be useful to explicitly address some corner cases, e.g., > long strings that end in &. > > SVALUE = 'This is a long string &' > CONTINUE 'really long &' > COMMENT 'But is this part of it?' > CONTINUE 'Or is this?' > > I guess that the intended value of SVALUE in this case is > 'This is a long string really long &' Correct. If a string value ends with an `&' and is NOT followed by a CONTINUE keyword, then the `&' character is interpreted literally as the last character of the string value). > You might also wish to explicitly note what is supposed > to be done with a CONTINUE statement that doesn't follow > a string ending in &. Presumably it is to be ignored. > > SVALUE = 'This is a short string' > CONTINUE 'And this is not part of it' Yes. Software should treat orphaned CONTINUE keywords the same as a COMMENT keyword. Software that doesn't support this convention at all should treat all the CONTINUE keywords this way. I'll revise the documentation to explicitly cover these cases. Bill Pence ************************************************************************ ************************************************************************ Date: Thu, 12 Jul 2007 15:54:51 -0400 From: William Pence Cc: FITSbits Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period Rob Seaman wrote: > Perhaps the discussion from 1993 and 1994 could be included verbatim, > or linked by thread off the web page? I'd hate to hold a debate on a > topic that was already batted around a dozen years ago. There is a link on the registry web page to the (very large!) set of comments that were made about this convention when it was first introduced, but the archived FITSBITS discussions for this time period are just broken up into monthly text files. You have to scan through the whole text file to pick out the relevant postings. It would take a lot of editing to make a file just containing the CONTINUE thread. > Is the > registry page the entire text of the proposed convention? Like Tom > said, you need to cover the obvious corner cases, at least by > verbiage like "it is undefined what string is conveyed if a card > ending is '&' is not followed by CONTINUE". What you see is the entire existing text of the convention, but I plan to augment it with some additional text, based on the comments that are made here during the public comment period. > 1) Couldn't the reordering question be dealt with via appending &NNN > instead of &? ... > 2) What is the reason for not strictly requiring that such a > continuation string end with ampersand-quote? ... These are valid questions, but they are a bit 'off-topic' since the purpose of the registry is to document existing conventions, not defend the way they were designed. If we were designing a new continuation convention now we might do things differently. > 3) One suspects a number of conventions will depend on valueless > keywords. Agreed. The ESO HIERARCH keyword (which will be submitted for inclusion in the Registry in the near future) is another well known example that is somewhat complementary to the CONTINUE convention. The HIERARCH keyword convention effectively allows keyword names much longer than the 8-character limit of real keywords > The CONTINUE convention, however, is different in providing a more > general capability that could be used, for instance, within any other > convention. Deleting a CONTINUE card could in principle change the > meaning of any header in arbitrary ways. As you further discussed in a subsequent posting, this CONTINUE convention could be used in principle with any string-valued keyword, including the mandatory or reserved keywords like TFORMn, TTYPEn, EXTNAME. In practice, however, the continue convention has mainly only been used for very instrument-specific keywords like TDDES12 = 'D[0~3] & E[0~63] C[(S[msLimit1]~S[msLimit2]),((S[msLi&' CONTINUE 'mit2]+1)~S[msLimit3]),((S[msLimit4]+1&' CONTINUE ')~S[msLimit5])] ' > I wonder if perhaps we > shouldn't rather be discussing how best to add this capability to the > core FITS standard. This was one of the reasons for setting up the registry in the first place. The first step is to simply document the existing convention in the registry; as a second step, this may lead to a wider discussion of how this sort of capability could be added to the FITS standard. Back in 1993 when this convention was first discussed, we failed to reach any consensus, largely, I think, because many people at that time felt that there was no compelling need for any keyword strings longer than 68-characters. In the rare cases where anyone need longer strings, ad-hoc work-arounds were devised, rather than coming up with a general solution to the problem. Maybe this thinking has changed in the past 14 years(?). If enough people agree that this is an issue worth addressing, it should not be hard to come up with a technical solution for handling long string keyword values (not necessarily identical to the current CONTINUE convention) that could be considered through the official approval process for inclusion in the FITS Standard. Bill Pence ************************************************************************ ************************************************************************ Date: Thu, 12 Jul 2007 16:10:24 -0400 From: William Pence To: fitsbits@nrao.edu Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period Walter Jaffe wrote: > While I think this is a useful convention, I note that this introduces > an implicit ordering of FITS headers, i.e. they have to be read > in a certain order if you want to interpret them correctly. As > far as I know, this is the first time such an implicit ordering > has been introduced and it could conceivable cause some problems > in copying/concatenating headers together. There are of course > a few explicit ordering conventions i.e. you have to start with > SIMPLE = T. Another case where ordering may be implicit is in a sequence of HISTORY keywords that document the order of data processing steps. In some cases time stamps may have been added to each keyword to disambiguate the order. This issue of keyword ordering was one of the main topics of debate back when this convention was first introduced back in 1993 and 1994. > I doubt this CONTINUE statement will cause much trouble in practise, > but before approving it we might want to consider the more > general problems of implicit ordering. Keep in mind that the approval process for adding a convention to the Registry is mainly only designed to ensure that the documentation is clear and complete. If a continuation convention were to be considered for the FITS Standard, then a much more rigorous and formal approval process would be required. Bill Pence ************************************************************************ ************************************************************************ Date: Thu, 12 Jul 2007 21:39:13 -0400 From: Tom McGlynn To: Tom McGlynn Cc: fitsbits@nrao.edu Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period Tom McGlynn wrote: > Mark Calabretta wrote: >> >> Does anyone know of any other syntax that uses a similar method of >> continuation? > IDL continuations are similar. > > ************************************************************************ ************************************************************************ From: Mark Calabretta To: William Pence CC: fitsbits@nrao.edu Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period Date: Fri, 13 Jul 2007 11:21:04 +1000 On Thu 2007/07/12 16:10:24 -0400, William Pence wrote in a message to: fitsbits@nrao.edu >Another case where ordering may be implicit is in a sequence of HISTORY >keywords that document the order of data processing steps. In some Another obvious example is that of COMMENT strings that span successive records and are meant to be read as a paragraph. An implicit example is where two header cards define the same keyword (this condition is not addressed by the current standard, though I note that the new draft specifically deprecates it). Since scanner/parsers are likely to read the header top-to-bottom the most likely result will be that the last occurrence prevails. >Keep in mind that the approval process for adding a convention to the >Registry is mainly only designed to ensure that the documentation is >clear and complete. If a continuation convention were to be considered Not withstanding that (and contrary to my normal practice), I'd like to make a couple of points: 1) The continuation character, '&', is redundant syntax. As described in the prologue of fitshdr.h (from WCSLIB, as appended), it only indicates continuation if the following card is CONTINUE otherwise it must be interpreted literally. 2) Making the continuation character part of the keyvalue, rather than a separate token, means that: a) Continuation is only defined for string keyvalues. b) When the parser tokenizes a header card, instead of being a separate token in itself the (redundant) continuation character must be removed from one of the other tokens. Consequently, the parser is obliged to modify the keyvalue. Does anyone know of any other syntax that uses a similar method of continuation? As alternatives think about the way it's done in Fortran, C/C++, and Bourne shell (or python, or Glish). Cobol is also quite interesting in this respect as it's card-oriented, so is relevant to FITS, and has elements of both C and Fortran continuation. Mark Calabretta ATNF >>> Extract from the prologue of fitshdr.h: >>> * 2) The parser supports a generalization of the OGIP Long String Keyvalue * Convention (v1.0) whereby strings may be continued onto successive * header cards. A card contains a segment of a continued string if and * only if * a) it contains the pseudo-keyword "CONTINUE", * b) columns 9 and 10 are both blank, * c) columns 11 to 80 contain what would be considered a valid string * keyvalue, including optional inline comment, if column 9 had * contained '=', * d) the previous card contained either a valid string keyvalue or a * valid CONTINUE card. * If any of these conditions is violated, the card is considered in * isolation. * * Syntax errors in inline comments in a continued string are treated * more permissively than usual; the '/' delimiter may be omitted * provided that parsing of the string keyvalue is not compromised. * However, the FITSHDR_COMMENT status bit will be set for the card (see * below). * * As for normal strings, trailing blanks in a continued string are not * significant. * * In the OGIP convention "the '&' character is used as the last non- * blank character of the string to indicate that the string is * (probably) continued on the following keyword". This additional * syntax is not required by fitshdr(), but if '&' does occur as the last * non-blank character of a continued string keyvalue then it will be * removed, along with any trailing blanks. However, blanks that occur * before the '&' will be preserved. ************************************************************************ ************************************************************************ From: Mark Calabretta To: Tom McGlynn Date: Fri, 13 Jul 2007 11:59:33 +1000 Cc: fitsbits@nrao.edu Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period On Thu 2007/07/12 21:39:13 -0400, Tom McGlynn wrote in a message to: Tom McGlynn and copied to: fitsbits@nrao.edu >>> Does anyone know of any other syntax that uses a similar method of >>> continuation? >> IDL continuations are similar. IDL provides yet another variant to add to the mix. However, syntactically it is quit different: 1) '$' must occur as the last character on the line. 2) It is a separate token, not part of another. 3) It is not redundant; when '$' occurs as the last character the continuation is unconditional, because... 4) ...the continued line does not contain additional syntax. Mark Calabretta ATNF ************************************************************************ ************************************************************************ Date: Fri, 13 Jul 2007 13:36:33 -0400 From: Doug Mink To: FITSBITS Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period William Pence wrote: > This is to announce the start of the 30-day Public Comment Period on > the CONTINUE keyword convention that has been submitted for inclusion in > the 'Registry of FITS Conventions' which is maintained by the IAU FITS > Working Group. As no one has mentioned it yet, I would like to point out that there is a huge amount of optical data in FITS files which use the IRAF multi-keyword continuation method. A 1-4 letter keyword is followed by an underscore and three-digit sequence number. The way I deal with them in WCSTools is to look for the base keyword and if it is not found look for _001. If that is found, I look for _002 and so forth, appending each string to the previous one, until a keyword is not found. An advantage is that the keywords do not have to be contiguous or even in the correct order because the order is documented in the keyword name. A typical usage is the storage of polynomial projection coefficients for the IRAF TNX world coordinate system (tangent plane + polynomial correction) as shown below. Support for reading this projection is included in WCSTools. -Doug Mink . . . WAT0_001= 'system=image' WAT1_001= 'wtype=tnx axtype=ra lngcor = "3. 4. 4. 2. -0.05837385199330483 0.070' WAT2_001= 'wtype=tnx axtype=dec latcor = "3. 4. 4. 2. -0.05837385199330483 0.07' TRIM = 'Jul 25 21:28 Trim data section is [1:2047,1:4093]' OVERSCAN= 'Jul 25 21:28 Overscan section is [2054:2100,100:4100] with mean=1064' CCDSEC = '[1:2047,1:4093]' CCDMEANT= 775260912 CCDPROC = 'Jul 25 22:15 CCD processing done' ZEROCOR = 'Jul 25 22:15 Zero level correction image is z1.fits' RADECSYS= 'FK5 ' CTYPE1 = 'RA---TNX' CTYPE2 = 'DEC--TNX' CRVAL1 = 53.3071726192818 CRVAL2 = -27.7173865733127 CRPIX1 = 1033.73520722181 CRPIX2 = 1673.69444794591 CD1_1 = 5.61383748782434E-5 CD1_2 = -1.5386025841724E-7 CD2_1 = 3.38555565395686E-7 CD2_2 = 5.59724101379682E-5 WAT1_002= '44519483406535 -0.09397437792791556 0.1451290667953512 -4.3273459822' WAT1_003= '97552E-5 4.525801441223172E-4 0.009912907609377613 0.185709640211661' WAT1_004= '6 1.874028266281251E-5 0.01797451858969125 0.04459426127294047 0.010' WAT1_005= '87988458562529 -0.1525431073290788 -0.00297118661606969 "' WAT2_002= '044519483406535 -0.09397437792791556 0.1451290667953512 -4.808546205' WAT2_003= '327283E-5 4.148865918405720E-4 0.01180811673900729 0.087140524550311' WAT2_004= '24 -9.417303440113919E-4 0.03105863709218507 0.3049773562603359 0.01' WAT2_005= '469707839868787 0.0440181026974359 0.1257086385879256 "' . . . ************************************************************************ ************************************************************************ Date: Fri, 13 Jul 2007 16:03:28 -0400 From: William Thompson To: FITSBITS Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period I'd like to make a couple of points about the CONTINUE convention. First, I recommend that wording be added to disallow the usage of the convention for any keyword relating to the physical organization of data within a FITS file. For example, one should not be able to continue keywords such as "TDIMnnn" or "EXTNAME". Second, I'd like to make some comments about keyword ordering. In solar physics it's common to convert FITS headers into structures upon reading. For example, the simple FITS header SIMPLE = T /Written by IDL: Fri Jul 13 15:52:15 2007 BITPIX = -32 /Real*4 (floating point) NAXIS = 2 / NAXIS1 = 100 / NAXIS2 = 100 / END would be converted into a structure with tag elements SIMPLE INT 1 BITPIX LONG -32 NAXIS LONG 2 NAXIS1 LONG 100 NAXIS2 LONG 100 In many cases, the arrangement of the structure is predetermined, and the FITS header is mined to extract the keywords corresponding to the desired parameters. In such cases, the original order of the keywords within the header is lost. I would envision that continuation lines would best be handled by the FITS reader, so that the resulting structure tag value would be the complete string. In that case, the question of the original ordering of the keyword would be moot. However, I doubt that any software would actually be changed to implement this until there was a real need for it. Bill Thompson ************************************************************************ ************************************************************************ From: Craig Markwardt Date: 14 Jul 2007 21:46:12 -0400 To: fitsbits@donar.cv.nrao.edu Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period Rob Seaman writes: > > The CONTINUE convention, however, is different in providing a more > > general capability that could be used, for instance, within any > > other convention. > > I don't think I emphasized this point enough. A continuation > convention, whether the one on the table or any other, changes the > meaning of other conventions. One could, for instance, recast a > CHECKSUM keyword: > > CHECKSUM= 'nADTnADQnADQnADQ' / ASCII 1's complement checksum > > as: > > CHECKSUM= 'nADTnADQ& ' / ASCII 1's complement checksum > CONTINUE 'nADQnADQ' > > This would apply to your favorite string valued keyword as well. The > example above would be a legal continuation but an illegal checksum > value. (More than illegal for this example since the byte ordering > needs to be preserved for the zeroing feature of the checksum to work.) Bill, could the convention be clarified to indicate that CONTINUE should not be used for keyword values that could fit in a single FITS card? And/or, that CONTINUE should be avoided for standard FITS keywords? Craig ************************************************************************ ************************************************************************ From: "LC's NoSpam Newsreading account" Newsgroups: sci.astro.fits Date: Fri, 13 Jul 2007 11:28:24 +0200 To: fitsbits@donar.cv.nrao.edu Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period Before I disappear, I give some brief comments : I think the material supplied to document the convention is clear enough (and the convention is sound and simple, although I remind everybody that inclusion in the Registry should NOT imply a discussion on the merit of a convention, but only on completeness of documentation, usefulness and actual use at one or more institution). I do not think the issue of keyword ordering and repetition of a keyword with same name is an issue, considering the proposed modifications to the Standard, and the nature of "commentary" keyword of CONTINUE. I am also pleased by the fact that this convention can be clearly spotted by a reader because of the presence of the "magic" string LONGSTRN= 'OGIP 1.0' So my only request for change is very formal. The explanatory document http://fits.gsfc.nasa.gov/registry/continue/continue_keyword.html is correctly titled "CONTINUE *Long String* Keyword Convention" I suggest this title (including the words *Long String*) be used even in higher level pages in the Registry instead of the shorter "CONTINUE Keyword Convention". The convention is clearly documented to deal with long STRINGS "only" (not numeric valued keywords which somebody mentioned), and is not characterized *just by the presence* of the CONTINUE keywords. But the specific ways it uses the CONTINUE keywords are flagged by LONGSTRN= 'OGIP 1.0'. So the proper name would be "CONTINUE *Long String* Keyword Convention" or "OGIP 1.0 *Long String* Keyword Convention" (assuming one could develop e.g. a LONGSTRN= 'Erewhon University 7.3' which uses a different continuation mechanism) Lucio Chiappetti - INAF IASF Milano ************************************************************************ ************************************************************************ From: Rob Seaman Date: Sun, 15 Jul 2007 21:42:26 -0700 To: fitsbits@donar.cv.nrao.edu Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period On Jul 14, 2007, at 6:46 PM, Craig Markwardt wrote: > Bill, could the convention be clarified to indicate that CONTINUE > should not be used for keyword values that could fit in a single > FITS card? I'm not sure how one ensures this given that a continued keyword might later be edited. > And/or, that CONTINUE should be avoided for standard FITS keywords? Which are the "standard" keywords - the ones mentioned in the standard? My example was a keyword from another convention. On Jul 13, 2007, at 2:28 AM, LC's NoSpam Newsreading account wrote: > I remind everybody that inclusion in the Registry should NOT imply > a discussion on the merit of a convention, but only on completeness > of documentation, usefulness and actual use at one or more > institution). Well, every previous convention has also generated a rousing discussion of the merits of the proposal. This seems rather healthy. I'm also not sure what the distinction is between "merit" and "usefulness". Presumably we are in a period of capturing and documenting as many local conventions as possible. The bigger question is what comes next. Some (I'll timidly suggest the checksum convention) seem mature enough to become part of the standard. Others, as with this current discussion, appear to have some real issues with broad adoption. Still others, like foreign encapsulation perhaps, might simply appear as some sort of appendix since a particular HDU either expresses such an object or does not. I guess I'm also wondering whether the point of the registry is to capture legacy conventions or to encourage new conventions. Personally, I would prefer that the staid, but reliable, FITS standard process remain the primary focus of FITS development efforts. We don't need novelty for its own sake. > I do not think the issue of keyword ordering and repetition of a > keyword with same name is an issue, considering the proposed > modifications to the Standard, I'm getting lost with this discussion. What proposed modifications are we talking about? > So the proper name would be "CONTINUE *Long String* Keyword > Convention" or "OGIP 1.0 *Long String* Keyword Convention" I disagree. Continuation is only one possible way to implement a long string capability. To be truly useful, a general facility for representing long strings needs to be applicable to any string valued keyword. On the other hand, continuing a keyword appears to have some real complications - some esoteric, like the need to preserve 32- bit alignment within ASCII encoded checksum strings - others more fundamental, such as ambiguous keyword ordering restrictions. But, if we're not to discuss merit, that drops us to two criteria for registration: documentation and prior use. Like all the other conventions to date, this one appears to have been actually used in some large or small number of files. The documentation seems too minimal and unconstraining, however, as expressed in the previous messages in this thread. Rob ************************************************************************ ************************************************************************ Date: Tue, 17 Jul 2007 16:17:06 -0400 From: William Pence To: FITSBITS Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period A disk crash here over the weekend wiped out all the files related to the CONTINUE long string keyword convention on the FITS Support Office web site, so I've had to recreate them. In the process, I've revised the documentation about this convention to try to address all the comments that have been received so far. Please see the new PDF (or postscript) documentation file available from the FITS Registry at http://fits.gsfc.nasa.gov/fits_registry.html. Here are a few more comments in response to recent postings: Rob Seaman wrote: > 1) Couldn't the reordering question be dealt with via appending &NNN > instead of &? Many different way of explicitly encoding the order of the continuation keywords were suggested back in 1993-4, but the HEASARC decided that this added complexity was not needed for it's applications. I don't recall a single instance in the past 13 years where users encountered problems with the keywords getting out of order. > > 2) What is the reason for not strictly requiring that such a > continuation string end with ampersand-quote? What advantage is > conveyed by permitting semantically null trailing whitespace that > merely confuses the handling? This was probably influenced by the FITS Standard, which says trailing spaces in keyword string values are not significant. > One could, for instance, recast a > CHECKSUM keyword: > > CHECKSUM= 'nADTnADQnADQnADQ' / ASCII 1's complement checksum > > as: > > CHECKSUM= 'nADTnADQ& ' / ASCII 1's complement checksum > CONTINUE 'nADQnADQ' > > This would apply to your favorite string valued keyword as well. The > example above would be a legal continuation but an illegal checksum > value. (More than illegal for this example since the byte ordering > needs to be preserved for the zeroing feature of the checksum to work.) I may be wrong, but I don't really see any technical problem with writing the CHECKSUM keyword as a continuation over 2 (or more) keywords. The algorithm for initializing the header and calculating the CHECKSUM keyword value(s) would need to be modified slightly, but it would still follow the same basic procedure. (Not to suggest that there is any reason to do this of course). Mark Calabretta wrote: > 1) The continuation character, '&', is redundant syntax. As described > in the prologue of fitshdr.h (from WCSLIB, as appended), it only > indicates continuation if the following card is CONTINUE otherwise > it must be interpreted literally. The redundancy is intentional in this case, and helps to avoid any possible confusion over whether the FITS writer really intended this convention to be used or not. > 2) Making the continuation character part of the keyvalue, rather than > a separate token, means that: > > a) Continuation is only defined for string keyvalues. Yes, because we assumed a single keyword is adequate for recording integer, logical, or floating point values, so continuations would not be needed in these cases. > b) When the parser tokenizes a header card, instead of being a > separate token in itself the (redundant) continuation character > must be removed from one of the other tokens. Consequently, the > parser is obliged to modify the keyvalue. It seemed safer and more convenient to embedded the token in the string, rather than perhaps as part of the comment field of the keyword. The token is less likely to get separated or inadvertently stripped off this way. Many FITS readers ignore the keyword comments, so putting vital information there is perhaps not a good idea. Doug Mink wrote: > As no one has mentioned it yet, I would like to point out that there is a > huge amount of optical data in FITS files which use the IRAF multi-keyword > continuation method. A 1-4 letter keyword is followed by an underscore and > three-digit sequence number. > . . . > WAT0_001= 'system=image' > WAT1_001= 'wtype=tnx axtype=ra lngcor = "3. 4. 4. 2. -0.05837385199330483 0.070' > WAT2_001= 'wtype=tnx axtype=dec latcor = "3. 4. 4. 2. -0.058373851993304 > TRIM = 'Jul 25 21:28 Trim data section is [1:2047,1:4093]' > OVERSCAN= 'Jul 25 21:28 Overscan section is [2054:2100,100:4100] with mean=1064' > WAT1_002= '44519483406535 -0.09397437792791556 0.1451290667953512 -4.3273459822' > WAT1_003= '97552E-5 4.525801441223172E-4 0.009912907609377613 0.185709640211661' > WAT1_004= '6 1.874028266281251E-5 0.01797451858969125 0.04459426127294047 0.010' > WAT1_005= '87988458562529 -0.1525431073290788 -0.00297118661606969 "' > WAT2_002= '044519483406535 -0.09397437792791556 0.1451290667953512 -4.808546205' > WAT2_003= '327283E-5 4.148865918405720E-4 0.01180811673900729 0.087140524550311' > WAT2_004= '24 -9.417303440113919E-4 0.03105863709218507 0.3049773562603359 0.01' > WAT2_005= '469707839868787 0.0440181026974359 0.1257086385879256 "' > . . . The main limitation with this of course is that the root keyword names can only be up to 4 characters long. The HEASARC needed a convention that could be used with keywords that describe table columns and hence would have a 1 to 3 digit column number following the root name. William Thompson wrote: > Second, I'd like to make some comments about keyword ordering. In solar physics > it's common to convert FITS headers into structures upon reading. ... > In many cases, the arrangement of the structure is predetermined, and the FITS > header is mined to extract the keywords corresponding to the desired parameters. > In such cases, the original order of the keywords within the header is lost. ... > I would envision that continuation lines would best be handled by the FITS > reader, so that the resulting structure tag value would be the complete string. > In that case, the question of the original ordering of the keyword would be > moot. However, I doubt that any software would actually be changed to implement > this until there was a real need for it. I agree. As long as the structure is capable of holding the full length of the long string keyword value, then the original order of the FITS keywords should not need to be preserved. Rob Seaman wrote: > Presumably we are in a period of capturing and documenting as many local > conventions as possible. The bigger question is what comes next. Some > (I'll timidly suggest the checksum convention) seem mature enough to > become part of the standard. Others, as with this current discussion, > appear to have some real issues with broad adoption. Still others, like > foreign encapsulation perhaps, might simply appear as some sort of > appendix since a particular HDU either expresses such an object or does not. At the moment, most of my effort is going into just documenting the existing conventions. At the current rate, we will probably have captured most of the common conventions (unless there are many more lurking around) within the next year. After that might be a good time to consider whether we want to promote, or further develop, some of the conventions and make them standards. > I guess I'm also wondering whether the point of the registry is to > capture legacy conventions or to encourage new conventions. The rules for what is appropriate for the current Registry state: "Generally speaking, proposed new FITS conventions that are not yet in actual use should not be submitted to the Registry until the proposal is fairly mature and has demonstratable support from at least a significant subset of the FITS user community." Finally, if anyone has read this far and is still interested in further background information about why this particular continuation convention was chosen, here is a copy of a FITSBITS posting made back in 1994: ------------------------------------------------------------------ The OGIP FITS Working Group met recently to decide how to handle long string keyword values in FITS files. We urgently need such a convention for our soon to be launched mission, so given that there is currently no consensus within the FITS community on a standard way to deal with this problem, we have adopted a local FITS convention for use within our own processing environment. Since this is not an agreed upon standard, we will not use this convention for any keywords that we would expect any external software systems would need to read or interpret. The following example illustrates the convention that we have adopted: BIGSTRNG= 'This is a long string value that is continued &' CONTINUE 'over three keywords in the& ' CONTINUE ' FITS header.' The '&' character is used as the last non-blank character of the string to indicate that the string is (probably) continued on the following keyword (if the next keyword name = 'CONTINUE' as described below). We adopted the '&' character instead of the '\' that had been previously proposed because of Barry Schlesinger's observation that the '\' is one of the 10 'National Use' positions which may not be interpreted as the same symbol by all nationalities. Also, some Fortran compilers treat the '\' character in a non-standard way making it difficult to write machine-independent code that uses this character. The '&' seems to be a reasonable alternative, especially since it is used as a continuation character in Fortran 90. Each continuation line has the keyword name = 'CONTINUE'. We adopted this keyword name, rather than the previous suggestion of an all blank keyword name, in order to provide an unambiguous way of identifying any continuation keywords even if they get separated from their parent keyword, and to try to avoid conflicts with other FITS software that may treat keywords with an all-blank name in a unique or special way. Since there is no equal sign in column 9, this keyword belongs to the same class of keywords as the COMMENT and HISTORY keywords that do not have a defined value. Under our convention the continuation of the character string value is enclosed in quotes starting in column 11 (or higher) of the 80-character record. Any other characters (e.g. a comment field) may optionally follow the closing quote character on any or all of the keywords. We also considered, but decided against, implementing a convention that would explicitly link together all the keywords so that the correct continuation sequence could be reconstructed even if the order of the keywords in the header were rearranged. (Several such schemes were suggested here on FITSBITS in the past month). Since there is even less consensus on the best way to implement such a scheme, or if it is even needed, we decided to simply rely on the physical order of the keywords in the header, which meets our own needs. (We did confirm with Barry Schlesinger that this reliance on the order of the keywords in the header does conform to the FITS Standard and hence is legal to use in a local FITS convention). If the FITS community later decides that a full forward and backward keyword linking capability is required to make this convention acceptable as a Standard FITS convention, then this can in principle be added with only minor modifications to our convention. -------------------------------------------------------------------- Bill Pence -- ************************************************************************ ************************************************************************ From: Rob Seaman Date: Tue, 17 Jul 2007 14:24:43 -0700 To: FITSBITS Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period Hi Bill, > I've revised the documentation about this convention to try to > address all the comments that have been received so far. Please > see the new PDF (or postscript) documentation file available from > the FITS Registry at > http://fits.gsfc.nasa.gov/fits_registry.html. Certainly an improvement. If we can agree that a broader discussion of implementing long string support within the standard will follow, I withdraw any objections from this end to the convention being registered. > I don't really see any technical problem with writing the CHECKSUM > keyword as a continuation over 2 (or more) keywords. The algorithm > for initializing the header and calculating the CHECKSUM keyword > value(s) would need to be modified slightly, but it would still > follow the same basic procedure. Indeed, the FITS checksum encoding could accommodate such a complication, but both writers (or generalized keyword editors) and readers would have to know about the 32-bit alignment requirements for CHECKSUM keywords. To modify my previous example a bit, this keyword: CHECKSUM= 'nADTnADQnADQnADQ' / ASCII 1's complement checksum is not the same as this expression of overtly the same value: CHECKSUM= 'nADTnADQn& ' / ASCII 1's complement checksum CONTINUE 'ADQnADQ' The 32-bit integer alignment is different, breaking the 1's complement legerdemain. Other keywords won't have this peculiar requirement, but additional semantic or logistical complications can be anticipated. - Rob ************************************************************************ ************************************************************************ From: Mark Calabretta To: William Pence Date: Thu, 19 Jul 2007 11:48:59 +1000 Cc: FITSBITS Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period On Tue 2007/07/17 16:17:06 -0400, William Pence wrote in a message to: FITSBITS >Mark Calabretta wrote: >> 1) The continuation character, '&', is redundant syntax. As described >> in the prologue of fitshdr.h (from WCSLIB, as appended), it only >> indicates continuation if the following card is CONTINUE otherwise >> it must be interpreted literally. > >The redundancy is intentional in this case, and helps to avoid any >possible confusion over whether the FITS writer really intended this >convention to be used or not. Bill, The CONTINUE convention differs significantly from the others offered for the registry because it defines basic functionality that people are likely to want to use. It will almost inevitably become a de facto standard, if it hasn't already, or at least strongly influence the way that continuation syntax might be standardised. Consequently, I think it is worth devoting some effort to settling on a syntax that we all feel comfortable with. My only concern with the convention as currently described is with the use of the '&' character, which, to reiterate, is redundant syntax. However, I don't advocate eliminating it. Instead I suggest making it optional in precisely the way described in the prologue of fitshdr.h (previously appended). In practice it can still serve the useful function of "guarding" trailing blanks that are to be preserved in a string value. The prologue of fitshdr.h describes how this form of CONTINUE-based continuation works in a parser that has been implemented. Tying continuation to a particular data type, apart from being unnecessary, must be unique amongst computer-based syntaxes. The argument that only string values are likely to be continued ignores possible future syntaxes. For example, record-valued keywords currently proposed by/for WCS Paper IV might be better implemented by extending the keyword syntax. They could easily be long enough to require continuation, and the continued portion could be a floating point value or something else. Against objections that have been raised relating to keyword ordering, it should be borne in mind that once a header has been parsed into a data structure it is largely inconsequential that the original ordering might be lost; if the data structure is later retranslated into a FITS header the keywords might appear in a different order (and with different inline comments, decimal precision, etc.), but the CONTINUE keywords would still have to be ordered correctly in the output header. The same applies to COMMENT, HISTORY, or other order-dependent keywords. Order must, and surely will be, preserved if and only if it matters. Also, in response to recent comments, I should point out that Paper IV advocates that the same record-valued keyword appear multiple times with different keyvalues (and that the ordering is not significant). Sect. 2.4.2 explains why. Regards, Mark ************************************************************************ ************************************************************************ Date: Thu, 19 Jul 2007 15:00:37 -0400 From: William Pence To: FITSBITS Subject: Re: [fitsbits] Start of the CONTINUE keyword Public Comment Period Hi Mark, Mark Calabretta wrote: > On Tue 2007/07/17 16:17:06 -0400, William Pence wrote > in a message to: FITSBITS > >> Mark Calabretta wrote: >>> 1) The continuation character, '&', is redundant syntax. As described >>> in the prologue of fitshdr.h (from WCSLIB, as appended), it only >>> indicates continuation if the following card is CONTINUE otherwise >>> it must be interpreted literally. >> The redundancy is intentional in this case, and helps to avoid any >> possible confusion over whether the FITS writer really intended this >> convention to be used or not. > > Bill, > > The CONTINUE convention differs significantly from the others offered > for the registry because it defines basic functionality that people are > likely to want to use. It will almost inevitably become a de facto > standard, if it hasn't already, or at least strongly influence the way > that continuation syntax might be standardised. Consequently, I think > it is worth devoting some effort to settling on a syntax that we all > feel comfortable with. Since this convention has been in use for ~13 years, and is in current use for at least 2 active missions (RXTE and Chandra), I'm assuming there should be no objection to documenting the current syntax of this convention in the Registry, once any remaining issues about the completeness or clarity of the documentation are resolved. Further discussion about whether some sort of continuation functionality should be incorporated into the FITS Standard is certainly welcome as far as I am concerned, but there is another somewhat related issue that should probably be considered at the same time: the 8-character limit on keyword names. These 2 limits (the keyword name length, and the length of character string keyword values) are arguably the 2 limitations of FITS that most affect (and irritate) data providers. Since both of these issues have to do with how to encode the required information into the 80-character header records, it seems to me these 2 issues should be considered together. In the near future (probably in September) the existing ESO HIERARCH keyword convention (which effectively allows longer keyword names) will be submitted to the Registry. That may provide a catalyst for wider discussion. In the meantime, here are a few comments on your previous comments: > > My only concern with the convention as currently described is with the > use of the '&' character, which, to reiterate, is redundant syntax. > However, I don't advocate eliminating it. Instead I suggest making it > optional in precisely the way described in the prologue of fitshdr.h > (previously appended). In practice it can still serve the useful > function of "guarding" trailing blanks that are to be preserved in a > string value. The prologue of fitshdr.h describes how this form of > CONTINUE-based continuation works in a parser that has been implemented. The '&' character serves 3 important functions, so I'm uneasy with making it optional: 1. The redundancy of requiring both the '&' at the end of the string followed by a CONTINUE keyword serves to prevent misinterpretation of the header values. To illustrate, consider the following keywords taken from an actual RXTE file: 1CTYP12 = 'CHANNEL ' 1CPIX12 = '(S[msLimit1]~S[msLimit2]),((S[msLimit2]+1)~S[msLimit&' CONTINUE '3]+1)~S[msLimit4]),((S[msLimit4]+1)~S[msLimit5])' If the 1CPIX12 keyword is now deleted without also deleting the following CONTINUE keyword (as could easily happen if the software does not support this convention) then you are left with this: 1CTYP12 = 'CHANNEL ' CONTINUE '3]+1)~S[msLimit4]),((S[msLimit4]+1)~S[msLimit5])' This will cause the 1CTYP12 keyword value to be misinterpreted if one does not require the '&' as part of this convention. Requiring that the first string end with an '&' makes is much less likely that an orphaned or misplaced CONTINUE keyword will cause any damage; it will simply be treated in the same way as a harmless COMMENT keyword. 2. The '&' also serves as an aid to FITS reading programs by providing an explicit indication that the keyword may be continued. Without this indicator, the reading program would always have to check the next keyword to see if it is a continuation, which could add more overhead. This may not be a particular issue for software like your wcslib parser, which passes through the whole header once, to populate an internal structure, but it is a significant issue for other software, like my CFITSIO library, which reads the keywords from the header on demand. 3. The third function of the '&', as you mentioned, is to allow trailing blanks to be included as a significant part of the string. Taken together, these 3 reasons seem compelling enough to me to require that the '&' be used as part of this convention. > Tying continuation to a particular data type, apart from being > unnecessary, must be unique amongst computer-based syntaxes. The > argument that only string values are likely to be continued ignores > possible future syntaxes. For example, record-valued keywords currently > proposed by/for WCS Paper IV might be better implemented by extending > the keyword syntax. They could easily be long enough to require > continuation, and the continued portion could be a floating point value > or something else. As currently defined, the record-valued keywords all have string values, so there shouldn't be any problem with using this continuation convention with them, if desired. I guess your point is that some other syntax might be invented in the future in which the keyword values are not strings. That may be, but without concrete examples, it is hard to see why the current convention is not adequate. Bill Pence