Date: Fri, 07 Sep 2007 11:44:00 -0400 From: Robert Hanisch To: William Pence , FITSBITS Subject: Re: [fitsbits] The CONTINUE and HIERARCH Conventions Public Comment Periods The description of the HIERARCH convention is inconsistent, I think, regarding the length of the component tokens. At the top of p. 2 it says "Under the ESO implementation of this convention, each token string which precede[s] the equals sign conforms to the requirements for a FITS keyword..." But in Section 2, this rule is abandoned and the same scheme is used for defining long and non-conforming keyword names. So, as written it is not really clear whether it is the ESO implementation that is being discussed, or the more general thing. That is, would the following conform to the convention? HIERARCH HST-ADVANCED-CAMERA APERTURE$Setting = 2 Finally, on p. 3 it says "This convention should not be used if the effective keyword name can be represented as a standard FITS keyword...". I think it can be argued that one can always find a standard FITS keyword representation, as has been done 1000s of times in FITS headers. The examples given, such as ESO.TEL.FOCU.SCALE and ESO.INS.OPTI-3.ID, would have quite obvious standard keyword values given the context of a FITS file originating from an ESO telescope and instrument, which would be documented elsewhere in the FITS header. I believe II share some responsibility for the invention of HIERARCH (I think I first suggested it at a FITS WCS meeting in Charlottesville, ca. 1989). I have regretted it at various times ever since. Bob ************************************************************************** Date: Mon, 10 Sep 2007 12:45:04 -0400 From: William Pence To: FITSBITS Subject: Re: [fitsbits] The CONTINUE and HIERARCH Conventions Public Comment Periods Robert Hanisch wrote: > The description of the HIERARCH convention is inconsistent, I think, > regarding the length of the component tokens. > > At the top of p. 2 it says > > "Under the ESO implementation of this convention, each token string which > precede[s] the equals sign conforms to the requirements for a FITS > keyword..." > > But in Section 2, this rule is abandoned and the same scheme is used for > defining long and non-conforming keyword names. So, as written it is not > really clear whether it is the ESO implementation that is being discussed, > or the more general thing. That is, would the following conform to the > convention? > > HIERARCH HST-ADVANCED-CAMERA APERTURE$Setting = 2 No, this does not conform this convention. The 2 tokens ("HST-ADVANCED-CAMERA" and "APERTURE$Setting") are not legal FITS keyword names, so this does not conform to the ESO use of the convention (although ESO bends the rules slightly, and does allow tokens longer than 8 characters). The more general HIERARCH convention that is described in the last half of the document disallows spaces in the "effective keyword name", so this example does not conform to that. If you replaced the space between the 2 tokens with an underscore, then it would conform to the more general convention: HIERARCH HST-ADVANCED-CAMERA_APERTURE$Setting = 2 > Finally, on p. 3 it says "This convention should not be used if the > effective keyword name can be represented as a standard FITS keyword...". I > think it can be argued that one can always find a standard FITS keyword > representation, as has been done 1000s of times in FITS headers. The statement you quote is intended to disallow usage like this: HIERARCH USERNAME = "John" In this case, "USERNAME" is a legal FITS keyword name, so there is no need to use the more complicated HIERARCH convention. > I believe I share some responsibility for the invention of HIERARCH (I > think I first suggested it at a FITS WCS meeting in Charlottesville, ca. > 1989). I have regretted it at various times ever since. I initially shared your dislike of the HIERARCH convention, but have changed my mind after seeing how well it has worked in the ESO FITS files. If you look at the sample FITS header listing from ESO given at http://fits.gsfc.nasa.gov/registry/hierarch_keyword.html (the fors.txt link) there are hundreds of keywords that use this convention in each file. It would be difficult to come up with clear and informative 8-character names for this many keywords. I think ESO has done a pretty good job of defining a self consistent set of hierarchical keyword names for all its dozens of telescope and instrument data systems. I agree however that it probably would not make sense to use the HIERARCH convention for just a couple keywords in a file. I don't particularly like the more generalized use of the HIERARCH convention to create keyword names that are longer than 8 characters or contain non-standard characters (even though I added support for this in the CFITSIO library). If we really want to extend FITS to allow longer keyword names, then I think a much simpler way to do this is to just allow free-format keyword records in which the "=" can appear anywhere after byte 9 of the record, such as: MY_LONG_KEYWORD = 17 / comment string Note that this keyword is perfectly legal under the current FITS Standard: existing FITS readers should interpret this as a commentary-type keyword, with the 8-character name "MY_LONG_" and the rest of the record treated as a comment string. Bill ************************************************************************** Date: Mon, 10 Sep 2007 13:25:42 -0400 From: Robert Hanisch To: William Pence , FITSBITS Subject: Re: [fitsbits] The CONTINUE and HIERARCH Conventions Public Comment Periods On 9/10/07 12:45 PM, "William Pence" wrote: > Robert Hanisch wrote: >> The description of the HIERARCH convention is inconsistent, I think, >> regarding the length of the component tokens. >> >> At the top of p. 2 it says >> >> "Under the ESO implementation of this convention, each token string which >> precede[s] the equals sign conforms to the requirements for a FITS >> keyword..." >> >> But in Section 2, this rule is abandoned and the same scheme is used for >> defining long and non-conforming keyword names. So, as written it is not >> really clear whether it is the ESO implementation that is being discussed, >> or the more general thing. That is, would the following conform to the >> convention? >> >> HIERARCH HST-ADVANCED-CAMERA APERTURE$Setting = 2 > > No, this does not conform this convention. The 2 tokens > ("HST-ADVANCED-CAMERA" and "APERTURE$Setting") are not legal FITS > keyword names, so this does not conform to the ESO use of the convention > (although ESO bends the rules slightly, and does allow tokens longer > than 8 characters). The more general HIERARCH convention that is > described in the last half of the document disallows spaces in the > "effective keyword name", so this example does not conform to that. But it does NOT disallow spaces, it just says they should be avoided in order to avoid confusion, and this was exactly the reason for my example. And if ESO bends their own rule and has tokens longer than 8 characters, then the definition of the convention is pretty sloppy. It would mean that HIERARCH DOMAINNAME SUBDOMAIN LONGKEYNAME = 2 both conforms and does not conform to the convention. To clarify things it seems this needs text that says that if a long keyword name is being specified, only one token after HIERARCH is allowed prior to the " = value ". But this then probably invalidates all those ESO files that use tokens longer than 8 characters. Despite my misgivings I am not really objecting to the convention itself, but rather noting that the description is confusing and inconsistent. Bob ************************************************************************** From: Rob Seaman Date: Mon, 10 Sep 2007 10:26:12 -0700 To: FITSBITS Subject: Re: [fitsbits] The CONTINUE and HIERARCH Conventions Public Comment Periods If the registry serves to bring order to chaos, it will be a net positive. If it serves, rather, to advertise conventions willy- nilly, it may be a net negative. I'm really nervous at the thought of random naive users including HIERARCH keywords in their documents. Bob's examples and Bill's comments demonstrate that even expert FITS users may differ on the legality of HIERARCH usage. On Sep 10, 2007, at 9:45 AM, William Pence wrote: > I initially shared your dislike of the HIERARCH convention, but have > changed my mind after seeing how well it has worked in the ESO FITS > files. If you look at the sample FITS header listing from ESO [...] > there are hundreds of keywords that use this convention in each file. One has always wondered at the project requirements that lead to including hundreds of keywords in a FITS file. > I agree however that it probably would not make sense to use the > HIERARCH > convention for just a couple keywords in a file. Can we hope to successfully legislate this? > I don't particularly like the more generalized use of the HIERARCH > convention to create keyword names that are longer than 8 > characters or > contain non-standard characters (even though I added support for > this in > the CFITSIO library). Me neither! > If we really want to extend FITS to allow longer > keyword names, then I think a much simpler way to do this is to just > allow free-format keyword records in which the "=" can appear anywhere > after byte 9 of the record, such as: > > MY_LONG_KEYWORD = 17 / comment string Bleh! No, no, no. Talk about needing to version FITS... > Note that this keyword is perfectly legal under the current FITS > Standard: existing FITS readers should interpret this as a > commentary-type keyword, with the 8-character name "MY_LONG_" > and the rest of the record treated as a comment string. I don't think we've ever tested this in anger. A lot of FITS applications likely still assume that the only commentary keywords are COMMENT, HISTORY and . We just had an issue with a technically legal keyword throwing an error due to an extraneous equals sign in column nine. If we want to address the issue of including elaborate header metadata in FITS, I suspect I wouldn't be alone in preferring a bintable based solution, perhaps located as the last HDU at the end of each file (where lots of folks wanted the header in the first place). This, too, is already perfectly legal FITS. Rob ************************************************************************** Date: Mon, 10 Sep 2007 15:34:35 -0400 From: William Pence To: FITSBITS Subject: Re: [fitsbits] The CONTINUE and HIERARCH Conventions Public Comment Periods Robert Hanisch wrote: >> The more general HIERARCH convention that is >> described in the last half of the document disallows spaces in the >> "effective keyword name", so this example does not conform to that. > > But it does NOT disallow spaces, it just says they should be avoided in > order to avoid confusion, and this was exactly the reason for my example. ... > To clarify things it seems this needs text that says that if a long keyword > name is being specified, only one token after HIERARCH is allowed prior to > the " = value ". I agree and will change the document to include that wording to clarify the convention. Of course, it would still be perfectly legal in FITS to have a keyword like your example: HIERARCH HST-ADVANCED-CAMERA APERTURE$Setting = 2 That keyword would not conform to THIS convention, as defined and used by ESO, but that doesn't prevent anyone from defining their own variation on this convention in which this usage could be allowed. Bill ************************************************************************** Date: Mon, 10 Sep 2007 17:29:46 -0400 From: William Pence To: FITSBITS Subject: Re: [fitsbits] The CONTINUE and HIERARCH Conventions Public Comment Periods Rob Seaman wrote: >> If we really want to extend FITS to allow longer >> keyword names, then I think a much simpler way to do this is to just >> allow free-format keyword records in which the "=" can appear anywhere >> after byte 9 of the record, such as: >> >> MY_LONG_KEYWORD = 17 / comment string > > Bleh! No, no, no. Talk about needing to version FITS... Why would versioning be necessary if this were to be adopted (not that this is being seriously proposed)? I would think that adding a version keyword to the new FITS files would only be necessary if there were existing FITS files that would be invalidated, or potentially misinterpreted, if read using the new free-format keyword parsing rules. >> Note that this keyword is perfectly legal under the current FITS >> Standard: existing FITS readers should interpret this as a >> commentary-type keyword, with the 8-character name "MY_LONG_" >> and the rest of the record treated as a comment string. > > I don't think we've ever tested this in anger. A lot of FITS > applications likely still assume that the only commentary keywords > are COMMENT, HISTORY and . Hopefully the FITS Standard is clear that if a keyword does not have a "= " value indicator in bytes 9 and 10 of the record, then the keyword does not have a formal value. The new wording in section 4.1.2.3 of the proposed new FITS Standard is: "In keyword records without a value indicator, bytes 9 through 80 should be interpreted as commentary text, however, this does not preclude conventions that interpret the contents of these bytes in other ways". The HIERARCH and CONTINUE keyword conventions are examples of this. As an important reminder, this proposed change in the wording of the FITS Standard, along with hundreds of other changes, are all available for review and public comment at http://fits.gsfc.nasa.gov/fits_draft.html until the end of this month (September). I would like to strongly encourage members of the FITS community to carefully read the entire new proposed Standard, and not just the short summary documents that the technical panel prepared, to double check that all of the proposed changes are clear and that they do not inadvertently introduce undesirable changes or ambiguities. Bill Pence ************************************************************************** Date: Mon, 10 Sep 2007 23:18:20 -0400 From: Don Wells Cc: FITSBITS Subject: Re: [fitsbits] The CONTINUE and HIERARCH Conventions Public Comment Periods Robert Hanisch wrote: > ..I believe II share some responsibility for the invention of HIERARCH (I > think I first suggested it at a FITS WCS meeting in Charlottesville, ca. > 1989).. I didn't remember who suggested the keyword HEIRARCH. I am not surprised to learn that it was you, Bob, and I am not surprised that the name HEIRARCH originated at that meeting. My memory is that the main purpose of that meeting (January 1989?) in Charlottesville was to discuss what we now call World Coordinate Systems, but we discussed many other ideas for FITS. However, there is more to say about the origin of the HEIRARCH idea. The basic idea of a heirarchical namespace for FITS is ten years older; it dates from the time of the first FITS files in 1979, when Eric Greisen prototyped the idea in NRAO production software for VLA mapping. Two years later Greisen's implementation was illustrated in the original Wells-Greisen-Harten FITS paper in A&A June-1981, in Figure-1 on p.370: INSTRUME= ‘VLA ' / NRAO(CV) VLA MAPPING.PROGRAMS DATE-MAP= ‘16/10/78’ / MAP CREATION DATE DD/MM/YY DATE = ‘27/05/79’ / MAP WRITiNG DATE DD/Vf/YY ORIGIN = 'NRAO(CV) PGM=DEC2FITS(V1)’ HISTORY VLACV MAP METHOD=’FFT’ DATA=’OBS. VISIBILITY’ HISTORY VLACV MAP WCONU= O.00000E+OO WCONV= O.000000E+O0 TCONU= 0 Greisen used HISTORY in his implementations of the idea, as shown in the last two header lines above. His implementation allowed the possibility of more than one keyword=value on a heirarchical keyword line. What we have in the example above are internal variables and options for an imaging software package called VLACV. (This was before the AIPS project; the code was batch software running on an IBM mainframe; note the DATE keyword values.) The keywords are for the 'MAP' sub-namespace of the 'VLACV' package. Anyone who has ever looked at a header written by NRAO's AIPS package in the past 25 years knows that Greisen later used the same scheme for AIPS. I have always regretted that Eric Greisen, Ron Harten and I did not include text in the original FITS paper that would have discussed those two header lines in Figure-1. I think that we should have advocated use of the heirarchical namespace idea by other astronomy groups. -Don ************************************************************************** Date: Tue, 11 Sep 2007 10:21:09 +0200 From: Walter Jaffe To: fitsbits@nrao.edu Subject: [fitsbits] Fwd: Re: The CONTINUE and HIERARCH Conventions Public Comment Periods Just for those who don't know the background of the HIERARCH keywords: ESO stores the 'state' of all its instrument subsystems in a tree-form, hierarchal database, with nodes, branches, leaves etc. They originally bought this database, including hardware, from HP, but meanwhile have implemented it themselves in a UNIX form. When recording the output data, as FITS images or tables, it is convenient for them to describe the status of the instrument by a more or less literal dump of the data base into the primary header, hence the HIERARCH keyword structure. The system programmers have some choice in which nodes they want to dump, but most of them are dumped automatically. The internal ESO database does not restrict keywords, formats etc to FITS standards, so if the system designer is not careful, you get conflicts. ESO discipline in this regard is not tight. My personal view of this is that if you want to record a database, be it relational, hierarchic, or whatever, you should choose a format well adapted to the database format. If you want to dump a relational database, use FITS tables; if you want to store a tree-base, invent a structure that represents this well and register this convention rather than misuse the FITS KEYWORD conventions. Walter ************************************************************************** From: "LC's NoSpam Newsreading account" Date: Tue, 18 Sep 2007 12:39:48 +0200 To: fitsbits@donar.cv.nrao.edu Subject: Re: [fitsbits] The CONTINUE and HIERARCH Conventions Public Comment Periods I have inspected the PDF description of the HIERARCH Convention in the registry and I have the following questions/comments. 0) Who is the author (or authority) which issued such document ? 1) I was of course aware of the usage of the HIERARCH convention at ESO and I think section 1 of the document provides an adequate documentation of the generalized hierarchical convention while leaving ESO specific details to the (quoted) DICB 2) The key characteristics of the general convention described in section 1 is that the first token defines a namespace. So any further details for namespace ESO are correctly referred to ESO documentation. Did anybody else use their own namespaces ? And who is the authority to prevent conflicts in the creation of namespaces ? 3) Although defined in a self-consisent manner, the content of section 2 is DEFINING AN ALTOGETHER DIFFERENT CONVENTION ! In section 2 there is no namespace, or any token is a namespace of its own. This seems to me a rather bad and confusing idea. It has nothing to with an "hierarchical" organization Unless such usage is already in widespread diffusion, I think we should try to stop it, and replace it with some cleaner alternative - define a specific namespace for long keywords e.g. one of HIERARCH LONGKWDS anysinglelongtoken = value / comment HIERARCH LONG anysinglelongtoken = value / comment (although this is a misuse of the name and a little space waste) - define an altogether new convention (with the same syntax of HIERARCH but eventually specifying there is a single token, and with the keyword name itself replaced by something else) LONGKWDS anysinglelongtoken = value / comment i.e. the 8-char kwd name is LONGKWDS, char 9 is blank so it is formally a commentary type keyword, and for the rest as for HIERARCH but ntoken=1 My idea is that we should register section 1 only as description of HIERARCH and perhaps register a separate LONGKWDS convention somehow replacing section 2. ************************************************************************** From: Andreas Wicenec Date: Wed, 19 Sep 2007 09:44:14 +0200 To: fitsbits@donar.cv.nrao.edu Subject: Re: [fitsbits] The CONTINUE and HIERARCH Conventions Public Comment Periods I fully agree with the points given here by [previous post] whoever that was... Andreas Wicenec ************************************************************************** Date: Thu, 20 Sep 2007 18:42:15 -0400 From: William Pence To: fitsbits@donar.cv.nrao.edu Subject: Re: [fitsbits] The CONTINUE and HIERARCH Conventions Public Comment Periods LC's NoSpam Newsreading account wrote: > I have inspected the PDF description of the HIERARCH Convention in the > registry and I have the following questions/comments. ... > > 3) Although defined in a self-consisent manner, the content of > section 2 is DEFINING AN ALTOGETHER DIFFERENT CONVENTION ! Both of these (the ESO hierarchical usage, and the long keyword name usage) are special cases of the more general convention that has been implemented in CFITSIO since 1999. The general convention looks like this: HIERARCH = value / Comment String where represents any string of ASCII text characters (except the equal sign character which is not allowed because it serves as the delimiter between the Effective Keyword Name and the value. Examples of this general convention are: HIERARCH Last Name = 'Pence' HIERARCH $PATH = '/usr/local/bin /usr/bin/' HIERARCH Minimum Disk Space Requirement = 3000000 / bytes In this general case there are no restrictions on what characters are allowed in the Effective Keyword Name, except for the equal sign character, and that it must fit within the 80-character keyword record. In the CFITSIO API, programs can read and write keywords such as "Last Name" or "$PATH" in exactly the same way as they would read or write a standard keyword like "OBJECT" or "DATE". The application program itself does not need to know how the HIERARCH convention works. In the ESO special case, the Effective Keyword Name consists of a series of tokens that each conform to the requirements of a FITS keyword name. The first token defines the name space, and the remaining tokens form a hierarchical classification of the keyword. In the long keyword name special case, embedded spaces are not allowed to avoid confusion with the ESO usage, and to avoid problems that can arise in handling the embedded spaces in certain circumstances (e.g., it can make it more difficult to parse the keyword record). Bill ************************************************************************** From: "LC's NoSpam Newsreading account" Date: Fri, 21 Sep 2007 09:12:59 +0200 To: fitsbits@donar.cv.nrao.edu Subject: Re: [fitsbits] The CONTINUE and HIERARCH Conventions Public Comment Periods On Thu, 20 Sep 2007, William Pence wrote: > Both of these (the ESO hierarchical usage, and the long keyword name usage) > are special cases of the more general convention that has been implemented in > CFITSIO since 1999. The general convention looks like this: > > HIERARCH = value / Comment String > > where represents any string of ASCII text characters > (except the equal sign character which is not allowed because it serves as the > delimiter between the Effective Keyword Name and the value. OK, if you have enough evidence that the full convention as supported by CFITSIO is in use (beyond the ESO and long keyword cases) then the definition above should appear FIRST in the document for the registry ! My only comment then could be that HIERARCH is a misnomer, but that should be accepted on historical grounds. > In the ESO special case, the Effective Keyword Name consists of a series of > tokens that each conform to the requirements of a FITS keyword name. > > In the long keyword name special case, embedded spaces are not allowed to > avoid confusion with the ESO usage, These should than be presented in the document as two clear subconventions of the general case. My suggestion are : - to call the convention "Generalized keyword (HIERARCH) convention" - to copy your general definition in a prominent place at beginning of the document - to call the first subconvention "hierarchical (tokenized) subconvention" ... the ESO case will be a sub-sub-case where the first token is 'ESO' ! - to call the second subconvention "long keyword subconvention" ************************************************************************** [Editorial Note] Following further discussion within the IAU-FWG, the description of the more general long keyword convention has been removed from this document mainly because it has not been widely used in publicly available FITS files. William Pence September 2009