This is a digest of the discussions on the IAUFWG email list regarding the INHERIT keyword convention. The original discussions may be seen in the IAUFWG archive at http://listmgr.cv.nrao.edu/pipermail/iaufwg/ ***************************************************************************** Date: Fri, 6 Apr 2007 14:36:41 -0700 From: Steve Allen Note that I redirect this to IAUFWG alone in order to reduce the distraction on FITSBITS. On Fri 2007-04-06T15:32:49 -0400, Arnold Rots hath writ: > That's neither here nor there for the present discussion, but I > mention it to indicate that I feel it would be a bad idea to change > our mind on the principle of self-sufficient HDU headers. Within the context of FITS I don't see any alternative to spreading important information across multiple HDUs. The only way to encode arbitrary amounts of structure in a FITS HDU right now is with tables. A table which can represent arbitrary amounts of structure is almost guaranteed not to be normalized in the sense of good database design. It is explicit in the FITS files produced by our multi-slit spectrograph that the slitmask design is attached using many separate table HDUs which are the normalized subset results of SQL queries on a normalized database. This scheme is used from the point of submitting the mask design, through its manufacture and use in the telescope, and in the data reduction pipeline. As such it is the only optical spectrograph on Keck whose images can be reduced without the use of data slaves manually transcribing data, and the grassroots never stop asking why the other spectrographs don't do it. I really wish that there were a conventional mechanism which described the primary and foreign key relations between all of the HDUs on these images, and which would thus serve to indicate that they all must be handled together in a self-consistent fashion. Even disregarding the attached binary tables, the 8 or 16 image HDUs have no convention for describing the reason they are all combined into a single FITS file. This in particular is a thorn in the side of Bill Joye as he tries to enhance ds9 to "just know" what to do with a FITS file that comes over a HTTP connection from far away. I really want to describe relations, but I don't have a good enough idea about how to do it in a simple but general way. This is not to say that I am sure we can, or should, try to stretch FITS to describe arbitrary relations between HDUs. But if not FITS, what sort of alternative will work best for the community? ***************************************************************************** Date: Tue, 10 Apr 2007 18:31:53 +0200 (CEST) From: Lucio Chiappetti On Fri, 6 Apr 2007, Doug Tody wrote on FITSBITS > 1) What is a convention, and what is our role in documenting them, and > 2) Is the INHERIT convention (most recently) a good idea Since the discussion has shifted a bit from FITSBITS to IAU-FWG I reply quickly here. > Regarding 1), I suggest that a convention is not an official recommended > standard, and if we are confused about this fact, we cannot proceed to > document conventions. I agree with Doug (and also with Walter). > The FITS registry of conventions should not "recommend" or "discourage" > any convention. If we start making such distinctions are we are > starting to repeat the standards process. I agree. The only thing we *could* add is a statement about how widespread in usage a convention is. But here too I have doubts, how can we get such info in a trusted way, if not by word of mouth of the submitters ? Also, even if a convention is used only at one site, it might be important (e.g. if used in the products produced by a MAJOR observatory or space mission). > Regarding 2), INHERIT is an established convention in current use, and > as such should be documented. It is absolutely fine if we do so with > various caveats about possible issues. Personally, being myself rather simplicist (i.e. I like single-extension FITS files, or multi-extension with general documentation kwds in the primary HDU), I find nothing bad in INHERIT (that's why I was silent). The only remark I could make, similar but not identical to one I raised for the FOREIGN convention, was that some of the material in the documentation was not "informative" or "syntactical" but "usage notes", and as such could be dropped. However since such usage notes are mostly the caveats written by the convention authors, I've nothing against the fact they remain in the documentation as it stands now. ***************************************************************************** Date: Fri, 13 Apr 2007 00:05:22 +0200 From: Francois Ochsenbein Hi, After reading the many comments regarding the 'INHERIT' convention, I have the impression that we are really very picky about what is a convention -- which is NOT a proposed addition to the FITS standard, as pointed out many times. Like Lucio Chiappetti, I can't see what is fundamentaly bad in this convention -- like any inheritance mechanism, it may lead to inconsistencies. If the convention is found interesting, imported and tried in several contexts, the pitfalls, limitations, problems etc will be discovered, and balanced against the advantages. By the way, Bill, do you plan to attach some "comment space" to each registered FITS convention, like discussion summary, pros and cons, list of software, persons, institutes ... making use of the convention ? --francois ***************************************************************************** Date: Thu, 12 Apr 2007 18:13:24 -0400 From: William Pence Francois Ochsenbein wrote: > By the way, Bill, do you plan to attach some "comment space" to each > registered FITS convention, like discussion summary, pros and cons, list > of software, persons, institutes ... making use of the convention ? Yes, to first order it will just be a concatenation of all the emails related to the convention from FITSBITS and the IAUFWG lists. It might evolve into something more elaborate as we gain more experience with this review process. Bill ***************************************************************************** From: Mark Calabretta Date: Fri, 13 Apr 2007 09:54:14 +1000 On Fri 2007/04/13 00:05:22 +0200, Francois Ochsenbein wrote in a message to: iaufwg@nrao.edu >as pointed out many times. Like Lucio Chiappetti, I can't see what is >fundamentaly bad in this convention -- like any inheritance mechanism, The discussion is (or should be) of the adequacy of the documentation, not that of the convention itself. Fundamentally bad conventions especially need to be documented! Cheers, Mark ***************************************************************************** Date: Tue, 10 Apr 2007 12:01:36 +0200 From: Walter Jaffe I will avoid discussing the optimum structures for transmitting data bases and confine myself to the role of the iaufwg in reviewing conventions. I don't think that it is feasible for the iaufwg to make deep reviews of the proposed conventions, and then expect or require the proposers to "improve" the submission. I think the review should confine itself to whether the proposal is formally consistent with the FITS standards and whether it meets the basic criteria for a convention: it has an application, it is reasonably stable, and it is sufficiently well documented that others can use it. This begs the question of whether the expertise present in the iaufwg can be used in fact to improve the convention. While this is probably desirable, I would separate this from the reviewing procedure. We might attach a blog and a revision history to each convention. In the first case people both in- and outside the iaufwg can make suggestions for improvements and point out hazards inherent in the convention. The second would document those changes that the authors have made. Walter ***************************************************************************** Date: Tue, 10 Apr 2007 08:12:29 -0400 From: Robert Hanisch Hi Walter. I have to say I disagree with you on this. In the case of these conventions, there really is no one else looking at them. The regional FITS committees are not involved in their review and endorsement. And, sorry to repeat myself, but despite the caveats on the website, I believe that the presence of a convention in the FWG registry does convey a certain imprimatur. There are a lot of things people can do that are legal FITS but bad ideas, and I don't think the FWG should recognize conventions simply because they do not break any rules. That said, I do agree with your second paragraph (but am neutral as to the mechanism for collection and dissemination of comments). A simple solution might be to add something to the registry such as "Comments and Recommendations" which would contain remarks from the FWG and perhaps a link to further discussion in a mail list or blog or whatever. Bob ***************************************************************************** From: "Dick Shaw" Date: Tue, 10 Apr 2007 09:12:05 -0700 Hi Bob, I have to admit that I don't understand what you are asking the WG to do. We agreed some months back to support the idea of a registry of FITS conventions; as I understood it, the main idea was to provide a clearing house of information about FITS usage in the community. It was not, as I recall, established to be a filter for conventions that have been in any way blessed by the IAUFWG. Also, the existing rules for submitting a convention to the registry, and the criteria by which they are judged, do not provide for objections based on whether the practice is in some way flawed. In the case of the INHERIT convention, it is pretty clear that it complies with the FITS standard, that datasets using this convention have been been produced in great number and for a long while, and that the submitters have followed the posted rules. In my view, the description of the convention is very well written, and describes the disadvantages very clearly. So my question is: are you suggesting that the rules for acceptance of conventions into the registry be changed? I agree with you that the presence of a convention in a registry that is maintained by the IAUFWG does more than document FITS usage: it can also, for example, encourage designers of data formats for new instruments to consider using these conventions. But I don't see a way around that--after all, data engineers can discover conventions whether we document them or not. I do not believe an uncontrolled forum for collecting comments would amount to anything more than a blog. I am worried, though, that a more formal review process would consume a lot of our time for very little gain. Regards, Dick ***************************************************************************** Date: Tue, 10 Apr 2007 12:18:11 -0400 From: William Pence Walter Jaffe wrote: > This begs the question of whether the expertise present in the iaufwg > can be used in fact to improve the convention. While this is probably > desirable, I would separate this from the reviewing procedure. We might > attach a blog and a revision history to each convention. In the first > case people both in- and outside the iaufwg can make suggestions for > improvements and point out hazards inherent in the convention. The > second would document those changes that the authors have made. I think the most important thing we can do (and it is also fairly easy to implement) is to capture all the discussions about a convention and make them easily available to future users from the registry web page. Since virtually all the discussions about a particular convention takes place during the 'public comment' period on FITSBITS, it is probably overkill to set up a separate wiki or blog for this purpose. Instead, at the end of the public comment period, I plan to append all the relevant emails from FITSBITS (and the IAUFWG list if appropriate) into a single text file that will be available from the registry web page (probably as a link labeled 'Comments and recommendations about this convention'). If more discussion about that convention occurs at a later date, then I will manually add them to the 'Comments and recommendations' text file for that convention. I don't think this will happen very often, so it shouldn't require much effort on my part. For this to work efficiently, it is important that everyone here make any comments that they have about the convention during the 'Public Comment' period on FITSBITS, and not wait until the last minute to raise any new issues here on the IAUFWG email list just before the IAUFWG formally approves adding the convention to the registry. I'll just add that I think it is perfectly acceptable to discuss implementation issues or criticisms, or suggestions for improving the convention during the public comment period on FITSBITS. When we first set up the registry, I thought that these sorts of discussions should be considered 'off topic' and should be discouraged, but now I've come to believe that these sorts of wide-ranging discussions are health and useful and may lead to the development of even better conventions in the future. These discussions should not affect whether the convention gets added into the registry, however, since the main purpose of the registry is to simply document how existing FITS conventions work. Bill ***************************************************************************** Date: Tue, 10 Apr 2007 10:32:17 -0600 (MDT) From: Doug Tody Hi Bill - This is fine with me. The only thing I might suggest, is that there is no reason to discourage a final round of comments on the IAUFWG list. That is, we could have the initial discussion on FITSBITS, to see what the community thinks, plus the possibility of some summary or concluding comments from the IAU group. We might ask though, that any suggestions for revisions to a specification occur early on in this process, in the FITSBITS forum, and that if there is a concluding IAUFWG comment period, this would be based on the final documentation. - Doug ***************************************************************************** Date: Wed, 11 Apr 2007 14:42:16 -0400 From: William Pence Doug, I agree as long as the final discussion on the IAUFWG is in the spirit of summarizing or restating the previous discussions on FITSBITS. I was more concerned with discouraging IAUFWG members from waiting until this final discussion period to raise entirely new issues about the convention which should have been more properly raised during the public comment period on FITSBITS. Bill ***************************************************************************** Date: Wed, 11 Apr 2007 00:24:51 +0100 From: Clive Page Robert Hanisch wrote: > repeat myself, but despite the caveats on the website, I believe that the > presence of a convention in the FWG registry does convey a certain > imprimatur. There are a lot of things people can do that are legal FITS but > bad ideas, and I don't think the FWG should recognize conventions simply > because they do not break any rules. Bob I don't think that the existence of the FWG registry implies approval. The main purpose was just to register the existence of each convention and document it in a location which everyone can find and use. I imagine that there are two main ways that people will use the registry: (a) If they encounter a FITS file which has used some convention it will make it easy for them to search for the documentation on it so that they can read it easily. (b) If they want to write a FITS file and need to develop a convention for (say) some new data structure, they might think of searching the registry to see if e.g. someone has mapped that structure to FITS before. They might then use the same convention, or perhaps, after reading the full gory details, they might invent something they think is better (and then register it, we hope). At least they should be able to avoid inventing a new convention which clashes in some way with an existing one, e.g. using the same keywords to mean different things. I'd have thought that both uses (a) and (b) are ones that we'd all want to support (but maybe you can think of reasons why not?) The next question is whether each convention can be annotated in some way to mark it as either approved or deprecated (perhaps with reasons if the latter, e.g. superseded by some later convention). That would also be a nice to have, but I don't think we have a mechanism for doing that at present. Maybe a working group could be set up to do the approving, but it might not be easy to reach consensus on each convention. It might also be politically difficult to deprecate a convention that is in active use by some small group of users. Regards ***************************************************************************** Date: Tue, 10 Apr 2007 18:09:52 -0600 (MDT) From: Doug Tody On Wed, 11 Apr 2007, Clive Page wrote: > The next question is whether each convention can be annotated in some > way to mark it as either approved or deprecated (perhaps with reasons > if the latter, e.g. superseded by some later convention). That would > also be a nice to have, but I don't think we have a mechanism for > doing that at present. Maybe a working group could be set up to do > the approving, but it might not be easy to reach consensus on each > convention. It might also be politically difficult to deprecate a > convention that is in active use by some small group of users. I don't think we can approve or deprecate a convention without getting into the standardization process, which is contrary to the point of a convention. Basically, if the IAUFWG "approves" or "recommends" a convention it becomes a standard. What would likely happen if we tried to do such a thing formally, is that we would throw out almost every single one of these conventions (I would not recommend HIERARCH, but someone else might not recommend INHERIT). I think defining the difference between a standard and a convention, and just documenting conventions uniformly, with formal comments from the IAUFWG, without attempting to make a recommendation, is sufficient. It is similar to an IVOA Note for example. Personally I have no problems with having both standards, and conventions layered upon standards. The existence of such conventions is in some degree a sign of a healthy and actively used standard. Often the conventions target some narrower problem, specializing the more generic standard for some more specific purpose, where a broad standard is not appropriate or would be too expensive and time consuming to produce. - Doug ***************************************************************************** From: Mark Calabretta Date: Wed, 11 Apr 2007 10:28:10 +1000 On Tue 2007/04/10 09:12:05 MST, "Dick Shaw" wrote in a message to: iaufwg@nrao.edu >I have to admit that I don't understand what you are asking the WG to do. We >agreed some months back to support the idea of a registry of FITS conventions; >as I understood it, the main idea was to provide a clearing house of >information about FITS usage in the community. It was not, as I recall, >established to be a filter for conventions that have been in any way blessed >by the IAUFWG. Also, the existing rules for submitting a convention to the That was also my understanding, i.e. that we were only required to validate the adequacy of the documentation, not the adequacy of the convention itself, that being the essential difference between a convention and a standard. Thus, if a significant body of data has been written in the guise of "FITS" then any convention it uses should be documented. If the convention violates the standard, whether unwittingly or not, then that should be noted. If it is likely to cause problems for general FITS interpreters then that should also be noted. While technically INHERIT may not violate the FITS standard, it must also be understood that the FITS standard was not written as a piece of loophole-free legislation. It seems to me that INHERIT exploits a loophole in the standard that will cause difficulties for general FITS readers. That should be noted. Cheers, Mark ***************************************************************************** Date: Tue, 10 Apr 2007 21:51:54 -0400 From: Robert Hanisch I agree with Mark and Clive's latest postings, though I still am concerned that some developers will take the presence of a convention in the registry as an implicit stamp of approval. That is why I want a way to convey the sense of the FWG in the registry. Yes, this exists. Yes, it is legal. Um, it has problems you should be aware of. Mark, in his usual incisive style, states it perfectly: " It seems to me that INHERIT exploits a loophole in the standard that will cause difficulties for general FITS readers. That should be noted." And I would like for the registry to be able to note such things at a visible level. I was sorry to see that Lucio disagreed with me on this, since virtually everything he has ever written to FITSBITS or this group I have been in concord with. Ok, this is my last word on this. I think INHERIT should be included in the registry of conventions. I think it should be noted at the top level of documentation, on the web site and not just in the document, that it may cause problems for FITS readers unaware of the convention. I think the FWG should be able to add such comments for other conventions, since there is no other review or validation mechanism other than the FWG. I do not accept the view that this body is incapable of making relevant observations on such matters. Fine. (The End) Bob