From heafits@legacy Wed May 12 16:44:30 1993 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["6760" "Wed" "12" "May" "1993" "16:42:20" "-0400" "William Pence" "pence@tetra " nil "123" "HK telemetry formats" "^From:" nil nil "5"]) Return-Path: Received: from legacy.gsfc.nasa.gov by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA03506; Wed, 12 May 93 16:44:28 EDT Received: from loopback by legacy.gsfc.nasa.gov (5.65/Ultrix3.0-C) id AA19408; Wed, 12 May 1993 16:42:20 -0400 Message-Id: <9305122044.AA04470@tetra.gsfc.nasa.gov> Originator: heafits@legacy.gsfc.nasa.gov Errors-To: oneel@arupa.gsfc.nasa.gov Reply-To: heafits@legacy Version: 5.5 -- Copyright (c) 1991/92, Anastasios Kotsikonas From: pence@tetra (William Pence) Sender: heafits@legacy To: dwells@fits.CV.NRAO.EDU Subject: HK telemetry formats Date: Wed, 12 May 1993 16:42:20 -0400 This is in reply to a couple of points raised by Lucio Chiappetti in his message dated 27 April 93 concerning the formats for science telemetry data and HK telemetry data. Lucio argued that these data tend to be highly mission specific, therefore they should not be the subject of discussion for standardization between different missions. However, from the experience we have had at the HEASARC in dealing with ROSAT, ASCA, and (in the near future) XTE data, we would argue that there are significant advantages in trying to develop standard FITS formats for telemetry data. These three different missions (ROSAT, ASCA, XTE) all have somewhat different requirements, and hence the resulting FITS files have slightly different formats. But the data files from all three missions share many common characteristics (most importantly they all use the BINTABLE format) which makes it easier to write multimission software which can deal with any of these data files. Below is a brief description of the FITS format for the HK telemetry files for each of the missions. Hopefully other future missions will find that one (or more) of these formats meets their requirements. ROSAT HK Files Planning for the ROSAT mission began before the HEASARC was formed, therefore we were not involved in the early data format decisions. It was decided early on that only a small set of HK parameters was of interest to the observer and this decision is being maintained in the new 'rationalized' FITS formats that are now being implemented. The values of these HK parameters are written to a FITS binary table extension with a separate column for each parameter. A new row of values is written to the table for each cycle of telemetry readouts from the spacecraft. Besides the set of parameter value columns, there is also a 'TIME' column in the table which records the time of all the other measurements on that row of the table. (The format of this TIME column is another big issue which should be discussed separately). In principle, if the values of all the parameters remained the same from one row to the next, the latter row could be deleted from the table without lose of information to reduce the size of the table, but this has not been done for ROSAT. ASCA HK Files During the early design of the ASCA FITS data formats it was decided that virtually all the HK parameters should be saved and given to the observer in the HK FITS file. Because there are about 600 different ASCA HK parameters it was deemed not practical to use the ROSAT format where each parameter is written to a separate column of the table. Instead the HK data are written to a much smaller binary table which contains only three columns: TIME, HK_NAME, and HK_VALUE. A new table is written for each observation. The initial value of each HK parameter is written at the beginnig of each table (which requires about 600 rows to list all the parameters), and then whenever the value of a parameter changes from its previous state, the new value, with its time stamp, is written to the table. Since many of the HK parameters do not change value very frequently, these files are only a fraction as large as if every parameter is written into a separate column. While this format has the advantage that is is quite compact, it is often more convenient for software to read the HK data if it is in a ROSAT-style format table. Therefore, we have written software which will convert FITS files between these two formats (usually after selecting just a small subset of the total number of HK parameters to be converted). There is also software available which will convert the raw telemetry value of each parameter into its true engineering units (e.g. Volts, or Degrees C). In general this conversion can only be applied to the ROSAT-style format files which allows the the datatype of each parameter to vary (e.g., integer or floating point). In the more compressed format HK files the HK_VALUE column is always a 4-byte integer. XTE The design of the formats of the XTE FITS files has just started, but one of the primary requirements is that we preserve as closely as possible the original telemetry format of the HK data (which has a packet format similar to SAX) with a minimum of reformating. This a partially driven by the desire to minimize the chances that data may be corrupted during this reformating process. Thus the ASCA format, which involves a considerable amount of reformating of the telemetry stream, is not suitable for XTE. One of the characteristics of this data is that there is a regular cycle of HK measurements in the telemetry which gets repeated at a well defined set interval (which I'll call a 'major cycle'). Some HK parameters get readout more frequently than others during a major cycle, so that while some parameters only occur once per major cycle, other parameters may be readout more frequently and appear multiple times in each major cycle. A fairly natural way to store this HK telemetry data in a binary FITS file is to represent each major cycle as a single row in the table, with a separate column for each HK parameter. Those parameters that occur more than once in each major cycle are stored in vector columns in the binary table. Very little processing needs to be done to convert the HK telemetry into this FITS format; it mainly consists of padding out the telemetry values (e.g., if the telemetry value has 1 to 8 bits, it is written as an 8-bit '1B' column in the FITS table). Note that the ROSAT format is a special case of the more general XTE format in which none of the columns contain vectors. Summary >From the example of these three missions, it can be seen that is fairly easy to preserve the HK telemetry data in FITS format files thus making it available to all observers in a standard, relatively easy to understand format. The HEASARC is developing software which can process these HK FITS files in various useful ways. For instance, we have software which will take a HK telemetry FITS file, along with a set of criteria on the values of various HK parameters, and will then output a list of time intervals during which the HK parameters satisfied those criteria (i.e., produces a list of 'Good Time Intervals'). This list of time intervals, which is also in a BINTABLE FITS format, can then be feed into another program which will select all the events from a Science FITS file which occur within one of those time intervals. In summary then, we would recommend that other missions consider adopting a FITS format similar to these for their HK telemetry data files. I can supply examples of these files, (at least for ROSAT and ASCA) if anyone wants to see more details. -Bill Pence HEASARC From heafits@legacy Fri May 14 22:29:26 1993 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["400" "Fri" "14" "May" "1993" "22:27:18" "-0400" "Usque ad mortem bibendum" "GEORGE@LHEAVX.GSFC.NASA.GOV (Ian M George, Code 668, NASA/GSFC, USA )" nil "10" "RE: about OGIP/93-001" "^From:" nil nil "5"]) Return-Path: Received: from legacy.gsfc.nasa.gov by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA05548; Fri, 14 May 93 22:29:25 EDT Received: from loopback by legacy.gsfc.nasa.gov (5.65/Ultrix3.0-C) id AA01291; Fri, 14 May 1993 22:27:18 -0400 Message-Id: <930514222451.202011c5@LHEAVX.GSFC.NASA.GOV> Originator: heafits@legacy.gsfc.nasa.gov Errors-To: oneel@arupa.gsfc.nasa.gov Reply-To: heafits@legacy Version: 5.5 -- Copyright (c) 1991/92, Anastasios Kotsikonas From: GEORGE@LHEAVX.GSFC.NASA.GOV (Ian M George, Code 668, NASA/GSFC, USA (Usque ad mortem bibendum)) Sender: heafits@legacy To: dwells@fits.CV.NRAO.EDU Subject: RE: about OGIP/93-001 Date: Fri, 14 May 1993 22:27:18 -0400 Just to reassure you all, Lorella & I have revised the above OGIP memo proposing a standard for the specification of unit strings within FITS in the light of all the comments received. I think we accomodate most of them. The document is currently circulating within the OGIP to get rid of as many obvious typos etc as possible, and we plan to make it public on 1993 May 21. Ian M George HEASARC From heafits@legacy.gsfc.nasa.gov Fri May 21 14:47:55 1993 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] [nil nil nil nil nil nil nil nil nil nil nil nil "^From:" nil nil nil]) Return-Path: Received: from inet-gw-2.pa.dec.com by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA22208; Fri, 21 May 93 14:47:53 EDT Received: by inet-gw-2.pa.dec.com; id AA06507; Fri, 21 May 93 11:47:16 -0700 Received: by legacy.gsfc.nasa.gov (5.65/DEC-Ultrix/4.3) id AA08531; Fri, 21 May 1993 14:45:30 -0400 Message-Id: <930521144312.20e0009e@ROSGIP.GSFC.NASA.GOV> Originator: heafits@legacy.gsfc.nasa.gov Errors-To: oneel@arupa.gsfc.nasa.gov Reply-To: heafits@legacy.gsfc.nasa.gov Version: 5.5 -- Copyright (c) 1991/92, Anastasios Kotsikonas From: GEORGE@ROSGIP.GSFC.NASA.GOV (Ian M George, Code 668, NASA/GSFC, USA (Usque ad mortem bibendum)) Sender: heafits@legacy.gsfc.nasa.gov To: dwells@fits.CV.NRAO.EDU Subject: OGIP/93-001 (Vers 1993 May 21) - The specification of unit strings Date: Fri, 21 May 1993 14:45:30 -0400 As promised, the latest version of the OGIP memo (OGIP/93-001) listing the OGIP-standard strings to specify physical units within FITS files is now available. We believe we have successfully (?) addressed most of the issues/corncerns which were brought to our attention, and we thank you all for your comments. The major change compared to previous versions is that due to popular demand we now allow SI prefixes (with the drawback that the strings are now case sensitive). The memo is available via anonymous ftp on legacy.gsfc.nasa.gov (128.183.8.233) in the .caldb/docs/memos directory: ogip_93_001.tex is the LaTeX source ogip_93_001.ps is a PS version of the complete doc. (I *think* it really is PS this time !) Constructive comments are again welcomed (I suggest to heafits). [However due to pressing time constraints, it should be noted that the HEASARC is adopting this proposal in its current state.] regards Ian M George (george@lheavx.gsfc.nasa.gov) Lorella Angelini (angelini@lheavx.gsfc.nasa.gov) HEASARC