1、HST Pipeline Concepts for OTFRDaryl A. SwadeSpace Telescope Science Instituteswadestsci.eduAbstract:The Space Telescope Science Institute (STScI) has deployed an On-the-Fly Reprocessing (OTFR) system for Hubble Space Telescope (HST) STIS, WFPC2, and NICMOS data. In addition to the basic functionalit
2、y of improving the quality of HST data, OTFR offers an opportunity to provide more advanced data products. This paper will focus on these potential new data services to be provided by the HST archive.1. The OTFR SystemOver the last few months the STScI has deployed an On-the-Fly Reprocessing system
3、for HST STIS (May 1, 2001), WFPC2 (May 16, 2001), and NICMOS (September 24, 2001) data. All future HST science instruments, ACS, COS, and WFC3, will have OTFR functionality as part of their initial data processing and archive systems.Prior to OTFR, HST data were processed into FITS format files and
4、calibrated within hours after execution on the spacecraft. These calibrated FITS files were statically stored on the archive media. Upon issuing an archive request, HST data users would receive the original processed version of the data, which required them to later perform re-calibration on their o
5、wn using STScI provided software.The basic OTFR system provides archive users with the latest data processing software including updated calibration algorithms and keyword updates, as well as the latest, most optimal, calibration reference files. The OTFR pipelines recreate calibrated FITS format da
6、ta from the raw telemetry (pod files) for each archive request. Figure 1 schematically represents the flow of data through the OTFR system. Further details about the OTFR system can be found in Swam, Hopkins, providing higher-level products derived from the calibrated data; generating object catalog
7、s from optimally calibrated data; grouping data for archive retrieval and further processing as a single unit; co-adding previously unrelated data; and providing user options for data processing. The following sections will focus on some of these potential new data services to be provided by the HST
8、 archive.OTFRdat requstArchiveSnCatlogOPUS cienDatrosigmetadtStarViewquerycalibrted FITS atsetpod filepod fileuncalibrted FITS atsetHigh Levl ProductGnatHigh Levl ProductsHigh Levl Productscalibrt FITS atetcalibrted FITS atsetengiring ifoArchive mdiametadt DAS istrbutionCalibrtionDat Colectrcalibt F
9、ITS atetngring ifongring ifongiringifoniingifngiringifoegingifoFigure 1. Schematic representation of the OTFR system data flow.2. Future NICMOS plansThe STScI NICMOS science instrument team plans two augmentations to calibration pipeline processing (Bushouse, Dickinson, & Bergeron 2001) that will gr
10、eatly enhance the quality of NICMOS data when retrieved through the OTFR system.The noiseless bias levels of the NICMOS detectors have two components, the overall bias level and the shading pattern, which are dependent on the detector temperature. These effects are stable and have been accurately ch
11、aracterized. Their removal in pipeline calibration requires the delivery of temperature dependent dark reference files.Previous NICMOS pipeline processing has left a residual net bias or pedestal level in the final image. A general algorithm to perform bias equalization and pedestal subtraction, usi
12、ng the STSDAS tasks biaseq and pedsub, has been found to significantly improve the pipeline processed data in most cases, and never degrade the quality of the final image (Bushouse 2001). These two processing steps will be incorporated into calnica and available through OTFR for all NICMOS data.3. E
13、ngineering data On HST, data from the spacecraft subsystems, engineering data, follows a separate path to the ground than data from the science instruments. Typically, the engineering data arrive at STScI 1-2 days after the science data. Hence, it is not possible to incorporate data from this teleme
14、try stream into the original data processing.The recent problem with the STIS side 1 electronics provides an example of how OTFR can integrate the engineering data into the pipeline processing to support science. STIS operation has now permanently switched to side 2 electronics, which does not have
15、a thermistor to control the CCD operating temperature. Hence, the CCD temperature can fluctuate a few degrees during an exposure. Dark current and other sensitivities can be calibrated if the CCD temperature as a function of time during the exposure is known. The science telemetry contains a single
16、readout of this temperature once every STIS memory dump. However, the engineering telemetry contains a readout of the STIS CCD housing temperature every 20 seconds. The current plan is to use an average of the low resolution CCD housing temperature from the science telemetry to calibrate the tempera
17、ture effect during the initial processing. From the engineering telemetry, the CCD housing temperature as a function of time will be extracted every 20 seconds during an exposure. In OTFR, the time resolved temperature function will be used to provide an improved calibration for the STIS CCD data.4.
18、 Data Quality InformationOTFR allows for the capture of information related to the quality of HST data in the science data files. Generally, data quality information is not available when the science data are initially processed. For example, information on problems with guide star acquisitions can
19、be obtained from the engineering telemetry that arrives after the initial science data processing (see section 3 above). Success of a target acquisition may require an analysis that takes place days after the initial data processing. By placing keywords in the science data headers that are populated
20、 during OTFR processing, the latest information on the quality of the retrieved data is available to the archive user.Data quality information is also available in the paper products, which summarize the results of a target visit with HST. These paper products are currently generated for every visit
21、 and placed on-line in pdf format for interested observers. Since all observers do not utilize these paper products, STScI plans on making them available on demand through OTFR. This will allow STScI to continue to provide this service, while saving resources by creating the product only when requir
22、ed.5. SHARERecognizing that the advent of OTFR will radically change the paradigm that has previously applied to the calibration and storage of HST data, STScI has commissioned a study to evaluate the scientific potential of a wide variety of possible enhancements to the scientific data products. Th
23、e Study of Hubble Archive and Reprocessing Enhancements (SHARE) committee at STScI has been meeting over the summer in an effort to draft a list of recommendations for HST data products and services (Kriss et al. 2001). A preliminary version of this list includes the following items. Enhancing the p
24、ost-data processing World Coordinate System parameters in the science headers through image analysis in order to improve astrometric accuracy. Current astrometric accuracy of HST images is limited by the uncertainties in the Guide Star Catalog. Combining long exposure HST images to form mosaic or di
25、thered products. Most of the images that are split into sub-exposures to remove cosmic rays, minimize the effects of hot pixels, or improve PSF sampling can be combined in an automated mode. Implementing software to perform object detection, classification, and photometry in a standard fashion (Whit
26、e 2000). Building on the previous item, generate object catalogs that will provide a well-defined sample of sources for statistical analysis.6. Community InputOTFR leads to the possibility of generating new data products that could not be realized in initial, near real-time processing. The planned a
27、nd potential enhancements listed above represent the first step in utilization of the OTFR system. STScI seeks astronomical community input for further data processing enhancements that could be realized through OTFR. What enhanced data products could be generated in a pipeline mode and useful to as
28、tronomical research? Which of these would be appropriate for STScI to uniformly apply to data in the Hubble Data Archive?References:Bushouse, H. 2001, in ASP Conf. Ser., Vol. 238, Astronomical Data Analysis Software and Systems X, eds. F. R. Harnden, Jr., F. A. Primini, & H. E. Payne (San Francisco:
29、 ASP), 427Bushouse, H., Dickinson, M., & Bergeron, E. 2001, STScI Instrument Science Report NICMOS 2001-02(http:/www.stsci.edu/instruments/nicmos/documents/isrs/isr_2001_002.pdf)Kriss, G., Bushouse, H., Dickinson, M., Donahue, M., Fraquelli, D., Giavalisco, M., Keyes, T., Koekemoer, A., Meurer, G.,
30、Padovani, P., Sparks, W., & Swade, D. 2001, in preparationSwade, D. A., Hopkins, E., & Swam, M. S. 2001, in ASP Conf. Ser., Vol. 238, Astronomical Data Analysis Software and Systems X, eds. F. R.Harnden, Jr., F. A. Primini, & H. E. Payne (San Francisco: ASP), 295Swam, M. S., Hopkins, E., & Swade, D.
31、 A. 2001, in ASP Conf. Ser., Vol. 238, Astronomical Data Analysis Software and Systems X, eds. F. R.Harnden, Jr., F. A. Primini, & H. E. Payne (San Francisco: ASP), 291White, R. L. 2000, in ASP Conf. Ser., Vol. 216, Astronomical Data Analysis Software and Systems IX, eds. N. Manset, C. Veillet, D. Crabtree (San Francisco: ASP), 577
