Subject: Final version of DSWG data system experiences for M. Schoeberl

From: [email-masked] (David Winningham)
Date: Fri, 19 Nov 93 16:00:29 CST

----- Begin Included Message -----

>From "34210::MX%"[email-masked]"" Fri Nov 19 15:14:45 1993
Date:    Fri, 19 Nov 1993 21:00:15 GMT
From: "34210::MX%\"[email-masked]\""@PEM.space.swri.edu
Subject: Final version of DSWG data system experiences for M. Schoeberl
To: [email-masked]
X-Vmsmail-To: pem::david
Content-Length: 20070

Sender: [email-masked]
Date: Fri, 19 Nov 1993 20:27:41 MST
From: Paul Bailey (303) 497-1410 <[email-masked]>
To: "cdhf2::graham"@uars.acd.ucar.edu, [email-masked], 
    "cdhf2::bove"@uars.acd.ucar.edu, "cdhf2::gates"@uars.acd.ucar.edu, 
    "cdhf2::pierce"@uars.acd.ucar.edu, [email-masked], 
    "cnesta::jalbaud"@uars.acd.ucar.edu, [email-masked], 
    [email-masked], [email-masked], 
    [email-masked], "virgo::knapp"@uars.acd.ucar.edu, 
    "virgo::eberle"@uars.acd.ucar.edu, "cdhf2::wincmn"@uars.acd.ucar.edu, 
    "windif::guinle"@uars.acd.ucar.edu, 
    "windif::charlot"@uars.acd.ucar.edu, 
    "cdhf2::nagatani"@uars.acd.ucar.edu, "cdhf2::huang"@uars.acd.ucar.edu, 
    "cdhf2::smallwood"@uars.acd.ucar.edu, 
    "cdhf2::laubert"@uars.acd.ucar.edu, "cdhf2::gill"@uars.acd.ucar.edu, 
    "cdhf2::wimple"@uars.acd.ucar.edu, "cdhf2::connellp"@uars.acd.ucar.edu, 
    "susim::reiser"@uars.acd.ucar.edu, "susim::floyd"@uars.acd.ucar.edu, 
    "susim::esfandiari"@uars.acd.ucar.edu, 
    "cdhf2::marshall"@uars.acd.ucar.edu, "sprlj::gell"@uars.acd.ucar.edu, 
    [email-masked]
X-VMSmail-To: @MAILLIST.DAT
Message-ID: <[email-masked]>
Subject: Final version of DSWG data system experiences for M. Schoeberl

DSWG Members

Attached is the text version of the DSWG experiences document being sent to
Mark Schoeberl. If you would like a more cosmetic version, there is a
PostScript file in

   ACDURS::DT:[BAILEY.DSWG]DSWG_STATEMENT.PS

A Macintosh Microsoft Word version in BinHex format is also in the same
directory.

I would like to thank all of you who contributed to the document and 
provided editorial comments on the draft. I recognize that it does not
express everyone's viewpoint exactly but I hope it conveys where there is
general agreement and where there are diverse opinions.

  Regards
  Paul Bailey
  NCAR

===========================================================

                             The UARS Data System

                         Shared Experiences of the UARS 
                            Data System Working Group

                                 Prepared for
                              Dr. Mark Schoeberl
                            UARS Project Scientist

                               November 19, 1993

This document describes the experiences of UARS Data System Working 
Group (DSWG) members in the pre and post launch mission phases.  It is 
intended to provide an assessment of how well the Central Data Handling 
Facility and associated systems have met the needs and goals of the Science 
Team in support of UARS data processing and analysis.  The DSWG is 
composed of a voting membership of PI Science Team representatives and 
affiliates from the UARS project, GSFC facilities, and UARS data system 
contractors.  The views expressed herein are based entirely on comments 
received from the PI Investigation Teams.

The DSWG was formed early in the UARS program to provide user input 
to the Project on the definition and development of the ground data system 
to be used to process the science data coming from the UARS instruments.  
The UARS scientist's desires regarding computing environment were 
formulated into goals and combined with NASA management needs to 
define the requirements for the overall UARS data system.  The intent was 
for the data system to be a tool to support study of the atmosphere and not 
to become a research project unto itself.  At the time, the concept proposed 
for the data system was considered by many to be a radical departure from 
the accepted norm.  Nay-sayers were common and NASA managers 
frequently felt compelled to adopt conservative positions.  However, the 
technological explosion of the past decade caught even the most visionary 
off guard.  Those now gifted with crystal clear hindsight may point to 
many areas where alternative approaches could have been employed to 
great advantage.  However, at the time the limiting decisions were made; 
the forks in the road were not so clearly marked.  What is important to 
note is that at each major decision point the DSWG and the Science Team 
had a very strong voice in the selection of the path taken.

The consensus of the DSWG is that the data system in use for UARS has in 
all respects met and, in some cases, exceeded its stated goals.  This cannot 
be over-emphasized.  That is not to say that nothing new was learned (or 
could not still be learned) from the undertaking.  In the following 
paragraphs, experiences with the major functional parts of the data system 
will be described.  It is difficult to express "lessons learned" without 
unduly emphasizing the negative aspects of an issue.  Please keep in mind 
that the UARS data system has delivered the functionality required; on the 
schedule required; within the budget allowed.  For NASA in recent times, 
this can be considered nothing short of an unqualified success.

1) Production Data Processing

Most teams felt that expectations were met or exceeded.  Most felt that the 
CDHF has consistently produced data on time in accordance with 
instrument team instructions.  Several groups believed that the requirement 
for central processing self-imposed much-needed discipline on the 
instrument team software developers.  As a result, the software delivered 
was more robust and better documented than might have occurred 
otherwise. This is thought to have resulted in fewer code errors.  Central 
processing has also resulted in data of a homogeneous nature and has 
provided a barrier to spurious data characteristics that would otherwise be 
introduced by software "tinkering".  A few felt the opposite; that the 
additional interfaces required were a hindrance but part of the cost 
necessary for central processing.

Most groups felt that scheduled deliveries worked to ensure data product 
availability for the non-instrument team members to begin development of 
validation and analysis tools. Many instrument team members felt relieved 
of the tedious task of routine processing.

However, some users felt that central processing was not flexible enough in 
the early phases of instrument/software checkout.  In their view, the 
resources required to satisfy the demands of a central facility could have 
been better spent in solving the start-up problems locally with the result of 
having scientifically useful products available in shorter overall time.

The entire CDHF has been taken over for extended periods of time on 
several occasions to support spacecraft emergencies.  This was never 
planned for in the initial concept in which the CDHF was viewed as a non-
mission-critical facility.  It was the feeling of several groups that this 
represents overkill.  In addition, downtime for hardware/software 
maintenance at the central facility affects a large number of people both at 
the CDHF and at RACs.  Even with advance notification and planning, 
considerable inefficiency results.

It is now apparent that more thought should have been given to 
requirements for reprocessing in the early stages of resource planning.  
Both the instrument teams and the data system planners share responsibility 
for this.  Lack of availability of CPU resources is not the only problem 
limiting the speed of reprocessing.  Slow data staging and awkward 
procedures for updating and managing calibration data are also factors.

In retrospect, several groups were of the opinion that the central data 
processing system should have striven more to remove 
machine/architecture dependence from the data processing software.

Some members point out that the current operations concept does not make 
maximum use of the overall system resource (CDHF + RACs) in that 
policy restricts data products created on the RACs from being assimilated 
into the CDHF.  However, efforts are currently under way to address some 
of these restrictions.

2) Data Cataloging and Archiving

Most felt that the UARS system for cataloging and archiving data products 
has worked well. The requirements for file-access attributes were well 
identified and the catalog well designed. The data file naming convention is 
excellent for catalogued files.  Some expressed the opinion that the 
capability should be extended to non-catalogued files and that the catalog 
should also include software, etc., upon which data files are dependent.  
Unfortunately, the file naming convention is not the same for jobs running 
under UCSS Analysis services and RSS as it is for production services.

Many felt that the centralization of this function has efficiently removed an 
unwanted burden from instrument teams in distributing data to members of 
the science team.  Mass storage in the central facility allows users 
reasonably quick access to the data without having to maintain it all on 
local storage.

Some point out that the inability or unwillingness of the CDHF/project to 
reclassify data produced in the engineering stream from "Test"  to 
"Production" status is wasteful of resources.  Others point out that the 
maintenance of calibration files is not as easy as it could be.

3) Data Access and Distribution

Most users indicate that their needs were satisfied and that the system 
works well.  The network bandwidth for data distribution is adequate in 
most cases.  A few felt that the use of DECNet "copy" for data distribution 
is not as robust as alternatives could be and point to the inefficient 
retransmission that is required when data links are interrupted.

Most believed that the Mass Storage System (MSS) function has worked 
reasonably well.  A few, in retrospect, felt that the MSS hardware was 
probably not the best choice.  A few felt that the "closed" system 
architecture of the CDHF had not been able to take maximum advantage of 
cheaper, more powerful CPUs and data storage technologies.  They point 
out that PI teams at RAC sites have found the advantages of other 
technologies outweigh the disadvantages due to lack of compatibility.

The vast majority found the UCSS command-line interface to be most 
useful especially when employed in conjunction with user DCL scripts.  
Most found the UCSS Menu system to be cumbersome, slow and not very 
useful. Some questioned continued maintenance of the menu system from a 
cost/benefit standpoint.

Several felt that the use of "Standard"  data formats and interfaces were 
good ideas but that more thought should have been given in implementation 
to the issues of data portability and extendibility.  Some point to 
unnecessary dependence on VMS file structures that has made widespread 
distribution and use of data difficult.  For example, the DAAC must 
reformat UARS data before general public distribution.  

Several people complained about the frequent UCSS downtime and felt that 
catalog query and data retrieval should have been implemented on a 
separate platform from other CDHF functions.  Also, several felt that 
destaging of data occurs too quickly.

Some groups pointed out that while the Near Real Time (NRT) data 
capability was added late in the design stage it has worked out very well. 
They have found that in practice, this source of information has been 
invaluable for instrument monitoring and operations support especially 
given the frequent needs to deactivate and reactivate instruments.  Several 
commented that this source of instrument information was better than the 
POCC KCRT that was originally intended to supply this function.

4) Remote Analysis Computers and Networking

Most groups found the homogeneous system environment very important 
to software development and felt the selection of DEC/VMS had been a 
good choice at the time the decision was made.  The RAC concept was 
highly praised in that it allowed the development of local data management 
systems that better support use of data at investigator sites.  Local control 
has allowed users to take advantage of new technologies and open 
architectures not supported at the CDHF and has enabled them to make use 
of unique resources at their home institutional facilities.  The result has 
been better utilization of the funding earmarked for data analysis.

Some users complained about the interactive response time between the 
CDHF and RACs indicating it to be adequate at best.  It was suggested that 
future interactive interfaces should consider alternative approaches such as 
"client/server"

One team pointed out that spacecraft integration and test could have been 
much more easily supported a RAC installed at the spacecraft integration 
site at GE.

5) Software Development Environment

Most groups found the homogeneous hardware/software environment to be 
very beneficial and indicated that it supported several products useful for 
software development.  The concept of local (RAC) development and 
testing and subsequent CDHF full-up engineering stream testing prior to 
production worked fairly well and caught almost all problems early on.  
Some groups expressed appreciation for CDHF features such as system-
wide pointers that made for easy access to the latest releases of 
UCSS/UAOS.  Development utilities supported at the CDHF were also 
found to be very helpful as well as the UCSS user guides.

The biggest disappointment expressed by most groups was with the RAC 
Simulated Services (RSS).  The simulated environment did not match the 
real environment of the CDHF adequately in many cases and required 
many work-arounds.  Many found the approach to implementing RSS in 
support of software development at the RACs to be very demanding of 
human resources and the required set-ups to be very fragile.  Pre-release 
testing of RSS functions and RSS documentation was not always adequate.  
The difficulties were not entirely confined to the RACs.  The functional 
non-equivalence of Production, Analysis and RAC-Simulated versions of 
the UCSS catalog access services made full-up testing more difficult than 
necessary even on the CDHF.  In retrospect, it is felt that requiring 
portability of all central services to RACs would be much more robust than 
simulation and probably more cost effective overall.

Many users would have liked to have had more disk space at the CDHF for 
pre-delivery testing.

6) Production Software Integration, Test and Management

Most software developers found the configuration control approach to be 
reasonably effective and the production software support team very 
responsive and professional.  After the initial shake-downs of the 
procedures, most instrument teams have found delivery of recent software 
versions to be rather straightforward and uneventful.

However, the first attempts at integration and test of production software 
were found to be painful for many.  It is felt that procedures should have 
been better thought out and tested in advance.  End-to-end simulation and 
testing requirements should have been defined much earlier, with more 
input from the instrument teams and supported with more resources.

A few felt that the configuration management approach should have had 
more flexibility and should have recognized levels of severity or criticality.  
Simple bug fixes that did not alter the data product, algorithm, or resource 
requirements should have been easier to implement.

Some felt that standards were not adequately enforced and that too much 
use of DEC specific language extensions had been allowed.

Some expressed the opinion that there is still not a good, consistent, 
universally accepted  definition of what constitutes a "new" version of 
software/data product.

7) User Support

DSWG members were unanimous in their praise of the personnel (NASA 
and contractor) supporting the data system.  They have continually 
displayed professionalism, openness, responsiveness and a genuine team 
philosophy.  It is felt that this enabled the various groups to work through 
some difficult periods without becoming adversarial.  The success of UARS 
results in large part from the hard work and dedication of the data system 
staff.

The only negative noted was that some key people who possessed a 
considerable amount of corporate memory were moved to other projects 
perhaps prematurely.

8) Mission Operations Support

While somewhat outside the normal purview of the DSWG, some team 
members wanted to recognize the good job done by the Data Capture 
Facility and other elements of the mission operations component.

Several instrument teams found that the POCC KCRT was useful to 
instrument operations but that interactive response was disappointing.  
Others found the POCC KCRT to be almost useless in their environments 
and relied mostly on NRT data from the CDHF.  Quicklook Orbits were 
found by some teams  to be very useful for mission analysis and they felt 
the capability could be further enhanced by quicklook on demand.

9) Project /Data System Management

The use of technical working groups to advise the UARS Project was an 
outgrowth of Science Team involvement in the definition of the mission.  
Several members felt that the early involvement of the DSWG was critical 
to the success of the data system.  While the DSWG concept worked well, 
some groups felt that other facets of mission operations did not have 
advisory groups with the same "critical mass".

A few groups felt that the CDHF suffers somewhat from fragmented 
authority.  Internal communication between operations, system 
management, contractors, etc., is not always as good as it could be.  
Problems reported to one of the contact points do not always get relayed to 
the others.  Others point out that all aspects of the UARS data system are 
not under single central management and that the configuration reflects 
more of the GSFC organizational structure than it should.  For example the 
Command Management System, daily planning aids, simulated 
orbit/attitude data files and Level-0 production data files all fall under 
different management systems with different associated data formats, 
conventions and management policies.  All of this filters down to the 
experimenters in one way or another and makes the overall interface more 
complicated than it should be.  One group was of the opinion that such 
problems are inherent in organizations of this magnitude.  They felt that 
such difficulties were overcome with minimal effort in the normal course 
of doing business and were generally pleased by the responsiveness of the 
Project/CDHF in smoothing out rough edges as the mission developed.

Several groups believed that unrealistic deadlines for "flight-ready" and 
"validated" production software were counter-productive.  Science data 
processing software is not the same as space hardware and should not be 
managed as such.  Failure to recognize this led to inefficiencies in the 
utilization of resources (human, time) in the near pre-launch and post-
launch periods and to incorrect expectations by CDHF/project management 
as to the time required to validate algorithms and data products.  Another 
group felt that such deadlines were beneficial to them as a way of focusing 
attention towards finding solutions to the essential problems and ensuring 
cost effective use of resources.

Finally, a few users felt that Project Management was slow to recognize 
that the overall UARS data system had successfully evolved into a 
distributed system.  Project policies have recently been revised to utilize 
this benefit (when appropriate) and the project has begun to define 
procedures by which data products created on RACs may be assimilated 
into the CDHF catalog/MSS and software/data management rules to which 
RACs must adhere in the creation of standard products.

Compiled  and edited by:
Paul Bailey
UARS DSWG Chairman

----- End Included Message -----