Section 2: User Scenarios

This section evaluates the input/output load on EOSDIS by users submitting ad hoc data requests. We demonstrate that the bandwidth required to fulfill their data needs is of the same order or larger than the bandwidth needed to accommodate the current product baseline. If the loads were equal, EOSDIS should be designed 50% for standard product generation (eager evaluation or push processing) and 50% for evaluation of ad hoc queries (lazy evaluation or pull processing).

However, if some standard products are evaluated only on demand, using our eager/lazy approach, the ad hoc component becomes relatively more important. If the requirements for standard products are cut in half but the ad hoc demands remain the same, then EOSDIS should be designed for 33% push processing and 66% pull processing. These figures are consistent with the observation in the Science Requirements Summary that EOSDIS should be designed to distribute electronically 3 times as much data as are generated.

Lastly, it is likely that additional ad hoc queries will be generated by users once they are accustomed to having online access to a very large repository. Ultimately, the usage of EOSDIS should be 80-90% ad hoc queries.

The computational resources needed to satisfy ad hoc user requests are less clear. This is because a large part will be devoted to data subsetting and visualization, tasks that are rare when standard products are created. For design purposes, we arbitrarily assert that the CPU load of ad hoc data requests is of the same order as that of the standard product machinery.

These conclusions are based on our interpretation of user scenarios, which are descriptions of activities that involve EOSDIS data. Scenarios are important to EOSDIS for several reasons and have been associated with its design and development almost from the beginning. From the developers' vantage point, scenarios are important because

We expect that users will continue to work with developers to describe scenarios because this is the best way to ensure their parochial needs become features of the system.

However, considering the scope and duration of EOSDIS, most users of the data will not be participants in the scenario development. Thus, each scenario is a surrogate for a large number of users, and the demographics of this unseen community need to be considered when metrics for individual scenarios are scaled to make predictions of system use.

A successful EOSDIS will engender new and unanticipated uses of the data and better and faster ways to access them. In short, there are scenarios we know nothing about at the moment, and the architecture must be able to cope with these unknowns.

We categorize users into 3 groups:

The needs of this group (which we term the "NASA 500") have been much studied. They will be heavy and sophisticated uses of EOSDIS data, but traditionally they have not been involved intimately in NASA flight programs. It is the needs of this group, which includes Project Sequoia 2000 researchers, that we primarily model in this section.

An interesting subset of this group consists of users who want to develop scientific algorithms and models, and install them within the EOSDIS system to be run concurrently with the generation of standard products. Our architecture supports this enhancement.

This section summarizes a number of technical studies that are provided in the appendices. Appendix 3 discusses 3 Project Sequoia 2000 scenarios. Two pertain to analysis of remote-sensing data, such as AVHRR and Landsat. The third is a GCM scenario that requires access to assimilated data for model validation. Appendix 4 is a discussion and re-analysis of a number of scenarios collected by the HAIS User Model Team. It has copious examples of how scenario activities can be expressed in SQL. Finally, Appendix 5 considers management of large objects in the context of the Project Sequoia 2000 architecture.

Section 2.1 is a discussion of our analysis method. Section 2.2 takes data requirements from 2 Project Sequoia 2000 scenarios and estimates the load on the data system. The load of each scenario is multiplied by our estimate of the number of users who are likely to submit similar data requests to derive the total load. Section 2.3 has a similar load estimate for a scenario involving a bio-geochemical problem over the Atlantic. This is scenario 19 from the HAIS collection. Finally, Section 2.4 summarizes the figures detailed in the earlier sections.

2.1 Approach

We have studied the scenarios of interdisciplinary Earth scientists with the objective of quantifying their contribution to the ad hoc, or "pull" load on EOSDIS. For this purpose, a scenario is taken to be a description of a miniature system. Scenarios involve transformations of data, external interfaces to data, and local storage of data. Thus, our approach is to use standard tools, data flow models and entity-relationship models to describe scenarios. These are recognized techniques in modeling information systems. All the technical detail resulting from the work is provided in appendices 3 (Project Sequoia 2000 Scenarios) and 4 (On ECS User Scenarios).

We introduce an important novelty, which is to phrase scenarios as sequences of SQL-* queries acting on a database model. This is a succinct and logically rigorous way to describe the selection of data and the processing of data. Our success at this validates the notion that SQL-* is the proper lingua franca for EOSDIS. It also begins the effort of defining the EOSDIS-specific data types and methods needed to support the types. This technique has been used extensively in our analysis of the HAIS scenarios, which is detailed in Appendix 4.

2.2 Project Sequoia 2000 Scenarios

Appendix 3 has 3 scenarios that illustrate important aspects of the way in which EOSDIS is likely to be used by the research community. These scenarios are based on current research by Earth scientists in the Project Sequoia 2000 project at several University of California campuses.

The first scenario involves the study of biomass burning through calculation of improved vegetation indices. This scenario, which is not abstracted here in this section, requires daily access to EOSDIS data. The second scenario involves a study of snow over mountain areas of the world. The scenario requires weekly or monthly access to EOSDIS data. The third scenario is a collection of studies in climate modeling. It requires access to assimilated EOSDIS data with a time-scale of hours. This last scenario is noteworthy because it is built around a DBMS.

2.2.1 Data Requirements of the Snow Cover Scenario

The scenario described in A3.2 is a study of snow cover derived from Landsat imagery. During each winter, 10 Landsat images are needed for 20 different mountain regions. Each image covers a 2-degree square with a ground resolution of 28.5 meters and is 250 MB in size. We assume Landsat images are indexed with a DBMS in the manner described in Appendix 5 and tiled in 100&endash;MB large objects. If the tiles exactly corresponded to the study area, it would be necessary to draw out 2.5 GB, or 25 tiles. In general, this will not be the case, and several adjacent tiles will need to be spliced and then cut to give the data needed. To account for this effect, we assume that twice as much data, 5 GB, is drawn from the archive. These will be pruned before delivery to the user.

The 5 GB needed by this query is just a small part of the available data for this interval. Assuming only cloud-free views of land taken during day time are saved, Landsat data would accumulate at the rate of 120 GB/day. In 6 months, this amounts to approximately 22 TB, which is 10,000 times larger than the data needed for the scenario.

Suppose, in the first case, the data are on the disk store. The system will read the data, join the tiles, trim to the study boundary, and transmit the subset to the user's facility. By 1997, it is likely there will be T3 circuits from EOSDIS data stores to researchers' desktops. Even so, the bandwidth of this circuit will be the limitation on the speed data can become available.

To send 2.5 GB over a T3 line will take approximately 8.3 minutes at the anticipated speed of 5.0 MB/s. The scenario involves 20 similar regions of the world, so it will take 2.8 hours to transfer the month's worth of data for all regions.

The average I/O load for push processing in our EOSDIS model is 30 MB/s (see Appendix 5), and the data of this snow scenario alone adds 16 % while the transfer is in progress. Averaged over a month, the interval between requests, the relative loading is .06%. Thus, if 100 users require EOSDIS data comparable to that of the Project Sequoia 2000 snow scenario, they will add a constant 6.4% to the I/O load for 6 months of every year.

On the other hand, suppose the data of this scenario were to be requested after they had migrated to nearline archive. Now, although the speed to access a single large object is slower (because it is read from magnetic tape rather than disk), the archive will have many tape transports that can fetch data in parallel. Whether the net effect is to lengthen or shorten the retrieval time depends on how many transports are available and how data are distributed across the tapes. If the data sets are distributed across 10 tapes that are read in parallel on 10 tape drives, the expected data retrieval time is 2.5 minutes. This assumes 15 MB/s tape drive read rates and 2 minutes to mount/rewind a tape. Thus, as long as data are delivered over a T3 link, the bandwidth of that circuit remains the bottleneck in the system. This means, regardless of whether the data is retrieved from disk or tape robots, 100 users issuing data request similar to this will use 6% of the available push-processing bandwidth for 6 months of the year.

2.2.2 Data Requirements of the GCM Scenario

Another Project Sequoia 2000 scenario (A3.3) involves validating climate models with smoothed observations such as those generated by the Data Assimilation Office and other organizations. An important aspect of the scenario is the need to extract only a few field variables at a few atmospheric levels from each time-slice. The scenario assumes that this will be performed by user-developed software to be installed on the DAAC but, with the Sequoia architecture, this would eventually be a built-in function of the DBMS.

The climate model scenario requires rapid access to numerous data sets in the 100-GB range. In the most favorable (but extremely unlikely) case, the data needed for a problem will be contained exactly on 5 20-GB tapes. Suppose these are read sequentially. Then the read time will be the time to mount (30 seconds) plus the time to read a tape end-to-end (0.4 hours). Hence, the elapsed time will be effectively 2 hours and the throughput 14 MB/s. These data could be streamed directly to the researcher's computers over a 155 Mb/s OC-3 circuit if one were available. During these 2 hours this load will amount to 50% of the baseline I/O load of EOSDIS.

Climate researchers in this group might submit a similar request for data once a week. This represents a sustained load on EOSDIS of 0.55% of the base I/O load. If similar data requests were made at the same frequency by 10 climate research groups, this would add 5% to the baseline load.

It is not likely that the data will be disposed so favorably on tape, in which case the elapsed time might be longer because more tapes will need to be mounted and non-pertinent data skipped. The exact amount of slowing will depend on how many transports are available to read tapes and how the tiles are physically distributed over them. In the worst case -- sequentially reading all of the data on a single drive with 1,000 tape mounts --the data retrieval time will still be less than 2 days. The total loading of the I/O subsystem (the time-bandwidth product) remains unaltered.

2.3 Scenarios of the User Model Team

Scenario 19 is a study of bio-geochemical equilibrium over two sections of the Atlantic Ocean. Section 1 is a 500-km by 500-km square straddling the sea between Florida and the Bahamas. Section 2 is a 500-km by 5,000-km rectangle spanning the entire width of the Atlantic along the 25-degree parallel. The data of the scenario are MODIS-derived measurements of CO2 flux, photochemical images and atmospheric data, sea-surface temperature, and 3 near-surface atmospheric parameters: wind speed, wind direction, and temperature. Several queries of this scenario are given in Appendix 4.

This scenario (with many others) requires extracting a small subset of MODIS data. In one month, MODIS generates 15 TB of level 1B data. These give calibrated radiances, geo-location, and other quantities for every pixel. If these data are tiled in 100-MB large objects, there will be 150,000 tiles. There are many ways to disperse the data over the tiles. We assume that the strategy is to maximize temporal locality. In this case, a tile covers about a 5-degree square.

Many users will desire only a few field variables over a small portion of the Earth for each day. For this scenario, one study area is a 5-degree square, which happens to correspond exactly to the supposed coverage of a single tile. Three field variables are required, amounting to approximately 3 MB/day of data. The scenario needs 1 month's worth of daily data, for a total of 100 MB; this is 1/150,000 of the data available for the time interval.

If the system were optimized for this exact scenario, the "cut" of the query would be extremely fast. All 100 MB would be stored on a single tile, and a single mount and read (and optional cache to disk) would suffice. This is the case with maximum data locality, and the cut operation would take just a few minutes.

Under data clustered by temporal locality, a more normal tiling strategy, each level 1B tile holds all the calibrated and geo-registered data for a single 5-degree square. If data are saved on 20-GB tapes, each tape holds 200 tiles, and the month's data are spread across at least 750 tapes. The boundaries of the study areas and the tiles rarely will coincide. Assume 4 tiles must be fetched to access the exact area of the scenario for each day. Then, 120 tiles, or 12 GB of data, must be read to find the 60-MB subset for a month's worth of data.

Following the reasoning described previously, if these data are read serially by 1 transport at a speed of 15 MB/second, the elapsed time would be 800 seconds, neglecting the time of mounting each tape and skipping to the proper tile. Over this 800 seconds, the load on the storage manager would be 1/2 the 30 MB/s baseline. If 500 users each month submitted a similar request for MODIS data, the sustained load would be

0.5 * (.22/720) * 500 = .08

or about 8% of the push load. In the preceding expression, the term in parenthesis is the fraction of a month a single 12-GB retrieve is running.

The larger study section is a 5-degree by 50-degree swath across the Atlantic. This is a typical regional-scale problem. With the same assumptions as before, it would take 10 tiles to cover the study area exactly. Assume that 20 must be retrieved each day of the month, for a total of 600 tiles, or 60 GB. This is 5 times more data than was involved in the previous example. Thus, if 100 users each month make a similar data request, they too would add 8% to the baseline push load.

2.4 Scenario Conclusions

These examples indicate the frequent need to extract data sets 10s of GB in size from the archive. Section 2.2 described 2 problems dealing directly with this much data. In general, a larger amount will need to be read than will be used because the tiling and physical layout will be sub-optimum for any given query.

Other researchers may need only a few 10s of MB, but again such queries may resolve into multi-GB reads as was explained in Section 2.3 because of the way data are saved on tape.

Table 2-1 summarizes the results of the preceding analyses. The column labeled "load" describes the input/output rate of the scenario expressed as a fraction of the baseline push-processing rate, 30 MB/s, and the duration this rate must be sustained to satisfy the data requirement. The column labeled "community" is our estimate of the number of users likely to submit queries for a like quantity of similar data at about the same frequency. The column labeled "total load" is the combined load, relative to a continuous 30 MB/s of the whole community of users. As is seen, these 3 problems alone, scaled to the larger community, will amount to a 27% use of the baseline push bandwidth.

Table 2-1: I/O Loads Relative to 30 MB/s (EOSDIS Baseline)

Scenario

Load

Community

Total load

Sequoia snow

16% for 8 minutes, every month

100

6%

Sequoia GCM

50% for 2 hours, every week

10

5%

Ocean CO2 small area)

50% for 13 minutes, every month

500

8%

Ocean CO2 (large area)

50% for 1 hour, every month

100

8%

Total

710

27%

These scenarios all assume data will be delivered to the remote user over a wide-area internetwork connection. This is a fundamental part of our architecture. If, on the other hand, data were to be provided on magnetic tape, there would be a larger load on the EOSDIS input/output system, and attendant staff would be needed to handle the media.

When the needs of additional users are taken into account, it is easy to imagine that aggregate pull processing could be 2-10 times larger than push processing. Together with a possible decrease in the size of the push-processing load, accomplished by moving it to compute-on-demand status, we conclude that EOSDIS will be an 80-90% pull system.