FIGURE 8The two-central-processor system of Rutgers-Bell.
FIGURE 8The two-central-processor system of Rutgers-Bell.
The initial success of the original RB SDS 910 data-acquisition system was soon tempered by a result of its popularity: during most experiments the computer was unavailable for program development or data analysis. Since most experiments required the use of displays and light pens in at least one stage of data analysis, the computer center could not handle the work.
FIGURE 9The new Rutgers-Bell Sigma 2-Sigma 5 system.
FIGURE 9The new Rutgers-Bell Sigma 2-Sigma 5 system.
The solution adopted was to acquire another computer with the same instruction set (an SDS 925) and to provide switches such that the line printer, card reader, and plotter could be run from either computer. No provision was made for direct transfer of data from one computer to the other.
In practice this system worked out quite well. There was complete interchangeability of programs from the 910 to the 925, which differed only in being five times faster. Normally the switchable peripherals were run from the 925; when the group taking data wished to print or plot current spectra, they consulted with the 925 users, then used the peripherals with little more difficulty than permanently attached units would have involved.
A further advantage of the switchable peripherals, in addition to the cost saving, was that the experiments associated with the 910 could proceed while the peripherals were being serviced. The 910 is exceedingly reliable, averaging less that one main frame failure per year, and the 925 is nearly as reliable. The vast majority of service calls have been occasioned by the peripherals and have competed with data analysis but not with accelerator utilization.
In addition to the switched peripherals, both computers were equipped with two magnetic tape transports, electric typewriter, and high-speed paper-tape reader and punch. While these units were also subject to downtime, the paper-tape system and the typewriter could be exchanged between the 910 and 925. Only the magnetic-tape transports required the use of the 910 CPU during servicing, and the presence of two transports has usually meant that the second one could carry the load until the weekly accelerator maintenance period.
While the reliability record of the central processors has been excellent, that of many of the peripherals has not. Here is an excellent justification for renting computing equipment: if units do not work well, they can be returned. For a time, a low-cost card reader (100 cards per minute) built by NCR for SDS was used. It was unacceptable in reliability and was replaced by the Univac reader which came with the 925. Another unit returned was a cartridge magnetic-tape system built by SDS. The Ampex TM-4 magnetic-tape transports on both the 910 and 925 have been consistently poor in reliability, but no other unit has been available to replace them. A manufacturer's name does not seem to be a guarantee of goodor bad quality—the line printer, also made by NCR, has been excellent both in reliability and print quality.
While the two-computer system generally rated high in user satisfaction, considerations of performance have led to the design of a larger and more powerful system with totally new components. The 925, without wired multiplication or floating-point operations, was too slow for theoretical computation or for many types of data analysis such as those using Monte Carlo methods. Interactive methods of analysis, using a display and light pen, have been found very effective in the cases where the 925 could accommodate them but have not been available through either the Bell Laboratories or Rutgers computer centers.
A further limitation on the earlier system was that only one person could use the 925 at a time. The generation of a display involved the full time of the CPU, and while multiprogramming might have been able to divert some CPU time, the 8k memory size did not permit it.
Data acquisition on the 910 was limited in array size to the capacity of the core memory. For multiparameter experiments, three, six, or even twelve 4096-channel arrays have been stored in core, but the advantages of live display available with core storage have discouraged anyone from handling large arrays by logging raw data on magnetic tape for analysis later. Memory expansion would have been desirable, but the necessity of making the expansion on both the 910 and the 925 effectively doubled the cost.
Limited flexibility, then, is a major drawback of this type of system. As long as only two users needed to be accommodated, and each could adapt to exactly half of the total core storage, it was satisfactory and provided redundant facilities to guard against experiment downtime due to computer failures.
In ordering a new computer powerful enough to handle most of the nuclear physics laboratory's data analysis and theoretical computing tasks, cost ruled out the acquisition of a pair of program-compatible computers. It was recognized that desirable features of the original system would have to be obtained in new ways. Accessibility of the system for programming couldbe improved by running a simple time-sharing monitor on it. Reliability could be enhanced by avoiding bargain peripherals and using only items of demonstrated high quality and by the capability of running the peripherals on either computer.
The use of a separate CPU for data collection still seemed particularly desirable, however. A combination of a large (by present standards) computer with a powerful small computer as a front end was designed. It includes a display disk for refreshing displays without CPU attention, as well as for storing data arrays too large to be kept in core. The computers selected were a 32k, 32-bit SDS Sigma 5 and a 12k, 16-bit Sigma 2.
The new system, with separate and nonequivalent computers, will have advantages over the old system in data analysis and general computation, because these will be done on the larger computer, either in time sharing or batch mode. Time sharing should enhance the flexibility of the system by making it easier to generate and debug new programs, in addition to improving the accessibility.
For the data-collection computer, RB will lose the advantage of a separate computer on which complete debugging of programs may be done. This loss can be tolerated since the fraction of the load carried by the Sigma 2 will be less than that carried by the 910 in the old system. In the old system, very few distinct data input or display programs were written. A few subroutines and their calling parameters sufficed for all needs for six years; the logic and I/O operations unique to each experiment were written in Fortran by the experimenters.
In the new system, the Sigma 2 will be concerned with the operations used in the data acquisition and formating of displays; most of the rest can be left in the Sigma 5, with routines sent over to the Sigma 2 under the time-sharing system. If the user should prefer, he can operate the Sigma 2 directly and make use of the Sigma 5 only for data storage.
Until very recently, program development on the Sigma 2 has been slow because it lacked means of getting program listings quickly. We have now developed an assembler for the Sigma 2 which runs on the Sigma 5. The availability of card reader input and line printer output has greatly speeded Sigma 2 software development. The loading of Sigma 2 programs is also much more convenient, since they can be stored on the Sigma 5 disk and loaded exactly as Sigma 5 programs. It seems highly desirable to have an assembler for any small data-acquisition computer capable of running on another machine; the means of transporting the object code to the small computer is of less importance.
The reliability of the new equipment has been excellent. Only the card reader has had any downtime of consequence, and modifications seem to have resolved its problems. The Sigma 5 main frame has had no failures in 12 months, and the Sigma 2 has had only one in the past year. If this record continues, the loss of the redundancy inherent in the old 910/925 system will not have any serious effects.
One component of the new system is taking on an increasingly important role, although it had not been a part of the original planning. That is the computer-independent data bus consisting of system controller, bin controller, and register units. Only the system controller is specific to a particular computer; moreover the same system controller design could be used on both the Sigma 2 and Sigma 5 by restricting the data path to 16 bits. The register units are used to interface external devices to the computer quite cheaply; a typical register used here to interface an existing Calcomp plotter to the new computers costs about $300 in parts and labor. Similar units are used to interface the Sigma 5 to the Sigma 2 and to the 910, to drive a temporary core-resident display on the Sigma 5, to read pushbutton inputs, and to read ADC's. The display disk controller now under construction uses these registers to furnish control information, although the data go directly to and from the core.
At the present time, the registers are read and written under program interrupt control, but the design is not limited to program-controlled operation. By substituting a controller designed to operate automatically (directly to core or to the I/O processor) speeds approaching 1 or 2 µsec per word transferred could be obtained. Such interfaces have been built for various computers using the European CAMAC bus system, which is conceptually similar.
The system is highly modular and is built into NIM bins with modified back connectors. Exchange of modular units has been very helpful in debugging the system, and presumably it will also be helpful in case of failures in operation. This is a much more satisfactory situation than that which was obtained with the ADC interface on which RB collaborated with Brookhaven. The latter unit was built with computer-type construction: commercial logic cards and wire-wrapped back panel. Debugging of that unit was exceedingly laborious because of the lack of modularity in its components.
The computer-independent bus system has not been expensive in manpower. It has required about 9 man-months in design and debugging and somewhat less time in construction. The registers cost about $300, as mentioned, and the controllers $1500 to $2000 depending on the need for cable drivers.
The costs of the RB multiple-computer system are given in Table 6. The figures are approximate and not the result of detailed accounting.
In 1965, a system based on an SDS 920 computer was put into operation at the Brookhaven National Laboratory to control data-acquisition processes involving eight neutron spectrometers and one x-ray spectrometer. The neutron spectrometers are located on the floor surrounding the High Flux Beam Reactor (HFBR); the x-ray spectrometer was placed in the same building in order to facilitate linking it to the computer. The system can control the execution of experiments on all nine sets of apparatus simultaneously, yet each experimenter feels that he is working essentially independently of all other users. The system controls all angular rotations of crystals and counters, all detector counting, the data displays, the input and output operations, and automatic error responses.
It can also perform most of the calculations necessary for real-time guidance of the course of the experiments. For example, the experimenter can mount a crystal on a goniometer at approximately the correct angular orientation, then he can specify to the computer where several peaks should be found, whereupon the computer will direct the execution of a trial experiment to find where the peaks do, in fact, occur, executing least-squares calculations in the process, after which the error in crystal orientation is known and the angular scales are automatically corrected. In another example, the computer is given as input information the crystal constants (unit cell) and the zone orientation of the crystal on the goniometer and is asked to produce a scanning of a given part of reciprocal space. The computer then calculates where to look, turns to a correct angle to check the intensity of a central peak, and performs the other necessary steps, making many decisions as it controls the execution of the entire experiment.
When it was first assembled, the system included only two teletypes, both located near the computer. Early in 1969, a communications network was added to permit the installation of a local, assigned typewriter at each of the nine spectrometer stations, as well as three assignable remote teletypes located in the Chemistry and Physics buildings. This network incorporates a Varian 620i computer. It permits any ordinary operation to be carried out from any of the 12 remote stations, except program loading, which still must be done via the high-speed paper-tape reader at the computer.
FIGURE 10The multiple-spectrometer control system at Brookhaven National Laboratory.
FIGURE 10The multiple-spectrometer control system at Brookhaven National Laboratory.
FIGURE 11Block diagram of a single-spectrometer control station of the MSCS shown in Figure 10. [From D. R. Beaucage, M. A. Kelley, D. Ophir, S. Rankowitz, R. J. Spinrad, and R. Van Norton, Nucl. Instrum. Methods40, 26 (1966).]
FIGURE 11Block diagram of a single-spectrometer control station of the MSCS shown in Figure 10. [From D. R. Beaucage, M. A. Kelley, D. Ophir, S. Rankowitz, R. J. Spinrad, and R. Van Norton, Nucl. Instrum. Methods40, 26 (1966).]
The major parts of the system (Figure 10) are the SDS 920 computer with a 16k, 24-bit memory, a bulk storage memory section comprising two magnetic tapes units and one 32,010-word drum, the communication network, and the nine local control stations (SCS) at the spectrometers. Each SCS (Figure 11) contains the stepping motors required for computer control of angular rotations of crystals and counters, together with shaft rotation encoders (optional, incremental type) to feed information back to the computer. Each SCS also includes manual controls, the electronic counters associated with the radiation detectors, counter displays, a decoding and control section, and other related equipment.
a. The system now does "all things imagined to be necessary."
b. The computer has proved to be remarkably reliable, with a record of about 40,000 hours of use without a breakdown.
c. A reasonable amount of preventive maintenance is done, mostly during the one week of four that the reactor is shut down.
d. One person serves as operator and programmer (for simple jobs). He also transports magnetic tapes to the computing center for off-line data processing and performs smaller tasks. The average user does not need to do any programming.
e. Fortunately, the people who have written most of the programs have remained in attendance and have updated the programs frequently. Machine-language programming has not proved to be a bad chore because the system is a fixed-hardware setup.
f. Modes of data collection can easily be changed.
g. The overall performance is excellent. The only problem is an occasional wiping out of a program due to the fact that there is no hardware memory-protection feature in the computer. These accidents are estimated to cost at most a loss of a few percent of the running time.
The costs in manpower and dollars of the MSCS are given in Table 7.
Although the use of a small data-acquisition and experiment-control computer on-line to a remote computing center machine is not uncommon in high-energy particle physics applications, we know of few such systems presently operating in low-energy nuclear physics.
For the purposes of this discussion, we define "general computing facility" to be a relatively large-scale centralized installation charged with the responsibility of servicing a wide range of computing needs. The typical university computing center is our model for such a facility.
In light of the fact that only a few years back the remote computer on-line to a general computing facility was considered to be the wave of the future, with plans for such systems under vigorous discussion at many low-energy physics installations, it is at first sight surprising that there is so little progress to report at this time. The Van de Graaff accelerator laboratory at the State University of New York at Stony Brook was one such facility planning to couple a PDP-9 on hand to an IBM System 360/67 available at the university computing center. It is instructive to examine what happened there. In 1967, with the completion of the new accelerator scheduled within a year, it was decided that the best way to acquire the desired power and flexibility in computing support was through a coupled system of the kind under discussion. Plans were formulated for a high-speed transmission line to a control unit on a selector channel at the computer center. Since true time-sharing of the System 360/67 was not in the offing, a 128k-byte partition of high-speed core storage was to be permanently dedicated to the needs of experimental physics (including the particle-physics group), and a high-speed program-swapping drum and at least one tape drive were to be assigned to the physics users as well.
What actually happened was that as funds became available to the low-energy physics group to implement its share of the remote link to the computer center, sentiment shifted to the point of view that the funds could more usefully be invested in a second PDP-9 installed at the accelerator, and the second small-to-intermediate class computer was in fact purchased.The two PDP-9's are coupled only by a switchable tape drive, with no plans at present for direct channel-to-channel communication. Plans for a remote link to the computing center have been completely dropped; any further funds for computing will be invested in larger high-speed core stores for the PDP-9's, at least in the foreseeable future.
Conversations with the principals involved in the operation of the Stony Brook low-energy physics facility fail to yield a clear and uniform explanation of the change in computing outlook. One cannot escape the impression that the group was not wildly enthusiastic about the proposed remote linkup in the first place, and that the evident immediate benefits to the group of a second PDP-9 on hand for program debugging and experiment setup while the second machine was running an experiment were irresistible when compared to the future promise of a remote link to the IBM 360/67. The physicists were not anxious to undertake what was expected to be a substantial systems program development task for the coupled system, being unconvinced that the result would be worth the effort. While they still wish to increase the computing power available to them on-site, they have elected to achieve that end by expanding high-speed core storage on their machines, at least until true time-sharing becomes available at the central computing facility.
The coupled system at the University of Manitoba cyclotron is representative of what was intended at Stony Brook. At that installation, the PDP-9 is linked to the computing center's IBM 360/65 by a control unit commercially available from DEC for about $15,000. The unit connects the PDP-9 (or its successor, the PDP-15) directly to a System 360 selector channel, without requiring an additional control unit. The maximum data-transfer rate at Manitoba over a 2000-foot twisted pair cable is 50k bytes/sec. A relatively unsophisticated set of system programs has been written to control communication and transfer of data between the two computers.
The only experiment to which the coupled system (as distinct from the stand-alone use of the PDP-9) has been applied is a p-p bremsstrahlung measurement, where the data are developed in wire spark chambers and plastic scintillation counters. Information from the wire chambers defines proton trajectories, and pulse heights from the counters determine their energies. The PDP-9 first tries to reconstruct a vertex from the proton trajectories. If a point of origin can be determined for the protons to the required accuracy, the relevant coordinates for the proton trajectories and the pulseheights are sent to the IBM 360/65 for full kinematic and statistical analysis of the individual event; otherwise, the event is rejected. The large computer also prepares displays and plots of physical interest that are returned to the PDP-9 for display on the local CRT or output on the localx-yplotter.
The remote computer operates in a multiprogramming rather than in a time-shared environment, with an assigned partition of 65k bytes. Because of the well-designed program overlay feature of the 360/65 operating system, the Manitoba group does not find itself restricted by this relatively small partition. Because of other demands on the computing center, however, they are restricted in the use of this partition to 16 hours/day and 5 days/week. The operation of the coupled system is controlled almost entirely from the PDP-9 teletype, with 360/65 operator intervention required only for initial loading of the partition, off-line printout, and, of course, mounting magnetic tapes at the computing center.
Users of the Manitoba system are pleased with the cooperation and service they have received from the computing center thus far, and they are anticipating no difficulties developing as their demands on the central computing facility increase. But while use of the coupled system for experiments other than that described is clearly possible and desirable, no information was available on plans for the future.
The Brookhaven on-line remote network (Brooknet), where a pair of CDC 6600 machines sharing a common one millionwordextended core storage unit may be interfaced over a high-speed channel to as many as 64 remote data-acquisition computers, can be considered an extreme example of a coupled system. Although the software for Brooknet is reported to be complete and debugged, the system has not yet begun routine operation, and the first remote computer intended for low-energy physics application (a PDP-15) has not yet been delivered. (The only Brooknet user at present is the Chemistry Department, which has a remote batch terminal: teletype, card reader, and printer.)
Why has linking data-acquisition computers directly to computing centers not proved as popular as the obvious advantage of having access to an extremely powerful computer would lead one to expect? There are a number of contributing factors:
1. Since the remote computer can be used only if it is in operating condition and if the necessary personnel are present, the physicist stands to lose some of his independence and flexibility of operation (often not four-shift operation).
2. Most remote computers operate on a multiprogramming basis, hence prompt interrupts are not available. The waiting time for attention might typically be several tenths of a second, therefore the computer in the physics laboratory should be fairly powerful in order to handle the preliminary processing and buffering. With such a computer at work the necessity for fairly rapid access to the large remote machine may entirely disappear, or else the experimenter may be able to store partly processed data on magnetic tape for subsequent further reduction off-line at the computing center.
3. The total amount of time available to one user of a shared-time system per day is always limited. The amount of access time guaranteed by the computing center may not be sufficient.
4. In some cases there is a question of charges, and the total expense of involvement with the computing center may be comparable over a period of several years with the extra cost of buying a sufficiently large local computer for the laboratory to be able to handle all the essential on-line calculations. Even though the calculations may take longer in terms of machine time, they may not require as much lapsed real time if there are stringent limitations on computer center access time.
In this chapter we present a review and an analysis of total expenditures for on-line computing in a large number of laboratories supported by the Atomic Energy Commission and the National Science Foundation through 1968. (Appendix B gives the background for this economic survey.)
Laboratory directors were requested to supply a separate report covering each data-acquisition system currently in use or under construction and, in addition, to supply an estimate of anticipated future requirements for the period 1970-1974. The high-energy field was excluded. Information was also requested on process-control applications, e.g., systems to control accelerator operation or to monitor progress and to execute control functions during the course of an experiment. In every case details were to be supplied regarding the nature and capability of the system and its cost in dollars and manpower during the design, construction, and operation phases.
In all, 46 different systems were reported by 22 different institutions (listed in Appendix B). Berkeley, Brookhaven, and Oak Ridge together reported 21. The various systems range in total cost (including manpower) from about $40,000 to about $1,000,000. Most are in operation, but a few are under construction, and a few others are in the advanced proposal or design stage. Plans for 16 substantial expansions and proposed expansions of existing systems were also reported. There was a wide range of thoroughness of compliance with the request; for example, cost estimates ranged from the most meticulous analyses down to one case where no cost information whatever was supplied. In assessing the reliability and completeness of the data the reviewer concluded that in general the costs of manufactured hardware items such as central processors (CPU's), line printers, card readers, rotating memory devices, etc. shouldbe regarded as reasonably accurate, while estimates of the amount of manpower used, and its cost, seemed much less reliable; in fact, the manpower item was frequently not covered, especially in connection with the preparation of systems software. Whenever a report was more or less complete, and there seemed to be a reasonable good basis for doing so, the reviewer estimated appropriate values for missing items by making use of figures given in more complete reports on similar systems constructed or operated under similar circumstances.
FIGURE 12Breakdown of system for analysis.
FIGURE 12Breakdown of system for analysis.
With regard to labor costs, government laboratory people seem to be in a much better position to supply figures than are university people. The reviewer got the impression that the university respondents have, on the average, a much less clear idea of the dollar value of people's time and a much less clear idea of how to estimate realistically the man-hours consumed by various projects.
Because of the nature of the data the reviewer separated each system into three parts for the purpose of analysis: (1) the data-acquisition central processor (CPU); (2) the standard computer input-output (I/O) devices such as magnetic tapes, disks, card readers, printers; (3) the complete data-acquisition subsystem (DAS). (See Figure 12.) This breakdown has the advantage that the costs of the first two parts of the system are usually fairly accurately known. The cost of the DAS includes the price of all manufactured units closely involved in its assembly, including scalers, ADC's, pulse-height analyzers, and the like (but not detection equipment), together with the expenses associated with all special construction, including engineering, fabrication, and parts. All engineering and fabrication costs associated with the entire system can logically be charged against the DAS,because the CPU and I/O parts, being assembled from standard manufactured items, generally are installed by the manufacturer without much effort or expense on the part of the laboratory personnel. Questions occasionally arose in connection with the assignment of the cost of interfacing the DAS to the CPU. Such costs were assigned to the DAS when the units involved were of a custom-built nature and to the CPU when they weremanufacturer's items incorporated in the computer frame. The very wide range of types of data-acquisition equipment in use necessarily contributes to the spread in DAS costs. Although a number of items of uncertain costs are lumped together in this definition of the DAS, the procedure adopted is believed to have led to a valuable overall picture of the pattern of expenditures.
A fourth item of importance in the analysis is the cost of system software programming. This is almost entirely a manpower item, assuming that program testing and debugging can be carried out without charge for the computer time involved. Here there is considerable uncertainty in the estimates, especially with respect to university installations as well as systems which have been in operation for a long time, e.g., the large system at Argonne.
The total cost of a system is taken to be the sum of the four items listed above, namely, the CPU, the standard I/O system, the DAS, and the system software expenditures. In all likelihood the total costs tend to be too small rather than too large because of incomplete assignments of charges of various sorts, especially manpower. In many cases the totals seem reliable to 10 or 20 percent, while in a few others an error of 30 or even 40 percent would not be surprising.
Table 8 gives a listing of the 27 different types of computers incorporated in the systems reported, together with the number of units of each type mentioned. Of the 27 types, 24 are machines designed with this general sort of application in mind; the exceptional three are the CDC 160A, the CDC 3100, and the IBM 7094. Evidently, the PDP machines are the most popular (24 units), followed by SDS types (8 units), and IBM types (5 units).
Of the 46 system reports, 35 were sufficiently complete to be useful in a detailed analysis. A histogram showing the distribution of these in total cost is given in Figure 13.One immediately sees that few systems cost less than $100,000; in fact only four were reported in this range. However, it must be pointed out that information was solicited regarding only those systems which had cost approximately $50,000 or more. The most common range is $100,000 to $200,000, with 12 examples. The total cost of the system at the Yale Van de Graaff laboratory was not known when the histogram was prepared, but the hardware is reported to cost about $750,000 to duplicate and about $655,000 to copy, so if allowance is made for the cost of developing the software and for other manpower uses the cost would rise substantially. (This system is not one of the 35. The conditions under which the Yale-IBM development are being carried out are so special that manpower costs cannot be assigned on the basis used in other cases.Chapter 2, Section E.)