FIGURE 1Basic data-acquisition system.
FIGURE 1Basic data-acquisition system.
b. A general-purpose system for use in an accelerator laboratory can be assembled from a medium-sized computer, two typewriters, four 13-bit (0-8192) ADC's, six 18-bit (0-262,144) counters, a 10 in. x 10 in. CRO display unit with light pen, two tape drives (for IBM tape), a Calcomp plotter, and a fast paper-tape reader for about $175,000 plus the cost of engineering service and programming. At least one full-time technician-programmer would be needed for maintenance and programming.
c. A large shared-time system of the smallest configuration which makes much sense consists of a large computer with a 16k memory, two typewriters, a fast punched-tape reader, four DEC tapes (or the equivalent), one IBM-compatible magnetic tape, one CRO with light pen, one incremental plotter, input devices for experimental data (ADC's, counters, etc.), plus an interfacing system to link the external input-output devices to the computer. The interfacing system may include a fixed-wired "front end," such as that used at Yale, a small computer, such as that used at Rochester, or both. The hardware would probably cost over $400,000 exclusive of engineering, and to this must be added a large expense for programming, even if the manufacturer supplies a satisfactory shared-time monitor plus all the usual software. Three men would be needed to assemble, maintain, and operate the system: an engineer, technician-programmer, and a full-time programmer, or some equivalent combination, assuming use of the system in a large laboratory with an active and continually developing research program. Thus the cost of this "stripped-down"system must be expected to reach $500,000 before it is in full operation, and the cost of keeping it going, including salaries, overhead, and replacement parts will likely exceed $50,000 per year, although this could perhaps be trimmed somewhat once the system is running. Furthermore, to run efficiently, the system would need additional components: another 16k (at least) of core memory, another IBM-compatible magnetic-tape drive, and a line printer. A rotating memory device would also be helpful. These would raise the cost by well over $100,000. It is apparent that large time-shared systems are so expensive that they can ordinarily be justified only in the largest, most lively research establishments.
In medium and large systems the use of small computers for coupling input and output devices to the main computer offers a number of attractive advantages, especially now that mass production and competition have brought the prices down so low that a large amount of hardware nearly ideal for the purpose is available at a bargain. Some advantages: (1) The small machine can control data acquisition, accumulating blocks of data while the large machine is doing background calculations, interrupting those calculations only occasionally to transfer raw or partially processed data. (2) The small machine can continuously control the monitor CRO. (3) It can control output devices such as a plotter, line printer, rotating memory, or tape drive. (4) It can carry out many logic operations on the incoming data. Experience has shown that such operations are numerous, and from the economic point of view they should not be allowed to tie up the larger machine, which, at the same time, can better be engaged in complicated calculations. In some cases the use of two small satellite computers can easily be justified. The chief disadvantage: Programming can be complicated. However, if the larger machine already has a time-shared monitor which recognizes the small machine as a typical input-output device (as is the case with the PDP-6 plus PDP-8 system at Rochester) the programming problem is not bad.
In the following five sections descriptions of a number of data-acquisition systems of various types and sizes will be given in order to illustrate concretely some practical system configurations. In each case a breakdown of costs and a discussion of the lessons learned in connection with planning, construction, and operation will be included. Thesystems are of the following types: two small, one medium, one large, two multiple-CPU, and one process control.
In 1966, two identical computer systems based on PDP-7 computers were set up in two different locations at the Lawrence Radiation Laboratory (LRL), to be used by several groups of experimenters (see Figure 2). Assembly of the hardware for the first system was completed 6 months after delivery of the computer. Assembly of the second system required only 3 months. Two years after operation commenced, the first satisfactory time-sharing monitor was completed and put into service. The basic use of these systems is pulse-height analysis. In principle many other types of operation are possible.
FIGURE 2PDP-7 data-acquisition system at Lawrence Radiation Laboratory
FIGURE 2PDP-7 data-acquisition system at Lawrence Radiation Laboratory
FIGURE 3A switch panel used for data taking and control of CRT display in conjunction with the PDP-7 computer. The switch-setting codes can be read into the PDP-7 accumulator under program control and are used to select branch points in the program. As many as eight of these units can be connected to the system. The lights are used to indicate program status.
FIGURE 3A switch panel used for data taking and control of CRT display in conjunction with the PDP-7 computer. The switch-setting codes can be read into the PDP-7 accumulator under program control and are used to select branch points in the program. As many as eight of these units can be connected to the system. The lights are used to indicate program status.
Data-reduction jobs currently possible in the shared-time operating mode include spectrum stripping, normalization, smoothing, storage and retrieval of data from magnetic tapes, graph plotting, printout, energy calibration, background fitting, peak integration, and transfer of data from a remote analyzer. Remote control of the computer from up to eight experimental locations is possible using inexpensive switch panels (Figure 3). Remote slave CRT display is also provided. Multiparameter pulse-height acquisition and analysis can be done on a time-shared basis but often requires all the computer's time and memory.
The hardware configuration is shown in Figure 2. The PDP-7 computer was supplied by the Digital Equipment Corporation with an 8k memory (18 bits) extended arithmetic hardware, microtape (Dectape), paper tape, teletype, and cathode-ray tube (CRT). The other items were built or interfaced at LRL.
Automatic memory increment and memory-protection hardware, together with suitable programming, allow a user to carry out simple data-reduction jobs with a live CRT display while two other users are independently acquiring separate, 2048-channel, pulse-height spectra in part of the computer memory, with computer-controlled-gain stabilization. ADC dead time per pulse is less that 40 µsec. Up To 6144 words of the memory can be used for data (one PHA channel per word), while machine language programs fill the remaining 2048 words of memory.
The system works well for pulse-height analysis, but for new applications, e.g., nuclear magnetic resonance magnet control, it needs additional hardware and programs.
Two groups of experimenters, doing chiefly pulse-height analysis experiments are very satisfied with the system. Another group, with a wider range of interests, has been dissatisfied because of the time lag to implement new experiments.
One programmer is now engaged full time preparing more programs.
Lack of free computer time has become a limitation for both users and programmers.
Provision for programming at the outset was inadequate. One full-time systems programmer should have been assigned to these systems for 18 months.
Experimenters need Fortran or similar language capability. A disk, or more core memory, would make this practical.
The memory size is totally inadequate for multiple users because of the large amount of data space needed for the high-resolution spectra now obtainable with Ge(Li) detectors. An external 8k memory is being acquired for data acquisition in each system so that more of the computer memory can be used for computing.
Memory CRT's are needed to provide independent displays for each user. A separate teletype for each user would be invaluable.
A disk memory is needed for rapid overlay of programs and for sorting of multiparameter data.
A "czar" should have been appointed for day-to-day assignment of facilities, consultations with users, and routine maintenance and upkeep of the hardware and programs. The "czar" could be a good electronics technician interested both in programming and in physics.
System costs for the PDP-7 with time-sharing are given in Table 1. Fabrication time is included as a dollar cost. Engineering and programming times shown are one half those charged against two identical systems.
Additional special-purpose experimental equipment commonly used with the system includes gain stabilizers, analog pulse derandomizers, amplifiers, pulse pileup rejection, low-noise preamplifiers, and Ge(Li) detectors.
This second example of a small computer system is also taken from experience at LRL. It was planned in February 1967 and first put into operation in the summer of 1967. Data were first taken with the aid of the system in the spring of 1968, and the system programming was completed in May 1969. The system is used extensively in experiments with the Bevatron.
FIGURE 4PDP-8 data-acquisition system at Lawrence Radiation Laboratory.
FIGURE 4PDP-8 data-acquisition system at Lawrence Radiation Laboratory.
The computer-system hardware consists of the items shown in Figure 4. The PDP-8 has 4k of memory. The disk is a Data Disc unit with a removable disk (128 tracks and a movable head) on the same shaft as a smaller disk and three fixed data heads. Two of the fixed-head tracks are devoted to the display: they drive a hardware-translator continuously. A single display track is used when the number of points does not exceed 2000. For larger displays two tracks are used alternately.
The display is controlled from the switch panel. On-line operating functions may be controlled both from the switch panel and from the teletype.
Three classes of programming have been completed:
System Programs: Symbolic Text Editor, Assembler, a general-purpose library system—all disk oriented.
Data-Taking Programs: These cause the ADC to be read, control elementary sorting, update histograms resident on the disk, write raw data on tape, and monitor the beam. The Bevatron has approximately 1 sec of beam every 6 sec. During a beam pulse the computer is devoted entirely to acquiring data, saving raw data in core, on the disk, and on tape. After a beam pulse, the 13-disk histograms are updated, then the display programs are read into core memory and the display is updated. The system is designed to be capable of accepting over 1000 events per beam burst, and it has met this requirement.
Simple Data-Analysis Programs: These compute displays (linear, log, isometric, and contour—all double precision), read out the sealers, monitor the real-time clock, allow resorting of raw data from tape, and generate tapes for remote plotting.
The system now functions as originally intended and does its job very satisfactorily.
The experimenter relies heavily on the main computer center for data processing.
In assembling this system now, one would buy the disk already interfaced by the computer manufacturer; furthermore the manufacturer now offers programs that would greatly reduce the programming costs.
A memory scope would eliminate the need for a disk-to-CRT display interface.
Less-expensive magnetic tapes are now available. However, it would be better to buy the tape already interfaced by the manufacturer of the computer.
The added cost of buying a 16-or 18-bit computer would have been almost completely offset by savings in the cost of programming.
The addition of a fast printer (e.g., Inktronic $5600) would have paid for itself in time saved during programming but would not be of much use in experiment.
The costs of the PDP-8 are given in Table 2.
FIGURE 5Block diagram of EMR 6130 data-acquisition system at Columbia University.
FIGURE 5Block diagram of EMR 6130 data-acquisition system at Columbia University.
An EMR 6130 computer system has been installed and is being prepared for use with Columbia University's Neutron Velocity Spectrometer data-acquisition and analysis system. The spectrometer is characterized by high data rates and many events per burst. At present, peak arrival rates are approximately 106events per second, with 40-60 events per burst and a burst rate of 70 Hz. The arrival distribution is random; therefore, 10 percent of the interarrival intervals are 100 nsec long, and 1 percent are 10 nsec long. In the future, peak arrival rates of 107events/sec and 400-600 events per burst are possible, with a burst rate of 300 Hz. With an appropriate time-of-flight "front end," the 6130 will be able to handle the anticipated faster rates.
The EMR 6130 is a 16-bit, 775-nsec computer. The memory has a multibus structure which permits each bus to communicate simultaneously with a separate memory module. Up to four memory buses may be purchased. The Columbia system has two memory buses. If a high-speed buffered data channel is used, block transfer may occur at memory cycle speeds. With two buses, data may be stored in two memory modules at rates up to twice memory speed. Alternatively, one bus, channel, and one or more memory modules may be dedicated to data acquisition, while the central processor and standard peripheral devices, using the second bus, simultaneously operate in the remaining memory modules.
A block diagram of the Columbia system is given in Figure 5. The system has three 8k core modules. Memory bus 1 is dedicated to a high-speed channel serving the time-of-flight acquisition system. Memory bus 2 serves both the central processor and a second high-speed channel. Low-speed input-output devices, such as the operator's console, teletype, card reader, and plotter communicate directly through the processor. The high-speed input-output devices, namely, a magnetic tape unit, line printer, fixed head disk, and interactive CRT display, communicate through the channel.
The box designated as "time-of-flight system" represents special-purpose electronics, including a 50-mHz clock, time-quantizingcircuits which "clock" an input event from one of the detectors to the nearest clock pulse following its arrival, a 50-mHz counter, and a 16-word derandomizing buffer capable of storing a new word of data (i.e., arrival time) every 20 nsec. The number of channels, nominally 16,000, is limited not by the front end but by the amount of core available for histogram storage in the 6130 system. (For the high data rates anticipated in the future, the time-of-flight clock speed and derandomizing buffer data acceptance rate will be increased to 100 mHz. At the same time, an accumulating buffer of several hundred words capacity, with a 20-mHz data acceptance rate, will be added to empty the derandomizing buffer and store temporarily the time-of-arrival data prior to its transmission to the 6130 system.)
FIGURE 6Diagram illustrating mode of utilization of core memory in the Columbia System shown in Figure 5.
FIGURE 6Diagram illustrating mode of utilization of core memory in the Columbia System shown in Figure 5.
During the time-of-flight experiment, memory is utilized as follows (see Figure 6). The channel dedicated to data acquisition writes on alternate bursts, into two buffer regions, of approximately 100 words each, in the top of memory module 3. The remaining parts of memory module 3 and all module 2 will be devoted to histogram storage (i.e., time-of-flight channels). Module 1 will contain a stripped-down monitor program and all data-handling programs, including buffer regions for the external devices other than the time-of-flight front end. Programs will be capable of referring to all module 1 or 2 in full concurrency with data acquisition. Reference to module 3 will also overlap data acquisition, except for a period of high input data rate of 100-to 200-µsec duration per burst. With the type of memory allocation described, the system will permit the use of all standard I/O devices, concurrent with the essential operations of input data buffering and histogram generation. Thus, new data may be stored on, or old data retrieved from, the disk or magnetic tape; either new or old data may be displayed on the CRT; and the same or other data may be output with the plotter or line printer. Control information will be input from the teletype, the operator's console, or from special-purpose switches. The importance to the physicist is that hard copy output is immediately available during data acquisition and may be used to monitor, or modify, the experiment.
Subsequent to the input data increase, a high-speed memory incrementing channel will be used to input time-of-flight data directly to the histogram area. With this channel the buffer area in module 3 will no longer be required. Histogram data will be stored in all modules 2 and 3, and no program intervention will be required for histogram generation.
Between data-acquisition runs, the system will be used for data analysis.
The computer, with two memory modules and one channel and bus, was delivered in July 1968. The remaining memory module channel and bus were delivered in the fall of 1968, the CRT arrived in June 1969, and the line printer (which was not purchased from EMR) came shortly afterward. The first time-of-flight run with this system was scheduled for December 1969. During the period from delivery to the first run, one full-timeprogrammer and approximately half the time of one physicist were devoted to the debugging of manufacturer-supplied programs and the writing of the on-line programs required for the run.
It has been hoped that the system would be used extensively for the analysis of previously acquired data, beginning shortly after delivery; however, very little such use has proved possible, essentially because of the unreliability of the 100-cpm card reader supplied by EMR. The lack of a line printer was also a factor. A more reliable reader has been purchased. The delivery of a line printer should rectify the second need.
The development of high-speed, buffered, time-of-flight front ends has been a continuing interest at Columbia. It is therefore difficult to estimate the precise costs of the time-of-flight system developed for use with the 6130. A rough estimate of the design and development time is approximately 3 engineer man years.
Columbia chose to order the EMR 6130, even though at that time (1966) it was not in production, because it seemed a very powerful machine which matched the needs of the system planners. The alternate possibility open was to order a larger, much more expensive machine of proven capability. As it turned out, difficulties in the development of the 6130 caused a delay of over a year in the delivery of the main frame and of over two years in the delivery of the CRT display. (When these delays became apparent, EMR loaned Columbia a 24-bit computer and also a small display for use during the interim period.) The EMR 6130 is perhaps the most powerful 16-bit computer available today, in spite of one or two changes in the original specifications, but in order to get it Columbia apparently traded time for money.
The costs of the Columbia University EMR 6130 system are given in Table 3.
Since early 1966, Yale and IBM Research have been engaged in a joint study in the application of computers to nuclear-data acquisition. The main goal was the production of an integrated hardware-software system which is fully under the control of the experimenter in the sense that he can define his entire data acquisition and analysis process with a Fortran program.
The joint study may be divided into four areas: (1) Development of a suitable general and powerful data-acquisition interface and control unit (front end) with a set of compatible nuclear instrumentation modules (scalers, ADC's, and general-purpose input registers). (2) Development of a suitable display system. (3) Development of a data-acquisition language (as an extension to Fortran) and the necessary library routines to support this language. (4) Development of a general-purpose multiprogramming system for the selected computer (the IBM System/360, Model 44) into which the data-acquisition system could be incorporated.
The first three areas became operational in July 1967, within three months after delivery of the computer, using the standard batch programming system for the 360/44 as a basis. Development continues on the multiprogramming system, which has now reached a state where users inexperienced in using the system can compile and execute Fortran programs, but the data-acquisition components are not yet operational.
Nuclear data are input by means of a general-purpose nuclear-data-acquisition interface and control unit, organized around the concept of an event, an occurrence in the real world which causes the outputs of a group of instruments selected by the user to be read into computer memory. Sixteen independent events are provided for, with each of which may be associated any or all of 30 different instruments (scalers, ADC's, or general-purpose monitor registers) by means of a diode plug board matrix. The instruments themselves, also designed and built by IBM, are modular and completely interchangeable and enable the experimenter to configure his experiment in any way desired, i.e., to determine not only which instruments are to be read but also in what order. Exclusion logic is provided to prevent processing of certain events if and when other defined events occur simultaneously (Figure 7).
The cathode-ray-tube display unit provides a 1024 x 1024 point plotting oscilloscope with seven levels of intensity, character-generation hardware, a light pen, and a programmed function keyboard, by means of which the user can call in programs by pushing buttons. Such programs can perform any function from changing displays gains to curve-fitting. They may be system-supplied or user-written and may be (and usually are) written Fortran. A parallel, high-resolution photographic system permits computer assembly of publication quality illustrations.
The data-acquisition and display-programming system is composed of a group of subroutines which may be called from Fortran programs for performing the various processes in data acquisition and display. For this purpose, a considerable number of additional statements have been added to the Fortran language. These statements perform such functions as defining multidimensional pulse-height analyzers in the computer memory, performing pulse-height analysis using incoming data as channel numbers, and defining separate programs to process each of the 16 classes of input events. All the special statements that make up the new data-acquisition language are implemented by means of a preprocessor which converts them into Fortran coding, which the standard 360/44 Fortran compiler then processes.
FIGURE 7Block diagram of the IBM 360/44 system at Yale.
FIGURE 7Block diagram of the IBM 360/44 system at Yale.
The general-purpose multiprogramming operating system is a multilevel priority system designed to provide access to the system simultaneously by an, in principle, unlimited number of users, each with unique priority. Unlimited means that there is no arbitrary restriction on the number of users; the nth user can always get access if the facilities his particular job requires are not already in use. Two types of user are recognized by the system: The basic unit of execution is the logical user, or task. Each logical user has a unique priority level. Switching between users is carried out as a response to I/O, timer, or external interrupts, at which time the highest priority user in a position to execute gains control of the central processor. The basic unit of memory protection is the physical user, composed of one or more logical users engaged in a common cause. Physical users correspond to real people doing independent work simultaneously. By dividing his work up among a group of logical users, a person may take advantage of the parallel processing capabilities of the system in a natural way. Since logical users within a physical user are not memory-protected against each other, they may communicate rapidly, at full machine speed. Communication between different physical users is also possible, via real or simulated I/O devices. While this system by no means guarantees execution time to any but the highest priority user, it is adequate in a single-experiment environment. The assumption is that the experimenter, who has actual control of the computer at all times, loads his logical users in the order in which he requires their priorities. Following this, other users load their jobs, getting whatever memory and I/O facilities remain. The amount of processor time available to the other users varies inversely with the experimenter's counting rate and the amount of processing he does on his data. In most experiments, the experimenter uses significantly less than 50 percent of the processor time simply because those experiments requiring the sophistication of the computer also have rather low counting rates.A priori, it is estimated that the simple priority algorithm described above is not only adequate but pays a dividend in terms of reduced system overhead time as compared to a more elaborate algorithm. It also guarantees that no data will be lost due to the lower priority users being in the machine.
In general the new system will provide all the facilities of 44PS within the multiprogramming framework, including execution of the Fortran compiler simultaneously with data acquisition.
To the laboratory interested in developing a system of the magnitude of the Yale system, but not a copy, it must be reiterated that neither Yale's out-of-pocket costs nor the cost of copying the system represents the total cost of development. IBM's development costs are not known, but they may be assumed to be very large.
From IBM's viewpoint, the ADC and scaler project is the least successful part of the whole project. Although those instruments are technically excellent, IBM is either unwilling or unable to sell them at a price competitive with the costs of front-end and interface equipment available from the traditional nuclear instrument manufacturers. However, ADC's and scalers available from the traditional sources can easily be interfaced to the front end (whose price is in keeping with its power and versatility). The University of Maryland has followed this procedure.
We therefore present the cost of copying the Yale system by some other laboratory. In Tables 4 and 5 following the example of Maryland, we have not selected IBM ADC's and scalers but rather less expensive components from traditional manufacturers, together with suitable adapters available commercially. The prices shown are to be considered strictly reference numbers and in no way constitute price quotations.
Starting by producing an operational data-acquisition software system running within the standard batch programming system for the 360/44 enabled the system to become operational within three months of the delivery of the computer. This not only enabled it to do useful work almost immediately but also enabled important experience to be gained which is being applied to the development of the multiprogrammed version.
One of the main lessons so far is that a batch-oriented system barely begins to tap the real-time potentials of a computer such as the 360/44. In a batch system, whatever analysis is needed during data acquisition must be somehow tied to the processing of events. If this is not possible, it is necessary to stop data acquisition in order to do analysis even though, on a millisecond time scale, plenty of CPU time is available during acquisition. Multiprogramming software is necessary in order to utilize this available time. This means that multiprogramming not only makes the machine available to several people at a time, but, more important, it makes large amounts of parallel processing power available to the experimenter.
It has also been shown quite conclusively that the ability of the physicist to program his own experiment (in Fortran) gives him enormous power, power which simply would not be available on a suitable time scale if he has to queue up for the services of a system programmer.
While the generalized event structure gives the experimenter considerable ability to deal with complex experimental situations, it has an overhead associated with it which limits it to about 5000 events per second. This is, of course, adequate for all experiments that demand such an event structure. For simple pulse-height analysis, it is unnecessary overhead, but it can be "turned off" in a trivial way, by simply defining the completion of filling of the buffer as an event and calling a special pulse-height-analysis program to process the entire buffer, bypassing the event sorting. This allows for close to 100,000 pulse-height analyses per second.
There are, however, few situations that justify using a computer as powerful as the 360/44 in a manner just described (i.e., doing nothing but simple addition 100,000 times per second). Therefore, such simple experiments will shortly be handled by means of a link between an existing multichannel analyzer and the computer.
The system does not suffer from having the front end directly connected to the 360/44. The data channel on the 360/44 is sufficiently sophisticated so that it performs all the functions that one might relegate to a small CPU placed between the front end and the 360/44, without any interference with the program currently running in the CPU. The one application described above, which does warrant a separate processor, is handled best by attaching the processor to a separate input port rather than by placing it between the front end and the computer. This enables it to do its intended job without acting as a bottleneck in jobs requiring the power of the front end. It also, incidentally, will function as a completely separate data input terminal if two simultaneous terminals should ever be required.
The particular display system employed has worked very well. Because the display list is in the main memory of the computer, programming of light-pen and other manipulative actions is extremely easy, but at the price of large amounts of memory being tied up. It is clear that the system cannot support two such terminals if they are to be truly independent of each other. It is equally clear that the display is as useful in data analysis as it is in data acquisition. A second display terminal is therefore being added. The selected unit (built by Computer Displays, Inc.) is oriented around the Tektronix 611 storage oscilloscope. It provides both alphameric and graphic display, as well as an interactive device (a cursor moved by means of a joystick) for a price of $8000 to $12,000 plus the cost of interfacing to the computer.
At the Rutgers-Bell (RB) nuclear physics laboratory, work has been done with two different two-CPU systems. The first of these represented essentially two duplicate processors (Figure 8), and the second, now in the process of implementation, two processors of different size and capability (Figure 9). While full data are not yet available on the actual performance of the second system, an outline of the experience to date will be given.