KineticSystems ::. Data Acquisition (DAQ)

Technical Notes

A New CAMAC and VXIbus High Performance Highway Interconnect

High performance, distributed data acquisition and control systems based on the CAMAC Serial Highway have been used extensively in the nuclear science field for 20 years. The need for higher I/O system throughput and the increased use of VX Ibus-based instrumentation has led to the development of a new highway interconnect. This interconnect supports CAMAC crates, VXI chassis or a mixture of CAMAC and VXI on a high-speed fiber-optic highway.

A typical highway interconnect system consists of a VMEbus or PCI bus host interface card, a fiber-optic ring configuration, CAMAC crate controllers and/or VXI Slot-0 controllers. The development of a VME-based I/O controller is being considered. Various throughput benchmarks and initial field expe rience are discussed. The crate controller supports the FASTCAMAC protocol being proposed to the NIM Committee. The maximum I/O throughput in FAST-CAMAC mode is 10 megabytes/s.

I. INTRODUCTION

The CAMAC Serial Highway system (IEEE Std. 595) has proven to be a valuable method for implementing multi-crate CAMAC systems since 1975 [1]. These systems can support up to 62 CAMAC crates. Enhanced Serial Highway modes developed by KineticSystems added block transfer capability within a single s erial message [2]. High-performance Serial Highway drivers that included list processors were developed. For example a driver that interfaced with the DRB32 data channel from Digital Equipment Corporation could achieve an effective I/O transfer rate of 2 megabytes/s for 16-bit CAMAC transfers, excluding the overhead of a single I/O call for the entire list [3].

VXIbus is an acronym for the VMEbus eXtensions for Instrumentation [4]. It has gained support from the major instrumentation manufacturers and is becoming a widely accepted standard, mainly for test systems. VXI-based systems are being considered for data acquisition and control applications associated with large physics experiments.

The highway interconnect system described in this paper is based on 20 years’ experience with the CAMAC Serial Highway system and its real-time highway protocol. Operation was further optimized by the use of components and techniques available today. Distributed VXI systems can be configured by using an interconnect Slot-0 controller in each VXI chassis. CAMAC systems can be configured by using an interconnect Crate Controller in each crate.

This highway interconnect system provides a deterministic data throughput up to 10 megabytes/s, supports up to 126 CAMAC or VXI chassis via fiber optics and includes hardware-driven list I/O address processing. Mixed VXI and CAMAC systems can also be configured. This provides an ideal upgrade path for laboratories that have an extensive inventory of CAMAC modules and wish to add VXI-based products. Also, since VXI uses the VME backplane protocol, a VME controller could be developed if sufficient need is found. Another possible future development would be a controller for the VME-P systems that are currently under development by a working group of the NIM Committee [5].

II. BACKGROUND

A common approach to creating distributed systems involves the use of Local Area Networks (LANs). These networks are ideal for the transfer of files or other non-real-time data from one network node to another. However, the timing of the movement of data over these networks is very difficult to predict. There are important classes of applications that require the acquisition of data as well as I/O control with rather high throughput and deterministic system-level response via multiple CAMAC and/or VXI I/O chassis. This highway interconnect system was developed for these applications.

III. GENERAL DESCRIPTION

A host interface card. Host interfaces are available for VMEbus (Model 2961) and PCI, the Peripheral Component Interface (Model 2962).

A fiber-optic "loop." This highway interconnects the host interface and the I/O chassis.

I/O chassis controllers. A CAMAC Crate Controller (Model 3972) and a VXI Slot-0 controller (Model V160) are available.

Various highway protocols were investigated as possible candidates for use in this interconnect. One obvious candidate was the FDDI (Fiber Distributed Da ta Interface) channel [6]. FDDI was developed for the high-speed interchange of data between a group of processors. FDDI is a recognized standard that typically operates at a highway clock rate of 125 Mbits/s and uses fiber optics as a means to transport the data between computers.

After review of the characteristics of FDDI and discussions with several members of the committee that developed the standard, we determined that, while FDDI exhibits a higher throughput than Ethernet, it cannot prov ide the deterministic response needed for this interconnect. For high performance real-time response, the data time "jitter" must be several microseconds or less. Additionally, no other standard highway protocol was found to meet this time-coherent requirement.

Because of the proven success of the CAMAC Serial Highway, the characteristics of that protocol were used as a basis for the interconnect development. Using fiber-optic adapters, CAMAC Serial Highway systems have been operating at a highway c lock rate of 50 Mbits/s. The interconnect host interfaces and I/O controllers described in this paper use TAXI (Transparent Asynchronous Xmitter-receiver Interface) ICs (integrated circuits) from Advanced Micro Devices to convert from parallel to serial words at the highway transmitters as well as from serial to parallel at the highway receivers. The serial data stream is divided into 12-bit "words" with 8 information transfer bits, 2 clock encoding bits, 1 byte-parity bit and 1 flag bit. The TAXI ICs perfo rm the serial encoding/decoding and are mated directly with fiber-optic transmitters and receivers. The fiber-optic loop operates at a 125 Mbit/s clock rate. Fiber-optic cables with core diameters from 50 µm to 100 µm can be used, and each segment can extend to 2 kilometers. The fiber-optic transmitters and receivers are part of the host interfaces and I/O controllers; therefore, separate port adapters are not required.

In order to assure time-coherent I/O response, this highway interconnect system uses a single-master protocol. The computer interface controls the timing of all messages that are transmitted over the highway, except for asynchronous event notification messages that are generated by the I/O controllers and inserted between host-generated messages.

IV. INTERCONNECT FEATURES

In order to provide a 10 Mbyte/s data throughput and deterministic response, no general-purpose processors are used in the host interfaces or in the I/O controllers. Each computer interface supp orts DMA block transfers and contains a hardware-driven list processor for I/O addressing. This list processor supports single operations as well as block transfers within a list. An optional multi-buffer memory allows data from a single I/O scan or many scans to be transferred to the host computer as a single DMA data block. Multi-rate data acquisition is supported with 16 levels of nesting. Both read and write data lists are supported.

In order to attain a highly deterministic response, the CAMAC Crate Controller and Slot-0 controller are slave devices on the interconnect highway. In addition to the list processor in the computer interface, these controllers also contain a hardware-driven list processor. To obtain maximum performance, the list processor in the computer interface would be used to transfer a block of data to or from each CAMAC or VXI chassis, and the list processors in the Crate Controllers or Slot-0 controllers would use downloaded lists to perform the module I/O addressing. B y this method, the highway transfers contain mostly I/O data (not addresses), which maximizes throughput. To provide maximum versatility, the list processor in the computer interface as well as the one in the Slot-0 controller each contain a DSP (digital signal processor). However, the primary list commands bypass the DSP for high throughput and low latency. The DSP is used for higher-level commands, such as branching, as well as for special firmware-driven commands.

The highway protocol initiated b y the computer interface is represented by a command message from the computer interface containing a header with an I/O address as well as other information such as data direction (read or write). As a command message enters a Crate Controller or Slot-0 controller, the controller checks for an address match. If the address does not match, that controller becomes "transparent" and immediately passes the message to the next controller. When the command message reaches the addressed controller, it splits the highway.

If read operations are requested, the controller performs the desired action and applies the resulting read data to the reply message sent to the computer interface. If write operations are requested, the controller strips the write data from the command message, performs the CAMAC or VXI module write operation(s), and sends a reply message to the computer interface. If the addresses of the module registers to be read or written had previously been downloaded to the list memory in the CAMAC or VXI controller, a large number of read or write operations can be enclosed in a single message, with the address information being extracted from the list. Also, there is a list command that supports block transfers. This command includes the block word count and allows the movement of a block of data to or from a memory-mapped module with a single list command.

In a list-driven data acquisition environment the time from the acquisition of a data word from a CAMAC or VXI module to the availabili ty of that data at the computer channel is typically 2 microseconds or less. That time will increase by about 0.2 microseconds for each controller through which the data must pass on its route to the computer interface. An additional 0.1 microseconds must be added for every 100 feet of fiber-optic cable. It is important to note that the variation in this delay from one block to the next is less than one microsecond.

The size of the list memories in the computer interfaces, the CAMAC Crate Con troller and the Slot-0 controller is 32k x 32. Each memory can contain an arbitrarily large number of lists, limited only by memory size. The lists can be downloaded from the host computer using DMA block transfers. Each of the host interfaces uses global memory on the host computer bus as a multibuffer. Multibuffer memories are also available for the Crate Controller (1 or 4 megabytes) and the Slot-0 controller (1, 4 or 8 megabytes). The multibuffers are fully dual-ported. When used with synchronous data a cquisition controlled by the computer interface, a buffer allows the extraction of read data while additional data is being collected. The buffer can be arranged to extract a single block of DMA data for n scans, further reducing the effects of computer overhead. The multibuffers in Slot-0 controllers allow data to be gathered in parallel from a number of VXI mainframes, then transmitted serially at 10 Mbytes/second to the host interface.

For synchronous data acquisition, each i nterconnect host interface contains a clock with programmable ‘tic’ rates. These clock pulses can be used to transmit periodic synchronization messages to the Slot-0 controllers. Methods are available to subtract the effects of delays around the loop as this master clock can be used to synchronize the programmable clocks in the Crate Controllers or Slot-0 controllers. Multi-rate data acquisition is supported with up to 16 levels of nesting.

As stated earlier, the high real-time performance is obtai ned by making the computer interface the source of highway traffic. The only exception to this is the AEN (Asynchronous Event Notification) message. This is a short message that is inserted on the highway to notify the host of asynchronous events such as interrupts. So as not to disturb normal highway traffic, a controller waits until the highway is free or an end-of-message is reached, inserts an elastic FIFO buffer and transmits the AEN message. Once sufficient "no message" bytes are seen, the buffer is r emoved from the highway path. This buffer switching allows the AEN messages to be transmitted between back-to-back messages without causing any highway collisions. This feature is used to respond to a CAMAC LAM (Look At Me) or VXI interrupt request, and the elastic buffer is an extension of the 3-byte delay used in the CAMAC Serial Highway protocol.

The highway protocol is highly efficient, particularly when the Crate Controller or Slot-0 controller lists are used to supply I/O module register addre ss information. In a multi-word read message, for example, there is an overhead for addressing, word count, etc., of 14 bytes plus one parity sum byte for every 256 data bytes. The theoretical block data throughput verses block size is shown in Table 1. This data represents the effective throughput of the interconnect itself and assumes that the CAMAC or VXI modules being written or read can transfer a 16-bit word in 200 nanoseconds. Further, this throughput is exclusive of any computer software overhead. A standard CAMAC Dataway cycle requires about 1 microsecond. The proposed FASTCAMAC protocol with multiple strobes should achieve a higher block rate during a DMA operation [7]. Actual throughput experience is described later in this paper.

Table 1. Theoretical block throughput vs. block size

Data Block Size (bytes)

Message Time (µs)

Theoretical Throughput (Mbytes/s)

1

1.5

0.66

10

2.4

4.16

100

11.4

8.77

1,000

101.8

9.48

10,000

1,005.4

9.82

100,000

10,041.4

9.96

1,000,000

100,401.4

9.96

V. OPERATIONAL EXPERIENCE

Benchmark tests have been run to determine the actual throughput obtained with the interconnect under a variety of operating conditions. The results are summarized in Table 2.

Table 2. Interconnect benchmark DMA throughput
Benchmark Target Module	Measured Throughput (Mbytes/s)
3516 CAMAC module, 16-bit transfers	2.0
3516 CAMAC module, 24-bit transfers	3.0
FAST 3516 module, 16-bit transfers	5.0
FAST 3516 module, 24-bit transfers	7.5
V207 VXI module, VXI block mode	5.0
V207 VXI module, VXI non-block mode	3.1
V200 VXI module, VXI block mode	9.2

These benchmarks were derived using a 2961 VME host interface, a 68030 processor, and a DMA block size of 10,000 bytes. The DMA block times were measured with a VME bus analyzer from the time when the host computer executed a start command to the 2961 until a block of data was in host memory (for I/O write operations) or the transfer was complete for I/O read operations. Similar results were obtained for both read and write blocks. If the command list resides in the controller list memory, no difference in throughput is experienced between the transfer of a block of data to or from a single module address and a list of I/O addresses. The CAMAC throughput is limited by Dataway cycle timing, while the VXI throughput is limited by the handshake timing of the particular module being used. Results were found using the 2962 PCI interface with a Pentium host computer. For both host platforms, the DMA performance approached the theoretical 10 m egabytes/s when reading or writing the multibuffer inside the CAMAC or VXI interconnect controller.

The "FAST" 3516 module was modified to support the FASTCAMAC protocol being proposed to the NIM Committee. That protocol, as used here, allows multiple S1 strobes within a single CAMAC cycle. Since that protocol was proposed after the interconnect Crate Controller was started, it is not fully optimized for FASTCAMAC. We anticipate that future changes to the controller could result in a DMA throughput approaching 10 megabytes/s.

The throughput just described covers the total time for a DMA cycle, but does not include the computer overhead to execute such a cycle. This interconnect is designed to optimize DMA block size by the use of the list processors. In a data acquisition sequence, data can be acquired from a large number of input channels associated with many I/O chassis at DMA rates similar to those in Table 2. The software I/O setup time for a DMA block varies widely. For a 68040 running th e VxWorks real-time kernel, this time is about 10 microseconds, and for an Alpha server running Digital UNIX, it ranges from 50 to 100 microseconds. Assuming a large-block DMA transfer rate of 9.2 megabytes/s and various I/O overhead values, effective throughputs were obtained as shown in Table 3 for various block sizes.

Table 3. Effective block throughput with host overhead
Block (bytes)	Mb/s w/ 10 us OHD	Mb/s w/ 50 us OHD	Mb/s w/ 100 us OHD
1	0.087	0.019	0.010
10	0.801	0.191	0.098
100	4.490	1.606	0.891
1,000	8.327	6.246	4.760
10,000	9.105	8.785	8.415
100,000	9.190	9.157	9.115
1,000,000	9.199	9.196	9.191

As can be seen from the above table, computer software overhead can have a significant effect on effective throughput for small block sizes. Where high effective throughput is needed, every attempt should be made to use the list processors and combine as many operations as possible from all I/O chassis in the system into a single block. In the case of synchronous data acquisition, the multibuffer should be used to store the results of a number of scans and present that i nformation as a single DMA block to the host computer.

VI. SOFTWARE

Software drivers for this highway interconnect have been developed for use on Alpha-based processors from Digital Equipment Corporation, running the Digital UNIX and OpenVMS operating system. Drivers for Windows NT are currently being developed. The software is written in a modular fashion to reduce the effort to support other software operating systems and hardware platforms. The CAMAC protocol is such that a single host dr iver is used for the entire I/O system. For VXI, however, instrument drivers are provided by the manufacturer for each module. To enhance the ease of use, particularly when I/O modules from a number of vendors are configured in a system, the VXI software drivers for the host interfaces are written to conform with standards developed by the VXIplug&play Systems Alliance [8].

This interconnect has been combined with Reality, our data acquisition implementation of EPICS (Experimental Ph ysics Industrial Control Software) obtained under license from the Department of Energy (DOE) [9]. The standard EPICS configuration, as used for accelerator control, uses host UNIX workstations connected by Ethernet to 680xx-based controllers in the I/O chassis performing the real-time tasks. For synchronous data acquisition, we are using a small VME chassis with a single 68040 or 68060 processor running VxWorks and our VME interconnect host interface to replace the embedded controllers [10]. VxWo rks software drivers have been developed, and this combination has been successfully implemented in the field.

VII. CONCLUSIONS

The interconnect products described in this paper were developed to maximize effective I/O throughput and to provide a high degree of time-coherence. The important features were shown to be:

A distributed fiber-optic loop that can interconnect up to 126 CAMAC crates and/or VXI mainframes with up to 2 kilometers between chassis.

A single-master protoc ol that produces I/O throughput of up to 10 megabytes/s with a latency of about 2 microseconds and a delay jitter of less than 1 microsecond.

Hardware-driven list processors that act as data movement "engines" to remove the addressing burden from the host computer.

Multibuffer memories that smooth the data flow and can be applied to increase system-level throughput. For data acquisition, all of the data from one scan or many scans can be transferred to the host as a single DMA block.

Programmable clocks to control synchronous data acquisition.

Compatibility with the FASTCAMAC standard being proposed.

Host software drivers to simplify the programming environment.

Benchmark tests show that a DMA block throughput of 3 megabytes/s can be achieved for standard CAMAC and 7.5 megabytes/s for FASTCAMAC. Also, a rate of 9.2 megabytes/s was achieved with a VXI module having good handshake response. The effective overall throughput must also include any computer op erating system overhead to set up the DMA operation. For applications that require high throughput, it is important to maximize the DMA block size.

This interconnect has been in service with a number of users for over a year. The field experience has been quite good with operation as anticipated.

VIII. REFERENCES

[1] IEEE-595, "CAMAC Serial Highway," Institute of Electrical and Electronic Engineers, 445 Hoes Lane, Piscataway, NJ 08855.

[2] R. T. Cleary, "Enhanced CAMAC Seri al Highway System," Nuclear Science Symposium, San Francisco, CA, Oct. 24, 1985.

[3] R. T. Cleary, "A High Speed CAMAC Interface using the DRB32," DECUS Symposium, Anaheim, CA, Oct. 20, 1988.

[4] IEEE-1155, "IEEE Standard for VMEbus Extensions for Instrumentation: VXIbus," Institute of Electrical and Electronic Engineers, 445 Hoes Lane, Piscataway, NJ 08855.

[5] Edward J. Barsotti, "The new VME64 Extensions Standard, Related VSO and IEEE Standards & VME International Physics Ass ociation (VIPA) Activities," Nuclear Science Symposium, Anaheim, CA, Nov. 3, 1996.

[6] Jay A. Sumana, "FDDI and FDDI-II, Architecture, Protocols and Performance," Artech House, 1994.

[7] S. Dahawan, "The proposal for a FASTCAMAC Standard,"Nuclear Science Symposium, Anaheim, CA, Nov. 3, 1996.

[8] VXIplug&play Systems Alliance, http://www.vxipnp.org.

[9] "EPICS, a Brief Description," Los Alamos National Lab, http://www.atdiv.lanl.gov/epics_description.htm l

[10] R. T. Cleary, "New Standards-based Software Enhances Real-time I/O Performance," AUTOTESTCON, Dayton, OH, September 17, 1996