ABSTRACT
A relatively inexpensive 16 Gbps data-recording system based on commercial off-the-shelf technology and open-source software has recently been developed. Combining this recorder with the parallel development of broadband Very Long Baseline Interferometer (VLBI) instrumentation is enabling dramatically improved sensitivity for both astronomical and geodetic VLBI. In this article, we describe the VLBI system and the results of a demonstration experiment that illustrates a number of cutting-edge technologies that can be deployed in the near future to significantly enhance the power of the VLBI technique.
Export citation and abstract BibTeX RIS
1. INTRODUCTION
The technique of Very Long Baseline Interferometry (VLBI) makes use of many globally distributed radio telescopes both to obtain high-resolution images of distant celestial objects and to measure the shape of the Earth and its orientation in space. Because the sources of radio emission are so weak, extremely sensitive receiving systems are required. This sensitivity can be improved by increasing the size of the radio antennas, reducing the noise contribution from the receivers, or, for continuum-frequency observations, increasing the recorded data rate. For existing antennas, or when contemplating the economics of new antenna systems, greater continuum sensitivity is often best achieved by increasing the observing bandwidth, which translates directly into higher data rate. In this article, we report a single-baseline dual-polarization VLBI experiment with a 2 GHz aggregate-bandwidth in which data were recorded at 16 Gbits s-1 at each station. This demonstration illustrates a number of cutting-edge technologies that can be deployed in the near future to enhance the sensitivity of astronomical and geodetic VLBI observing arrays.
In contrast to the current and recent generations of recorders used for VLBI, which are based on proprietary hardware components and controlling firmware, the new Mark 6 system uses commercial-off-the-shelf (COTS) technology and open-source software. The high data rate is achieved by adopting modern high-performance motherboards and components writing simultaneously to 32 conventional magnetic disks.
2. APPLICATIONS
2.1. Astronomy
As an astronomical imaging technique, VLBI is unrivaled for providing high angular resolution, as well as for providing a means to study high-brightness-temperature cosmic phenomena with exceptional detail. At 2 cm wavelength on intercontinental baselines (5000 km) resolutions of 0.8 mas are achieved, and at 1.3 mm wavelength on similar baselines a resolution of 50 μ as can be obtained. (The resolution is given by λ/D; where λ is the observing wavelength and D is the projected baseline length.) The sensitivity of VLBI arrays is proportional to B1/2 (B is total recorded bandwidth), and historically the bandwidth limit has been set by the capability of VLBI backends and recorders, not by radio telescope receivers, which typically have bandwidths of several gigahertz. By contrast, the data capture rate of VLBI recorders was for many years limited by magnetic tape technology to 512 Mb s-1 (Whitney 1996), which corresponds to 128 MHz of bandwidth (Nyquist sampled, 2 bits sample-1). The adoption of hard disk storage media and industry-standard hardware has begun to successfully address this mismatch in bandwidth between the telescope front end and VLBI instrumentation (Whitney et al. 2010), enabling dramatic sensitivity improvements in VLBI networks including the Very Long Baseline Array (Walker et al. 2007). These efforts have led to advances in astrophysical research that require both high angular resolution and high sensitivity, including detection of the faint afterglows of gamma-ray bursts (Pihlström et al. 2007), searches for active galactic nuclei (AGN) in starburst galaxies (Alexandroff et al. 2012), and investigation of the radio-quiet population of AGN (Bontempi et al. 2012).
For VLBI observations at the shortest wavelengths (≤ 1.3 mm) continued expansion of recorded bandwidth is particularly useful and critical for several important science objectives. Because of surface accuracy requirements, radio telescopes are typically smaller at these short wavelengths than at centimeter wavelengths, and increased bandwidth compensates for the smaller collecting area to some extent. In addition, atmospheric turbulence often limits the VLBI integration time at these very short millimeter wavelengths to typically ≲10 s, so that recording a wide bandwidth is essential in order to allow detection of faint sources (Rogers et al. 1995). State-of-the-art millimeter and submillimeter receivers have bandwidths of ≳4 GHz, requiring VLBI recorders that can reach sustained data rates of order 16 Gb s-1 in order to achieve maximum sensitivity.
Of particular scientific interest in this short-wavelength regime are VLBI studies of nearby supermassive black holes for which emission near the event horizon can be resolved and imaged. For SgrA*, the four million solar mass black hole at the Milky Way center (Reid 2009; Genzel et al. 2010), the event horizon subtends an angle of 10 μ as, presenting us with the opportunity to observe general relativistic (GR) effects in a strong gravity regime. Recent 1.3 mm VLBI observations have detected structures on the scale of a few Schwarzschild radii in the direction of SgrA* which can be interpreted in the context of strong gravitational lensing at the event horizon (Doeleman et al. 2008; Fish et al. 2011; Broderick et al. 2009). For M87, the giant elliptical galaxy whose core harbors a ∼6.2 billion solar mass black hole, 1.3 mm VLBI has been used to detect a similar sized structure (∼5 Schwarzschild radii) at the base of the relativistic jet produced at the galaxy's core. The size of this structure, which is on the same scale as the innermost stable circular orbit (ISCO) of the black hole, has now set limits on the black hole spin and orbital direction of the accretion disk (Doeleman et al. 2012). Extension of this work towards true imaging of these sources, which would allow tests of GR (Johannsen & Psaltis 2010; Johannsen et al. 2012), requires the higher bandwidth systems described herein.
2.2. Geodesy
Geodesy is the study of the shape and rotation of the Earth and their changes with time. VLBI provides the most accurate measurements of the shape of the Earth on a global scale. Twice-weekly VLBI observations using a global network coordinated by the International VLBI Service for Geodesy and Astrometry (Schlüter & Behrend 2007), in combination with the satellite laser ranging network and the Global Navigation Satellite System network of antennas, provide the International Terrestrial Reference Frame (ITRF) to an accuracy of a few centimeters or better, with a goal in the few-millimeter range for the next-generation VLBI2010 system currently under development (Petrachenko et al. 2009). The ITRF has become the basis for the civil surveying infrastructure in most countries, replacing less accurate country-wide or continent-wide reference systems that on occasion led to embarrassing inconsistencies, such as bridges that did not meet at boundaries. The IVS observations are used to determine (1) UT1 minus UTC, which allows the maintenance of civil time at the accuracy needed by GPS for greatest accuracy, (2) the International Celestial Reference Frame, composed of the positions of several thousand extragalactic radio sources, and (3) nutation and precession of the Earth's spin axis, which tells us much about the structure of the Earth's core (Koot et al. 2010).
Geodetic VLBI requires observations of several hundred extragalactic broadband radio sources distributed as uniformly around the sky as possible (Schuh & Behrend 2012). Fast-slewing antennas are needed to rapidly move around the sky, encouraging the adoption of smaller antennas for economic reasons. To make up for the lower sensitivity resulting from the smaller collecting area, a higher VLBI data rate is needed to obtain the same sensitivity. To increase the uniformity of the source distribution generally requires making use of weaker candidate sources, which also encourages adoption of the highest possible data rate.
While the antenna area, receiver temperature, and data rate determine the sensitivity per unit time, the sensitivity per "observation" can generally be increased by integrating for a longer time. However, for both of the applications that have provided the driver for the development of the new Mark 6 data-acquisition system, the integration time is limited. For astronomical observations made at frequencies of several hundred gigahertz, atmosphere turbulence typically limits coherent integrations to no more than a few tens of seconds, depending on weather conditions. For geodetic applications, studies made in preparing the specifications for the VLBI2010 system demonstrated that an observing rate of one or two scans per minute is desirable, which, with the move time required to get from source to source, provides a limit of ∼10 s on source per scan for the fully developed system.
2.3. General
For both astronomy and geodesy applications, ultrawideband VLBI observations may pose challenges to high-fidelity imaging for sources with large spectral indices, particularly when combined with frequency-dependent source structure. This is especially relevant when the observation bandwidth is a large fraction of the median observing frequency, such as is planned for geodetic observations with the VLBI2010 system. Techniques for dealing with these issues are important but are outside the scope of this article.
3. HIGH-LEVEL SYSTEM DESCRIPTION
The observations reported here were made in June 2012 using a 12 m antenna recently installed at the Goddard Geophysical and Astronomical Observatory (GGAO), Goddard Space Flight Center, Maryland, and the 18 m Westford antenna at Haystack Observatory, Massachusetts; both antennas are part of the global geodetic VLBI network. The signal chain for each of these antennas (Fig. 1) consists of (1) a broadband dual linear polarization feed; (2) a broadband low-noise amplifier (LNA) for each polarization; (3) four-way splitters for each polarization; (4) four frequency converters for translation of the signals from radio frequency (RF) to intermediate frequency bands; (5) two digital backend units (Niell et al. 2010), and (6) the 16 Gbps Mark 6 digital recorder. The signal chain also includes a phase-calibration system which was not used in the experiments described in this article.
Fig. 1.— Diagram of 2 GHz bandwidth dual-polarization system (4 GHz aggregate bandwidth) which records to the Mark 6 data recorder at 16 Gbps.
Except for the feeds, the receiving system is the same for each of the antennas. For the 12 m antenna the feed is a quadruple-ridged flared horn (Akgiray et al. 2011), while for the Westford antenna an ETS-Lindgren 3164-05 quad-ridge feed was used. For both systems, the feed and LNAs are cooled to approximately 25 K in a dewar. The LNAs for both systems are Caltech 2, 12 GHz single-ended monolithic microwave integrated circuits (MMICs). Figure 2 displays the typical system temperature performance of the receiver front-ends installed on the Westford and GGAO antennas. Note that the system temperature stays below ∼50 K over this entire range, except for some regions of radio interference near 2 and 6 GHz.
Fig. 2.— Receiver/frontend Tsys for horizontal and vertical polarizations as a function of frequency over the continuous band 2–14 GHz; regions of interference are visible near 2 and 6 GHz. Measurements were made on ground (off-antenna) with receiver pointing vertically; on-antenna Tsys is somewhat higher due to ground pickup. The shaded area indicates the 2 GHz frequency range over which observations reported in this article were conducted.
The RF for the two linear polarizations is brought to the control room on optical fibers. In the control room each polarization is split into four paths. A pair of polarizations is fed to each of four up/down converters (UDCs) which translate both polarizations from four different RF bands to an IF in the range 0.5–2.5 GHz. For these observations, the four 512 MHz bands were selected to span the RF range 8592–10,640 MHz. Anti-aliasing filters in the output path of each UDC select a subset of the IF range from 512 to 1024 MHz corresponding to the second Nyquist zone of the digital back ends which follow.
Each DBE accepts the two polarizations from two bands, quantizes all four 512 MHz bands to two-bit samples, time-tags and formats the data into VLBI data interchange format (VDIF) format (Whitney et al. 2009), and outputs the formatted data to two separate 10 Gigabit Ethernet datastreams at 4 Gbps each, one for each band. The four Ethernet 4 Gbps streams (two from each DBE) are fed to the Mark 6 recorder through two dual-port 10 Gigabit Ethernet network interface cards.
3.1. Up/Down Converter
The up/down converter (UDC) translates an arbitrarily selected 2-GHz-wide slice of the RF spectrum to IF. The frequency translation is accomplished in two mixing stages (Fig. 3): an initial upconversion about a tunable LO frequency is followed by a downconversion about a fixed frequency. The output IF passband is defined by the bandpass filter between the two mixers. By varying the first LO frequency, the spectral slice of the RF signal translated to IF can be varied. The downconversion is net upper sideband.
Fig. 3.— Simplified block diagram of UpDown converter (UDC).
The advantages of the UDC architecture include wide input and output frequency ranges, flexibility in tuning, and excellent image rejection. In the current implementation the rejection of the image and other unwanted responses is >70 dB. High-rejection is particularly desirable when the RF input signal is broadband and has strong RFI at some frequencies.
3.2. Digital Backend
The wide-band single-band ROACH digital backend (RDBE-S) (see Fig. 4) utilizes a Casper ROACH (Reconfigurable Open Architecture Computing Hardware) board (Parsons et al. 2008) and supports two dual-input analog-to-digital converter (iADC) cards covering 2048 MHz of spectrum. The FPGA personality accepts four 512 MHz analog bands, quantizes to 2 bits sample-1, and outputs an aggregate of 8 Gbps of data through two 10 Gigabit Ethernet ports. The output data are encapsulated in a UDP/IP payload for either long-haul transmission or transmission to a local storage repository. The RDBE-S utilizes a standard 1-Gigabit Ethernet interface for control and monitoring.
Fig. 4.— Block diagram of RDBE-S digital backend unit; large outline encloses FPGA-based ROACH board. Four 512 MHz bandwidth input IFs are converted into two 4 Gbps dual-threaded VDIF format data threads.
A separate FPGA personality (RDBE-H) can be used to configure the same ROACH hardware to process each of two independent 512-MHz-wide analog bands through a polyphase-filter-bank to produce 32 MHz bandwidth bands more suited to geodetic-VLBI requirements.
3.3. Mark 6 Recorder
The Mark 6 VLBI data system is a joint collaborative development effort among MIT Haystack Observatory, NASA/GSFC High-End Network Computing group, and the Conduant Corporation, and is designed to support a sustained 16 Gbps data rate written to an array of disks. The hardware components were carefully selected for high performance and for compatibility with other system elements and use only COTS technology. The system operates under a fully open source Debian Linux distribution with the application software (also open source) written primarily in C++/C for the data plane and in Python for the command and control interface.
The input to the Mark 6 can accommodate up to four 10 GigE data ports, each operating independently at up to ∼7 Gbps, with a maximum aggregate rate of ∼16 Gbps. The system supports SATA-interface disks packaged into modules (8 disks per module) that are inserted into the Mark 6 chassis and connected to the disk controllers via COTS external SATA cables. Each external SATA cable supports four disks, so that each module requires the connection of two such cables. Depending on the record data rate, different numbers of simultaneously operating disk modules are required. A single 8-disk module with modern disks will support 4 Gbps of continuous recording; two modules (16 disks) are required for 8 Gbps, 4 modules (32 disks) are required for 16 Gbps. A prototype Mark 6 system is shown in Figure 5.
Fig. 5.— Photograph of prototype Mark 6 system.
VLBI has always pushed the technology of digital recording to the highest possible data rates primarily due to the fact that, for the majority of VLBI observations, the achievable signal-to-noise ratio of a given system increases as the square root of the recording bandwidth. Figure 6a shows the evolution of VLBI recording capability from the origins of VLBI in the late 1960s through the present. Over this period, the record data-rate capability has increased by more than four orders of magnitude from less than 1 Mbps in 1967 (magnetic tape) to 16 Gbps today (magnetic disks). At the same time, as shown in Figure 6b, the cost per Gbps of capability has dropped by almost five orders of magnitude and moved from highly proprietary tape-based systems to semiproprietary disk-based systems (Mark 5 series), and now to the current 16 Gbps Mark 6 system which utilizes almost fully COTS hardware with specially developed open-source software.
Fig. 6.— (a) Evolution of VLBI recording-rate capability from 1967 to 2012, progressing more than 4 orders-of-magnitude from original 1967 magnetic tape to modern magnetic disks, (b) parallel evolution of cost of recording in k$ per Gbps, which has dropped by almost 5 orders-of-magnitude during the same period.
4. OBSERVATIONS
Several well known sources of differing correlated flux density were observed with the dual-polarization (horizontal/vertical)4 receiver system at each of the two sites (GGAO and Westford). The UDC frequencies were set to span the RF band from ∼8 to 10 GHz (see Fig. 2) in four adjacent 512 MHz bands. The programmable UDC attenuators were set manually to obtain approximately the proper level required for the samplers in the DBEs; a second level of gain adjustment, done for each scan, was applied digitally in the RDBE-S to achieve close-to-nominal sample-state statistics. Data recording was done using the Mark 6 system via four 4 Gbps Ethernet datastreams with Ethernet packet size of 8192 data bytes, plus 16 bytes of identifying header, in standard VDIF format; the aggregate 16 Gbps datastream at each station (2 GHz bandwidth in each of two polarizations) was recorded to four Mark 6 disk modules, each populated by eight standard 3.5 inch magnetic disks.
Observation sessions were scheduled for 2012 May 17–18 and 2012 June 19. Unfortunately, the May set of observations suffered from an equipment failure at the GGAO station that limited good data to be produced from only six of the planned eight 512 MHz bands. The June observations were cut short after less than an hour by an emergency need to put Westford back into normal geodetic-VLBI service, so the only observation suitable for processing was on 3C84, the first source on the schedule.
After completion of the observations the data from both telescopes were electronically transferred to Haystack Observatory for processing.
4.1. Correlation and Fringe Fitting
The data were crosscorrelated on a 'DiFX' software-based correlation system (Deller et al. 2011), processing horizontal-to-horizontal and vertical-to-vertical polarizations between the two stations. Four passes through the correlator, one for each IF band, were required to process the complete dataset.
Following correlation, the outputs from all four passes were transformed into a format compatible with the Haystack-developed fourfit fringe-fitting software. This fringe-fitting processing was unconventional and challenging due to the very large bandwidth and to the requirement to coherently combine the constituent frequency bands in both polarizations. Relative phase adjustments between correlated bands were determined manually in order to coherently combine the four bands in each polarization, then again manually to combine the two polarizations. In the future, for the VLBI2010 system (operating at centimeter wavelengths), this phase adjustment will be done automatically by measuring the phases of weak phase-calibration tones injected into the receiver; for millimeter VLBI observations, the generation of proper calibration tones is difficult, and relative instrumental phases are sometimes more robustly determined by phase-referencing to a nearby stronger source.
5. RESULTS
Figures 7 and 8 show the results of a 16 Gbps station-1 observation of 3C84 of 10 s duration made on 2012 June 19. Although there was no ability to do careful calibration of the system, the correlation amplitude and signal-to-noise-ratio of the scan (Fig. 7) are within the expected range. The time-segmented band-by-band amplitude and residual-phase data (Fig. 8) also appear nominal.
Fig. 7.— Correlation amplitude vs. residual delay rate (nanoseconds per second) from 10 s observation of 3C84 at 16 Gbps per station on 2012 June 19. Correlation amplitude is ∼5.5 × 10-3 (units in plot are 10-4) with a signal-to-noise ratio of ∼940; HH and VV cross-correlations for all four 512 MHz bands were combined coherently for this result.
Fig. 8.— Plot of correlation amplitude (lower traces, in blue) and residual phase (upper traces, in red) vs. time for each of the four 512-MHz-bandwidth bands ("a" through "d") and vector sum ("All") over the 10-s duration of the observation.
Although the results of the 2012 May 17–18 observations were somewhat compromised by the failure of one of the UDC units, 16 Gbps were recorded for a full 60 s on the Mark 6 system at each station, though only 12 Gbps of the recorded 16 Gbps could be processed to obtain fringes on the weak (∼0.2 Jy) source 0550 + 356 (Figs. 9 and 10); as in the 3C84 observation above, only HH and VV correlations were done. Manual band-to-band adjustment phases for this source were obtained from observations of the nearby brighter source 0552 + 398 (7 Jy at 5 GHz).
Fig. 9.— Results from 60 s observation of 0550 + 356 on 2012 May 18 showing correlation amplitude as function of residual delay rate in nanoseconds per second (scale at bottom). Correlation amplitude at peak is ∼4.0 × 10-5 with a signal-to-noise ratio of ∼14; HH and VV cross-correlation for the three good bands were combined coherently for this result.
Fig. 10.— Plot of correlation amplitude (blue, connected dots) and residual phase (red, unconnected dots) vs. time for each for each of the three 512-MHz-bandwidth bands ("a", "c", "d") and vector sum ("All") over the 60-s duration of the observation.
6. PROSPECTS, PLANS, AND FUTURE WORK
The 16 Gigagit s-1 system used in these experiments, as well as some components of the broadband receiver system, are still in a process of maturation, with work continuing in the following areas:
- 1.The Mark 6 prototype system used in these experiments is an early RAID0 implementation and is susceptible to "slow" disks impeding the sustainable record rate. For some applications, such as geodetic VLBI that tend to use sub-30-second scans, the RAM buffers supply sufficient capacity in the case that the sustained disk-writing rate is lower than the RAM capture rate, but may not work well for longer sustained observations. Work is currently in progress to develop a custom (but open source) file system that is relatively tolerant of one or a few disks not recording fully to speed, allowing the system to sustain a high throughput rate under these conditions.
- 2.A phase-calibration system has been developed that injects very short weak pulses into the broadband receiver to calibrate the relative phases of the individually-correlated frequency bands so they can be coherently combined without manual intervention. This is in use for all geodetic observations.
Chopo Ma of NASA/GSFC has been an unflagging supporter of the development of the suite of technologies necessary for the broadband VLBI system. The Mark 6 recording system development has been aided by discussions with Bill Fink, Pat Gary (deceased), and Paul Lang at the NASA/GSFC High End Computer Network group and by development of specific disk-module-handling hardware at Conduant Corporation. The development of the RDBE-S digital backend unit was a collaborative effort with the ROACH (Reconfigurable Open Architecture Computing Hardware) group at UC Berkeley, the National Radio Astronomy Observatory, and the MeerKAT development group in Capetown, South Africa; embedded software development for the RDBE-S was aided by Mikael Taveniku of XCube, Inc. The DiFX correlator originated with Adam Deller, then at Swinburne University of Technology in Melbourne, Australia, with work carried forward by the international DiFX consortium of developers. The development of the broadband RF systems includes contributions by Dave Fields, Michael Poirier, and Bruce Whittier (deceased) at MIT Haystack Observatory, Sander Weinreb, Hamdi Mani, and Ahmed Akgiray of Caltech and the Jet Propulsion Laboratory, and Wendy Avelar at ITT Exelis. Modifications of the control/monitor software to support these observations were done by Ed Himwich of NVI Inc.
Footnotes
- 4
Normally, circular polarization is preferred for VLBI observations. However, these observations were made with a "broadband system" that continuously spans ∼2–14 GHz with a single antenna feedhorn. Feedhorns supporting these very large fractional bandwidths currently exist only for linear polarization. For millimeter-VLBI observations, suitable circular-polarization systems are normally used. Depending on the application, correlation of all four possible polarization pairs may be necessary.