GSB

The GMRT software backend (GSB) file format is the standard output of the initial correlator of the Giant Metrewave Radio Telescope (GMRT). The GSB design is described by Roy et al. (2010, Exper. Astron. 28:25-60) with further specifications and operating procedures given on the relevant GMRT/GSB pages.

File Structure

A GSB dataset consists of an ASCII file with a sequence of headers, and one or more accompanying binary data files. Each line in the header and its corresponding data comprise a data frame, though these do not have explicit divisions in the data files.

Baseband currently supports two forms of GSB data: rawdump, for storing real-valued raw voltage timestreams, and phased, for storing complex pre-channelized data from the GMRT in phased array baseband mode.

Data in rawdump format is stored in a binary file representing the voltage stream from one polarization of a single dish. Each such file is accompanied by a header file which contains GPS timestamps, in the form:

YYYY MM DD HH MM SS 0.SSSSSSSSS

In the default rawdump observing setup, samples are recorded at a rate of 33.3333… megasamples per second (Msps). Each sample is 4 bits in size, and two samples are grouped into bytes such that the oldest sample occupies the least significant bit. Each frame consists of 4 megabytes of data, or \(2^{23}\), samples; as such, the timespan of one frame is exactly 0.25165824 s.

Data in phased format is normally spread over four binary files and one accompanying header file. The binary files come in two pairs, one for each polarization, with the pair contain the first and second half of the data of each frame.

When recording GSB in phased array voltage beam (ie. baseband) mode, the “raw”, or pre-channelized, sample rate is either 33.3333… Msps at 8 bits per sample or 66.6666… Msps at 4 bits per sample (in the latter case, sample bit-ordering is the same as for rawdump). Channelization via fast Fourier transform sets the channelized complete sample rate to the raw rate divided by \(2N_\mathrm{F}\), where \(N_\mathrm{F}\) is the number of Fourier channels (either 256 or 512). The timespan of one frame is 0.25165824 s, and one frame is 8 megabytes in size, for either raw sample rate.

The phased header’s structure is:

<PC TIME> <GPS TIME> <SEQ NUMBER> <MEM BLOCK>

where <PC TIME> and <GPS TIME> are the less accurate computer-based and exact GPS-based timestamps, respectively, with the same format as the rawdump timestamp; <SEQ NUMBER> is the frame number; and <MEM BLOCK> a redundant modulo-8 shared memory block number.

Usage Notes

This section covers reading and writing GSB files with Baseband; general usage is covered in the Getting Started section. While Baseband features the general baseband.open and baseband.file_info functions, these cannot read GSB binary files without the accompanying timestamp file (at which point it is obvious the files are GSB). baseband.file_info, however, can be used on the timestamp file to determine if it is in rawdump or phased format.

The examples below use the samplefiles in the baseband/data/gsb/ directory, and the numpy, astropy.units and baseband.gsb modules:

>>> import numpy as np
>>> import astropy.units as u
>>> from baseband import gsb
>>> from baseband.data import (
...     SAMPLE_GSB_RAWDUMP, SAMPLE_GSB_RAWDUMP_HEADER,
...     SAMPLE_GSB_PHASED, SAMPLE_GSB_PHASED_HEADER)

A single timestamp file can be opened with open in text mode:

>>> ft = gsb.open(SAMPLE_GSB_RAWDUMP_HEADER, 'rt')
>>> ft.read_timestamp()
<GSBRawdumpHeader gps: 2015 04 27 18 45 00 0.000000240>
>>> ft.close()

Reading payloads requires the samples per frame or sample rate. For phased the sample rate is:

sample_rate = raw_sample_rate / (2 * nchan)

where the raw sample rate is the pre-channelized one, and nchan the number of Fourier channels. The samples per frame for both rawdump and phased is:

samples_per_frame = timespan_of_frame * sample_rate

Note

Since the number of samples per frame is an integer number while both the frame timespan and sample rate are not, it is better to separately caculate samples_per_frame rather than multiplying timespan_of_frame with sample_rate in order to avoid rounding issues.

Alternatively, if the size of the frame buffer and the frame rate are known, the former can be used to determine samples_per_frame, and the latter used to determine sample_rate by inverting the above equation.

If samples_per_frame is not given, Baseband assumes it is the equivalent of 4 megabytes of data for rawdump, or 8 megabytes if phased. If sample_rate is not given, it is calculated from samples_per_frame assuming timespan_of_frame = 0.25165824 (see File Structure above).

A single payload file can be opened with open in binary mode. Here, for our sample file, we have to take into account that in order to keep these files small, their sample size has been reduced to only 4 or 8 kilobytes worth of samples per frame (for the default timespan). So, we define their sample rate here, and use that to calculate payload_nbytes, the size of one frame in bytes. Since rawdump samples are 4 bits, payload_nbytes is just samples_per_frame / 2:

>>> rawdump_samples_per_frame = 2**13
>>> payload_nbytes = rawdump_samples_per_frame // 2
>>> fb = gsb.open(SAMPLE_GSB_RAWDUMP, 'rb', payload_nbytes=payload_nbytes,
...               nchan=1, bps=4, complex_data=False)
>>> payload = fb.read_payload()
>>> payload[:4]
array([[ 0.],
       [-2.],
       [-2.],
       [ 0.]], dtype=float32)
>>> fb.close()

(payload_nbytes for phased data is the size of one frame divided by the number of binary files.)

Opening in stream mode allows timestamp and binary files to be read in concert to create data frames, and also wraps the low-level routines such that reading and writing is in units of samples, and provides access to header information.

When opening a rawdump file in stream mode, we pass the timestamp file as the first argument, and the binary file to the raw keyword. As per above, we also pass samples_per_frame:

>>> fh_rd = gsb.open(SAMPLE_GSB_RAWDUMP_HEADER, mode='rs',
...                  raw=SAMPLE_GSB_RAWDUMP,
...                  samples_per_frame=rawdump_samples_per_frame)
>>> fh_rd.header0
<GSBRawdumpHeader gps: 2015 04 27 18 45 00 0.000000240>
>>> dr = fh_rd.read()
>>> dr.shape
(81920,)
>>> dr[:3]
array([ 0., -2., -2.], dtype=float32)
>>> fh_rd.close()

To open a phased fileset in stream mode, we package the binary files into a nested tuple with the format:

((L pol stream 1, L pol stream 2), (R pol stream 1, R pol stream 2))

The nested tuple is passed to raw (note that we again have to pass a non-default sample rate):

>>> phased_samples_per_frame = 2**3
>>> fh_ph = gsb.open(SAMPLE_GSB_PHASED_HEADER, mode='rs',
...                  raw=SAMPLE_GSB_PHASED,
...                  samples_per_frame=phased_samples_per_frame)
>>> header0 = fh_ph.header0     # To be used for writing, below.
>>> dp = fh_ph.read()
>>> dp.shape
(80, 2, 512)
>>> dp[0, 0, :3]    
array([30.+12.j, -1. +8.j,  7.+19.j], dtype=complex64)
>>> fh_ph.close()

To set up a file for writing, we need to pass names for both timestamp and raw files, as well as sample_rate, samples_per_frame, and either the first header or a time object. We first calculate sample_rate:

>>> timespan = 0.25165824 * u.s
>>> rawdump_sample_rate = (rawdump_samples_per_frame / timespan).to(u.MHz)
>>> phased_sample_rate = (phased_samples_per_frame / timespan).to(u.MHz)

To write a rawdump file:

>>> from astropy.time import Time
>>> fw_rd = gsb.open('test_rawdump.timestamp',
...                  mode='ws', raw='test_rawdump.dat',
...                  sample_rate=rawdump_sample_rate,
...                  samples_per_frame=rawdump_samples_per_frame,
...                  time=Time('2015-04-27T13:15:00'))
>>> fw_rd.write(dr)
>>> fw_rd.close()
>>> fh_rd = gsb.open('test_rawdump.timestamp', mode='rs',
...                  raw='test_rawdump.dat',
...                  sample_rate=rawdump_sample_rate,
...                  samples_per_frame=rawdump_samples_per_frame)
>>> np.all(dr == fh_rd.read())
True
>>> fh_rd.close()

To write a phased file, we need to pass a nested tuple of filenames or filehandles:

>>> test_phased_bin = (('test_phased_pL1.dat', 'test_phased_pL2.dat'),
...                    ('test_phased_pR1.dat', 'test_phased_pR2.dat'))
>>> fw_ph = gsb.open('test_phased.timestamp',
...                  mode='ws', raw=test_phased_bin,
...                  sample_rate=phased_sample_rate,
...                  samples_per_frame=phased_samples_per_frame,
...                  header0=header0)
>>> fw_ph.write(dp)
>>> fw_ph.close()
>>> fh_ph = gsb.open('test_phased.timestamp', mode='rs',
...                  raw=test_phased_bin,
...                  sample_rate=phased_sample_rate,
...                  samples_per_frame=phased_samples_per_frame)
>>> np.all(dp == fh_ph.read())
True
>>> fh_ph.close()

Baseband does not use the PC time in the phased header, and, when writing, simply uses the same time for both GPS and PC times. Since the PC time can drift from the GPS time by several tens of milliseconds, test_phased.timestamp will not be identical to SAMPLE_GSB_PHASED, even though we have written the exact same data to file.

Reference/API

baseband.gsb Package

GMRT Software Backend (GSB) data reader.

See http://gmrt.ncra.tifr.res.in/gmrt_hpage/sub_system/gmrt_gsb/index.htm

Functions

open(name[, mode]) Open GSB file(s) for reading or writing.

Classes

GSBFrame(header, payload[, valid, verify]) Frame encapsulating GSB rawdump or phased data.
GSBHeader(words[, mode, nbytes, utc_offset, …]) GSB Header, based on a line from a timestamp file.
GSBPayload(words[, sample_shape, bps, …]) Container for decoding and encoding GSB payloads.

Class Inheritance Diagram

Inheritance diagram of baseband.gsb.frame.GSBFrame, baseband.gsb.header.GSBHeader, baseband.gsb.payload.GSBPayload

baseband.gsb.header Module

Definitions for GSB Headers, using the timestamp files.

Somewhat out of data description for phased data: http://gmrt.ncra.tifr.res.in/gmrt_hpage/sub_system/gmrt_gsb/GSB_beam_timestamp_note_v1.pdf and for rawdump data http://gmrt.ncra.tifr.res.in/gmrt_hpage/sub_system/gmrt_gsb/GSB_rawdump_data_format_v2.pdf

Classes

TimeGSB(val1, val2, scale, precision, …[, …]) GSB header date-time format YYYY MM DD HH MM SS 0.SSSSSSSSS.
GSBHeader(words[, mode, nbytes, utc_offset, …]) GSB Header, based on a line from a timestamp file.
GSBRawdumpHeader(words[, mode, nbytes, …]) GSB rawdump header.
GSBPhasedHeader(words[, mode, nbytes, …]) GSB phased header.

Class Inheritance Diagram

Inheritance diagram of baseband.gsb.header.TimeGSB, baseband.gsb.header.GSBHeader, baseband.gsb.header.GSBRawdumpHeader, baseband.gsb.header.GSBPhasedHeader

baseband.gsb.payload Module

Definitions for GSB payloads.

Implements a GSBPayload class used to store payload blocks, and decode to or encode from a data array.

See http://gmrt.ncra.tifr.res.in/gmrt_hpage/sub_system/gmrt_gsb/index.htm

Classes

GSBPayload(words[, sample_shape, bps, …]) Container for decoding and encoding GSB payloads.

Class Inheritance Diagram

Inheritance diagram of baseband.gsb.payload.GSBPayload

baseband.gsb.frame Module

Classes

GSBFrame(header, payload[, valid, verify]) Frame encapsulating GSB rawdump or phased data.

Class Inheritance Diagram

Inheritance diagram of baseband.gsb.frame.GSBFrame

baseband.gsb.base Module

Functions

open(name[, mode]) Open GSB file(s) for reading or writing.

Classes

GSBFileReader(fh_raw, payload_nbytes[, …]) Simple reader for GSB data files.
GSBFileWriter(fh_raw) Simple writer for GSB data files.
GSBStreamReader(fh_ts, fh_raw[, …]) GSB format reader.
GSBStreamWriter(fh_ts, fh_raw[, header0, …]) GSB format writer.

Class Inheritance Diagram

Inheritance diagram of baseband.gsb.base.GSBFileReader, baseband.gsb.base.GSBFileWriter, baseband.gsb.base.GSBStreamReader, baseband.gsb.base.GSBStreamWriter