MARK 4

The Mark 4 format is the output format of the MIT Haystack Observatory’s Mark 4 VLBI magnetic tape-based data acquisition system, and one output format of its successor, the Mark 5A hard drive-based system. The format’s specification is in the Mark IIIA/IV/VLBA design specifications.

Baseband currently only supports files that have been parity-stripped and corrected for barrel roll and data modulation.

File Structure

Mark 4 files contain up to 64 concurrent data “tracks”. Tracks are divided into 22500-bit “tape frames”, each of which consists of a 160-bit header followed by a 19840-bit payload. The header includes a timestamp (accurate to 1.25 ms), track ID, sideband, and fan-out/in factor (see below); the details of these can be found in 2.1.1 - 2.1.3 in the design specifications. The payload consists of a 1-bit stream. When recording 2-bit elementary samples, the data is split into two tracks, with one carrying the sign bit, and the other the magnitude bit.

The header takes the place of the first 160 bits of payload data, so that the first sample occurs fanout * 160 sample times after the header time. This means that a Mark 4 stream is not contiguous in time. The length of one frame ranges from 1.25 ms to 160 ms in octave steps (which ensures an integer number of frames falls within 1 minute), setting the maximum sample rate per track to 18 megabits/track/s.

Data from a single channel may be distributed to multiple tracks - “fan-out” - or multiple channels fed to one track - “fan-in”. Fan-out is used when sampling at rates higher than 18 megabits/track/s. Baseband currently only supports tracks using fan-out (“longitudinal data format”).

Baseband reconstructs the tracks into channels (reconstituting 2-bit data from two tracks into a single channel if necessary) and combines tape frame headers into a single data frame header.

Usage

This section covers reading and writing Mark 4 files with Baseband; general usage can be found under the Using Baseband section. For situations in which one is unsure of a file’s format, Baseband features the general baseband.open and baseband.file_info functions, which are also discussed in Using Baseband. The examples below use the small sample file baseband/data/sample.m4, and the numpy, astropy.units, astropy.time.Time, and baseband.mark4 modules:

>>> import numpy as np
>>> import astropy.units as u
>>> from astropy.time import Time
>>> from baseband import mark4
>>> from baseband.data import SAMPLE_MARK4

Opening a Mark 4 file with open in binary mode provides a normal file reader but extended with methods to read a Mark4Frame. Mark 4 files generally do not start (or end) at a frame boundary, so in binary mode one has to find the first header using find_header (which will also determine the number of Mark 4 tracks, if not given explicitly). Since Mark 4 files do not store the full time information, one must pass either the the decade the data was taken, or an equivalent reference Time object:

>>> fb = mark4.open(SAMPLE_MARK4, 'rb', decade=2010)
>>> fb.find_header()  # Locate first header and determine ntrack.
<Mark4Header bcd_headstack1: [0x3344]*64,
             bcd_headstack2: [0x1122]*64,
             headstack_id: [0, ..., 1],
             bcd_track_id: [0x2, ..., 0x33],
             fan_out: [0, ..., 3],
             magnitude_bit: [False, ..., True],
             lsb_output: [True]*64,
             converter_id: [0, ..., 7],
             time_sync_error: [False]*64,
             internal_clock_error: [False]*64,
             processor_time_out_error: [False]*64,
             communication_error: [False]*64,
             _1_11_1: [False]*64,
             _1_10_1: [False]*64,
             track_roll_enabled: [False]*64,
             sequence_suspended: [False]*64,
             system_id: [108]*64,
             _1_0_1_sync: [False]*64,
             sync_pattern: [0xffffffff]*64,
             bcd_unit_year: [0x4]*64,
             bcd_day: [0x167]*64,
             bcd_hour: [0x7]*64,
             bcd_minute: [0x38]*64,
             bcd_second: [0x12]*64,
             bcd_fraction: [0x475]*64,
             crc: [0xea6, ..., 0x212]>
>>> fb.ntrack
64
>>> fb.tell()
2696
>>> frame = fb.read_frame()
>>> frame.shape
(80000, 8)
>>> frame.header.time
<Time object: scale='utc' format='yday' value=2014:167:07:38:12.47500>
>>> fb.close()

Opening in stream mode automatically finds the first frame, and wraps the low-level routines such that reading and writing is in units of samples. It also provides access to header information. Here we pass a reference Time object within 4 years of the observation start time to ref_time, rather than a decade:

>>> fh = mark4.open(SAMPLE_MARK4, 'rs', ref_time=Time('2013:100:23:00:00'))
>>> fh
<Mark4StreamReader name=... offset=0
    sample_rate=32.0 MHz, samples_per_frame=80000,
    sample_shape=SampleShape(nchan=8), bps=2,
    start_time=2014-06-16T07:38:12.47500>
>>> d = fh.read(6400)
>>> d.shape
(6400, 8)
>>> d[635:645, 0].astype(int)  # first channel
array([ 0,  0,  0,  0,  0, -1,  1,  3,  1, -1])
>>> fh.close()

As mentioned in the File Structure section, because the header takes the place of the first 160 samples of each track, the first payload sample occurs fanout * 160 sample times after the header time. The stream reader includes these overwritten samples as invalid data (zeros, by default):

>>> np.array_equal(d[:640], np.zeros((640,) + d.shape[1:]))
True

When writing to file, we can just pass in a complete header, as the decade and number of tracks can be inferred from it. However, since the header actually written to the file does not store this, we have to pass in the decade when reading back:

>>> fw = mark4.open('sample_mark4_segment.m4', 'ws', header0=frame.header,
...                 sample_rate=32*u.MHz)
>>> fw.write(frame.data)
>>> fw.close()
>>> fh = mark4.open('sample_mark4_segment.m4', 'rs',
...                 sample_rate=32.*u.MHz, decade=2010)
>>> np.all(fh.read(80000) == frame.data)
True
>>> fh.close()

Note that above we had to pass in the sample rate even when opening the file for reading; this is because there is only a single frame in the file, and hence the sample rate cannot be inferred automatically.

Reference/API

baseband.mark4 Package

Mark 4 VLBI data reader.

Code inspired by Walter Brisken’s mark5access. See https://github.com/difx/mark5access.

The format itself is described in detail in https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf

Functions

info(name, **kwargs)

Collect Mark4 file information.

open(name[, mode])

Open Mark4 file(s) for reading or writing.

Classes

Mark4Frame(header, payload[, valid, verify])

Representation of a Mark 4 frame, consisting of a header and payload.

Mark4Header(words[, ntrack, decade, ...])

Decoder/encoder of a Mark 4 Header, containing all streams.

Mark4Payload(words[, header, sample_shape, ...])

Container for decoding and encoding Mark 4 payloads.

Class Inheritance Diagram

Inheritance diagram of baseband.mark4.frame.Mark4Frame, baseband.mark4.header.Mark4Header, baseband.mark4.payload.Mark4Payload

baseband.mark4.header Module

Definitions for VLBI Mark 4 Headers.

Implements a Mark4Header class used to store header words, and decode/encode the information therein.

For the specification of tape Mark 4 format, see https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf

A little bit on the disk representation is at https://ui.adsabs.harvard.edu/abs/2003ASPC..306..123W

Functions

stream2words(stream[, track])

Convert a stream of integers to uint32 header words.

words2stream(words)

Convert a set of uint32 header words to a stream of integers.

Classes

Mark4TrackHeader(words[, decade, ref_time, ...])

Decoder/encoder of a Mark 4 Track Header.

Mark4Header(words[, ntrack, decade, ...])

Decoder/encoder of a Mark 4 Header, containing all streams.

Variables

CRC12

CRC polynomial used for Mark 4 Headers.

crc12

Cyclic Redundancy Check for a bitstream.

Class Inheritance Diagram

Inheritance diagram of baseband.mark4.header.Mark4TrackHeader, baseband.mark4.header.Mark4Header

baseband.mark4.payload Module

Definitions for VLBI Mark 4 payloads.

Implements a Mark4Payload class used to store payload words, and decode to or encode from a data array.

For the specification, see https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf

Functions

reorder32(x)

Reorder 32-track bits to bring signs & magnitudes together.

reorder64(x)

Reorder 64-track bits to bring signs & magnitudes together.

init_luts()

Set up the look-up tables for levels as a function of input byte.

decode_8chan_2bit_fanout4(frame)

Decode payload for 8 channels using 2 bits, fan-out 4 (64 tracks).

encode_8chan_2bit_fanout4(values)

Encode payload for 8 channels using 2 bits, fan-out 4 (64 tracks).

Classes

Mark4Payload(words[, header, sample_shape, ...])

Container for decoding and encoding Mark 4 payloads.

Class Inheritance Diagram

Inheritance diagram of baseband.mark4.payload.Mark4Payload

baseband.mark4.frame Module

Definitions for VLBI Mark 4 payloads.

Implements a Mark4Payload class used to store payload words, and decode to or encode from a data array.

For the specification, see https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf

Classes

Mark4Frame(header, payload[, valid, verify])

Representation of a Mark 4 frame, consisting of a header and payload.

Class Inheritance Diagram

Inheritance diagram of baseband.mark4.frame.Mark4Frame

baseband.mark4.file_info Module

The Mark4FileReaderInfo property.

Includes information about what is needed to calcuate times, number of tracks and offset of first header.

Classes

Mark4FileReaderInfo([parent])

Standardized information on Mark 4 file readers.

Class Inheritance Diagram

Inheritance diagram of baseband.mark4.file_info.Mark4FileReaderInfo

baseband.mark4.base Module

Functions

open(name[, mode])

Open Mark4 file(s) for reading or writing.

info(name, **kwargs)

Collect Mark4 file information.

Classes

Mark4FileReader(fh_raw[, ntrack, decade, ...])

Simple reader for Mark 4 files.

Mark4FileWriter(fh_raw)

Simple writer for Mark 4 files.

Mark4StreamReader(fh_raw[, sample_rate, ...])

VLBI Mark 4 format reader.

Mark4StreamWriter(fh_raw, header0[, ...])

VLBI Mark 4 format writer.

Class Inheritance Diagram

Inheritance diagram of baseband.mark4.base.Mark4FileReader, baseband.mark4.base.Mark4FileWriter, baseband.mark4.base.Mark4StreamReader, baseband.mark4.base.Mark4StreamWriter