# MARK 4¶

The Mark 4 format is the output format of the MIT Haystack Observatory’s Mark 4 VLBI magnetic tape-based data acquisition system, and one output format of its successor, the Mark 5A hard drive-based system. The format’s specification is in the Mark IIIA/IV/VLBA design specifications.

Baseband currently only supports files that have been parity-stripped and corrected for barrel roll and data modulation.

## File Structure¶

Mark 4 files contain up to 64 concurrent data “tracks”. Tracks are divided into 22500-bit “tape frames”, each of which consists of a 160-bit header followed by a 19840-bit payload. The header includes a timestamp (accurate to 1.25 ms), track ID, sideband, and fan-out/in factor (see below); the details of these can be found in 2.1.1 - 2.1.3 in the design specifications. The payload consists of a 1-bit stream. When recording 2-bit elementary samples, the data is split into two tracks, with one carrying the sign bit, and the other the magnitude bit.

The header takes the place of the first 160 bits of payload data, so that the first sample occurs fanout * 160 sample times after the header time. This means that a Mark 4 stream is not contiguous in time. The length of one frame ranges from 1.25 ms to 160 ms in octave steps (which ensures an integer number of frames falls within 1 minute), setting the maximum sample rate per track to 18 megabits/track/s.

Data from a single channel may be distributed to multiple tracks - “fan-out” - or multiple channels fed to one track - “fan-in”. Fan-out is used when sampling at rates higher than 18 megabits/track/s. Baseband currently only supports tracks using fan-out (“longitudinal data format”).

Baseband reconstructs the tracks into channels (reconstituting 2-bit data from two tracks into a single channel if necessary) and combines tape frame headers into a single data frame header.

## Usage¶

This section covers reading and writing Mark 4 files with Baseband; general usage can be found under the Using Baseband section. For situations in which one is unsure of a file’s format, Baseband features the general baseband.open and baseband.file_info functions, which are also discussed in Using Baseband. The examples below use the small sample file baseband/data/sample.m4, and the numpy, astropy.units, astropy.time.Time, and baseband.mark4 modules:

>>> import numpy as np
>>> import astropy.units as u
>>> from astropy.time import Time
>>> from baseband import mark4
>>> from baseband.data import SAMPLE_MARK4


Opening a Mark 4 file with open in binary mode provides a normal file reader but extended with methods to read a Mark4Frame. Mark 4 files generally do not start (or end) at a frame boundary, so in binary mode one has to find the first header using find_header (which will also determine the number of Mark 4 tracks, if not given explicitly). Since Mark 4 files do not store the full time information, one must pass either the the decade the data was taken, or an equivalent reference Time object:

>>> fb = mark4.open(SAMPLE_MARK4, 'rb', decade=2010)
bcd_track_id: [0x2, ..., 0x33],
fan_out: [0, ..., 3],
magnitude_bit: [False, ..., True],
lsb_output: [True]*64,
converter_id: [0, ..., 7],
time_sync_error: [False]*64,
internal_clock_error: [False]*64,
processor_time_out_error: [False]*64,
communication_error: [False]*64,
_1_11_1: [False]*64,
_1_10_1: [False]*64,
track_roll_enabled: [False]*64,
sequence_suspended: [False]*64,
system_id: [108]*64,
_1_0_1_sync: [False]*64,
sync_pattern: [0xffffffff]*64,
bcd_unit_year: [0x4]*64,
bcd_day: [0x167]*64,
bcd_hour: [0x7]*64,
bcd_minute: [0x38]*64,
bcd_second: [0x12]*64,
bcd_fraction: [0x475]*64,
crc: [0xea6, ..., 0x212]>
>>> fb.ntrack
64
>>> fb.tell()
2696
>>> frame.shape
(80000, 8)
<Time object: scale='utc' format='yday' value=2014:167:07:38:12.47500>
>>> fb.close()


Opening in stream mode automatically finds the first frame, and wraps the low-level routines such that reading and writing is in units of samples. It also provides access to header information. Here we pass a reference Time object within 4 years of the observation start time to ref_time, rather than a decade:

>>> fh = mark4.open(SAMPLE_MARK4, 'rs', ref_time=Time('2013:100:23:00:00'))
>>> fh
sample_rate=32.0 MHz, samples_per_frame=80000,
sample_shape=SampleShape(nchan=8), bps=2,
start_time=2014-06-16T07:38:12.47500>
>>> d.shape
(6400, 8)
>>> d[635:645, 0].astype(int)  # first channel
array([ 0,  0,  0,  0,  0, -1,  1,  3,  1, -1])
>>> fh.close()


As mentioned in the File Structure section, because the header takes the place of the first 160 samples of each track, the first payload sample occurs fanout * 160 sample times after the header time. The stream reader includes these overwritten samples as invalid data (zeros, by default):

>>> np.array_equal(d[:640], np.zeros((640,) + d.shape[1:]))
True


When writing to file, we can just pass in a complete header, as the decade and number of tracks can be inferred from it. However, since the header actually written to the file does not store this, we have to pass in the decade when reading back:

>>> fw = mark4.open('sample_mark4_segment.m4', 'ws', header0=frame.header,
...                 sample_rate=32*u.MHz)
>>> fw.write(frame.data)
>>> fw.close()
>>> fh = mark4.open('sample_mark4_segment.m4', 'rs',
True
>>> fh.close()


Note that above we had to pass in the sample rate even when opening the file for reading; this is because there is only a single frame in the file, and hence the sample rate cannot be inferred automatically.

## Reference/API¶

### baseband.mark4 Package¶

Code inspired by Walter Brisken’s mark5access. See https://github.com/demorest/mark5access.

The format itself is described in detail in https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf

#### Functions¶

 info(name, **kwargs) Collect Mark4 file information. open(name[, mode]) Open Mark4 file(s) for reading or writing.

#### Classes¶

 Mark4Frame(header, payload[, valid, verify]) Representation of a Mark 4 frame, consisting of a header and payload. Mark4Header(words[, ntrack, decade, …]) Decoder/encoder of a Mark 4 Header, containing all streams. Mark4Payload(words[, header, sample_shape, …]) Container for decoding and encoding Mark 4 payloads.

#### Class Inheritance Diagram¶

Definitions for VLBI Mark 4 Headers.

Implements a Mark4Header class used to store header words, and decode/encode the information therein.

For the specification of tape Mark 4 format, see https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf

A little bit on the disk representation is at https://ui.adsabs.harvard.edu/abs/2003ASPC..306..123W

#### Functions¶

 stream2words(stream[, track]) Convert a stream of integers to uint32 header words. words2stream(words) Convert a set of uint32 header words to a stream of integers.

#### Classes¶

 Mark4TrackHeader(words[, decade, ref_time, …]) Decoder/encoder of a Mark 4 Track Header. Mark4Header(words[, ntrack, decade, …]) Decoder/encoder of a Mark 4 Header, containing all streams.

#### Variables¶

 CRC12 CRC polynomial used for Mark 4 Headers. crc12 Cyclic Redundancy Check for a bitstream.

#### Class Inheritance Diagram¶

Definitions for VLBI Mark 4 payloads.

Implements a Mark4Payload class used to store payload words, and decode to or encode from a data array.

For the specification, see https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf

#### Functions¶

 Reorder 32-track bits to bring signs & magnitudes together. Reorder 64-track bits to bring signs & magnitudes together. Set up the look-up tables for levels as a function of input byte. Decode payload for 8 channels using 2 bits, fan-out 4 (64 tracks). Encode payload for 8 channels using 2 bits, fan-out 4 (64 tracks).

#### Classes¶

 Mark4Payload(words[, header, sample_shape, …]) Container for decoding and encoding Mark 4 payloads.

### baseband.mark4.frame Module¶

Definitions for VLBI Mark 4 payloads.

Implements a Mark4Payload class used to store payload words, and decode to or encode from a data array.

For the specification, see https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf

#### Classes¶

 Mark4Frame(header, payload[, valid, verify]) Representation of a Mark 4 frame, consisting of a header and payload.

### baseband.mark4.file_info Module¶

Includes information about what is needed to calcuate times, number of tracks and offset of first header.

#### Classes¶

 Mark4FileReaderInfo([parent]) Standardized information on Mark 4 file readers.

### baseband.mark4.base Module¶

#### Functions¶

 open(name[, mode]) Open Mark4 file(s) for reading or writing. info(name, **kwargs) Collect Mark4 file information.

#### Classes¶

 Mark4FileReader(fh_raw[, ntrack, decade, …]) Simple reader for Mark 4 files. Mark4FileWriter(fh_raw) Simple writer for Mark 4 files. Mark4StreamReader(fh_raw[, sample_rate, …]) VLBI Mark 4 format reader. Mark4StreamWriter(fh_raw, header0[, …]) VLBI Mark 4 format writer.