MARK 4¶
The Mark 4 format is the output format of the MIT Haystack Observatory’s Mark 4 VLBI magnetic tape-based data acquisition system, and one output format of its successor, the Mark 5A hard drive-based system. The format’s specification is in the Mark IIIA/IV/VLBA design specifications.
Baseband currently only supports files that have been parity-stripped and corrected for barrel roll and data modulation.
File Structure¶
Mark 4 files contain up to 64 concurrent data “tracks”. Tracks are divided into 22500-bit “tape frames”, each of which consists of a 160-bit header followed by a 19840-bit payload. The header includes a timestamp (accurate to 1.25 ms), track ID, sideband, and fan-out/in factor (see below); the details of these can be found in 2.1.1 - 2.1.3 in the design specifications. The payload consists of a 1-bit stream. When recording 2-bit elementary samples, the data is split into two tracks, with one carrying the sign bit, and the other the magnitude bit.
The header takes the place of the first 160 bits of payload data, so that the
first sample occurs fanout * 160
sample times after the header time. This
means that a Mark 4 stream is not contiguous in time. The length of
one frame ranges from 1.25 ms to 160 ms in octave steps (which ensures an
integer number of frames falls within 1 minute), setting the maximum sample
rate per track to 18 megabits/track/s.
Data from a single channel may be distributed to multiple tracks - “fan-out” - or multiple channels fed to one track - “fan-in”. Fan-out is used when sampling at rates higher than 18 megabits/track/s. Baseband currently only supports tracks using fan-out (“longitudinal data format”).
Baseband reconstructs the tracks into channels (reconstituting 2-bit data from two tracks into a single channel if necessary) and combines tape frame headers into a single data frame header.
Usage¶
This section covers reading and writing Mark 4 files with Baseband; general
usage can be found under the Using Baseband section.
For situations in which one is unsure of a file’s format, Baseband features the
general baseband.open
and baseband.file_info
functions, which are also
discussed in Using Baseband. The examples below use
the small sample file baseband/data/sample.m4
, and the numpy
,
astropy.units
, astropy.time.Time
, and baseband.mark4
modules:
>>> import numpy as np
>>> import astropy.units as u
>>> from astropy.time import Time
>>> from baseband import mark4
>>> from baseband.data import SAMPLE_MARK4
Opening a Mark 4 file with open
in binary mode provides
a normal file reader but extended with methods to read a
Mark4Frame
. Mark 4 files generally do not start (or end)
at a frame boundary, so in binary mode one has to find the first
header using find_header
(which will
also determine the number of Mark 4 tracks, if not given explicitly). Since
Mark 4 files do not store the full time information, one must pass either the
the decade the data was taken, or an equivalent reference Time
object:
>>> fb = mark4.open(SAMPLE_MARK4, 'rb', decade=2010)
>>> fb.find_header() # Locate first header and determine ntrack.
<Mark4Header bcd_headstack1: [0x3344]*64,
bcd_headstack2: [0x1122]*64,
headstack_id: [0, ..., 1],
bcd_track_id: [0x2, ..., 0x33],
fan_out: [0, ..., 3],
magnitude_bit: [False, ..., True],
lsb_output: [True]*64,
converter_id: [0, ..., 7],
time_sync_error: [False]*64,
internal_clock_error: [False]*64,
processor_time_out_error: [False]*64,
communication_error: [False]*64,
_1_11_1: [False]*64,
_1_10_1: [False]*64,
track_roll_enabled: [False]*64,
sequence_suspended: [False]*64,
system_id: [108]*64,
_1_0_1_sync: [False]*64,
sync_pattern: [0xffffffff]*64,
bcd_unit_year: [0x4]*64,
bcd_day: [0x167]*64,
bcd_hour: [0x7]*64,
bcd_minute: [0x38]*64,
bcd_second: [0x12]*64,
bcd_fraction: [0x475]*64,
crc: [0xea6, ..., 0x212]>
>>> fb.ntrack
64
>>> fb.tell()
2696
>>> frame = fb.read_frame()
>>> frame.shape
(80000, 8)
>>> frame.header.time
<Time object: scale='utc' format='yday' value=2014:167:07:38:12.47500>
>>> fb.close()
Opening in stream mode automatically finds the first frame, and wraps the
low-level routines such that reading and writing is in units of samples. It
also provides access to header information. Here we pass a reference
Time
object within 4 years of the observation start time to
ref_time
, rather than a decade
:
>>> fh = mark4.open(SAMPLE_MARK4, 'rs', ref_time=Time('2013:100:23:00:00'))
>>> fh
<Mark4StreamReader name=... offset=0
sample_rate=32.0 MHz, samples_per_frame=80000,
sample_shape=SampleShape(nchan=8), bps=2,
start_time=2014-06-16T07:38:12.47500>
>>> d = fh.read(6400)
>>> d.shape
(6400, 8)
>>> d[635:645, 0].astype(int) # first channel
array([ 0, 0, 0, 0, 0, -1, 1, 3, 1, -1])
>>> fh.close()
As mentioned in the File Structure section, because the header
takes the place of the first 160 samples of each track, the first payload
sample occurs fanout * 160
sample times after the header time. The stream
reader includes these overwritten samples as invalid data (zeros, by default):
>>> np.array_equal(d[:640], np.zeros((640,) + d.shape[1:]))
True
When writing to file, we can just pass in a complete header, as the
decade and number of tracks can be inferred from it. However, since
the header actually written to the file does not store this, we have
to pass in the decade
when reading back:
>>> fw = mark4.open('sample_mark4_segment.m4', 'ws', header0=frame.header,
... sample_rate=32*u.MHz)
>>> fw.write(frame.data)
>>> fw.close()
>>> fh = mark4.open('sample_mark4_segment.m4', 'rs',
... sample_rate=32.*u.MHz, decade=2010)
>>> np.all(fh.read(80000) == frame.data)
True
>>> fh.close()
Note that above we had to pass in the sample rate even when opening the file for reading; this is because there is only a single frame in the file, and hence the sample rate cannot be inferred automatically.
Reference/API¶
baseband.mark4 Package¶
Mark 4 VLBI data reader.
Code inspired by Walter Brisken’s mark5access. See https://github.com/demorest/mark5access.
The format itself is described in detail in https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf
Functions¶
|
Collect Mark4 file information. |
|
Open Mark4 file(s) for reading or writing. |
Classes¶
|
Representation of a Mark 4 frame, consisting of a header and payload. |
|
Decoder/encoder of a Mark 4 Header, containing all streams. |
|
Container for decoding and encoding Mark 4 payloads. |
Class Inheritance Diagram¶
baseband.mark4.header Module¶
Definitions for VLBI Mark 4 Headers.
Implements a Mark4Header class used to store header words, and decode/encode the information therein.
For the specification of tape Mark 4 format, see https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf
A little bit on the disk representation is at https://ui.adsabs.harvard.edu/abs/2003ASPC..306..123W
Functions¶
|
Convert a stream of integers to uint32 header words. |
|
Convert a set of uint32 header words to a stream of integers. |
Classes¶
|
Decoder/encoder of a Mark 4 Track Header. |
|
Decoder/encoder of a Mark 4 Header, containing all streams. |
Variables¶
CRC polynomial used for Mark 4 Headers. |
|
Cyclic Redundancy Check for a bitstream. |
Class Inheritance Diagram¶
baseband.mark4.payload Module¶
Definitions for VLBI Mark 4 payloads.
Implements a Mark4Payload class used to store payload words, and decode to or encode from a data array.
For the specification, see https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf
Functions¶
|
Reorder 32-track bits to bring signs & magnitudes together. |
|
Reorder 64-track bits to bring signs & magnitudes together. |
Set up the look-up tables for levels as a function of input byte. |
|
|
Decode payload for 8 channels using 2 bits, fan-out 4 (64 tracks). |
|
Encode payload for 8 channels using 2 bits, fan-out 4 (64 tracks). |
Classes¶
|
Container for decoding and encoding Mark 4 payloads. |
Class Inheritance Diagram¶
baseband.mark4.frame Module¶
Definitions for VLBI Mark 4 payloads.
Implements a Mark4Payload class used to store payload words, and decode to or encode from a data array.
For the specification, see https://www.haystack.mit.edu/tech/vlbi/mark5/docs/230.3.pdf
Classes¶
|
Representation of a Mark 4 frame, consisting of a header and payload. |
Class Inheritance Diagram¶
baseband.mark4.file_info Module¶
The Mark4FileReaderInfo property.
Includes information about what is needed to calcuate times, number of tracks and offset of first header.
Classes¶
|
Standardized information on Mark 4 file readers. |
Class Inheritance Diagram¶
baseband.mark4.base Module¶
Functions¶
|
Open Mark4 file(s) for reading or writing. |
|
Collect Mark4 file information. |
Classes¶
|
Simple reader for Mark 4 files. |
|
Simple writer for Mark 4 files. |
|
VLBI Mark 4 format reader. |
|
VLBI Mark 4 format writer. |