GUPPI

The GUPPI format is the output of the Green Bank Ultimate Pulsar Processing Instrument and any clones operating at other telescopes, such as PUPPI at the Arecibo Observatory. Baseband specifically supports GUPPI data taken in baseband mode, and is based off of DSPSR’s implementation. While general format specifications can be found on Paul Demorest’s site, some of the header information could be invalid or not applicable, particularly with older files.

Baseband currently only supports 8-bit elementary samples.

File Structure

Each GUPPI file contains multiple (typically 128) frames, with each frame consisting of an ASCII header composed of 80-character entries, followed by a binary payload (or “block”). The header’s length is variable, but always ends with “END” followed by 77 spaces.

How samples are stored in the payload depends on whether or not it is channels-first. A channels-first payload stores each channel’s stream in a contiguous data block, while a non-channels-first one groups the components of a complete sample together (like with other formats). In either case, for each channel polarization samples from the same point in time are stored adjacent to one another. At the end of each channel’s data is a section of overlap samples identical to the first samples in the next payload. Baseband retains these redundant samples when reading individual GUPPI frames, but skips them when reading files as a stream, keeping only the overlap of the last frame.

Usage

This section covers reading and writing GUPPI files with Baseband; general usage is covered in the Using Baseband section. For situations in which one is unsure of a file’s format, Baseband features the general baseband.open and baseband.file_info functions, which are also discussed in Using Baseband. The examples below use the sample PUPPI file baseband/data/sample_puppi.raw, and the the astropy.units and baseband.guppi modules:

>>> from baseband import guppi
>>> import astropy.units as u
>>> from baseband.data import SAMPLE_PUPPI

Single files can be opened with open in binary mode, which provides a normal file reader, but extended with methods to read a GUPPIFrame:

>>> fb = guppi.open(SAMPLE_PUPPI, 'rb')
>>> frame = fb.read_frame()
>>> frame.shape
(1024, 2, 4)
>>> frame[:3, 0, 1]    
array([-32.-10.j, -15.-14.j,   9.-13.j], dtype=complex64)
>>> fb.close()

Since the files can be quite large, the payload is mapped (with numpy.memmap), so that if one accesses part of the data, only the corresponding parts of the encoded payload are loaded into memory (since the sample file is encoded using 8 bits, the above example thus loads 6 bytes into memory).

Opening in stream mode wraps the low-level routines such that reading and writing is in units of samples, and provides access to header information:

>>> fh = guppi.open(SAMPLE_PUPPI, 'rs')
>>> fh
<GUPPIStreamReader name=... offset=0
    sample_rate=250.0 Hz, samples_per_frame=960,
    sample_shape=SampleShape(npol=2, nchan=4), bps=8,
    start_time=2018-01-14T14:11:33.000>
>>> fh.header0
<GUPPIHeader SRC_NAME= 'J1810+1744'
             OBSERVER= 'NikhilMahajan'
             RA_STR  = '18:10:37.2800'
             DEC_STR = '+17:44:37.3801'
             TELESCOP= 'Arecibo '
...
>>> d = fh.read()
>>> d.shape
(3904, 2, 4)
>>> d[:3, 0, 1]    
array([-32.-10.j, -15.-14.j,   9.-13.j], dtype=complex64)
>>> fh.close()

Note that fh.samples_per_frame represents the number of samples per frame excluding overlap samples, since the stream reader works on a linearly increasing sequence of samples. Frames themselves have access to the overlap, and fh.header0.samples_per_frame returns the number of samples per frame including overlap. Furthermore, the overlap in the last frame is used for the stream, which is the reason that the shape above is not an integer multiple of fh.samples_per_frame, but rather 960*4+64=3904.

To set up a file for writing as a stream is possible as well. Overlap must be zero when writing (so we set samples_per_frame to its stream reader value from above and ignore the extra 64 samples we got from the reader):

>>> from astropy.time import Time
>>> fw = guppi.open('puppi_test.{file_nr:04d}.raw', 'ws',
...                 frames_per_file=2, sample_rate=250*u.Hz,
...                 samples_per_frame=960, pktsize=1024,
...                 time=Time(58132.59135416667, format='mjd'),
...                 npol=2, nchan=4)
>>> fw.write(d[:-64])
>>> fw.close()
>>> fr = guppi.open('puppi_test.{file_nr:04d}.raw', 'rs')
>>> d2 = fr.read()
>>> (d2 == d[:-64]).all()
True
>>> fr.close()

Here we show how to write a sequence of files by passing a string template to open, which prompts it to create and use a filename sequencer generated with GUPPIFileNameSequencer. One may also pass a time-ordered list or tuple of filenames to open. Unlike when writing DADA files, which have one frame per file, we specify the number of frames in one file using``frames_per_file``. Note that typically one does not have to pass PKTSIZE, the UDP data packet size (set by the observing mode), but the sample file has small enough frames that the default of 8192 bytes is too large. Baseband only uses PKTSIZE to double-check the sample offset of the frame, so PKTSIZE must be set to a value such that each payload, excluding overlap samples, contains an integer number of packets. (See API links for further details on how to read and write file sequences.)

Reference/API

baseband.guppi Package

Green Bank Ultimate Pulsar Processing Instrument (GUPPI) format reader/writer.

Functions

info(name, **kwargs)

Collect GUPPI file information.

open(name[, mode])

Open GUPPI file(s) for reading or writing.

Classes

GUPPIFileNameSequencer(template[, header])

List-like generator of GUPPI filenames using a template.

GUPPIFrame(header, payload[, valid, verify])

Representation of a GUPPI file, consisting of a header and payload.

GUPPIHeader(*args[, verify, mutable])

GUPPI baseband file format header.

GUPPIPayload(words, *[, header, ...])

Container for decoding and encoding GUPPI payloads.

Class Inheritance Diagram

Inheritance diagram of baseband.guppi.base.GUPPIFileNameSequencer, baseband.guppi.frame.GUPPIFrame, baseband.guppi.header.GUPPIHeader, baseband.guppi.payload.GUPPIPayload

baseband.guppi.header Module

Definitions for GUPPI headers.

Implements a GUPPIHeader class that reads & writes FITS-like headers from file.

Classes

GUPPIHeader(*args[, verify, mutable])

GUPPI baseband file format header.

Class Inheritance Diagram

Inheritance diagram of baseband.guppi.header.GUPPIHeader

baseband.guppi.payload Module

Payload for GUPPI format.

Classes

GUPPIPayload(words, *[, header, ...])

Container for decoding and encoding GUPPI payloads.

Class Inheritance Diagram

Inheritance diagram of baseband.guppi.payload.GUPPIPayload

baseband.guppi.frame Module

Classes

GUPPIFrame(header, payload[, valid, verify])

Representation of a GUPPI file, consisting of a header and payload.

Class Inheritance Diagram

Inheritance diagram of baseband.guppi.frame.GUPPIFrame

baseband.guppi.file_info Module

The GuppiFileReaderInfo property.

Overrides what can be gotten from the first header.

Classes

GUPPIFileReaderInfo([parent])

Class Inheritance Diagram

Inheritance diagram of baseband.guppi.file_info.GUPPIFileReaderInfo

baseband.guppi.base Module

Functions

open(name[, mode])

Open GUPPI file(s) for reading or writing.

info(name, **kwargs)

Collect GUPPI file information.

Classes

GUPPIFileNameSequencer(template[, header])

List-like generator of GUPPI filenames using a template.

GUPPIFileReader(fh_raw)

Simple reader for GUPPI files.

GUPPIFileWriter(fh_raw)

Simple writer/mapper for GUPPI files.

GUPPIStreamBase()

Provides samples_per_frame and fast index getting/setting.

GUPPIStreamReader(fh_raw[, squeeze, subset, ...])

GUPPI format reader.

GUPPIStreamWriter(fh_raw, header0[, squeeze])

GUPPI format writer.

Class Inheritance Diagram

Inheritance diagram of baseband.guppi.base.GUPPIFileNameSequencer, baseband.guppi.base.GUPPIFileReader, baseband.guppi.base.GUPPIFileWriter, baseband.guppi.base.GUPPIStreamBase, baseband.guppi.base.GUPPIStreamReader, baseband.guppi.base.GUPPIStreamWriter