Data Streaming

Each data stream on a site is uniquely specified by two ids. The stream_id, which is the id given to a particular smurf-slot. Often this will be the the uxm assembly ID, but currently defaults to something like crate1slot2. The session_id is the id given to a particular session for a given slot. This is determined by the smurf-streamer, and is simply the timestamp that the first G3Frame was received.

Data Taking

Sodetlib provides two methods for taking data. The take_g3_data function can be used to stream data for a specified length of time. The stream_g3_on and stream_g3_off functions can be used to and stop a stream manually, which can be used to take data for indeterminite amounts of time.

All three of these functions return the session_id of the stream they corresond to. The stream_id can be accessed through cfg.stream_id using the DetConfig object which will always be defined for smurf operation.

Note

The streaming methods defined in pysmurf will still generate G3 files, so any data streams produced by pysmurf functions or older sodetlib functions will be archived and loadable through the sodetlib load methods, though you may need to search out the corresponding stream-id.

However, by default the sodetlib stream methods no longer generate .dat files (the native pysmurf format) to save space on smurf-servers. If you want to save .dat files as well as G3, you can run the sodetlib stream methods with make_datfile=True.

Data Loading

Sodetlib also provides functions for loading data streams into axis-managers using the sotodlib load_smurf framework. To load a particular stream, you can simply call load_session which takes the stream_id and session_id as input, and will return an axis-manager for the corresponding stream session. If running on a system that is not a smurf-server such as simons1, the base_dir option will need to be passed with the base directory of timestreams, since the default is only true for smurf-servers.

To get the SmurfStatus of a given session, you can use the load_session_status function which takes the stream_id, session_id, and optionally the base_dir and will return the corresponding SmurfStatus object which has relevent metadata info from the g3 stream.

Example

Here’s a simple example of taking and loading data using the sodetlib functions:

import sodetlib as sdl
from sodetlib.det_config import DetConfig

cfg  = DetConfig()
cfg.load_config_files(slot=2)
S = cfg.get_smurf_control()

sid = sdl.take_g3_data(S, 30)
am = sdl.load_session(cfg.stream_id, sid)
status = sdl.load_session_status(cfg.stream_id, sid)

API

sodetlib.stream.get_session_files(stream_id, session_id, idx=None, base_dir='/data/so/timestreams')

Gets a list of all files on the system corresponding to a given streaming session.

Parameters:

  • stream_id (str) – stream_id for the stream you wish to load. Often this will be in cfg.stream_id

  • session_id (int) – Session id corresonding with the stream session you wish to load. This is what is returned by the stream-taking functions.

  • idx (int, list(int), optional) – Index of file you wish to load. Long streams are chunked into 10-minute files, and this parameter can used to isolate a smaller part of a long stream.

  • base_dir (str) – Base directory where timestreams are stored. Defaults to /data/so/timestreams.

sodetlib.stream.load_session(stream_id, session_id, idx=None, base_dir='/data/so/timestreams', show_pb=False, **kwargs)

Loads a stream-session into an axis manager. Any additional keyword arguments will be passed to the load_smurf.load_file function.

Parameters:

  • stream_id (str) – stream_id for the stream you wish to load. Often this will be in cfg.stream_id

  • session_id (int) – Session id corresonding with the stream session you wish to load. This is what is returned by the stream-taking functions.

  • idx (int, list(int), optional) – Index of file you wish to load. Long streams are chunked into 10-minute files, and this parameter can used to isolate a smaller part of a long stream.

  • base_dir (str) – Base directory where timestreams are stored. Defaults to /data/so/timestreams.

sodetlib.stream.load_session_status(stream_id, session_id, base_dir='/data/so/timestreams')

Gets SmurfStatus object for a given stream.

Parameters:

  • stream_id (str) – stream_id for the stream you wish to load. Often this will be in cfg.stream_id

  • session_id (int) – Session id corresonding with the stream session you wish to load. This is what is returned by the stream-taking functions.

  • base_dir (str) – Base directory where timestreams are stored. Defaults to /data/so/timestreams.

sodetlib.stream.stream_g3_off(S, emulator=False)

Stops the G3 data-stream. Returns the session-id corresponding with the data stream.

Parameters:

  • S (S) – Pysmurf control object

  • emulator (bool) – If True, will enable the emulator source data-generator. Defaults to False.

Returns:

session_id – Id used to read back streamed data

Return type:

int

sodetlib.stream.stream_g3_on(S, make_freq_mask=False, emulator=False, tag=None, channel_mask=None, filter_wait_time=2, make_datfile=False, downsample_factor=None, downsample_mode=None, filter_disable=False, filter_order=None, filter_cutoff=None, stream_type=None, subtype='stream', enable_compression=None)

Starts the G3 data-stream. Returns the session-id corresponding with the data stream.

Parameters:

  • S (S) – Pysmurf control object

  • make_freq_mask (bool, optional) – Tell pysmurf to write and register the current freq mask

  • emulator (bool) – If True, will enable the emulator source data-generator. Defaults to False.

  • tag (optional(string)) – If set, this will attach the tag to the g3 stream

  • channel_mask (optional(list)) – List of channels to stream. If None, this will stream all channels with a probe tone

  • filter_wait_time (float) – Seconds to wait after resetting the filter before writing data to disk

  • bool (make_datfile) – If True, will create a datfile

  • downsample_factor (optional(int)) – Downsample factor. If None, this will be pulled from the device cfg.

  • downsample_mode (optional(string)) – Downsample mode, can either be ‘internal’ or ‘external’. If not set, this will be pulled from the device cfg.

  • filter_disable (bool) – If true, will disable the downsample filter before streaming.

  • filter_order (int) – Order of the downsample filter. Read from device config by default.

  • filter_cutoff (float) – The cutoff frequency in Hz for the downsample filter. Read from device config by default.

  • stream_type (optional(string)) – Type of stream. This should either be “obs” or “oper”. If None, it will be determined based off of the subtype. (‘stream’, ‘cmb’, and ‘cal’, will automatically be set to ‘obs’ type)

  • subtype (optional(string)) – Stream subtype. This defaults to ‘stream’.

  • enable_compression (optional(bool)) – If true, will tell the smurf-streamer to compress data.

Returns:

session_id – Id used to read back streamed data

Return type:

int

sodetlib.stream.take_g3_data(S, dur, **stream_kw)

Takes data for some duration

Parameters:

  • S (SmurfControl) – Pysmurf control object

  • dur (float) – Duration to take data over

Returns:

session_id – Id used to read back stream data

Return type:

int