TrueTrace / Data format

Stereo egocentric data format

How a TrueTrace stereo episode is laid out on disk, which frame and clock everything is expressed in, and what each file contains. Examples are taken from our evaluation sample (one 75 s episode of fine handwork). For custom collection the same structure is used unless you specify your own schema.

1. Package layout

package/
├── README.md  LICENSE.txt  MANIFEST.sha256
├── episodes/episode_000000/
│   ├── cam0.mp4  cam1.mp4            # stereo pair, frame i ↔ frame i
│   ├── imu.csv                       # accelerometer + gyroscope
│   ├── calibration.json              # intrinsics, extrinsics, IMU info
│   ├── hands_3d.jsonl                # one line per frame
│   └── stereo_hand3d_meta.json       # method and summary statistics
├── annotations/
│   ├── SCHEMA.json
│   └── episode_000000_segments.jsonl
├── preview/depth_preview.mp4
└── qc/
    ├── index.html  metrics.json      # QC report and raw numbers
    ├── refined_extrinsics.json
    └── rect/                         # rectified sample pairs

One folder per episode. Every file in the package is listed in MANIFEST.sha256; run shasum -a 256 -c MANIFEST.sha256 to verify a delivery.

2. Coordinate frame and timeline

• Reference frame: the left camera, cam0, in the OpenCV convention — x right, y down, z forward. All 3D quantities are in metres.
• Timeline: the cam0 frame index. Frame i of cam0.mp4 pairs with frame i of cam1.mp4; the pairing is checked on sensor timestamps (see Quality).
• Clip vs. source: an episode can be cut from a longer recording. clip_frame and t_s count from the start of the clip; source_frame is the frame number in the original recording.

3. calibration.json

{
  "model": "kannala_brandt",                 # fisheye model, 4 distortion coefficients
  "cameras": {
    "cam0": { "width": 1920, "height": 1080,
              "K": [fx, 0, cx, 0, fy, cy, 0, 0, 1],   # 3×3, row-major
              "D": [k1, k2, k3, k4] },
    "cam1": { … }
  },
  "extrinsics": {
    "cam0->cam1": { "t": [x, y, z], "q": [x, y, z, w] },   # metres, unit quaternion
    "cam0->imu0": { "t": [x, y, z], "q": [x, y, z, w] }
  },
  "extrinsics_convention": "child pose expressed in parent frame …",
  "baseline_mm": 62.18,
  "imu": { "rate_hz": 476.87, "time_offset_to_cam0_s": -0.000775 },
  "refined_stereo_extrinsics": { "R": [[…], […], […]], "T": [ … ] }
}

Factory extrinsics are stored as the child pose expressed in the parent frame: X_cam1 = Rᵀ (X_cam0 − t), with R from the cam0->cam1 quaternion. Refined stereo extrinsics, present when our epipolar check flags drift, use the OpenCV stereo convention X_cam1 = R·X_cam0 + T. Both are delivered; 3D hands and depth use the refined values when present. time_offset_to_cam0_s is the camera–IMU time offset reported by the device.

4. imu.csv

t_s,frame_index,accel_x,accel_y,accel_z,gyro_x,gyro_y,gyro_z
0.000791,300,-2.815581,-0.354342,9.500193,-0.024435,0.017104,-0.012217
0.002890,300,-2.829947,-0.368707,9.500193,-0.024435,0.014661,-0.009774

• t_s — seconds from the start of the clip.
• frame_index — the cam0 frame (source numbering) the sample falls in, so IMU can be joined to video without resampling.
• accel_* — linear acceleration in m/s², including gravity; gyro_* — angular rate as reported by the device.
• About 477 samples per second on the reference rig.

5. hands_3d.jsonl

{"clip_frame": 99, "source_frame": 399, "t_s": 3.3,
 "hands": [{
   "hand": "Right", "confidence": 0.983,
   "keypoints_2d_cam0_px":        [[1178.7, 337.3], …],   # 21 × 2, cam0 pixels
   "keypoints_3d_cam0_m":         [[0.0890, -0.0701, 0.3009], …],   # 21 × 3, registered shape
   "keypoints_3d_triangulated_m": [[0.1005, -0.0650, 0.2916], …],   # 21 × 3, raw stereo
   "reprojection_error_px":       [ … ],   # 21
   "shape_fit_residual_mm":       [ … ]    # 21
 }]}

• Keypoint order (common 21-point hand convention): 0 wrist, 1–4 thumb, 5–8 index, 9–12 middle, 13–16 ring, 17–20 little.
• keypoints_3d_triangulated_m — raw points triangulated from the two views. keypoints_3d_cam0_m — a 21-point hand shape rigidly registered to those points, for consistent proportions. shape_fit_residual_mm is the per-joint distance between the two.
• A frame lists only the hands that passed the reprojection gate; an empty hands array means no hand was kept for that frame.
• Left/right labels come from the 2D model and can be swapped in the egocentric view.

6. segments.jsonl

{"episode_index": 0, "segment_id": "seg_000",
 "start_s": 0, "end_s": 33, "start_frame": 0, "end_frame": 989,
 "task": "hair_clip_carding", "action": "attach_clips_to_card",
 "objects": "metal hair clips; display card", "hands": "both",
 "description": "Left hand holds the display card; right hand picks a clip …",
 "review_status": "draft", "boundary_tolerance_s": 3}

Frame indices are inclusive, on the cam0 clip timeline. review_status is draft or reviewed, and boundary_tolerance_s states the precision of the boundaries. The segments in the evaluation sample are drafts; label sets and boundary precision are agreed per project. Field definitions ship in annotations/SCHEMA.json.

7. qc/ and MANIFEST.sha256

qc/metrics.json holds the raw numbers behind every quality gate — per-pair sync differences, per-frame intervals for each camera, IMU sample intervals, epipolar disparity per sampled pair — plus a verdicts list with pass/fail per gate. qc/index.html renders them with synchronized playback and charts, offline. See Quality for how each gate is computed.

8. HDF5 export

On request each episode is also delivered as one HDF5 file with every stream on the cam0 frame timeline:

/meta                       attrs: fps, coordinate_frame, keypoint_order
/video/cam0, /video/cam1    attrs: mp4 file name (video kept as mp4 by default)
/frames/cam0, /frames/cam1  JPEG per frame (optional, embedded)
/time/t_s                   float64 [N]
/hands/keypoints_3d_m       float32 [N, 2, 21, 3]   # [:,0] left, [:,1] right; NaN if absent
/hands/keypoints_2d_px      float32 [N, 2, 21, 2]
/hands/reprojection_px      float32 [N, 2, 21]
/hands/present              bool    [N, 2]
/imu/t_s, /imu/frame_index, /imu/accel, /imu/gyro
/calibration                attrs: calibration.json as JSON
/annotations/segments       JSON;  /annotations/per_frame_action  str [N]

For the evaluation sample this gives N = 2,250 frames and 35,724 IMU samples in a single file.

9. Other formats

• MCAP — the raw sensor recording, with the device's own calibration and IMU channels.
• Trajectory HDF5 — one trajectory per file with meta_info/, JPEG frames per camera, int64 nanosecond timestamps and per-frame language annotations, for pipelines that expect that layout.
• LeRobot or your own schema — on request.