McGill
AI Lab
Join
sEMGxRoboticHand

In Progress

Device-Agnostic sEMG (open-sEMG-16 + Domain-General Pose Encoder)

Recent advances in surface electromyography (sEMG) decoding, such as Meta’s EMG2Pose, EMG2QWERTY datasets, and their associated pretrained models, have demonstrated high-accuracy hand-pose and typing reconstruction. However, these breakthroughs rely on Meta’s proprietary acquisition hardware (sEMG-RD), limiting reproducibility and broader utility for independent research and open development. This repo aims to make the hardware, data, model pipeline transparent, modifiable, and benchmarkable.

  • Custom acquisition hardware (open-sEMG-16): a 16-channel, high-fidelity, wrist-wearable sEMG platform built from commercially available components, with design files + firmware to enable end-to-end reproducibility.
  • A device-/domain-robust encoder model that maps multi-channel muscular activity to hand pose (joint angles / kinematic representation), with an explicit focus on out-of-domain generalization across:
  • unseen users (anatomical variation),
  • electrode shifts / rotations / contact changes,
  • session-to-session drift,
  • dataset / protocol mismatch.

High-level modules (see top-level folders):

  • Hardware/ — open-sEMG-16 hardware stack (schematics/PCB, electrode layout, enclosure notes, firmware hooks, bring-up docs).
  • src/ — acquisition + preprocessing utilities (streaming, windowing, filtering, normalization, dataset I/O, evaluation harness).
  • data/ — dataset organization, conversion scripts, and format docs (raw → aligned → windowed → model-ready).
  • hand-joint-labeling/ — tooling for pose labeling / alignment (e.g., joint definitions, coordinate frames, annotation utilities).
  • LUNA/ — encoder model code (feature encoder + pose head) and training/eval entrypoints.

Status: still in active development; APIs and folder contents may shift. We aim to keep experiments config-driven and results reproducible as the codebase stabilizes.

Acquisition to Dataset to Model: End-to-End Pipeline

This repo is organized around a reproducible pipeline:

1) Acquisition & Streaming

  • synchronized multi-channel sampling
  • timestamping + packetization
  • stream integrity checks (drop detection, reordering, CRC if available)
  • host-side capture and persistent storage

2) Signal Processing

  • band-pass filtering (and optional notch)
  • per-channel normalization (robust stats recommended)
  • windowing (fixed-length frames with overlap)
  • optional time–frequency transforms (e.g., STFT / filterbanks) for encoder variants
  • augmentation hooks to simulate electrode shift / noise / drift

3) Label Alignment (pose supervision)

  • consistent hand model definition (joint ordering, DoFs, coordinate frame)
  • alignment utilities for pose timestamps vs EMG timestamps
  • split generation for cross-user / cross-session / cross-placement evaluation

4) Encoder + Pose Head

We focus on an encoder that learns a shift-tolerant representation of sEMG:

Input: X ∈ R^{C×T} (C channels, T samples)
Encoder: temporal feature extractor + cross-channel mixing
Head: regression to pose representation (joint angles / keypoints / low-dim hand state)

Contributors

  • Katherine Lambert
  • Emir Sahin
  • Lia Brahami
  • Karen Chen Lai

References