Jashwanth Bamidi

Embedded Hackathon prototype

VisionVest

A vest that guides its wearer with vibration, leaving their ears free. A phone camera decides the direction; the vest delivers it.

Role
Firmware, protocol and hardware integration
When
StarkHacks 2026
Tools
ESP32-S3, C++, BLE, JSON, Ultrasonic sensors, PWM, Core ML, React
Three-quarter VisionVest render with a front-mounted camera phone, ultrasonic sensors, yellow tactile motors, an ESP32 powered by a USB power bank, and a separate 9 V battery connected to the motor driver.
Conceptual illustration. A vest that communicates by touch. Front-mounted phone, USB power bank for the ESP32 and a separate 9 V motor-driver battery. Illustrative reconstruction; mounting positions are approximated.

What I did

I built the vest side: ESP32-S3 firmware, three ultrasonic rangefinders, four motor channels and a Wi-Fi telemetry dashboard.

The firmware checks every phone command and lets it expire after 300 ms.

Result

  • Demonstrated at StarkHacks 2026, with a live dashboard of commands, motors and hazards.
  • Every command type- and range-checked; duplicates rejected; stale commands expire after 300 ms.
  • Nonblocking three-sensor ultrasonic sensing with five-sample filtering.
  • A hackathon prototype: no user study, clinical validation or perception-accuracy figures.

Devpost write-upPrototype video

Design decision

Commands that expire

Each phone command lives for 300 ms after it arrives. If the phone pauses or disconnects, the motor stops instead of buzzing on an old decision.

Try it

Pick a case and drag the time slider.

A front cue with a 300 ms lifetime, never refreshed. Ultrasonic sensing is off in this mode.

frontfrom behindFront motorBack motorRight motorRangefindersESP32-S3Left motorback, left, rightLeft and right are the wearer's.
  1. Phone sentfront, intensity 150, lifetime 300 ms, mode object_nav
  2. Command statelive
  3. Local sensorsignored in this mode
  4. Output owned byLive phone command, steady
MotorRequestedApplied duty
front150255 (on)
back—0
left—0
right—0

Active motors run at full duty, 255, whatever intensity is requested.

Illustrative. Requested versus applied. Built from the firmware's documented rules; not a full firmware emulator.

More detail

Why split the phone and the vest

The phone is good at camera perception; a microcontroller is good at sensors and motors. BLE between them carries small decisions, not video, and isn't deterministic.

Firmware and command handling

Each BLE write is one JSON command: mode, direction, intensity, pattern, priority, lifetime, confidence, distance and sequence number. The firmware checks each field and stamps the arrival time.

Every loop it expires stale commands, reads the sensors, then arbitrates: local danger alert, live phone command, local caution, idle. Ultrasonic hazards only count in awareness mode.

This firmware version drives every active motor at full duty: a command asking for 150 produces 255. Direction and pulse pattern tell cues apart, and the dashboard shows requested and applied values side by side.

Other design decisions

  • A narrow interface

    Perception on the phone, sensing and motors on the ESP32, joined by one small JSON command. Instead of: streaming frames or doing everything on the phone.

    Easy to log, but the JSON has to fit the BLE write size.

  • Show requested and applied output

    Telemetry reports both the command and what the motors do. Instead of: debugging by feel.

    Makes overrides and full-duty output visible at a glance.

  • FPGA Edge DetectionLive camera video in, edge outlines out on HDMI, all in FPGA hardware across three clock domains.
  • Spatial Audio ResearchHeadphone audio that stays fixed in the room while the listener turns, with head-pose prediction.