Hand Tetris: Playing Tetris With Just Your Hands

Why I Built It

Some side projects start with a grand vision. This one started with a question: can I make Tetris playable with just my hands? I had been experimenting with MediaPipe in the browser and was struck by how much signal you could extract from a webcam feed in real time. Finger positions, pinch detection, hand orientation. Most demos stopped at drawing a skeleton overlay. I wanted to take that signal somewhere useful. Tetris felt like the right game. The controls are simple enough to map to gestures (move left, move right, rotate, drop), but demanding enough that any lag or imprecision becomes immediately frustrating. If I could make gesture-controlled Tetris feel good, not just functional, I would have actually solved something. So I built it. From scratch. As the designer and the developer.

The Design

Retro Arcade in the Browser

I wanted the game to feel like a cabinet you would find in an arcade, not a browser tab. That meant committing to a visual language: deep purple-black backgrounds (#0e0a14), a warm gold accent (#f5b651), pixel-perfect typography using Silkscreen for headings and JetBrains Mono for data readouts. The UI is intentionally dense. Score, level, lines cleared, and the next-piece queue are all visible at once. Arcade games do not hide information from you. They throw it in your face and let you build the mental model.

Why 3D for a 2D Game

Classic Tetris is 2D. Blocks fall, rows clear, game ends. I chose to render the playfield in 3D anyway, and it was not for novelty. The 3D perspective gives the hand tracking system something to work with visually. When you are steering a falling piece with your physical hand in real space, a flat grid feels disconnected. A camera angled down into a 3D well, with depth, shadows, and a subtle floor, makes the mapping feel grounded. Your hand is in 3D space. The game should look like it is too. I used React Three Fiber with instanced meshes so the renderer stays performant even as locked blocks pile up. Each piece type has its own color. A ghost piece renders as a wireframe at the projected landing position so you always know where the piece will land. The crown jewel of the 3D work: line-clear physics. When you clear a row, the cells do not just disappear. Each cleared block explodes outward as a tumbling shard, with random lateral velocity, an upward pop, then gravity pulls it down while it spins on all three axes. It lasts 750ms and makes clearing lines feel visceral.

Designing the Gesture System

This was the hardest design problem in the project. Hand tracking gives you a stream of raw coordinates. 21 landmarks per hand, updated 30 times per second. The naive approach is to map X position directly to piece column. The result is unplayable: the piece thrashes left and right as your hand naturally wobbles. The solution was a layered approach: Smoothing. I apply exponential smoothing (alpha=0.35) to the raw X coordinate before doing anything else. This removes high-frequency jitter without adding noticeable lag. Hysteresis. Column changes only register after the hand has moved 0.65 units from the current column center. Small wobbles do not trigger movement. You have to mean it. Pinch to rotate. Closing your thumb and index finger rotates the piece. But detecting a pinch reliably required more than just measuring distance. I normalize the thumb-index gap against the full hand size so it works at any camera distance, and require the middle, ring, and pinky fingers to be extended. A closed fist does not rotate. An intentional pinch does. Post-pinch freeze. After rotating, movement is frozen for 200ms. Without this, the act of pinching would shift the hand slightly and move the piece. A phantom drift that felt like a bug even though it was not. Drop zone. When your hand drops below 70% of the frame height, the piece enters fast-fall at a 50ms drop interval. No gesture needed. Just lower your hand. Together, these made gesture control feel controllable. Not perfect. You are still fighting the physics of your own hand. But that friction turned out to be part of the fun.

The Build

Architecture

I kept the game logic strictly separated from everything else. The engine is a set of pure functions: no React, no rendering, no IO. It takes a game state and an action and returns a new state. This made it easy to test, debug, and reason about independently from the chaos of hand tracking and 3D rendering. On top of that:

useGameController bridges inputs (gesture or keyboard) to the engine. It reads the gesture state, computes column targets, detects pinch rising edges, and dispatches moves to the engine on a timer.
Playfield3D owns all Three.js rendering. It reads game state and updates instanced mesh positions, never touching game logic.
VisionFeed handles the camera, runs MediaPipe, and draws the hand skeleton overlay on a canvas.
The leaderboard lives in its own module and talks to Supabase through RPC functions.

Clean separation meant I could work on gesture tuning without touching the renderer, and vice versa.

Audio

Browser audio is a minefield. Autoplay policies block sound until a user gesture. Audio files fail silently on some devices. Overlapping sounds cut each other off. I built a dual-mode audio system. The primary mode uses a pool of three HTMLAudioElement instances per sound type, cycling through them so overlapping sounds do not cut each other off. If MP3 loading fails, the system falls back to a procedural Web Audio API synth. Each action has a defined tone:

Rotate: a rising square-wave chirp (520 to 880Hz, 70ms)
Lock: a two-layer thunk (low square + higher triangle)
Line clear: a four-note arpeggio (C, E, G, C)
Game over: a descending minor melody

The synth sounds intentionally retro and works completely offline.

The Leaderboard Anti-Cheat

A public leaderboard on a browser game is an invitation for abuse. I spent real time on this. All writes go through Supabase RPC functions marked SECURITY DEFINER. Direct table writes are blocked at the RLS level. The flow works like this:

Player registers a name once. The server validates the name (charset, length, profanity check), then issues a 32-character hex token stored in localStorage.
Every score submission requires that token. The server re-validates it before writing.
Score plausibility is checked against a formula: (lines + 4) x 800 x level. Scores outside that range are rejected.
Play time is validated. Submissions require at least 600ms per cleared line. Instant-play injection gets rejected.
A 10-second rate limit prevents score flooding.

It is not unbreakable, but it is meaningfully harder to abuse than nothing. For a browser game leaderboard, that is the right bar.

The Hard Problems

Making imprecise input feel intentional. The gesture system went through five or six iterations before it stopped feeling broken. The breakthrough was accepting that hand tracking would never be pixel-perfect and designing around that. Larger dead zones, hysteresis, post-rotation freezes. The goal was not precision. It was a feeling of control. Static deployment with a live backend. The game is a fully static Next.js export hosted on GitHub Pages, with no server. But it has a real leaderboard. The trick: all the server logic lives in Supabase's RPC layer. The client is dumb. The database enforces everything. Line-clear physics at 60fps. Running physics simulations for 12 shards per cleared row, across potentially 4 rows at once, while maintaining 60fps in Three.js required instanced meshes and careful animation loop management. The shards share geometry. Only transforms update per frame.

Platform Evolution

Week 1

Pure game engine

Game logic written as pure functions with no UI coupling. State in, state out. No React, no rendering. This made every later layer easier to debug in isolation.

Week 2

3D playfield

React Three Fiber with instanced meshes, a ghost piece wireframe, and line-clear physics. Each cleared block explodes as a tumbling shard for 750ms.

Week 3

Gesture system

MediaPipe hand tracking with exponential smoothing, column hysteresis, pinch-to-rotate, and a post-rotation freeze. Five iterations before it felt controllable, not broken.

Week 4

Leaderboard + anti-cheat

Supabase RPC layer with one-time token auth, score plausibility math, play-time validation, and a 10-second rate limit. Hard enough to matter, light enough for a browser game.

Week 5

Audio system

Dual-mode audio: MP3 pool for primary sounds, Web Audio API synth as fallback. Each action has a defined procedural tone that works completely offline.

Mar 2025

Launch on GitHub Pages

Fully static Next.js export. No server. The leaderboard logic lives entirely in Supabase's RPC layer so the client stays dumb and deployment stays free.

A Miniclip for gesture-based games

Hand Tetris is the first game in a planned platform: a browser arcade built entirely around gesture controls. Each game would be a different design challenge for the same input system, a hand in front of a camera.

Lessons

Design the input before the interface I nearly built the full UI before realising the gesture system was broken. Fix the input first. Everything else follows.
Imprecision is a constraint, not a failure Hand tracking will never be pixel-perfect. The job was designing around that with dead zones and hysteresis, not fighting it.
Friction can be the feature Steering a piece with your physical hand turned out to be part of the fun. Not every interaction needs to be frictionless.
Fallbacks are real features Keyboard controls, synth audio, local leaderboard. Each started as a backup. Each ended up being genuinely used.
Separation pays off early Pure engine functions with no UI coupling meant I could iterate on gestures without touching the renderer, and vice versa.
Anti-cheat is design, not just engineering A public leaderboard without trust is useless. The validation layer exists so the scores mean something.

FAQ

Yes. Keyboard controls are fully supported: arrow keys to move and rotate, spacebar to drop. The gesture system is the primary experience but the keyboard fallback is first-class, not an afterthought.

Any modern Chromium-based browser works best. Firefox supports MediaPipe but has slightly lower hand tracking performance. Safari on iOS does not support WebRTC in the way MediaPipe requires, so mobile is keyboard-only.

MediaPipe runs on a separate canvas thread and targets 30fps for landmark detection. The game itself renders at 60fps via Three.js. The two loops are intentionally decoupled so a slow camera frame does not stutter the game.

Determined enough, anyone can bypass it. The goal was not an impenetrable system. It was making casual abuse harder than it is worth. Token validation, plausibility math, and rate limiting stop the obvious attacks. The leaderboard is for fun, not for money, so that bar is the right one.

The leaderboard runs on Supabase with RPC functions doing all the validation server-side. If you fork the repo, you would need to create your own Supabase project, run the migration SQL, and update the environment variables. The game works completely without a leaderboard if you skip that setup.

The gesture system is the reason. Your hand moves in real 3D space. A flat grid feels disconnected from that. A playfield with depth, a camera angle, and physical shards when rows clear makes the hand-to-game mapping feel grounded in the same space your hand is in.