Thousand Worlds — Hierarchical Predictive World Model

01Motivation

The brain doesn't reconstruct the world. It predicts it.

Jeff Hawkins' Thousand Brains Theory proposes that the neocortex consists of many parallel predictive models, each operating at different timescales and levels of abstraction. Predictions flow down the cortical hierarchy; prediction errors flow up. Learning is driven entirely by the discrepancy between what was predicted and what was observed.

This architecture implements that principle for video understanding. A frozen visual backbone encodes frames; a three-level learned hierarchy compresses time at multiple scales, predicts future states in latent space (never pixels), and uses top-down context from abstract levels to constrain concrete predictions.

The approach shares principles with LeCun's JEPA (Joint Embedding Predictive Architecture), particularly the commitment to latent-space prediction over pixel reconstruction. Where V-JEPA 2 operates at a single temporal scale, this project builds the multi-scale temporal hierarchy that both Hawkins and LeCun theorized but neither has fully implemented.

Cortical Hierarchy

Multiple levels at different timescales. Top-down predictions constrain bottom-up processing. Prediction error is the only learning signal. Directly implements Hawkins' theory.

Latent-Space Prediction

No pixel reconstruction during training. Encoders learn what to discard. The model predicts abstract states, not observations. Overlaps with JEPA principles.

Emergent Actions

Transition codes discovered via prediction bottleneck, not labels. Different levels learn different action vocabularies — from optical flow to scene-level goals.

02Notation

Symbols used throughout the architecture description.

Symbol	Meaning
$x_t$	Raw sensory frame at time $t$
$z_t^{(\ell)}$	Latent state at level $\ell$, time $t$
$a_t^{(\ell)}$	Inferred action / transition code at level $\ell$
$E^{(\ell)}$	Encoder from level $\ell{-}1$ representations to level $\ell$
$P^{(\ell)}$	Predictor (transition model) at level $\ell$
$D^{(\ell)}$	Top-down decoder from level $\ell$ to level $\ell{-}1$
$T_\ell$	Temporal stride of level $\ell$
$\Omega(z)$	Sparsity penalty (SDR constraint)
$\rho$	Target activation fraction (e.g. 0.05 = 5% of units active)

Level 0 is the pixel level: $z_t^{(0)} = x_t$. A frozen DINOv2 backbone encodes frames into $z_t^{(1)}$.

03The Core Loop

At every level, the same four operations repeat.

Encode

Compress a sequence of lower-level states into a single latent:

z_t^{(\ell)} = E^{(\ell)}\!\left(z_{t-T_\ell+1}^{(\ell-1)},\; \ldots,\; z_t^{(\ell-1)}\right)

The encoder $E^{(\ell)}$ is a learned temporal pooling operation — a causal transformer that attends to the most informative moments in the window and produces a fixed-size output, forcing compression.

Infer Action

The transition code $a_t^{(\ell)}$ captures what changed between consecutive latent states:

a_t^{(\ell)} = f_a^{(\ell)}\!\left(z_t^{(\ell)},\; z_{t+T_\ell}^{(\ell)}\right)

This is learned jointly — $a_t^{(\ell)}$ is a bottleneck that must be informative enough to support prediction but compact enough to generalize. It is not a motor command; it's whatever abstract transition descriptor the level needs.

Predict

Given current state and action, predict next state, biased by top-down context from the level above:

\hat{z}_{t+T_\ell}^{(\ell)} = P^{(\ell)}\!\left(z_t^{(\ell)},\; a_t^{(\ell)}\right) + \alpha \cdot c_t^{(\ell+1)}

where $c_t^{(\ell+1)} = D^{(\ell+1)}(z_k^{(\ell+1)})$ is the top-down context and $\alpha$ is a learned scalar.

Learn from Error

\mathcal{L}^{(\ell)} = \left\| z_{t+T_\ell}^{(\ell)} - \hat{z}_{t+T_\ell}^{(\ell)} \right\|^2 + \lambda_s \, \Omega\!\left(z_t^{(\ell)}\right) + \lambda_a \, \Omega\!\left(a_t^{(\ell)}\right)

The $\Omega$ terms are sparsity penalties. Prediction error drives all learning — there is no reconstruction loss back to pixels.

04Temporal Hierarchy

Each level consumes $T_\ell$ steps of the level below and produces one latent. Total receptive field: $T_1 \times T_2 \times T_3 = 64$ frames.

Level 1
$T_1 = 1$

Frame-level. Encodes each frame independently via frozen DINOv2 → learned projection. Latent dim 256, action dim 64. Predicts $z_{t+1}^{(1)}$ — what objects are where next. Timescale: ~33ms.

Level 2
$T_2 = 8$

Action-level. A causal transformer compresses 8 level-1 latents into one vector $z_k^{(2)}$ capturing the gist of what happened ("ball moved left", "hand reached for cup"). Latent dim 128, action dim 32. Timescale: ~250ms.

Level 3
$T_3 = 8$

Scene-level. Compresses 8 level-2 latents (= 64 frames) into a scene summary $z_m^{(3)}$ ("person is making coffee"). Latent dim 64, action dim 16. Timescale: ~2s.

Data Flow

Frame x_t │ ▼ E¹: encode single frame → z_t^(1) [every frame, ~33ms] │ ├─→ P¹: predict z_{t+1}^(1) from z_t^(1), a_t^(1), c^(2) │ loss: ‖z_{t+1}^(1) - ẑ_{t+1}^(1)‖² │ ▼ (every 8 frames) E²: encode 8 z^(1)'s → z_k^(2) [every 8 frames, ~250ms] │ ├─→ P²: predict z_{k+1}^(2) from z_k^(2), a_k^(2), c^(3) ├─→ D²: project z_k^(2) → c^(2) for level 1 [top-down] │ ▼ (every 64 frames) E³: encode 8 z^(2)'s → z_m^(3) [every 64 frames, ~2s] │ ├─→ P³: predict z_{m+1}^(3) ├─→ D³: project z_m^(3) → c^(3) for level 2 [top-down]

All arrows carry gradients. The whole model trains end-to-end from video with one loss.

05Joint Action–Sensory Learning

Actions are not labels. They emerge from a prediction bottleneck.

Action codes $a_t^{(\ell)}$ are inferred as the minimal information needed to predict the next state given the current state:

a_t^{(\ell)} = f_a^{(\ell)}\!\left(z_t^{(\ell)},\; z_{t+T_\ell}^{(\ell)}\right)

At level 1, $a_t^{(1)}$ learns optical flow or local motion vectors. At level 2, $a_k^{(2)}$ represents "reach", "push", or "turn". At level 3, $a_m^{(3)}$ represents "go to kitchen" or "start new task".

The action space is not predefined — it emerges from the prediction bottleneck. If a code carries more information than needed, the sparsity penalty compresses it. If too little, prediction error increases and the model expands its use.

When real motor commands are available (e.g., from a robot), they can be fed as additional input to $P^{(\ell)}$ at the lowest level, and $a_t^{(1)}$ learns to align with them.

06Top-Down Prediction

Higher levels constrain lower levels via a context signal.

c_t^{(\ell+1)} = D^{(\ell+1)}\!\left(z_k^{(\ell+1)}\right)

The predictor at level $\ell$ receives this as an additive bias: $\hat{z} = P^{(\ell)}(z, a) + \alpha \cdot c^{(\ell+1)}$. The additive form is simplest; multiplicative gating gives the higher level more control but is harder to train.

Effect: When level 3 has confidently predicted "person continues making coffee", this biases level 2 toward coffee-making sub-actions, which biases level 1 toward the corresponding motion patterns. Surprise at any level propagates up as prediction error.

07Training Objective

A single loss trains everything end-to-end.

\mathcal{L} = \sum_{\ell=1}^{L} \gamma_\ell \, \mathbb{E}_t\!\left[\left\| z_{t+T_\ell}^{(\ell)} - \hat{z}_{t+T_\ell}^{(\ell)} \right\|^2\right] + \lambda_s \sum_{\ell=1}^{L} \mathbb{E}_t\!\left[\Omega\!\left(z_t^{(\ell)}\right)\right] + \lambda_a \sum_{\ell=1}^{L} \mathbb{E}_t\!\left[\Omega\!\left(a_t^{(\ell)}\right)\right]

The weights $\gamma_\ell$ balance levels (all 1.0 by default). All encoders, predictors, top-down decoders, and action inference networks receive gradients from this single objective.

Sparsity / SDR Constraints

Without explicit constraints, the model can collapse: all latents converge to the same point.

\Omega(z) = \left(\frac{\|z\|_1}{d} - \rho\right)^2

where $d$ is the latent dimension and $\rho$ is a target activation fraction (e.g., $\rho = 0.05$). This encourages sparsity (most units near zero), distribution (the which 5% varies across inputs, giving $\binom{d}{k}$ possible patterns), and disentanglement (sparse codes tend to separate independent factors).

08Vision Decoder (Phase 2)

A separate decoder learns to read the frozen latent space — not reshape it.

Phase 1 learns purely in latent space. Phase 2 trains a standalone decoder $D_\text{pixel}$ that maps level-1 latents back to frames. All Phase 1 weights are frozen.

\hat{x}_t = D_\text{pixel}\!\left(z_t^{(1)}\right)

The full decoder objective combines MSE, perceptual, and (optionally) adversarial losses:

\mathcal{L}_\text{decoder} = \left\| x_t - \hat{x}_t \right\|^2 + \lambda_p \sum_l \left\| \phi_l(x_t) - \phi_l(\hat{x}_t) \right\|^2 + \lambda_\text{adv} \, \mathcal{L}_\text{GAN}

Multi-Resolution Decoding

Level	Expected Output	Use Case
$z^{(1)}$	Sharp frame (full detail)	Primary decoder
$z^{(2)}$	Blurry scene layout	Visualize what the 250ms-level "sees"
$z^{(3)}$	Abstract scene type	Visualize episode-level understanding

09Applications

What the model can do once training converges.

Next-State Prediction

Feed frames, predict what comes next at every level. Low levels predict motion; mid levels predict actions; high levels predict scene transitions. Inference is training without the weight update.

Anomaly Detection

Prediction error is a free anomaly signal. High error at low levels = visual glitch. Mid levels = unusual behavior. Top levels = structurally anomalous event. Full spectrum without training separate detectors.

Learned Representations

Latent states as feature vectors for activity recognition, scene classification, object tracking, temporal segmentation. Many tasks need only a linear probe — no fine-tuning.

Action Vocabulary

Each level develops its own transition vocabulary. Low: optical flow. Mid: "pick up", "open". Top: "go to kitchen". Emerges from prediction, not labels. Enables video search by action code.

Mental Simulation

Run the predictor forward without sensory input. Explore branching futures by conditioning on different action codes. Enables counterfactual reasoning and model-based RL.

Dreaming

Cut sensory input entirely. Top-down flow keeps trajectories coherent; without bottom-up error correction, details drift and scenes morph. Useful for data augmentation, planning, and model consolidation.

10Datasets

Streams from HuggingFace. Only the requested samples are downloaded.

Name	HuggingFace Path	Description	Status
`ucf101`	sayakpaul/ucf101-subset	UCF101 action recognition — tiny, ideal for smoke tests	open
`disney`	Wild-Heart/Disney-VideoGeneration-Dataset	Disney animated video clips (640×360)	open
`open-sora`	LanguageBind/Open-Sora-Plan-v1.1.0	Open-Sora video generation clips (1080p)	open
`kinetics400-sample`	JackWong0911/kinetic-400_450samples	Kinetics-400 sample — 450 clips with raw mp4	open
`epic-kitchens`	awsaf49/epic_kitchens_100	EPIC-KITCHENS-100 — 268 full kitchen videos (501 GB)	open
`finevideo`	HuggingFaceFV/finevideo	43K YouTube videos (~3,400 h)	gated
`egocentric-10k`	builddotai/Egocentric-10K	10K egocentric factory video samples (1080p)	gated

Gated datasets require accepting Terms of Service on HuggingFace and setting HF_TOKEN. Any dataset with a Video feature works — pass a raw path: --dataset org/name.

11Implementation

Built with PyTorch, HuggingFace Transformers, and torchcodec.

Visual Backbone

DINOv2

Frozen self-supervised ViT. Extracts CLS token per frame.

Audio Backbone

Whisper

Optional frozen speech encoder for multimodal training.

Vision Decoder

ConvTranspose

Phase 2. Transposed CNN mapping latents to pixels.

Hierarchical World Model

3-Level Predictive Hierarchy

Encoder + ActionHead + Predictor + TopDownDecoder per level. Trained end-to-end.

Project Structure

worlds1k/
  model/
    world_model.py      # Hierarchical predictive model
    world_layer.py      # Single hierarchy level
    encoder_base.py     # Abstract base classes + factories
    vision_encoder.py   # DINOv2 visual encoder
    audio_encoder.py    # Whisper audio encoder + AudioVideoEncoder
    vision_decoder.py   # Vision decoder (phase 2)
    audio_decoder.py    # Audio decoder (phase 2, mel spectrograms)
  train/
    world_model.py      # Phase 1: world model training + CLI
    decoder.py          # Phase 2: frame + audio decoder training + CLI
  inference/
    dream.py            # Dreaming (autoregressive rollout) + CLI
  data.py               # Dataset registry + streaming with disk cache
resources/              # Architecture docs + design notes

12Quick Start

Requires Python 3.11+, FFmpeg, and uv.

# Clone and install
git clone https://github.com/max-gartz/worlds1k
cd worlds1k
uv sync

# List available datasets
uv run python -m worlds1k.train.world_model --list-datasets

# Phase 1: train world model (100K frames)
uv run python -m worlds1k.train.world_model \
  --dataset disney \
  --max-frames 100000 \
  --output-dir checkpoints/

# Phase 2: train decoders
uv run python -m worlds1k.train.decoder \
  --world-model checkpoints/latest.pt \
  --dataset disney \
  --max-frames 50000 \
  --output-dir checkpoints/decoders

# Dream (generates HTML with video)
uv run python -m worlds1k.inference.dream \
  --world-model checkpoints/latest.pt \
  --vision-decoder checkpoints/decoders/vision_decoder.pt \
  --input video.mp4 \
  --dream-steps 20