Read videos like images: index once locally, query the text layer, fetch frames on demand.
Index a video once. Query it cheaply, forever, with any LLM.
videoctx ("video context") turns any video into an open, portable Video
Context index (.vctx, see SPEC.md) using only local, model-free
tools: ffmpeg keyframes, PySceneDetect shot boundaries, faster-whisper
transcripts, and optional one-line visual captions from any cheap VLM. An
agent answers most questions from the index's text layer for fractions of a
cent, and pulls individual frames on demand only when a question needs pixels.
videoctx index lecture.mp4 # one-time, local, free
videoctx show lecture.vctx # the text view the model reads
videoctx ask lecture.vctx "What diagram is shown when she explains backprop?"
Ask questions about videos the way you'd ask about a document:
Who this is for: developers and anyone comfortable running a few terminal commands. You'll need Python, ffmpeg, and an Anthropic API key for question-answering (indexing itself is free and offline). If that's not you yet, star/watch the repo: a no-install version is on the roadmap.
Native video ingestion re-tokenizes the whole video on every question: an hour of footage is ~250-350k tokens, every single time. Proprietary video platforms index into closed representations behind per-minute pricing. videoctx is the third option: an open on-disk format any model can consume, produced by free tools, where the marginal cost of a follow-up question on an hour-long video is about $0.02.
Full protocol and honest cost accounting (including prompt-cache write premiums) in paper/draft.md. Query model: Claude Opus 4.8.
Short videos, held-out test half (n=150):
| Method | Accuracy | $/question |
|---|---|---|
| videoctx | 83.3% | $0.019* |
| Uniform frame sampling | 74.0% | $0.0172 |
*Correction 2026-10-18: videoctx short-video costs originally published as $0.0067/question undercounted prompt-cache write premiums (a metering bug, since fixed); $0.019 is the honestly-metered dev-split figure. Accuracy figures and baseline costs were unaffected. Long/medium figures were measured after the fix and stand as published.
Identical 30-question subset, all methods:
| Method | Accuracy | Latency/question |
|---|---|---|
| videoctx | 76.7% | 2-20s |
| Uniform frame sampling | 60.0% | ~5s |
| Gemini Flash native ingestion | 53.3% | ~40s |
Long videos, 30-60 minutes (n=30):
| Method | Accuracy | $/question | Follow-up $/question |
|---|---|---|---|
| videoctx | 80.0% | $0.096 | $0.024 |
| Uniform frame sampling (60-frame cap) | 70.0% | $0.133 | $0.133 |
| Native ingestion (published Pro rates, est.) | pending | ~$0.50 | ~$0.50 |
One-time index cost: ~$0.10 per hour of video (captions), pennies otherwise.
git clone <repo> && cd videoctx
uv venv && uv pip install -e .
Requires ffmpeg on PATH. videoctx index and the MCP server's core tools
run fully locally with no API key. videoctx ask uses the Anthropic API
(ANTHROPIC_API_KEY or an ant auth login profile).
claude mcp add videoctx -- $(pwd)/.venv/bin/videoctx-mcp
The calling agent's own model reads the text layer and frames directly; the
server makes no model calls and needs no key. Tools: video_overview (cheap
text layer), survey_video (one thumbnail per shot), get_frames(path, t0, t1, detail), transcript_search, index_video. The server's instructions
carry the evidence rule: visual answers must cite fetched frames, never
general knowledge.
uv pip install -e ".[eval]"
python eval/prepare_videomme.py --duration short --videos 100 --out eval/data
python eval/split_dev_test.py eval/data/videomme_short.jsonl
python eval/run.py eval/data/videomme_short_test.jsonl --methods videoctx,naive
eval/methods.py documents each baseline. Cost accounting counts cache
writes at their real premium (1.25x / 2x by TTL); see the paper's Section 7.3
for the accounting traps this avoids.
The .vctx directory is plain JSON + JPEG: manifest.json (provenance,
parameters, source hash), index.json (shots, keyframes, transcript,
captions), frames/. Consumers must ignore unknown keys; enrichment layers
(OCR, embeddings) are reserved in SPEC.md v0.1.
Working and benchmarked: indexing, query loop, MCP server, eval harness. Next: medium-duration subset, Pro-tier native baseline, cross-model portability runs (same indexes via non-Claude models), OCR and per-shot embedding layers, spec v1.0.
Apache-2.0. Copyright 2026 Fred Dunyo.
Source-derived launch command. Check the maintainer’s required arguments and credentials before running:
uvx videoctxMerge this template into ~/Library/Application Support/Claude/claude_desktop_config.json. Keep existing servers. Add any arguments, credentials, and permissions required by the maintainer; this template has not been install-tested.
{
"mcpServers": {
"io-github-dunfred-videoctx": {
"command": "uvx",
"args": [
"videoctx"
]
}
}
}Restart Claude Desktop completely for changes to take effect. Confirm the server appears connected in the client’s tool list, then try a read-only example from its documentation.
Claude Desktop setup referencevideoctxpypivideoctx works with any MCP-compatible client. Copy the config snippet from the Configuration section above and add it to the file shown for your client, then restart the application.
~/Library/Application Support/Claude/claude_desktop_config.jsonRestart Claude Desktop completely for changes to take effect.~/.cursor/mcp.jsonRestart Cursor for changes to take effect..vscode/mcp.jsonReload VS Code window for changes to take effect.~/.codeium/windsurf/mcp_config.jsonRestart Windsurf for changes to take effect..mcp.jsonSave at the project root, then start Claude Code in that project and review the MCP server approval prompt. Keep real credentials out of shared files.