How TopVault identifies a card from a photo with image embeddings
TopVault used to identify a scanned card by reading the text off it. It now compares the picture itself against a precomputed catalog of reference images, and only falls back to reading text when the picture is not a confident match. This is a walk through...
TopVault used to identify a scanned card by reading the text off it. It now compares the picture itself against a precomputed catalog of reference images, and only falls back to reading text when the picture is not a confident match. This is a walk through the whole path, from the model on your phone to the ranking loop in the Go service, with the real numbers and thresholds involved.
If you have not read Use AI models on-device with Ionic, React, and ONNX Runtime, that post covers the character recognition half of this system and how models get onto the device in the first place. This one is the sequel it promised.
Why reading the text was the wrong place to start
Optical character recognition (called OCR) works by finding text regions in an image and recognizing the characters in each one. For a card that is flat, evenly lit, and square to the camera it works well. Cards are frequently none of those things. They are glossy, they are held at an angle, they sit under a lamp that blows out half the surface, and the text a collector cares about is often 6pt type at the bottom edge.
Here is a real example from the development of this feature. A card photographed at a modest angle, run through the OCR pipeline, produced this as its recognized text:
Cores nay mlus. fhnclThe card was Colress's Tenacity. Every character that mattered was destroyed by perspective and the holo finish. There is no lookup table in the world that recovers the right card from that string, because the failure happened before the lookup.
The important observation is that the rest of the photo was completely fine. The artwork was legible, the border and layout were unmistakable, the color palette was right. OCR throws all of that away and keeps only the part of the image that degrades the fastest.
What an embedding is
An embedding model takes an image and returns a fixed-length list of numbers, a vector. It is trained so that images that look alike produce vectors that point in similar directions, and images that look different produce vectors that point in different directions. It never reads anything. It has no notion of "Pokémon card" or "057/064". It just turns a picture into a point in a high-dimensional space.
TopVault uses a quantized DINOv3 export, labeled dinov3-q4. It is 14 MB, takes a 224×224 image, and returns 384 numbers. That is the entire representation of your card that the rest of the system works with.
Once both your photo and every catalog image are points in the same space, "which card is this?" becomes "which catalog point is closest to my point?" The measure is cosine similarity, the cosine of the angle between two vectors, which is 1.0 when they point in exactly the same direction. Because every vector is L2-normalized on the way in (scaled to length 1), cosine similarity reduces to a plain dot product, which is worth knowing because it makes the hot loop in the matcher very cheap.
The invariant that makes or breaks the whole thing
Cosine similarity between two vectors is only meaningful if both came from the same model with the same preprocessing. Mismatch them and you do not get an error. You get numbers. They are just meaningless numbers, and every match silently gets worse in ways no test catches.
This is the single most dangerous property of the system, so it is designed around defensively. The preprocessing contract does not live in the app source, where it could drift from whatever generated the catalog vectors. It travels with the model, in the hosted model manifest:
json
"card-embed": {
"id": "card-embed",
"version": "7d32eb5f1be3",
"url": "https://vault.top/app/assets/embed/dinov3-q4.onnx",
"expectedSha256": "7d32eb5f1be3...",
"expectedBytes": 14820384,
"params": {
"kind": "embedding",
"modelLabel": "dinov3-q4",
"inputName": "pixel_values",
"outputName": "last_hidden_state",
"inputSize": 224,
"dimensions": 384,
"pooling": "cls",
"normalize": "l2"
}
}The app downloads the model, verifies the checksum, and reads how to use it from the same document. Swapping the embedding model is a manifest change, not an app release, and the client cannot drift into a different vector space than the catalog.
The preprocessing itself is deliberately boring, and every line of it matters:
ts
// Stretch to size×size. No letterbox, no aspect preservation.
// This matches the corpus precompute exactly.
ctx.drawImage(img, 0, 0, size, size);
const { data } = ctx.getImageData(0, 0, size, size); // RGBA uint8
const plane = size * size;
const tensorData = new Float32Array(3 * plane);
for (let i = 0; i < plane; i += 1) {
tensorData[i] = data[i * 4] / 255; // R plane
tensorData[i + plane] = data[i * 4 + 1] / 255; // G plane
tensorData[i + 2 * plane] = data[i * 4 + 2] / 255; // B plane
}A plain stretch to 224×224, RGB divided by 255, arranged in CHW planes. It is tempting to letterbox instead, to preserve the card's aspect ratio. That would be better preprocessing in the abstract and wrong here, because the offline script that embedded the catalog uses an ImageMagick resize with ignoreAspectRatio. The two sides have to agree more than either side has to be optimal.
DINOv3 emits a token sequence with no pooler head, so the 384-vector is the CLS token, element 0 of the sequence, which is then L2-normalized. Both of those steps are named in the manifest params rather than hardcoded.
Preparing the catalog
The other half of the vector space is built offline, ahead of any user scanning anything.
TopVault's catalog is collectdb, which is open source, with the binary images in a sibling asset tree. A script walks every front.webp reference image, embeds it with the same ONNX model on the same preprocessing path, and writes a sibling file next to it:
front.webp
front.dinov3-q4.emb <- 384 floats, L2-normalized, base64The model label is in the filename on purpose. A .emb file is permanently tied to one specific model, and naming it after the label makes it impossible to confuse two models' outputs, or to quietly reuse the wrong ones when evaluating a candidate model. There are currently 41,237 of these for English Pokémon cards and 6,395 for Japanese.
The API Service ingests those sidecars into the catalog entry's details, keyed by label, and seeds a cache with one vector per entry. After a catalog refresh it pushes the whole set to Match Service as a named corpus:
pokemon-card-english-dinov3-q4Collection type, region, and model label, all three in the name. That naming is what makes model migrations survivable, which is the last section of this post.
On the device: detect, crop, embed
When you scan a card, four models are potentially involved and only two of them usually run.
First an oriented bounding box detector finds the card in the frame and returns four corners, not an axis-aligned rectangle, a card photographed at an angle is a quadrilateral, and treating it as a rectangle would either clip corners or include a lot of table. The app then shows you the suggested crop with draggable corners so you can correct it:
Then, and only then, the embedding model runs on the cropped region. ONNX Runtime Web executes it in WASM. Both the session creation and the inference are serialized behind mutexes because the runtime is configured single-threaded, with explicit wasmPaths and SIMD disabled, a combination required to get ORT loading inside the iOS WKWebView that Capacitor uses.
The OCR models do not run at this point. That is the part worth emphasizing: on the common path, the two PaddleOCR models are never even initialized, which saves both their download and roughly the time it takes to run text detection and recognition over the card.
What actually leaves your device
The match request carries the embedding, not the picture:
json
{
"collectionType": "pokemon-card",
"regions": ["english"],
"match": {
"matches": [{
"embedding": {
"modelLabel": "dinov3-q4",
"vector": [-0.0072888128, -0.1129669885, -0.0268763921, ...]
},
"hashes": [],
"details": { "note": "canvas-resize" }
}]
}
}That is 384 floating point numbers and a model label. The photo itself never goes to the server for identification. This is consistent with how photo uploads are handled, those are end-to-end encrypted and TopVault cannot read them, and it means card identification adds no new place where your photos could be exposed. An embedding is not reversible into the original image, and in any case the server has no reason to keep it: it is used to rank a corpus and then discarded.
Match Service: cosine over a packed corpus
Match Service is a small Go service holding the corpus in memory. Corpus vectors are stored packed as FP16 rather than float32, which roughly halves the memory. The precision loss is irrelevant for ranking, the gaps between a right answer and a wrong answer are far larger than FP16 rounding.
The ranking loop is exactly as simple as it sounds:
go
// cosineFromPacked computes the dot product between a float32 query and an
// FP16-packed corpus vector. Both sides are L2-normalized on the way in, so
// the dot product is the cosine similarity.
func rankEmbeddings(query []float32, items []storedEmbedding) []EmbeddingMatchResult {
results := make([]EmbeddingMatchResult, 0, len(items))
for _, item := range items {
results = append(results, EmbeddingMatchResult{
ItemID: item.itemID,
Score: cosineFromPacked(query, item.packed),
})
}
sort.Slice(results, func(i, j int) bool {
return results[i].Score > results[j].Score
})
return results
}A linear scan over every vector in the corpus. No approximate nearest neighbor index, no vector database. At tens of thousands of 384-dimension vectors this is fast enough that adding an index would be optimizing the wrong thing, and a brute force scan has the pleasant property of being exactly correct.
If the query's dimension count does not match the corpus, the service returns no results rather than garbage. It also returns at most 5 results, and it deliberately does not decide whether any of them are good. It returns ranked scores. The policy question of what counts as a confident match lives in API Service, in one place, next to the fallback logic that depends on it.
The confidence bar, and the near-tie window
Two constants decide what happens with those scores:
ts
// The lowest cosine similarity at which an embedding match is trusted.
const EMBEDDING_MATCH_MIN_CONFIDENCE = 0.82;
// Keeps near-tie matches within this cosine margin of the best.
const EMBEDDING_MATCH_FALLOFF = 0.03;If the best score across all searched regions is below 0.82, the embedding result is discarded entirely, not returned as a weak guess. Returning a plausible wrong card is worse than returning nothing, because a collector who trusts it puts the wrong entry in their collection.
If the best score clears the bar, everything within 0.03 cosine of it comes back too. This second constant is not a technicality; it is the most visible behavior in the whole feature. Consider what happens when you photograph a Tinkaton from Paldea Evolved:
Six results, all the same card. The normal print, the non-holo, the Cosmos Holo, the Reverse Holo, a Paldea Evolved stamped copy, and a GameStop stamped copy. This is correct, and it is the honest limit of a purely visual match: those printings are nearly the same image. A holo pattern or a small foil stamp moves the vector a little, not a lot. The model has done its job by narrowing the whole catalog down to the six entries that genuinely are this card, and the collector, who is holding the thing and can see whether it is reverse holo, picks the printing.
Widening the falloff would return more variants and more noise. Narrowing it would arbitrarily amputate the cluster and hide the printing you actually own. 0.03 is a judgement about where that tradeoff sits, and it is one number in one file.
When the picture is not enough
OCR did not go away. It became the fallback, and the handoff is the interesting design decision.
The server never tells the client which signal to send next. It reports a modality-agnostic outcome:
matched, results came back, with asourcenaming which signal produced them.no-match, a search ran against the corpus and nothing cleared the bar.unavailable, the search could not be performed: the corpus is not loaded, the dimensions disagree, or the backend errored.
Embedding-path errors are deliberately swallowed into unavailable rather than failing the request, so a visual match that cannot run degrades into a text match instead of into an error message.
The client owns the fallback decision:
ts
// 1) Embedding-only, when at least one request carries an embedding.
if (requests.some(request => request.embedding)) {
const embeddingResult = await api.matchMarketItems(/* ... */);
if (embeddingResult.items.length > 0) {
return embeddingResult;
}
}
// 2) OCR-text fallback. Resolve the text lazily, this is the first
// point at which the OCR models are needed at all.
const fallbackText = requests[0]?.text ?? (await resolveText());
return api.matchMarketItems(ctx, collectionType, regions, { matches: textMatches });Keeping the outcome status modality-agnostic, rather than having the server return something like retryWithText, means the response contract does not have to change as signals are added or removed. When a third signal is added, or when OCR is eventually retired, no field named after a retired modality is left behind in a public API.
The cost is one extra network round trip on the fallback path. That is acceptable precisely because the fallback is the uncommon case, and the alternative, always computing OCR so it can be batched into the first request, would impose the OCR cost on every scan to save a round trip on a minority of them.
Migrating models without a flag day
Everything above depends on the client's vectors and the corpus vectors coming from the same model. That sounds like it forces a synchronized deployment: publish the new model, ship the new app build, rebuild the corpus, all at once, or matching breaks for everyone in between.
It does not, because of that corpus naming scheme. The corpus is per label:
pokemon-card-english-dinov3-q4
pokemon-card-english-<next-model>The API Service ingests an explicit allow-list of labels and serves each as its own corpus. The client sends its model label with every request, and gets matched against the corpus for that label. So during a migration both corpora exist at once. An app build that has not updated keeps matching perfectly against the old vectors while updated builds use the new ones, and the old label is dropped from the allow-list only once nothing is asking for it. The vectors from two different models never touch each other, which is exactly the invariant the whole system is built to protect.
Where this goes next
The embedding path is live for Pokémon cards, English and Japanese. It will expand to the other collectible types as their reference images are embedded; until then those types match on text exactly as before.
The obvious open problem is the one the Tinkaton screenshot shows. Distinguishing printings that share artwork is not something a whole-card embedding can do well, because the difference is a foil pattern or a stamp measuring a few millimeters. That is a different kind of signal, localized, high-frequency, and not what a 224×224 stretch of the whole card preserves. Whether that becomes a second model, a crop of a specific region, or something else entirely is undecided.
If you want to look at the data side of this, collectdb is public.