In Part 1, one paragraph made its way through the encoding pipeline: a tokenizer split it into 106 token IDs, and an embedding model turned those IDs into one 384-dimension vector. This post picks up from there.

The cat sat on the mat and watched the birds fly south for winter. Every autumn, the swallows left the garden in a loose flock and flew toward the coast, where the marshlands held enough insects and seeds to carry them through the cold months. The cat knew the pattern. It had watched the same migration for seven seasons from its mat by the kitchen door, and each year the birds departed a few days earlier. When the last swallow vanished past the old oak, the garden fell quiet and the cat went inside.

Three questions remain before the paragraph can be searched: what those 384 dimensions actually capture, how a long document gets split into embeddable chunks, and which distance metric ranks the results.

Table of Contents

1. What Dimensions Actually Capture — model architecture and vector length
2. Chunking — the two limits and the splitter I use
3. Similarity Search — cosine, L2, inner product

---

1. What Dimensions Actually Capture

Part 1 built the chain text → token → vector. The tokenizer split text into token IDs, and the embedding model turned each ID into a vector. A vector, in this context, is simply an ordered list of numbers. The row for “the” is one such list, 384 numbers long.

Each number in that list is one dimension, so “384 dimensions” means the vector holds 384 numbers. You can also think of each dimension as one axis of a coordinate system and the vector as a point in that space: a 384-dimension vector is a point in 384-dimensional space. The dimension count is the size of the vector, how many values it takes to describe one data point, and word embedding models produce exactly this, with each token becoming one vector with one value per dimension, as DataKnobs’s vector database primer explains.

The specific lookup worked like this in Part 1: the tokenizer mapped “the cat” to the IDs [1996, 4937], and the embedding model looked up each ID in its 30,522 × 384 matrix. Each row is a 384-number vector, so “the” and “cat” each become a point in the same 384-dimension space.

The Part 1 output printed only the first 8 values to keep the terminal readable. The full row for “the” is 384 numbers:

// row(1996) is the vector for "the" — first 8 of 384 values shown
[
  "-0.0089",
  "-0.0102",
  "-0.0419",
  "0.0178",
  "-0.0057",
  "0.0192",
  "-0.0153",
  "-0.0306",
  // ... 376 more
];

row(1996).length returns 384. That is the 384 this post keeps returning to: every token, every query, and every chunk becomes a vector of this exact size.

Here is that relationship made visual. Each box is one dimension; the whole strip is the vector:

IBM’s primer on vector embeddings defines “dimension” for a vector the same way I just used it:

When describing a vector, it refers to how many components—individual numbers—that vector contains.

For row(1996), that count is 384.

The numbers themselves carry no labels. You cannot open the model and point at dimension #47 and say “that is the humor dimension.” The 384 axes are learned together from millions of sentences, so the meaning of a text lives in the whole vector, not in any single slot.

Part 1 already showed the result. “Where did the birds go?” and the running paragraph meet at cosine similarity 0.60, nearly the same direction. “The stock market crashed and investors fled to bonds” meets it at 0.14, a quarter of the way there. Both are grammatical English sentences, and the space separates them cleanly: each text earns a position, and texts about the same thing land at nearby positions.

So 384 is not an implementation detail. It is the size of the space the model reasons in, and the architect fixes it before training starts:

Who Decides the Vector Length

Where does 384 come from? Not from the text. “the” does not arrive with 384 attributes attached; the number is a design decision the architect makes before a single sentence is seen. It is the product of two knobs the architect turns, the number of attention heads and the width of each head:

num_attention_heads  ×  head_dimension  =  hidden_size (vector length)

The hidden size IS the vector length. Every token gets a vector of this length.

Most transformer models follow the “Attention Is All You Need” paper (2017) and use a head dimension of 64, but not all of them. Once an architect picks the number of heads and the head dimension, the vector length locks in.

For example, here is the actual config for all-MiniLM-L6-v2 on HuggingFace:

https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/blob/main/config.json

{
  "hidden_size": 384,
  "num_attention_heads": 12,
  "num_hidden_layers": 6,
  "intermediate_size": 1536,
  "max_position_embeddings": 512
}

head_dimension is not stored in the config; it is derived. The config gives hidden_size: 384 and num_attention_heads: 12, and dividing them yields the missing number: 384 ÷ 12 = 32. Plug it back into the formula and the loop closes: 12 attention heads × 32 head dimension = 384 hidden size. Every token gets a 384-dim vector.

384 is not a size sentence-transformers picked at random. It comes from Microsoft’s MiniLM, a smaller, distilled cousin of BERT built to be fast. The hidden size is baked into the model’s own name — MiniLM-L12-H384 — where the H384 stands for “384-dimensional hidden state.” all-MiniLM-L6-v2 is this same family, slimmed down to six layers and fine-tuned on a billion sentence pairs. So the 384 you have been staring at is a trade-off: big enough to hold meaning, small enough to run fast. Here is Microsoft’s model card for the source of that number.

MiniLM is the same BERT architecture, compressed. Here is the full-size sibling it was distilled from, BERT-base, for comparison:

https://huggingface.co/google-bert/bert-base-uncased/blob/main/config.json

{
  "hidden_size": 768,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "intermediate_size": 3072
}

12 heads × 64 = 768.

And BERT-large:

{
  "hidden_size": 1024,
  "num_attention_heads": 16,
  "num_hidden_layers": 24,
  "intermediate_size": 4096
}

16 heads × 64 = 1024.

The architect picks these values based on tradeoffs. More heads = more parameters = more compute = higher quality. Fewer heads = smaller model = faster inference = runs on cheaper hardware.

Model	Heads	Head dim	Hidden	Parameters	Reference
all-MiniLM-L6-v2	12	32	384	22.7M	HuggingFace
BERT-base	12	64	768	110M	Devlin et al.
BERT-large	16	64	1024	340M	Devlin et al.
OpenAI ada-002	24	64	1536	unknown	OpenAI docs
text-embedding-3	48	64	3072	unknown	OpenAI docs

---

2. Chunking

Chunking exists because of two limits, one on the model and one on the content.

The model limit is the token window. all-MiniLM-L6-v2 caps at 512 tokens. Feed it a 2,000-word document and it truncates the excess before embedding, dropping the tail from the vector.

The content limit is searchability. One vector represents everything you embed. A whole chapter dilutes every specific fact; a single sentence matches sharply, but it rarely answers a question on its own.

Pinecone’s chunking guide offers a usable test: if a chunk makes sense to a human on its own, it makes sense to the embedding model. Apply that test to every boundary you pick.

Chunking splits long documents into pieces that fit the model and carry one coherent idea.

My Strategy

Pinecone’s guide surveys fixed-size, recursive, structure-based, semantic, and contextual splitting, each with its tradeoffs. I use fixed-size chunking with overlap: the splitter divides by token count, slides a window forward, and steps back a few tokens so a fact cut at a boundary survives in the neighbor chunk. Pinecone recommends starting here.

Chonkie’s TokenChunker is the battle-tested implementation, and it is exactly what the live demos use. It measures windows in the embedding model’s own tokens: I hand it the AutoTokenizer from @huggingface/transformers so a 45-token chunk is 45 of all-MiniLM-L6-v2’s WordPiece tokens, not some other tokenizer’s approximation:

import { AutoTokenizer } from "@huggingface/transformers";
import { TokenChunker } from "@chonkiejs/core";

const tokenizer = await AutoTokenizer.from_pretrained("Xenova/all-MiniLM-L6-v2");
const wrap = {
  encode: (t) => {
    const out = tokenizer.encode(t, { add_special_tokens: false });
    return Array.isArray(out) ? out : out.input_ids;
  },
  decode: (t) => tokenizer.decode(t, { skip_special_tokens: true }),
};

const chunker = await TokenChunker.create({
  tokenizer: wrap,
  chunkSize: 45, // tokens per chunk
  chunkOverlap: 10, // tokens shared with the previous chunk
});

const chunks = await chunker.chunk(longDocument);

Why not LangChain’s TokenTextSplitter? Its esm.sh build pulls in langsmith, whose transformed output re-exports itself in a circle, so the __version__ export the SDK needs is never provided and the whole module graph fails to load in the browser. Since these demos run entirely in the browser, that is a hard stop — I reached for a splitter with a clean browser graph instead (esm.sh#1391).

The Example Paragraph, Chunked

The example paragraph runs 94 words. The model’s tokenizer counts them as 106 tokens, which sits comfortably under the 512-token limit.

This paragraph does not need chunking. At 106 tokens it fits the model with room to spare, and a coherent paragraph is exactly what you would embed whole. Chunking matters when a document outgrows the window or mixes topics a single vector would blur. I split it anyway: the three chunks below are the same chunks Part 3 searches and ranks.

The same chunker with chunkSize = 45 and chunkOverlap = 10:

const chunks = await chunker.chunk(paragraph);

chunks.forEach((chunk, i) => {
  console.log(
    `chunk ${i} (${chunk.text.trim().split(/\s+/).length} words, ${chunk.tokenCount} tokens): "${chunk.text.trim()}"`,
  );
});

// chunk 0 (41 words, 45 tokens):  "the cat sat on the mat and watched the birds fly south for winter.
//                                  every autumn, the swallows left the garden in a loose flock and
//                                  flew toward the coast, where the marshlands held enough insects
//                                  and seeds to carry them"
// chunk 1 (41 words, 45 tokens):  "marshlands held enough insects and seeds to carry them through
//                                  the cold months. the cat knew the pattern. it had watched the
//                                  same migration for seven seasons from its mat by the kitchen door,
//                                  and each year the birds departed a"
// chunk 2 (30 words, 34 tokens):  "kitchen door, and each year the birds departed a few days
//                                  earlier. when the last swallow vanished past the old oak, the
//                                  garden fell quiet and the cat went inside."

The run shows three things:

1. Chunks overlap. “marshlands held enough insects and seeds to carry them” closes chunk 0 and opens chunk 1, and “kitchen door, and each year the birds departed a” spans chunks 1 and 2. The overlap is intentional: a fact split at a boundary survives in a neighbor.

2. Windows cut mid-sentence. Token counts ignore sentence structure. Chunk 0 ends “carry them” mid-clause, the sentence continues “through the cold months” in chunk 1, and chunk 1 ends mid-phrase “departed a” (finished “a few days earlier” in chunk 2). Chunks 1 and 2 begin mid-phrase with “marshlands held” and “kitchen door,”.

3. No stub this time. Three chunks tile the paragraph’s 104 chunked tokens exactly — 45 + 45 + 34, with the 20 shared by overlaps — nothing left over. Real documents rarely divide this evenly. A leftover shorter than the window becomes a stub, and most chunkers drop stubs under a minimum length, losing the tail from the index.

The same split running live: the paragraph cut into fixed-size chunks, each embedded with Xenova/all-MiniLM-L6-v2 into a 384-float vector and stored in an in-memory SQLite database:

Part 3 searches this exact table: same schema, same splitter, same model. Three chunks, each embedded once. The mid-sentence cuts are visible in the table: chunk 1 begins “marshlands held enough…” and chunk 2 “kitchen door, and each year the birds departed…”, because the window slides across tokens, not sentences.

Chunk Size Tradeoffs

Size	Use Case	Risk
128-256 tokens	Precise answer extraction	Too narrow for context-dependent answers
512-1024 tokens	General Q&A, summarization	May dilute specific facts
Whole document	Very short texts only	Lost-in-the-middle problem for long docs

Test multiple sizes against your queries. Revisit as your data changes. You are also not stuck with the boundaries at query time: chunk expansion retrieves the neighbors of each matched chunk, so the LLM in Part 3 sees surrounding context alongside the hit.

---

3. Similarity Search

The query is a vector too. The same model embeds “where did the birds go?” into 384 dimensions, so searching becomes a geometry problem: rank chunks by distance from the query vector.

Cosine distance is the default for text embeddings because it compares direction and ignores length. Pinecone’s guide to similarity metrics covers cosine, L2, and inner product. The walkthrough below calls a ready-made function, so the math stays in the library.

Walkthrough: Querying the Paragraph

The paragraph is chunked, every chunk is embedded once, and the query is embedded with the same model. ml-distance ranks the chunks by cosine distance:

import { AutoTokenizer } from "@huggingface/transformers";
import { TokenChunker } from "@chonkiejs/core";
import { similarity } from "ml-distance";

// The example paragraph, chunked with the model's own tokenizer (45 tokens, 10 overlap)
const tokenizer = await AutoTokenizer.from_pretrained("Xenova/all-MiniLM-L6-v2");
const chunker = await TokenChunker.create({
  tokenizer: {
    encode: (t) => {
      const out = tokenizer.encode(t, { add_special_tokens: false });
      return Array.isArray(out) ? out : out.input_ids;
    },
    decode: (t) => tokenizer.decode(t, { skip_special_tokens: true }),
  },
  chunkSize: 45,
  chunkOverlap: 10,
});
const chunks = await chunker.chunk(paragraph);

// Embed every chunk once, at index time
const chunkVecs = await Promise.all(chunks.map((c) => embed(c.text)));

// A query embedded with the same model
const queryVec = await embed("where did the birds go?");

// Rank chunks by cosine distance
const ranking = chunkVecs
  .map((v, i) => ({ i, d: 1 - similarity.cosine(queryVec, v) }))
  .sort((a, b) => a.d - b.d);
for (const { i, d } of ranking) {
  console.log(i, d.toFixed(3), chunks[i].text);
}

Results with all-MiniLM-L6-v2:

0 0.348  "the cat sat on the mat and watched the birds fly south for winter.
          every autumn, the swallows left the garden in a loose flock and flew
          toward the coast, where the marshlands held enough insects and seeds
          to carry them"
2 0.392  "kitchen door, and each year the birds departed a few days earlier.
          when the last swallow vanished past the old oak, the garden fell quiet
          and the cat went inside."
1 0.531  "marshlands held enough insects and seeds to carry them through the
          cold months. the cat knew the pattern. it had watched the same
          migration for seven seasons from its mat by the kitchen door, and
          each year the birds departed a"

The query embeds closest to the chunk that answers it. “Where did the birds go?” lands on chunk 0, which contains “birds fly south for winter” and “flew toward the coast”, the whole answer in one chunk. Chunk 2 ranks second (0.392) because it holds the departure (“the birds departed”, “the last swallow vanished”) and the query reads as “they are gone”. Chunk 1 (0.531) repeats the marshlands detail from chunk 0 but carries less of the migration narrative.

A query about something unrelated to the paragraph, like “The cat likes to eat fish.”, leaves every chunk far away (best distance 0.704). Distance separates relevance from noise, but it does not guarantee relevance. The vector index returns the closest chunks; the RAG system in Part 3 decides whether they are good enough to feed the LLM.

Run the same search live. The demo below is the walkthrough’s exact pipeline: the same chunks indexed in an in-memory SQLite table, ranked by cosine distance, with the SQL that runs for every query. Try one of the suggested questions or type your own:

---

What We Covered So Far

Till now the encoding side of the pipeline is complete. Text becomes tokens: Part 1 walked the 94 words of the example paragraph through a tokenizer into 106 token IDs. Tokens become vectors: the embedding model projected those IDs into 384 dimensions, and this post explained what a dimension captures and why the model needs exactly that many. And chunking makes it practical: long documents are split into overlapping, token-sized windows that fit the model and carry one coherent idea, each embedded once so a query can find the right part instead of the whole document.

---

Further Reading

Every demo in this post runs in the browser: tokenizer, embedding model, and SQLite, all on the reader’s CPU. That works because models like all-MiniLM-L6-v2 fit in a few megabytes and WASM runs them anywhere.

ternlight pushes the same idea further. It is a sentence-embedding model trained with ternary weights and shipped as one Rust-to-WASM bundle of about 7 MB. It embeds in around 5 ms per sentence on a CPU, with no server, no API, no GPU. The site runs a live cosine search in the tab and the blog posts explain how the model was built. If the demos here showed how pleasant in-browser embeddings are, ternlight shows how small and fast they can get. Worth a read.

More from around the web:

• How do embedding models convert text into vectors? (Milvus)
• Embedding Models Rundown (Pinecone)
• Chunking Strategies (Pinecone)
• Understanding Hierarchical Navigable Small Worlds (HNSW) for Vector Search (Milvus)
• Running embeddings in the browser (Ternlight)