A vector in LLM is a way of representing the meaning of words or sentences as a list of numbers. For example, the word “cat” might be represented as [0.2, -0.5, 0.7, …] in a vector of 300 dimensions.
Representation of a vector:
Each number in the vector corresponds to a learned feature or attribute of the language, such as the part of speech, the sentiment, the topic, etc. The vector can capture the similarity or difference between words or sentences based on their meaning and context.
To understand how language models work, you first need to understand how they represent words. Humans represent English words with a sequence of letters, like C-A-T for “cat.” Language models use a long list of numbers called a “word vector.” For example, here’s one way to represent cat as a vector:[0.0074, 0.0030, -0.0105, 0.0742, 0.0765, -0.0011, 0.0265, 0.0106, 0.0191, 0.0038, -0.0468, -0.0212, 0.0091, 0.0030, -0.0563, -0.0396, -0.0998, -0.0796, …, 0.0002]
<The full vector is 300 numbers long—to see it all, click here and then click “show the raw vector.”>
Why use such a baroque notation? Here’s an analogy. Washington, DC, is located at 38.9 degrees north and 77 degrees west. We can represent this using a vector notation:
- Washington, DC, is at [38.9, 77]
- New York is at [40.7, 74]
- London is at [51.5, 0.1]
- Paris is at [48.9, -2.4]
This is useful for reasoning about spatial relationships. You can tell New York is close to Washington, DC, because 38.9 is close to 40.7 and 77 is close to 74. By the same token, Paris is close to London. But Paris is far from Washington, DC.
<Source of above explanation found here>
Vectors are stored in a Vector Database, examples of Vector Databases: PineCone, Redis, Milvus, Chroma, Faiss.
Graphical overview:
