From the course: LLM Foundations: Vector Databases for Caching and Retrieval Augmented Generation (RAG)

Query and search in Milvus

From the course: LLM Foundations: Vector Databases for Caching and Retrieval Augmented Generation (RAG)

Query and search in Milvus

“

Query and such are key capabilities that any database needs to support. Let's explore what capabilities are available in Milvus in this regard. Let's start with queries. In queries, we extract entities using scalar field-based filtering. This is very similar to the retrieval process in RDBMS, where SQL statements are executed to extract data that match specific filter conditions. When running a query in Milvus, we can specify the output fields needed and the limits on the number of rows. We can also specify offsets, which is the number of rows to skip before returning the remaining data. This helps with pagination type queries. We can restrict the queries to specific partitions. This can be done by specifying the partition name in the query or using the partition key in the filter. For aggregations, only a count(*) feature is available. Other capabilities like sum or average are not available. Do note that the scalar querying capabilities are severely limited in Milvus when compared to RDBMSs. When executing queries, we can also specify filters on which rows to fetch from the database. Milvus supports several popular filtering operators on field values. Comparison operators can be used to compare numeric data. Logical operators can combine multiple filters to form a composite filter. Like can be used as a match operator for string filtering. In addition, array and JSON operators are available for filtering on array and JSON datatypes. For more details on the filters, please refer to the Milvus documentation in the link provided. Next comes search on vector fields. Milvus provides the ability to search on any vector field using distance measures. An input string can be compared to strings in the database, and related strings can be extracted with semantic search. For this, the input string or the search query should first be converted to a vector using the same embedding model as the one used when ingesting the vector field. The metric used for comparison should be the same metric that was used when creating the index for the vector field. Do note that the index is a prerequisite before search can be performed on the vector field. We can also specify the limit on the number of rows returned and an offset from which -- sorry. Repeating. We can also specify the limit on the number of rows returned and an offset from which to return rows. The query returns results in ascending order of distance to the search string. So if we only need the top five matches, we specify a limit of five. To further restrict the query and filtered based on the distance between the input string and the vectors in the database, we can also specify the maximum allowed distance in the radius parameter. The smaller the distance, the higher the similarity. Do note that the range of values for distance will vary based on the metric type used, so radius needs to be adjusted for that. When returning results, the computed distance is also returned in addition to the query results. We will see examples of queries in the next chapter. Let's first get set up with Milvus and exercise files in the next video. You can read more about Milvus capabilities and commands by using the online documentation available in the website milvus.io/docs.

Contents