Vector Store

Overview

The VectorStore is a core component of DeepIntShield’s framework package that provides a unified interface for vector database operations. It enables plugins to store embeddings, perform similarity searches, and build AI-powered features like semantic caching, content recommendations, and knowledge retrieval.

Key Capabilities:

• Vector Similarity Search: Find semantically similar content using embeddings
• Namespace Management: Organize data into separate collections with custom schemas
• Flexible Filtering: Query data with complex filters and pagination
• Multiple Backends: Support for Weaviate, Redis/Valkey-compatible, Qdrant, and Pinecone vector stores
• High Performance: Optimized for production workloads
• Scalable Storage: Handle millions of vectors with efficient indexing

Supported Vector Stores

DeepIntShield currently supports four vector store implementations:

• Weaviate: Production-ready vector database with gRPC support and advanced querying
• Redis: High-performance in-memory vector store using RediSearch/Valkey Search
• Qdrant: Rust-based vector search engine with advanced filtering
• Pinecone: Managed vector database service with serverless and pod-based options

VectorStore Interface Usage

Creating Namespaces

Create collections (namespaces) with custom schemas:

// Define properties for your data
properties := map[string]vectorstore.VectorStoreProperties{
    "content": {
        DataType:    vectorstore.VectorStorePropertyTypeString,
        Description: "The main content text",
    },
    "category": {
        DataType:    vectorstore.VectorStorePropertyTypeString,
        Description: "Content category",
    },
    "tags": {
        DataType:    vectorstore.VectorStorePropertyTypeStringArray,
        Description: "Content tags",
    },
}

// Create namespace
err := store.CreateNamespace(ctx, "my_content", 1536, properties)
if err != nil {
    log.Fatal("Failed to create namespace:", err)
}

Storing Data with Embeddings

Add data with vector embeddings for similarity search:

// Your embedding data (typically from an embedding model)
embedding := []float32{0.1, 0.2, 0.3 } // example 3-dimensional vector

// Metadata associated with this vector
metadata := map[string]interface{}{
    "content":  "This is my content text",
    "category": "documentation",
    "tags":     []string{"guide", "tutorial"},
}

// Store in vector database
err := store.Add(ctx, "my_content", "unique-id-123", embedding, metadata)
if err != nil {
    log.Fatal("Failed to add data:", err)
}

Similarity Search

Find similar content using vector similarity:

// Query embedding (from user query)
queryEmbedding := []float32{0.15, 0.25, 0.35, ...}

// Optional filters
filters := []vectorstore.Query{
    {
        Field:    "category",
        Operator: vectorstore.QueryOperatorEqual,
        Value:    "documentation",
    },
}

// Perform similarity search
results, err := store.GetNearest(
    ctx,
    "my_content",        // namespace
    queryEmbedding,      // query vector
    filters,             // optional filters
    []string{"content", "category"}, // fields to return
    0.7,                 // similarity threshold (0-1)
    10,                  // limit
)

for _, result := range results {
    fmt.Printf("Score: %.3f, Content: %s\n", *result.Score, result.Properties["content"])
}

Data Retrieval and Management

Query and manage stored data:

// Get specific item by ID
item, err := store.GetChunk(ctx, "my_content", "unique-id-123")
if err != nil {
    log.Fatal("Failed to get item:", err)
}

// Get all items with filtering and pagination
allResults, cursor, err := store.GetAll(
    ctx,
    "my_content",
    []vectorstore.Query{
        {Field: "category", Operator: vectorstore.QueryOperatorEqual, Value: "documentation"},
    },
    []string{"content", "tags"}, // select fields
    nil,  // cursor for pagination
    50,   // limit
)

// Delete items
err = store.Delete(ctx, "my_content", "unique-id-123")

Weaviate

Weaviate is a production-ready vector database solution that provides advanced querying capabilities, gRPC support for high performance, and flexible schema management for production deployments.

Key Features

• gRPC Support: Enhanced performance with gRPC connections
• Advanced Filtering: Complex query operations with multiple conditions
• Schema Management: Flexible schema definition for different data types
• Cloud & Self-Hosted: Support for both Weaviate Cloud and self-hosted deployments
• Scalable Storage: Handle millions of vectors with efficient indexing

Setup & Installation

Weaviate Cloud:

• Sign up at cloud.weaviate.io
• Create a new cluster
• Get your API key and cluster URL

Local Weaviate:

# Using Docker
docker run -d \
  --name weaviate \
  -p 8080:8080 \
  -e QUERY_DEFAULTS_LIMIT=25 \
  -e AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED='true' \
  -e PERSISTENCE_DATA_PATH='/var/lib/weaviate' \
  semitechnologies/weaviate:latest

Configuration Options

Go SDK

// Configure Weaviate vector store
vectorConfig := &vectorstore.Config{
    Enabled: true,
    Type:    vectorstore.VectorStoreTypeWeaviate,
    Config: vectorstore.WeaviateConfig{
        Scheme: "http",           // "http" for local, "https" for cloud
        Host:   "localhost:8080", // Your Weaviate host
        APIKey: "your-weaviate-api-key", // Required for Weaviate Cloud; optional for local/self-hosted

        // Enable gRPC for improved performance (optional)
        GrpcConfig: &vectorstore.WeaviateGrpcConfig{
            Host:    "localhost:50051", // gRPC port
            Secured: false,             // true for TLS
        },
    },
}

// Create vector store
store, err := vectorstore.NewVectorStore(context.Background(), vectorConfig, logger)
if err != nil {
    log.Fatal("Failed to create vector store:", err)
}

config.json

Local Setup:

{
  "vector_store": {
    "enabled": true,
    "type": "weaviate",
    "config": {
      "scheme": "http",
      "host": "localhost:8080"
    }
  }
}

Cloud Setup with gRPC:

{
  "vector_store": {
    "enabled": true,
    "type": "weaviate",
    "config": {
      "scheme": "https",
      "host": "your-weaviate-host",
      "api_key": "your-weaviate-api-key",
      "grpc_config": {
        "host": "your-weaviate-grpc-host",
        "secured": true
      }
    }
  }
}

Note

gRPC host should include the port. If no port is specified, port 80 is used for insecured connections and port 443 for secured connections.

Advanced Features

gRPC Performance Optimization: Enable gRPC for better performance in production:

vectorConfig := &vectorstore.Config{
    Type: vectorstore.VectorStoreTypeWeaviate,
    Config: vectorstore.WeaviateConfig{
        Scheme: "https",
        Host:   "your-weaviate-host",
        APIKey: "your-api-key",

        // Enable gRPC for better performance
        GrpcConfig: &vectorstore.WeaviateGrpcConfig{
            Host:    "your-weaviate-grpc-host:443",
            Secured: true,
        },
    },
}

Production Considerations

Note

Performance: For production environments, consider using gRPC configuration for better performance and enable appropriate authentication mechanisms for your Weaviate deployment.

Caution

Authentication: Always use API keys for Weaviate Cloud deployments and configure proper authentication for self-hosted instances in production.

---

Redis

Redis provides high-performance in-memory vector storage using RediSearch-compatible APIs, ideal for applications requiring sub-millisecond response times and fast semantic search capabilities. Valkey deployments that expose compatible FT.* commands are supported through the same configuration.

Key Features

• High Performance: Sub-millisecond cache retrieval with Redis’s in-memory storage
• Cost Effective: Open-source solution with no licensing costs
• HNSW Algorithm: Fast vector similarity search with excellent recall rates
• Connection Pooling: Advanced connection management for high-throughput applications
• TTL Support: Automatic expiration of cached entries
• Streaming Support: Full streaming response caching with proper chunk ordering
• Flexible Filtering: Advanced metadata filtering with exact string matching

Setup & Installation

Redis Cloud:

• Sign up at cloud.redis.io
• Create a new database with RediSearch module enabled
• Get your connection details

Local Redis with RediSearch:

# Using Docker with Redis Stack (includes RediSearch)
docker run -d --name redis-stack -p 6379:6379 redis/redis-stack:latest

Local Valkey Bundle:

# Example Valkey bundle with search/vector support
docker run -d --name valkey-bundle -p 6379:6379 valkey/valkey-bundle:9.0.0

Configuration Options

Go SDK

// Configure Redis-compatible vector store (Redis or Valkey endpoint)
vectorConfig := &vectorstore.Config{
    Enabled: true,
    Type:    vectorstore.VectorStoreTypeRedis, // Keep type as "redis" for Valkey too
    Config: vectorstore.RedisConfig{
        Addr:     "localhost:6379",        // Redis/Valkey server address - REQUIRED
        Username: "",                      // Optional: Redis username
        Password: "",                      // Optional: Redis password
        DB:       0,                       // Optional: Redis database number (default: 0)

        // Optional: Connection pool settings
        PoolSize:        10,               // Maximum socket connections
        MaxActiveConns:  10,               // Maximum active connections
        MinIdleConns:    5,                // Minimum idle connections
        MaxIdleConns:    10,               // Maximum idle connections

        // Optional: Timeout settings
        DialTimeout:     5 * time.Second,  // Connection timeout
        ReadTimeout:     3 * time.Second,  // Read timeout
        WriteTimeout:    3 * time.Second,  // Write timeout
        ContextTimeout:  10 * time.Second, // Operation timeout
    },
}

// Create vector store
store, err := vectorstore.NewVectorStore(context.Background(), vectorConfig, logger)
if err != nil {
    log.Fatal("Failed to create vector store:", err)
}

config.json

{
  "vector_store": {
    "enabled": true,
    "type": "redis",
    "config": {
      "addr": "localhost:6379",
      "username": "",
      "password": "",
      "db": 0,
      "pool_size": 10,
      "max_active_conns": 10,
      "min_idle_conns": 5,
      "max_idle_conns": 10,
      "dial_timeout": "5s",
      "read_timeout": "3s",
      "write_timeout": "3s",
      "context_timeout": "10s"
    }
  }
}

For Redis Cloud or Valkey service endpoints:

{
  "vector_store": {
    "enabled": true,
    "type": "redis",
    "config": {
      "addr": "your-redis-host:port",
      "username": "your-username",
      "password": "your-password",
      "db": 0,
      "context_timeout": "10s"
    }
  }
}

Redis-Specific Features

Vector Search Algorithm: Redis uses the HNSW (Hierarchical Navigable Small World) algorithm for vector similarity search, which provides:

• Fast Search: O(log N) search complexity
• High Accuracy: Excellent recall rates for similarity search
• Memory Efficient: Optimized for in-memory operations
• Cosine Similarity: Uses cosine distance metric for semantic similarity

Connection Pool Management: Redis provides extensive connection pool configuration:

config := vectorstore.RedisConfig{
    Addr:            "localhost:6379",
    PoolSize:        20,                    // Max socket connections
    MaxActiveConns:  20,                    // Max active connections
    MinIdleConns:    5,                     // Min idle connections
    MaxIdleConns:    10,                    // Max idle connections
    ConnMaxLifetime: 30 * time.Minute,      // Connection lifetime
    ConnMaxIdleTime: 5 * time.Minute,       // Idle connection timeout
    DialTimeout:     5 * time.Second,       // Connection timeout
    ReadTimeout:     3 * time.Second,       // Read timeout
    WriteTimeout:    3 * time.Second,       // Write timeout
    ContextTimeout:  10 * time.Second,      // Operation timeout
}

Performance Optimization

Connection Pool Tuning: For high-throughput applications, tune the connection pool settings:

{
  "vector_store": {
    "config": {
      "pool_size": 50,           // Increase for high concurrency
      "max_active_conns": 50,    // Match pool_size
      "min_idle_conns": 10,      // Keep connections warm
      "max_idle_conns": 20,      // Allow some idle connections
      "conn_max_lifetime": "1h", // Refresh connections periodically
      "conn_max_idle_time": "10m" // Close idle connections
    }
  }
}

Memory Optimization:

• TTL: Use appropriate TTL values to prevent memory bloat
• Namespace Cleanup: Regularly clean up unused namespaces

Batch Operations: Redis supports efficient batch operations:

// Batch retrieval
results, err := store.GetChunks(ctx, namespace, []string{"id1", "id2", "id3"})

// Batch deletion
deleteResults, err := store.DeleteAll(ctx, namespace, queries)

Production Considerations

Note

Search Module Required: Redis/Valkey integration requires a search module/API that supports FT.* commands (index creation and vector search). If FT.INFO or FT.SEARCH is unavailable, semantic caching will not work.

Caution

Production Considerations:

• Use Redis AUTH for production deployments
• Configure appropriate connection timeouts
• Monitor memory usage and set appropriate TTL values

---

Qdrant

Qdrant is a high-performance vector search engine built in Rust.

Setup & Installation

Local Qdrant:

# Using Docker
docker run -d \
  --name qdrant \
  -p 6333:6333 \
  -p 6334:6334 \
  -v $(pwd)/qdrant_storage:/qdrant/storage \
  qdrant/qdrant:latest

Qdrant Cloud: Sign up at cloud.qdrant.io

Configuration Options

Go SDK

vectorConfig := &vectorstore.Config{
    Enabled: true,
    Type:    vectorstore.VectorStoreTypeQdrant,
    Config: vectorstore.QdrantConfig{
        Host:   "localhost",
        Port:   6334,
        APIKey: "",
        UseTLS: false,
    },
}

store, err := vectorstore.NewVectorStore(context.Background(), vectorConfig, logger)

config.json

Local Setup:

{
  "vector_store": {
    "enabled": true,
    "type": "qdrant",
    "config": {
      "host": "localhost",
      "port": 6334
    }
  }
}

Cloud Setup:

{
  "vector_store": {
    "enabled": true,
    "type": "qdrant",
    "config": {
      "host": "your-qdrant-cluster.cloud.qdrant.io",
      "port": 6334,
      "api_key": "your-qdrant-api-key",
      "use_tls": true
    }
  }
}

Note

Qdrant uses port 6334 for gRPC and port 6333 for REST. DeepIntShield uses the gRPC port.

Caution

Qdrant requires all IDs to be valid UUIDs. Use uuid.New().String() to generate IDs.

---

Pinecone

Pinecone is a managed vector database service designed for machine learning applications, offering both serverless and pod-based deployment options.

Key Features

• Managed Service: Fully managed with no infrastructure to maintain
• Serverless Option: Pay-per-use pricing with automatic scaling
• High Performance: Optimized for low-latency vector search
• Metadata Filtering: Advanced filtering on vector metadata
• Namespaces: Organize vectors into separate namespaces within an index

Setup & Installation

Pinecone Cloud:

• Sign up at pinecone.io
• Create a new index with the desired dimensions
• Get your API key and index host URL from the console

Local Development (Pinecone Local):

docker run -d \
  --name pinecone-local \
  -p 5081:5081 \
  ghcr.io/pinecone-io/pinecone-index:latest

Configuration Options

Go SDK

vectorConfig := &vectorstore.Config{
    Enabled: true,
    Type:    vectorstore.VectorStoreTypePinecone,
    Config: vectorstore.PineconeConfig{
        APIKey:    "your-pinecone-api-key",
        IndexHost: "your-index-host.svc.environment.pinecone.io",
    },
}

store, err := vectorstore.NewVectorStore(context.Background(), vectorConfig, logger)

config.json

Cloud Setup:

{
  "vector_store": {
    "enabled": true,
    "type": "pinecone",
    "config": {
      "api_key": "your-pinecone-api-key",
      "index_host": "your-index-host.svc.environment.pinecone.io"
    }
  }
}

Local Development:

{
  "vector_store": {
    "enabled": true,
    "type": "pinecone",
    "config": {
      "api_key": "pclocal",
      "index_host": "localhost:5081"
    }
  }
}

Note

For local development with Pinecone Local, any API key value works (e.g., “pclocal”). The index host should point to localhost:5081 by default.

Caution

Pinecone requires all IDs to be unique strings. Namespaces are created automatically when you first upsert vectors.

---

Use Cases

Semantic Caching

Build intelligent caching systems that understand query intent rather than just exact matches.

Applications:

• Customer support systems with FAQ matching
• Code completion and documentation search
• Content management with semantic deduplication

Knowledge Base & Search

Create intelligent search systems that understand user queries contextually.

Applications:

• Document search and retrieval systems
• Product recommendation engines
• Research paper and knowledge discovery platforms

Content Classification

Automatically categorize and tag content based on semantic similarity.

Applications:

• Email classification and routing
• Content moderation and filtering
• News article categorization and clustering

Recommendation Systems

Build personalized recommendation engines using vector similarity.

Applications:

• Product recommendations based on user preferences
• Content suggestions for media platforms
• Similar document or article recommendations

Related Documentation

Topic	Documentation	Description
Framework Overview	What is Framework	Understanding the framework package and VectorStore interface
Semantic Caching	Semantic Caching	Using VectorStore for AI response caching