Multinode Deployment

Overview

Running multiple DeepIntShield nodes provides high availability, load distribution, and fault tolerance for your AI gateway. This guide covers the recommended approach for deploying multiple DeepIntShield nodes in OSS deployments.

OSS vs Enterprise

Aspect	OSS Approach	Enterprise Approach
Configuration Source	Shared config.json file	Database with P2P sync
Sync Mechanism	File sharing (ConfigMap, volumes)	Gossip protocol (real-time)
Config Updates	Modify file + restart nodes	UI/API with automatic propagation

---

How It Works

All configuration in DeepIntShield is loaded into memory at startup. For OSS multinode deployments, the recommended approach is to use config.json without config_store enabled.

config.json as Single Source of Truth

When you deploy without config_store:

• No database involved - config.json is the only configuration source
• Shared file - All nodes read from the same config.json file
• Identical configuration - Since the source is shared, all nodes automatically have the same configuration
• No sync needed - The shared file itself ensures consistency

---

Why not to use config_store for Multinode OSS?

Using config_store (database-backed configuration) with multiple nodes in OSS creates a synchronization problem:

1. Config changes are local - When you update configuration via the UI or API, it updates the database and the in-memory config on that specific node only
2. No propagation mechanism - Other nodes don’t know about the change; they keep their existing in-memory configuration
3. Nodes become out of sync - Different nodes end up with different configurations
4. Restart required - You’d have to restart all nodes after every config change to bring them back in sync

This defeats the purpose of having database-backed configuration with real-time updates.

Caution

Without P2P clustering (Enterprise feature), there’s no mechanism to notify other nodes of configuration changes. For OSS multinode deployments, use the shared config.json approach instead.

Enterprise Solution

DeepIntShield Enterprise includes P2P clustering with gossip protocol that automatically syncs configuration changes across all nodes in real-time. See the Clustering documentation for details.

---

Setting Up Multinode OSS Deployment

Example config.json

Create a config.json without config_store or logs_store:

Note

If you use PostgreSQL for logs_store, ensure the target database is UTF8 encoded. See PostgreSQL UTF8 Requirement.

{
  "$schema": "https://www.getbifrost.ai/schema",
  "client": {
    "drop_excess_requests": false,
    "enable_logging": false
  },
  "config_store": {
    "enabled": false
  },
  "logs_store": {
    "enabled": true,
    "type": "postgres",
    "config": {...}
  },
  "providers": {
    "openai": {
      "keys": [
        {
          "name": "openai-primary",
          "value": "env.OPENAI_API_KEY",
          "models": ["gpt-4o", "gpt-4o-mini"],
          "weight": 1.0
        }
      ]
    },
    "anthropic": {
      "keys": [
        {
          "name": "anthropic-primary",
          "value": "env.ANTHROPIC_API_KEY",
          "models": ["claude-sonnet-4-20250514", "claude-3-5-haiku-20241022"],
          "weight": 1.0
        }
      ]
    }
  }
}

Note

Notice config_store is disabled. This ensures all configuration comes from the file only.

Kubernetes Deployment

Use a ConfigMap to share the same configuration across all pods:

apiVersion: v1
kind: ConfigMap
metadata:
  name: deepintshield-config
  namespace: default
data:
  config.json: |
    {
      "$schema": "https://www.getbifrost.ai/schema",
      "client": {
        "drop_excess_requests": false,
        "enable_logging": false
      },
      "config_store": {
        "enabled": false
      },
      "logs_store": {
        "enabled": true,
        "type": "postgres",
        "config": {...}
      },
      "providers": {
        "openai": {
          "keys": [
            {
              "name": "openai-primary",
              "value": "env.OPENAI_API_KEY",
              "models": ["gpt-4o", "gpt-4o-mini"],
              "weight": 1.0
            }
          ]
        }
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deepintshield
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: deepintshield
  template:
    metadata:
      labels:
        app: deepintshield
    spec:
      containers:
      - name: deepintshield
        image: maximhq/deepintshield:latest
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: provider-secrets
              key: openai-api-key
        volumeMounts:
        - name: config
          mountPath: /app
          readOnly: true
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
          limits:
            cpu: 1000m
            memory: 1Gi
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
      volumes:
      - name: config
        configMap:
          name: deepintshield-config
---
apiVersion: v1
kind: Service
metadata:
  name: deepintshield
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app: deepintshield
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http

Docker Compose

Share the configuration using a bind mount:

version: '3.8'

services:
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - deepintshield-1
      - deepintshield-2
      - deepintshield-3

  deepintshield-1:
    image: maximhq/deepintshield:latest
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
    volumes:
      - ./config.json:/app/config.json:ro
    expose:
      - "8080"

  deepintshield-2:
    image: maximhq/deepintshield:latest
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
    volumes:
      - ./config.json:/app/config.json:ro
    expose:
      - "8080"

  deepintshield-3:
    image: maximhq/deepintshield:latest
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
    volumes:
      - ./config.json:/app/config.json:ro
    expose:
      - "8080"

nginx.conf for load balancing:

events {
    worker_connections 1024;
}

http {
    upstream deepintshield {
        least_conn;
        server deepintshield-1:8080;
        server deepintshield-2:8080;
        server deepintshield-3:8080;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://deepintshield;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_connect_timeout 60s;
            proxy_send_timeout 60s;
            proxy_read_timeout 60s;
        }

        location /health {
            access_log off;
            return 200 "healthy\n";
        }
    }
}

Bare Metal / VM Deployment

For bare metal or VM deployments, distribute the configuration file using:

• NFS mount - Mount a shared NFS directory containing config.json
• rsync - Sync the config file from a central location to all nodes
• Configuration management - Use Ansible, Chef, or Puppet to deploy identical configs

Example with rsync:

# On config server - push to all nodes
for node in node1 node2 node3; do
  rsync -avz /etc/deepintshield/config.json $node:/etc/deepintshield/config.json
done

# Restart nodes after config update
for node in node1 node2 node3; do
  ssh $node "systemctl restart deepintshield"
done

---

Updating Configuration

To update configuration in a multinode OSS deployment:

1. Modify the shared config.json file

   • Update the ConfigMap (Kubernetes)
   • Edit the shared file (Docker Compose / bare metal)

2. Restart the nodes

   • Rolling restart is supported - nodes can be restarted one at a time
   • Each node picks up the new configuration on startup

Kubernetes Rolling Restart

# Update ConfigMap
kubectl apply -f configmap.yaml

# Trigger rolling restart
kubectl rollout restart deployment/deepintshield

# Watch the rollout
kubectl rollout status deployment/deepintshield

Docker Compose Restart

# After updating config.json
docker-compose restart deepintshield-1
docker-compose restart deepintshield-2
docker-compose restart deepintshield-3

---

Best Practices

Use Environment Variables for Secrets

Never put API keys directly in config.json. Use the env. prefix to reference environment variables:

{
  "providers": {
    "openai": {
      "keys": [
        {
          "value": "env.OPENAI_API_KEY"
        }
      ]
    }
  }
}

Then provide the actual keys via environment variables or Kubernetes secrets.

Load Balancer Configuration

Always put a load balancer in front of your DeepIntShield nodes:

• Kubernetes: Use a Service with type: LoadBalancer or an Ingress
• Docker/VMs: Use nginx, HAProxy, or a cloud load balancer

Health Checks

Configure health checks to ensure traffic only goes to healthy nodes:

• Liveness endpoint: GET /health
• Readiness endpoint: GET /health

Resource Allocation

For production deployments:

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 2000m
    memory: 2Gi

---

Summary

Scenario	Recommendation
Single node	Use config_store for UI access
Multinode OSS	Use shared config.json without config_store
Multinode Enterprise	Use P2P clustering with config_store

For OSS multinode deployments, the shared config.json approach provides a simple, reliable way to keep all nodes in sync without the complexity of database synchronization.