Garnet Distributed Caching

Version: v102.0 Status: Available since v102.0 Audience: Administrators, DevOps

---

Overview

Cesivi integrates Microsoft Garnet as an optional embedded distributed cache. Garnet is a high-performance, Redis-compatible cache-store from Microsoft Research that runs in-process within each Cesivi.Server instance.

Key benefits: - No external infrastructure — Garnet runs embedded in the Cesivi process, no separate Redis/Valkey server needed - AOF persistence — Append-Only File ensures cache data survives restarts - Cluster mode — Multiple Cesivi servers auto-form a Garnet cluster for distributed caching - Redis compatible — Uses the Redis Serialization Protocol (RESP), compatible with standard Redis clients - Sub-millisecond latency — In-process access eliminates network overhead for local reads

Architecture

Each Cesivi.Server instance embeds a Garnet server node as an IHostedService. The application connects to its own localhost Garnet via StackExchange.Redis. When multiple Cesivi servers are deployed, Garnet nodes discover each other (via config or DNS-SRV) and form a cluster automatically.

[Cesivi.Server A]                    [Cesivi.Server B]
+-----------------------------+      +-----------------------------+
| ASP.NET Core                |      | ASP.NET Core                |
|   ICacheProviderFactory     |      |   ICacheProviderFactory     |
|   -> GarnetCacheProvider    |      |   -> GarnetCacheProvider    |
|   -> StackExchange.Redis    |      |   -> StackExchange.Redis    |
|      -> localhost:6310      |      |      -> localhost:6310      |
|                             |      |                             |
|   Embedded GarnetServer     |      |   Embedded GarnetServer     |
|   Port: 6310                |<---->|   Port: 6310                |
|   Cluster: enabled          |gossip|   Cluster: enabled          |
|   AOF: enabled              |      |   AOF: enabled              |
+-----------------------------+      +-----------------------------+

---

Quick Start

1. Enable Garnet

In appsettings.json, set Cesivi:Garnet:Enabled to true:

{
  "Cesivi": {
    "Garnet": {
      "Enabled": true
    }
  }
}

That's it. Garnet starts with the application using sensible defaults (port 6310, AOF enabled, 256MB memory).

2. Verify It's Running

Check the application logs at startup:

[INF] GarnetHostedService: Starting embedded Garnet server on 127.0.0.1:6310
[INF] GarnetHostedService: AOF enabled (commit every 100ms)
[INF] GarnetHostedService: Garnet server started successfully

You can also connect with any Redis client:

redis-cli -p 6310 PING
# PONG

---

Configuration Reference

All Garnet settings live under Cesivi:Garnet in appsettings.json:

{
  "Cesivi": {
    "Garnet": {
      "Enabled": false,
      "Port": 6310,
      "Address": "127.0.0.1",
      "MemorySize": "256m",
      "IndexSize": "32m",
      "EnableAOF": true,
      "AofCommitFrequencyMs": 100,
      "AofSizeLimit": "256m",
      "CheckpointDir": "./data/garnet",
      "Recover": true,
      "Password": "",
      "EnableCluster": false,
      "Peers": [],
      "DnsSrv": {
        "Enabled": false,
        "ServiceName": "_garnet._tcp.cesivi.local",
        "RefreshIntervalSeconds": 60
      }
    }
  }
}

Settings Detail

Setting	Type	Default	Description
Enabled	bool	false	Master toggle. When false, InMemory caching is used (no Garnet process).
Port	int	6310	TCP port for the embedded Garnet server. Override via .branch-config.json garnetPort field.
Address	string	"127.0.0.1"	Bind address. Use "0.0.0.0" to accept connections from other hosts (cluster mode).
MemorySize	string	"256m"	Maximum memory for the main store. Supports k, m, g suffixes.
IndexSize	string	"32m"	Memory for the hash index. Supports k, m, g suffixes.
EnableAOF	bool	true	Enable Append-Only File for persistence. Recommended for production.
AofCommitFrequencyMs	int	100	How often AOF commits to disk (milliseconds). Lower = more durable, slightly lower throughput.
AofSizeLimit	string	"256m"	Maximum AOF file size before compaction.
CheckpointDir	string	"./data/garnet"	Directory for AOF and checkpoint files.
Recover	bool	true	Recover data from AOF/checkpoint on startup. Set false to start fresh.
Password	string	""	Optional password for Redis AUTH. Leave empty for no authentication (localhost-only).
EnableCluster	bool	false	Enable Garnet cluster mode for multi-server deployments. See Cluster Setup.
Peers	string[]	[]	Static list of peer Garnet nodes (e.g., ["server-b:6310", "server-c:6310"]).
DnsSrv.Enabled	bool	false	Enable DNS-SRV record discovery for peer nodes.
DnsSrv.ServiceName	string	"_garnet._tcp.cesivi.local"	DNS-SRV service name to query for peer discovery.
DnsSrv.RefreshIntervalSeconds	int	60	How often to re-query DNS-SRV for new/removed peers.

---

AOF Persistence

Garnet's Append-Only File (AOF) records every write operation to disk. On restart, Garnet replays the AOF to restore the cache state.

How It Works

1. Every write (SET, DELETE, etc.) is appended to the AOF
2. The AOF is committed to disk every AofCommitFrequencyMs milliseconds (default: 100ms)
3. On startup with Recover: true, Garnet replays the AOF to restore state
4. When the AOF exceeds AofSizeLimit, Garnet compacts it

Best Practices

• Leave AOF enabled for production (default). Cache warm-up after restart is automatic.
• Set CheckpointDir to a fast local disk (SSD recommended).
• Monitor disk usage — AOF files grow with write volume. The AofSizeLimit triggers compaction.
• For ephemeral caches (dev/test), set EnableAOF: false to avoid disk I/O.

Disk Space Requirements

Approximate AOF sizes based on cache usage:

Cached Items	Approximate AOF Size
1,000	~1-5 MB
10,000	~10-50 MB
100,000	~100-500 MB

Actual size depends on value sizes and write patterns.

---

Cluster Setup

For multi-server Cesivi deployments, Garnet cluster mode enables distributed caching across all instances. The cluster uses 16,384 hash slots (same as Redis Cluster) to distribute keys.

Option 1: Static Peer Configuration

List all peer servers in appsettings.json:

{
  "Cesivi": {
    "Garnet": {
      "Enabled": true,
      "Address": "0.0.0.0",
      "EnableCluster": true,
      "Peers": [
        "cesivi-server-b:6310",
        "cesivi-server-c:6310"
      ]
    }
  }
}

Each server lists the other servers as peers. On startup, the GarnetClusterManager issues CLUSTER MEET commands to join the cluster.

Option 2: DNS-SRV Auto-Discovery

For dynamic environments (Docker Swarm, Kubernetes), use DNS-SRV records:

{
  "Cesivi": {
    "Garnet": {
      "Enabled": true,
      "Address": "0.0.0.0",
      "EnableCluster": true,
      "DnsSrv": {
        "Enabled": true,
        "ServiceName": "_garnet._tcp.cesivi.local",
        "RefreshIntervalSeconds": 30
      }
    }
  }
}

DNS-SRV record example:

_garnet._tcp.cesivi.local. 60 IN SRV 0 0 6310 cesivi-a.cesivi.local.
_garnet._tcp.cesivi.local. 60 IN SRV 0 0 6310 cesivi-b.cesivi.local.
_garnet._tcp.cesivi.local. 60 IN SRV 0 0 6310 cesivi-c.cesivi.local.

The GarnetClusterManager periodically queries DNS-SRV and automatically meets new peers or handles removed nodes.

Network Requirements

• All Garnet nodes must be able to reach each other on the configured port (default: 6310)
• Garnet uses a gossip protocol on the same port for cluster state
• Firewall rules: Allow TCP traffic on port 6310 (or configured port) between all Cesivi servers

Cache Invalidation

When cluster mode is active, Cesivi uses Garnet's Pub/Sub mechanism for cache invalidation across all nodes. This replaces the HTTP-based farm broadcast used in single-server mode.

• Channel: cesivi:cache:invalidation
• Automatic: No configuration needed beyond enabling the cluster
• Fallback: If Garnet cluster is unavailable, falls back to HTTP broadcast

---

Migration from InMemory to Garnet

Switching from InMemory caching to Garnet is seamless — no data migration required:

1. Set Cesivi:Garnet:Enabled: true in appsettings.json
2. Restart Cesivi.Server
3. The cache starts empty and warms up naturally as requests are served
4. All existing ICacheProvider consumers transparently use Garnet

Zero-Downtime Migration (Multi-Server)

For multi-server deployments with a load balancer:

1. Update appsettings.json on all servers (but don't restart yet)
2. Restart servers one at a time (rolling restart)
3. Each restarted server starts with Garnet; the cache warms up from client requests
4. Once all servers are restarted, enable cluster mode for shared caching

Rollback

To revert to InMemory caching:

1. Set Cesivi:Garnet:Enabled: false
2. Restart the server
3. InMemory caching resumes immediately (cache starts empty)

---

Non-Serializable Caches

Some Cesivi caches contain objects that cannot be serialized to Garnet. These remain in-memory regardless of the Garnet setting:

Cache	Reason	Impact
ReflectionCache	Contains compiled expression trees and delegate references	None — per-process optimization, not shared between servers
SessionCache	Contains non-serializable session-specific objects (HttpContext references, CSOM state)	None — sessions are per-server by design

All other caches (CamlQueryCache, CacheService, ObjectPathCache, DirectoryScanCache) transparently use Garnet when enabled.

---

Branch Configuration

Each branch in .branch-config.json has a garnetPort field to avoid port conflicts when running multiple branches simultaneously:

Branch	HTTP Port	Garnet Port
main	5010	6310
main1	5020	6320
main2	5030	6330
main3	5040	6340
main4	5050	6350
main5	5060	6360

The GarnetHostedService reads garnetPort from .branch-config.json and uses it as the Garnet server port. If not specified, the default port from Cesivi:Garnet:Port (6310) is used.

---

Troubleshooting

Garnet won't start

Check: Port already in use

# Check if port 6310 is occupied
netstat -ano | findstr :6310
# On Linux:
ss -tlnp | grep 6310

Fix: Change the port in appsettings.json or kill the process using the port.

AOF recovery fails

Symptom: Failed to recover from AOF in logs Fix: Delete the AOF files in CheckpointDir and restart. The cache will start empty.

rm -rf ./data/garnet/*

Cluster nodes not connecting

Check: 1. Firewall rules allow TCP traffic on the Garnet port between all servers 2. DNS names resolve correctly: nslookup cesivi-server-b 3. Garnet is listening on 0.0.0.0 (not 127.0.0.1) for cluster mode 4. All nodes use the same Garnet password (if set)

Diagnostic:

# From server A, try connecting to server B's Garnet
redis-cli -h cesivi-server-b -p 6310 PING

High memory usage

Garnet's memory is controlled by MemorySize (default: 256MB). If Cesivi's cache requirements grow:

1. Increase MemorySize in appsettings.json
2. Monitor with redis-cli INFO memory
3. Consider enabling cluster mode to distribute keys across servers

Performance diagnostics

Use any Redis-compatible client to inspect Garnet:

redis-cli -p 6310 INFO stats       # General statistics
redis-cli -p 6310 INFO memory      # Memory usage
redis-cli -p 6310 DBSIZE           # Number of keys
redis-cli -p 6310 INFO keyspace    # Key distribution

---

See Also

• Multi-Server Deployment — Deploying multiple Cesivi instances
• Storage Providers — Storage backend configuration
• Performance Tuning Guide — Performance optimization
• appsettings.json Reference — Full configuration reference

---

Last Updated: 2026-03-09 Introduced in v102.0 (PLAN-537, PLAN-538, PLAN-539)