Storage Provider Guide

Home → Documentation → Reference → Storage Providers

Overview

Cesivi Server supports pluggable storage backends through the IStorageService interface. This allows you to choose between different storage implementations based on your needs.

Available Providers

FileSystemStorageService (Default)

Stores all SharePoint data in the file system as JSON files and binary blobs.

Pros: - Persistent across restarts - Human-readable JSON files - Easy to inspect and debug - Supports version history - Works with export/import tools

Cons: - Slower than in-memory (disk I/O) - Not suitable for high-concurrency scenarios - File system size limitations

Configuration:

{
  "Cesivi": {
    "StorageProvider": "FileSystem",
    "DataRootPath": "../MockData"
  }
}

InMemoryStorageService

Stores all data in thread-safe in-memory collections. Data is lost on restart.

Pros: - Extremely fast (no disk I/O) - Thread-safe using ConcurrentDictionary - Perfect for unit tests - No file system dependencies

Cons: - Data lost on restart - Memory consumption grows with data size - Not suitable for production scenarios requiring persistence

Configuration:

{
  "Cesivi": {
    "StorageProvider": "InMemory"
  }
}

SqliteStorageService

Stores all data in a SQLite database file. Best balance of performance and persistence.

Pros: - Fast queries (10x faster than FileSystem for complex queries) - ACID transactions - Single file for entire storage - Lower memory usage than InMemory - Supports concurrent access

Cons: - Binary format (not human-readable) - Requires SQLite libraries - Database file can grow large

Configuration:

{
  "Cesivi": {
    "StorageProvider": "Sqlite",
    "SqlitePath": "./Data/sharepoint.db"
  }
}

Or via environment variable:

export CESIVI_SQLITE_PATH=/var/data/sharepoint.db

LiteDbStorageService

Stores all data in a LiteDB embedded NoSQL database file. MongoDB-like API with document storage.

Pros: - Single DLL deployment (~350kb), no native dependencies - MongoDB-like API with BSON format - LINQ query support - Cross-platform (works everywhere .NET runs) - FileStorage API for binary content - Lower resource usage than SQLite

Cons: - Binary format (not human-readable) - Less mature than SQLite - Fewer tooling options

Configuration:

{
  "Cesivi": {
    "StorageProvider": "LiteDb",
    "LiteDbPath": "./Data/sharepoint.litedb"
  }
}

Or via environment variable:

export CESIVI_LITEDB_PATH=/var/data/sharepoint.litedb

SqlServerStorageService

Stores all data in a Microsoft SQL Server database. Enterprise-grade storage for production deployments.

Pros: - Enterprise-grade reliability (AlwaysOn, failover clustering) - High availability and disaster recovery support - Advanced query optimization and indexing - Centralized database for multiple SPM instances - Works with SQL Server 2016+, Azure SQL Database - Full transaction support with ACID guarantees - Supports heavy concurrent access

Cons: - Requires SQL Server instance (licensed or Express) - Binary format (not human-readable) - More complex setup than embedded databases - Network latency if database is remote

Configuration:

{
  "Cesivi": {
    "StorageProvider": "SqlServer",
    "SqlServerConnectionString": "Server=localhost;Database=Cesivi;Trusted_Connection=True;TrustServerCertificate=True"
  }
}

Or via environment variable:

export CESIVI_SQLSERVER_CONNECTIONSTRING="Server=localhost;Database=Cesivi;Trusted_Connection=True;TrustServerCertificate=True"

Database Schema: The database schema is automatically created on first run. The provider uses a hybrid approach: - Relational structure for hierarchy (WebApplications, SiteCollections, Webs, Lists) - JSON columns for full object serialization (backward compatible with existing models) - Binary columns for document content and attachments - Indexes on commonly queried fields (ServerRelativeUrl, Title, UniqueId)

Use Cases: - Enterprise deployments with centralized database infrastructure - High availability scenarios requiring SQL Server AlwaysOn - Multi-server Cesivi instances sharing data - Advanced querying leveraging SQL Server's query optimizer - Existing SQL infrastructure to minimize operational overhead

MySqlStorageService

Stores all data in a MySQL/MariaDB database. Open-source database for cost-effective production deployments.

Pros: - Open source with no licensing costs - Works with MySQL 8.0+ and MariaDB 10.6+ - True async I/O with MySqlConnector library - Cross-platform (Linux, Windows, macOS) - Cloud-ready (AWS RDS MySQL, Azure Database for MySQL, Google Cloud SQL) - Good performance with InnoDB engine - Full Unicode support (utf8mb4) - Active community and ecosystem

Cons: - Binary format (not human-readable) - Requires MySQL/MariaDB server installation - Network latency if database is remote - Less enterprise features than SQL Server

Configuration:

{
  "Cesivi": {
    "StorageProvider": "MySQL",
    "MySqlConnectionString": "Server=localhost;Database=Cesivi;User=spmock;Password=xxx;SslMode=Preferred"
  }
}

Or via environment variable:

export CESIVI_MYSQL_CONNECTIONSTRING="Server=localhost;Database=Cesivi;User=spmock;Password=xxx"

Database Schema: The database schema is automatically created on first run. Key features: - CHAR(36) for GUID storage (MySQL has no native GUID type) - JSON type for structured data columns - LONGBLOB for binary document storage (up to 4GB) - DATETIME(6) with microsecond precision timestamps - ENGINE=InnoDB with utf8mb4 character set for full Unicode - INSERT...ON DUPLICATE KEY UPDATE for upsert operations

Use Cases: - Open source stack deployments without SQL Server licensing - Docker/Linux environments (official MySQL Docker images) - Cloud MySQL services (AWS RDS, Azure, Google Cloud) - MariaDB environments (compatible alternative) - Cost-sensitive deployments requiring enterprise-grade database

Docker Compose Example:

version: '3.8'
services:
  spm-server:
    image: Cesivi:latest
    environment:
      - CESIVI__StorageProvider=MySQL
      - CESIVI_MYSQL_CONNECTIONSTRING=Server=mysql;Database=spmock;User=spmock;Password=spmock123
    ports:
      - "5000:5000"
    depends_on:
      mysql:
        condition: service_healthy

  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: rootpassword
      MYSQL_DATABASE: spmock
      MYSQL_USER: spmock
      MYSQL_PASSWORD: spmock123
    ports:
      - "3306:3306"
    volumes:
      - mysql_data:/var/lib/mysql
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  mysql_data:

Switching Providers

To switch storage providers, update appsettings.json:

{
  "Cesivi": {
    "StorageProvider": "Sqlite"  // or "FileSystem", "InMemory", "LiteDb", "SqlServer", "MySQL"
  }
}

The server will automatically instantiate the correct provider on startup.

Storage Converter Tool

Use the Storage Converter CLI to migrate data between providers:

# FileSystem → SQLite
CesiviStorageConverter convert -s filesystem -t sqlite

# FileSystem → LiteDB
CesiviStorageConverter convert -s filesystem -t litedb

# SQLite → FileSystem (backup)
CesiviStorageConverter convert -s sqlite -t filesystem --target-path ./Backup

# LiteDB → SQLite (provider migration)
CesiviStorageConverter convert -s litedb -t sqlite

# Selective conversion
CesiviStorageConverter convert -s filesystem -t sqlite --webapp "Intranet"

See Storage Converter Guide for full documentation.

Creating Custom Storage Providers

You can implement your own storage provider (e.g., SQL, MongoDB, graph database) by implementing the IStorageService interface.

Step 1: Implement IStorageService

using Cesivi.Services;

namespace MyCompany.Cesivi.Storage
{
    public class SqlStorageService : IStorageService
    {
        private readonly string _connectionString;

        public SqlStorageService(IConfiguration configuration)
        {
            _connectionString = configuration["Cesivi:SqlConnectionString"];
        }

        // Implement all 190+ methods from IStorageService
        public async Task<WebApplication?> LoadWebApplicationAsync(string name)
        {
            // Your SQL implementation
            using var connection = new SqlConnection(_connectionString);
            // ...
        }

        // ... implement remaining methods
    }
}

Step 2: Register in Program.cs

Update Program.cs to include your custom provider:

builder.Services.AddSingleton<IStorageService>(sp =>
{
    var config = sp.GetRequiredService<IConfiguration>();
    var providerType = config["Cesivi:StorageProvider"] ?? "FileSystem";

    return providerType.ToLower() switch
    {
        "inmemory" => new InMemoryStorageService(),
        "filesystem" => new FileSystemStorageService(config),
        "sql" => new SqlStorageService(config),  // Your custom provider
        _ => new FileSystemStorageService(config)
    };
});

Step 3: Configure in appsettings.json

{
  "Cesivi": {
    "StorageProvider": "Sql",
    "SqlConnectionString": "Server=localhost;Database=Cesivi;..."
  }
}

IStorageService Interface Methods

The IStorageService interface includes 190+ methods organized by entity type:

Core Entities

• WebApplication (4 methods): Load, Save, Delete, Exists
• SiteCollection (4 methods): Load, Save, Delete, Exists
• Web (4 methods): Load, Save, Delete, Exists
• List (4 methods): Load, Save, Delete, Exists
• ListItem (4 methods): Load, Save, Delete, Exists
• Document (7 methods): Load, Save, Delete, Exists, LoadMetadata, SaveMetadata, DeleteMetadata
• Folder (4 methods): Load, Save, Delete, Exists

User & Permission Management

• User (6 methods): Load, LoadByLoginName, GetAll, Save, Delete
• UserProfile (6 methods): Load, Save, Delete, GetAll, GetByIndex
• Group (8 methods): Load, LoadByName, GetAll, Save, Delete, AddUser, RemoveUser
• Role (6 methods): Load, LoadByName, GetAll, Save, Delete
• RoleAssignment (3 methods): Load, Save, Delete

Content Structure

• ContentType (4 methods): Load, GetAll, Save, Delete
• Field (5 methods): Load, GetAll, LoadListFields, LoadWebFields, Save, Delete
• View (6 methods): Load, GetAll, LoadListViews, Save, SaveListView, Delete

Files & Folders

• SPFile (6 methods): Load, LoadByUrl, GetAll, Save, SaveContent, LoadContent, Delete
• SPFolder (5 methods): Load, LoadByUrl, GetAll, Save, Delete
• FileVersion (6 methods): Get, Save, Delete, DeleteByLabel, DeleteAll, Restore
• Attachment (4 methods): Get, Save, Delete, Load

Taxonomy

• GlobalTermStore (4 methods): Load, Save, Delete, Exists
• SiteCollectionTermStore (4 methods): Load, Save, Delete, Exists
• TermStore (2 methods): Load, Save
• TermGroup (3 methods): Load, Save, Delete
• TermSet (3 methods): Load, Save, Delete
• Term (3 methods): Load, Save, Delete

Search

• SearchIndex (4 methods): Load, Save, UpdateDocument, RemoveDocument

Recycle Bin

• RecycleBinItem (5 methods): GetAll, Save, Load, Restore, Delete

Document Workspaces

• DocumentWorkspace (5 methods): Load, Save, Delete, Exists, GetAll

Utilities

• GetMockDataPath (1 method): Returns root storage path

Best Practices

1. Thread Safety: Ensure your storage implementation is thread-safe
2. Async Operations: Use async/await for all I/O operations
3. Error Handling: Throw meaningful exceptions for storage errors
4. Performance: Consider caching for frequently accessed data
5. Testing: Write unit tests using the InMemoryStorageService
6. Transactions: Consider transaction support for multi-step operations

Performance Comparison

Provider	Read Speed	Write Speed	Memory	Persistence	Best For
InMemory	⚡⚡⚡⚡⚡	⚡⚡⚡⚡⚡	High	❌	Unit tests, dev
FileSystem	⚡⚡⚡	⚡⚡	Low	✅	Development, debugging
SQLite	⚡⚡⚡⚡	⚡⚡⚡	Low	✅	Production, high performance
LiteDB	⚡⚡⚡⚡	⚡⚡⚡	Low	✅	Single-file deployment, embedded
SQL Server*	⚡⚡⚡⚡	⚡⚡⚡	Low	✅	Enterprise
MongoDB*	⚡⚡⚡⚡	⚡⚡⚡⚡	Low	✅	High concurrency

*Not yet implemented - custom provider needed

InMemory Provider Note

The InMemory provider does NOT persist data to disk. Data is lost when the server restarts.

Use cases: - Unit testing - Integration testing - Temporary development scenarios - Storage Converter validation

NOT suitable for: - Production workloads requiring persistence - Long-running development sessions - Data migration (as a target for permanent storage)

Example: Using InMemory Provider for Tests

[Fact]
public async Task TestListCreation()
{
    // Arrange
    var config = new ConfigurationBuilder()
        .AddInMemoryCollection(new Dictionary<string, string>
        {
            ["Cesivi:StorageProvider"] = "InMemory"
        })
        .Build();

    var storage = new InMemoryStorageService();

    // Act
    var list = new List
    {
        Title = "Test List",
        BaseType = 0
    };
    await storage.SaveListAsync("webapp", "site", "TestList", list);
    var loaded = await storage.LoadListAsync("webapp", "site", "TestList");

    // Assert
    Assert.NotNull(loaded);
    Assert.Equal("Test List", loaded.Title);
}

Migration Between Providers

Option 1: Storage Converter CLI (Recommended)

Use the Storage Converter for direct provider-to-provider migration:

# FileSystem → SQLite
CesiviStorageConverter convert -s filesystem -t sqlite

# SQLite → FileSystem
CesiviStorageConverter convert -s sqlite -t filesystem

See Storage Converter Guide for full options.

Option 2: Export/Import via ZIP

For backup/restore scenarios:

1. Export storage as ZIP: POST /_api/admin/storage/export
2. Switch to new provider in configuration
3. Import ZIP: POST /_api/admin/storage/import

See Export/Import Guide for details.

Troubleshooting

Provider Not Found

Error: Provider 'XYZ' not recognized Solution: Check appsettings.json - valid values are "FileSystem", "InMemory", "Sqlite", or "LiteDb"

Memory Issues with InMemory

Error: OutOfMemoryException Solution: Switch to FileSystem provider or increase available memory

File Access Denied

Error: Access denied to C:\MockData\... Solution: Ensure the application has read/write permissions to DataRootPath

Future Enhancements

Planned storage providers: - SQL Server (tracked as separate goal) - PostgreSQL (tracked as separate goal) - MongoDB (tracked as separate goal) - Redis (for caching layer) - Azure Blob Storage (for cloud scenarios)

See Also

• Storage Converter Guide - Convert between storage providers
• Export/Import Guide - ZIP-based backup/restore
• Migration Guide - Migrate from real SharePoint
• Architecture - Overall system architecture