Storage Migration Guide¶

This guide explains how to migrate Cesivi Server data between different storage providers using the CesiviStorageConverter tool.

Overview¶

The CesiviStorageConverter tool allows you to: - Migrate from development to production storage (FileSystem/LiteDb → SQL Server/PostgreSQL) - Switch between storage providers without data loss - Create backups by exporting to FileSystem - Validate data integrity after migration - Filter migration by web application or site collection

Installation¶

The CesiviStorageConverter tool is included with Cesivi Server. Build it from source:

cd Cesivi.StorageConverter
dotnet build
dotnet run -- convert --help

Or run directly from the build output:

./Cesivi.StorageConverter/bin/Debug/net10.0/CesiviStorageConverter convert --help

Common Migration Scenarios¶

1. Development → Production (LiteDb → SQL Server)¶

Use Case: You've been developing with LiteDb and are ready to deploy to production with SQL Server.

CesiviStorageConverter convert \
  -s litedb \
  --source-path "./Data/sharepoint.litedb" \
  -t sqlserver \
  --target-connection "Server=prodserver;Database=Cesivi;User Id=spmuser;Password=SecurePassword;TrustServerCertificate=true"

Steps: 1. Backup source: Copy sharepoint.litedb to safe location 2. Create target database:

CREATE DATABASE Cesivi;

3. Run migration (command above) 4. Validate: Check counts match (see Validation section) 5. Update app config to use SQL Server 6. Test application 7. Keep backup until fully validated

2. Development → Production (FileSystem → PostgreSQL)¶

Use Case: You've been developing with FileSystem (JSON files) and want to migrate to PostgreSQL.

CesiviStorageConverter convert \
  -s filesystem \
  --source-path "./MockData" \
  -t postgresql \
  --target-connection "Host=postgres-server;Database=Cesivi;Username=spmuser;Password=SecurePassword"

Steps: 1. Backup source: Copy entire MockData folder 2. Create target database:

CREATE DATABASE Cesivi;

3. Run migration (command above) 4. Validate: Check entity counts 5. Update app config to use PostgreSQL 6. Test application

3. Cross-Platform Migration (SQL Server → PostgreSQL)¶

Use Case: Migrating from Windows/SQL Server to Linux/PostgreSQL.

CesiviStorageConverter convert \
  -s sqlserver \
  --source-connection "Server=windowsserver;Database=Cesivi;Integrated Security=true;TrustServerCertificate=true" \
  -t postgresql \
  --target-connection "Host=linuxserver;Database=Cesivi;Username=postgres;Password=SecurePassword"

Steps: 1. Backup SQL Server database 2. Create PostgreSQL database 3. Run migration (command above) 4. Validate (very important for cross-platform) 5. Test thoroughly (data type differences) 6. Gradual cutover (run parallel for a period)

4. Backup to FileSystem¶

Use Case: Export production data to human-readable JSON files for backup or inspection.

CesiviStorageConverter convert \
  -s sqlserver \
  --source-connection "Server=.;Database=Cesivi;Integrated Security=true;TrustServerCertificate=true" \
  -t filesystem \
  --target-path "./Backup-2026-01-11"

Result: Creates a folder with all data as JSON files that you can: - Commit to version control - Inspect manually - Use for debugging - Archive for compliance

5. Incremental Migration (Filter by Site)¶

Use Case: Migrate only specific site collections (large dataset, want to test first).

# Migrate only specific web application
CesiviStorageConverter convert \
  -s litedb --source-path "./sharepoint.litedb" \
  -t sqlserver --target-connection "..." \
  --webapp "IntranetApp"

# Migrate only specific site collections
CesiviStorageConverter convert \
  -s litedb --source-path "./sharepoint.litedb" \
  -t sqlserver --target-connection "..." \
  --site "TeamSite" --site "ProjectSite"

Command-Line Options¶

Basic Options¶

CesiviStorageConverter convert [OPTIONS]

Option	Description	Example
`-s, --source <TYPE>`	Source storage type	`-s litedb`
`-t, --target <TYPE>`	Target storage type	`-t sqlserver`
`--source-path <PATH>`	Source path (filesystem/db file)	`--source-path "./MockData"`
`--target-path <PATH>`	Target path (filesystem/db file)	`--target-path "./Backup"`
`--source-connection <CONNSTR>`	Source connection string	`--source-connection "Server=...;Database=..."`
`--target-connection <CONNSTR>`	Target connection string	`--target-connection "Host=...;Database=..."`

Storage Types¶

Supported values for -s and -t: - filesystem - JSON files in folders - inmemory - In-memory (testing only, not useful for migrations) - litedb - LiteDb embedded database - sqlite - SQLite database - sqlserver - Microsoft SQL Server - postgresql - PostgreSQL

Filtering Options¶

Option	Description	Example
`--webapp <NAME>`	Filter by web application name	`--webapp "Cesivi"`
`--site <NAME>`	Filter by site collection name	`--site "TeamSite" --site "ProjectSite"`

Note: Can specify multiple --webapp or --site options.

Document Options¶

Option	Description	Default
`--no-documents`	Skip document binary content	Include documents
`--max-doc-size <MB>`	Maximum document size to convert (MB)	0 (unlimited)

Use Case: For large file libraries, skip large files or all binary content to speed up migration.

# Skip all documents (metadata only)
CesiviStorageConverter convert -s litedb -t sqlserver --no-documents

# Skip documents larger than 10MB
CesiviStorageConverter convert -s litedb -t sqlserver --max-doc-size 10

Behavior Options¶

Option	Description	Default
`--no-clear`	Don't clear target storage before migration	Clear target
`--stop-on-error`	Stop on first error	Continue on error

Use Case: - --no-clear: Append data to existing target (advanced) - --stop-on-error: Ensure 100% success or rollback

# Strict migration (stop if any error)
CesiviStorageConverter convert -s litedb -t sqlserver --stop-on-error

# Append to existing data (dangerous!)
CesiviStorageConverter convert -s filesystem -t sqlserver --no-clear

Migration Process¶

What Gets Migrated¶

The tool migrates all SharePoint entities in this order:

Web Applications - Server-level configuration
Site Collections - Top-level sites
Webs - Sites and subsites
Lists - Lists and libraries
List Items - All items with metadata
Documents - Files with binary content
Folders - Folder hierarchy
Users - User accounts
Groups - SharePoint groups
Content Types - Content type definitions
Fields - Field definitions
Permissions - Role definitions and assignments

Migration Output¶

During migration, you'll see real-time progress:

  ____  ____    ____  _
 / ___||  _ \  / ___|| |_ ___  _ __ __ _  __ _  ___
 \___ \| |_) | \___ \| __/ _ \| '__/ _` |/ _` |/ _ \
  ___) |  __/   ___) | || (_) | | | (_| | (_| |  __/
 |____/|_|     |____/ \__\___/|_|  \__,_|\__, |\___|
                                          |___/
  ____                          _
 / ___|___  _ ____   _____ _ __| |_ ___ _ __
| |   / _ \| '_ \ \ / / _ \ '__| __/ _ \ '__|
| |__| (_) | | | \ V /  __/ |  | ||  __/ |
 \____\___/|_| |_|\_/ \___|_|   \__\___|_|

Source: litedb (./sharepoint.litedb)
Target: sqlserver (Server=.;Database=Cesivi;...)

✓ Storage validation successful!

Converting storage... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%

Completion Report¶

After migration completes, you'll see a detailed report:

┌─────────────────┬────────┐
│ Entity Type     │  Count │
├─────────────────┼────────┤
│ Web Applications│      1 │
│ Site Collections│      3 │
│ Webs            │     12 │
│ Lists           │     45 │
│ List Items      │  8,432 │
│ Documents       │    234 │
│ Users           │     89 │
│ Groups          │     15 │
│ Content Types   │     28 │
│ Fields          │    156 │
│ Total           │  9,015 │
└─────────────────┴────────┘

Duration: 00:02:34.567
Status: Success

Validation¶

Automatic Validation¶

The tool automatically validates: - ✅ Source storage is readable - ✅ Target storage is writable - ✅ Connection strings are valid - ⚠️ Entity counts (reported but doesn't fail)

Manual Validation¶

After migration, manually verify:

1. Entity Counts

Compare entity counts between source and target:

# Source (LiteDb)
- Open LiteDb with LiteDB Studio
- Count documents in each collection

# Target (SQL Server)
SELECT
    'Lists' AS Entity, COUNT(*) AS Count FROM Lists
UNION ALL
SELECT 'ListItems', COUNT(*) FROM ListItems
UNION ALL
SELECT 'Files', COUNT(*) FROM Files
-- etc.

2. Sample Data

Verify a few entities manually: - Open a list in source - Open same list in target - Compare item counts, metadata, files

3. Application Testing

Test your application: - Browse lists/libraries - Upload/download files - Search for items - Verify permissions

4. File Checksums

For critical files, compare checksums:

# Not built-in yet, but you can:
# 1. Export file to filesystem (source)
# 2. Export same file to filesystem (target)
# 3. Compare SHA256 hashes

Troubleshooting¶

Common Issues¶

1. Target Already Has Data

Error: Target storage already contains data

Solution: By default, the tool clears target before migration. If you see this error, the clear failed. Options: - Manually clear target (SQL: TRUNCATE TABLE ...) - Use --no-clear (advanced, may cause duplicates)

2. Connection Failed

Error: Could not validate source storage
Error: Could not connect to database

Solution: - Verify connection string format - Check database server is running - Verify credentials - Check firewall rules - For SQL Server, check TCP/IP is enabled - For PostgreSQL, check pg_hba.conf

3. Migration Incomplete

Status: Failed
Errors: 12 errors occurred

Solution: - Check error list at end of migration - By default, migration continues on errors (--continue-on-error) - Use --stop-on-error for strict migration - Fix errors and re-run migration

4. Document Upload Failed

Error: Could not save file {filename}: File too large

Solution: - Use --max-doc-size to skip large files - Use --no-documents to skip all files - Increase target database limits (SQL Server: FILESTREAM, PostgreSQL: max_allowed_packet)

5. Performance Too Slow

Migration taking hours for large datasets?

Solution: - Ensure indexes are created on target (SQL Server/PostgreSQL) - Use bulk operations (built-in for SQL Server/PostgreSQL) - Run migration on fast network (local or same datacenter) - Consider incremental migration (--webapp filter) - Disable full-text indexing during migration (re-enable after)

Performance Tips¶

For Large Datasets (100K+ items):

Disable constraints temporarily (SQL Server/PostgreSQL):

-- SQL Server
ALTER TABLE ListItems NOCHECK CONSTRAINT ALL;
-- Run migration
ALTER TABLE ListItems CHECK CONSTRAINT ALL;

Increase timeouts:

# Modify connection string
--target-connection "...;Command Timeout=300"

Use local database:
Run migration on same machine as database server
Reduces network latency

Incremental migration:

# Migrate one web app at a time
for webapp in App1 App2 App3; do
  CesiviStorageConverter convert ... --webapp "$webapp"
done

Migration Checklist¶

Use this checklist for production migrations:

Pre-Migration¶

[ ] Backup source data (database dump, folder copy, etc.)
[ ] Create target database
[ ] Test connection to source
[ ] Test connection to target
[ ] Document current entity counts (source)
[ ] Plan maintenance window (downtime)
[ ] Notify users of downtime

During Migration¶

[ ] Stop application (prevent writes to source)
[ ] Run migration command
[ ] Monitor progress (logs, network, disk)
[ ] Document start time

Post-Migration¶

[ ] Verify entity counts match
[ ] Test sample data manually
[ ] Update application configuration
[ ] Start application
[ ] Smoke test (browse, upload, search)
[ ] Monitor for errors
[ ] Document completion time
[ ] Keep backup until fully validated

Rollback Plan¶

If migration fails:

[ ] Restore source from backup
[ ] Clear target database
[ ] Document failure reason
[ ] Fix issue
[ ] Retry migration

Advanced Scenarios¶

Parallel Migration (Multiple Site Collections)¶

For very large datasets, migrate site collections in parallel:

# Terminal 1
CesiviStorageConverter convert ... --site "Site1" --site "Site2"

# Terminal 2
CesiviStorageConverter convert ... --site "Site3" --site "Site4"

# Terminal 3
CesiviStorageConverter convert ... --site "Site5" --site "Site6"

Note: Ensure each site collection is migrated only once (no overlap).

Dry Run (Test Migration)¶

Before migrating to production, test with InMemory target:

CesiviStorageConverter convert \
  -s sqlserver --source-connection "..." \
  -t inmemory

Result: Validates source data can be read, counts entities, but doesn't persist to target.

Selective Entity Migration¶

Want to migrate only lists (no documents)?

CesiviStorageConverter convert ... --no-documents

Want to migrate only metadata (no users/groups)?

Not currently supported, but you can: 1. Migrate to FileSystem 2. Manually delete Users/Groups folders 3. Migrate from FileSystem to target