Cesivi Server - Migration Guide

Purpose: Guide for migrating SharePoint data from real SharePoint servers to the Mock Server Target Audience: Developers, testers, system administrators Last Updated: 2026-03-28 (verified)

---

Table of Contents

1. Overview
2. Prerequisites
3. Cesivi.MigrationTool (Recommended)
4. Migration Strategies
5. Manual Export Methods
6. PnP PowerShell Export
7. Mock Data Structure
8. Import to Mock Server
9. Validation & Testing
10. Troubleshooting

---

Overview

What is Migration?

Migration is the process of exporting SharePoint data from a real SharePoint server and importing it into the Mock Server's file-based storage (@MockData folder).

When to Migrate

• Testing: Use real production data for realistic testing
• Development: Develop against actual data structures
• Demos: Create demo environments with representative data
• Learning: Understand SharePoint data models

What Gets Migrated

✅ Fully Supported: - Site collections and webs - Lists and libraries - List items and documents - File content and metadata - Users and groups - Permissions and role assignments - Content types - Site columns (fields) - Views - Taxonomy (term stores, term sets, terms)

⚠️ Partially Supported: - User profiles (basic properties only) - Custom actions (metadata only) - Event receivers (metadata only)

❌ Not Supported: - Workflow instances (deprecated) - Social data (deprecated) - App packages - Enterprise service application data

---

Prerequisites

Source SharePoint Server

• Supported Versions:
• SharePoint Server Subscription Edition
• SharePoint Server 2019
• SharePoint Server 2016

• SharePoint Online (with limitations)

• Required Permissions:

• Site Collection Administrator (minimum)

• Farm Administrator (for full migration)

• Test Server Example:

• URL: https://intranet21.sharepoint.farm
• Credentials: SHAREPOINT\administrator / Passw0rd

Tools Required

1. PnP PowerShell (Recommended)

   Install-Module -Name "PnP.PowerShell" -Force
   

2. SharePoint Client SDK (Alternative)

3. Install via NuGet: Microsoft.SharePointOnline.CSOM

4. Mock Server

5. Ensure Mock Server is installed and @MockData folder is accessible

---

Cesivi.MigrationTool (Recommended)

Overview

The Cesivi.MigrationTool is a dedicated CLI tool for exporting SharePoint data to the Mock Server's @MockData folder. It provides automated, robust data migration with real-time progress reporting.

Features

✅ Automated Export: Complete site export with one command ✅ Real-time Progress: Beautiful CLI progress bars and status updates ✅ Error Handling: Robust error handling with detailed logging ✅ Flexible Configuration: Multiple export strategies and filters ✅ File Support: Downloads document library files with metadata ✅ Users & Groups: Exports site users and group memberships

Quick Start

# Navigate to migration tool
cd Cesivi.MigrationTool

# Run export (test server example)
dotnet run -- export \
  --url "https://intranet21.sharepoint.farm" \
  --username "SHAREPOINT\administrator" \
  --password "Passw0rd" \
  --target "@MockData"

Command-Line Options

Option	Description	Required	Default
--url <url>	SharePoint site URL	✅ Yes	-
--username <user>	Username for authentication	✅ Yes	-
--password <pwd>	Password for authentication	✅ Yes	-
--target <path>	Target export folder	❌ No	@MockData
--strategy <type>	Export strategy (Full/Selective/Incremental)	❌ No	Full
--lists <names>	Comma-separated list names to export	❌ No	All lists
--include-hidden	Include hidden lists	❌ No	false
--no-files	Skip file content from libraries	❌ No	Include files
--max-items <n>	Maximum items per list (0 = unlimited)	❌ No	0
--max-file-size <mb>	Maximum file size in MB (0 = unlimited)	❌ No	0

Usage Examples

Basic Full Export

dotnet run -- export \
  --url "https://intranet21.sharepoint.farm" \
  --username "SHAREPOINT\administrator" \
  --password "Passw0rd"

Selective Export (Specific Lists)

dotnet run -- export \
  --url "https://intranet21.sharepoint.farm" \
  --username "SHAREPOINT\administrator" \
  --password "Passw0rd" \
  --strategy Selective \
  --lists "Documents,Tasks,Calendar"

Export with Limits

dotnet run -- export \
  --url "https://intranet21.sharepoint.farm" \
  --username "SHAREPOINT\administrator" \
  --password "Passw0rd" \
  --max-items 1000 \
  --max-file-size 50

Export to RAM Disk (Fast Testing)

dotnet run -- export \
  --url "https://intranet21.sharepoint.farm" \
  --username "SHAREPOINT\administrator" \
  --password "Passw0rd" \
  --target "R:\MockData"

What Gets Exported

The tool exports: - ✅ Web metadata (title, URL, language, template) - ✅ Lists and libraries (metadata, settings) - ✅ List items (all fields and values) - ✅ Document library files (content + metadata) - ✅ Users (site users with properties) - ✅ Groups (groups with memberships) - ✅ Content types (definitions without hidden types)

Output Structure

@MockData/
├── sites/
│   └── site1/
│       └── webs/
│           └── web1/
│               ├── web.json                 # Web metadata
│               ├── lists/
│               │   ├── Documents/
│               │   │   ├── list.json        # List metadata
│               │   │   ├── items/
│               │   │   │   └── item_*.json  # List items
│               │   │   └── files/
│               │   │       ├── file.pdf     # File content
│               │   │       └── file.pdf.metadata.json
│               │   └── Tasks/
│               │       └── ...
│               ├── users/
│               │   └── user_*.json
│               ├── groups/
│               │   └── group_*.json
│               └── contenttypes/
│                   └── *.json

Progress Reporting

The tool provides real-time progress with: - Connection status - Export progress bars - Item/file counters - Summary statistics - Error reporting

Error Handling

• Continues on individual item failures
• Reports all errors at completion
• Shows first 10 errors in summary
• Logs detailed error information

Performance Tips

1. Use RAM Disk: Export to RAM disk (e.g., R:\MockData) for faster performance
2. Limit Items: Use --max-items for large lists
3. Skip Files: Use --no-files for metadata-only export
4. Selective Export: Use --lists to export specific lists only

Troubleshooting

Connection Issues: - Verify SharePoint URL is accessible - Check username/password are correct - Ensure user has Site Collection Administrator rights

Large Export Issues: - Use --max-items to limit items - Use --max-file-size to skip large files - Export to RAM disk for better performance - Use selective strategy for specific lists only

---

Migration Strategies

Strategy 1: Full Site Collection Export (Recommended)

Best For: Complete site replication, comprehensive testing

Steps: 1. Export entire site collection structure 2. Export all lists and libraries 3. Export all documents and list items 4. Export users, groups, and permissions 5. Export taxonomy and content types

Time Estimate: 2-4 hours for typical site

---

Strategy 2: Selective Export

Best For: Specific feature testing, smaller data sets

Steps: 1. Identify required lists/libraries 2. Export only specific lists 3. Export minimal user set 4. Skip unnecessary data

Time Estimate: 30 minutes - 2 hours

---

Strategy 3: Incremental Export

Best For: Ongoing development, data updates

Steps: 1. Perform initial full export 2. Export only changes/new data 3. Use change tracking to identify deltas 4. Merge with existing Mock Data

Time Estimate: 15-30 minutes per update

---

Manual Export Methods

Method 1: PowerShell with CSOM

# Connect to SharePoint
$siteUrl = "https://intranet21.sharepoint.farm"
$username = "SHAREPOINT\administrator"
$password = ConvertTo-SecureString "Passw0rd" -AsPlainText -Force
$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $password)

Add-Type -Path "C:\Program Files\SharePoint Client Components\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\SharePoint Client Components\Microsoft.SharePoint.Client.Runtime.dll"

$context = New-Object Microsoft.SharePoint.Client.ClientContext($siteUrl)
$context.Credentials = $credentials

# Export web metadata
$web = $context.Web
$context.Load($web)
$context.ExecuteQuery()

$webData = @{
    Title = $web.Title
    Description = $web.Description
    Url = $web.Url
    ServerRelativeUrl = $web.ServerRelativeUrl
    Created = $web.Created
    Language = $web.Language
}

# Save to JSON
$webData | ConvertTo-Json -Depth 10 | Out-File "@MockData\sites\site1\webs\web1\web.json"

Method 2: REST API with Invoke-RestMethod

# Connect to SharePoint
$siteUrl = "https://intranet21.sharepoint.farm"
$username = "SHAREPOINT\administrator"
$password = "Passw0rd"

# Create credential
$securePassword = ConvertTo-SecureString $password -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential($username, $securePassword)

# Get web metadata via REST
$webEndpoint = "$siteUrl/_api/web"
$webData = Invoke-RestMethod -Uri $webEndpoint -Credential $credential -Method Get

# Save to JSON
$webData | ConvertTo-Json -Depth 10 | Out-File "@MockData\sites\site1\webs\web1\web.json"

---

PnP PowerShell Export

Full Site Collection Export Script

# Configuration
$sourceUrl = "https://intranet21.sharepoint.farm"
$targetPath = "C:\Source\_AI\Cesivi\@MockData"
$username = "SHAREPOINT\administrator"
$password = ConvertTo-SecureString "Passw0rd" -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential($username, $password)

# Connect to source SharePoint
Connect-PnPOnline -Url $sourceUrl -Credentials $credential

# 1. Export Web Metadata
Write-Host "Exporting web metadata..." -ForegroundColor Green
$web = Get-PnPWeb
$webPath = "$targetPath\sites\site1\webs\web1"
New-Item -ItemType Directory -Path $webPath -Force | Out-Null

$webMetadata = @{
    Id = $web.Id
    Title = $web.Title
    Description = $web.Description
    Url = $web.Url
    ServerRelativeUrl = $web.ServerRelativeUrl
    Created = $web.Created
    Language = $web.Language
    WebTemplate = $web.WebTemplate
}
$webMetadata | ConvertTo-Json -Depth 10 | Out-File "$webPath\web.json"

# 2. Export Lists
Write-Host "Exporting lists..." -ForegroundColor Green
$lists = Get-PnPList
$listsPath = "$webPath\lists"
New-Item -ItemType Directory -Path $listsPath -Force | Out-Null

foreach ($list in $lists) {
    if ($list.Hidden -eq $false) {
        Write-Host "  - Exporting list: $($list.Title)" -ForegroundColor Cyan

        $listMetadata = @{
            Id = $list.Id
            Title = $list.Title
            Description = $list.Description
            BaseTemplate = $list.BaseTemplate
            ItemCount = $list.ItemCount
            Created = $list.Created
            LastItemModifiedDate = $list.LastItemModifiedDate
        }

        $listPath = "$listsPath\$($list.Title)"
        New-Item -ItemType Directory -Path $listPath -Force | Out-Null
        $listMetadata | ConvertTo-Json -Depth 10 | Out-File "$listPath\list.json"

        # Export list items
        $items = Get-PnPListItem -List $list.Title -PageSize 1000
        $itemsPath = "$listPath\items"
        New-Item -ItemType Directory -Path $itemsPath -Force | Out-Null

        foreach ($item in $items) {
            $itemData = @{
                Id = $item.Id
                Title = $item["Title"]
                Created = $item["Created"]
                Modified = $item["Modified"]
                Author = $item["Author"].LookupValue
                Editor = $item["Editor"].LookupValue
                Fields = @{}
            }

            # Export all fields
            foreach ($field in $item.FieldValues.Keys) {
                $itemData.Fields[$field] = $item[$field]
            }

            $itemData | ConvertTo-Json -Depth 10 | Out-File "$itemsPath\item_$($item.Id).json"
        }
    }
}

# 3. Export Users and Groups
Write-Host "Exporting users and groups..." -ForegroundColor Green
$usersPath = "$webPath\users"
$groupsPath = "$webPath\groups"
New-Item -ItemType Directory -Path $usersPath -Force | Out-Null
New-Item -ItemType Directory -Path $groupsPath -Force | Out-Null

$users = Get-PnPUser
foreach ($user in $users) {
    $userData = @{
        Id = $user.Id
        LoginName = $user.LoginName
        Title = $user.Title
        Email = $user.Email
        IsSiteAdmin = $user.IsSiteAdmin
    }
    $userData | ConvertTo-Json -Depth 10 | Out-File "$usersPath\user_$($user.Id).json"
}

$groups = Get-PnPGroup
foreach ($group in $groups) {
    $groupData = @{
        Id = $group.Id
        Title = $group.Title
        Description = $group.Description
        Owner = $group.Owner
        Users = @()
    }

    $groupUsers = Get-PnPGroupMember -Identity $group.Title
    foreach ($user in $groupUsers) {
        $groupData.Users += @{
            Id = $user.Id
            LoginName = $user.LoginName
            Title = $user.Title
        }
    }

    $groupData | ConvertTo-Json -Depth 10 | Out-File "$groupsPath\group_$($group.Id).json"
}

# 4. Export Content Types
Write-Host "Exporting content types..." -ForegroundColor Green
$contentTypesPath = "$webPath\contenttypes"
New-Item -ItemType Directory -Path $contentTypesPath -Force | Out-Null

$contentTypes = Get-PnPContentType
foreach ($ct in $contentTypes) {
    $ctData = @{
        Id = $ct.StringId
        Name = $ct.Name
        Description = $ct.Description
        Group = $ct.Group
        Fields = @()
    }

    # Export fields
    $fields = Get-PnPField -ContentType $ct.Name -ErrorAction SilentlyContinue
    foreach ($field in $fields) {
        $ctData.Fields += @{
            InternalName = $field.InternalName
            Title = $field.Title
            FieldTypeKind = $field.TypeAsString
        }
    }

    $ctData | ConvertTo-Json -Depth 10 | Out-File "$contentTypesPath\$($ct.StringId).json"
}

# 5. Export Taxonomy
Write-Host "Exporting taxonomy..." -ForegroundColor Green
$taxonomyPath = "$webPath\taxonomy"
New-Item -ItemType Directory -Path $taxonomyPath -Force | Out-Null

$termStores = Get-PnPTermStore
foreach ($store in $termStores) {
    $storeData = @{
        Id = $store.Id
        Name = $store.Name
        DefaultLanguage = $store.DefaultLanguage
        TermSets = @()
    }

    $termGroups = Get-PnPTermGroup
    foreach ($group in $termGroups) {
        $termSets = Get-PnPTermSet -TermGroup $group.Name
        foreach ($termSet in $termSets) {
            $tsData = @{
                Id = $termSet.Id
                Name = $termSet.Name
                Description = $termSet.Description
                Terms = @()
            }

            $terms = Get-PnPTerm -TermSet $termSet.Name -TermGroup $group.Name
            foreach ($term in $terms) {
                $tsData.Terms += @{
                    Id = $term.Id
                    Name = $term.Name
                    Description = $term.Description
                }
            }

            $storeData.TermSets += $tsData
        }
    }

    $storeData | ConvertTo-Json -Depth 10 | Out-File "$taxonomyPath\termstore.json"
}

# 6. Export Documents (for libraries)
Write-Host "Exporting documents..." -ForegroundColor Green
$documentLibraries = $lists | Where-Object { $_.BaseTemplate -eq 101 }

foreach ($library in $documentLibraries) {
    Write-Host "  - Exporting library: $($library.Title)" -ForegroundColor Cyan
    $libraryPath = "$listsPath\$($library.Title)\files"
    New-Item -ItemType Directory -Path $libraryPath -Force | Out-Null

    $files = Get-PnPFolderItem -FolderSiteRelativeUrl $library.RootFolder.ServerRelativeUrl -ItemType File
    foreach ($file in $files) {
        Write-Host "    - Downloading: $($file.Name)" -ForegroundColor Gray
        $filePath = "$libraryPath\$($file.Name)"
        Get-PnPFile -Url $file.ServerRelativeUrl -Path $libraryPath -Filename $file.Name -AsFile -Force

        # Export file metadata
        $fileMetadata = @{
            Name = $file.Name
            ServerRelativeUrl = $file.ServerRelativeUrl
            Length = $file.Length
            TimeCreated = $file.TimeCreated
            TimeLastModified = $file.TimeLastModified
            Title = $file.Title
        }
        $fileMetadata | ConvertTo-Json -Depth 10 | Out-File "$libraryPath\$($file.Name).metadata.json"
    }
}

Write-Host "`nExport completed successfully!" -ForegroundColor Green
Write-Host "Data saved to: $targetPath" -ForegroundColor Green

# Disconnect
Disconnect-PnPOnline

---

Mock Data Structure

Directory Layout

@MockData/
├── sites/
│   └── site1/                      # Site collection
│       ├── site.json               # Site collection metadata
│       └── webs/
│           └── web1/               # Web (site)
│               ├── web.json        # Web metadata
│               ├── lists/
│               │   ├── Documents/
│               │   │   ├── list.json
│               │   │   ├── items/
│               │   │   │   ├── item_1.json
│               │   │   │   └── item_2.json
│               │   │   └── files/
│               │   │       ├── document1.pdf
│               │   │       ├── document1.pdf.metadata.json
│               │   │       └── document2.docx
│               │   └── CustomList/
│               │       ├── list.json
│               │       └── items/
│               ├── users/
│               │   ├── user_1.json
│               │   └── user_2.json
│               ├── groups/
│               │   ├── group_1.json
│               │   └── group_2.json
│               ├── contenttypes/
│               │   └── 0x0101.json
│               ├── fields/
│               │   ├── field_1.json
│               │   └── field_2.json
│               └── taxonomy/
│                   └── termstore.json

JSON Format Examples

web.json

{
  "Id": "12345678-1234-1234-1234-123456789012",
  "Title": "Team Site",
  "Description": "Collaborative team workspace",
  "Url": "http://mocksharepoint.local",
  "ServerRelativeUrl": "/",
  "Created": "2025-01-01T00:00:00Z",
  "Language": 1033,
  "WebTemplate": "STS",
  "Configuration": 0
}

list.json

{
  "Id": "87654321-4321-4321-4321-210987654321",
  "Title": "Documents",
  "Description": "Shared documents",
  "BaseTemplate": 101,
  "ItemCount": 25,
  "Created": "2025-01-01T00:00:00Z",
  "LastItemModifiedDate": "2025-10-06T00:00:00Z",
  "EnableVersioning": true,
  "EnableFolderCreation": true
}

item_1.json

{
  "Id": 1,
  "Title": "Project Proposal",
  "Created": "2025-01-15T10:30:00Z",
  "Modified": "2025-01-20T14:45:00Z",
  "Author": "John Doe",
  "Editor": "Jane Smith",
  "Fields": {
    "Status": "Approved",
    "Priority": "High",
    "DueDate": "2025-02-01T00:00:00Z",
    "AssignedTo": "John Doe"
  }
}

---

Import to Mock Server

Automatic Import

The Mock Server automatically loads data from @MockData folder on startup.

Steps: 1. Place exported JSON files in @MockData folder following structure above 2. Restart Mock Server 3. Verify data loaded via logs

Manual Import via API

You can also import data via REST/SOAP APIs:

# Connect to Mock Server
Connect-PnPOnline -Url "http://mocksharepoint.local"

# Create list
New-PnPList -Title "Imported List" -Template GenericList

# Add items
Add-PnPListItem -List "Imported List" -Values @{
    "Title" = "Item 1"
    "Status" = "Active"
}

Validation

After import, verify data:

# Verify web
Get-PnPWeb | Select-Object Title, Url

# Verify lists
Get-PnPList | Select-Object Title, ItemCount

# Verify users
Get-PnPUser | Select-Object Title, Email

# Verify groups
Get-PnPGroup | Select-Object Title, Description

---

Validation & Testing

Validation Checklist

• [ ] Web metadata correct (title, URL, language)
• [ ] All expected lists present
• [ ] List item counts match
• [ ] File content accessible
• [ ] Users and groups exist
• [ ] Permissions applied correctly
• [ ] Content types available
• [ ] Taxonomy terms present
• [ ] Search returns results

Test Scripts

# Test 1: Verify list access
$lists = Get-PnPList
Write-Host "Lists found: $($lists.Count)"

# Test 2: Verify items
$items = Get-PnPListItem -List "Documents"
Write-Host "Documents count: $($items.Count)"

# Test 3: Verify file download
$file = Get-PnPFile -Url "/Shared Documents/document1.pdf" -AsFile
Write-Host "File size: $($file.Length) bytes"

# Test 4: Verify search
$results = Submit-PnPSearchQuery -Query "proposal"
Write-Host "Search results: $($results.ResultRows.Count)"

---

Troubleshooting

Issue: JSON Parsing Errors

Symptom: Mock Server fails to load data, JSON parse errors in logs

Solutions: 1. Validate JSON syntax: Test-Json -Path "web.json" 2. Check for special characters (escape them) 3. Ensure UTF-8 encoding without BOM 4. Verify JSON depth not exceeding limits

Issue: Missing Data After Import

Symptom: Some data not visible in Mock Server

Solutions: 1. Check file paths match expected structure 2. Verify GUIDs are valid format 3. Ensure required fields present (Id, Title, etc.) 4. Check Mock Server logs for warnings

Issue: Permission Errors

Symptom: Access denied when querying data

Solutions: 1. Mock Server accepts all credentials by default 2. Verify authentication middleware configured 3. Check user exists in exported user list

Issue: File Content Missing

Symptom: File metadata present but content missing

Solutions: 1. Verify binary files copied to files/ folder 2. Check file permissions (read access required) 3. Ensure file path in metadata matches actual file

Issue: Taxonomy Terms Not Loading

Symptom: Term store or terms not available

Solutions: 1. Check termstore.json structure 2. Verify term GUIDs are valid 3. Ensure hierarchy (Store → Group → TermSet → Term) correct

---

Best Practices

1. Start Small

• Export single list first
• Validate import works
• Gradually expand to full site

2. Document Mappings

• Keep log of GUID mappings (source → mock)
• Document any data transformations
• Track exported vs imported item counts

3. Version Control Mock Data

• Commit @MockData to Git (if appropriate)
• Tag exports with source date
• Use .gitignore for large files

4. Automate Exports

• Schedule regular exports for ongoing development
• Use incremental exports to reduce time
• Script validation tests

5. Sanitize Data

• Remove sensitive information
• Anonymize user data if needed
• Reduce file sizes for performance

---

Additional Resources

• PnP PowerShell Documentation: https://pnp.github.io/powershell/
• SharePoint REST API: https://docs.microsoft.com/sharepoint/dev/sp-add-ins/get-to-know-the-sharepoint-rest-service
• Mock Server Architecture: See _docs/ARCHITECTURE.md
• Testing Guide: See _docs/TESTING.md

---

Support

For issues or questions: 1. Check _docs/TROUBLESHOOTING.md 2. Review Mock Server logs 3. Consult _docs/ARCHITECTURE.md for data structure 4. Open issue on GitHub (if applicable)