Cesivi Server - Migration Guide

Home → Documentation → Usage → Migration

Guide for importing data from real SharePoint environments to Cesivi Server.

Table of Contents

1. Overview
2. Quick Start
3. Using the Migration Tool
4. Manual Export Methods
5. Migration Strategies
6. Validation
7. Troubleshooting

---

Overview

What is Migration?

Migration is the process of exporting data from a real SharePoint server and importing it into Cesivi Server's file-based storage.

When to Migrate

• Testing - Use real production data for realistic testing
• Development - Develop against actual data structures
• Demos - Create demo environments with representative data
• Learning - Understand SharePoint data models

What Gets Migrated

Fully Supported: - Site collections and webs - Lists and libraries - List items and documents - File content and metadata - Users and groups - Permissions and role assignments - Content types - Site columns (fields)

Partially Supported: - User profiles (basic properties only) - Custom actions (metadata only)

Not Supported: - Workflow instances (deprecated) - Social data (deprecated) - App packages

---

Quick Start

Prerequisites

Source SharePoint Server: - SharePoint Online, 2016, 2019, or Subscription Edition - Site Collection Administrator permissions

Tools Required: - Migration tool (included) OR - PnP PowerShell module

5-Minute Migration

# 1. Navigate to migration tool
cd Cesivi.MigrationTool

# 2. Run migration (replace with your SharePoint URL and credentials)
dotnet run -- export \
  --url "https://your-sharepoint.com/sites/team" \
  --username "domain\username" \
  --password "YourPassword" \
  --target "../@MockData"

# 3. Start Mock Server
cd ../Cesivi.Server
dotnet run

# 4. Verify data loaded

---

Using the Migration Tool

Overview

The Cesivi.MigrationTool is a CLI tool that automates the export of SharePoint data to the Mock Server's storage format.

Features: - Automated complete site export - Real-time progress reporting - Robust error handling - Flexible filtering options - File content download

Command-Line Options

Option	Description	Required	Default
--url <url>	SharePoint site URL	Yes	-
--username <user>	Username for authentication	Yes	-
--password <pwd>	Password for authentication	Yes	-
--target <path>	Target export folder	No	@MockData
--strategy <type>	Export strategy (Full/Selective)	No	Full
--lists <names>	Comma-separated list names	No	All lists
--include-hidden	Include hidden lists	No	false
--no-files	Skip file content	No	Include files
--max-items <n>	Max items per list (0=unlimited)	No	0
--max-file-size <mb>	Max file size in MB	No	0

Basic Examples

Full Site Export:

dotnet run -- export \
  --url "https://intranet.company.com/sites/team" \
  --username "DOMAIN\admin" \
  --password "P@ssw0rd"

Selective Export (Specific Lists):

dotnet run -- export \
  --url "https://intranet.company.com/sites/team" \
  --username "DOMAIN\admin" \
  --password "P@ssw0rd" \
  --strategy Selective \
  --lists "Documents,Tasks,Calendar"

Export with Limits (for testing):

dotnet run -- export \
  --url "https://intranet.company.com/sites/team" \
  --username "DOMAIN\admin" \
  --password "P@ssw0rd" \
  --max-items 100 \
  --max-file-size 10

Metadata-Only Export (no files):

dotnet run -- export \
  --url "https://intranet.company.com/sites/team" \
  --username "DOMAIN\admin" \
  --password "P@ssw0rd" \
  --no-files

Output Structure

The tool exports data to the following structure:

@MockData/
├── sites/
│   └── site1/
│       └── webs/
│           └── web1/
│               ├── web.json              # Web metadata
│               ├── lists/
│               │   ├── Documents/
│               │   │   ├── list.json     # List metadata
│               │   │   ├── items/
│               │   │   │   ├── item_1.json
│               │   │   │   └── item_2.json
│               │   │   └── files/
│               │   │       ├── doc.pdf
│               │   │       └── doc.pdf.metadata.json
│               │   └── Tasks/
│               │       ├── list.json
│               │       └── items/
│               ├── users/
│               │   └── user_*.json
│               ├── groups/
│               │   └── group_*.json
│               └── contenttypes/
│                   └── *.json

Progress Reporting

The tool shows real-time progress:

Connecting to SharePoint...
✓ Connected to https://intranet.company.com/sites/team

Exporting web metadata...
✓ Web: Team Site

Exporting lists...
[1/5] Documents................ [████████████] 100% (150/150 items)
[2/5] Tasks.................... [████████████] 100% (45/45 items)
[3/5] Calendar................ [████████████] 100% (20/20 items)
[4/5] Contacts................ [████████████] 100% (80/80 items)
[5/5] Links................... [████████████] 100% (12/12 items)

Exporting users and groups...
✓ 25 users exported
✓ 8 groups exported

Export Summary:
- Lists: 5
- Items: 307
- Files: 150
- Users: 25
- Groups: 8
- Duration: 2m 34s
✓ Export completed successfully!

Performance Tips

1. Use RAM Disk - Export to RAM disk for faster I/O

   --target "R:\MockData"
   

2. Limit Items - For large lists, use --max-items

   --max-items 1000
   

3. Skip Files - For structure-only exports

   --no-files
   

4. Selective Export - Export only needed lists

   --strategy Selective --lists "Documents,Tasks"
   

---

Manual Export Methods

Method 1: PnP PowerShell Export

Full Site Export Script:

# Configuration
$sourceUrl = "https://intranet.company.com/sites/team"
$targetPath = "C:\Projects\Cesivi\@MockData"
$username = "DOMAIN\admin"
$password = ConvertTo-SecureString "P@ssw0rd" -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential($username, $password)

# Connect
Connect-PnPOnline -Url $sourceUrl -Credentials $credential

# Export web
Write-Host "Exporting web..." -ForegroundColor Green
$web = Get-PnPWeb
$webPath = "$targetPath\sites\site1\webs\web1"
New-Item -ItemType Directory -Path $webPath -Force | Out-Null

$webData = @{
    Id = $web.Id
    Title = $web.Title
    Description = $web.Description
    Url = $web.Url
    ServerRelativeUrl = $web.ServerRelativeUrl
    Created = $web.Created
    Language = $web.Language
}
$webData | ConvertTo-Json -Depth 10 | Out-File "$webPath\web.json"

# Export lists
Write-Host "Exporting lists..." -ForegroundColor Green
$lists = Get-PnPList | Where-Object { -not $_.Hidden }
$listsPath = "$webPath\lists"
New-Item -ItemType Directory -Path $listsPath -Force | Out-Null

foreach ($list in $lists) {
    Write-Host "  - $($list.Title)" -ForegroundColor Cyan

    $listPath = "$listsPath\$($list.Title)"
    New-Item -ItemType Directory -Path $listPath -Force | Out-Null

    # List metadata
    $listData = @{
        Id = $list.Id
        Title = $list.Title
        Description = $list.Description
        BaseTemplate = $list.BaseTemplate
        ItemCount = $list.ItemCount
    }
    $listData | ConvertTo-Json -Depth 10 | Out-File "$listPath\list.json"

    # Export items
    $items = Get-PnPListItem -List $list.Title -PageSize 1000
    $itemsPath = "$listPath\items"
    New-Item -ItemType Directory -Path $itemsPath -Force | Out-Null

    foreach ($item in $items) {
        $itemData = @{
            Id = $item.Id
            Title = $item["Title"]
            Created = $item["Created"]
            Modified = $item["Modified"]
            Fields = @{}
        }

        foreach ($field in $item.FieldValues.Keys) {
            $itemData.Fields[$field] = $item[$field]
        }

        $itemData | ConvertTo-Json -Depth 10 | Out-File "$itemsPath\item_$($item.Id).json"
    }
}

# Export users
Write-Host "Exporting users..." -ForegroundColor Green
$usersPath = "$webPath\users"
New-Item -ItemType Directory -Path $usersPath -Force | Out-Null

$users = Get-PnPUser
foreach ($user in $users) {
    $userData = @{
        Id = $user.Id
        LoginName = $user.LoginName
        Title = $user.Title
        Email = $user.Email
    }
    $userData | ConvertTo-Json -Depth 10 | Out-File "$usersPath\user_$($user.Id).json"
}

# Export groups
Write-Host "Exporting groups..." -ForegroundColor Green
$groupsPath = "$webPath\groups"
New-Item -ItemType Directory -Path $groupsPath -Force | Out-Null

$groups = Get-PnPGroup
foreach ($group in $groups) {
    $groupData = @{
        Id = $group.Id
        Title = $group.Title
        Description = $group.Description
        Users = @()
    }

    $members = Get-PnPGroupMember -Identity $group.Title
    foreach ($member in $members) {
        $groupData.Users += @{
            Id = $member.Id
            LoginName = $member.LoginName
            Title = $member.Title
        }
    }

    $groupData | ConvertTo-Json -Depth 10 | Out-File "$groupsPath\group_$($group.Id).json"
}

Write-Host "`nExport completed!" -ForegroundColor Green
Disconnect-PnPOnline

Method 2: CSOM Export (C#)

using Microsoft.SharePoint.Client;

var siteUrl = "https://intranet.company.com/sites/team";
var username = "DOMAIN\\admin";
var password = "P@ssw0rd";

using (var context = new ClientContext(siteUrl))
{
    var securePassword = new SecureString();
    foreach (char c in password) securePassword.AppendChar(c);
    context.Credentials = new SharePointOnlineCredentials(username, securePassword);

    // Get web
    var web = context.Web;
    context.Load(web);
    context.ExecuteQuery();

    var webData = new
    {
        Title = web.Title,
        Url = web.Url,
        ServerRelativeUrl = web.ServerRelativeUrl,
        Created = web.Created,
        Language = web.Language
    };

    // Save to JSON
    var json = JsonConvert.SerializeObject(webData, Formatting.Indented);
    File.WriteAllText(@"@MockData\sites\site1\webs\web1\web.json", json);

    // Export lists (similar pattern)
    var lists = context.Web.Lists;
    context.Load(lists);
    context.ExecuteQuery();

    foreach (var list in lists)
    {
        if (!list.Hidden)
        {
            // Export list data...
        }
    }
}

---

Migration Strategies

Strategy 1: Full Site Export

Best For: Complete site replication, comprehensive testing

Steps: 1. Export entire site structure 2. Export all lists and libraries 3. Export all documents and items 4. Export users, groups, permissions

Pros: - Complete data fidelity - All relationships preserved - Realistic testing environment

Cons: - Longer export time - Larger storage footprint - May include unnecessary data

Strategy 2: Selective Export

Best For: Specific feature testing, smaller datasets

Steps: 1. Identify required lists/libraries 2. Export only specific lists 3. Export minimal user set

Pros: - Faster export - Smaller storage - Focused testing

Cons: - May miss dependencies - Less realistic

Strategy 3: Incremental Export

Best For: Ongoing development, data updates

Steps: 1. Perform initial full export 2. Export only changes/new data 3. Merge with existing data

Pros: - Quick updates - Keeps data current

Cons: - More complex process - Requires change tracking

---

Validation

Validate Export Success

After migration, verify data loaded correctly:

Check Web:

Connect-PnPOnline -Url "http://localhost:5000" -UseWebLogin
Get-PnPWeb | Select-Object Title, Url, Language

Check Lists:

Get-PnPList | Select-Object Title, ItemCount, BaseTemplate

Check Items:

Get-PnPListItem -List "Documents" | Select-Object Id, Title

Check Users:

Get-PnPUser | Select-Object Title, Email

Check Groups:

Get-PnPGroup | Select-Object Title, Description

Validation Checklist

• [ ] Web metadata correct (title, URL, language)
• [ ] All expected lists present
• [ ] List item counts match source
• [ ] File content accessible
• [ ] Users and groups exist
• [ ] Permissions applied correctly

---

Troubleshooting

Connection Issues

Problem: Cannot connect to SharePoint

Solutions: 1. Verify SharePoint URL is correct and accessible 2. Check username/password are correct 3. Ensure user has Site Collection Administrator rights 4. For SharePoint Online, use app-only credentials if MFA enabled

SharePoint Online with MFA:

# Use app-only authentication
dotnet run -- export \
  --url "https://tenant.sharepoint.com/sites/team" \
  --clientid "your-app-id" \
  --clientsecret "your-app-secret"

Large Export Issues

Problem: Export times out or fails on large sites

Solutions:

1. Use item limits:

   --max-items 1000
   

2. Skip large files:

   --max-file-size 50
   

3. Export to RAM disk:

   --target "R:\MockData"
   

4. Use selective strategy:

   --strategy Selective --lists "Documents,Tasks"
   

Missing Data

Problem: Some data not visible in Mock Server

Solutions: 1. Check file paths match expected structure 2. Verify JSON files are valid (Test-Json in PowerShell) 3. Check Mock Server logs for errors 4. Ensure required fields present (Id, Title, etc.)

Permission Errors

Problem: Access denied when accessing migrated data

Solutions: 1. Mock Server accepts all credentials by default 2. Verify authentication is configured 3. Check user exists in exported user list

---

Related Documentation

• Basic Operations - Working with migrated data
• Export/Import Guide - Backup and restore
• API Reference - API documentation

---

Navigation: - ← Usage Guide - Documentation Home

Last Updated: 2025-11-15