Cesivi Migration Tool - Troubleshooting Guide¶

Home → Documentation → Troubleshooting → Migration Tool

Purpose: Diagnose and resolve common issues when exporting data from real SharePoint servers to Cesivi Server.

Related Documentation: - Migration Tool README - Usage guide and examples - Migration Tool Deployment - Deployment instructions - General Troubleshooting - Cesivi Server troubleshooting

Quick Diagnostics¶

Before troubleshooting, verify:

✅ Source SharePoint server is accessible
✅ Credentials have sufficient permissions
✅ Target directory (@MockData) is writable
✅ Network connectivity is stable
✅ Sufficient disk space available

Enable verbose logging for diagnostics:

dotnet run -- export \
  --url "https://your-sharepoint.site" \
  --username "user" \
  --password "pass" \
  --log-json \
  --log-text \
  --log-path "./logs"

Common Errors¶

1. Authentication Errors¶

Error: "Access denied" or "401 Unauthorized"¶

Symptoms:

[ERROR] Failed to connect to SharePoint: Access denied
[ERROR] The request failed with HTTP status 401: Unauthorized

Common Causes: - Invalid username or password - Account locked or disabled - Insufficient permissions on source site - Multi-factor authentication (MFA) enabled

Solutions:

For invalid credentials:

# Verify credentials work with direct login
# Use DOMAIN\username format for on-premises
dotnet run -- export \
  --url "https://sharepoint.farm" \
  --username "DOMAIN\administrator" \
  --password "YourPassword"

For SharePoint Online with MFA: - MFA not supported by basic auth - Use app-only authentication (requires app registration) - Alternative: Export from on-premises SharePoint

For permission issues: - User needs at least Read permissions on all lists/libraries - User needs Site Collection Administrator for full site export - Verify: Site Settings > Site Permissions > Check Permissions

Error: "403 Forbidden"¶

Symptoms:

[ERROR] Request forbidden: You don't have permission to access this resource

Causes: - User account has limited permissions - IP address blocked/restricted - Site requires additional authentication

Solutions: 1. Grant user Site Collection Administrator role temporarily 2. Use service account with broad permissions 3. Check firewall rules allow your IP address 4. Verify site is not in lockdown mode

2. Network and Timeout Errors¶

Error: "Request timeout" or "Connection failed"¶

Symptoms:

[ERROR] The operation has timed out
[ERROR] Unable to connect to the remote server
[ERROR] A connection attempt failed because the connected party did not properly respond

Common Causes: - Network latency or slow connection - Large files timing out - Firewall blocking requests - Server overloaded

Solutions:

For timeouts on large files:

# Limit file size to avoid timeouts
dotnet run -- export \
  --url "https://sharepoint.site" \
  --username "user" \
  --password "pass" \
  --max-file-size 50  # Skip files > 50MB

For network reliability:

# Export specific lists one at a time
dotnet run -- export \
  --url "https://sharepoint.site" \
  --username "user" \
  --password "pass" \
  --strategy Selective \
  --lists "Documents"  # Export one list

For connection issues: - Verify URL is accessible via browser - Check DNS resolution: nslookup sharepoint.site - Test connectivity: curl https://sharepoint.site - Verify proxy settings if using corporate network

Error: "SSL/TLS certificate errors"¶

Symptoms:

[ERROR] The SSL connection could not be established
[ERROR] The remote certificate is invalid according to the validation procedure

Causes: - Self-signed certificate on SharePoint server - Expired certificate - Certificate chain not trusted

Solutions:

For development/testing (UNSAFE for production): - Accept self-signed certificates at your own risk - For production, use proper SSL certificates

Proper solution: 1. Install SharePoint server's root CA certificate 2. Ensure certificate chain is valid 3. Update certificate if expired

3. Data Migration Errors¶

Error: "Unsupported content type" or "Feature not implemented"¶

Symptoms:

[WARN] Content type 'Document Set' not fully supported - basic fields only
[WARN] Feature 'Publishing Pages' not implemented in mock server

Causes: - SharePoint feature not implemented in mock server - Complex content types with custom properties - Third-party solutions or apps

Solutions:

Check supported features: - Standard lists and libraries: ✅ Supported - Document libraries with files: ✅ Supported - Custom lists: ✅ Supported - Content types: ✅ Basic support - Workflows: ❌ Not supported - Publishing pages: ⚠️ Limited support - InfoPath forms: ❌ Not supported

Workarounds:

# Skip unsupported features
# Migration tool automatically skips what it can't handle
# Check logs for warnings about skipped items

Known Limitations: - Workflows not migrated (design/instance) - Publishing infrastructure (pages/images) - Search configuration - InfoPath form templates - Apps/solutions - Sub-web recursive export (TODO in code)

Error: "Large file download failed"¶

Symptoms:

[ERROR] Failed to download file 'LargeVideo.mp4': Stream was too long
[ERROR] Out of memory exception processing file

Causes: - File exceeds memory limits - Network interruption during download - File locked on server

Solutions:

Set file size limits:

dotnet run -- export \
  --url "https://sharepoint.site" \
  --username "user" \
  --password "pass" \
  --max-file-size 100  # Skip files > 100MB

For very large libraries:

# Export without files first
dotnet run -- export \
  --url "https://sharepoint.site" \
  --username "user" \
  --password "pass" \
  --include-files false  # Metadata only

Then manually copy large files if needed.

Error: "Permission migration failed"¶

Symptoms:

[WARN] Failed to export permissions for item: Access denied
[WARN] Role assignment could not be read

Causes: - User lacks permission to read permission assignments - Broken inheritance issues - Missing role definitions

Solutions:

Grant required permissions: - User needs Manage Permissions at site collection level - Or use account with Full Control

Skip permissions if not needed:

dotnet run -- export \
  --url "https://sharepoint.site" \
  --username "user" \
  --password "pass" \
  --include-item-permissions false \
  --include-folder-permissions false \
  --include-file-permissions false

4. Storage and File System Errors¶

Error: "Path too long" or "File name invalid"¶

Symptoms:

[ERROR] The specified path, file name, or both are too long
[ERROR] Illegal characters in path

Causes: - Windows MAX_PATH limit (260 characters) - Special characters in SharePoint item names - Deep folder nesting

Solutions:

Use shorter target path:

# Use short base path
dotnet run -- export \
  --url "https://sharepoint.site" \
  --username "user" \
  --password "pass" \
  --target "C:\Mock"  # Instead of C:\VeryLongPathName\...

Enable long paths in Windows 10/11: 1. Run gpedit.msc (Local Group Policy Editor) 2. Navigate to: Computer Configuration > Administrative Templates > System > Filesystem 3. Enable: "Enable Win32 long paths" 4. Restart computer

Alternative - use mapped drive:

# Map short drive letter
net use M: C:\Your\Long\Path\MockData
dotnet run -- export --target "M:\"

Error: "Disk space insufficient"¶

Symptoms:

[ERROR] There is not enough space on the disk
[ERROR] Failed to write file: Disk full

Solutions:

Check available space:

# Windows
wmic logicaldisk get caption,freespace,size

# Estimate space needed:
# - SharePoint site size + 20% overhead
# - Check SharePoint site storage quota

Reduce export size:

# Skip large files
dotnet run -- export \
  --max-file-size 50 \
  --max-items 10000

# Or selective export
dotnet run -- export \
  --strategy Selective \
  --lists "ImportantDocuments"

Error: "File access denied" or "Cannot create directory"¶

Symptoms:

[ERROR] Access to the path 'C:\MockData\...' is denied
[ERROR] Could not find a part of the path

Causes: - Insufficient permissions on target directory - Directory doesn't exist - Antivirus blocking write operations - File/folder locked by another process

Solutions:

Run as Administrator (Windows)
Grant write permissions to target directory
Create directory if it doesn't exist:
```
mkdir C:\MockData
```
Temporarily disable antivirus for target directory

Check file locks:

# Windows - check what's using the file
# Use Process Explorer or Resource Monitor

Performance Issues¶

Slow Migration Speed¶

Symptoms: - Migration takes hours for small sites - Progress bar barely moves - High network latency

Diagnostic Steps:

Check network bandwidth:

# Test download speed from SharePoint
# Use browser to download sample file
# Measure time vs file size

Monitor resource usage:
CPU usage should be low-moderate
Memory usage increases with large libraries
Disk I/O during file downloads
Check SharePoint server load:
Peak usage times cause slowdowns
Contact SharePoint admin if server is slow

Optimization Tips:

Reduce item count:

# Export fewer items for testing
dotnet run -- export \
  --max-items 1000  # Limit total items

Skip version history:

# Version history adds significant overhead
dotnet run -- export \
  --include-version-history false  # Skip versions

Selective export:

# Export specific lists only
dotnet run -- export \
  --strategy Selective \
  --lists "CriticalData" "ImportantDocs"

Limit file size:

# Skip large media files
dotnet run -- export \
  --max-file-size 25  # Skip > 25MB files

Out of Memory Errors¶

Symptoms:

[ERROR] System.OutOfMemoryException
[ERROR] Insufficient memory to continue

Causes: - Very large files loaded into memory - Too many items processed at once - Memory leak (rare)

Solutions:

Reduce batch size:

# Process fewer items at once
dotnet run -- export \
  --max-items 5000  # Smaller batches

Limit file downloads:

dotnet run -- export \
  --max-file-size 50  # Skip large files

Increase available memory: - Close other applications - Use 64-bit process (default in .NET) - Add more RAM to machine if needed

Known Limitations¶

Sub-Web Recursive Export¶

Status: Not implemented (TODO in code - line 1267 DataExporter.cs)

Current Behavior: - Only exports the specified site - Sub-webs are not recursively exported

Workaround:

# Export each sub-web separately
dotnet run -- export --url "https://site.com/subweb1"
dotnet run -- export --url "https://site.com/subweb2"

SharePoint Version Compatibility¶

Fully Supported: - SharePoint Server 2019 - SharePoint Server Subscription Edition - SharePoint Server 2016 (mostly)

Limited Support: - SharePoint Server 2013 (older API) - SharePoint Online (authentication challenges)

Not Supported: - SharePoint 2010 and earlier

Feature Gaps¶

Not Migrated: - Workflows (SharePoint Designer/Power Automate) - Publishing infrastructure - Search configuration - Managed metadata service configuration - InfoPath form templates - Custom solutions/apps - Site collection features (some) - Audit logs - Usage analytics

Partially Migrated: - Content types (basic fields only) - Site columns (custom properties may be lost) - Permissions (complex inheritance may have issues)

Diagnostic Procedures¶

Step-by-Step Troubleshooting¶

1. Verify Source Connectivity

# Test basic connectivity
curl https://your-sharepoint.site

# Test authentication (should return 200 or redirect)
curl -u "DOMAIN\user:password" https://your-sharepoint.site/_api/web

2. Run Test Export (Small Scope)

# Start with minimal export
dotnet run -- export \
  --url "https://your-sharepoint.site" \
  --username "user" \
  --password "pass" \
  --strategy Selective \
  --lists "ListName" \
  --max-items 10 \
  --include-files false

3. Enable Comprehensive Logging

# Enable all logging
dotnet run -- export \
  --url "https://your-sharepoint.site" \
  --username "user" \
  --password "pass" \
  --log-json \
  --log-text \
  --log-path "./migration-logs"

4. Review Error Logs

Check log files in order: 1. Console output (real-time errors) 2. migration-log-{date}.json - Structured errors 3. migration-log-{date}.txt - Human-readable errors

Look for patterns: - Repeated errors on specific list/library - Authentication failures - Network timeouts - File access errors

5. Isolate the Problem

If specific list fails:

# Try different list
# Identify if issue is list-specific or global

If all lists fail:

# Check authentication
# Check network/firewall
# Check SharePoint server status

If only large files fail:

# Use --max-file-size to skip them
# Export metadata first, files separately

6. Check Target Environment

# Verify target directory
ls -la @MockData  # Linux/Mac
dir @MockData     # Windows

# Check disk space
df -h @MockData       # Linux/Mac
wmic logicaldisk ...  # Windows

# Check permissions
# Ensure write access to target

FAQ¶

Q1: How do I resume a failed migration?¶

A: The migration tool does not have built-in resume functionality. However:

Workaround for full export: 1. Check which lists were successfully exported 2. Use selective export for remaining lists:

dotnet run -- export \
  --strategy Selective \
  --lists "FailedList1" "FailedList2"

Workaround for large libraries: 1. Export in batches using --max-items 2. Manually combine exports 3. Or run full export again (overwrites existing)

Q2: How do I verify migration success?¶

Check migration logs:

# Review log files for errors
grep -i error ./migration-logs/migration-log-*.txt

# Count warnings vs errors
grep -c WARN ./migration-logs/migration-log-*.txt
grep -c ERROR ./migration-logs/migration-log-*.txt

Verify file structure:

# Check MockData structure
ls -R @MockData/

# Compare list counts
# Source: SharePoint list item count
# Target: Count JSON files in MockData

Test in Mock Server: 1. Start Cesivi Server 2. Connect with PnP PowerShell or CSOM 3. Verify lists exist: Get-PnPList 4. Check item counts: Get-PnPListItem -List "ListName" 5. Test file downloads from libraries

Q3: How do I handle very large SharePoint sites (>100GB)?¶

Strategy 1 - Selective Export:

# Export critical lists only
dotnet run -- export \
  --strategy Selective \
  --lists "CriticalList1" "CriticalList2" \
  --include-files true

Strategy 2 - Metadata First:

# Step 1: Export all metadata (fast)
dotnet run -- export \
  --include-files false

# Step 2: Selectively add file content
dotnet run -- export \
  --strategy Selective \
  --lists "ImportantDocs" \
  --include-files true

Strategy 3 - Size Limits:

# Skip large files
dotnet run -- export \
  --max-file-size 100 \
  --max-items 50000

Strategy 4 - Time-based:

# Export recent items only (if tool supports)
# Or export by list creation/modification date

Q4: Can I migrate SharePoint Online sites?¶

Challenges: - Modern authentication (OAuth) not supported by tool - Basic authentication deprecated by Microsoft - Multi-factor authentication (MFA) blocks basic auth

Workarounds:

Option 1 - App-Only Authentication: - Requires code changes to support app registration - Currently not implemented

Option 2 - Hybrid Approach: - Export from on-premises SharePoint - Or use SharePoint Online Management Shell - Then manually copy to mock server

Option 3 - Third-Party Tools: - Use SharePoint migration tools to export - Import exported data to mock server

Recommended: - Use on-premises SharePoint for migration source - Or wait for OAuth support in future tool version

Q5: Why are some items showing as warnings instead of errors?¶

Understanding Log Levels:

ERROR (Critical issues): - Migration cannot proceed - Data loss or corruption - Authentication/connection failures

WARN (Non-critical issues): - Item skipped but migration continues - Feature not supported - Partial data migrated - Performance degradation

INFO (Informational): - Normal progress updates - Successful operations - Configuration settings

Common Warnings:

[WARN] Content type 'X' not fully supported
[WARN] Skipping workflow 'Y' - not implemented
[WARN] Version history limited to N versions

Action Required: - Review warnings after migration - Determine if skipped items are critical - Manual intervention may be needed for important data

Q6: How do I troubleshoot specific error codes?¶

Enable verbose logging:

dotnet run -- export \
  --log-json \
  --log-text \
  --log-path "./logs"

Common HTTP Status Codes: - 401 Unauthorized - Authentication failure (see Authentication Errors above) - 403 Forbidden - Permission denied (see Authentication Errors above) - 404 Not Found - URL incorrect or item doesn't exist - 500 Internal Server Error - SharePoint server issue - 503 Service Unavailable - SharePoint overloaded or maintenance

Lookup Error in Logs:

# Search for specific error
grep "error-message" ./logs/migration-log-*.txt

# Find context around error
grep -A 5 -B 5 "error-message" ./logs/migration-log-*.txt

Review JSON logs for details: - Open migration-log-{date}.json - Find error entries - Check Exception, StackTrace, Message properties - Look for SourceContext to identify which component failed

Q7: What permissions does my account need?¶

Minimum Permissions (Metadata Only): - Read permission on all lists/libraries - View permission on site

Recommended Permissions (Full Export): - Site Collection Administrator role (temporary) - Or Full Control permission level

Specific Permission Requirements:

For lists/items: - Read List Items - View Application Pages

For files: - Open Items - View Items

For permissions: - Manage Permissions (to read permission assignments) - Enumerate Permissions

For users/groups: - Manage Web Site - Enumerate Permissions

Grant Temporary Admin:

Site Settings > Site Collection Administrators
Add user > Run migration > Remove user

Recursive Sub-Webs Export¶

Overview¶

Feature: Recursive site hierarchy export (added in S1.148)

The Migration Tool now supports full recursive export of site hierarchies. When --no-subwebs is NOT specified (default behavior), the tool will automatically export all sub-webs recursively, preserving the complete site structure.

Default Behavior¶

By default, IncludeSubWebs is true, meaning sub-webs are exported automatically:

# This will export the parent site AND all sub-webs recursively
dotnet run -- export \
  --url "https://sharepoint/sites/parent" \
  --username "user" \
  --password "pass"

Result: Exports /sites/parent, /sites/parent/subweb1, /sites/parent/subweb1/subsubweb, etc.

Disabling Recursive Export¶

To export only the specified site without sub-webs:

# Export only the parent site, skip all sub-webs
dotnet run -- export \
  --url "https://sharepoint/sites/parent" \
  --username "user" \
  --password "pass" \
  --no-subwebs

Folder Structure¶

Recursive export preserves site hierarchy in the folder structure:

@MockData/
└── sites/
    └── site1/
        └── webs/
            ├── web1/           # Root web
            │   ├── lists/
            │   ├── users/
            │   └── web.json
            └── webs/           # Sub-webs of web1
                ├── subweb1/
                │   ├── lists/
                │   └── webs/   # Sub-webs of subweb1
                │       └── subsubweb1/
                └── subweb2/

Circular Reference Protection¶

The tool automatically detects and prevents circular references:

[INFO] Exporting sub-web 'SubWeb1'...
[INFO] Exporting sub-web 'SubWeb2'...
[WARN] Skipping sub-web 'SubWeb1' - circular reference detected (URL: https://sharepoint/sites/parent/subweb1)

How it works: - Maintains a HashSet<string> of visited URLs - Uses case-insensitive URL comparison - Continues export of other sub-webs if circular reference found

Progress Reporting¶

During recursive export, progress updates show:

[INFO] Exporting web 1 of 1...
[INFO] Processing 5 lists...
[INFO] Exporting sub-web 'SubWeb1' (2 of 5)...
[INFO] Processing 3 lists...
[INFO] Exporting sub-web 'SubSubWeb1' (3 of 5)...

Performance Considerations¶

Large Site Hierarchies: - Each sub-web requires separate SharePoint connection - Export time scales linearly with number of sub-webs - Consider using --no-subwebs for faster partial exports

Bandwidth: - Recursive export downloads content from all sub-webs - Monitor network usage for large hierarchies - Use --max-file-size to limit bandwidth if needed

Common Scenarios¶

Scenario 1: Export entire site collection

# Export root site and all nested sub-webs (default)
dotnet run -- export \
  --url "https://sharepoint/sites/intranet" \
  --username "admin" \
  --password "pass"

Scenario 2: Export single site only

# Export specific site without its children
dotnet run -- export \
  --url "https://sharepoint/sites/intranet/team1" \
  --username "admin" \
  --password "pass" \
  --no-subwebs

Scenario 3: Export with file size limits

# Recursive export with 50MB file size limit
dotnet run -- export \
  --url "https://sharepoint/sites/intranet" \
  --username "admin" \
  --password "pass" \
  --max-file-size 50

Troubleshooting Recursive Export¶

Issue: Sub-web export fails with "Access Denied"

Solution: - Ensure user has permissions on ALL sub-webs - Some sub-webs may have unique permissions - Use Site Collection Administrator account - Check error log for specific sub-web URLs

Issue: Export takes very long time

Diagnosis:

# Enable logging to see progress
dotnet run -- export \
  --url "https://sharepoint/sites/intranet" \
  --username "admin" \
  --password "pass" \
  --log-text

Solutions: - Use --no-subwebs for faster partial export - Use --max-items to limit items per list - Export sub-webs individually if needed

Issue: "Skipping sub-web - circular reference detected"

Explanation: - This is expected behavior, not an error - SharePoint can have cross-references between webs - Tool automatically skips to prevent infinite loops - Other sub-webs continue to export normally

Best Practices¶

For Production Migrations: 1. Test with --no-subwebs first on root site 2. Verify single-site export works correctly 3. Then enable recursive export (default behavior) 4. Monitor first few sub-webs for issues 5. Use --log-text for audit trail

For Development/Testing: 1. Start with --no-subwebs for faster iteration 2. Use --max-items 10 to limit export size 3. Test recursive export on small site hierarchy first

For Large Site Collections (100+ sites): 1. Consider exporting in batches 2. Use --max-file-size to control bandwidth 3. Run during off-peak hours 4. Monitor disk space on target system

Version History¶

Added in S1.148 (2025-11-05): - Full recursive sub-web export implementation - Circular reference protection with URL tracking - --no-subwebs command-line flag - Complete sub-web content export (lists, users, groups, etc.) - Preservation of site hierarchy in folder structure

Previously (v1.0): - Basic sub-web metadata export only - No recursive traversal - Limited to direct child webs

Additional Resources¶

Migration Tool Documentation: - Main README - Usage and examples - Deployment Guide - Setup instructions

Cesivi Server: - General Troubleshooting - Server issues - API Reference - Supported APIs - Getting Started - Initial setup

SharePoint Resources: - SharePoint CSOM Reference - SharePoint Permission Levels

Getting Help¶

Before requesting support: 1. ✅ Enable logging and review error logs 2. ✅ Try diagnostic procedures above 3. ✅ Check FAQ for similar issues 4. ✅ Search existing GitHub issues

When reporting issues: - Include migration command used (redact credentials!) - Include relevant error messages from logs - Include SharePoint version (2016/2019/Online) - Include migration tool version - Describe what was expected vs actual behavior

Support Channels: - GitHub Issues: Project Issues - Documentation: This guide and related docs

Document Version: 1.0 Last Updated: 2025-11-05 Related Tool Version: Cesivi.MigrationTool 1.0