Tutorial H — SPO Local Backup (Variant B Incremental Sync)¶

Audience: Cesivi administrators
Time estimate: 60–90 minutes (first-time setup on a fresh Cesivi instance with a new M365 dev tenant)
Prerequisites: Cesivi instance installed and reachable, access to an M365 tenant with SharePoint Online

Section 1: Prerequisites¶

Before starting, confirm you have:

Cesivi installed and the server running (follow Tutorial A–D for first-time install)
Microsoft 365 tenant — free via Microsoft 365 Developer Program
Azure Portal / Entra ID access — requires a Global Administrator or Application Administrator role on the tenant
CesiviMigrationTool.exe accessible from your server's PATH or a known folder
A SharePoint Online site to back up (e.g. https://contoso.sharepoint.com/sites/hr)

Section 2: Entra App Registration¶

The migration tool connects to SPO using OAuth2 app credentials. You need a Entra application with Sites.FullControl.All app permission.

2.1 Register the application¶

Open Azure Portal → Entra ID → App registrations → New registration
Name: Cesivi-SPO-Backup | Supported account types: Accounts in this organizational directory only (Single tenant)
Click Register
Note the Application (client) ID and Directory (tenant) ID — you'll need them throughout

2.2 Grant SharePoint permission¶

In your app: API permissions → Add a permission → SharePoint
Choose Application permissions → Sites → Sites.FullControl.All
Click Grant admin consent for [your tenant] (requires Global Admin)

2.3 Create credentials (choose one)¶

Option A — Client secret (simpler): 1. Certificates & secrets → New client secret 2. Set a description (cesivi-backup-secret), choose expiry (12 or 24 months) 3. Copy the secret Value immediately — it's hidden after navigation

Option B — Certificate (recommended for production): 1. Generate a self-signed certificate:

$cert = New-SelfSignedCertificate -Subject "CN=CesiviSPOBackup" `
    -CertStoreLocation "Cert:\LocalMachine\My" -NotAfter (Get-Date).AddYears(2)
Write-Host "Thumbprint: $($cert.Thumbprint)"

2. In your app: Certificates & secrets → Upload certificate → upload the exported .cer

Section 3: Setup Wizard Configuration¶

Open the Cesivi Setup wizard
Navigate to Archive → SPO Source (Variant B Backup)
Fill in:
Entra tenant ID — the Directory (tenant) ID from Section 2
App (client) ID — the Application (client) ID from Section 2
SharePoint Online site URL — e.g. https://contoso.sharepoint.com/sites/hr
Authentication mode — choose Client secret or Certificate (device code is for interactive/testing only)
Client secret (if auth=secret) — the secret value from Section 2.3A
Certificate thumbprint (if auth=cert) — from Section 2.3B
Local backup target folder — e.g. C:\Cesivi\SpoBackup
Default sync cadence — daily recommended for production
Click Test connection — wait for "Connected successfully!" before proceeding
Complete the wizard to generate appsettings.json

The wizard writes the following block to appsettings.json:

"Cesivi": {
  "Archive": {
    "Source": {
      "SPO": {
        "TenantId": "<your-tenant-id>",
        "SiteUrl": "https://contoso.sharepoint.com/sites/hr",
        "AppId": "<your-app-id>",
        "AuthMode": "secret",
        "ClientSecret": "<redacted>",
        "TargetPath": "C:\\Cesivi\\SpoBackup"
      }
    }
  }
}

Section 4: First Bulk Import¶

Run a full one-shot bulk import to establish the initial baseline. This downloads all lists, items, versions, and attachments.

CesiviMigrationTool.exe import ^
  --source spo ^
  --site https://contoso.sharepoint.com/sites/hr ^
  --tenant <tenantId> ^
  --app-id <appId> ^
  --auth secret ^
  --secret <secret> ^
  --target C:\Cesivi\SpoBackup

For certificate auth, replace --auth secret --secret <s> with --auth cert --cert-thumbprint <thumbprint>.

The tool will create a folder structure under C:\Cesivi\SpoBackup:

SpoBackup/
  Documents/        ← per-list folders
    items.json
    files/
    versions/
  Tasks/
    items.json
  _checkpoints/     ← watermark files (do not delete!)
  _web.json

After the import completes, the tool records the current SPO change token for each list in _checkpoints/. This is the baseline for incremental sync.

Expected runtime: 2–15 minutes depending on site size and throttling. The tool respects SPO throttle limits automatically (Retry-After header).

Section 5: Schedule Incremental Sync¶

Run incremental sync on a schedule to keep the local backup current. The sync reads change tokens stored in _checkpoints/ and applies only the changes since the last sync.

One-shot sync (test it first):¶

CesiviMigrationTool.exe sync --once ^
  --site https://contoso.sharepoint.com/sites/hr ^
  --tenant <tenantId> --app-id <appId> --auth secret --secret <secret> ^
  --target C:\Cesivi\SpoBackup

Scheduled (Windows Task Scheduler):¶

Create a scheduled task that runs daily at 02:00:

$action = New-ScheduledTaskAction -Execute "CesiviMigrationTool.exe" `
    -Argument "sync --once --site https://contoso.sharepoint.com/sites/hr ..."
$trigger = New-ScheduledTaskTrigger -Daily -At "02:00"
Register-ScheduledTask -TaskName "CesiviSpoBackup" -Action $action -Trigger $trigger -RunLevel Highest

Or use the --schedule daily flag for the built-in foreground scheduler:

CesiviMigrationTool.exe sync ^
  --site https://contoso.sharepoint.com/sites/hr ^
  --tenant <tenantId> --app-id <appId> --auth secret --secret <secret> ^
  --target C:\Cesivi\SpoBackup ^
  --schedule daily

This process blocks and fires at the next 09:00 UTC each day. Run it as a Windows Service for production.

Section 6: Verify Sync in ControlCenter¶

Open the Cesivi ControlCenter in a browser
Navigate to Archive → SPO Sync Status (/_admin/archive-spo-sync.aspx)
The table shows per-list: last sync timestamp, watermark, cadence, items changed, status (OK/Failed/Pending)
To test manually: click Sync now next to a list → status changes to "Pending" then "OK"

After editing an item in SPO and running sync, the new version appears in C:\Cesivi\SpoBackup\Documents\items.json with a new version entry.

Section 7: Webhook Setup (Optional — Near-Real-Time)¶

SPO webhooks enable near-real-time sync — a change on SPO fires within ~60 seconds. This is opt-in per-list.

Requirements¶

A public HTTPS URL reachable from SPO (cannot use localhost in production)
For development/testing: ngrok or Cloudflare Tunnel

7.1 Start the webhook daemon¶

CesiviMigrationTool.exe webhook-daemon ^
  --site https://contoso.sharepoint.com/sites/hr ^
  --tenant <tenantId> --app-id <appId> --auth secret --secret <secret> ^
  --prefix http://+:9000/spo-webhook/ ^
  --target C:\Cesivi\SpoBackup ^
  --debounce 5

Note: HttpListener requires URL ACL. If access is denied, run once as admin or register: netsh http add urlacl url=http://+:9000/spo-webhook/ user=DOMAIN\CesiviServiceAccount

7.2 Expose the daemon via a tunnel (dev) or reverse proxy (prod)¶

Development (ngrok):

ngrok http 9000
# Note the HTTPS URL: https://abc123.ngrok.io

Production: Configure your reverse proxy (IIS ARR, nginx, Caddy) to forward https://cesivi.yourdomain.com/spo-webhook/ → http://localhost:9000/spo-webhook/.

CesiviMigrationTool.exe webhook subscribe ^
  --site https://contoso.sharepoint.com/sites/hr ^
  --tenant <tenantId> --app-id <appId> --auth secret --secret <secret> ^
  --list Documents ^
  --notification-url https://abc123.ngrok.io/spo-webhook/ ^
  --expiry-days 180

The subscription ID is printed and saved to webhook-subscriptions.json. The daemon auto-renews subscriptions 30 days before expiry.

7.4 Verify the webhook¶

Edit any item in the Documents list on SPO (via browser)
Within ~60 s, the daemon terminal logs: Webhook triggered sync for resource ...
Check the ControlCenter SPO Sync Status page — the timestamp should update

Section 8: Troubleshooting¶

Symptom	Cause	Fix
`AADSTS700016: Application not found`	Wrong tenant ID or app ID	Re-check both from Azure Portal → App registrations
`AADSTS7000218: The request body must contain ... client_assertion`	App registration was deleted or app type wrong	Recreate as multi-tenant or single-tenant app
`Token expired, full rescan required`	SPO change token is >60 days old	Re-run bulk import to reset watermarks
Throttling (HTTP 429)	SPO is rate-limiting the tool	The tool retries automatically via `Retry-After` header; reduce parallelism with `--lists` flag
Webhook challenge handshake timeout	Daemon not reachable from SPO	Confirm your public HTTPS URL is reachable externally; check firewall rules
Subscription expired	Renewal daemon was not running	Restart the `webhook-daemon`; it renews on startup. If subscription is fully expired, re-run `webhook subscribe`
`Access denied` to HttpListener port	HttpListener URL ACL not configured	Run `netsh http add urlacl url=http://+:PORT/spo-webhook/ user=NETWORK SERVICE` as admin
Certificate thumbprint not found	Certificate not in correct store	The cert must be in `LocalMachine\My` or `CurrentUser\My` on the machine running the tool