Multi-Web-Application — Operations Runbook¶

Operational procedures for a Cesivi farm hosting multiple web applications / host-named site collections behind nginx (Linux) or IIS/Kestrel-direct (Windows).

Creating or converting a host-named site collection? See the dedicated hnsc-guide.md walkthrough. This page covers the TLS/cert, farm-scripting, and load-balancer concerns around HNSCs once they exist.

TLS / HTTP hardening (PLAN-1790)¶

Cesivi enforces a single TLS baseline on every shipped host:

Protocols: TLS 1.2 + TLS 1.3 only. TLS 1.1/1.0, SSL 3/2 are refused at the handshake.
Ciphers: AEAD + ECDHE only (forward secrecy). No RC4, 3DES, CBC, or static-RSA key exchange.
HSTS: max-age=63072000; includeSubDomains; preload (2 years → preload-eligible).
HTTP→HTTPS: port 80 returns 301 to the https URL.
Security headers: X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy, nonce-based CSP.

Where the baseline is enforced¶

Surface	Mechanism
Kestrel (all HTTPS listeners: base, per-host SNI, dedicated ports)	`Cesivi.Common.Security.TlsHardening.Apply` via `ConfigureHttpsDefaults` in both `Cesivi.Server/Program.cs` and `Cesivi.WebUI/Program.cs`
nginx (Linux edge / TLS termination)	`deployment/server-scripts/setup-nginx-cesivi.sh` — explicit `ssl_protocols` / `ssl_ciphers` / `ssl_stapling` / HSTS `add_header`
HSTS + security headers (app)	`SecurityHeadersMiddleware` (Server + WebUI)
Auth/session cookie `Secure` flag	`Cesivi.Common.Security.CookieSecurity.IsSecureRequest` (honours `X-Forwarded-Proto`)
HTTP→HTTPS 301	nginx (Linux); WebUI `UseHttpsRedirection` (301); Server `UseHttpsRedirection` gated by `Cesivi:Security:HttpsRedirect`

Note (cipher policy by OS): CipherSuitesPolicy is honoured on Linux/macOS only. On Windows the Schannel system policy governs cipher selection — restrict ciphers there via Group Policy (Computer Configuration → Administrative Templates → Network → SSL Configuration Settings), not in code. Kestrel still pins the protocol set to TLS 1.2/1.3 on Windows.

Verifying a host (closure gate)¶

Run the executable handshake matrix from any Linux box (or WSL):

bash Scripts/tls-handshake-test.sh demo1.cesivi.com 443 80

It asserts: TLS 1.3 ✓, TLS 1.2 ✓, TLS 1.1 ✗, TLS 1.0 ✗, port-80 → 301 https, HSTS present.

Then run a live SSL Labs scan and confirm grade A (or higher): https://www.ssllabs.com/ssltest/analyze.html?d=demo1.cesivi.com&hideResults=on

Renewing certificates¶

Let's Encrypt certs (issued by setup-nginx-cesivi.sh via certbot) auto-renew through the certbot.timer systemd unit. To verify / force:

sudo certbot renew --dry-run     # validate renewal config
sudo certbot renew               # renew anything within 30 days of expiry
sudo systemctl reload nginx      # pick up the new cert (certbot's deploy hook usually does this)

For a host-named SC added as a SAN on a parent WA cert, re-run certbot with the full -d list so the new SAN is included, then reload nginx.

After any cert change, re-run Scripts/tls-handshake-test.sh <host> to confirm the baseline still holds.

If SSL Labs grade drops below A¶

Work the checklist in order — each maps to a concrete config surface:

Protocol leak (TLS 1.0/1.1 accepted). An edge or listener lost the protocol pin.
nginx: confirm ssl_protocols TLSv1.2 TLSv1.3; is present and no include options-ssl-nginx.conf re-widened it. Reload nginx.
Kestrel direct: confirm ConfigureHttpsDefaults(TlsHardening.Apply) is still wired (a new listener added outside ConfigureHttpsDefaults would miss it).
Weak cipher (CBC/3DES/RC4 offered). Check ssl_ciphers in nginx matches the AEAD/ECDHE list in setup-nginx-cesivi.sh. On Windows hosts, check the Schannel Group Policy cipher order.
No forward secrecy. A static-RSA suite slipped in — same fix as (2); the allowed list is ECDHE-only.
HSTS missing / short. Confirm both the nginx add_header Strict-Transport-Security ... always; and the app SecurityHeadersMiddleware emit max-age=63072000.
Cert chain incomplete / OCSP stapling off. Ensure ssl_certificate points at fullchain.pem (not cert.pem) and ssl_stapling on; ssl_stapling_verify on; ssl_trusted_certificate .../chain.pem; are present with a working resolver.
Re-test: Scripts/tls-handshake-test.sh <host> then re-scan SSL Labs.

HSTS preload submission¶

A 2-year max-age with includeSubDomains; preload makes a host eligible for the browser preload list. Submitting a domain to https://hstspreload.org/ is the customer's decision for their own apex domain (it is hard to reverse and affects all subdomains). We ship preload-eligible headers; we do not submit a customer's domain on their behalf. For *.cesivi.com demo hosts, submission is an internal ops decision, not automated by deploy.

Out of scope (named)¶

mTLS, client-side certificate pinning, HPKP (deprecated), and TLS 1.3 0-RTT are intentionally not enabled — see PLAN-1790 §Out-of-scope.

Per-WA dedicated ports + SNI (multi-listener Kestrel, PLAN-1771)¶

Each web application in Cesivi:Farm:WebApplications gets a HostBindings array — an Alternate-Access-Mapping entry per host, with an optional dedicated port and TLS certificate. Web applications that don't specify a port share the base :443 listener and are routed by SNI (each still gets its own cert); a WA that specifies a distinct port gets its own dedicated Kestrel listener.

{
  "Cesivi": {
    "Farm": {
      "WebApplications": [
        {
          "Name": "Intranet",
          "CertificateRef": "wa-intranet",              // WA-level SAN cert (covers all its hosts + HNSCs)
          "HostBindings": [
            { "Host": "sp.contoso.com", "Port": 443, "Scheme": "https", "IsPrimary": true }
          ]
        },
        {
          "Name": "CentralAdmin",
          "HostBindings": [
            {
              "Host": "ca.contoso.com", "Port": 9443, "Scheme": "https", "IsPrimary": true,
              "TlsCertificateRef": "ca-cert"             // per-binding cert overrides the WA-level one
            }
          ]
        }
      ]
    },
    "Tls": {
      "Certificates": {
        "wa-intranet": { "Path": "/opt/cesivi/certs/intranet.pfx", "Password": "..." },
        "ca-cert":     { "Path": "/opt/cesivi/certs/ca.pfx",       "Password": "..." }
      }
    }
  }
}

Certificate resolution order (highest wins): per-binding TlsCertificatePath (direct file) > per-binding TlsCertificateRef (named config, or store:LocalMachine/My:thumbprint=... / subject=... for the Windows cert store) > WA-level CertificateRef (SAN cert covering all the WA's hosts). A host with no resolvable cert falls back to the listener's base/dev cert and logs a warning — it does not fail startup. Config is validated fail-fast at startup (duplicate host+port bindings, unresolvable cert refs, etc. all refuse to boot with the exact conflict named).

Full reference (validation rules, dev-VM/loopback setup, SSL-Labs verification, known limitations): _docs_dev/multi-webapp-deploy.md.

Managing web applications via the Admin UI¶

Everything in this runbook is also reachable click-through, without PowerShell, at /_admin/webapplications.aspx (FarmAdmin role required):

/_admin/webapplications.aspx — the web-application list: name, host bindings, managed paths, anonymous-access mode, and site-collection count per row. New Web Application opens the create form.
/_admin/webapplicationcreate.aspx — create form: Name (the storage-key identifier), Display Name, Base URL, Host Bindings (one host or host:port per line — the wildcard * is reserved for the Default web application), Anonymous Access mode, and an optional dedicated Storage backend (isolates this WA's data from every other WA; leave on "inherit farm default" unless you need per-WA isolation).
/_admin/webapplicationsettings.aspx?name={wa} — click a WA name in the list to reach its settings page. Sections, top to bottom:
General / Host Bindings / Anonymous Access / Storage — editable (except the Default WA's host bindings, which are read-only: it owns the wildcard fallback). Save Changes applies; Delete appears only for a non-Default WA with zero site collections.
Managed Paths and Site Collections — read-only summaries here (managed paths are added implicitly when a site collection is created under one; see USER-GUIDE-SITE-COLLECTIONS.md).
Content Databases — list + per-row status changer (Online/ReadOnly/etc.) + Create physical DB (for a CDB with its own connection-string reference) + Add Content Database form (Id, Name, Status, Max site count, optional connection-string reference, optional "create physical database now"). Mirrors the New/Get/Set/Remove-CSContentDatabase cmdlets below.
Host-Named Site Collections — create/list/convert HNSCs under this WA. See the dedicated hnsc-guide.md walkthrough for this section.

There is no Admin-UI equivalent (yet) for backup/restore or site-collection move/duplicate — those are PowerShell/REST-only (below).

Farm scripting with PnP cmdlets (PLAN-1773)¶

Operators script the farm — web applications, content databases, backups — with two interchangeable surfaces over the same /_admin/webapps... REST endpoints, both requiring the CentralAdmin (FarmAdmin) role:

Primary — native Cesivi.PowerShell snap-in (the Microsoft.SharePoint.PowerShell analogue): *-CSWebApplication, *-CSContentDatabase, Backup/Restore-CSContentDatabase (with *-CesiviWebApplication / *-CesiviContentDatabase aliases). Connect with Connect-Cesivi.
Secondary — PnP compatibility module (CesiviPnP): *-PnPWebApplication, *-PnPContentDatabase, Backup/Restore-PnPContentDatabase. Connect with Connect-PnPOnline.

# Native (primary)
Import-Module Cesivi.PowerShell
Connect-Cesivi -Server https://sp.contoso.com -Credential (Get-Credential)   # must be a FarmAdmin

# PnP (secondary)
Import-Module CesiviPnP
Connect-PnPOnline -Url https://sp.contoso.com -CurrentCredentials            # must be a FarmAdmin

Web applications¶

# Native (primary)
New-CSWebApplication -Name Intranet -DisplayName 'Intranet' -Url https://sp.contoso.com -HostHeader sp.contoso.com
New-CSWebApplication -Name Intranet -Host sp.contoso.com -StorageBackendId Postgres -StorageConfig @{ ConnectionStringRef = 'Inet' }
Get-CSWebApplication | Format-Table Name,Url,SiteCollectionCount
Set-CSWebApplication -Identity Intranet -HostHeader sp.contoso.com,teams.contoso.com
Remove-CSWebApplication -Identity Intranet -Force        # blocked while it owns site collections / for Default

# PnP (secondary) — identical semantics
New-PnPWebApplication -Name Intranet -DisplayName 'Intranet' -Url https://sp.contoso.com -HostHeader sp.contoso.com
Get-PnPWebApplication | Format-Table Name,Url,SiteCollectionCount
Set-PnPWebApplication -Identity Intranet -HostHeader sp.contoso.com,teams.contoso.com
Remove-PnPWebApplication -Identity Intranet -Force

-HostHeader entries accept host, host:port or scheme://host:port. -ManagedPath entries accept path or path:Explicit|Wildcard.

Content databases¶

# Native (primary)
New-CSContentDatabase -WebApplication Intranet -Name Inet2 -ConnectionStringRef Inet2
New-CSContentDatabase -WebApplication Intranet -Name Inet2 -ConnectionStringRef Inet2 -CreatePhysicalDatabase
Get-CSContentDatabase | Format-Table Name,SiteCount,MaxSiteCount,Status
Set-CSContentDatabase -Identity Inet1 -Status ReadOnly   # -WebApplication optional when the id is farm-unique
Remove-CSContentDatabase -WebApplication Intranet -Identity Inet2 -Force

# PnP (secondary) — identical semantics
New-PnPContentDatabase -WebApplication Intranet -Name Inet2 -ConnectionStringRef Inet2 -CreatePhysicalDatabase
Get-PnPContentDatabase | Format-Table Name,SiteCount,MaxSiteCount,Status
Set-PnPContentDatabase -Identity Inet1 -Status ReadOnly
Remove-PnPContentDatabase -WebApplication Intranet -Identity Inet2 -Force

-ConnectionStringRef names a configured connection looked up under Cesivi:ConnectionStrings:{Ref} (so the real connection string never lives in appsettings.json). Omit it to inherit the WA/farm backend.

Backup & restore (disaster recovery)¶

Backup/restore operates at content-database granularity using each backend's native tooling: SQLite is a consistent file copy (SQLite Online Backup API); SQL Server is native BACKUP DATABASE / RESTORE DATABASE. A content database that inherits the WA/farm backend has no separate physical database to dump — the cmdlet reports Unsupported. The backup path is on the database host (for SQL backends, a path the SQL Server service account can write).

Postgres / MySql: these providers shell out to the native client tools — pg_dump/pg_restore (Postgres, in the Cesivi.Storage.PostgreSql package, registered where that package is referenced, the same model as the physical-database provisioner) and mysqldump/mysql (MySql, in core, registered by default). The client tools must be on the server's PATH (or set Cesivi:Postgres:ClientToolsPath / Cesivi:MySql:ClientToolsPath). Passwords are passed via PGPASSWORD / MYSQL_PWD, never on the command line. If a Postgres deployment hasn't registered the Postgres backup provider, Backup-PnPContentDatabase reports Unsupported — register it alongside PostgresPhysicalDatabaseProvisioner.

Scripted disaster-recovery scenario:

# 1. Back up before maintenance   (native primary — or Backup-PnPContentDatabase)
Backup-CSContentDatabase -Identity Inet1 -Path L:/backups/inet1-2026-06-30.bak

# 2. (disaster) the physical database is lost

# 3. Restore — replaces the content database's contents from the backup
Restore-CSContentDatabase -Identity Inet1 -Path L:/backups/inet1-2026-06-30.bak -Force

# 4. All site collections previously in Inet1 are operational again
Get-CSContentDatabase -Identity Inet1 | Format-Table Name,SiteCount,Status

Site-collection move / duplicate (PLAN-1787)¶

Two operations relocate or clone a whole site collection. They flow through the farm-admin REST surface /_admin/sites (CentralAdmin-gated), the primary native cmdlets Move-CSSite / Copy-CSSite (Cesivi.PowerShell, with Move-CesiviSite / Copy-CesiviSite aliases), and the secondary PnP wrappers Move-PnPSite / Copy-PnPSite. Move is scheduled-downtime (the SC is taken offline for the relocation); copy is atomic (the destination is fully usable or absent).

Migrate a customer's SC from content database A to content database B (same web application):

# Re-pin the 'eng' site collection to content database 'CDB2'. Same URL; data served from the new CDB.
Connect-PnPOnline -Url https://demo1.cesivi.com -Interactive       # a CentralAdmin (FarmAdmin) account
Move-PnPSite -Identity eng -TargetContentDatabase CDB2
#   …or natively:
Move-CSSite  -Identity eng -TargetContentDatabase CDB2

For content databases that share or inherit the WA/farm backend (the default), the move is a metadata re-pin (SiteCollection.ContentDatabaseId) plus a backend-pool eviction — the SC stays reachable at the same URL. For content databases backed by physically distinct connections (e.g. one Postgres database per CDB), the row copy across the two connections is the documented integration path (reachable Postgres + client tools required).

Move an SC to a different web application (route table updated; old URL 301-redirects for 30 days):

Move-PnPSite -Identity eng -TargetWebApplication Intranet
# An explicit managed path '/eng' is added on 'Intranet'; the source's managed path is removed; the moved SC
# remembers its old URL (PreviousManagedPathUrl + RedirectGraceUntilUtc = now+30d) for the grace redirect.

Duplicate an SC (clone webs, lists, libraries, items, files, ACLs, content types, fields, principals into a new SC — atomic, principal Ids remapped, CT/field GUIDs preserved):

Copy-PnPSite -Source eng -Destination eng-copy
Copy-PnPSite -Source eng -Destination eng-archive -NewOwner 'i:0#.w|contoso\archivist'
#   …or natively:  Copy-CSSite -SourceUrl eng -DestinationUrl eng-copy

The remap table (old→new principal Ids) is recorded as an audit artifact in the destination's /Lists/_DuplicateAuditLog. Recent operations are queryable: GET /_admin/sites/{id}/operations?webApp=Default and GET /_admin/sites/operations.

Non-goals: zero-downtime online move (scheduled-downtime only); cross-farm move/copy (use Backup-PnPContentDatabase / Restore-PnPContentDatabase, above); resumable partial move (atomicity is at the SC level).

The engine is covered by Cesivi.Server.Tests/MultiWebApp/SiteMoveCopyTests (the six closure-gate scenarios: move same-WA-different-CDB, move cross-WA + 301 grace, atomic copy + independence, ACL remap, CT-id preservation, copy rollback). Live Move-PnPSite / Copy-PnPSite against demo1 + intranet21 parity are the manual acceptance gate (same framing as the PLAN-1773 backup/restore live gate).

Smoke-testing the cmdlets¶

Cesivi.PnP.Commands.Tests/Farm.Cmdlets.Tests.ps1 is a Pester suite that exercises the whole surface against a running server (set CESIVI_URL / CESIVI_USER / CESIVI_PASS). The deterministic engine logic is covered by the xUnit suites Cesivi.Tests.Storage/ContentDatabaseBackupTests and Cesivi.Server.Tests/MultiWebApp/ContentDatabaseBackupAdminTests.

The native parallel suite Tests/Cesivi.PowerShell.FarmCmdlets.Tests.ps1 covers the *-CSWebApplication / *-CSContentDatabase / Backup/Restore-CSContentDatabase cmdlets (structural registration always; the live WA/CDB lifecycle when the FarmAdmin /_admin/webapps surface is reachable).

Cmdlet-surface note (PLAN-1774 parity audit, resolved PLAN-1795). Web-application and content-database management now ships in BOTH surfaces, matching the site-collection cmdlets. The primary native Cesivi.PowerShell snap-in provides Get/New/Set/Remove-CSWebApplication, Get/New/Set/Remove-CSContentDatabase and Backup/Restore-CSContentDatabase (with *-CesiviWebApplication / *-CesiviContentDatabase aliases per DESIGN §2.11.1); the secondary PnP module provides the *-PnPWebApplication / *-PnPContentDatabase wrappers. Both are thin clients over the same /_admin/webapps REST surface.

Load-balancer deployment¶

A clustered Cesivi farm (2+ servers sharing storage + distributed DataProtection, PLAN-1772) sits behind a single corp-net endpoint that terminates TLS once and routes by Host: header to a healthy backend. Two supported shapes:

Option A — BYO load balancer (nginx / HAProxy / IIS ARR / cloud LB) — SHIPPING¶

The fully-supported shape today. Any L7 LB that can route by host header works. The farm exposes its routing table at GET /_admin/farm/topology (FarmAdmin-gated) — WA + HNSC host list, cert SAN list, cluster members, per-node health — so config can be generated from it instead of hand-maintained.

# nginx: terminate TLS, route every WA/HNSC host to the backend pool (health-checked).
upstream cesivi_farm { server 10.0.0.11:8443; server 10.0.0.12:8443; }   # cluster members
server {
    listen 443 ssl http2;
    server_name sp.contoso.com teams.contoso.com hr.contoso.com;          # WA hosts ∪ HNSC hosts (= cert SANs)
    ssl_certificate     /etc/letsencrypt/live/contoso/fullchain.pem;       # one SAN cert per WA (see § Renewing certificates)
    ssl_certificate_key /etc/letsencrypt/live/contoso/privkey.pem;
    # TLS hardening (§ above): ssl_protocols TLSv1.2 TLSv1.3; AEAD/ECDHE ciphers; HSTS; OCSP stapling.
    location / { proxy_pass https://cesivi_farm; proxy_set_header Host $host; }   # Host header preserved → WA resolution
}

The only hard requirement is that the LB preserves the inbound Host: header (Cesivi resolves the WA/HNSC from it — § DESIGN invariant #3) and forwards X-Forwarded-Proto: https (so secure-cookie detection works behind TLS termination — CookieSecurity.IsSecureRequest). No sticky sessions are needed: distributed DataProtection (PLAN-1772) lets any backend serve any request.

Option B — `cesivi-lb` config generator — SHIPPED (PLAN-1788, 2026-07-05)¶

cesivi-lb (project Cesivi.LoadBalancer) is a config generator, not a runtime LB: it reads /_admin/farm/topology and emits a ready-to-run nginx or Caddy reverse-proxy config — one server block per web application + per HNSC, with the right SNI cert per host (WA SAN cert; HNSC inherits it unless it has a CertificateRefOverride), least_conn + no session affinity (PLAN-1772 shares the session state), and health probes on /_vti_bin/health. nginx / Caddy runs the emitted config; the tool does not proxy traffic.

# generate (from a live farm or a saved topology.json)
cesivi-lb generate --target nginx --from-url https://sp.contoso.com --auth bearer:<token> \
    --profile profile.json --out /etc/nginx/conf.d/cesivi.conf
cesivi-lb generate --target caddy --from-file topology.json --profile profile.json --out /etc/caddy/cesivi.caddy

cesivi-lb generate ... --check          # drift mode (exit 1 if the deployed config fell behind the topology)
cesivi-lb generate ... --apply --reload-cmd "systemctl reload nginx"   # validate -> atomic swap -> reload
cesivi-lb watch --from-url ... --out ... --reload-cmd "..."            # poll + auto-apply on every change

The --apply pipeline validates with the target's own checker (nginx -t / caddy validate) and only atomically swaps the file in on success — a broken config never lands. See Cesivi.LoadBalancer/README.md for the site-profile schema (backends, cert locations) and the full flag reference. BYO-LB operators (Option A) can keep hand-written nginx/HAProxy — both flows stay supported.

Verifying farm topology¶

Connect-PnPOnline -Url https://sp.contoso.com -CurrentCredentials   # FarmAdmin
Invoke-RestMethod https://sp.contoso.com/_admin/farm/topology -UseDefaultCredentials | ConvertTo-Json -Depth 6
# → web applications, their host bindings, HNSC hosts, cert SAN union, cluster members, per-node health.