Production Deployment Checklist¶

Version: 1.0 Last Updated: 2026-01-18

Overview¶

This checklist provides a step-by-step guide for deploying Cesivi Server to production environments. It covers pre-deployment preparation, deployment execution, and post-deployment verification.

Related Documentation: - DEPLOYMENT_GUIDE.md - General deployment instructions - SECURITY_DEPLOYMENT_CHECKLIST.md - Security-specific deployment steps - SECURITY_HARDENING_GUIDE.md - Security hardening recommendations - PERFORMANCE_TUNING_GUIDE.md - Performance optimization guide

Pre-Deployment Phase¶

1. Backup Current Environment¶

[ ] Backup current application binaries
Location: /var/www/cesivi/ or configured deployment directory
Archive name: cesivi-backup-YYYYMMDD-HHMMSS.tar.gz
Retention: Keep last 3 production deployments
[ ] Backup database (if using SQL Server/PostgreSQL)
Full database backup with timestamp
Verify backup integrity (test restore on non-production DB)
Store backup in secure location separate from application server
[ ] Backup configuration files
appsettings.Production.json
web.config (IIS deployments)
SSL/TLS certificates
Plugin configurations
[ ] Document current application state
Current version: GET /_health/version
Current metrics: GET /_diagnostics/metrics
Active sessions count
Database size and record counts

2. Review Changes¶

[ ] Code review completed
All pull requests approved
No outstanding high-priority issues
Security scan passed (if applicable)
[ ] Review release notes
Breaking changes identified
Migration scripts required (if any)
New features documented
Bug fixes verified
[ ] Review configuration changes
New appsettings keys required
Deprecated settings removed
Environment-specific values updated
[ ] Verify dependencies
.NET 10 runtime installed on target server
Database version compatible
Required system packages installed

3. Test in Staging Environment¶

[ ] Deploy to staging environment
Use identical deployment process
Same configuration structure (different credentials)
Same hardware/infrastructure profile
[ ] Run smoke tests
Application starts successfully
Health endpoint responds: GET /_health
Authentication works
Basic CRUD operations succeed
[ ] Run full test suite
Unit tests: dotnet test Cesivi.Tests
Integration tests: dotnet test Cesivi.Tests.RestSoap
CSOM tests: dotnet test Cesivi.Tests.Csom
PnP tests: dotnet test Cesivi.Tests.PnP
[ ] Performance validation
Load testing with expected production traffic
Response times within SLA (< 200ms for simple queries)
Memory usage stable over 1+ hour test
No connection leaks
[ ] User acceptance testing (UAT)
Key stakeholders approve staging deployment
Critical workflows verified
No blocking issues reported

4. Prepare Deployment Window¶

[ ] Schedule maintenance window
Communicate to users (email, status page)
Duration: Typically 30-60 minutes
Off-peak hours preferred
Approval from change management board (if required)
[ ] Prepare deployment team
Primary deployer assigned
Backup deployer available
DBA on standby (if database changes)
Network/infrastructure team notified
[ ] Prepare rollback plan
Documented rollback steps (see section below)
Rollback decision criteria defined
Rollback tested in staging

Deployment Phase¶

5. Pre-Deployment Checks¶

[ ] Stop application gracefully

# Systemd (Linux)
sudo systemctl stop cesivi.service

# IIS (Windows)
Stop-WebAppPool -Name "Cesivi"

[ ] Verify no active sessions
Check application logs for last request timestamp
Wait 30 seconds for in-flight requests to complete
Verify database connections closed
[ ] Take final backup
Incremental backup since pre-deployment backup
Quick sanity check (file sizes, record counts)

6. Deploy Application¶

[ ] Upload new binaries

# Example: Deploy from build artifact
cd /var/www/cesivi
tar -xzf /tmp/cesivi-release-v2.5.0.tar.gz

[ ] Update configuration files
Merge new appsettings keys
Update connection strings (if changed)
Verify environment variables set correctly

[ ] Update database schema (if required)

# Run migration scripts
dotnet ef database update --project Cesivi.Server

[ ] Set correct permissions

# Linux example
chown -R www-data:www-data /var/www/cesivi
chmod -R 755 /var/www/cesivi
chmod 600 /var/www/cesivi/appsettings.Production.json

[ ] Verify file integrity
Check MD5/SHA256 checksums of critical files
Ensure all required files present

7. Start Application¶

[ ] Start application

# Systemd (Linux)
sudo systemctl start cesivi.service

# IIS (Windows)
Start-WebAppPool -Name "Cesivi"

[ ] Monitor startup logs

# Watch logs for errors
tail -f /var/log/cesivi/app.log

# Or journalctl (systemd)
journalctl -u cesivi.service -f

[ ] Wait for application ready
Startup typically takes 5-15 seconds
Watch for "Application started" log message

Post-Deployment Phase¶

8. Health Check Verification¶

[ ] Basic health check

curl http://localhost:5000/_health
# Expected: {"status":"Healthy","timestamp":"..."}

[ ] Detailed diagnostics

curl http://localhost:5000/_diagnostics/health
# Verify all components healthy (database, cache, storage)

[ ] Version verification

curl http://localhost:5000/_health/version
# Verify version matches deployed version

9. Functional Smoke Testing¶

[ ] Authentication test

# Test Basic authentication
curl -u admin:password http://localhost:5000/_api/web

[ ] Web operations test

# Get current web
curl -u admin:password http://localhost:5000/_api/web \
  -H "Accept: application/json;odata=verbose"

# Expected: Web object with Title, Url, Created properties

[ ] List operations test

# Get all lists
curl -u admin:password http://localhost:5000/_api/web/lists \
  -H "Accept: application/json;odata=verbose"

# Expected: List collection

[ ] Item creation test

# Create test item in existing list
curl -X POST http://localhost:5000/_api/web/lists/getbytitle('TestList')/items \
  -u admin:password \
  -H "Content-Type: application/json;odata=verbose" \
  -H "Accept: application/json;odata=verbose" \
  -d '{"__metadata":{"type":"SP.ListItem"},"Title":"Deployment Test"}'

# Expected: HTTP 201 Created

[ ] OData query test

# Test OData filtering
curl -u admin:password \
  "http://localhost:5000/_api/web/lists/getbytitle('TestList')/items?\$filter=Title eq 'Deployment Test'"

# Expected: Item created in previous test

10. Performance Verification¶

[ ] Response time check
Simple query: < 100ms
Complex query: < 500ms
List creation: < 200ms

[ ] Memory usage check

# Linux
ps aux | grep cesivi

# Expected: < 500MB initial memory (varies by workload)

[ ] Database connection pool

# Query diagnostics endpoint
curl http://localhost:5000/_diagnostics/metrics | jq '.database'

# Verify connections < max pool size

11. Monitoring Setup¶

[ ] Verify metrics collection
Application metrics endpoint: /_diagnostics/metrics
System metrics (CPU, memory, disk)
Database metrics (connections, query time)
[ ] Verify alerting configured
High error rate alerts
High response time alerts
Database connection pool exhaustion alerts
Disk space alerts
[ ] Verify log aggregation
Application logs flowing to centralized logging (if configured)
Error logs captured
Audit logs captured

12. User Notification¶

[ ] Announce deployment complete
Email to users
Update status page
Internal team notification
[ ] Provide release notes
New features summary
Breaking changes (if any)
Known issues

Rollback Plan¶

When to Rollback¶

Rollback immediately if: - Application fails to start within 5 minutes - Health endpoint returns Unhealthy status - Critical functionality broken (authentication, list operations) - Database corruption detected - Security vulnerability introduced

Consider rollback if: - Performance degradation > 50% - Non-critical features broken affecting > 10% of users - Unexpected errors in logs > 10/minute

Rollback Steps¶

Stop current application
```
sudo systemctl stop cesivi.service
```

Restore previous binaries

cd /var/www/cesivi
rm -rf *
tar -xzf /backup/cesivi-backup-YYYYMMDD-HHMMSS.tar.gz

Restore database (if schema changed)

# Restore from backup
# SQL Server example:
sqlcmd -S localhost -U sa -P password -Q "RESTORE DATABASE CesiviDB FROM DISK='/backup/CesiviDB.bak'"

Restore configuration files

cp /backup/appsettings.Production.json /var/www/cesivi/

Start application
```
sudo systemctl start cesivi.service
```
Verify rollback successful
Health check passes
Version matches previous version
Basic functionality works
Notify stakeholders
Deployment rolled back
Investigation in progress
ETA for resolution

Post-Deployment Monitoring (First 24 Hours)¶

Hour 1: Critical Monitoring¶

[ ] Monitor error logs every 15 minutes
Watch for new error patterns
Investigate any HTTP 500 errors immediately
[ ] Monitor response times
Compare to baseline from staging
Alert if > 2x baseline
[ ] Monitor resource usage
CPU usage stable (< 80%)
Memory usage not growing continuously
Disk I/O within normal range

Hour 1-4: Active Monitoring¶

[ ] Check error logs every 30 minutes
[ ] Review user feedback channels
Support tickets
User reports
Social media mentions
[ ] Verify scheduled jobs running (if applicable)
Background workers started
Cleanup jobs executing

Hour 4-24: Passive Monitoring¶

[ ] Check dashboards every 2-4 hours
[ ] Review daily metrics summary
Total requests
Error rate
Average response time
Unique users
[ ] Plan post-mortem (if issues occurred)
Document issues encountered
Root cause analysis
Prevention measures

Deployment Success Criteria¶

Deployment considered successful when: - ✅ Application running for 24 hours without critical issues - ✅ Error rate < 1% of requests - ✅ Average response time within 10% of baseline - ✅ No rollback required - ✅ No critical bugs reported by users - ✅ All health checks passing - ✅ Monitoring and alerting functional

Appendix: Common Issues¶

Issue 1: Application Fails to Start¶

Symptoms: - Service fails to start - "Application startup exception" in logs

Common Causes: - Missing configuration keys - Database connection string incorrect - Port already in use - File permissions incorrect

Resolution:

# Check logs
journalctl -u cesivi.service -n 50

# Verify configuration
cat /var/www/cesivi/appsettings.Production.json

# Test database connection
dotnet ef database update --dry-run

Issue 2: High Error Rate After Deployment¶

Symptoms: - HTTP 500 errors in logs - Users reporting errors

Common Causes: - Breaking API changes - Missing database migration - Cache invalidation needed

Resolution:

# Check recent errors
tail -f /var/log/cesivi/app.log | grep ERROR

# Clear cache
curl -X POST http://localhost:5000/_diagnostics/clearcache

# Restart application
sudo systemctl restart cesivi.service

Issue 3: Performance Degradation¶

Symptoms: - Slow response times - High CPU/memory usage

Common Causes: - Missing database indexes - Cache not configured correctly - Connection pool exhausted

Resolution:

# Check metrics
curl http://localhost:5000/_diagnostics/metrics

# Review database query performance
# (Use database-specific query profiling tools)

# Adjust connection pool settings in appsettings

Last Updated: 2026-01-18 Version: 1.0 For Support: See README.md