# Migrate your data from AWS S3 to Quake AI Source: https://docs.quake.ai/docs/platform/migration/migrate-from-s3 Markdown: https://docs.quake.ai/docs/platform/migration/migrate-from-s3.md --- # Migrate your data from AWS S3 to Quake AI In this migration guide, you'll move data from an AWS S3 bucket to Quake AI object storage using rclone. By the end, you will have migrated a complete bucket with integrity verification, identified where Quake AI's S3 compatibility has limits, and have a tested procedure you can apply to the rest of your buckets. You'll walk through the full migration methodology: inventory your source data, check for compatibility gaps, run a pilot migration on one bucket, validate that every object arrived intact, and then update your application to use the new endpoint. Each step explains why it matters so you can adapt the process to your own situation. **What you will learn:** - How to inventory an AWS S3 bucket and estimate migration time - Which S3 features Quake AI supports and which it does not (and what to do about gaps) - How to configure rclone to copy data between two S3-compatible providers - How to validate a migration with checksum verification - How to update your application's S3 endpoint with zero data loss - How to run both endpoints in parallel during a transition period **Time estimate:** 45–60 minutes (excluding data transfer time, which depends on bucket size)
```d2 direction: right src: "AWS S3\nsource bucket" {shape: cylinder} inventory: "1. Inventory\nobjects + size" compat: "2. Check S3\ncompatibility" pilot: "3. Pilot migration\n(rclone copy)" validate: "4. Validate\nchecksums" cutover: "5. Update app\nendpoint" dst: "Quake AI\nbucket" {shape: cylinder} src -> inventory inventory -> compat compat -> pilot compat -> pilot: adapt for gaps {style.stroke-dash: 3} pilot -> validate validate -> pilot: re-run on mismatch {style.stroke-dash: 3} validate -> cutover pilot -> dst: writes objects cutover -> dst: app traffic ```
## Prerequisites You need: - An **AWS account** with an S3 bucket containing data you want to migrate - An **AWS IAM user or role** with `s3:GetObject`, `s3:ListBucket`, and `s3:GetBucketLocation` permissions on the source bucket - A **Quake AI account** with [S3 credentials](/docs/object/how-to/create-s3-credentials) created - [rclone](https://rclone.org/install/) v1.65+ installed on your local machine - A stable internet connection (migration speed depends on your upload bandwidth) AWS offers [free data transfer out when migrating away from AWS](https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-internet-when-moving-out-of-aws/). You will not be charged egress fees for this migration if you follow the AWS migration fee waiver process. Check the linked page for current eligibility requirements. ## Why migrate With AWS S3 pricing, you pay for storage, requests, data retrieval, and egress separately, with different rates for different storage classes. Quake AI charges a flat rate per TB with unlimited data transfer included. No egress fees, no request-tier pricing. ## Step 1: Inventory your source bucket Before copying anything, we need to know what we are migrating. Counting objects and measuring total size upfront helps you estimate transfer time and catch surprises (like a bucket that is larger than expected). List the bucket contents and count objects: ```bash aws s3 ls s3://YOUR_SOURCE_BUCKET --recursive --summarize ``` The `--summarize` flag prints total object count and total size at the end. Note both numbers. For a more detailed breakdown by prefix (folder), use: ```bash aws s3 ls s3://YOUR_SOURCE_BUCKET --recursive | awk '{sum += $3; count++} END {printf "Objects: %d\nTotal size: %.2f GB\n", count, sum/1024/1024/1024}' ``` Record these numbers. We use them after the migration to verify nothing was lost. **Estimate transfer time:** At a typical home connection (50 Mbps upload), 100 GB takes about 4.5 hours. At a datacenter connection (1 Gbps), the same transfer takes about 15 minutes. Plan your migration window accordingly. ## Step 2: Check for compatibility gaps Quake AI's object storage is S3-compatible, but not S3-identical. It runs on OpenStack Swift with an S3 gateway. Most S3 operations work without modification, but some AWS-specific features are not supported. **What works:** - `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject`, `s3:ListBucket`: all standard CRUD operations - Multipart uploads (rclone handles this automatically for large files) - Bucket policies for public-read access - Pre-signed URLs for time-limited access - Server-side encryption with SSE-S3 - Object metadata (Content-Type, Cache-Control, custom headers) **What does not work:** | AWS feature | Quake AI status | What to do | |---|---|---| | S3 Lifecycle policies | Not supported | Manage retention manually or with rclone scripts | | S3 Event Notifications (SNS/SQS/Lambda) | Not supported | Use application-level polling or cron jobs | | S3 Object Lock / WORM | Not supported | Use bucket policies for access control | | IAM-style bucket policies (complex conditions) | Partial | Basic Allow/Deny works; condition keys are limited | | S3 Storage Classes (Glacier, IA) | Not supported | All objects use standard storage | | S3 Inventory reports | Not supported | Use `rclone ls` or `aws s3 ls` for inventory | | Cross-region replication | Not supported | Use rclone sync between regions if needed | If your application depends on lifecycle policies or event notifications. You will need to replace those features before switching over. For most applications (file storage, static assets, backups, media hosting), the compatibility is sufficient. For the complete compatibility matrix, see the [migration planning guide](/docs/object/migration/plan-your-migration). ## Step 3: Configure rclone for both providers rclone acts as a bridge between AWS and Quake AI. We configure a remote for each provider so rclone can read from one and write to the other. Edit your rclone configuration: ```bash nano ~/.config/rclone/rclone.conf ``` Add both remotes: ```ini [aws] type = s3 provider = AWS access_key_id = YOUR_AWS_ACCESS_KEY secret_access_key = YOUR_AWS_SECRET_KEY region = us-east-1 [quakeai] type = s3 provider = Ceph access_key_id = YOUR_RUMBLE_ACCESS_KEY secret_access_key = YOUR_RUMBLE_SECRET_KEY endpoint = object.us-east-1.rumble.cloud ``` Replace the placeholder values with your actual credentials. Quake AI's object storage runs on OpenStack Swift with an S3 gateway. The `provider = Ceph` setting is a hint that adjusts how rclone shapes its requests (region handling, list-buckets behavior, retry semantics) for non-AWS S3-compatible services, and it is the recommended setting for Quake AI because it avoids a handful of AWS-specific edge cases. `provider = AWS` against the same `endpoint = object.us-east-1.rumble.cloud` also works, so either value will list buckets and copy objects. Protect the config file. The block you pasted contains plaintext AWS and Quake AI S3 secrets, and a default Linux file mode leaves the config world-readable to other local accounts. ```bash chmod 600 ~/.config/rclone/rclone.conf ``` The rclone config file stores plaintext AWS and Quake AI S3 secret keys. Set its mode to `0600` so other local users cannot read it: ```bash chmod 600 ~/.config/rclone/rclone.conf ``` On a multi-user host or a shared migration VM, this step is mandatory. For production deployments, load credentials from environment variables or a secret manager and remove the long-lived secrets from disk entirely. Verify both remotes work: ```bash rclone lsd aws: rclone lsd quakeai: ``` Each command should list your buckets on the respective provider. If either fails, check the credentials and endpoint. ## Step 4: Create the destination bucket Create a bucket on Quake AI to receive the migrated data. We use the same name as the source bucket when possible: this simplifies endpoint switching later. ```bash rclone mkdir quakeai:YOUR_SOURCE_BUCKET ``` Quake AI bucket names are unique within your project, not globally unique like AWS S3. Unless a bucket with the source name already exists inside your own Quake AI project, the name is available and you can keep it. Reusing the source name makes the endpoint swap in Step 8 a one-line change. If you do hit a name collision in your own project (for example, you created a test bucket with that name earlier), pick a different name and record the mapping: ```bash rclone mkdir quakeai:YOUR_NEW_BUCKET_NAME ``` AWS S3 enforces a global bucket namespace. Every bucket name in the world has to be unique, which is why short names are usually taken. Quake AI scopes bucket names to your project. Two projects on Quake AI can each own a bucket named `images`, `backups`, or `test` without conflict. You can keep your AWS bucket name when you migrate; only an existing bucket inside your own project would force a rename. ## Step 5: Run a pilot migration We start with a small subset of data, not the entire bucket. This validates that the configuration works, metadata transfers correctly, and the transfer speed meets expectations. Copy a single prefix (folder) from AWS to Quake AI: ```bash rclone copy aws:YOUR_SOURCE_BUCKET/sample-folder quakeai:YOUR_SOURCE_BUCKET/sample-folder \ --transfers 8 \ --checkers 16 \ --progress \ --metadata ``` What each flag does: - `copy` transfers files without deleting anything at the destination - `--transfers 8` runs 8 parallel file uploads - `--checkers 16` runs 16 parallel integrity checkers - `--progress` shows real-time transfer statistics - `--metadata` preserves object metadata (Content-Type, timestamps, custom headers) Watch the output. You should see files transferring with speed and progress indicators. When complete, verify the pilot: ```bash echo "Source:" rclone ls aws:YOUR_SOURCE_BUCKET/sample-folder | wc -l echo "Destination:" rclone ls quakeai:YOUR_SOURCE_BUCKET/sample-folder | wc -l ``` If the counts match, the pilot succeeded. ## Step 6: Run the full migration With the pilot verified, copy the entire bucket: ```bash rclone copy aws:YOUR_SOURCE_BUCKET quakeai:YOUR_SOURCE_BUCKET \ --transfers 16 \ --checkers 32 \ --progress \ --metadata \ --fast-list \ --log-file ~/migration.log \ --log-level INFO ``` New flags for the full migration: - `--transfers 16` and `--checkers 32` increase parallelism for faster throughput - `--fast-list` uses fewer API calls to list objects (faster for large buckets, uses more memory) - `--log-file` writes a detailed log for post-migration review - `--log-level INFO` logs each transferred file For buckets over 100 GB, consider running the migration from a VM in the same region as either the source or destination. This avoids routing data through your local machine and can be 10x faster. For buckets over 1 TB, run the migration in stages by prefix. This lets you resume without re-transferring everything if the connection drops: ```bash rclone copy aws:YOUR_SOURCE_BUCKET/prefix-a quakeai:YOUR_SOURCE_BUCKET/prefix-a ... rclone copy aws:YOUR_SOURCE_BUCKET/prefix-b quakeai:YOUR_SOURCE_BUCKET/prefix-b ... ``` rclone's `copy` command is idempotent: running it again only transfers files that are missing or changed at the destination. ## Step 7: Validate the migration Counting files is a sanity check. Checksum verification is proof. rclone can compare the source and destination byte-for-byte: ```bash rclone check aws:YOUR_SOURCE_BUCKET quakeai:YOUR_SOURCE_BUCKET \ --one-way \ --fast-list ``` `rclone check --one-way` verifies that every file in the source exists at the destination with matching size and checksum. It does not transfer any data; it only compares. The output reports: - Files that match (good) - Files that differ in size or checksum (investigate) - Files that exist in the source but not the destination (re-transfer) If all files match, the migration is verified. If some files differ, re-run the `rclone copy` command: it will only transfer the mismatched files. For a final count comparison: ```bash echo "AWS objects:" rclone size aws:YOUR_SOURCE_BUCKET echo "Quake AI objects:" rclone size quakeai:YOUR_SOURCE_BUCKET ``` Both should report the same object count and total size. ## Step 8: Update your application endpoint With the data verified on Quake AI, the last step is pointing your application at the new endpoint. The beauty of S3 compatibility is that this is a configuration change, not a code change. **Before (AWS):** ```yaml endpoint: s3.us-east-1.amazonaws.com bucket: my-app-assets access_key: AKIA... secret_key: ... ``` **After (Quake AI):** ```yaml endpoint: object.us-east-1.rumble.cloud bucket: my-app-assets access_key: YOUR_RUMBLE_ACCESS_KEY secret_key: YOUR_RUMBLE_SECRET_KEY ``` The S3 API calls (`GetObject`, `PutObject`, `ListBucket`, etc.) remain identical. Only the endpoint and credentials change. **For applications using the AWS SDK**, set the endpoint URL explicitly: ```python import boto3 s3 = boto3.client( "s3", endpoint_url="https://object.us-east-1.rumble.cloud", aws_access_key_id="YOUR_RUMBLE_ACCESS_KEY", aws_secret_access_key="YOUR_RUMBLE_SECRET_KEY", ) ``` ```javascript const { S3Client } = require("@aws-sdk/client-s3"); const s3 = new S3Client({ endpoint: "https://object.us-east-1.rumble.cloud", region: "us-east-1", credentials: { accessKeyId: "YOUR_RUMBLE_ACCESS_KEY", secretAccessKey: "YOUR_RUMBLE_SECRET_KEY", }, forcePathStyle: true, }); ``` For zero-downtime migrations, run a dual-write pattern during the transition: write new objects to both AWS and Quake AI, fetch from Quake AI. Once you confirm fetches are stable, stop writing to AWS. This gives you a rollback path if anything unexpected surfaces. For retrieval, switch your application to fetch from Quake AI first and fall back to AWS: ```python def get_object(key): try: return rumble_s3.get_object(Bucket=BUCKET, Key=key) except Exception: return aws_s3.get_object(Bucket=BUCKET, Key=key) ``` After a week with zero AWS fallbacks. You can remove the dual-read logic. ## What you learned In this migration guide, you: - **Inventoried a source bucket** to understand what you were migrating and estimate transfer time - **Checked S3 compatibility** to identify features that work differently on Quake AI (lifecycle policies, event notifications) - **Configured rclone** as a bridge between AWS S3 and Quake AI - **Ran a pilot migration** on a single prefix to validate the configuration before committing to the full transfer - **Migrated the entire bucket** with parallel transfers, metadata preservation, and logging - **Validated integrity** with checksum comparison to prove every object arrived intact - **Updated the application endpoint**: a configuration change, not a code rewrite This process works for any S3-compatible source, not only AWS. Replace the `[aws]` remote with any provider rclone supports: DigitalOcean Spaces, Backblaze B2, Google Cloud Storage, Hetzner, Wasabi. The same steps apply. ## Next steps - [Set up automated backups to Quake AI](/docs/tutorials/automated-backups): protect your migrated data with rclone and restic - [S3 feature compatibility reference](/docs/object/migration/plan-your-migration): the complete compatibility matrix - [Multi-cloud sync](/docs/object/migration/multi-cloud-sync): keep data synchronized across providers during a phased migration - [Update your application endpoint](/docs/object/migration/update-application-endpoint): the how-to reference for endpoint switching across languages and frameworks ## Clean up This migration moves real data that you likely want to keep. Nothing needs to be torn down on the Quake AI side. On AWS, once you are confident the migration is complete and your application is running on Quake AI: 1. Keep the AWS bucket in read-only mode for 30 days as a safety net. 2. After 30 days with no issues, delete the AWS bucket contents and then the bucket itself: ```bash aws s3 rm s3://YOUR_SOURCE_BUCKET --recursive aws s3 rb s3://YOUR_SOURCE_BUCKET ``` Do not delete the source bucket until you are certain the migration is complete and your application is fully operational on Quake AI.