Run any operation asynchronously

Overview#

The asynchronous API stores work and answers at once, so a large batch, a slow source or a retry does not hold a connection open. Each item names one of the API's own operations and carries that operation's normal JSON body, so the same metering, provenance and error contract apply as on a direct call. You poll the job, receive a signed webhook when it finishes, or have it uploaded to your own S3 or GCS bucket, once or on a schedule.

Every route takes the same API key as the rest of the API. A job belongs to the account: any active key of that account can read or cancel it, and a key from another account gets 404, which never confirms that the id exists.

Name the work#

GET /v1/async/operations lists every operation that can run asynchronously, with its operation id, method, path and credit cost. Only operations with a price can be submitted; account, billing and job-management routes cannot. An item names its operation in one of three ways:

{"operation": "google_calendar", "body": {"token": "ChUIoben2Mv6-CYaCi9tLzA3czVwbjQQAQ", "days": 90}}
{"operation": "POST /v1/calendar", "body": {"token": "ChUIoben2Mv6-CYaCi9tLzA3czVwbjQQAQ", "days": 90}}
{"path": "/v1/calendar", "method": "POST", "body": {"token": "ChUIoben2Mv6-CYaCi9tLzA3czVwbjQQAQ", "days": 90}}

body is exactly what the synchronous endpoint takes; query carries query-string parameters for a GET operation, and label is echoed back on the item.

Submit a job#

Put one operation's fields at the top level, or send items with 1 to 500 of them. A job also takes priority (0 to 9, default 5; first in, first out within a priority), meta (echoed on every status response and the webhook), max_credits, a callback_url and a destination_id.

curl -X POST "https://api.scrapercompany.com/v1/async/jobs" \
  -H "x-api-key: $SCRAPERCOMPANY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "google_calendar",
    "body": {"token": "ChUIoben2Mv6-CYaCi9tLzA3czVwbjQQAQ", "days": 90, "currency": "USD", "market": "US"},
    "callback_url": "https://example.com/webhooks/scrapercompany",
    "meta": {"batch": "q3-comp-set"}
  }'

The answer is 202 with the stored job. It is written before any work starts, so a restart on our side does not lose the submission.

{
  "job_id": "asyncjob_0123456789abcdef0123456789abcdef",
  "status": "queued",
  "priority": 5,
  "status_url": "/v1/async/jobs/asyncjob_0123456789abcdef0123456789abcdef",
  "poll_url": "/v1/async/jobs/asyncjob_0123456789abcdef0123456789abcdef",
  "callback_url": "https://example.com/webhooks/scrapercompany",
  "callback_status": "pending",
  "meta": {
    "batch": "q3-comp-set"
  },
  "max_credits": 100,
  "credits_used": 0,
  "items_total": 2,
  "items_succeeded": 0,
  "items_failed": 0,
  "items": [
    {
      "index": 0,
      "operation": "POST /v1/calendar",
      "label": "Boston Marriott Copley Place",
      "status": "queued",
      "attempts": 0,
      "credits": 0
    },
    {
      "index": 1,
      "operation": "POST /v1/offers",
      "label": "Same property, seller by seller",
      "status": "queued",
      "attempts": 0,
      "credits": 0
    }
  ]
}

Poll and cancel#

GET /v1/async/jobs/{job_id} returns the job and each item's status (queued, running, succeeded, failed, cancelled), attempts, http_status, credits, request_id and error. Add include_results=true for each item's response body. The job is succeeded when every item succeeded, partial when some did, and failed when none did. A failed item carries the route's own error and HTTP status, and never hides the others.

curl "https://api.scrapercompany.com/v1/async/jobs/asyncjob_0123456789abcdef0123456789abcdef?include_results=true" \
  -H "x-api-key: $SCRAPERCOMPANY_API_KEY"

{
  "job_id": "asyncjob_0123456789abcdef0123456789abcdef",
  "status": "succeeded",
  "credits_used": 8,
  "items_total": 1,
  "items_succeeded": 1,
  "items_failed": 0,
  "delivery_status": "delivered",
  "delivery_uri": "s3://acme-rate-archive/scrapercompany/async-jobs/2026-10-22/asyncjob_0123456789abcdef0123456789abcdef/manifest.json",
  "finished_at": "2026-10-22T14:05:02Z",
  "items": [
    {
      "index": 0,
      "operation": "POST /v1/calendar",
      "label": "Boston Marriott Copley Place",
      "status": "succeeded",
      "attempts": 1,
      "http_status": 200,
      "credits": 8,
      "request_id": "req_0123456789abcdef0123456789abcdef",
      "result": {
        "token": "ChUIoben2Mv6-CYaCi9tLzA3czVwbjQQAQ",
        "currency": "USD",
        "days": 90,
        "rates": [
          "…"
        ]
      }
    }
  ]
}

GET /v1/async/jobs lists the account's jobs newest first, without item bodies (limit up to 100, then cursor from next_cursor). DELETE /v1/async/jobs/{job_id} stops queued and running items: work that had not finished is not billed, a charge already taken for an item that had started is returned, and the completion webhook is still sent. Cancelling a job that already finished is 409.

A stored item body is capped at 4 MB (a larger one is replaced by {"truncated": true, …}) and a job's stored results at 32 MB (later items keep their status and credits but store {"omitted": true, …}). Jobs and their results are deleted after seven days.

Retries and cost control#

Only retryable failures are retried: HTTP 408, 425, any 5xx and transport errors, up to three attempts with a backoff of 1, 2, then 4 seconds. A 4xx input error is final. Every item counts against your key's per-minute rate limit like a direct call; a 429 waits and does not use up the item's attempts, so a large batch spreads over time.

max_credits is a reservation, not a check after the fact: an item starts only when the credits already spent, plus the listed price of the items in flight, plus this item's listed price fit under it. The listed price is the synchronous cost, and the charge is never higher. Items that do not fit are cancelled and the job reports max_credits exceeded. When your account's credits are enforced, a batch the account cannot afford is refused at submission with 402.

Webhooks#

Set callback_url (or "callback": {"type": "webhook", "url": "https://…"}) to a public HTTPS URL. One signed async_job.completed event is POSTed when the job reaches a terminal state, cancellation included. items is a summary: fetch results_url for the bodies, or set include_results: true on the job to have them in the webhook when they fit in 256 KB. delivery_uri is present only after a bucket upload.

{
  "event": "async_job.completed",
  "job_id": "asyncjob_0123456789abcdef0123456789abcdef",
  "status": "succeeded",
  "items_total": 2,
  "items_succeeded": 2,
  "items_failed": 0,
  "credits_used": 18,
  "poll_url": "/v1/async/jobs/asyncjob_0123456789abcdef0123456789abcdef",
  "results_url": "/v1/async/jobs/asyncjob_0123456789abcdef0123456789abcdef?include_results=true",
  "delivery_uri": "s3://acme-rate-archive/scrapercompany/async-jobs/2026-10-22/asyncjob_0123456789abcdef0123456789abcdef/manifest.json",
  "finished_at": "2026-10-22T14:05:02Z",
  "meta": {
    "batch": "q3-comp-set"
  },
  "items": [
    {
      "index": 0,
      "operation": "POST /v1/calendar",
      "status": "succeeded",
      "credits": 8
    },
    {
      "index": 1,
      "operation": "POST /v1/offers",
      "status": "succeeded",
      "credits": 10
    }
  ]
}

Verify the signature#

Read the raw body before parsing JSON, compare each comma-separated value in constant time, and reject a timestamp more than five minutes off.

import hashlib, hmac, time

def verify(secret: str, headers, body: bytes, tolerance: int = 300) -> bool:
    timestamp = headers["x-scrapercompany-timestamp"]
    signature = headers["x-scrapercompany-signature"]
    if abs(time.time() - int(timestamp)) > tolerance:
        return False
    expected = "sha256=" + hmac.new(
        secret.encode(), timestamp.encode() + b"." + body,
        hashlib.sha256).hexdigest()
    ok = False
    for part in signature.split(","):
        ok = hmac.compare_digest(expected, part.strip()) or ok
    return ok

Which secret signs it#

• If the callback URL is an enabled HTTPS endpoint you registered for webhooks on your account, the event is signed with that endpoint's own whsec_… secret.
• Any other callback URL is signed with the account callback secret. GET /v1/async/callback-secret returns {secret, created_at, rotated_at}; the secret is created the first time you read it and can be read again. The dashboard's Jobs page shows the same value.
• POST /v1/async/callback-secret/rotate replaces it and keeps the previous secret signing for 24 hours. During that window the signature header is sha256=<current>,sha256=<previous>, current first.

A schedule's runs use the callback URL stored on the schedule and the same rule. Delivery is attempted up to three times. Submitting a callback while signing is unavailable on our side is 503.

Deliver to S3 or GCS#

Save a bucket once with POST /v1/async/destinations, then pass its id as destination_id on a job or a schedule. kind: "s3" covers AWS and any S3-compatible store (R2, B2, MinIO: set endpoint, and path_style where needed); kind: "gcs" takes a Google Cloud Storage HMAC key and always writes to https://storage.googleapis.com. Credentials are encrypted at rest and never returned: reads show the last four characters of the access key.

curl -X POST "https://api.scrapercompany.com/v1/async/destinations" \
  -H "x-api-key: $SCRAPERCOMPANY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "nightly-comp-set",
    "kind": "s3",
    "bucket": "acme-rate-archive",
    "prefix": "scrapercompany",
    "region": "us-east-1",
    "format": "jsonl",
    "access_key_id": "AKIAEXAMPLE123456",
    "secret_access_key": "replace-with-the-secret"
  }'

format is json (the default), jsonl or csv. Each item is one gzipped object, and a small manifest lists them; delivery_uri points at the manifest:

<prefix>/async-jobs/<YYYY-MM-DD>/<job_id>/manifest.json
<prefix>/async-jobs/<YYYY-MM-DD>/<job_id>/items/<index>.<format>.gz

A finished job is uploaded once, whatever its scrape status, and a bucket outage never changes what was collected. The job reports delivery_status (not_configured, pending, delivered or failed), delivery_uri and delivery_error, a short category such as credentials rejected, bucket not found or could not be reached. Pending and failed uploads are retried in the background. Deleting a destination stops future uploads; jobs and schedules that referenced it keep running.

The key only needs to create objects under the prefix, not to list or delete. On AWS, for prefix scrapercompany/:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::acme-rate-archive/scrapercompany/*"
    }
  ]
}

For GCS, create the HMAC key on a service account that holds only storage.objects.create on the bucket. Saving an 11th destination, or a second one with the same name, is 409.

Schedules#

POST /v1/async/schedules stores a job template and a cadence. The body is the same as a job submission plus name, cadence, run_hour, weekday, timezone and enabled. Each due slot becomes an ordinary async job with the same metering, retries, webhook and delivery, and reports its schedule_id.

curl -X POST "https://api.scrapercompany.com/v1/async/schedules" \
  -H "x-api-key: $SCRAPERCOMPANY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "nightly-comp-set",
    "cadence": "daily",
    "run_hour": 6,
    "timezone": "America/New_York",
    "destination_id": "asyncdst_0123456789abcdef0123456789abcdef",
    "items": [
      {"operation": "google_calendar",
       "body": {"token": "ChUIoben2Mv6-CYaCi9tLzA3czVwbjQQAQ", "days": 90, "currency": "USD", "market": "US"}}
    ],
    "max_credits": 100,
    "meta": {"batch": "nightly"}
  }'

• cadence is hourly, every_6h, twice_daily, daily (the default), weekly or manual. daily and weekly run at run_hour in the IANA timezone (default UTC); weekday 0 is Monday. hourly is the fastest, and manual only runs when you ask. Missed slots are skipped, never replayed in a burst.
• PUT /v1/async/schedules/{schedule_id} replaces the whole schedule and recomputes the next run; enabled: false pauses it.
• POST /v1/async/schedules/{schedule_id}/run creates one job now without changing the cadence, and answers 202 with that job.
• DELETE /v1/async/schedules/{schedule_id} stops future runs; jobs already created keep running.
• The schedule shows next_run_at, last_run_at, last_job_id and last_error. An 11th schedule, a name already in use, an unknown timezone or an unknown destination_id is 422.

Errors#

Field-level details are on each endpoint's page under API reference → Async jobs. For calendar-only batches with a required idempotency key, see Monitor a comp set.