Monitor a comp set with jobs and webhooks
Queue calendar collection for up to 50 properties, poll or receive a signed webhook, and schedule it daily.
Overview
A calendar job collects forward calendars for up to 50 properties in one request. The job is stored before the API answers, runs in the background, survives restarts, and can call your webhook when it finishes.
Calls | |
Cost | Each item that succeeds is priced like POST /v1/calendar, surcharges included (5 for a default item), charged as it finishes; submitting and polling are free |
Limits | 1–50 items per job; each item takes the same fields as POST /v1/calendar plus an optional label |
Submit a job
Every submission needs an Idempotency-Key header. Reusing a key with the same body returns the original job (safe to retry after a timeout); reusing it with a different body returns 409. A daily run can use a key like compset-2026-10-14.
curl -X POST "https://api.scrapercompany.com/v1/jobs" \
-H "x-api-key: $SCRAPERCOMPANY_API_KEY" \
-H "Idempotency-Key: compset-$(date +%F)" \
-H "Content-Type: application/json" \
-d '{
"items": [
{"token": "ChUIoben2Mv6-CYaCi9tLzA3czVwbjQQAQ", "label": "Hilton Chicago", "days": 90, "currency": "USD", "market": "US"},
{"token": "ChcI7LnnsZLasKYOGgsvZy8xdHJzejFiORAB", "label": "Comp hotel 2", "days": 90, "currency": "USD", "market": "US"}
],
"callback_url": "https://example.com/webhooks/scrapercompany"
}'Poll for results
Poll poll_url until status is terminal: succeeded, partial (some items failed) or failed. While it runs it is queued or running. Add include_results=true to get each item's calendar in items[].result (the same shape as a calendar response).
import time
while True:
r = requests.get(f"{API}{job['poll_url']}", headers=HEADERS, params={"include_results": "true"}, timeout=60)
r.raise_for_status()
job = r.json()
if job["status"] in ("succeeded", "partial", "failed"):
break
time.sleep(10)
print(job["status"], job["items_succeeded"], "of", job["items_total"])
for item in job["items"]:
if item["status"] == "succeeded":
rates = (item.get("result") or {}).get("rates", [])
print(item["label"], len(rates), "nights, cheapest", min(row["rate"] for row in rates) if rates else None)
else:
print(item["label"], item["status"], item["error"])Receive a webhook
Add callback_url (a public HTTPS URL) and the API POSTs a job.completed event when the job finishes, retrying up to 5 times with backoff. The event is a summary; fetch the results from poll_url.
{
"event": "job.completed",
"finished_at": "2026-10-14 06:00:41.512+00",
"items_failed": 0,
"items_succeeded": 12,
"items_total": 12,
"job_id": "refreshjob_0123456789abcdef0123456789abcdef",
"poll_url": "/v1/jobs/refreshjob_0123456789abcdef0123456789abcdef",
"status": "succeeded"
}| Header | Value |
|---|---|
x-scrapingme-event | job.completed |
x-scrapingme-delivery | The job id (use it to de-duplicate deliveries). |
x-scrapingme-timestamp | Unix seconds when the delivery was signed. |
x-scrapingme-signature | sha256= + hex HMAC-SHA256 of {timestamp}.{raw body} |
# Flask example
import hashlib
import hmac
import os
import time
import requests
from flask import Flask, abort, request
app = Flask(__name__)
SIGNING_SECRET = os.environ.get("SCRAPERCOMPANY_WEBHOOK_SECRET") # optional, from support
seen = set()
@app.post("/webhooks/scrapercompany")
def job_completed():
raw = request.get_data()
timestamp = request.headers.get("x-scrapingme-timestamp", "0")
if abs(time.time() - int(timestamp)) > 300:
abort(400)
if SIGNING_SECRET:
expected = "sha256=" + hmac.new(SIGNING_SECRET.encode(), timestamp.encode() + b"." + raw, hashlib.sha256).hexdigest()
if not hmac.compare_digest(expected, request.headers.get("x-scrapingme-signature", "")):
abort(401)
delivery = request.headers.get("x-scrapingme-delivery")
if delivery in seen:
return "", 204
seen.add(delivery)
event = request.get_json()
job = requests.get(
f"https://api.scrapercompany.com{event['poll_url']}",
headers={"x-api-key": os.environ["SCRAPERCOMPANY_API_KEY"]},
params={"include_results": "true"},
timeout=60,
).json()
print(job["status"], job["items_succeeded"], "items")
return "", 204Run it every day
Schedule the submit script once a day (cron, a CI schedule or your job runner) and use the date in the Idempotency-Key so accidental double runs don't create two jobs:
# 06:00 UTC every day
0 6 * * * cd /srv/compset && python submit_job.py >> job.log 2>&1- Split comp sets larger than 50 properties into several jobs with distinct keys.
- Watch
items_failed: failed items are free, and resubmitting just those tokens (with a new key) is cheaper than rerunning the whole set. - Stored rates from jobs can be read back later without new collection via GET/v1/rates/stored (free).