v1.2.12026-09-19
本次更新主要围绕文档站的发布流程展开。推送 v 标签时,CI 会自动构建文档站镜像 ghcr.io/any4ai/anycrawl-docs(含 :latest 与 :<tag>,支持 linux/arm64),并在配置好 COOLIFYDOCSDEPLOYURL 和 COOLIFYAPITOKEN 两个仓库密钥后,自动触发 docs.anycrawl.dev 的部署。手动触发工作流可重建 main(打 :sha-<短哈希> 标签)或指定标签,其他分支一律拒绝。
apps/docs/Dockerfile 也做了精简,只安装文档包自身的依赖,产出独立运行的非 root 镜像并带健康检查;独立输出仅在 DOCS_STANDALONE=1 时启用,next dev/next start 行为不变。
升级方面无需任何操作:服务端代码与数据库结构均未改动,自托管部署可直接跳过。服务端 workspace 包升至 1.2.1,独立版本的 JS SDK 仍为 0.0.9。
v1.2.02026-09-19
1.2.0:数据集保留策略改为读取时过滤
retentionpolicy 现在以读取时视图的方式生效:itemdays 隐藏窗口期外未活跃的条目及其之前创建的导出,changedays 隐藏超期的变更和运行告警,活跃条目始终可见。只隐藏不删除,调整或清空策略后下次读取即生效,可随时回退。itemcount 按可见条目计数,被隐藏的导出不再返回下载地址。
文档补齐了托管版限额说明:Scrape、Search、Map 标注 429 Concurrency limit reached,Scrape、Search 另标注 403 stealthproxynot_allowed;Proxy Rules 列出各套餐可用的代理模式,Monitors 新增「Plan Limits (Hosted Service)」章节。
升级须知:无新数据库迁移,API 与 workers
v1.1.02026-09-19
Added
- Per-plan limits for the hosted service — concurrency, stealth proxy access, monitor count and minimum monitor check interval are enforced per API key tier. Controlled by ANYCRAWLAPIPLANLIMITSENABLED, which follows the existing ANYCRAWLAPIAUTHENABLED / ANYCRAWLAPICREDITSENABLED convention: unset means off, so self-hosted deployments see no behaviour change. ANYCRAWLPLANLIMITS_JSON overrides the default table per deployment and is validated at boot.
| Tier | Concurrency | Stealth proxy | Monitors | Fastest monitor check |
|---|---|---|---|---|
| free | 2 | no | 1 | every 6 hours |
| hobby | 5 | yes | 10 | hourly |
| pro | 20 | yes | 50 | every 15 minutes |
| business | 50 | yes | 200 | every 15 minutes |
AI formats (json, summary) remain available on every tier.
Fixed
- Default missing GeoIP metadata (timezone to UTC, locale to en-US) after the proxy exit IP resolves, instead of failing the browser launch. The same exit IP is reused for WebRTC; a proxy gateway address is no longer substituted for an unobserved exit IP.
- Isolate browser startup failures to the failing request, respect retry budgets, and stop terminal task states from being overwritten. Queue handoffs and crawl seed counts are deduplicated.
- Copy worker dependency patches to the workspace root in every Docker install stage, so worker images resolve the CloakBrowser patch.
- Auto-crawl reads crawl controls and nested page options from the normalized API schema output, keeps each child crawl bounded to one page, and respects same-hostname selection.
- Crawl sends flat page options to scrape workers, which consume them flat; nested options were silently ignored.
Upgrade notes
- Plan limits are off unless you set ANYCRAWLAPIPLANLIMITSENABLED=true together with ANYCRAWLAPIAUTH_ENABLED=true. Self-hosted deployments that set neither need no action.
- When enabled, concurrency is tracked in Redis on a dedicated fail-fast connection. A Redis outage lets requests through rather than blocking them.
- Plan checks run on resolved options, so they also cover monitor targets, scheduled-task payloads and template reqOptions, not only direct request bodies.
- Monitor frequency floors are evaluated in UTC, so a wall-clock schedule in a DST timezone is not rejected for the hour the clock moves.
- No new database migrations. Server workspace packages move to 1.1.0; the independently versioned JS SDK remains at 0.0.9.
v1.0.02026-09-09
这版为爬虫代理层引入了可选的粘性代理生命周期管理:浏览器会话有独立身份、统一 TTL 与租约准入规则,正常上下文可复用,让同一会话在多请求间保持稳定。同时补上了浏览器评分工具和 CloakBrowser 运行时的回归测试。
修复集中在失败处理上。内容出错不再误伤健康的浏览器身份,能区分源站限制与代理故障,回退也严格限制在已配置权限内;同一上下文同一源站的 Cloudflare 验证改为共享,各页面各自恢复。抽取前会先识别可观测的内容故障,允许一次受控的同会话重试,并让 HTML 及其衍生格式共用同一份校验过的快照。恢复过程中代理认证与传输错误会被保留,缓存载荷不再外泄校验元数据。
升级须知:粘性管理默认关闭,启用需代理用户名包含 {sessionId},且供应商会话窗口要覆盖完整请求预算加安全余量。共享站点状态必须用普通上下文,该模式会接管浏览器租约并禁用 Crawlee 的 SessionPool 与 cookie 重放;单次会话失败不会拉黑整个代理模板。相对 beta.37 无数据库迁移,服务端包升至 1.0.0,JS SDK 仍为 0.0.9。
回归套件 393 通过、7 项可选跳过。ScienceDirect 验证覆盖 50 请求矩阵与 20 分钟身份复用,但 120 分钟供应商寿命未完整观测;此前偶发的 79 字符响应未再出现,诱因仍未确认。
v1.0.0-beta.372026-09-07
Highlights
- Persist monitoring checks and notification intents across queue failures and worker restarts, serialize checks through post-processing, and recover unfinished work from the database.
- Add cursor pagination, an owner-scoped changes feed, full snapshot details, and check/notification history to the monitoring API and JS SDK 0.0.9.
- Add Facebook, Instagram, and YouTube scraper templates, with validated PostgreSQL/SQLite SQL generation and template registration tooling.
- Add per-request Playwright humanization using CloakBrowser, including default/careful presets and pointer warm-up.
Fixes
- Report search upstream parameter rejection as HTTP 400 (SEARCHINVALIDREQUEST) and other upstream/transport failures as HTTP 502 (SEARCHUPSTREAMERROR) instead of successful empty results. Await page callbacks and prevent partial searches from dispatching follow-up scrapes or being charged.
- Repair output-format and LLM prompt/schema test contracts, validate actual JSON output, and use a controlled real HTTP fixture for status handling while retaining external browser/proxy/LLM checks.
- Enforce webhook owner isolation, validate the final merged monitor configuration, preserve configuration revisions and complete comparison content, and report unavailable AI judgments as unknown.
- Recover email and webhook delivery using stable IDs and retryable intent records; protect pending work during retention and prevent hidden checks from orphaned or inactive managed tasks.
- Make SQLite/PostgreSQL transactions and migrations consistent, including the SQLite job-billing schema repair.
- Declare the missing direct monitor-test dependency and isolate production compilation and Docker context from test files while retaining full type checks and tests.
Upgrade notes
- Search clients must handle explicit upstream errors; successful searches with no matches still return an empty array. Multi-page searches fail if any page fails, and failed requests are not charged.
- Stop and drain old monitoring producers, back up and migrate the target database, then start the new API, Scheduler, and Workers together. Do not run old and new monitoring producers concurrently.
- New migrations: PostgreSQL 0026monitorworkflow.sql; SQLite 0021monitorworkflow.sql and 0022repairsqlitejobbilling.sql. Use pnpm --filter @anycrawl/db db:migrate from source, or the container's db:migrate:docker entrypoint.
- Historical alerts are not replayed; incomplete historical snapshots remain visible but are not used as new comparison baselines. Keep the new workflow tables if rolling back.
- Monitor country pinning is explicitly unsupported and new country requests return HTTP 400. ignore_selectors filters literal text lines. Email alerts require recipients; price/JSON/mixed monitoring requires an extraction schema.
- Cursor-pagination methods in JS SDK 0.0.9 require the updated server; the existing array-returning history methods remain available.
Docker image: ghcr.io/any4ai/anycrawl:v1.0.0-beta.37 (amd64/arm64). Separate API, Cheerio, Playwright, and Puppeteer images use the same version tag.
The separate Puppeteer image is amd64-only under the existing build configuration. The All-in-One arm64 image is built with Puppeteer disabled; use its Playwright or Cheerio engine.
Full comparison: https://github.com/any4ai/AnyCrawl/compare/v1.0.0-beta.36...v1.0.0-beta.37
What's Changed
* fix: harden monitoring and include scraper runtime updates by @QThans in https://github.com/any4ai/AnyCrawl/pull/99
* fix(search): surface upstream errors and release beta.37 by @QThans in https://github.com/any4ai/AnyCrawl/pull/100
Full Changelog: https://github.com/any4ai/AnyCrawl/compare/v1.0.0-beta.36...v1.0.0-beta.37
v1.0.0-beta.36预发布2026-08-11
anyCrawl 1.0.0-beta.36 更新:
模板运行(L3)上线:新增模板专用接口,支持异步编排运行,涵盖修订管理、配置契约、inputs/urlmode 与 returnmodes 输出,配套 OrchestratedRunner(max_concurrency)、template-run worker 及 run-request/run-event 账本。
v1.0.0-beta.352026-08-05
本次更新主打批量抓取。新增异步的 Batch Scrape API:用一次任务提交大量已知 URL,共享同一套抓取参数,包含 POST /v1/batch/scrape 及查询状态、取结果、取消任务三个配套接口,返回结构与单条 POST /v1/scrape 独立。计费按成功抓取的 URL 逐条扣减,失败不收费,余额不足时会在预检阶段直接返回 402。配合 templateid 和 variables,模板可套用到每个 URL,同时支持变量映射、默认值、逐条变换与域名白名单。任务全生命周期提供六种 Webhook 事件,方便外部系统跟踪进度。并发与上限通过环境变量调节:ANYCRAWLBATCHSCRAPEMAXURLS(默认 10000)、ANYCRAWLBATCHSCRAPECONCURRENCY(默认 10)、ANYCRAWLBATCHSCRAPEWAITBUFFER_MS(默认 60000)。
JS SDK 同步发布 @anycrawl/js-sdk 0.0.7,提供 batchScrape()(创建、轮询、汇总一条龙)及各步骤的独立方法。
另外修复了 QueueManager 的 Redis 异常处理,清理任务轮询逻辑,避免监听器和定时器泄漏。
v1.0.0-beta.332026-07-13
这次 1.0.0 版本把此前 alpha/beta 阶段的零散改动一并收拢,核心能力有明显补齐。抓取方面:新增 File API,可从 S3 或本地存储取出文件;html 转 markdown 效果增强;错误处理统一改进,响应支持 json 格式,抓取引擎的错误日志也更清晰。计费逻辑重做——只有抓取成功才扣积分,CrawlController 加了额度校验,避免额度不足仍发起任务。代理支持更细粒度的高级配置,截图路径改为挂上公网域名路由,方便直接访问。AI 侧接入了模型调用并完善 LLMExtract,可按结构抽取内容。进度管理支持任务入队,搜索能力也有优化。另外重构了 index.ts 的导出结构,整理了各包依赖与版本号,并修复了一批 js-sdk 的问题:crawl 选项重新分层封装抓取参数,README 与类型定义同步更新。
v1.0.0-beta.322026-06-30
本次更新集中在功能补全与计费、SDK 三条线上。抓取侧:新增 File API,可从 S3 或本地存储取回文件;html 转 Markdown 的效果进一步优化;截图路径统一加上公网域名前缀,返回的链接可直接访问;搜索引擎能力有所改进。AI 侧:接入 AI 模型并完成 LLMExtract,错误处理与响应格式也做了强化,支持 JSON 输出。代理方面新增高级配置项,可按需指定更细粒度的代理策略。计费逻辑修正了两个实际问题——只有抓取成功才扣减额度,且 Crawl 接口会在执行前校验用户余额,避免超额任务。调度上,BaseEngine 与 ProgressManager 支持任务入队。JS SDK 同步重构,crawl 选项改为封装式结构,类型定义和 README 一并更新,并修复了多处缺陷。版本号已推进至 1.0.0-beta.4,各子包版本统一。
v0.0.1-beta.202026-03-06
v0.0.1-beta.20 版本更新内容可参见完整变更日志。