Scrapinghub Reference Documentation

Crawlera Errors Reference

When an error occurs, Crawlera sends a response containing an error code.

When using the Proxy API, the error code is returned in the X-Crawlera-Error header, with a human-readable description in the response body.

Errors can be classified into two main categories:

  • Bans. These are the errors produced when Crawlera couldn’t find enough unblocked IPs, and are plotted in red in the Crawlera Usage Stats.

  • Failures. These are all the remaining errors, some of them produced by errors on the user side and other produced by errors on Crawlera server side. These are plotted in gray in the Crawlera Usage Stats.

In both of these cases requests don’t count towards your monthly successful quota.

Note

These error codes are subject to change at any time, so they should not be relied on and only used for debugging.

Ban Errors (Not Enough Capacity)

X-Crawlera-Error

Response Code

Error Message

How to fix or debug

banned, noslaves

503

No available proxies to fulfill request.

Trying again usually works. If you keep getting this message, you may need to consider a Crawlera Enterprise plan with dedicated capacity.

Failure Errors (User Side)

X-Crawlera-Error

Response Code

Error Message

How to fix or debug

bad_session_id

400

Bad session ID

Create a new session, this one probably expired.

user_session_limit

400

Session limit exceeded

Create less sessions or upgrade your plan.

invalid_request

400

Invalid request (typically malformed)

The client-side technology you’re using with Crawlera may have some incompatibilities. Contact our support team for help.

bad_uri

401

Malformed URL

Check the URL you’re passing to Crawlera. Contact our support team if you need help.

user_suspended

401

User account suspended

Extend the overage limit in your account through the Billing section

bad_proxy_auth

407

Incorrect authentication data

Make sure you’re using the right Crawlera credentials.

too_many_conns

429

The number of parallel connections has exceeded the concurrency limit.

Reduce concurrency or upgrade your Crawlera plan

header_auth

470

Unauthorized header

You’re using a Crawlera header (or header value) that is not allowed in your current plan. Consider upgrading to a higher plan or contact our support team for alternatives.

client_conn_closed

499

Connection closed early by client

Check the timeout used when connecting to Crawlera from your code. It may need to be increased.

Failure Errors (Server Side)

X-Crawlera-Error

Response Code

Error Message

How to fix or debug

nxdomain

502

Domain not found

Check the domain of the URL you’re trying to fetch, it may be wrong.

ehostunreach

502

Host is unreachable

The domain you’re trying to access may be down. Contact our support team if this is not the case.

econnrefused

502

Targey website is refusing connection

Check the website works in your browser, and contact support it if does.

econnreset

502

Connection with website lost

Trying again usually works. unless the target website is having some issues. If the issue persists, contact our support team.

socket_closed_remotely

502

Target website closed the connection

Trying again usually works, unless the target website is having some issues. If the issue persists, contact our support team.

ssl_tunnel_error

502

Failed to initialize SSL tunnel. This contemplates both client SSL errors trying to connect to Crawlera and Crawlera trying to connect to target websites.

Make sure you have the Crawlera CA file installed in your trusted certificates. Contact support if you still have issues.

serverbusy

503

Server busy: too many outstanding requests

Try again after some minutes

timeout

504

Timeout trying to establishe the TCP connection with the target domain

Check if the URL you’re trying to access actually works, contact support if it does.

msgtimeout

504

Timeout receiving data from the domain, after the TCP connection is established.

Check if the URL works well in your browser, and contact our support team if it does.

domain_forbidden

523

The domain you are trying to access is restricted in Crawlera

Contact support to check if the restriction can be lifted

bad_header

540

Invalid Crawlera header value

Check the value you passed in the X-Crawlera header and make sure it’s valid according to the documentation.

data_error

541

Response size is too big. The maximum response size in Crawlera is 500 Mb.

Fetch a URL that is smaller in size.

bad_endpoint

542

You’re accessing the wrong API endpoint for Crawlera.

If you’re on an Enterprise plan, make sure you’re using your Enteprise hostname (eg. acme.crawlera.com).