Scrapinghub API Reference

Logs API

The logs API lets you work with logs from your crawls.

Log object

Field Description Required
message Log message. Yes
level Integer log level as defined in the table below. Yes
time UNIX timestamp of the message, in milliseconds. No

Log levels

Value Log level
10 DEBUG
20 INFO
30 WARNING
40 ERROR
50 CRITICAL

logs/:project_id/:spider_id/:job_id

Retrieve or upload logs for a given job.

Parameter Description Required
format Results format. See Result formats. No

Note

Pagination and meta parameters are supported, see Pagination and Meta parameters.

Method Description Supported parameters
GET Retrieve logs. format
POST Upload logs.  

Retrieving logs

HTTP:

$ curl -u APIKEY: https://storage.scrapinghub.com/logs/1111111/1/1/
{"time":1444822757227,"level":20,"message":"Log opened."}
{"time":1444822757229,"level":20,"message":"[scrapy.log] Scrapy 1.0.3.post6+g2d688cd started"}

Submitting logs

HTTP:

$ curl -u APIKEY: https://storage.scrapinghub.com/logs/53/34/7 -X POST -T log.jl