4
$\begingroup$

Background

Taking an example from Decoding GZIP encoded Body, BodyBytes (ByteArray) and BodyBytesArray from URLRead:

URLRead[
 "https://api.stackexchange.com/2.2/info?site=mathematica", "Headers"
]

{ ... 
, "content-type" -> "application/json; charset=utf-8"
, "content-encoding" -> "gzip"
, ...
}

Import @ "https://api.stackexchange.com/2.2/info?site=mathematica"

returns string json which can be later put to ImportString.

Problem

URLRead though throws a bunch of decoding errors suggesting it ignores "gzip" spec and goes directly to charset specified in content-type.

URLRead[
 "https://api.stackexchange.com/2.2/info?site=mathematica", "Body"
]

Workaround

is already shown in linked topic:

URLRead[
    "https://api.stackexchange.com/2.2/info?site=mathematica"
  , "BodyBytes"
] // FromCharacterCode // ImportString[#, {"gzip", "RawJSON"}] &

Question

Should that be the case? Is it a bug or am I missing the purpose of URLRead Body?

URLFetch behaves the same so I'm surprised it wasn't asked before. 

URLFetch[
    "https://api.stackexchange.com/2.2/info?site=mathematica"
  , "Content"
]

Related

Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

Improve this question
$\endgroup$
9
  • 2
    $\begingroup$ Note that Import actually does not need the "content-encoding" -> "gzip" header for recognizing gzip compressed data. You can check it with file=URLDownload["https://api.stackexchange.com/2.2/info?site=mathematica"];Import@file. The created file is a binary gzip-compressed file and Import recognizes the compression method from the first few bytes of the file, the HTTP "content-encoding" header isn't necessary at all. $\endgroup$ Commented Aug 29, 2017 at 14:20
  • 1
    $\begingroup$ @AlexeyPopkov good point, so no one cares about content-encoding :) $\endgroup$ Commented Aug 29, 2017 at 14:21
  • $\begingroup$ I would like to understand this too (+1). We knew that Import worked, from the quoted question.. BTW, "As of Version 11, URLFetch has been superseded by URLRead and URLExecute." $\endgroup$ Commented Aug 29, 2017 at 14:23
  • 1
    $\begingroup$ @rhermans and Alexey, now it works, does it mean it was a bug? $\endgroup$ Commented Oct 17, 2017 at 21:04
  • $\begingroup$ @Kuba I think it was a bug since ability to recognize gzip is of crucial importance for such function as URLRead. As I wrote above, I would expect it to recognize gzip even without the explicit "content-encoding" -> "gzip". Probably the latter can be checked using the file: protocol. $\endgroup$ Commented Oct 18, 2017 at 9:52

1 Answer 1

Reset to default
2
$\begingroup$

As of V11.2 Content-Encoding specification is respected by URLRead and e.g.

URLRead[
  "https://api.stackexchange.com/2.2/info?site=mathematica", "Body"
]

Works well.

For earlier versions one needs to use a workaround like one shown above.

Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.