1

I want to retrieve content from web page. However, I tried above method but the error still come when the query string contain Chinese character.

code


$json = Get-Content -Encoding utf8 -Path "./kw.json" | ConvertFrom-Json

$kw = $json.name
$numPages = $json.page
# $encodingKW = [System.Text.Encoding]::UTF8.GetString([System.Text.Encoding]::GetEncoding("UTF-8").GetBytes($kw))
Write-Output $encodingKW

for ($i = 0; $i -lt $numPages; $i ++) {
    $pn = $i * 50
    
    [string]$url = "https://tieba.baidu.com/f?kw=${kw}&ie=utf-8&pn=${pn}"
    # https://tieba.baidu.com/f?kw=%E5%88%80%E5%8A%8D%E7%A5%9E%E5%9F%9F&ie=utf-8&pn=0
    
    Write-Output $url
    $html = Invoke-WebRequest -Method Get -Uri $url
    # $content = [System.Text.Encoding]::UTF8.GetString([System.Text.Encoding]::GetEncoding("iso-8859-1").GetBytes($html.Content))

    Write-Output $html
    break;
}
# [string]$url = "https://tieba.baidu.com/f?kw=${kw}"

# Write-Output $json.name
# Write-Output $json.page
# Write-Output $url

The url is

https://tieba.baidu.com/f?kw=地理&ie=utf-8&pn=0

It give me Invoke-WebRequest error. I think the url is successfully encoding since the url is in encoding form ,backurl:'https://tieba.baidu.com/f?kw=%E5%9C%B0%E7%90%86&ie=utf-8&pn=0',.

But the error is

$html = Invoke-WebRequest -Method Get -Uri $url CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest] WebException FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

And Is there alternative method as python url = baseurl+"f?kw="+parse.quote(kw) + "&ie=utf-8&pn=" + str(i * 50)?

3
  • Your primary problem is one of lack of authentication, not encoding; the response is a 403 (Forbidden)` error response. Commented Nov 14 at 17:58
  • I see. So I need to set cookies? Commented Nov 15 at 2:29
  • The Invoke-WebRequest cmdlet has authentication-related parameters, driven by the -Authentication parameter. Commented Nov 15 at 2:40

0

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.