Use a separate webclient_context for tunnel establishment.
I chose this way (instead of having tunnelling steps in
the state machine of a single webclient_context) because
I want to allow tunnelling of non-HTTP protocols sooner or later.
Add a primitive API for tunnel establishment.
(WEBCLIENT_FLAG_TUNNEL and webclient_get_tunnel)
I plan to use this to implement https proxy support.
That is, the primary user will be webclient itself.
When making the following change, I haven't noticed that
g_httpuseragentfields contains a "Connection" header.
```
commit 092ce81444
Author: YAMAMOTO Takashi <yamamoto@midokura.com>
Date: Mon Mar 7 09:30:23 2022 +0900
webclient: Always use "connection: close" for HTTP 1.1 for now
* This matches the HTTP 1.0 behavior.
* Persistent connection doesn't make much sense with the current API.
```
It seems that some servers are not happy with the duplicated header
and ignore them. (and do the default keep-alive for HTTP 1.1)
eg. Azure Blob (global)
* webclient_perform
* Add a new flag to use non-blocking mode (WEBCLIENT_FLAG_NON_BLOCKING)
* Implement restarting
* Add a few associated API functions
* webclient_get_poll_info: get the descriptor info for poll/select
* webclient_abort: abort the operation (instead of restarting)
Similarly to the fix in wget_parseheaders.
But simply always bail out as i guess it's very rare to see
that long status line.
Tested with an aritifically small CONFIG_WEBCLIENT_MAXHTTPLINE=20,
which is smaller than "HTTP/1.1 301 Moved Permanently".
* Detect a long header line which doesn't fit the buffer.
* If the header line in question doesn't seem important for us,
just ignore it.
* Otherwise, bail out with -E2BIG.
The overrun was found by LLVM UBSan on macOS.
The following is an example of a long header line,
which doesn't fit the default CONFIG_WEBCLIENT_MAXHTTPLINE=200.
```
pacetanuki% curl -v https://www.midokura.com
:
:
< HTTP/2 200
< server: nginx
< date: Fri, 14 May 2021 02:16:24 GMT
< content-type: text/html; charset=UTF-8
< content-length: 131313
< x-powered-by: PHP/7.4.18
< link: <https://www.midokura.com/wp-json/>; rel="https://api.w.org/", <https://www.midokura.com/wp-json/wp/v2/pages/7>; rel="alternate"; type="application/json", <https://www.midokura.com/>; rel=shortlink
< x-powered-by: PleskLin
<
```
from:
return - ECONNABORTED;
to:
return -ECONNABORTED;
The latter style is more commonly used within NuttX code base.
Note: nxstyle doesn't complain on either of them.
This happened frequently for me with Docker Desktop's
/var/run/docker.sock on macOS.
But I believe it can happen on other environments, even with TCP.
Alternatively, this case can be handled by the callback
implementations. But it's simpler to handle the corner case here.
This fixes the case when webclient_set_static_body is used
for data larger than webclient_context::buflen.
Note: as of writing this, webclient_set_static_body is the
only user of body_callback in NuttX apps tree.
Highlights:
* TLS support (a hook to allow users to provide TLS implementation)
* ability to add extra request headers
* ability to use PUT method
* ability to report http status
* error handling improvements
Proposed on the ML while ago:
https://www.mail-archive.com/dev@nuttx.apache.org/msg03803.html
The original API is kept for now.
I plan to remove them after adapting the existing users.
(examples in this repo)
When recv() failed, the current code assumes the return value
of recv() is a negative errno. It's wrong. Actually the return value
in case of error is -1. The wrong assumption ends up with reporting
EPERM, as EPERM happens to be -(-1).
This commit changes the code to leave the errno set by the failed
recv() as it is. It should be fine as wget_base() has the same
error returning convention as recv(). That is, return -1 (ERROR)
and set errno in the case of failure.
NOTE: the close() after the errout label can also fail and overwrite
the errno. I don't feel it's a big problem as wget_base() doesn't have
any promise about which error should be reported.
http -> https redirection is rather common. The old code was
just broken in that case.
Also, this commit is a step towards https support.
* Switch to netlib_parseurl
* Fix error propagation in wget_parseheaders
* Bail out on a redirect to a URL with unsupported scheme
1.Remove void cast for function because many place ignore the returned value witout cast
2.Replace void cast for variable with UNUSED macro
Change-Id: Ie644129a563244a6397036789c4c3ea83c4e9b09
Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com>
