Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use accept-encoding instead of content-encoding for requests favoring gzip #250

Closed
wants to merge 1 commit into from

Conversation

shunf4
Copy link

@shunf4 shunf4 commented Jun 28, 2024

Per https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding , a request accepting gzip response ususally uses Accept-Encoding header, not Content-Encoding (which is used by the response).

Such request with invalid request header is forbidden by some mirrror sites:

$ curl -L --http1.1 -H 'content-encoding: gzip' -H 'content-type: application/json' -i 'https://registry.npmmirror.com/tar/-/tar-6.2.1.tgz'
HTTP/1.1 400 Bad Request
Server: Tengine
Content-Type: text/html; charset=utf-8
Content-Length: 24
Connection: keep-alive
Strict-Transport-Security: max-age=5184000
Date: *
Vary: Origin
Via: *
Ali-Swift-Global-Savetime: *
X-Cache: MISS TCP_MISS dirn:-2:-2
X-Swift-Error: orig response 4XX error
X-Swift-SaveTime: *
X-Swift-CacheTime: 0
Timing-Allow-Origin: *
EagleId: *

<h2>400 Bad Request</h2>

The original code of commit 340abe0 is likely a typo.

…oring gzip

Per https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding ,
a request accepting gzip response ususally uses `Accept-Encoding`
header, not `Content-Encoding` (which is used by the response).

The original code is likely a typo.
@shunf4 shunf4 requested a review from a team as a code owner June 28, 2024 13:50
@wraithgar
Copy link
Member

For the response body, the underlying minipass-fetch handles the accept headers, and unzipping if needed.

@wraithgar
Copy link
Member

wraithgar commented Jun 28, 2024

I don't think this is a mistake. This is for sending a gzipped request. The rest of the if statement is gzipping the request body.

@wraithgar wraithgar closed this Jun 28, 2024
@shunf4
Copy link
Author

shunf4 commented Jun 28, 2024

Yes, also realized that almost the same time you replied.

This PR came about as I was looking into an error message complained by npm install using a http proxy. The header content-encoding: gzip was sent in a fetch to https://registry.npmmirror.com/tar/-/tar-6.2.1.tgz, causing the 400 response, but it's not the case when no http proxy was set.

Digging into the code I found that the request to https://registry.npmmirror.com/tar/-/tar-6.2.1.tgz was never meant to be using gzip: true, nor the content-encoding: gzip header ever passed in opts.header field. It's the agent, that (unnecessarily and unwantedly?) contained the content-encoding: gzip header passed from the first request (which was the one meant to be using gzipped request body), and reused by caching mechanism for following requests. However, agent.headers will be picked up if and only if an http proxy is configured (https://github.com/npm/agent/blob/21c19874834fb00c7ab37268b385fb84deb2df04/lib/agents.js#L175 ).

The most obvious fix seems to be removing headers field in agent's normalizeOptions; an agent is not expected to remember the headers, in usual sense.

I've created npm/agent#105 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants