A new /control/config endpoint provides configuration reload function
with feedback returned to the user on the outcome.
Example:
$ curl 127.0.0.1:1234/1/control/config -X PATCH -v | jq
< HTTP/1.1 200 OK
< ...
<
{
"logs": []
}
$ curl 127.0.0.1:1234/1/control/config -X PATCH -v | jq
< HTTP/1.1 422 Unprocessable Entity
< ...
<
{
"logs": [
"...: invalid number of arguments in \"listen\" directive"
]
}
Co-authored-by: Pavel Pautov <p.pautov@f5.com>
A predicate location name starts with '$' and is interpreted as a
string evaluated at runtime, and its value is used to route the
request into the location. A predicate location matches if the
string evaluates to a non-empty value not equal to "0". At each
nesting level, the first matching predicate location in the
configuration order is chosen. Predicate locations are matched
after prefix and regex locations. They are not matched if an exact
or a regex location matched, or if the longest matching prefix location
has the "^~" modifier. Predicate locations can be nested inside
other location types and can contain nested locations of their own.
Example:
map $arg_foo $pred {
bar 1;
qux 1;
}
location $pred {
root html;
}
When a body filter returned an error, ngx_event_pipe_drain_chains()
returned all buffers to p->free_raw_bufs, including those in p->busy,
which are still owned by the downstream. If upstream EOF was detected
in the same ngx_event_pipe() pass, ngx_event_pipe_read_upstream() then
freed them while still referenced. With HTTP/2 only the stream's fake
connection is marked on error, so ngx_http_v2_send_output_queue() kept
sending DATA frames pointing at the freed buffers, a use-after-free.
Fixed by keeping the p->busy chains intact, the buffers are freed with
the request pool. This also stops the pipe from reusing them for new
upstream reads, which previously could overwrite response body data
still queued for the client. Additionally, the pipe no longer treats
a downstream it has given up on as merely blocked, so the loop is
still guaranteed to terminate.
Co-authored-by: Roman Arutyunyan <arut@nginx.com>
When the byte range is passed in the client Range header, the range
start is accepted up to NGX_MAX_OFF_T_VALUE, while
ngx_http_slice_range_variable() adds the slice size to it when
formatting the slice end. With start values close to the maximum
this overflowed.
Now such range starts are ignored, similarly to other ranges not
handled by ngx_http_slice_get_start().
If a CRYPTO frame was received after the handshake, ngx_quic_handshake()
ran the handshake completion for the second time. With the check added
in 6f8c520f4, this is limited to short header packets.
Since a client has nothing to send in a CRYPTO frame once the handshake
is complete, such frames are now rejected with the "unexpected_message"
connection error, as does TLS 1.3 for handshake messages received out of
order, and QUIC requires for KeyUpdate.
The length in the "VALUE" line was accepted up to NGX_MAX_OFF_T_VALUE,
while ngx_http_memcached_filter_init() adds the size of the end marker
to it. With lengths close to the maximum this overflowed u->length.
Now such lengths are rejected, similarly to other invalid lengths.
Previously, when the worker file descriptor table was full, a listen
connection might be disabled and a timeout might be set for re-enabling
it. If during this period the worker received the NGX_CMD_QUIT signal,
the accept connection would be recycled, but the timer would remain set.
On timer expiration, an attempt to accept a new connection would result
in "accept4() failed (9: Bad file descriptor)" alert because c->fd was
reset to -1.
Fix is to delete accept timer when closing listen connections.
If a worker process' file descriptor table is full, recvmsg() is not
able to read out the channel fd sent from the master. This results in
different behavior on different platforms.
On Linux, recvmsg() succeeds, returns the payload and drops the fd.
On BSD systems, the first recvmsg() call fails with EMSGSIZE/EMFILE,
but the next call succeeds and returns the payload without the fd.
On Solaris, recvmsg() fails with EMFILE and drops both the payload
and the fd.
Previously, all of these paths resulted in fatal errors. The worker
channel was closed and the worker was no longer capable of receiving
messages from the master. In particular, the NGX_CMD_QUIT command
could not be delivered anymore. As a result, the worker never exited
after nginx reload and kept running and accepting new connections
forever until killed by a signal.
Fix is to consider those errors non-fatal. An additional check is
added to the NGX_CMD_CLOSE_CHANNEL handler to make sure the worker
has received a file descriptor for this entry before. Also, the
MSG_CTRUNC check is removed since a proper ancillary data size
validation is now implemented.
Added the http_json_module with a json_set directive that
extracts values from JSON stored in nginx variables and exposes them
as new nginx variables. Parsed results are cached per request, so
multiple json_set directives using the same source expression trigger
only one parse. The directive is available in the http main context.
Example usage:
http {
json_set $user_name $arg_data user.name;
json_set $user_age $arg_data user.age;
json_set $tag0 $arg_data tags[0];
server {
listen 8080;
location / {
return 200 "name=$user_name age=$user_age "
"tag0=$tag0\n";
}
}
# quoted keys for names containing '.', '[', ']', or that
# are empty; the whole argument is single-quoted so that the
# inner '"' reach the path parser intact
json_set $host $arg_data '["a.b.com"]';
json_set $nested $arg_data 'obj["a.b"].c[2]';
json_set $quoted $arg_data 'obj["a\\"b"]';
}
Co-authored-by: Dmitry Volyntsev <d.volyntsev@f5.com>
RFC 8259 permits any JSON value as a complete document, not just
objects and arrays. The start state now accepts strings, numbers,
and the true/false/null literals as well. A bare top-level number,
having no closing delimiter, is emitted when input ends.
An event-driven JSON parser implementing the grammar of RFC 8259.
The full grammar is recognized: objects, arrays, strings with \uXXXX
escapes (including UTF-16 surrogate pairs), numbers, and the
true/false/null literals. Tokens are reported to a caller-supplied
handler, with nesting bounded by a configurable maximum depth.
Based on previous work by Maxim Dounin.
Co-authored-by: Dmitry Volyntsev <d.volyntsev@f5.com>
Decodes JSON string escape sequences in place as defined by
RFC 8259, Section 7: the two-character escapes and \uXXXX,
with non-BMP code points encoded as UTF-16 surrogate pairs
and emitted as UTF-8.
The directive accepts predicates. When enabled, request body is read
immediately after the headers and before choosing a location.
Body read settings from the server block are used in this case.
Example:
http {
map $http_content_type $is_json {
application/json 1;
}
server {
listen 8000;
client_body_early_read $is_json;
client_max_body_size 256;
client_body_buffer_size 256;
return 200 $request_body;
}
}
The name length of a configured parameter was truncated to one byte
before the per-parameter length field was written, so that a
uwsgi_param name of 256 bytes or longer was encoded with a wrong
length while the name itself was copied in full. The resulting request
was malformed, and upstream servers responded by closing the connection.
While long names are barely used in practice, it is easy to support
them. The length is now properly encoded up to the allowed length.
The name length of a configured parameter was truncated to one byte
before the per-parameter length field was written, so that a
fastcgi_param name of 128 bytes or longer was encoded with a wrong
length while the name itself was copied in full. The resulting request
was malformed, and upstream servers responded by closing the connection.
While long names are barely used in practice, it is easy to support
them. The length is now properly encoded up to the allowed length.
Add ngx_proxy_protocol_v2_write() that writes the fixed 16-byte
PPv2 header and address block, and appends a list of TLVs.
An address block is written using IPv4/IPv6 mixed-family promotion:
the highest family present on either side (IPv6 > IPv4) is used for
the header, and the other address is promoted into it. Both v4-mapped
addresses are demoted to IPv4.
When PROXY protocol v2 is configured and the downstream connection
is SSL, standard PROXY protocol v2 TLVs are populated:
PP2_TYPE_ALPN from SSL_get0_alpn_selected()
PP2_TYPE_AUTHORITY from SSL_get_servername()
PP2_SUBTYPE_SSL_VERSION from SSL_get_version()
PP2_SUBTYPE_SSL_CIPHER from SSL_get_cipher_name()
PP2_SUBTYPE_SSL_CN from client cert subject CN
PP2_SUBTYPE_SSL_SIG_ALG from server certificate
PP2_SUBTYPE_SSL_KEY_ALG from server certificate
The CRC32c TLV is written last and covers the entire header in place.
Co-authored-by: Vadim Zhestikov <v.zhestikov@f5.com>
This change implements the Castagnoli polynomial code 0x11EDC6F41,
as described in RFC4960, Appendix B. It will be used to calculate
PROXY protocol v2 CRC32c checksums.
The HTTP/2 proxy and gRPC upstream clients accepted WINDOW_UPDATE
frames with a zero flow-control window increment. RFC 9113 requires
such frames to be handled as PROTOCOL_ERROR at stream or connection
scope.
Reject these frames as invalid upstream responses.
HTTP/2 and gRPC requests are now always sent with the ":authority"
pseudo-header, while the "Host" header is no longer passed.
Where a host value is set explicitly with "proxy_set_header Host"
or "grpc_set_header Host", that value is conveyed in ":authority".
As per RFC 9113, Section 8.3.1, clients and intermediaries MUST
generate an ":authority" pseudo-header to convey authority information.
For HTTP/1.1, HTTP/2 and gRPC, if the explicit value is empty, then
the value of $proxy_host is used instead. As per RFC 9110, Sections
4.2.1 and 4.2.2, the "http" and "https" URI schemes MUST contain a
non-empty host identifier. The change enforces this rule by
introducing a default.
Closes: https://github.com/nginx/nginx/issues/1426
Previously, the response length was explicitly set to zero if the
initial HEADERS frame was seen with the END_STREAM flag. This was
meant to properly pass responses in the Trailers-Only format as
introduced in 6559a42013. This handling was too broad though,
passing incomplete responses that carry a non-zero Content-Length
to the client as if they were complete and empty.
Now such responses are caught by the existing checks later on and
are passed as incomplete, similarly to how this is handled elsewhere.
For instance, on HTTP/2 client connections this now results in proper
signalling, an incomplete stream followed by stream reset.
In a8289aa69c, we introduced a new code to finalize the evaluation of
a complex value, deferring the stack update to this new code. However,
the change inadvertently broke compatibility with several third-party
modules that were reusing ngx_http_script_complex_value_code.
This change relegates omitted ngx_http_script_complex_value_end_code
from crash to a potential read of uninitialized bytes at the end of the
allocated buffer.
As per RFC 9000, Section 4.5:
Once a final size for a stream is known, it cannot change. If a
RESET_STREAM or STREAM frame is received indicating a change in the
final size for the stream, an endpoint SHOULD respond with an error of
type FINAL_SIZE_ERROR. A receiver SHOULD treat receipt of data at or
beyond the final size as an error of type FINAL_SIZE_ERROR, even after
a stream is closed.
Previously, the stream final size was validated only after other checks
that could return earlier and skip validation: the receive state check
and, for STREAM frames, the check for data below the current receive
offset. As a result, a STREAM or RESET_STREAM frame carrying a final
size error was silently ignored once the receiving part of the stream
had reached a state in which the frame would otherwise be discarded, or
when a STREAM frame did not advance the receive offset. Now the final
size is validated first, so a FINAL_SIZE_ERROR is generated in these
cases as well.
Previously, stream flow control was not applied to the RESET_STREAM
final_size, which allowed a client to exceed it. The excess had to be
within the connection flow control limits anyway.
Reported by Tony Wang.
The check in ngx_select_init_conf() can be insufficient because it
doesn't cover descriptors for local files, which are not accounted
for in cycle->connection_n. For instance, to reach this condition
with new connections, no reusable connections must be available for
recycling in ngx_get_connection().
The fix adopts a similar code from the win32 select module.
Since d9b7669666, the single-SV zero copy path in $r->print() no
longer required the SV to be read-only. Reference counting introduced
by that change keeps the SV alive, but not its string buffer: modifying
a mutable scalar after $r->print() overwrites the buffer in place or
reallocates it, while nginx might still have it queued in r->out, since
small responses are postponed by ngx_http_write_filter(). This resulted
in corrupted responses, and in freed memory being sent to the client
when the scalar was grown.
The SV is now required to be read-only again. Reference counting is
still needed here, as a read-only SV in an eval() might be freed with
the eval's op tree.
Directory listing entries (file names, metadata) were allocated
from the request pool and lived until request teardown. Use a
temporary pool instead, destroying it immediately after the
response buffer is built.
The size parser and the image decoders used ctx->length, which is the
allocation size (set to image_filter_buffer when the upstream response
omits Content-Length), as the amount of valid data, so a truncated
response without Content-Length could be parsed or decoded past the
received bytes into uninitialized buffer memory.
The length is now adjusted to the actually-read size once the response
body has been read.
Reported by dukesp69 and YLChen-007.
