enh(Net,Foundation): multipart parsing performance and stream bulk-read optimization - #5290
Merged
Conversation
matejk
force-pushed
the
5288-multipart-fixes-improvements
branch
from
April 3, 2026 09:52
501a689 to
be73cbe
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
matejk
marked this pull request as draft
April 5, 2026 20:29
matejk
force-pushed
the
5288-multipart-fixes-improvements
branch
from
April 8, 2026 08:58
7429582 to
6e39cfb
Compare
Add xsgetn() override to BasicBufferedStreamBuf that copies directly from the internal buffer via char_traits::copy + setg, delegating refill to underflow(). Eliminates per-byte virtual call overhead when consumers use istream::read() or StreamCopier (up to 25x faster for multipart Content-Length reads). Fix UnbufferedStreamBuf::xsgetn to catch exceptions mid-read and return bytes already copied, rather than losing all data. This fixes StreamCopier::copyToString returning empty through Base64Decoder on streams that throw during decoding (e.g., multipart without final boundary marker).
matejk
marked this pull request as ready for review
April 8, 2026 12:01
matejk
force-pushed
the
5288-multipart-fixes-improvements
branch
from
April 8, 2026 12:01
6e39cfb to
b469ca6
Compare
Fix OSS-Fuzz timeout (#5288) caused by O(parts x content_type_params) complexity -- a crafted message with 2086 Content-Type parameters and empty boundary caused 60s+ timeout. Cache multipart state, validate empty boundary, add MAX_PARTS (100000) DoS limit. Add Content-Length bulk-read optimization (#4118) using sgetn fast path when Content-Length is present in part headers, with 32-bit overflow protection. Increase STREAM_BUFFER_SIZE from 1KB to 16KB. Introduce ReadWindow sliding buffer at MultipartReader level. All reads (parseHeader, readContent, boundary scanning) go through a single 16KB window with zero-copy in-buffer boundary scanning via scanForBoundary(). Over-read bytes from boundary detection stay in the window for the next part's header parsing -- no replay, no holdback vectors, no PrependStreamBuf. Replace char-by-char stream drain with istr.ignore(). Modernize with C++17: [[nodiscard]], = delete/default, structured bindings, std::none_of, if-with-initializer, enum class, static constexpr.
Add 7 new MailMessage tests: - testReadMultiPartEmptyBoundary: empty boundary throws (#5288) - testReadMultiPartWithContentLength: Content-Length bulk-read (#4118) - testReadMultiPartWithZeroContentLength: Content-Length: 0 edge case - testReadMultiPartManyParts: 60000 parts correctness + timing - testReadMultiPartTooManyParts: MAX_PARTS (100000) limit enforced - testReadMultiPartLargeWithContentLength: 200x500KB with CL + timing - testReadMultiPartLargeWithoutContentLength: 200x500KB no CL + timing Update StringPartHandler to use StreamCopier::copyToString for bulk reads and add override/[[nodiscard]] annotations.
matejk
force-pushed
the
5288-multipart-fixes-improvements
branch
from
April 8, 2026 16:41
b469ca6 to
235722b
Compare
matejk
added a commit
that referenced
this pull request
Apr 9, 2026
…ad optimization (#5290) * enh(Foundation): bulk-read xsgetn and exception safety for stream bufs Add xsgetn() override to BasicBufferedStreamBuf that copies directly from the internal buffer via char_traits::copy + setg, delegating refill to underflow(). Eliminates per-byte virtual call overhead when consumers use istream::read() or StreamCopier (up to 25x faster for multipart Content-Length reads). Fix UnbufferedStreamBuf::xsgetn to catch exceptions mid-read and return bytes already copied, rather than losing all data. This fixes StreamCopier::copyToString returning empty through Base64Decoder on streams that throw during decoding (e.g., multipart without final boundary marker). * enh(Net): multipart parsing performance and robustness (#5288, #4118) Fix OSS-Fuzz timeout (#5288) caused by O(parts x content_type_params) complexity -- a crafted message with 2086 Content-Type parameters and empty boundary caused 60s+ timeout. Cache multipart state, validate empty boundary, add MAX_PARTS (100000) DoS limit. Add Content-Length bulk-read optimization (#4118) using sgetn fast path when Content-Length is present in part headers, with 32-bit overflow protection. Increase STREAM_BUFFER_SIZE from 1KB to 16KB. Introduce ReadWindow sliding buffer at MultipartReader level. All reads (parseHeader, readContent, boundary scanning) go through a single 16KB window with zero-copy in-buffer boundary scanning via scanForBoundary(). Over-read bytes from boundary detection stay in the window for the next part's header parsing -- no replay, no holdback vectors, no PrependStreamBuf. Replace char-by-char stream drain with istr.ignore(). Modernize with C++17: [[nodiscard]], = delete/default, structured bindings, std::none_of, if-with-initializer, enum class, static constexpr. * test(Net): add multipart parsing tests for boundary, limits, and perf Add 7 new MailMessage tests: - testReadMultiPartEmptyBoundary: empty boundary throws (#5288) - testReadMultiPartWithContentLength: Content-Length bulk-read (#4118) - testReadMultiPartWithZeroContentLength: Content-Length: 0 edge case - testReadMultiPartManyParts: 60000 parts correctness + timing - testReadMultiPartTooManyParts: MAX_PARTS (100000) limit enforced - testReadMultiPartLargeWithContentLength: 200x500KB with CL + timing - testReadMultiPartLargeWithoutContentLength: 200x500KB no CL + timing Update StringPartHandler to use StreamCopier::copyToString for bulk reads and add override/[[nodiscard]] annotations.
This was referenced Apr 20, 2026
matejk
added a commit
that referenced
this pull request
Apr 20, 2026
Revert the mid-read try/catch added to BasicUnbufferedStreamBuf::xsgetn in #5290. Swallowing uflow() exceptions after a partial copy left istream::read unable to set badbit, so Base64Decoder/HexBinaryDecoder errors became silent no-ops on read() consumers. Add read()-based regression blocks to Base64Test and HexBinaryTest; the existing invalid-input blocks only exercised operator>> (sbumpc).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
xsgetn()bulk-read override toBufferedStreamBufand fix exception handling inUnbufferedStreamBuf::xsgetn()ReadWindowsliding buffer atMultipartReaderlevel for zero-copy boundary scanningMailMessage,MultipartReader, andPartHandlerwith C++17 featuresCloses #5288
Closes #4118
Commits
1.
enh(Foundation): bulk-read xsgetn and exception safety for stream bufsxsgetn()inBasicBufferedStreamBufto copy directly from internal buffer viachar_traits::copy+setg, delegating refill tounderflow()UnbufferedStreamBuf::xsgetnto catch exceptions mid-read and return bytes already copied (fixesStreamCopier::copyToStringthroughBase64Decoderon truncated streams)2.
enh(Net): multipart parsing performance and robustness (#5288, #4118)makeMultipart(), validate empty boundary, addMAX_PARTS(100,000) DoS limitsgetnfast path whenContent-Lengthpresent in part headers, with 32-bit overflow protectionReadWindowsliding buffer atMultipartReaderlevel: all reads (parseHeader,readContent, boundary scanning) go through a single 32KB window with zero-copy in-buffer boundary scanning. Over-read bytes stay in the window for the next part's header parsing -- no replay, no holdback vectorsSTREAM_BUFFER_SIZEfrom 1KB to 32KBistr.ignore()[[nodiscard]],= delete/default, structured bindings,std::none_of, if-with-initializer,enum class,static constexpr3.
test(Net): add multipart parsing tests for boundary, limits, and perf7 new MailMessage tests: empty boundary, Content-Length path, Content-Length: 0, 60K parts, MAX_PARTS limit, 200x500KB with/without Content-Length. Update
StringPartHandlerto useStreamCopier::copyToString.Measured Performance
Production handler (
StreamCopier::copyToString, 200 x 500KB parts, ~97 MB)Null handler (
stream.ignore, 200 x 500KB parts, ~97 MB)Small parts (60,000 parts, 5.5 MB, null handler)
No regression for small parts.
OSS-Fuzz #5288 (46 KB crafted message)
Side-effect performance improvements
BufferedStreamBuf::xsgetn(affects all BufferedStreamBuf-derived streams)The
xsgetn()override copies directly from the internal buffer instead of callingsbumpc()per byte. This benefits every consumer usingistream::read()orStreamCopieron anyBufferedStreamBuf-derived stream:UnbufferedStreamBuf::xsgetnexception safetyFixes
StreamCopier::copyToStringreturning empty when reading throughBase64Decoderon streams that throw mid-read (e.g., multipart without final boundary). Previously, an exception duringxsgetndiscarded all bytes read so far.HTTP server impact
For HTTP multipart form uploads (
HTMLForm::readMultipart):read()-based consumer on HTTPFixedLengthStreamTest plan