🚧 feat: Support Guardrail Config Option streamProcessingMode

[dlew]

Summary

streamProcessingMode affects how guardrail processes the stream from the model. If it's in "sync" mode, it chunks up the response and processes them before returning them to the user. If it's in "async" mode, it both processes the chunk & sends it to the user at the same time, allowing for smoother streaming (at the cost of guardrail only reacting after offending content starts to stream, in some cases).

Related documentation change here: LibreChat-AI/librechat.ai#564

Change Type

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update (here: docs: guardrail config's streamProcessingMode LibreChat-AI/librechat.ai#564)

Testing

I added streamProcessingMode to an existing test to make sure it's processed by the guardrail configuration code.

I also verified that setting it to async in my librechat.yaml would result in a faster streaming response from Bedrock.

Checklist

Please delete any irrelevant options.

• My code adheres to this project's style guidelines
• I have performed a self-review of my own code
• I have made pertinent documentation changes
• My changes do not introduce new warnings
• I have written tests demonstrating that my changes are effective or that my feature works
• Local unit tests pass with my changes
• A pull request for updating the documentation has been submitted (here: docs: guardrail config's streamProcessingMode LibreChat-AI/librechat.ai#564)

[danny-avila]

@dlew lgtm, can we maybe make async the default behavior? seems better for most use cases

[dlew]

@dlew lgtm, can we maybe make async the default behavior? seems better for most use cases

@danny-avila The issue is that it's inherently less safe to use async, since it could (briefly) let through content that should've been blocked. So for something like content filtering, I think the default should be whatever is safest (and let people opt into a less safe mode).

Though if you feel strongly about it, I'm fine setting the default the other way.

[danny-avila]

fair enough, merged!