feat: add prompt caching support for nova models in bedrock#1816
feat: add prompt caching support for nova models in bedrock#1816mehtarac wants to merge 1 commit intostrands-agents:mainfrom
Conversation
|
|
||
| # Models that support prompt caching | ||
| # Anthropic Claude and Amazon Nova models on Bedrock support prompt caching | ||
| _MODELS_SUPPORTING_CACHING = [ |
There was a problem hiding this comment.
why do we keep a list? why not just let bedrock throw the exceptions? is keeping a list from our side bring any benefits?
There was a problem hiding this comment.
i think helps us maintain which models we want to support with this feature. Follows similar code pattern to _MODELS_INCLUDE_STATUS for tool result status.
There was a problem hiding this comment.
i think helps us maintain which models we want to support with this feature
Any reason we wouldn't want to support all models that support caching? I don't like that we have to keep a feature matrix for bedrock in our codebase. What's the exception bedrock throws if we didn't have this?
There was a problem hiding this comment.
I've implemented the approach you suggested - removing the hardcoded model list and letting Bedrock validate. When a user tries to use caching with an unsupported model, Bedrock will return a clear ValidationException explaining which models are supported, which is better than a silent warning in the logs.
| model_id = self.config.get("model_id", "").lower() | ||
| if "claude" in model_id or "anthropic" in model_id: | ||
| if any(pattern in model_id for pattern in _MODELS_SUPPORTING_CACHING): | ||
| return "anthropic" |
There was a problem hiding this comment.
going to update this as this would also return anthropic for nova models
|
/strands I want to see what's the devx without the hard coded model list. what errrors do customers get, is it clear? |
DevX Investigation: Removing Hardcoded Model ListBased on @mkmeral's feedback, I've investigated the developer experience without the hardcoded model list and implemented the changes. Before (hardcoded list):
After (let Bedrock validate):
Changes Made:
All 121 bedrock tests pass ✅ |
Description
Extends automatic prompt caching to Amazon Nova models in the Bedrock provider. Previously, only Anthropic Claude models supported automatic cache point injection when using CacheConfig(strategy="auto"). This is consistent with the docs https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html
Related Issues
#1817
Documentation PR
Type of Change
New feature
Testing
How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli
hatch run prepareChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.