feat: add and update openai reasoning_effort tests #4815
Conversation
Signed-off-by: Alexandros Pappas <apappascs@gmail.com>
apappascs
force-pushed
the
test/validate-reasoning-effort-parameter-4804
branch
from
8a29f9d to
aec4bf8
Compare
Signed-off-by: Alexandros Pappas <apappascs@gmail.com>
… in OpenAiChatModelIT Signed-off-by: Alexandros Pappas <apappascs@gmail.com>
| } | ||
|
|
||
| @Test | ||
| @Disabled("The reasoning_effort option is only available in o1 models.") |
There was a problem hiding this comment.
This test result isn't consistent with the current settings as the reasoning tokens end up being "0" in some cases when the model doesn't seem to use reasoning tokens for simple problems unless being more specific with the prompt message.
There was a problem hiding this comment.
I will update the prompt when merging.
There was a problem hiding this comment.
Thank you @ilayaperumalg for pointing this out.
just updated with the example from claude https://docs.claude.com/en/docs/build-with-claude/extended-thinking
and set it to high. already run it 10 times without failure, but I am not sure if there is a case to have reasoning tokens end up being "0"
| } | ||
|
|
||
| @Test | ||
| void testReasoningEffortWithDifferentLevels() { |
There was a problem hiding this comment.
This test can be an overhead as this doesn't exactly verify the reasoning tokens from the output. What do you think?
There was a problem hiding this comment.
I agree just wanted to verify all reasoning levels work. lemme update the PR
Signed-off-by: Alexandros Pappas <apappascs@gmail.com>
|
@apappascs Thank you for the quick follow up and updates. The changes look good. Rebased, squashed and merged as e157775 |
2 participants
#4804
Thank you for taking time to contribute this pull request!
You might have already read the contributor guide, but as a reminder, please make sure to:
git commit -s) per the DCOmainbranch and squash your commitsFor more details, please check the contributor guide.
Thank you upfront!