How to evaluate models like claude / chatgpt？

[Bruno-Lee]

Hello, I've read the repo code，but I can not find how to chat with source model like openai gpt-4o，claude .etc and evaluate their chat completion results，can u help?

[NamCyan]

Hi @Bruno-Lee, for closed source models like GPT-4o and Claude, we access their APIs to generate solutions. The prompts for these models remain similar to those presented in the paper, with slight modifications to make easier extraction of the answers. I will update the script within the next two days and notify you once it is ready.

[Bruno-Lee]

@NamCyan Tks. I'm looking forward to your sharing of more in-depth content.

[NamCyan]

Sorry for the delay, @Bruno-Lee. PhD life has been keeping me busy :(
I've added the scripts for running closed-source models. Please check the Note in both the LLMs Generation and Process Output guidelines in README for more details. Let me know if you have any questions.

[Bruno-Lee]

@NamCyan Thanks for sharing. Over the past few weeks, I have been deeply studying the RepoExec evaluation system and scoring logic, and I have read related papers, gaining many insights into model evaluation. I look forward to having more in-depth discussions with you. 😄

[NamCyan]

@Bruno-Lee That’s awesome to hear! 😄