LLM Evaluation Report

ModelDateTotal Response Time (s)Tests PassedMean CodeBLEU (0-1)Mean Usefulness Score (0-4)Mean Functional Correctness Score (0-4)

claude-3-5-sonnet-20240620

2024-06-24

257.051

114

0.301656

3.53659

3.38415

gpt-4o

2024-06-24

249.677

124

0.315538

3.57317

3.53049

gpt-4-turbo

2024-06-24

420.44

116

0.324327

3.48171

3.51829

gpt-4-turbo-preview

2024-06-24

426.862

105

0.318952

3.46341

3.46951

claude-3-opus-20240229

2024-06-24

936.962

70

0.285935

3.22561

3.09756

gemini-1.5-pro-preview-0514

2024-06-24

574.591

94

0.34067

3.31098

3.14024

Total Response Time (s): The total time taken by the model to generate all the outputs.

Tests passed: The number of unit tests that the model has passed during evaluation, out of a total of 164 tests.

Mean CodeBLEU: Average CodeBLEU score, a metric for evaluating code generation quality based on both syntactic and semantic correctness.

Mean Usefulness Score: Average rating of the model's output usefulness as rated by a LLM model.

  • 0: Snippet is not at all helpful, it is irrelevant to the problem.

  • 1: Snippet is slightly helpful, it contains information relevant to the problem, but it is easier to write the solution from scratch.

  • 2: Snippet is somewhat helpful, it requires significant changes (compared to the size of the snippet), but is still useful.

  • 3: Snippet is helpful, but needs to be slightly changed to solve the problem.

  • 4: Snippet is very helpful, it solves the problem.

Mean Functional Correctness Score: Average score of the functional correctness of the model's outputs, assessing how well the outputs meet the functional requirements, rated by a LLM model.

  • 0 (failing all possible tests): The code snippet is totally incorrect and meaningless.

  • 4 (passing all possible tests): The code snippet is totally correct and can handle all cases.

Last updated