2) models have gotten too good at this task to compare them meaningfully (I tried doing a blind test and couldn't reliably tell 6.1 Astra, Sol, and Terra apart at reasoning levels that approximately match in cost)
>> Are the pelicans still a useful benchmark at all?
If you are using these models for software development, then I ask you this question…… when you are interviewing job candidates to join your team, do you ask them to write a program to draw pelicans?
I would guess that
1) they're probably in the training data, and
2) models have gotten too good at this task to compare them meaningfully (I tried doing a blind test and couldn't reliably tell 6.1 Astra, Sol, and Terra apart at reasoning levels that approximately match in cost)
If you are using these models for software development, then I ask you this question…… when you are interviewing job candidates to join your team, do you ask them to write a program to draw pelicans?
Thus you have your answer.