Public bake-off · classify
Best small model for Classify GitHub issue titles as bug, feature_request, or question
Winner: cross-encoder/nli-deberta-v3-xsmall incumbent
Resident family incumbents were compared on the submitted labels.
Bake-off bo_c23303f14990692e4abfb768b476459b ·
Scorecard
| Rank | Model | Metrics | Predicted vs expected | Latency | Status |
|---|---|---|---|---|---|
| 1 | cross-encoder/nli-deberta-v3-xsmallincumbentwinner | accuracy 0.625macro_f1 0.5238n 8 | Not retained | 1247 ms | scored |
| 2 | distilbert/distilbert-base-uncased-finetuned-sst-2-englishincumbent | accuracy 0.0macro_f1 0.0n 8 | Not retained | 152 ms | scored |
| 3 | cardiffnlp/twitter-roberta-base-sentiment-latestincumbent | accuracy 0.0macro_f1 0.0n 8 | Not retained | 1743 ms | scored |
Signed receipt
This result is bound to its task, candidates, scorecard, and winner.
sha256:a28f71f2fa4d5af7db41fc357be238157b04cefee1af542b9b22a6d7c54b2ec9
Winner badge
Copy-paste Markdown:
[](https://run.huggingbay.xyz/bakeoffs/bo_c23303f14990692e4abfb768b476459b)Reproduce it
The first bake-off is anonymous. Run the same labeled request directly:
curl -sS https://run.huggingbay.xyz/v1/bakeoff -H 'Content-Type: application/json' --data '{"task_description":"Classify GitHub issue titles as bug, feature_request, or question","examples":[{"input":"App crashes on startup after upgrading to v2.3.1","expected":"bug"},{"input":"Add dark mode support to the settings page","expected":"feature_request"},{"input":"How do I configure a custom retry policy?","expected":"question"},{"input":"Memory leak when streaming large responses","expected":"bug"},{"input":"Support exporting reports as CSV","expected":"feature_request"},{"input":"Is there a way to run this offline?","expected":"question"},{"input":"TypeError: cannot read property '"'"'id'"'"' of undefined in worker.js","expected":"bug"},{"input":"Please add webhook signatures for delivery verification","expected":"feature_request"}]}'