Beam Korean Bench
How well does Beam-1.0-501B-A23B handle the parts of Korean that a rule can check? Gold answers in the Korean split are computed by code (particle allomorphy from the final consonant, Sino vs native numeral reading, Hangul counts), or derived by code from a hand-written verb table (irregular conjugation). Beam is compared with four other open models, run on Belebele in English and Korean, and its tokenizer is priced against Korean-native tokenizers.
1. Korean split: accuracy by category
Free-response unless noted; answers are compared after removing spaces and punctuation. Hover a dot for the exact score.
Table view
2. Where Beam goes wrong
Two patterns explain most of Beam's misses. Particles: when a loanword or a letter name ends in the epenthetic vowel ㅡ (버스 from "bus", 에스 for "S"), Beam picks the form for a final consonant. Its reasoning says why: "버스 ends with ㅅ (consonant)", "키보드 ends with ㄷ", "BTS ends with S (consonant)". It drops the ㅡ and judges by how the English word ends. High effort does not fix it; one sentence in the prompt mostly does (below). Numerals: with native-Korean counters (살, 명, 개) it often reads the number in Sino-Korean (삼십이 살 for 서른두 살, 사 명 for 네 명).
Table view
3. Belebele: English vs Korean reading comprehension
The same 900 parallel questions (facebook/belebele) in three modes, reasoning_effort=low.