Research findings

Is wordfreq useful for our Arabic decks?

Short answer: as a coverage sanity-check against written Arabic, yes. As a priority list for a spoken Palestinian course, no — it would teach the wrong word first.

Measured against wordfreq 3.1.1 (Robyn Speer), Arabic “large” list of 619,906 words. Its ten most frequent Arabic tokens: في · من · على · أن · لا · إلى · و · ما · عن · هذا.

Applicable — as a gap-check

Coverage sanity-check ✓

wordfreq recognises 98.9% of our Palestinian single words and 100.0% of the MSA ones. It’s a fair way to spot common written words we haven’t taught.

Not applicable — as a priority list

Frequency ranking ✗

Its data is Modern Standard Arabic. Where a Levantine word differs from the textbook word, the textbook word ranks higher 78.7% of the time.

01The coverage numbers

We compared two of our decks against wordfreq: the Palestinian deck (our real, native-reviewed dialect deck) and the MSA deck (the fair like-for-like, since wordfreq’s Arabic is mostly written MSA). Percentages are over single-word entries; multi-word idioms are counted separately.

265
Palestinian single words analysed (+23 idioms)
43.0%
of them sit in wordfreq’s top 5,000 written words
51.8%
of the MSA deck sits in the top 5,000 — the register gap
78.7%
of differing word-pairs: the MSA form is commoner in writing
wordfreq strips diacritics and recognises almost everything we teach — so “does it know our words?” is the wrong question. The signal is where words sit. Our Palestinian words cluster lower in the frequency distribution than the MSA words, because everyday spoken forms are simply rarer in the Wikipedia-and-news text wordfreq is built from.

02Same meaning, different word — who wins in writing?

Our decks are concept-aligned, so for every idea taught in both we can line the Levantine word up against the MSA word. Across 47 concepts where the two differ, the written (MSA) form is more common 37 times; the spoken (dialect) form wins only 2. Longer bar = more common in written Arabic (Zipf scale, higher = commoner).

Palestinian (spoken) MSA (written)
بقديش b'addeish · How much? (alternative)
0.0
5.4
زغير zghiir · Small, little
2.2
5.0
زلمة zalame · Man, bloke
2.7
5.5
عيلة 'iile · Family
2.8
5.2
زهقان zah'aan · Bored, fed up
1.8
4.0
مهضوم mahdoom · Charming, likeable
2.4
4.4
مرا mara · Woman
3.2
5.2
منيح mniih · Good
3.1
5.1
مزنوق maznoo' · Pressed for time; in a tight spot
1.6
3.4
قلقان 'al'aan · Worried
2.7
4.5
مصاري masaari · Money
2.3
4.0
بكرا bukra · Tomorrow
3.5
5.1
تقيل t'iil · Heavy
2.7
4.1
فرحان farhaan · Glad, joyful
3.8
5.3
تلفون tilifoon · Phone
3.2
4.6
فروج farrooj · Chicken
2.7
4.0

This is the crux: sort a Levantine syllabus by wordfreq and you’d front-load صغير, جيد, غدا over the words people actually say — زغير, منيح, بكرا.

03How common are our words, really?

The same 265 Palestinian and 164 MSA single words, bucketed by Zipf value. Zipf 6 ≈ a word you meet about once per thousand; Zipf 3 ≈ once per million. The MSA deck leans right (commoner in writing); the Palestinian deck spreads left, with a genuinely dialectal tail wordfreq barely sees.

Palestinian deck

by Zipf band · median 4.28
30 (unknown)
40.01–2
252–3
663–4
1134–5
505–6
46+

MSA deck

by Zipf band · median 4.47
00 (unknown)
00.01–2
32–3
403–4
824–5
395–6
06+

04What that tail looks like

Words we teach that are common in writing too

  • هُوّ huwwe He #16
  • هِيّ hiyye She #30
  • تِمّ timm Mouth #43
  • يَوْم yoom Day #51
  • أَنَا ana I #96
  • كِيف kiif How #99
  • وَاحَد waahad One #121
  • هُمّ humme They #146
  • سَنَة sane Year #157
  • كْبِير kbiir Big, large #163

Core Levantine words wordfreq barely knows

  • تُقْبُرْنِي tu'burni I’d rather die first than lose you Zipf 0.0
  • بْقَدِّيش b'addeish How much? (alternative) Zipf 0.0
  • إِيمْتَى eimta When Zipf 0.0
  • مَزْنُوق maznoo' Pressed for time; in a tight spot Zipf 1.6
  • زَهْقَان zah'aan Bored, fed up Zipf 1.8
  • سِرْفِيس serviis Shared taxi Zipf 1.9
  • تْمَانْيَة tmaanye Eight Zipf 2.0
  • عَطَى 'ata To give Zipf 2.0
  • إِجَا ija To come Zipf 2.1
  • زْغِير zghiir Small, little Zipf 2.2
Look at the right-hand column: تمانية (eight), عطى (to give), إجا (to come), زغير (small). These are beginner essentials that wordfreq ranks as obscure — the clearest sign its ranking can’t drive a dialect syllabus.

05Common written words neither deck teaches

wordfreq’s most frequent Arabic words that appear in neither deck. Read the caveat first: most are MSA grammar words (الذي “which”, لم negation, قد a verb particle) that Levantine either doesn’t use or forms differently — so “missing” here often means “correctly absent from a spoken course”, not a real gap.

06What wordfreq is (and its expiry date)

wordfreq is a Python library by Robyn Speer giving word-frequency data for 40+ languages. Its Arabic data is pooled from Wikipedia, news crawls, film subtitles, the OSCAR web corpus and Twitter — overwhelmingly Modern Standard Arabic. It’s the standard open frequency source, but it stopped in 2024:

“I don’t think anyone has reliable information about post-2021 language usage by humans… large language models generate text that masquerades as real language.” — Robyn Speer, wordfreq sunset note, Sept 2024
Language
Arabic (ar) — one shared code for all dialects and MSA
Register
Mostly written / Modern Standard Arabic
Frozen at
~2021 language usage; no further data updates
Diacritics
Stripped — it matches bare forms, so we normalise our vocalised text before comparing
Clitics
Not split — the definite article ال and prefixes stay attached to the token
Licence
Apache-2.0 (code); data CC-BY-SA-4.0
API used
zipf_frequency, top_n_list

07Recommendation