disambig_mle_calima-msa-r13-0.2.5 is 85mb — after replacing escape codes with unicode it is 51mb.
The command I used to remove escape codes: jq -c . < model.json > model-nofmt.json. In python, pass ensure_ascii=False to json.dumps.
- Escaped unicode:
{"\u0641\u064a": {"diac": "\u0641\u0650\u064a"
- Unicode:
{"في":{"diac":"فِي"
disambig_mle_calima-msa-r13-0.2.5is 85mb — after replacing escape codes with unicode it is 51mb.The command I used to remove escape codes:
jq -c . < model.json > model-nofmt.json. In python, passensure_ascii=Falsetojson.dumps.{"\u0641\u064a": {"diac": "\u0641\u0650\u064a"{"في":{"diac":"فِي"