sqlglot[c] leaks memory every time a ParseError gets raised and caught. Pure-python sqlglot doesn't. If you parse a lot of SQL and some of it is invalid, RSS keeps climbing until the process gets OOM-killed. Hit this in a long-running service that parses user SQL.
# pip install "sqlglot[c]==30.8.0" -> RSS climbs -> even newer versions have this issue
# pip install "sqlglot==30.8.0" -> flat
import gc, psutil, sqlglot
from sqlglot.errors import ParseError
def rss(): return psutil.Process().memory_info().rss / 1024 / 1024
def op(i):
try: sqlglot.parse_one(f"SELECT FROM WHERE {i} GROUP")
except ParseError: pass
for i in range(1000): op(i)
gc.collect(); base = rss()
for i in range(40000): op(i + 100000)
gc.collect(); print(f"{rss()-base:+.1f} MB")sqlglot[c] prints about +38 MB, pure python ~0. gc.collect() doesn't get it back. Same on 30.8.0, 30.11, 30.12, and main.
The real bug is upstream in mypyc, not sqlglot: mypyc-compiled Exception subclasses never free their attributes, so every ParseError hangs onto its message and context. I filed it with a small repro at python/mypy#21716.
If you want to knock most of it out now without waiting on mypyc: drop errors.py from SOURCE_FILES in sqlglotc/setup.py so ParseError doesn't get compiled. It's not hot code so it shouldn't cost anything. That took it from ~25.8 MB down to ~5.7 MB over 20k parses for me (~78% off). A little still leaks from the exception passing through the other compiled modules, but that part needs the mypyc fix.
I tested on Python 3.11, Linux.
sqlglot[c]leaks memory every time a ParseError gets raised and caught. Pure-python sqlglot doesn't. If you parse a lot of SQL and some of it is invalid, RSS keeps climbing until the process gets OOM-killed. Hit this in a long-running service that parses user SQL.sqlglot[c] prints about +38 MB, pure python ~0. gc.collect() doesn't get it back. Same on 30.8.0, 30.11, 30.12, and main.
The real bug is upstream in mypyc, not sqlglot: mypyc-compiled Exception subclasses never free their attributes, so every ParseError hangs onto its message and context. I filed it with a small repro at python/mypy#21716.
If you want to knock most of it out now without waiting on mypyc: drop
errors.pyfromSOURCE_FILESinsqlglotc/setup.pyso ParseError doesn't get compiled. It's not hot code so it shouldn't cost anything. That took it from ~25.8 MB down to ~5.7 MB over 20k parses for me (~78% off). A little still leaks from the exception passing through the other compiled modules, but that part needs the mypyc fix.I tested on Python 3.11, Linux.