[refactor] Consider making public and private pieces of the library #1015
Description
Today, almost everything that is exportable is able to be touched by callers of the library. I don't believe that we have though through if this is valid, and it also pollutes the documentation or discoverability of the library. Because of this, we should consider having distinct internal and external packages that have defined audiences. For instance, the "engine" code should be usable by an integrator (kantra/kai) and should be focused on making it easy/flexable to run analysis and get progress/output in a standard reasonable way. While the "provider" bit's should be focused on getting providers started easily. This may mean that the lsp/base package and provider package need to be refactored and combined into a single package.
This can only happen after moving folks to the refactored entrypoint in #992