Use PURL for inputs

[pombredanne]

I would like to use a PURL as a project input like with pkg:pypi/psycopg2@2.9.9

[tdruez]

It would be nice to use fetchcode for this purpose instead of yet another implementation in ScanCode.io

This does not look supported:

>>> fetch('pkg:pypi/psycopg2@2.9.9')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/fetchcode/__init__.py", line 108, in fetch
    raise Exception("Not a supported/known scheme.")
Exception: Not a supported/known scheme.

The best I could find is a wheel distribution link using:

>>> list(package.info('pkg:pypi/psycopg2@2.9.9'))
[Package(type='pypi', namespace=None, name='psycopg2', version='2.9.9')]
>>> p = list(package.info('pkg:pypi/psycopg2@2.9.9'))[0]
>>> print(p)
Package(type='pypi', namespace=None, name='psycopg2', version='2.9.9')
>>> print(p.download_url)
https://files.pythonhosted.org/packages/a2/14/2767d963915f957c07f5d4c3d9c5c9a407415289f5cde90b82cb3e8c2a12/psycopg2-2.9.9-cp310-cp310-win32.whl

@pombredanne @keshav-space could you add proper support for fetching PURLs in fetchcode? Also, it would be better to return a sdist link when possible, such as "https://files.pythonhosted.org/packages/c9/5e/dc6acaf46d78979d6b03458b7a1618a68e152a6776fce95daac5e0f0301b/psycopg2-2.9.9.tar.gz"

[keshav-space]

@tdruez

could you add proper support for fetching PURLs in fetchcode?

FetchCode already supports retrieving package metadata along with download_url using PURL. We can add support for more ecosystems there as needed.

Also, it would be better to return a sdist link when possible, such as

Yes, we should return the sdist link in the repository_download_url which is misisng here.

[tdruez]

FetchCode already supports retrieving package metadata along with download_url using PURL.

@keshav-space Great, could you provide a link to the documentation on how to get the download_url from the suggested PURL input: pkg:pypi/psycopg2@2.9.9?

[keshav-space]

Great, could you provide a link to the documentation on how to get the download_url from the suggested PURL input: pkg:pypi/psycopg2@2.9.9?

@tdruez fetchcode.package.info that you tried earlier is the way to get package metadata, and it includes the download_url field. We're already using this in PurlDB. We don't have proper documentation for fetchcode only the README, which needs to be updated.

>>> list(package.info('pkg:pypi/psycopg2@2.9.9'))
[Package(type='pypi', namespace=None, name='psycopg2', version='2.9.9')]
>>> p = list(package.info('pkg:pypi/psycopg2@2.9.9'))[0]
>>> print(p)
Package(type='pypi', namespace=None, name='psycopg2', version='2.9.9')
>>> print(p.download_url)
https://files.pythonhosted.org/packages/a2/14/2767d963915f957c07f5d4c3d9c5c9a407415289f5cde90b82cb3e8c2a12/psycopg2-2.9.9-cp310-cp310-win32.whl

[tdruez]

@keshav-space

list(package.info('pkg:pypi/psycopg2@2.9.9'))[0].download_url

This is not a great API...

The following should be implemented:

fetch('pkg:pypi/psycopg2@2.9.9')

[keshav-space]

@tdruez

list(package.info('pkg:pypi/psycopg2@2.9.9'))[0].download_url

This is not a great API...

We can also call this API with version-less PURL, and in that case, it will return package metadata for all the existing versions of the package.

>>> list(package.info('pkg:pypi/psycopg2'))
[Package(type='pypi', namespace=None, name='psycopg2', version='2.0.10'),
 Package(type='pypi', namespace=None, name='psycopg2', version='2.0.11'),
 Package(type='pypi', namespace=None, name='psycopg2', version='2.0.12'),
 Package(type='pypi', namespace=None, name='psycopg2', version='2.0.13'),
 ...

The following should be implemented:

fetch('pkg:pypi/psycopg2@2.9.9')

So, how should the new API behave? Should it return only the download_url, or the full package metadata? And should the version-less PURL be allowed as an input?

[tdruez]

Implemented based on purl2url.get_download_url in:

• Add support for using Package URL (purl) as project input. #1686

Now, since PyPI is not supported in purl2url, leveraging FetchCode still make sense.

---

@keshav-space

So, how should the new API behave?

I think we should have clean, simple low-level API functions that support every use case: package metadata, list all download URL, etc..

We should also offer a simple fetch or similar name, that providing a full purl (including version) will fetch the best option for you.
It should also be possible to provide more context to that function to fetch a more precise package.

Some design ideas:

fetch('pkg:pypi/psycopg2@2.9.9')
-> Return the source by default: `https://files.pythonhosted.org/packages/62/51/2007ea29e605957a17ac6357115d0c1a1b60c8c984951c19419b3474cdfd/psycopg2-2.9.10.tar.gz`

fetch('pkg:pypi/django@5.2.3', extension="whl")
-> https://files.pythonhosted.org/packages/1b/11/7aff961db37e1ea501a2bb663d27a8ce97f3683b9e5b83d3bfead8b86fa4/django-5.2.3-py3-none-any.whl

We could also support a filename argument for when multiple file with the same extension are present, psycopg has multiple wheel files: https://pypi.org/project/psycopg2/#files

And if a download URL cannot be properly choosen, raise an exception.

To be refined during implementation.

And should the version-less PURL be allowed as an input?

Maybe but that's not major though. Getting general info makes sense on a version-less purl, but fetching a version-less does not seem critical.