Migrate EPSS importer for advisory V2

[ziadhany]

Issue:

• VCIO-API-V2: Migrate EPSS importer for advisory V2 format handling #2042

[ziadhany]

EPSSImporterPipeline Logs:

Importing data using epss_importer_v2
INFO 2025-12-24 10:04:07.326175 UTC Pipeline [EPSSImporterPipeline] starting
INFO 2025-12-24 10:04:07.326303 UTC Step [fetch_db] starting
INFO Fetching EPSS database from https://epss.cyentia.com/epss_scores-current.csv.gz
INFO 2025-12-24 10:04:10.149981 UTC Step [fetch_db] completed in 3 seconds
INFO 2025-12-24 10:04:10.150150 UTC Step [collect_and_store_advisories] starting
INFO 2025-12-24 10:04:10.150200 UTC Collecting 307,389 advisories
INFO 2025-12-24 10:11:42.582399 UTC Progress: 10% (30739/307389) ETA: 4072 seconds (1.1 hours)
INFO 2025-12-24 10:19:01.305331 UTC Progress: 20% (61478/307389) ETA: 3565 seconds (59.4 minutes)
INFO 2025-12-24 10:26:27.749857 UTC Progress: 30% (92217/307389) ETA: 3121 seconds (52.0 minutes)
INFO 2025-12-24 10:33:54.212909 UTC Progress: 40% (122956/307389) ETA: 2676 seconds (44.6 minutes)
INFO 2025-12-24 10:41:16.240522 UTC Progress: 50% (153695/307389) ETA: 2226 seconds (37.1 minutes)
INFO 2025-12-24 10:49:15.940224 UTC Progress: 60% (184434/307389) ETA: 1804 seconds (30.1 minutes)
INFO 2025-12-24 10:57:16.890067 UTC Progress: 70% (215173/307389) ETA: 1366 seconds (22.8 minutes)
INFO 2025-12-24 11:05:29.408529 UTC Progress: 80% (245912/307389) ETA: 920 seconds (15.3 minutes)
INFO 2025-12-24 11:13:47.299311 UTC Progress: 90% (276651/307389) ETA: 464 seconds (7.7 minutes)
INFO 2025-12-24 11:22:24.677970 UTC Successfully collected 307,387 advisories
INFO 2025-12-24 11:22:24.678160 UTC Step [collect_and_store_advisories] completed in 4695 seconds (1.3 hours)
INFO 2025-12-24 11:22:24.678262 UTC Pipeline completed in 4697 seconds (1.3 hours)

[TG1999]

@ziadhany thanks! Can you do a quick check to see if two advisories does not share same AVID ? Please run the query and share the results. Also do it for the next set of importers that you are gonna do as well. Thanks!

[ziadhany]

I ran the epss twice and this is the result in my local database:

>>  from vulnerabilities.models import AdvisoryV2
>>  from django.db.models import Count
>> duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
>> print(len(duplicates))
332422

[TG1999]

I ran the epss twice and this is the result in my local database:

>>  from vulnerabilities.models import AdvisoryV2

>>  from django.db.models import Count

>> duplicates = (

    AdvisoryV2.objects

    .values('avid')

    .annotate(count=Count('id'))

    .filter(count__gt=1)

)

>> print(len(duplicates))

332422

Clear the DB, run it once. And then let me know. Does it have any dupe AVIDs. Thanks!

[ziadhany]

There is no duplication in EPSS Importer v2 when the importer is run once.

>> from vulnerabilities.models import AdvisoryV2
>> from django.db.models import Count
>> duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
>> duplicates
Out[10]: <AdvisoryQuerySet []>
>> len(duplicates)
Out[11]: 0

>> AdvisoryV2.objects.count()
Out[21]: 307387

Logs:

Importing data using epss_importer_v2
INFO 2025-12-24 13:26:19.085930 UTC Pipeline [EPSSImporterPipeline] starting
INFO 2025-12-24 13:26:19.086081 UTC Step [fetch_db] starting
INFO Fetching EPSS database from https://epss.cyentia.com/epss_scores-current.csv.gz
INFO 2025-12-24 13:26:21.094138 UTC Step [fetch_db] completed in 2 seconds
INFO 2025-12-24 13:26:21.094349 UTC Step [collect_and_store_advisories] starting
INFO 2025-12-24 13:26:21.094429 UTC Collecting 307,389 advisories
INFO 2025-12-24 13:30:09.905309 UTC Progress: 10% (30739/307389) ETA: 2059 seconds (34.3 minutes)
INFO 2025-12-24 13:34:43.573540 UTC Progress: 20% (61478/307389) ETA: 2010 seconds (33.5 minutes)
INFO 2025-12-24 13:39:36.892927 UTC Progress: 30% (92217/307389) ETA: 1857 seconds (30.9 minutes)
INFO 2025-12-24 13:44:31.029478 UTC Progress: 40% (122956/307389) ETA: 1635 seconds (27.2 minutes)
INFO 2025-12-24 13:49:17.964310 UTC Progress: 50% (153695/307389) ETA: 1377 seconds (22.9 minutes)
INFO 2025-12-24 13:54:03.047150 UTC Progress: 60% (184434/307389) ETA: 1108 seconds (18.5 minutes)
INFO 2025-12-24 13:58:50.244036 UTC Progress: 70% (215173/307389) ETA: 835 seconds (13.9 minutes)
INFO 2025-12-24 14:03:40.148023 UTC Progress: 80% (245912/307389) ETA: 560 seconds (9.3 minutes)
INFO 2025-12-24 14:08:29.004852 UTC Progress: 90% (276651/307389) ETA: 281 seconds (4.7 minutes)
INFO 2025-12-24 14:13:17.860258 UTC Successfully collected 307,387 advisories
INFO 2025-12-24 14:13:17.860431 UTC Step [collect_and_store_advisories] completed in 2817 seconds (46.9 minutes)
INFO 2025-12-24 14:13:17.860510 UTC Pipeline completed in 2819 seconds (47.0 minutes)