Skip to content

[Bug] DeepLense_Classification_Transformers: Evaluation metrics bugs — micro_auroc always NaN, missing softmax dim, hardcoded W&B entity #192

@Ganglet

Description

@Ganglet

Description

Several bugs in the evaluation pipeline of
DeepLense_Classification_Transformers_Archil_Srivastava/ cause incorrect metrics
and prevent contributors from running evaluation.

Issues Found

1. micro_auroc is always NaN (eval.py:44, eval.py:68)

micro_auroc list is declared but never appended to. np.mean([]) silently
returns nan, which gets logged to W&B as the micro_auroc metric — completely
incorrect and silent.

2. softmax called without dim= (eval.py:172)

torch.nn.functional.softmax(metrics["logits"])  # missing dim=

PyTorch raises:
UserWarning: Implicit dimension choice for softmax has been deprecated.
Line 158 already does it correctly with dim=-1. Line 172 doesn't.

3. Hardcoded W&B entity (train.py:237, eval.py:96)

entity="_archil"

Any other contributor running this will either get an auth error or accidentally
log to the original author's W&B account. Should be configurable via CLI arg or
environment variable.

Proposed Fix

  • Append micro_auroc values in the evaluation loop
  • Add dim=-1 to the softmax call on line 172
  • Replace hardcoded entity with --entity CLI argument defaulting to
    os.environ.get("WANDB_ENTITY", None)

I'll submit a PR addressing all of these.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions