NeuronsPlusPlus

Reconstruct visual content (images and videos) from human fMRI signals.

Pipeline: fMRI voxels → CLIP-aligned visual/text features → diffusion conditioning → reconstructed image/video. Image work uses the NSD dataset; video work uses CC2017 (and optionally HCP).

Setup

conda activate neurons_v2
pip install -r requirements.txt
cd generative_models && pip install . && cd ..   # SGM fork (SDXL/SVD)

The two diffusers versions conflict and share one env — the run scripts pin/swap the version at runtime:

• diffusers==0.23.0 — training + image reconstruction + unCLIP/SGM
• diffusers==0.11.0 — video synthesis (AnimateDiff stage only)

HF runs offline (HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1); weights must be pre-cached under pretrained_weights/.

Running

Stage-runner shell scripts drive everything. stage is a substring match, so digits combine (12 runs stages 1+2).

bash run_neuronsv2_image.sh <exp> <stage> <mode> <subj>
bash run_neuronsv2_video.sh <exp> <stage> <mode> <subj> [dataset]   # dataset: cc2017 (default) | hcp

For any run that includes a training stage, exp must be literally image or video (the training stages invoke train_${exp}.py). If mode contains enhance, recon/video stages switch to their *_enhance variant.

Image stages (run_neuronsv2_image.sh):

stage	action	script
1	multi-subject train	train_image.py --multi_subject
2	single-subject fine-tune	train_image.py
3	reconstruction	eval/recon_image.py (or _enhance)
4	metrics	eval/run_metrics_image.py
5	caption	eval/caption_image.py
6	img2img refine	eval/recon_i2i_refine.py
7	metrics on refined	eval/run_metrics_image.py --i2i_refine

Video stages (run_neuronsv2_video.sh):

stage	action	script	diffusers
1	backbone train	train_video.py	0.23.0
2	keyframe reconstruction	eval/recon_keyframe_video.py (or _enhance)	0.23.0
3	caption	eval/caption_image.py	0.23.0
4	video synthesis (AnimateDiff)	eval/neuroclips_video.py (or _enhance)	0.11.0
5	metrics	eval/run_metrics_video.py	—

Layout

train_image.py / train_video.py   # training entry points
utils.py                           # shared helpers + per-dataset class dicts
model_variants/                    # Neurons model + video decoder
autoencoder/                       # convnext backbone
modeling_git.py                    # GIT captioning model
eval/                              # reconstruction, captioning, metrics
generative_models/                # Stability AI SGM fork (SDXL/SD-Turbo/SVD)
animatediff/                       # AnimateDiff + datasets (NSD / CC2017)
configs/                           # OMEGACONF YAML (training / inference / prompts)
tasks_construction/                # dataset download, HDF5 conversion, CLIP embeds, captioning
Data_preprocess/                   # CC2017 + HCP_v3 preprocessing steps

Notes

• Cluster AFS paths are hardcoded throughout (/mnt/afs/...). Local runs require editing these.
• Outputs land in EXP/exp_<...>/subj_<subj>/inference_results-<mode>/ (gitignored).
• Distributed: HF accelerate for training, torchrun for recon.
• Subject IDs are restricted to {1, 2, 5, 7} in the training argparsers.