# Lux TDDFT Santander Field v0.2.0 Runbook

This directory is the active 1,000-job production campaign for Lux-owned
TDDFT UV/Vis priors for Santander, Colombia field spectroscopy. It is not a
published release yet. Do not publish this dataset on `data.luxnode.org` until
`tools/lux_tddft_campaign.py merge --min-records 1000` succeeds.

## Authoritative Files

- `molecules.csv`: curated Santander field-work candidates resolved through
  PubChem/RDKit.
- `campaign_jobs.jsonl`: the 1,000-job TDDFT queue with frozen geometries and
  six method profiles.
- `results/*.json`: one completed spectrum per successful job.
- `failures/*.json`: resumable failure records, if any.
- `locks/*.lock`: live or reserved work items for Mac and DGX workers.
- `remote_batches/*.jsonl`: DGX batch reservation files.
- `legacy_screening_release_20260601T1646Z/`: preserved older
  1,000-record screening/TDA artifact. It is not the active six-method
  PySCF/GPU4PySCF campaign release.

## Current State

The campaign has 570 locally validated completed records as of the latest sync
and is still running. Use the status command below for the exact live number.

Active worker layout:

- Local Mac: three fast lock-aware workers excluding M06-2X.
- Local Mac: one slower M06-2X worker.
- Local supervisor: `luxdata-tddft-supervisor`, target `3 fast + 1 M06`.
- DGX Spark: fourteen CPU-only v0.2 workers running small reserved batches while
  the separate refined v0.3 GPU job occupies the GB10.

Planned queue summary:

- Jobs: 1,000.
- Unique molecules in queue: 134.
- Method balance: B3LYP/6-31G* 167, PBE0/6-31G* 167,
  CAM-B3LYP/6-31G* 167, B3LYP/def2-SVP 167, wB97X/6-31G* 166,
  M06-2X/6-31G* 166.
- Major domains: agrochemicals, soil organic acids, plant pigments,
  petroleum NSO heterocycles, PAHs/oil/coal, lignin proxies, cacao/coffee
  markers, metal ligands, BTEX, mining ligands, sulfur ligands, and
  plant-water chromophores.

## Local Commands

From `/Users/luismantilla/Desktop/LuxNode/LuxData`:

```bash
.venv/bin/python tools/lux_tddft_campaign.py status \
  --jobs datasets/lux-tddft-santander-field/v0.2.0/campaign_jobs.jsonl \
  --outdir datasets/lux-tddft-santander-field/v0.2.0
```

```bash
.venv/bin/python tools/lux_tddft_tmux.py status \
  --root /Users/luismantilla/Desktop/LuxNode/LuxData
```

```bash
.venv/bin/python tools/lux_tddft_tmux.py supervise \
  --root /Users/luismantilla/Desktop/LuxNode/LuxData \
  --fast-workers 3 --m06-workers 1 --threads 1 --max-memory-mb 6000 \
  --interval-sec 300
```

If tmux dies and leaves stale locks, remove only locks whose local PID is dead
and whose job has no result/failure file, then relaunch a smaller worker pool.

## DGX Commands

Check remote sessions and Lux jobs:

```bash
.venv/bin/python tools/lux_tddft_dgx.py status
```

Only reserve a new v0.2 DGX GPU batch when `status` shows no competing
`run_gpu4pyscf`/`lux_tddft_campaign` process on the remote GPU. The helper now
enforces this by default for `--engine gpu`: `reserve` and `launch` refuse to
proceed while a remote Lux/GPU4PySCF process is active unless
`--allow-busy-gpu` is explicitly passed.

CPU-only DGX batches are allowed while the GPU lane is active. Keep them small,
single-threaded, and light-element/organic so they do not starve GPU4PySCF:

```bash
.venv/bin/python tools/lux_tddft_dgx.py reserve \
  --root /Users/luismantilla/Desktop/LuxNode/LuxData \
  --batch-id dgx-cpu-v02-YYYYMMDDTHHMMSSZ --limit 8 --min-index 160 \
  --max-heavy-atoms 18 --engine cpu --exclude-m06
```

```bash
.venv/bin/python tools/lux_tddft_dgx.py launch \
  --batch-id dgx-cpu-v02-YYYYMMDDTHHMMSSZ \
  --session lux-tddft-dgx-cpu-v02-YYYYMMDDTHHMMSSZ \
  --engine cpu --threads 1 --max-memory-mb 10000 \
  --grid-level 1 --job-timeout-sec 7200
```

Harvest any completed CPU batch records without slow per-missing-file rsyncs:

```bash
.venv/bin/python tools/lux_tddft_dgx.py sync-all \
  --root /Users/luismantilla/Desktop/LuxNode/LuxData \
  --prefix dgx-cpu-v02-
```

Run the full checkpoint monitor to sync DGX outputs, sync the GPU-refined lane,
validate raw records, rebuild the static catalog, deploy the public data site,
and verify `data.luxnode.org` without exposing unfinished raw campaign files:

```bash
.venv/bin/python tools/lux_tddft_monitor.py once \
  --root /Users/luismantilla/Desktop/LuxNode/LuxData
```

For unattended checkpoints:

```bash
tmux new-session -d -s lux-tddft-monitor \
  'cd /Users/luismantilla/Desktop/LuxNode/LuxData && .venv/bin/python tools/lux_tddft_monitor.py loop --interval-sec 900 --root /Users/luismantilla/Desktop/LuxNode/LuxData --target-dgx-cpu-workers 18 --top-up-max-new-workers 2 --top-up-batch-size 8 --top-up-max-heavy-atoms 18 --dgx-max-load-per-core 0.9 --dgx-min-available-mb 30000 2>&1 | tee -a datasets/lux-tddft-santander-field/v0.2.0/logs/lux-tddft-monitor.log'
```

The top-up path is intentionally CPU-only and excludes M06-2X by default. It
skips launches when the DGX already has the target number of v0.2 CPU sessions,
when one-minute load is above `nproc * 0.9`, or when available memory is below
30 GB. Monitor output also includes `gpu_next_task`, which is the first
GPU-refined v0.3 task that has not yet produced a record or failure after the
latest sync, and a `throughput` block with recent local completion counts,
estimated records/hour, and a rough ETA at the observed rate.

Summarize reserved DGX batches without transferring files:

```bash
.venv/bin/python tools/lux_tddft_dgx.py batch-status \
  --root /Users/luismantilla/Desktop/LuxNode/LuxData \
  --prefix dgx-cpu-v02-
```

GPU batches use the same helper with `--engine gpu`:

```bash
.venv/bin/python tools/lux_tddft_dgx.py reserve \
  --root /Users/luismantilla/Desktop/LuxNode/LuxData \
  --batch-id dgx-gpu-prod-001 --limit 25 --min-index 220 \
  --max-heavy-atoms 18 --exclude-m06
```

```bash
.venv/bin/python tools/lux_tddft_dgx.py launch \
  --batch-id dgx-gpu-prod-001 --threads 1 --max-memory-mb 16000 \
  --grid-level 2 --job-timeout-sec 3600
```

```bash
.venv/bin/python tools/lux_tddft_dgx.py sync \
  --root /Users/luismantilla/Desktop/LuxNode/LuxData \
  --batch-id dgx-gpu-prod-001
```

Remote status:

```bash
.venv/bin/python tools/lux_tddft_dgx.py status
ssh dgx 'tmux capture-pane -pt lux-tddft-dgx-gpu-prod-001 -S -60'
```

`dgx-gpu-prod-001` was attempted once, but the remote tmux session died before
writing either a result or a failure for the first reserved molecule. Its 25
local reservation locks were released so the active Mac pool can proceed and a
future DGX batch can be chosen more conservatively.

## DGX Setup Notes

The reachable DGX Spark host is `dgx` / `spark-a846`.

- CUDA: `/usr/local/cuda-13.0`.
- GPU: NVIDIA GB10, compute capability 12.1.
- Python venv: `~/.venvs/lux-tddft`.
- Installed: PySCF 2.13.0, CuPy CUDA 13, GPU4PySCF 1.7.1.

GPU4PySCF was installed from upstream source, not the PyPI source tarball. The
PyPI tarball declared CUDA 13 metadata but lacked the expected CMake source
tree. The upstream checkout was patched locally on the DGX to build gas-phase
TDDFT support without the SMD solvent submodule:

- top-level CMake language list changed from `C CXX CUDA Fortran` to
  `C CXX CUDA`;
- build flags include `-DBUILD_SOLVENT=OFF`;
- build flags include `-DBUILD_LIBXC=ON`;
- build flags include `-DCUDA_ARCHITECTURES=121-real`.

Validated GPU smoke tests:

- Water B3LYP/6-31G* GPU SCF converged in 8.23 s; 4-state TDDFT completed in
  15.54 s.
- Lux runner job `pyridine-c0-wb97x_6-31gstar_tddft` completed with
  `--engine gpu` in 13.8 s and was imported into local `results/` with
  `engine=GPU4PySCF`, `gpu4pyscf_version=1.7.1`, and `host=spark-a846`.

DGX CPU is slower per job than the local Mac for very small molecules, but it is
useful for extra single-threaded throughput while the GB10 GPU is reserved for
the refined v0.3 lane. Use conservative CPU batches, sync them often, and stop
adding workers if the v0.3 GPU process slows or memory pressure rises.

## Release Gate

Merge only when the target release is ready:

```bash
.venv/bin/python tools/lux_tddft_campaign.py merge \
  --outdir datasets/lux-tddft-santander-field/v0.2.0 \
  --min-records 1000
```

Expected behavior before completion: this command must fail with
`Only N completed records; need 1000`.
