Errors on loading the checkpoint for indexing the corpus

[Jun-Kai-Zhang]

Hi, I am trying to run the model with provided checkpoint. I download the PyTorch lightening checkpoint from hugginface and save them to ReProver/ckpts/leandojo-pl-ckpts, However, when I ran

python retrieval/index.py --ckpt_path ckpts/leandojo-pl-ckpts/retriever_random.ckpt --corpus-path data/leandojo_benchmark_4/corpus.jsonl --output-path indexed_corpus

I got error like this:

[2024-07-01 11:20:02,800] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The default cache directory for DeepSpeed Triton autotune, /home/zhang/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
2024-07-01 11:20:09.174 | INFO     | __main__:main:26 - Namespace(ckpt_path='/home/zhang/ReProver/ckpts/leandojo-pl-ckpts/retriever_random.ckpt', corpus_path='data/leandojo_benchmark_4/corpus.jsonl', output_path='/home/zhang/ReProver/indexed_corpus', batch_size=128)
Traceback (most recent call last):
  File "/home/zhang/ReProver/retrieval/index.py", line 45, in <module>
    main()
  File "/home/zhang/ReProver/retrieval/index.py", line 33, in main
    model = PremiseRetriever.load(args.ckpt_path, device, freeze=True)
  File "/home/zhang/ReProver/retrieval/model.py", line 51, in load
    return load_checkpoint(cls, ckpt_path, device, freeze)
  File "/home/zhang/ReProver/common.py", line 470, in load_checkpoint
    convert_zero_checkpoint_to_fp32_state_dict(ckpt_path, path)
  File "/home/zhang/miniforge3/envs/ReProver/lib/python3.10/site-packages/pytorch_lightning/utilities/deepspeed.py", line 81, in convert_zero_checkpoint_to_fp32_state_dict
    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
  File "/home/zhang/miniforge3/envs/ReProver/lib/python3.10/site-packages/deepspeed/utils/zero_to_fp32.py", line 519, in get_fp32_state_dict_from_zero_checkpoint
    raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
FileNotFoundError: Directory '/home/zhang/ReProver/ckpts/leandojo-pl-ckpts/retriever_random.ckpt/version https://git-lfs.github.com/spec/v1
oid sha256:47320987f9a49d5b00119b960f247a956773f57543982b8bfcb6da5bb3afd9ef
size 10' doesn't exist

I would really appreciate if you can share your thoughts on possible solutions!!!

The packages' version are:

Package                   Version
------------------------- ---------------
absl-py                   2.1.0
aiohttp                   3.9.5
aiohttp-cors              0.7.0
aiosignal                 1.3.1
annotated-types           0.7.0
antlr4-python3-runtime    4.9.3
anyio                     4.4.0
asttokens                 2.4.1
async-timeout             4.0.3
attrs                     23.2.0
bitsandbytes              0.43.1
cachetools                5.3.3
certifi                   2024.6.2
cffi                      1.16.0
charset-normalizer        3.3.2
click                     8.1.7
colorful                  0.5.6
contourpy                 1.2.1
cryptography              42.0.8
cycler                    0.12.1
decorator                 5.1.1
deepspeed                 0.14.4
Deprecated                1.2.14
distlib                   0.3.8
distro                    1.9.0
docstring_parser          0.16
exceptiongroup            1.2.0
executing                 2.0.1
filelock                  3.13.1
fonttools                 4.53.0
frozenlist                1.4.1
fsspec                    2024.2.0
google-api-core           2.19.1
google-auth               2.30.0
googleapis-common-protos  1.63.2
grpcio                    1.64.1
h11                       0.14.0
hjson                     3.1.0
httpcore                  1.0.5
httpx                     0.27.0
huggingface-hub           0.23.4
hydra-core                1.3.2
idna                      3.7
importlib_resources       6.4.0
ipython                   8.25.0
jedi                      0.19.1
Jinja2                    3.1.3
jsonargparse              4.31.0
jsonschema                4.22.0
jsonschema-specifications 2023.12.1
kiwisolver                1.4.5
lean-dojo                 1.9.0
lightning-utilities       0.11.3.post0
linkify-it-py             2.0.3
loguru                    0.7.2
lxml                      5.2.2
Markdown                  3.6
markdown-it-py            3.0.0
MarkupSafe                2.1.5
matplotlib                3.9.0
matplotlib-inline         0.1.7
mdit-py-plugins           0.4.1
mdurl                     0.1.2
memray                    1.13.2
mpmath                    1.3.0
msgpack                   1.0.8
multidict                 6.0.5
networkx                  3.2.1
ninja                     1.11.1.1
numpy                     1.26.4
nvidia-cublas-cu12        12.1.3.1
nvidia-cuda-cupti-cu12    12.1.105
nvidia-cuda-nvrtc-cu12    12.1.105
nvidia-cuda-runtime-cu12  12.1.105
nvidia-cudnn-cu12         8.9.2.26
nvidia-cufft-cu12         11.0.2.54
nvidia-curand-cu12        10.3.2.106
nvidia-cusolver-cu12      11.4.5.107
nvidia-cusparse-cu12      12.1.0.106
nvidia-ml-py              12.555.43
nvidia-nccl-cu12          2.20.5
nvidia-nvjitlink-cu12     12.1.105
nvidia-nvtx-cu12          12.1.105
omegaconf                 2.3.0
openai                    1.35.7
opencensus                0.11.4
opencensus-context        0.1.3
packaging                 24.1
parso                     0.8.4
pexpect                   4.9.0
pickleshare               0.7.5
pillow                    10.3.0
pip                       24.0
platformdirs              4.2.2
prometheus_client         0.20.0
prompt_toolkit            3.0.46
proto-plus                1.24.0
protobuf                  4.25.3
psutil                    6.0.0
ptyprocess                0.7.0
pure-eval                 0.2.2
py-cpuinfo                9.0.0
py-spy                    0.3.14
pyasn1                    0.6.0
pyasn1_modules            0.4.0
pycparser                 2.22
pydantic                  2.7.4
pydantic_core             2.18.4
PyGithub                  2.3.0
Pygments                  2.18.0
PyJWT                     2.8.0
PyNaCl                    1.5.0
pyparsing                 3.1.2
python-dateutil           2.9.0.post0
python-dotenv             1.0.1
pytorch-lightning         2.3.1
PyYAML                    6.0.1
rank-bm25                 0.2.2
ray                       2.31.0
referencing               0.35.1
regex                     2024.5.15
requests                  2.32.3
rich                      13.7.1
rpds-py                   0.18.1
rsa                       4.9
safetensors               0.4.3
setuptools                70.0.0
six                       1.16.0
smart-open                7.0.4
sniffio                   1.3.1
stack-data                0.6.2
sympy                     1.12
tensorboard               2.17.0
tensorboard-data-server   0.7.2
tensorboardX              2.6.2.2
textual                   0.71.0
tokenizers                0.19.1
toml                      0.10.2
torch                     2.3.1+cu121
torchmetrics              1.4.0.post0
tqdm                      4.66.4
traitlets                 5.14.3
transformers              4.42.3
triton                    2.3.1
types-psutil              6.0.0.20240621
types-toml                0.10.8.20240310
typeshed_client           2.5.1
typing_extensions         4.12.1
uc-micro-py               1.0.3
urllib3                   2.2.2
virtualenv                20.26.3
wcwidth                   0.2.13
Werkzeug                  3.0.3
wheel                     0.43.0
wrapt                     1.16.0
yarl                      1.9.4

[Jun-Kai-Zhang]

I have solved this problem. If someone encounter the same problem as mine, please try changing the content in the file 'latest' to 'checkpoint'.

[yangky11]

It looks like PyTorch Lightning or DeepSpeed has changed their checkpoint format recently, and some old checkpoints are no longer loadable by the new version. I'm trying to re-train all models, which will finish in the next week or so.

[amit9oct]

Can you tell the version of PyTorch Lightning and DeepSpeed which was used in the older version?

[yangky11]

I just updated the evaluation script to use Hugging Face checkpoints instead of PyTorch Lightning checkpoints, which hopefully can eliminate this issue. Please follow the instructions in the new README. Feel free to re-open it if you have other questions.